angsd - User contributions [en]

Abbababa2

2017-03-07T14:55:47Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 1=use the last group of bam files as outgroup

</pre>
This function will counts the number of ABBA and BABA sites of the 4-population trees that can be built from the data, where the outgroup is fixed.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). Columns 4 and 5 are the numerator and denominator of the D-statistic for their specific block. Column 6 is the number of sites containing data in that block. The other 256 columns are the normalized counts of the 256 allele patterns between the 4 populations, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the trees that are built.
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove ancient transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -useLast [int]
1: use the last group of bam files as outgroup for the D-stat analysys. Default: 0 (use the fasta file as outgroup)
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads.
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts. Default 100.
; -sizeFile [fileName]
file that specifies number of individuals in each population (more than 4 populations can be defined). If not provided, it is assumed that each population has only one individual.
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
This tutorial require having Samtools previously installed, and the library 'pracma' previously installed in R.
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R DSTAT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test.
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -doCounts 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

==Generate files for the error correction==
We will apply error correction to the group with 3 individuals, using "perfectSampleCEU" as high-quality reference genome.
The population containing 3 individuals affected by transition error goes from line 6 to line 8 in the file bam.filelist. We select those individuals and write them in another file.
<pre>
sed -n 6,8p bam.filelist > bamWithErrors.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later as input for the R script that calculates the D-statistic. "doAncError" applies the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out errorFile -bam bamWithErrors.filelist
</pre>

==4-population test==
In this tutorial we perform the ABBABABA test on all the combinations of 4 populations amongst 6 populations of size 1,2,2,3,2,1 individuals, where the last population is fixed as outgroup (so that there are 30 possible combinations). The last population is represented by the fasta file defined with the option -anc, of which we enable the use as outgroup by the option -useLast 0. One can use the last population of .bam files as outgroup with the option -useLast 1. Create a file named sizeFile.size and write the size of each population (IT IS NECESSARY to define the size of the -anc outgroup population, that is always 1):
<pre>
1
2
2
3
2
1
</pre>
We decide to target three chromosomes, one of the three with loci between position 10Mb and 15Mb. Thus create a file called regions.txt in which is written
<pre>
1:
5:
16:10000000-15000000
</pre>
The output of ANGSD will show no data about chromosome 1. This happens when all blocks within that chromosome contained no data and therefore where not printed.

After running ANGSD to count ABBA and BABA combinations, we will call the R script who applies error correction to the ABBA and BABA allele combinations and produces the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -sizeFile sizeFile.size -doCounts 1 -out bam.Angsd -anc chimpHg19.fa -rf regions.txt -useLast 0 -minQ 20 -minMapQ 30 -p 1
</pre>
[[The output file is]]
[[bam.Angsd.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1) for one of the possible 30 trees (so each block is written on 30 lines), a start position (Column 2), an end postion (Column 3). Columns 4,5 and 6 are the numerator, denominator and number of sites analyzed in the block. The next 256 columns are the counted patterns of alleles in the tree, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the error files for the population with 3 individuals. This is done defining the error files in each populations inside a text file (including a line for the outgroup population). If a population has no error file, it is sufficient to write NA. Create a file called errorList.error with written
<pre>
NA
NA
NA
./errorFile.ancError
NA
NA
</pre>
Create a file popNames.name with written
<pre>
Population1
Population2
Population3
PopWithError
Population4
Chimpanzee
</pre>
Run the Rscript with the command
<pre>
Rscript DSTAT angsdFile="bam.Angsd" out="result" sizeFile=sizeFile.size errFile=errorList.error nameFile=popNames.name
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities for all 30 4-populations trees. Note: If error correction is not needed, it is sufficient to avoid specifying any error file. If no names need to be provided, the script will assign Population_* as standard name. If no size file is provided, the script assigns 1 to each population. At least one between the name file and the size file is needed. It is possible to recycle the size file used in ANGSD.

The D-statistics and other informations are contained in four distinct files depending on the application of error correction and ancient transition removal. The files are named as follow:
;[[1)result.Observed.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
;[[2) result.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
;[[3) result.ErrorCorr.TransRem.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
;[[4) result.TransRem.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal

Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBlocks=number of blocks with observed data, H*=the names of the four populations for the specific tree. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

Abbababa2

2017-02-28T17:51:27Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 1=use the last group of bam files as outgroup

</pre>
This function will counts the number of ABBA and BABA sites of the 4-population trees that can be built from the data, where the outgroup is fixed.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). Columns 4 and 5 are the numerator and denominator of the D-statistic for their specific block. Column 6 is the number of sites containing data in that block. The other 256 columns are the normalized counts of the 256 allele patterns between the 4 populations, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the trees that are built.
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove ancient transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -useLast [int]
1: use the last group of bam files as outgroup for the D-stat analysys. Default: 0 (use the fasta file as outgroup)
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads.
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts. Default 100.
; -sizeFile [fileName]
file that specifies number of individuals in each population (more than 4 populations can be defined). If not provided, it is assumed that each population has only one individual.
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
This tutorial require having Samtools previously installed, and the library 'pracma' previously installed in R.
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R DSTAT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test.
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -doCounts 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

==Generate files for the error correction==
We will apply error correction to the group with 3 individuals, using "perfectSampleCEU" as high-quality reference genome.
The population containing 3 individuals affected by transition error goes from line 6 to line 8 in the file bam.filelist. We select those individuals and write them in another file.
<pre>
sed -n 6,8p bam.filelist > bamWithErrors.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later as input for the R script that calculates the D-statistic. "doAncError" applies the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out errorFile -bam bamWithErrors.filelist
</pre>

==4-population test==
In this tutorial we perform the ABBABABA test on all the combinations of 4 populations amongst 6 populations of size 1,2,2,3,2,1 individuals, where the last population is fixed as outgroup (so that there are 30 possible combinations). The last population is represented by the fasta file defined with the option -anc, of which we enable the use as outgroup by the option -useLast 0. One can use the last population of .bam files as outgroup with the option -useLast 1. Create a file named sizeFile.size and write the size of each population (IT IS NECESSARY to define the size of the -anc outgroup population, that is always 1):
<pre>
1
2
2
3
2
1
</pre>
We decide to target three chromosomes, one of the three with loci between position 10Mb and 15Mb. Thus create a file called regions.txt in which is written
<pre>
1:
5:
16:10000000-15000000
</pre>
The output of ANGSD will show no data about chromosome 1. This happens when all blocks within that chromosome contained no data and therefore where not printed.

After running ANGSD to count ABBA and BABA combinations, we will call the R script who applies error correction to the ABBA and BABA allele combinations and produces the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -sizeFile sizeFile.size -doCounts 1 -out bam.Angsd -anc chimpHg19.fa -rf regions.txt -useLast 0 -minQ 20 -minMapQ 30 -p 1
</pre>
[[The output file is]]
[[bam.Angsd.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1) for one of the possible 30 trees (so each block is written on 30 lines), a start position (Column 2), an end postion (Column 3). Columns 4,5 and 6 are the numerator, denominator and number of sites analyzed in the block. The next 256 columns are the counted patterns of alleles in the tree, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the error files for the population with 3 individuals. This is done defining the error files in each populations inside a text file (including a line for the outgroup population). If a population has no error file, it is sufficient to write NA. Create a file called errorList.error with written
<pre>
NA
NA
NA
./errorFile.ancError
NA
NA
</pre>
We also want to study the effect of error correction if we add individually to each population an error rate between -0.005 and 0.005 of steps 0.001 and involving transition A->T. You can run more transitions separating the letters with a comma. We need to create a file with the names of the 6 populations. Create a file popNames.name with written
<pre>
Population1
Population2
Population3
PopWithError
Population4
Chimpanzee
</pre>
Run the Rscript with the command
<pre>
Rscript DSTAT angsdFile="bam.Angsd" out="result" sizeFile=sizeFile.size errFile=errorList.error nameFile=popNames.name addErr="-0.005,0.005,0.001;A;T"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities for all 30 4-populations trees. Note: If error correction is not needed, it is sufficient to avoid specifying any error file. If no names need to be provided, the script will assign Population_* as standard name. If no size file is provided, the script assigns 1 to each population. At least one between the name file and the size file is needed. It is possible to recycle the size file used in ANGSD.

The plots of error rates for type specific errors and added errors are in the folder result.errorDataFolder as pdf files (only if you have provided an error rate file and added error rates in the options for the R script). The other text files are used by the Rscript for the necessary computations and do not contain useful information.
The D-statistics are contained in four distinct files for each combination of populations. For example, for the tree (((Population1, Population2)PopWithError)Chimpanzee) the files containing the results are:
;[[1)result.Population1.Population2.PopWithError.Chimpanzee.Observed.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
;[[2) result.Population1.Population2.PopWithError.Chimpanzee.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
;[[3) result.Population1.Population2.PopWithError.Chimpanzee.TransRem.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
;[[4) result.Population1.Population2.PopWithError.Chimpanzee.TransRem.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal

Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, the names of the four populations. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.Population1.Population2.PopWithError.Chimpanzee.pdf showing a barplot of the error rates
;-the file plotAddErr.A2T.Population1.Population2.PopWithError.Chimpanzee.pdf showing the effect of error correction on transition A-->T

Abbababa2

2017-02-28T17:49:40Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 1=use the last group of bam files as outgroup

</pre>
This function will counts the number of ABBA and BABA sites of the 4-population trees that can be built from the data, where the outgroup is fixed.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). Columns 4 and 5 are the numerator and denominator of the D-statistic for their specific block. Column 6 is the number of sites containing data in that block. The other 256 columns are the normalized counts of the 256 allele patterns between the 4 populations, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the trees that are built.
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove ancient transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -useLast [int]
1: use the last group of bam files as outgroup for the D-stat analysys. Default: 0 (use the fasta file as outgroup)
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads.
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts. Default 100.
; -sizeFile [fileName]
file that specifies number of individuals in each population (more than 4 populations can be defined). If not provided, it is assumed that each population has only one individual.
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
This tutorial require having Samtools previously installed, and the library 'pracma' previously installed in R.
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R DSTAT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test.
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -doCounts 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

==Generate files for the error correction==
We will apply error correction to the group with 3 individuals, using "perfectSampleCEU" as high-quality reference genome.
The population containing 3 individuals affected by transition error goes from line 6 to line 8 in the file bam.filelist. We select those individuals and write them in another file.
<pre>
sed -n 6,8p bam.filelist > bamWithErrors.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later as input for the R script that calculates the D-statistic. "doAncError" applies the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out errorFile -bam bamWithErrors.filelist
</pre>

==4-population test==
In this tutorial we perform the ABBABABA test on all the combinations of 4 populations amongst 6 populations of size 1,2,2,3,2,1 individuals, where the last population is fixed as outgroup (so that there are 30 possible combinations). The last population is represented by the fasta file defined with the option -anc, of which we enable the use as outgroup by the option -useLast 0. One can use the last population of .bam files as outgroup with the option -useLast 1. Create a file named sizeFile.size and write the size of each population (IT IS NECESSARY to define the size of the -anc outgroup population, that is always 1):
<pre>
1
2
2
3
2
1
</pre>
We decide to target three chromosomes, one of the three with loci between position 10Mb and 15Mb. Thus create a file called regions.txt in which is written
<pre>
1:
5:
16:10000000-15000000
</pre>
After running ANGSD to count ABBA and BABA combinations, we will call the R script who applies error correction to the ABBA and BABA allele combinations and produces the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -sizeFile sizeFile.size -doCounts 1 -out bam.Angsd -anc chimpHg19.fa -rf regions.txt -useLast 0 -minQ 20 -minMapQ 30 -p 1
</pre>
[[The output file is]]
[[bam.Angsd.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1) for one of the possible 30 trees (so each block is written on 30 lines), a start position (Column 2), an end postion (Column 3). Columns 4,5 and 6 are the numerator, denominator and number of sites analyzed in the block. The next 256 columns are the counted patterns of alleles in the tree, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the error files for the population with 3 individuals. This is done defining the error files in each populations inside a text file (including a line for the outgroup population). If a population has no error file, it is sufficient to write NA. Create a file called errorList.error with written
<pre>
NA
NA
NA
./errorFile.ancError
NA
NA
</pre>
We also want to study the effect of error correction if we add individually to each population an error rate between -0.005 and 0.005 of steps 0.001 and involving transition A->T. You can run more transitions separating the letters with a comma. We need to create a file with the names of the 6 populations. Create a file popNames.name with written
<pre>
Population1
Population2
Population3
PopWithError
Population4
Chimpanzee
</pre>
Run the Rscript with the command
<pre>
Rscript DSTAT angsdFile="bam.Angsd" out="result" sizeFile=sizeFile.size errFile=errorList.error nameFile=popNames.name addErr="-0.005,0.005,0.001;A;T"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities for all 30 4-populations trees. Note: If error correction is not needed, it is sufficient to avoid specifying any error file. If no names need to be provided, the script will assign Population_* as standard name. If no size file is provided, the script assigns 1 to each population. At least one between the name file and the size file is needed. It is possible to recycle the size file used in ANGSD.

The plots of error rates for type specific errors and added errors are in the folder result.errorDataFolder as pdf files (only if you have provided an error rate file and added error rates in the options for the R script). The other text files are used by the Rscript for the necessary computations and do not contain useful information.
The D-statistics are contained in four distinct files for each combination of populations. For example, for the tree (((Population1, Population2)PopWithError)Chimpanzee) the files containing the results are:
;[[1)result.Population1.Population2.PopWithError.Chimpanzee.Observed.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
;[[2) result.Population1.Population2.PopWithError.Chimpanzee.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
;[[3) result.Population1.Population2.PopWithError.Chimpanzee.TransRem.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
;[[4) result.Population1.Population2.PopWithError.Chimpanzee.TransRem.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal

Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, the names of the four populations. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.Population1.Population2.PopWithError.Chimpanzee.pdf showing a barplot of the error rates
;-the file plotAddErr.A2T.Population1.Population2.PopWithError.Chimpanzee.pdf showing the effect of error correction on transition A-->T

Abbababa2

2017-02-28T17:09:25Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 1=use the last group of bam files as outgroup

</pre>
This function will counts the number of ABBA and BABA sites of the 4-population trees that can be built from the data, where the outgroup is fixed.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). Columns 4 and 5 are the numerator and denominator of the D-statistic for their specific block. Column 6 is the number of sites containing data in that block. The other 256 columns are the normalized counts of the 256 allele patterns between the 4 populations, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the trees that are built.
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove ancient transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -useLast [int]
1: use the last group of bam files as outgroup for the D-stat analysys. Default: 0 (use the fasta file as outgroup)
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads.
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts. Default 100.
; -sizeFile [fileName]
file that specifies number of individuals in each population (more than 4 populations can be defined). If not provided, it is assumed that each population has only one individual.
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
This tutorial require having Samtools previously installed, and the library 'pracma' previously installed in R.
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R DSTAT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test.
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -doCounts 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

==Generate files for the error correction==
We will apply error correction to the group with 3 individuals, using "perfectSampleCEU" as high-quality reference genome.
The population containing 3 individuals affected by transition error goes from line 6 to line 8 in the file bam.filelist. We select those individuals and write them in another file.
<pre>
sed -n 6,8p bam.filelist > bamWithErrors.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later as input for the R script that calculates the D-statistic. "doAncError" applies the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out errorFile -bam bamWithErrors.filelist
</pre>

==4-population test==
In this tutorial we perform the ABBABABA test on all the combinations of 4 populations amongst 6 populations of size 1,2,2,3,2,1 individuals, where the last population is fixed as outgroup (so that there are 30 possible combinations). The last population is represented by the fasta file defined with the option -anc, of which we enable the use as outgroup by the option -useLast 0. One can use the last population of .bam files as outgroup with the option -useLast 1. Create a file named sizeFile.size and write the size of each population (IT IS NECESSARY to define the size of the -anc outgroup population, that is always 1):
<pre>
1
2
2
3
2
1
</pre>
We decide to target two chromosomes, one of the two with loci between position 10Mb and 15Mb. Thus create a file called regions.txt in which is written
<pre>
1:
16:10000000-15000000
</pre>
After running ANGSD to count ABBA and BABA combinations, we will call the R script who applies error correction to the ABBA and BABA allele combinations and produces the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -sizeFile sizeFile.size -doCounts 1 -out bam.Angsd -anc chimpHg19.fa -rf regions.txt -useLast 0 -minQ 20 -minMapQ 30 -p 1
</pre>
[[The output file is]]
[[bam.Angsd.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1) for one of the possible 30 trees (so each block is written on 30 lines), a start position (Column 2), an end postion (Column 3). Columns 4,5 and 6 are the numerator, denominator and number of sites analyzed in the block. The next 256 columns are the counted patterns of alleles in the tree, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the error files for the population with 3 individuals. This is done defining the error files in each populations inside a text file (including a line for the outgroup population). If a population has no error file, it is sufficient to write NA. Create a file called errorList.error with written
<pre>
NA
NA
NA
./errorFile.ancError
NA
NA
</pre>
We also want to study the effect of error correction if we add individually to each population an error rate between -0.005 and 0.005 of steps 0.001 and involving transition A->T. You can run more transitions separating the letters with a comma. We need to create a file with the names of the 6 populations. Create a file popNames.name with written
<pre>
Population1
Population2
Population3
PopWithError
Population4
Chimpanzee
</pre>
Run the Rscript with the command
<pre>
Rscript DSTAT angsdFile="bam.Angsd" out="result" sizeFile=sizeFile.size errFile=errorList.error nameFile=popNames.name addErr="-0.005,0.005,0.001;A;T"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities for all 30 4-populations trees. Note: If error correction is not needed, it is sufficient to avoid specifying any error file. If no names need to be provided, the script will assign Population_* as standard name. If no size file is provided, the script assigns 1 to each population. At least one between the name file and the size file is needed.

The plots of error rates for type specific errors and added errors are in the folder result.errorDataFolder as pdf files (only if you have provided an error rate file and added error rates in the options for the R script). The other text files are used by the Rscript for the necessary computations and do not contain useful information.
The D-statistics are contained in four distinct files for each combination of populations. For example, for the tree (((Population1, Population2)PopWithError)Chimpanzee) the files containing the results are:
;[[1)result.Population1.Population2.PopWithError.Chimpanzee.Observed.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
;[[2) result.Population1.Population2.PopWithError.Chimpanzee.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
;[[3) result.Population1.Population2.PopWithError.Chimpanzee.TransRem.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
;[[4) result.Population1.Population2.PopWithError.Chimpanzee.TransRem.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal

Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, the names of the four populations. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.Population1.Population2.PopWithError.Chimpanzee.pdf showing a barplot of the error rates
;-the file plotAddErr.A2T.Population1.Population2.PopWithError.Chimpanzee.pdf showing the effect of error correction on transition A-->T

Abbababa2

2017-02-24T08:50:28Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 1=use the last group of bam files as outgroup

</pre>
This function will counts the number of ABBA and BABA sites of the 4-population trees that can be built from the data, where the outgroup is fixed.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). Columns 4 and 5 are the numerator and denominator of the D-statistic for their specific block. Column 6 is the number of sites containing data in that block. The other 256 columns are the normalized counts of the 256 allele patterns between the 4 populations, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the trees that are built.
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove ancient transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -useLast [int]
1: use the last group of bam files as outgroup for the D-stat analysys. Default: 0 (use the fasta file as outgroup)
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads.
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts. Default 100.
; -sizeFile [fileName]
file that specifies number of individuals in each population (more than 4 populations can be defined). If not provided, it is assumed that each population has only one individual.
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
This tutorial require having Samtools previously installed, and the library 'pracma' previously installed in R.
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R DSTAT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test.
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -doCounts 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

==Generate files for the error correction==
We will apply error correction to the group with 3 individuals, using "perfectSampleCEU" as high-quality reference genome.
The population containing 3 individuals affected by transition error goes from line 6 to line 8 in the file bam.filelist. We select those individuals and write them in another file.
<pre>
sed -n 6,8p bam.filelist > bamWithErrors.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later as input for the R script that calculates the D-statistic. "doAncError" applies the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out errorFile -bam bamWithErrors.filelist
</pre>

==4-population test==
In this tutorial we perform the ABBABABA test on all the combinations of 4 populations amongst 6 populations of size 1,2,2,3,2,1 individuals, where the last population is fixed as outgroup (so that there are 30 possible combinations). The last population is represented by the fasta file defined with the option -anc, of which we enable the use as outgroup by the option -useLast 0. One can use the last population of .bam files as outgroup with the option -useLast 1. Create a file named sizeFile.size and write the size of each population (it is NOT necessary to define the size of the -anc fasta outgroup):
<pre>
1
2
2
3
2
</pre>
We decide to target three chromosomes with loci in the range 10000000 to 15000000 for chromosomes 1,2 and 3. Thus create a file called regions.txt in which is written
<pre>
1:10000000-15000000
2:10000000-15000000
3:10000000-15000000
</pre>
After running ANGSD to count ABBA and BABA combinations, we will call the R script who applies error correction to the ABBA and BABA allele combinations and produces the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -sizeFile sizeFile.size -doCounts 1 -out bam.Angsd -anc chimpHg19.fa -rf regions.txt -useLast 0 -minQ 20 -minMapQ 30 -p 1
</pre>
[[The output file is]]
[[bam.Angsd.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1) for one of the possible 30 trees (so each block is written on 30 lines), a start position (Column 2), an end postion (Column 3). Columns 4,5 and 6 are the numerator, denominator and number of sites analyzed in the block. The next 256 columns are the counted patterns of alleles in the tree, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the error files for the population with 3 individuals. This is done defining the error files in each populations inside a text file (including a line for the outgroup population). If a population has no error file, it is sufficient to write NA. Create a file called errorList.error with written
<pre>
NA
NA
NA
./errorFile.ancError
NA
NA
</pre>
We also want to study the effect of error correction if we add individually to each population an error rate between -0.005 and 0.005 of steps 0.001 and involving transition A->T. You can run more transitions separating the letters with a comma. We need to create a file with the names of the 6 populations. Create a file popNames.name with written
<pre>
Population1
Population2
Population3
PopWithError
Population4
Chimpanzee
</pre>
Run the Rscript with the command
<pre>
Rscript DSTAT angsdFile="bam.Angsd" out="result" sizeFile=sizeFile.size errFile=errorList.error nameFile=popNames.name addErr="-0.005,0.005,0.001;A;T"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities for all 30 4-populations trees. Note: If error correction is not needed, it is sufficient to avoid specifying any error file. If no names need to be provided, the script will assign Population_* as standard name. If no size file is provided, the script assigns 1 to each population. At least one between the name file and the size file is needed.

The plots of error rates for type specific errors and added errors are in the folder result.errorDataFolder as pdf files (only if you have provided an error rate file and added error rates in the options for the R script). The other text files are used by the Rscript for the necessary computations and do not contain useful information.
The D-statistics are contained in four distinct files for each combination of populations. For example, for the tree (((Population1, Population2)PopWithError)Chimpanzee) the files containing the results are:
;[[1)result.Population1.Population2.PopWithError.Chimpanzee.Observed.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
;[[2) result.Population1.Population2.PopWithError.Chimpanzee.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
;[[3) result.Population1.Population2.PopWithError.Chimpanzee.TransRem.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
;[[4) result.Population1.Population2.PopWithError.Chimpanzee.TransRem.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal

Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, the names of the four populations. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.Population1.Population2.PopWithError.Chimpanzee.pdf showing a barplot of the error rates
;-the file plotAddErr.A2T.Population1.Population2.PopWithError.Chimpanzee.pdf showing the effect of error correction on transition A-->T

Abbababa2

2017-02-15T11:03:35Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 set fasta file defined by -anc as outgroup in the analysis

</pre>
This function will counts the number of ABBA and BABA sites of the possible 4-population trees that can be built from the data.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the combinations of 4 populations that are possible to create from the data (with one population fixed as outgroup)
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove ancient transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -useLast [int]
1: use the file defined by -anc as outgroup of the D-stat analysys. Default: 0 (use last population as a fixed outgroup)
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads.
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts. Default 100.
; -sizeFile [fileName]
file that specifies number of individuals in each population (more than 4 populations can be defined)
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
This tutorial require having Samtools previously installed, and the library 'pracma' previously installed in R.
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R DSTAT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test.
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -doCounts 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

==Generate files for the error correction==
We will apply error correction to the group with 3 individuals, using "perfectSampleCEU" as high-quality reference genome.
The population containing 3 individuals affected by transition error goes from line 6 to line 8 in the file bam.filelist. We select those individuals and write them in another file.
<pre>
sed -n 6,8p bam.filelist > bamWithErrors.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later as input for the R script that calculates the D-statistic. "doAncError" applies the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out errorFile -bam bamWithErrors.filelist
</pre>

==4-population test==
In this tutorial we perform the ABBABABA test on all the combinations of 4 populations amongst 6 populations of size 1,2,2,3,2,1 individuals, where the last population is fixed as outgroup (so that there are 30 possible combinations). The last population is represented by the fasta file defined with the option -anc, of which we enable the use as outgroup by the option -useLast 1. One can use the last population of .bam files as outgroup with the option -useLast 0. Create a file named sizeFile.size and write the size of each population (it is NOT necessary to define the size of the -anc fasta outgroup):
<pre>
1
2
2
3
2
</pre>
We decide to target three chromosomes with loci in the range 10000000 to 15000000 for chromosomes 1,2 and 3. Thus create a file called regions.txt in which is written
<pre>
1:10000000-15000000
2:10000000-15000000
3:10000000-15000000
</pre>
After running ANGSD to count ABBA and BABA combinations, we will call the R script who applies error correction to the ABBA and BABA allele combinations and produces the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -sizeFile sizeFile.size -doCounts 1 -out bam.Angsd -anc chimpHg19.fa -rf regions.txt -useLast 1 -minQ 20 -minMapQ 30 -p 3
</pre>
[[The output file is]]
[[bam.Angsd.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1) for one of the possible 30 combinations (so each block is written on 30 lines), a start position (Column 2), an end postion (Column 3). Columns 4,5 and 6 are the numerator, denominator and number of sites analyzed. The next 256 columns are the counted patterns of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the error files for the population with 3 individuals. This is done defining the error files in each populations inside a text file (including a line for the outgroup population). If a population has no error file, it is sufficient to write NA. Create a file called errorList.error with written
<pre>
NA
NA
NA
./errorFile.ancError
NA
NA
</pre>
We also want to study the effect of error correction if we add individually to each population an error rate between -0.005 and 0.005 of steps 0.001 and involving transition A->T. You can run more transitions separating the letters with a comma. We need to create a file with the names of the 6 populations. Create a file popNames.name with written
<pre>
Population1
Population2
Population3
PopWithError
Population4
Chimpanzee
</pre>
Run the Rscript with the command
<pre>
Rscript DSTAT angsdFile="bam.Angsd" out="result" sizeFile=sizeFile.size errFile=errorList.error nameFile=popNames.name addErr="-0.005,0.005,0.001;A;T"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities for all 30 4-populations trees.
The plots of error rates for type specific errors and added errors are in the folder result.errorDataFolder as pdf files (only if you have provided an error rate file and added error rates in the options for the R script). The other text files are used by the Rscript for the necessary computations and do not contain useful information.
The D-statistics are contained in four distinct files for each combination of populations. For example, for the tree (((Population1, Population2)PopWithError)Chimpanzee) the files containing the results are:
;[[1)result.Population1.Population2.PopWithError.Chimpanzee.Observed.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
;[[2) result.Population1.Population2.PopWithError.Chimpanzee.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
;[[3) result.Population1.Population2.PopWithError.Chimpanzee.TransRem.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
;[[4) result.Population1.Population2.PopWithError.Chimpanzee.TransRem.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal

Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, the names of the four populations. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.Population1.Population2.PopWithError.Chimpanzee.pdf showing a barplot of the error rates
;-the file plotAddErr.A2T.Population1.Population2.PopWithError.Chimpanzee.pdf showing the effect of error correction on transition A-->T

Abbababa2

2017-02-06T12:53:06Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 include fasta file defined by -anc in the analysis

</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the combinations of 4 populations that are possible to do create from the data
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove ancient transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -useLast [int]
1: include the file defined by -anc in the D-stat analysys. Default: 0
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads.
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts. Default 100.
; -sizeFile [fileName]
file that specifies number of individuals in each population (more than 4 populations can be defined)
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R DSTAT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -doCounts 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

==Generate files for the error correction==
We will apply error correction to the group with 3 individuals, using "perfectSampleCEU" as high-quality reference genome.
The population containing 3 individuals affected by transition error goes from line 6 to line 8 in the file bam.filelist. We select those individuals and write them in another file.
<pre>
sed -n 6,8p bam.filelist > bamWithErrors.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later as input for the R script that calculates the D-statistic. "doAncError" applies the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out errorFile -bam bamWithErrors.filelist
</pre>

==4-population test==
In this tutorial we perform the ABBABABA test on all the combinations of 4 populations amongst 6 populations of size 1,2,2,3,2,1 individuals (so that there are 360 possible combinations). The last population is represented by the fasta file defined with the option -anc, of which we enable the use by the option -useLast 1. Create a file named sizeFile.size and write the size of each population (skip the last referred to the -anc fasta file):
<pre>
1
2
2
3
2
</pre>
We decide to target three chromosomes with loci in the range 10000000 to 15000000 for chromosomes 1,2 and 3. Thus create a file called regions.txt in which is written
<pre>
1:10000000-15000000
2:10000000-15000000
3:10000000-15000000
</pre>
After running ANGSD we will call the R script who applies error correction to the ABBA and BABA allele combinations and produces the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -sizeFile sizeFile.size -doCounts 1 -out bam.Angsd -anc chimpHg19.fa -rf regions.txt -useLast 1 -minQ 20 -minMapQ 30 -p 3
</pre>
[[The output file is]]
[[bam.Angsd.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1) for one of the possible 360 combinations (so each block is written on 360 lines), a start position (Column 2), an end postion (Column 3). Columns 4,5 and 6 are the numerator, denominator and number of sites analyzed. The next 256 columns are the counted patterns of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the error files for the population with 3 individuals. This is done defining the error files in each populations inside a text file. If a population has no error file, it is sufficient to write NA. Create a file called errorList.error with written
<pre>
NA
NA
NA
./errorFile.ancError
NA
NA
</pre>
We also want to study the effect of error correction if we add individually to each population an error rate between -0.005 and 0.005 with step 0.001 and involving transition A->T. You can run more transitions separating the letters with a comma. We need to create a file with the names of the 6 populations. Create a file popNames.name with written
<pre>
Population1
Population2
Population3
PopWithError
Population4
Chimpanzee
</pre>
Run the Rscript with the command
<pre>
Rscript DSTAT angsdFile="bam.Angsd" out="result" sizeFile=sizeFile.size errFile=errorList.error nameFile=popNames.name addErr="-0.005,0.005,0.001;A;T"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities for all combinations.
The plots of error rates for type specific errors and added errors are in the folder result.errorDataFolder as pdf files. The other files are used by the Rscript for the necessary computations and do not contain useful information.
The D-statistics are contained in four distinct files for each combination of populations. For example, for the populations Population1, Chimpanzee, Population2, Population3 the files containing the results are:
;[[1)result.Population1.Chimpanzee.Population2.Population3.Observed.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
;[[2) result.Population1.Chimpanzee.Population2.Population3.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
;[[3) result.Population1.Chimpanzee.Population2.Population3.TransRem.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
;[[4) result.Population1.Chimpanzee.Population2.Population3.RemTrans.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal

Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, the names of the four populations. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.Population1.Chimpanzee.Population2.Population3.pdf showing a barplot of the error rates
;-the file plotAddErr.A2T.Population1.Chimpanzee.Population2.Population3.pdf showing the effect of error correction on transition A-->T

Abbababa2

2017-02-06T12:50:55Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 include fasta file defined by -anc in the analysis

</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the combinations of 4 populations that are possible to do create from the data
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove ancient transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -useLast [int]
1: include the file defined by -anc in the D-stat analysys. Default: 0
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads.
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts. Default 100.
; -sizeFile [fileName]
file that specifies number of individuals in each population (more than 4 populations can be defined)
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R DSTAT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -doCounts 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on all the combinations of 4 populations amongst 6 populations of size 1,2,2,3,2,1 individuals (so that there are 360 possible combinations). The last population is represented by the fasta file defined with the option -anc, of which we enable the use by the option -useLast 1. Create a file named sizeFile.size and write the size of each population (skip the last referred to the -anc fasta file):
<pre>
1
2
2
3
2
</pre>

==Generate files for the error correction==
We will apply error correction to the group with 3 individuals, using "perfectSampleCEU" as high-quality reference genome.
The population containing 3 individuals affected by transition error goes from line 6 to line 8 in the file bam.filelist. We select those individuals and write them in another file.
<pre>
sed -n 6,8p bam.filelist > bamWithErrors.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later as input for the R script that calculates the D-statistic. "doAncError" applies the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out errorFile -bam bamWithErrors.filelist
</pre>

==4-population test==

Now, we want to run the four population test using the 6 groups of 1,2,2,3,2,1 individuals (where the last one is the fasta file). We decide to target two regions between loci 10000000 and 15000000 for chromosomes 1 and 3. Thus create a file called regions.txt in which is written
<pre>
1:10000000-15000000
2:10000000-15000000
3:10000000-15000000
</pre>
After running ANGSD we will call the R script who applies error correction to the ABBA and BABA allele combinations and produces the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -sizeFile sizeFile.size -doCounts 1 -out bam.Angsd -anc chimpHg19.fa -rf regions.txt -useLast 1 -minQ 20 -minMapQ 30 -p 3
</pre>
[[The output file is]]
[[bam.Angsd.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1) for one of the possible 360 combinations (so each block is written on 360 lines), a start position (Column 2), an end postion (Column 3). Columns 4,5 and 6 are the numerator, denominator and number of sites analyzed. The next 256 columns are the counted patterns of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the error files for the population with 3 individuals. This is done defining the error files in each populations inside a text file. If a population has no error file, it is sufficient to write NA. Create a file called errorList.error with written
<pre>
NA
NA
NA
./errorFile.ancError
NA
NA
</pre>
We also want to study the effect of error correction if we add individually to each population an error rate between -0.005 and 0.005 with step 0.001 and involving transition A->T. You can run more transitions separating the letters with a comma. We need to create a file with the names of the 6 populations. Create a file popNames.name with written
<pre>
Population1
Population2
Population3
PopWithError
Population4
Chimpanzee
</pre>
Run the Rscript with the command
<pre>
Rscript DSTAT angsdFile="bam.Angsd" out="result" sizeFile=sizeFile.size errFile=errorList.error nameFile=popNames.name addErr="-0.005,0.005,0.001;A;T"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities for all combinations.
The plots of error rates for type specific errors and added errors are in the folder result.errorDataFolder as pdf files. The other files are used by the Rscript for the necessary computations and do not contain useful information.
The D-statistics are contained in four distinct files for each combination of populations. For example, for the populations Population1, Chimpanzee, Population2, Population3 the files containing the results are:
;[[1)result.Population1.Chimpanzee.Population2.Population3.Observed.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
;[[2) result.Population1.Chimpanzee.Population2.Population3.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
;[[3) result.Population1.Chimpanzee.Population2.Population3.TransRem.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
;[[4) result.Population1.Chimpanzee.Population2.Population3.RemTrans.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal

Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, the names of the four populations. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.Population1.Chimpanzee.Population2.Population3.pdf showing a barplot of the error rates
;-the file plotAddErr.A2T.Population1.Chimpanzee.Population2.Population3.pdf showing the effect of error correction on transition A-->T

Abbababa2

2017-02-06T12:43:39Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 include fasta file defined by -anc in the analysis

</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the combinations of 4 populations that are possible to do create from the data
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove ancient transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -useLast [int]
1: include the file defined by -anc in the D-stat analysys. Default: 0
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads.
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts. Default 100.
; -sizeFile [fileName]
file that specifies number of individuals in each population (more than 4 populations can be defined)
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R DSTAT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -doCounts 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on all the combinations of 4 populations amongst 6 populations of size 1,2,2,3,2,1 individuals (so that there are 360 possible combinations). The last population is represented by the fasta file defined with the option -anc, of which we enable the use by the option -useLast 1. Create a file named sizeFile.size and write the size of each population (skip the last referred to the -anc fasta file):
<pre>
1
2
2
3
2
</pre>

==Generate files for the error correction==
We will apply error correction to the group with 3 individuals, using "perfectSampleCEU" as high-quality reference genome.
The population containing 3 individuals affected by transition error goes from line 6 to line 8 in the file bam.filelist. We select those individuals and write them in another file.
<pre>
sed -n 6,8p bam.filelist > bamWithErrors.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later as input for the R script that calculates the D-statistic. "doAncError" applies the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out errorFile -bam bamWithErrors.filelist
</pre>

==4-population test==

Now, we want to run the four population test using the 6 groups of 1,2,2,3,2,1 individuals (where the last one is the fasta file). We decide to target two regions between loci 10000000 and 15000000 for chromosomes 1 and 3. Thus create a file called regions.txt in which is written
<pre>
1:10000000-15000000
2:10000000-15000000
3:10000000-15000000
</pre>
After running ANGSD we will call the R script who applies error correction to the ABBA and BABA allele combinations and produces the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -sizeFile sizeFile.size -doCounts 1 -out bam.Angsd -anc chimpHg19.fa -rf regions.txt -useLast 1 -minQ 20 -minMapQ 30 -p 3
</pre>
[[The output file is]]
[[bam.Angsd.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1) for one of the possible 360 combinations (so each block is written on 360 lines), a start position (Column 2), an end postion (Column 3). Columns 4,5 and 6 are the numerator, denominator and number of sites analyzed. The next 256 columns are the counted patterns of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the error files for the population with 3 individuals. This is done defining the error files in each populations inside a text file. If a population has no error file, it is sufficient to write NA. Create a file called errorList.error with written
<pre>
NA
NA
NA
./errorFile.ancError
NA
NA
</pre>
We also want to study the effect of error correction if we add individually to each population an error rate between -0.005 and 0.005 with step 0.001 and involving transition A->T. You can run more transitions separating the letters with a comma. We need to create a file with the names of the 6 populations. Create a file popNames.name with written
<pre>
Population1
Population2
Population3
PopWithError
Population4
Chimpanzee
</pre>
Run the Rscript with the command
<pre>
Rscript DSTAT angsdFile="bam.Angsd" out="result" sizeFile=sizeFile.size errFile=errorList.error nameFile=popNames.name addErr="-0.005,0.005,0.001;A;T"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities for all combinations.
The plots of error rates for type specific errors and added errors are in the folder result.errorDataFolder as pdf files. The other files are used by the Rscript for the necessary computations and do not contain useful information.
The D-statistics are contained in four distinct files for each combination of populations. For example, for the populations Population1, Chimpanzee, Population2, Population3 the files containing the results are:
;[[1)result.Observed.Population1.Chimpanzee.Population2.Population3.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
;[[2) result.Population1.Chimpanzee.Population2.Population3.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
;[[3) result.Population1.Chimpanzee.Population2.Population3.TransRem.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
;[[4) result.RemTrans.Population1.Chimpanzee.Population2.Population3.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal

Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, the names of the four populations. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.Population1.Chimpanzee.Population2.Population3.pdf showing a barplot of the error rates
;-the file plotAddErr.A2T.Population1.Chimpanzee.Population2.Population3.pdf showing the effect of error correction on transition A-->T

Abbababa2

2017-02-06T12:41:01Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 include fasta file defined by -anc in the analysis

</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the combinations of 4 populations that are possible to do create from the data
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove ancient transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -useLast [int]
1: include the file defined by -anc in the D-stat analysys. Default: 0
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads.
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts. Default 100.
; -sizeFile [fileName]
file that specifies number of individuals in each population (more than 4 populations can be defined)
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R DSTAT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -doCounts 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on all the combinations of 4 populations amongst 6 populations of size 1,2,2,3,2,1 individuals (so that there are 360 possible combinations). The last population is represented by the fasta file defined with the option -anc, of which we enable the use by the option -useLast 1. Create a file named sizeFile.size and write the size of each population (skip the last referred to the -anc fasta file):
<pre>
1
2
2
3
2
</pre>

==Generate files for the error correction==
We will apply error correction to the group with 3 individuals, using "perfectSampleCEU" as high-quality reference genome.
The population containing 3 individuals affected by transition error goes from line 6 to line 8 in the file bam.filelist. We select those individuals and write them in another file.
<pre>
sed -n 6,8p bam.filelist > bamWithErrors.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later as input for the R script that calculates the D-statistic. "doAncError" applies the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out errorFile -bam bamWithErrors.filelist
</pre>

==4-population test==

Now, we want to run the four population test using the 6 groups of 1,2,2,3,2,1 individuals (where the last one is the fasta file). We decide to target two regions between loci 10000000 and 15000000 for chromosomes 1 and 3. Thus create a file called regions.txt in which is written
<pre>
1:10000000-15000000
2:10000000-15000000
3:10000000-15000000
</pre>
After running ANGSD we will call the R script who applies error correction to the ABBA and BABA allele combinations and produces the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -sizeFile sizeFile.size -doCounts 1 -out bam.Angsd -anc chimpHg19.fa -rf regions.txt -useLast 1 -minQ 20 -minMapQ 30 -p 3
</pre>
[[The output file is]]
[[bam.Angsd.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1) for one of the possible 360 combinations (so each block is written on 360 lines), a start position (Column 2), an end postion (Column 3). Columns 4,5 and 6 are the numerator, denominator and number of sites analyzed. The next 256 columns are the counted patterns of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the error files for the population with 3 individuals. This is done defining the error files in each populations inside a text file. If a population has no error file, it is sufficient to write NA. Create a file called errorList.error with written
<pre>
NA
NA
NA
./errorFile.ancError
NA
NA
</pre>
We also want to study the effect of error correction if we add individually to each population an error rate between -0.005 and 0.005 with step 0.001 and involving transition A->T. You can run more transitions separating the letters with a comma. We need to create a file with the names of the 6 populations. Create a file popNames.name with written
<pre>
Population1
Population2
Population3
PopWithError
Population4
Chimpanzee
</pre>
Run the Rscript with the command
<pre>
Rscript DSTAT angsdFile="bam.Angsd" out="result" sizeFile=sizeFile.size errFile=errorList.error nameFile=popNames.name addErr="-0.005,0.005,0.001;A;T"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities for all combinations.
The plots of error rates for type specific errors and added errors are in the folder result.errorDataFolder as pdf files. The other files are used by the Rscript for the necessary computations and do not contain useful information.
The D-statistics are contained in four distinct files for each combination of populations. For example, for the populations Population1, Chimpanzee, Population2, Population3 the files containing the results are:
;[[1)result.Observed.Population1.Chimpanzee.Population2.Population3.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
;[[2) result.Population1.Chimpanzee.Population2.Population3.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
;[[3) result.Population1.Chimpanzee.Population2.Population3.TransRem.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
;[[4) result.RemTrans.Population1.Chimpanzee.Population2.Population3.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal

Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.Population1.Chimpanzee.Population2.Population3.pdf showing a barplot of the error rates
;-the file plotAddErr.A2T.Population1.Chimpanzee.Population2.Population3.pdf showing the effect of error correction on transition A-->T (plot shown below)

Abbababa2

2017-02-06T12:32:11Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 include fasta file defined by -anc in the analysis

</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the combinations of 4 populations that are possible to do create from the data
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove ancient transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -useLast [int]
1: include the file defined by -anc in the D-stat analysys. Default: 0
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads.
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts. Default 100.
; -sizeFile [fileName]
file that specifies number of individuals in each population (more than 4 populations can be defined)
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R DSTAT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -doCounts 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on all the combinations of 4 populations amongst 6 populations of size 1,2,2,3,2,1 individuals (so that there are 360 possible combinations). The last population is represented by the fasta file defined with the option -anc, of which we enable the use by the option -useLast 1. Create a file named sizeFile.size and write the size of each population (skip the last referred to the -anc fasta file):
<pre>
1
2
2
3
2
</pre>

==Generate files for the error correction==
We will apply error correction to the group with 3 individuals, using "perfectSampleCEU" as high-quality reference genome.
The population containing 3 individuals affected by transition error goes from line 6 to line 8 in the file bam.filelist. We select those individuals and write them in another file.
<pre>
sed -n 6,8p bam.filelist > bamWithErrors.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later as input for the R script that calculates the D-statistic. "doAncError" applies the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out errorFile -bam bamWithErrors.filelist
</pre>

==4-population test==

Now, we want to run the four population test using the 6 groups of 1,2,2,3,2,1 individuals (where the last one is the fasta file). We decide to target two regions between loci 10000000 and 15000000 for chromosomes 1 and 3. Thus create a file called regions.txt in which is written
<pre>
1:10000000-15000000
2:10000000-15000000
3:10000000-15000000
</pre>
After running ANGSD we will call the R script who applies error correction to the ABBA and BABA allele combinations and produces the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -sizeFile sizeFile.size -doCounts 1 -out bam.Angsd -anc chimpHg19.fa -rf regions.txt -useLast 1 -minQ 20 -minMapQ 30 -p 3
</pre>
[[The output file is]]
[[bam.Angsd.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1) for one of the possible 360 combinations (so each block is written on 360 lines), a start position (Column 2), an end postion (Column 3). Columns 4,5 and 6 are the numerator, denominator and number of sites analyzed. The next 256 columns are the counted patterns of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the error files for the population with 3 individuals. This is done defining the error files in each populations inside a text file. If a population has no error file, it is sufficient to write NA. Create a file called errorList.error with written
<pre>
NA
NA
NA
./errorFile.ancError
NA
NA
</pre>
We also want to study the effect of error correction if we add individually to each population an error rate between -0.005 and 0.005 with step 0.001 and involving transition A->T. You can run more transitions separating the letters with a comma. We need to create a file with the names of the 6 populations. Create a file popNames.name with written
<pre>
Population1
Population2
Population3
PopWithError
Population4
Chimpanzee
</pre>
Run the Rscript with the command
<pre>
Rscript DSTAT angsdFile="bam.Angsd" out="result" sizeFile=sizeFile.size errFile=errorList.error nameFile=popNames.name addErr="0,0.005,0.001;A;T"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities for all combinations.
The plots of error rates for type specific errors and added errors are in the folder result.errorDataFolder as pdf files. The other files are used by the Rscript for the necessary computations and do not contain useful information.
The D-statistics are contained in four distinct files for each combination of populations. For example, for the populations Population1, Chimpanzee, Population2, Population3 the files containing the results are:
;[[1)result.Observed.Population1.Chimpanzee.Population2.Population3.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
;[[2) result.Population1.Chimpanzee.Population2.Population3.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
;[[3) result.Population1.Chimpanzee.Population2.Population3.TransRem.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
;[[4) result.RemTrans.Population1.Chimpanzee.Population2.Population3.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal

Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.Population1.Chimpanzee.Population2.Population3.pdf showing a barplot of the error rates
;-the file plotAddErr.A2T.Population1.Chimpanzee.Population2.Population3.pdf showing the effect of error correction on transition A-->T (plot shown below)

Abbababa2

2017-02-06T12:31:35Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 include fasta file defined by -anc in the analysis

</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the combinations of 4 populations that are possible to do create from the data
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove ancient transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -useLast [int]
1: include the file defined by -anc in the D-stat analysys. Default: 0
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads.
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts. Default 100.
; -sizeFile [fileName]
file that specifies number of individuals in each population (more than 4 populations can be defined)
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -doCounts 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on all the combinations of 4 populations amongst 6 populations of size 1,2,2,3,2,1 individuals (so that there are 360 possible combinations). The last population is represented by the fasta file defined with the option -anc, of which we enable the use by the option -useLast 1. Create a file named sizeFile.size and write the size of each population (skip the last referred to the -anc fasta file):
<pre>
1
2
2
3
2
</pre>

==Generate files for the error correction==
We will apply error correction to the group with 3 individuals, using "perfectSampleCEU" as high-quality reference genome.
The population containing 3 individuals affected by transition error goes from line 6 to line 8 in the file bam.filelist. We select those individuals and write them in another file.
<pre>
sed -n 6,8p bam.filelist > bamWithErrors.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later as input for the R script that calculates the D-statistic. "doAncError" applies the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out errorFile -bam bamWithErrors.filelist
</pre>

==4-population test==

Now, we want to run the four population test using the 6 groups of 1,2,2,3,2,1 individuals (where the last one is the fasta file). We decide to target two regions between loci 10000000 and 15000000 for chromosomes 1 and 3. Thus create a file called regions.txt in which is written
<pre>
1:10000000-15000000
2:10000000-15000000
3:10000000-15000000
</pre>
After running ANGSD we will call the R script who applies error correction to the ABBA and BABA allele combinations and produces the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -sizeFile sizeFile.size -doCounts 1 -out bam.Angsd -anc chimpHg19.fa -rf regions.txt -useLast 1 -minQ 20 -minMapQ 30 -p 3
</pre>
[[The output file is]]
[[bam.Angsd.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1) for one of the possible 360 combinations (so each block is written on 360 lines), a start position (Column 2), an end postion (Column 3). Columns 4,5 and 6 are the numerator, denominator and number of sites analyzed. The next 256 columns are the counted patterns of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the error files for the population with 3 individuals. This is done defining the error files in each populations inside a text file. If a population has no error file, it is sufficient to write NA. Create a file called errorList.error with written
<pre>
NA
NA
NA
./errorFile.ancError
NA
NA
</pre>
We also want to study the effect of error correction if we add individually to each population an error rate between -0.005 and 0.005 with step 0.001 and involving transition A->T. You can run more transitions separating the letters with a comma. We need to create a file with the names of the 6 populations. Create a file popNames.name with written
<pre>
Population1
Population2
Population3
PopWithError
Population4
Chimpanzee
</pre>
Run the Rscript with the command
<pre>
Rscript DSTAT angsdFile="bam.Angsd" out="result" sizeFile=sizeFile.size errFile=errorList.error nameFile=popNames.name addErr="0,0.005,0.001;A;T"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities for all combinations.
The plots of error rates for type specific errors and added errors are in the folder result.errorDataFolder as pdf files. The other files are used by the Rscript for the necessary computations and do not contain useful information.
The D-statistics are contained in four distinct files for each combination of populations. For example, for the populations Population1, Chimpanzee, Population2, Population3 the files containing the results are:
;[[1)result.Observed.Population1.Chimpanzee.Population2.Population3.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
;[[2) result.Population1.Chimpanzee.Population2.Population3.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
;[[3) result.Population1.Chimpanzee.Population2.Population3.TransRem.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
;[[4) result.RemTrans.Population1.Chimpanzee.Population2.Population3.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal

Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.Population1.Chimpanzee.Population2.Population3.pdf showing a barplot of the error rates
;-the file plotAddErr.A2T.Population1.Chimpanzee.Population2.Population3.pdf showing the effect of error correction on transition A-->T (plot shown below)

Abbababa2

2017-02-06T12:22:42Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 include fasta file defined by -anc in the analysis

</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the combinations of 4 populations that are possible to do create from the data
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove ancient transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -useLast [int]
1: include the file defined by -anc in the D-stat analysys. Default: 0
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads.
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts. Default 100.
; -sizeFile [fileName]
file that specifies number of individuals in each population (more than 4 populations can be defined)
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -doCounts 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on all the combinations of 4 populations amongst 6 populations of size 1,2,2,3,2,1 individuals (so that there are 360 possible combinations). The last population is represented by the fasta file defined with the option -anc, of which we enable the use by the option -useLast 1. Create a file named sizeFile.size and write the size of each population (skip the last referred to the -anc fasta file):
<pre>
1
2
2
3
2
</pre>

==Generate files for the error correction==
We will apply error correction to the group with 3 individuals, using "perfectSampleCEU" as high-quality reference genome.
The population containing 3 individuals affected by transition error goes from line 6 to line 8 in the file bam.filelist. We select those individuals and write them in another file.
<pre>
sed -n 6,8p bam.filelist > bamWithErrors.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later as input for the R script that calculates the D-statistic. "doAncError" applies the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out errorFile -bam bamWithErrors.filelist
</pre>

==4-population test==

Now, we want to run the four population test using the 6 groups of 1,2,2,3,2,1 individuals (where the last one is the fasta file). We decide to target two regions between loci 10000000 and 15000000 for chromosomes 1 and 3. Thus create a file called regions.txt in which is written
</pre>
1:10000000-15000000
2:10000000-15000000
3:10000000-15000000
<pre>
After running ANGSD we will call the R script who applies error correction to the ABBA and BABA allele combinations and produces the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -sizeFile sizeFile.size -doCounts 1 -out bam.Angsd -anc chimpHg19.fa -rf regions.txt -useLast 1 -minQ 20 -minMapQ 30 -p 3
</pre>
[[The output file is]]
[[bam.Angsd.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1) for one of the possible 360 combinations (so each block is written on 360 lines), a start position (Column 2), an end postion (Column 3). Columns 4,5 and 6 are the numerator, denominator and number of sites analyzed. The next 256 columns are the counted patterns of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the error files for the population with 3 individuals. This is done defining the error files in each populations inside a text file. If a population has no error file, it is sufficient to write NA. Create a file called errorList.error with written
</pre>
NA
NA
NA
./errorFile.ancError
NA
NA
<pre>
We also want to study the effect of error correction if we add individually to each population an error rate between -0.005 and 0.005 with step 0.001 and involving transition A->T. You can run more transitions separating the letters with a comma. We need to create a file with the names of the 6 populations. Create a file popNames.name with written
</pre>
Population1
Population2
Population3
PopWithError
Population4
Chimpanzee
<pre>
Run the Rscript with the command
<pre>
Rscript DSTAT angsdFile="bam.Angsd" out="result" sizeFile=sizeFile.size errFile=errorList.error nameFile=popNames.name addErr="0,0.005,0.001;A;T"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities for all combinations.
The plots of error rates for type specific errors and added errors are in the folder result.errorDataFolder as pdf files. The other files are used by the Rscript for the necessary computations and do not contain useful information.
The D-statistics are contained in four distinct files for each combination of populations. For example, for the populations Population1, Chimpanzee, Population2, Population3 the files containing the results are:
;[[1)result.Observed.Population1.Chimpanzee.Population2.Population3.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
;[[2) result.Population1.Chimpanzee.Population2.Population3.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
;[[3) result.Population1.Chimpanzee.Population2.Population3.TransRem.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
;[[4) result.RemTrans.Population1.Chimpanzee.Population2.Population3.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal

Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.Population1.Chimpanzee.Population2.Population3.pdf showing a barplot of the error rates
;-the file plotAddErr.A2T.Population1.Chimpanzee.Population2.Population3.pdf showing the effect of error correction on transition A-->T (plot shown below)

Abbababa2

2017-02-06T10:41:25Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 include fasta file defined by -anc in the analysis

</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the combinations of 4 populations that are possible to do create from the data
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove ancient transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -useLast [int]
1: include the file defined by -anc in the D-stat analysys. Default: 0
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads.
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts. Default 100.
; -sizeFile [fileName]
file that specifies number of individuals in each population (more than 4 populations can be defined)
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -doCounts 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on all the combinations of 4 populations amongst 6 populations of size 1,2,2,3,2,1 individuals (so that there are 360 possible combinations). The last population is represented by the fasta file defined with the option -anc, of which we enable the use by the option -useLast 1. Create a file named sizeFile.size and write the size of each population (skip the last referred to the -anc fasta file):
<pre>
1
2
2
3
2
</pre>

==Generate files for the error correction==
We will apply error correction to the group with 3 individuals, using "perfectSampleCEU" as high-quality reference genome.
The population containing 3 individuals affected by transition error goes from line 6 to line 8 in the file bam.filelist. We select those individuals and write them in another file.
<pre>
sed -n 6,8p bam.filelist > bamWithErrors.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later as input for the R script that calculates the D-statistic. "doAncError" applies the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out errorFile -bam bamWithErrors.filelist
</pre>

==4-population test==

Now, we want to run the four population test using the 6 groups of 1,2,2,3,2,1 individuals (where the last one is the fasta file). We decide to target two regions between loci 10000000 and 15000000 for chromosomes 1 and 3. Thus create a file called regions.txt in which is written
</pre>
1:10000000-15000000
2:10000000-15000000
3:10000000-15000000
<pre>
After running ANGSD we will call the R script who applies error correction to the ABBA and BABA allele combinations and produces the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -sizeFile sizeFile.size -doCounts 1 -out bam.Angsd -anc chimpHg19.fa -rf regions.txt -useLast 1 -minQ 20 -minMapQ 30 -p 3
</pre>
[[The output file is]]
[[bam.Angsd.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1) for one of the possible 360 combinations (so each block is written on 360 lines), a start position (Column 2), an end postion (Column 3). Columns 4,5 and 6 are the numerator, denominator and number of sites analyzed. The next 256 columns are the counted patterns of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the error files for the population with 3 individuals. This is done defining the error files in each populations inside a text file. If a population has no error file, it is sufficient to write NA. Create a file called errorList.error with written
</pre>
NA
NA
NA
./errorFile.ancError
NA
NA
<pre>
We also want to study the effect of error correction if we add individually to each population an error rate between -0.005 and 0.005 with step 0.001 and involving transition A->T. You can run more transitions separating the letters with a comma. We need to create a file with the names of the 6 populations. Create a file popNames.name with written
</pre>
Population1
Population2
Population3
PopWithError
Population4
Chimpanzee
<pre>
Run the Rscript with the command
<pre>
Rscript DSTAT angsdFile="bam.Angsd" out="result" sizeFile=sizeFile.size errFile=errorList.error nameFile=popNames.name addErr="0,0.005,0.001;A;T"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities for all combinations.
The plots of error rates for type specific errors and added errors are in the folder result.errorDataFolder.
The D-statistics are contained in four distinct files for each population.

Those results are also contained in four distinct files
;[[1)result.Observed.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;[[2) result.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;[[3) result.TransRemErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;[[4) result.RemTrans.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.pdf showing a barplot of the error rates (plot shown below)
;-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T (plot shown below)
;-the file errorRates.txt showing in each line transition errors for each population, respectively
;-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2017-02-06T10:15:29Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 include fasta file defined by -anc in the analysis

</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the combinations of 4 populations that are possible to do create from the data
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove ancient transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -useLast [int]
1: include the file defined by -anc in the D-stat analysys. Default: 0
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads.
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts. Default 100.
; -sizeFile [fileName]
file that specifies number of individuals in each population (more than 4 populations can be defined)
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -doCounts 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on all the combinations of 4 populations amongst 6 populations of size 1,2,2,3,2,1 individuals (so that there are 360 possible combinations). The last population is represented by the fasta file defined with the option -anc, of which we enable the use by the option -useLast 1. Create a file named sizeFile.size and write the size of each population (skip the last referred to the -anc fasta file):
<pre>
1
2
2
3
2
</pre>

==Generate files for the error correction==
We will apply error correction to the group with 3 individuals, using "perfectSampleCEU" as high-quality reference genome.
The population containing 3 individuals affected by transition error goes from line 6 to line 8 in the file bam.filelist. We select those individuals and write them in another file.
<pre>
sed -n 6,8p bam.filelist > bamWithErrors.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later as input for the R script that calculates the D-statistic. "doAncError" applies the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out errorFile -bam bamWithErrors.filelist
</pre>

==4-population test==

Now, we want to run the four population test using the 6 groups of 1,2,2,3,2,1 individuals (where the last one is the fasta file)
After running ANGSD we will call the R script who applies error correction to the ABBA and BABA allele combinations and produces the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -sizeFile sizeFile.size -doCounts 1 -out bam.Angsd -anc chimpHg19.fa -useLast 1 -minQ 20 -minMapQ 30
</pre>
[[The output file is]]
[[bam.AllelePatterns.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T. You can run more transitions separating the letters with a comma. It is also possible to specify the names of H1,H2,H3 to be seen on the plot (do not specify them to keep standard names). In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;[[1)result.Observed.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;[[2) result.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;[[3) result.TransRemErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;[[4) result.RemTrans.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.pdf showing a barplot of the error rates (plot shown below)
;-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T (plot shown below)
;-the file errorRates.txt showing in each line transition errors for each population, respectively
;-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2017-02-06T09:24:35Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[D stat and Z scores{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeFile (null) file with sizes of the populations
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-useLast 0 include fasta file defined by -anc in the analysis

</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. Every block is repeated a number of times corresponding to the combinations of 4 populations that are possible to do create from the data
This file is used as input for the R script estAvgError.R.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [int]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeFile [fileName]
1: specifies number of individuals in each population
; -Aanc [int]
1: H4 allele is A in each site.

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on H1,H2,H3,H4 consisting respectively of 3,5,2,1 individuals. In this case we use a fasta file for the outgroup H4 (we use a bam file in next tutorial). We will apply error correction to groups H1 and H2 assuming one of the samples from H3 as high-quality reference one.

==Generate files for the error correction==

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

==4-population test==

Now, we want to run the four population test using:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
[[The output file is]]
[[bam.AllelePatterns.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T. You can run more transitions separating the letters with a comma. It is also possible to specify the names of H1,H2,H3 to be seen on the plot (do not specify them to keep standard names). In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;[[1)result.Observed.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;[[2) result.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;[[3) result.TransRemErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;[[4) result.RemTrans.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.pdf showing a barplot of the error rates (plot shown below)
;-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T (plot shown below)
;-the file errorRates.txt showing in each line transition errors for each population, respectively
;-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2016-08-31T19:47:17Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Ancient Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

;2)*.abbbababa2counts (optional file)
The columns of this file are the same as above. The file shows the sum of allele patterns observed between any possible combination of 4 genomes in the tree. This file has NO USE in the 4-population test. To print this file, see the option -combFile in the summary.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events between single individuals (however, this file is NOT USED for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on H1,H2,H3,H4 consisting respectively of 3,5,2,1 individuals. In this case we use a fasta file for the outgroup H4 (we use a bam file in next tutorial). We will apply error correction to groups H1 and H2 assuming one of the samples from H3 as high-quality reference one.

==Generate files for the error correction==

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

==4-population test==

Now, we want to run the four population test using:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
[[The output file is]]
[[bam.AllelePatterns.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T. You can run more transitions separating the letters with a comma. It is also possible to specify the names of H1,H2,H3 to be seen on the plot (do not specify them to keep standard names). In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;[[1)result.Observed.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;[[2) result.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;[[3) result.TransRemErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;[[4) result.RemTrans.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
;-the file barPlotErrors.pdf showing a barplot of the error rates (plot shown below)
;-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T (plot shown below)
;-the file errorRates.txt showing in each line transition errors for each population, respectively
;-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2016-08-29T05:49:22Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Ancient Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

;2)bam.AllelePatterns.abbbababa2counts (optional file)
The columns of this file are the same as above. The file shows the sum of allele patterns observed between any possible combination of 4 genomes in the tree. This file has NO USE in the 4-population test. To print this file, see the option -combFile in the summary.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events between single individuals (however, this file is NOT USED for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on H1,H2,H3,H4 consisting respectively of 3,5,2,1 individuals. In this case we use a fasta file for the outgroup H4 (we use a bam file in next tutorial). We will apply error correction to groups H1 and H2 assuming one of the samples from H3 as high-quality reference one.

==Generate files for the error correction==

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

==4-population test==

Now, we want to run the four population test using:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
[[The output file is]]
[[bam.AllelePatterns.abbbababa2 (used for the 4-population test)]]
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T. You can run more transitions separating the letters with a comma. It is also possible to specify the names of H1,H2,H3 to be seen on the plot (do not specify them to keep standard names). In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
[[1)result.Observed.txt]]
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
[[2) result.ErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
[[3) result.TransRemErrorCorr.txt]]
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
[[4) result.RemTrans.txt]]
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder [[result.errorDataFolder]] containing:
-the file barPlotErrors.pdf showing a barplot of the error rates (plot shown below)
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T (plot shown below)
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2016-08-29T05:37:08Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Ancient Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

;2)bam.AllelePatterns.abbbababa2counts (optional file)
The columns of this file are the same as above. The file shows the sum of allele patterns observed between any possible combination of 4 genomes in the tree. This file has NO USE in the 4-population test. To print this file, see the option -combFile in the summary.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events between single individuals (however, this file is NOT USED for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=
== Prepare BAM and FASTA files ==
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the 4-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on H1,H2,H3,H4 consisting respectively of 3,5,2,1 individuals. In this case we use a fasta file for the outgroup H4 (we use a bam file in next tutorial). We will apply error correction to groups H1 and H2 assuming one of the samples from H3 as high-quality reference one.

==Generate files for the error correction==

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

==4-population test==

Now, we want to run the four population test using:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
[[The output file is]]
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T. You can run more transitions separating the letters with a comma. It is also possible to specify the names of H1,H2,H3 to be seen on the plot (do not specify them to keep standard names). In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates (plot shown below)
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T (plot shown below)
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2016-08-29T05:33:06Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Ancient Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)*.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

;2)bam.AllelePatterns.abbbababa2counts (optional file)
The columns of this file are the same as above. The file shows the sum of allele patterns observed between any possible combination of 4 genomes in the tree. This file has NO USE in the 4-population test. To print this file, see the option -combFile in the summary.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events between single individuals (however, this file is NOT USED for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial for the ABBABABA (Multipop) test=

;[[Some preparation steps before using ANGSD]]
== Headline text ==

Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the four-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on H1,H2,H3,H4 consisting respectively of 3,5,2,1 individuals. In this case we use a fasta file for the outgroup H4 (we use a bam file in next tutorial). We will apply error correction to groups H1 and H2 assuming one of the samples from H3 as high-quality reference one.

;Prepare files for the estimation of type specific error rates

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

;ABBABABA test

Now, we want to run the four population test using:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
The output file is
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T and C-->T. It is also possible to specify the names of H1,H2,H3 to be seen on the plot. In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A,C;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates (plot shown below)
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T (plot shown below)
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2016-08-26T13:59:23Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Ancient Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

;2)bam.AllelePatterns.abbbababa2counts (optional file)
As above each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. This file is not used as input for the ABBABABA test.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events before weighting the samples (however, this file is not used for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial of the ABBABABA (Multipop) test=

;Some preparation steps before using ANGSD

Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist
</pre>
<pre>
bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the four-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on H1,H2,H3,H4 consisting respectively of 3,5,2,1 individuals. In this case we use a fasta file for the outgroup H4 (we use a bam file in next tutorial). We will apply error correction to groups H1 and H2 assuming one of the samples from H3 as high-quality reference one.

;Prepare files for the estimation of type specific error rates

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

;ABBABABA test

Now, we want to run the four population test using:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
The output file is
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T and C-->T. It is also possible to specify the names of H1,H2,H3 to be seen on the plot. In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A,C;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates (plot shown below)
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T (plot shown below)
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2016-08-26T13:58:10Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Ancient Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

;2)bam.AllelePatterns.abbbababa2counts (optional file)
As above each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. This file is not used as input for the ABBABABA test.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events before weighting the samples (however, this file is not used for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial of the ABBABABA (Multipop) test=

;Some preparation steps before using ANGSD

Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist

bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the four-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on H1,H2,H3,H4 consisting respectively of 3,5,2,1 individuals. In this case we use a fasta file for the outgroup H4 (we use a bam file in next tutorial). We will apply error correction to groups H1 and H2 assuming one of the samples from H3 as high-quality reference one.

;Prepare files for the estimation of type specific error rates

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

;ABBABABA test

Now, we want to run the four population test using:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
The output file is
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T and C-->T. It is also possible to specify the names of H1,H2,H3 to be seen on the plot. In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A,C;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates (plot shown below)
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T (plot shown below)
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2016-08-26T13:53:23Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Ancient Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.
=Output=
;1)bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

;2)bam.AllelePatterns.abbbababa2counts (optional file)
As above each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. This file is not used as input for the ABBABABA test.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events before weighting the samples (however, this file is not used for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial of the ABBABABA (Multipop) test=

;Some preparation steps before using ANGSD

Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist

bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
<\pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the four-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on H1,H2,H3,H4 consisting respectively of 3,5,2,1 individuals. In this case we use a fasta file for the outgroup H4 (we use a bam file in next tutorial). We will apply error correction to groups H1 and H2 assuming one of the samples from H3 as high-quality reference one.

;Prepare files for the estimation of type specific error rates

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

;ABBABABA test

Now, we want to run the four population test using:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
The output file is
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T and C-->T. It is also possible to specify the names of H1,H2,H3 to be seen on the plot. In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A,C;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates (plot shown below)
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T (plot shown below)
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2016-08-26T13:50:50Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Ancient Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events before weighting the samples (however, this file is not used for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial of the ABBABABA (Multipop) test=

;Some preparation steps before using ANGSD

Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist

bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
<\pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the four-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
In this tutorial we perform the ABBABABA test on H1,H2,H3,H4 consisting respectively of 3,5,2,1 individuals. In this case we use a fasta file for the outgroup H4 (we use a bam file in next tutorial). We will apply error correction to groups H1 and H2 assuming one of the samples from H3 as high-quality reference one.

;Prepare files for the estimation of type specific error rates

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

;ABBABABA test

Now, we want to run the four population test using:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
The output file is
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.
Optionally, one can also produce
;bam.AllelePatterns.abbbababa2counts (optional file)
As above each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. This file is not used as input for the ABBABABA test.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T and C-->T. It is also possible to specify the names of H1,H2,H3 to be seen on the plot. In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A,C;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates (plot shown below)
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T (plot shown below)
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2016-08-26T13:15:49Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Ancient Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events before weighting the samples (however, this file is not used for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial of the ABBABABA (Multipop) test=
;Some preparation steps before using ANGSD

Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
This is how the file bam.filelist looks like
<pre>
cat bam.filelist

bams/smallNA06985.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA06994.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07000.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07056.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA07357.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11829.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11830.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11831.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11832.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam
<\pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the four-population test. One can use a bam file as well (see in one of the other examples after the tutorial how to do it).
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

;Prepare files for the estimation of type specific error rates

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

;ABBABABA test

Now, we want to run the four population test on:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
The output file is
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.
Optionally, one can also produce
;bam.AllelePatterns.abbbababa2counts (optional file)
As above each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. This file is not used as input for the ABBABABA test.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T and C-->T. It is also possible to specify the names of H1,H2,H3 to be seen on the plot. In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A,C;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates (plot shown below)
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T (plot shown below)
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2016-08-26T13:12:30Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Ancient Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events before weighting the samples (however, this file is not used for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial of the ABBABABA (Multipop) test=
;Some preparation steps before using ANGSD

Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the four-population test. One can use a bam file as well.
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

;Prepare files for the estimation of type specific error rates

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

;ABBABABA test

Now, we want to run the four population test on:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
The output file is
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.
Optionally, one can also produce
;bam.AllelePatterns.abbbababa2counts (optional file)
As above each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. This file is not used as input for the ABBABABA test.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T and C-->T. It is also possible to specify the names of H1,H2,H3 to be seen on the plot. In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A,C;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates (plot shown below)
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T (plot shown below)
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2016-08-26T13:12:08Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Ancient Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events before weighting the samples (however, this file is not used for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial of the ABBABABA (Multipop) test=
;Some preparation steps before using ANGSD

Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the four-population test. One can use a bam file as well.
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

;Prepare files for the estimation of type specific error rates

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

;ABBABABA test

Now, we want to run the four population test on:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
The output file is
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.
Optionally, one can also produce
;bam.AllelePatterns.abbbababa2counts (optional file)
As above each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. This file is not used as input for the ABBABABA test.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T and C-->T. It is also possible to specify the names of H1,H2,H3 to be seen on the plot. In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A,C;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates (plot shown below)
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T (plot shown below)
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]ciao]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2016-08-26T13:11:02Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Ancient Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events before weighting the samples (however, this file is not used for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial of the ABBABABA (Multipop) test=
;Some preparation steps before using ANGSD

Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the four-population test. One can use a bam file as well.
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

;Prepare files for the estimation of type specific error rates

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

;ABBABABA test

Now, we want to run the four population test on:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
The output file is
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.
Optionally, one can also produce
;bam.AllelePatterns.abbbababa2counts (optional file)
As above each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. This file is not used as input for the ABBABABA test.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T and C-->T. It is also possible to specify the names of H1,H2,H3 to be seen on the plot. In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A,C;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T
-the file plotAddErr.C2T.pdf showing the effect of error correction on transition C-->T
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2016-08-26T13:09:04Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events before weighting the samples (however, this file is not used for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial of the ABBABABA (Multipop) test=
;Some preparation steps before using ANGSD

Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the four-population test. One can use a bam file as well.
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

;Prepare files for the estimation of type specific error rates

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

;ABBABABA test

Now, we want to run the four population test on:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
The output file is
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.
Optionally, one can also produce
;bam.AllelePatterns.abbbababa2counts (optional file)
As above each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. This file is not used as input for the ABBABABA test.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T and C-->T. It is also possible to specify the names of H1,H2,H3 to be seen on the plot. In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A,C;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T
-the file plotAddErr.C2T.pdf showing the effect of error correction on transition C-->T
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorJPGs.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

Abbababa2

2016-08-26T13:08:06Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events before weighting the samples (however, this file is not used for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial of the ABBABABA (Multipop) test=
;Some preparation steps before using ANGSD

Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the four-population test. One can use a bam file as well.
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

;Prepare files for the estimation of type specific error rates

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

;ABBABABA test

Now, we want to run the four population test on:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
The output file is
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.
Optionally, one can also produce
;bam.AllelePatterns.abbbababa2counts (optional file)
As above each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. This file is not used as input for the ABBABABA test.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T and C-->T. It is also possible to specify the names of H1,H2,H3 to be seen on the plot. In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A,C;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T
-the file plotAddErr.C2T.pdf showing the effect of error correction on transition C-->T
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf.

[[File:barPlotErrorsJPG.jpg]]
[[File:PlotAddErr.A2TJPG.jpg]]

File:PlotAddErr.A2TJPG.jpg

2016-08-26T13:06:50Z

Samuele: Error correction A to T doAbbababa2

Error correction A to T doAbbababa2

File:BarPlotErrorJPGs.jpg

2016-08-26T13:05:21Z

Samuele: bar plot error estimation abbababa2

bar plot error estimation abbababa2

Abbababa2

2016-08-26T13:00:55Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events before weighting the samples (however, this file is not used for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial of the ABBABABA (Multipop) test=
;Some preparation steps before using ANGSD

Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the four-population test. One can use a bam file as well.
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

;Prepare files for the estimation of type specific error rates

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

;ABBABABA test

Now, we want to run the four population test on:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
The output file is
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.
Optionally, one can also produce
;bam.AllelePatterns.abbbababa2counts (optional file)
As above each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. This file is not used as input for the ABBABABA test.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T and C-->T. It is also possible to specify the names of H1,H2,H3 to be seen on the plot. In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A,C;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T
-the file plotAddErr.C2T.pdf showing the effect of error correction on transition C-->T
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf and plotAddErr.C2T.pdf.

[[File:barPlotErrors.pdf]]

File:PlotAddErr.C2T.pdf

2016-08-26T12:59:36Z

Samuele: (Samuele) Error correction C to T doAbbababa2

(Samuele) Error correction C to T doAbbababa2

File:PlotAddErr.A2T.pdf

2016-08-26T12:59:20Z

Samuele: (Samuele) Error correction A to T doAbbababa2

(Samuele) Error correction A to T doAbbababa2

File:BarPlotErrors.pdf

2016-08-26T12:58:35Z

Samuele: (Samuele)bar plot error estimation

(Samuele)bar plot error estimation

Abbababa2

2016-08-26T12:52:17Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.RemTrans.txt|D stat and Z scores with Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events before weighting the samples (however, this file is not used for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial of the ABBABABA (Multipop) test=
;Some preparation steps before using ANGSD

Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the four-population test. One can use a bam file as well.
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

;Prepare files for the estimation of type specific error rates

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

;ABBABABA test

Now, we want to run the four population test on:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
The output file is
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.
Optionally, one can also produce
;bam.AllelePatterns.abbbababa2counts (optional file)
As above each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. This file is not used as input for the ABBABABA test.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T and C-->T. It is also possible to specify the names of H1,H2,H3 to be seen on the plot. In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A,C;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T
-the file plotAddErr.C2T.pdf showing the effect of error correction on transition C-->T
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf and plotAddErr.C2T.pdf.

Abbababa2

2016-08-26T12:51:12Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRem.txt|D stat and Z scores with Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events before weighting the samples (however, this file is not used for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial of the ABBABABA (Multipop) test=
;Some preparation steps before using ANGSD

Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, position them in the folder ./bams/ and create a file bam.filelist listing the pathnames of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done #index bam files
ls bams/*.bam > bam.filelist
rm bams.tar.gz #remove zipped file
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the four-population test. One can use a bam file as well.
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
rm chimpHg19.fa.gz
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>

;Prepare files for the estimation of type specific error rates

Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>

;ABBABABA test

Now, we want to run the four population test on:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
The output file is
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.
Optionally, one can also produce
;bam.AllelePatterns.abbbababa2counts (optional file)
As above each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. This file is not used as input for the ABBABABA test.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T and C-->T. It is also possible to specify the names of H1,H2,H3 to be seen on the plot. In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A,C;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T
-the file plotAddErr.C2T.pdf showing the effect of error correction on transition C-->T
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf and plotAddErr.C2T.pdf.

Abbababa2

2016-08-26T12:46:15Z

Samuele:

The ''ABBABABA (multipop)'' compute the abbababa test (aka D-statistic), that means testing for an ancient admixture (or wrong tree topology).
Differently from ABBABABA (D_stat) multiple individuals for each one of the groups are allowed. As all methods in ANGSD we require that the header of the BAM files are the same.
; some of the options only works for the developmental version availeble from github
; if you use -rf to specify regions. These MUST appear in the same ordering as your fai file.

__TOC__
<classdiagram type="dir:LR">
[*.bam and/or *.cram| NGS genome datasets{bg:orange}]->[Sequence data|All bases or Random bases]
[Sequence data]->[Elaborate multiple genomes per population]
[Sequence data]->[*.abbababa2counts|ABBA and BABA intermediate counts file {bg:blue}]
[Elaborate multiple genomes per population]->[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]
</classdiagram>

<classdiagram type="dir:LR">
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.Observed.txt|Observed D stat and Z scores{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.ErrorCorr.txt|D stat and Z scores Error Corrected{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRemErrorCorr.txt|D stat and Z scores Error Corrected with ancient Transition Removal{bg:blue}]
[*.abbababa2|weighted ABBA and BABA counts file {bg:blue}]->estAvgError.R[*.TransRem.txt|D stat and Z scores with Transition Removal{bg:blue}]
</classdiagram>

=Method=

[[File:tree.png|800px]]

=Brief Overview=
<pre>
> ./angsd -doAbbababa2

--------------
abcDstat2.cpp:
-doAbbababa2 0 run the abbababa analysis
-rmTrans 0 remove transitions
-blockSize 5000000 size of each block in bases
-anc (null) fasta file with outgroup
-sample 0 sample a single base in each individual
-maxDepth 100 max depth of each site allowed
-sizeH1 1 num of individuals in group H1
-sizeH2 1 num of individuals in group H2
-sizeH3 1 num of individuals in group H3
-sizeH4 1 num of individuals in group H4
-enhance 0 only analyze sites where outgroup H4 is non poly
-Aanc 0 set H4 outgroup allele as A in each site
-combFile 0 create an optional *.abbababa2counts file where are printed the
numbers of alleles combinations without having weighted the individuals
</pre>
This function will counts the number of ABBA and BABA sites.

=Options=
;-doAbbababa2 1:
take all bases at each position.
;-rmTrans [int]
0; use all reads (default), 1 Remove transitions (important for ancient DNA)
;-blockSize [INT]
Size of each block. Choose a number that is higher than the LD in the populations. For human 5Mb (5000000) is usually used.
; -anc [fileName.fa]
Include an outgroup in fasta format.
; -doCounts 1
use -doCounts 1 in order to count the bases at each sites after filters.
; -enhance [int]
1: use only sites where the reads for the outgroup has the same base for all reads (If you want to use it, it must be only when you have one genome in the outgroup, it won't work otherwise).
; -sample [int]
1: sample only one base at each position for every individual 0: all bases at each position are used for the ABBABABA test
; -maxDepth [int]
allows for a maximum depth in each site to avoid overflow of the ABBA BABA counts
; -sizeH* [int]
decide how many individuals are in each group (the file list must contain the BAM files ordered from population 1 to 4).
If you are using a fasta file (option -anc) for population H4, leave -sizeH4 at its default value
; -Aanc [int]
1: H4 allele is A in each site.
; -combFile [int]
1: create an intermediate *.abbababa2counts to obtain the allele events before weighting the samples (however, this file is not used for the estimation of the D-statistic).

In order to do fancy filtering of bases based on quality scores see the [[Alleles_counts|Allele counts]] options.

=Tutorial of the ABBABABA Multipop test=

;Some preparation steps before using ANGSD
Download the latest version of angsd in your working folder from the github repository
<pre>
https://github.com/ANGSD/angsd.git
</pre>
Create symbolic links to angsd and the necessary R script
<pre>
ln -s ./angsd/angsd ANGSD
ln -s ./angsd/R/estAvgError.R RSCRIPT
</pre>
Get 10 example .bam datasets, positioned in the folder ./bams/ and create a file bam.filelist containing the pathname of those datasets
<pre>
wget http://popgen.dk/software/download/angsd/bams.tar.gz
tar xf bams.tar.gz
for i in bams/*.bam;do samtools index $i;done
ls bams/*.bam > bam.filelist
rm bams.tar.gz
</pre>
Download a fasta file for the chimpanzee. This is going to be used as the outgroup for the four-population test. One can use a bam file as well.
<pre>
wget http://popgen.dk/software/download/angsd/hg19ancNoChr.fa.gz
mv hg19ancNoChr.fa.gz chimpHg19.fa.gz
gunzip chimpHg19.fa.gz
samtools faidx chimpHg19.fa #indexing the fasta file
</pre>
Now, generate a fasta file for one of our 10 bam file. We assume such a genome has very high quality and we can use it as a reference for estimating error rates in others of our datasets.
<pre>
./ANGSD -i bams/smallNA11840.mapped.ILLUMINA.bwa.CEU.low_coverage.20111114.bam -doFasta 1 -out perfectSampleCEU
gunzip perfectSampleCEU.fa.gz
samtools faidx perfectSampleCEU.fa
</pre>
;Prepare files for the estimation of type specific error rates
Assume population H1 consists of the first two genomes of our list, while population H2 consists of the genomes 3 to 7. We want to apply error correction to those genomes, because we know they have been subjected to contamination. We generate two files containing the pathnames of the genomes of H1 and H2 on which we want to apply error correction.
<pre>
sed -n 1,2p bam.filelist > bamH1.filelist
sed -n 3,7p bam.filelist > bamH2.filelist
</pre>
and then we use "doAncError" to generate the intermediate files that we will use later to estimate the error rates for the two groups H1 and H2. "doAncError" apply the so called "perfect individual assumption", based on which error rates are estimate using a high quality genome (option -ref) and an outgroup (option -anc), both in fasta format. We have already prepared the two fasta files in our preparation phase.
<pre>
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH1 -bam bamH1.filelist
./ANGSD -doAncError 1 -anc chimpHg19.fa -ref perfectSampleCEU.fa -out bamH2 -bam bamH2.filelist
</pre>
Now, we want to run the four population test on:
H1: first 2 bam files
H2: bam files from 3 to 7
H3: bam files from 8 to 10
H4: chimpHg19.fa file
After running ANGSD we will call the R script who apply error correction to the ABBA and BABA allele combinations and produce the final output files.
<pre>
./ANGSD -doAbbababa2 1 -bam bam.filelist -doCounts 1 -out bam.AllelePatterns -sizeH1 2 -sizeH2 5 -sizeH3 3 -anc chimpHg19.fa -minQ 20 -minMapQ 30
</pre>
The output file is
;bam.AllelePatterns.abbbababa2 (used for the 4-population test)
Each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T.
This file is used as input for the R script estAvgError.R.
Optionally, one can also produce
;bam.AllelePatterns.abbbababa2counts (optional file)
As above each line represents a block with a chromsome name (Column 1), a start position (Column 2), an end postion (Column 3). The new columns are the counts of all 256 counted combination of alleles, starting from X0000=AAAA,X0001=AAAC,....,X3333=TTTT, with the correspondence 0=A,1=C,2=G,3=T. This file is not used as input for the ABBABABA test.

We run the R script specifying the intermediate error files for populations H1 and H2. We also want to study the effect of error correction if we add individually to each population an error rate between 0 and 0.005 with step 0.001 and involving transitions A->T and C-->T. It is also possible to specify the names of H1,H2,H3 to be seen on the plot. In this case we use the generic names CEU1,CEU2,CEU3. When at least an error file is given as input, the script will apply error correction.
<pre>
Rscript RSCRIPT angsdFile="bam.AllelePatterns" out="result" file1="bamH1.ancError" file2="bamH2.ancError" addErr="0,0.005,0.001;A,C;T;CEU1,CEU2,CEU3"
</pre>
The script will show the calculated D statistic along with Z-score, Pvalues, Standard deviation and other quantities.
<pre>
--- Table of Results ---
---------------------------------------------------------------------------------
Mode |Dstat |sd(Dstat) |Djack |Zscore |Pvalue
---------------------------------------------------------------------------------
Observed |-6.323e-02 |6.985e-02 |-6.323e-02 |-0.905 |3.7e-01
---------------------------------------------------------------------------------
Err Corr |-6.430e-02 |7.226e-02 |-6.431e-02 |-0.890 |3.7e-01
---------------------------------------------------------------------------------
No Trans |-1.141e-02 |6.311e-02 |-1.141e-02 |-0.181 |8.6e-01
---------------------------------------------------------------------------------
Err Corr | | | | |
and |-1.494e-02 |6.615e-02 |-1.496e-02 |-0.226 |8.2e-01
No Trans | | | | |
---------------------------------------------------------------------------------
plots with effect of removed errors and D statistic files for all the removed errors are in folder result.errorDataFolder
</pre>
Those results are also contained in four distinct files
;1) result.Observed.txt
D-statistic calculated WITHOUT Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.063233 -0.063233 0.004878 -0.905320 0.365296 246.033565 279.248560 292.834879
</pre>
;2) result.ErrorCorr.txt
D-statistic calculated WITH Error Correction and WITHOUT Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.064295 -0.064309 0.005221 -0.889833 0.373555 238.242964 270.983960 293.326044
</pre>
;3) result.TransRemErrorCorr.txt
D-statistic calculated WITH Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.014939 -0.014959 0.004376 -0.225829 0.821335 81.636843 84.112983 293.326044
</pre>
;4) result.RemTrans.txt
D-statistic calculated WITHOUT Error Correction and WITH Ancient Transition removal
<pre>
mean(D) JK-D V(JK-D) Z pvalue nABBA nBABA nBBAA
-0.011406 -0.011406 0.003983 -0.180730 0.856580 85.730478 87.708709 292.834879
</pre>
Specifically, the values contained in the four files are: mean(D)=average D-stat, JK-D=jackknife estimate of the D-stat, V(JK-D)=variance of the D-stat, Z=Z score, pvalue=pvalue from the Z score, nABBA=number of ABBA patterns observed, nBABA=number of BABA patterns observed, nBBAA=all the other observed patterns. Note that the number of patterns might not be integer because of how ANGSD treats multiple genomes per populations.

In case of error correction, the R script also creates the folder result.errorDataFolder containing:
-the file barPlotErrors.pdf showing a barplot of the error rates
-the file plotAddErr.A2T.pdf showing the effect of error correction on transition A-->T
-the file plotAddErr.C2T.pdf showing the effect of error correction on transition C-->T
-the file errorRates.txt showing in each line transition errors for each population, respectively
-all the files related to the addition of error correction to H1,H2,H3, necessary to plot the files plotAddErr.A2T.pdf and plotAddErr.C2T.pdf.

Abbababa2

2016-08-26T11:48:25Z

Samuele:

Abbababa2

2016-08-26T11:45:55Z

Samuele:

Abbababa2

2016-08-26T11:42:17Z

Samuele:

Abbababa2

2016-08-26T11:41:16Z

Samuele:

Abbababa2

2016-08-26T11:36:48Z

Samuele:

Abbababa2

2016-08-26T11:36:32Z

Samuele:

Abbababa2

2016-08-26T11:35:16Z

Samuele:

Abbababa2

2016-08-26T11:32:26Z

Samuele:

Abbababa2

2016-08-26T11:31:56Z

Samuele:

Abbababa2

2016-08-26T11:30:29Z

Samuele:

Abbababa2

2016-08-26T11:29:49Z

Samuele: