Conventions and Notation

Every factor of two in population genetics, stated in both conventions side by side, with the primary source for each — plus a nine-question checklist to run against a draft.

"It should be emphasized that nothing is involved here other than convenience of notation." — Warren Ewens, introducing three different ways to write the same fitness triplet1

"This formulation is proposed at the risk of some confusion because t is now the selection coefficient against aa rather than the time in generations." — Hartl and Clark, apologising for their own symbol collision2

Why this page exists

Population genetics is full of factors of two that depend on bookkeeping rather than biology.  A paper that writes s = 0.01 and a paper that writes s = 0.01 can mean numbers that differ by two, and both are right.

This is not pedantry, and the proof is that Fisher got one of them wrong and Wright had to correct him.  Wright, 1931:  Fisher's 1922 rate of loss of heterozygosis "comes out 1/4N... This is just half the rate indicated by the method referred to above."3  The right answer is 1/(2N).

This page is the binding reference for the population-genetics article and for every other article on this site that uses these quantities.  It states each convention, names the primary source that uses it, and gives the conversion.  Section 4 is a checklist to run against a draft.

1. The three switches almost every stray 2 comes from

  1. Ploidy. Diploid: 2N gene copies, drift variance p(1−p)/(2Ne). Haploid: N gene copies, variance p(1−p)/Ne.
  2. Where s is measured. On one allele copy (shet) or on the homozygote (shom). With no dominance, shom = 2·shet.
  3. Which allele h describes. The deleterious one (Crow & Kimura, Hartl & Clark) or the favourable one (Ewens). hEwens = 1 − hCrow.

Everything in section 3 is a consequence of those three.

2. The symbol table

Read the last column as a warning, not a menu.  The middle column is what this site uses everywhere.

2.1 Population size

Symbol This site means Also in the literature
N census number of breeding adults Wright 1931 p. 110 restricts it explicitly. Conservation papers often mean total headcount, 3-10× larger. In haploid models N is the number of gene copies, so neutral fixation is 1/N, not 1/(2N)
Ne effective size: the size of an ideal Wright-Fisher population that drifts as fast as this one At least five: inbreeding, variance, eigenvalue, coalescent, linkage-disequilibrium. They differ without limit outside an isolated constant-size population. Wright 1931 has no symbol at all — he writes "the effective N" in words
Nm, Nf breeding males, breeding females Crow & Kimura eq. 3.13.1, p. 109
n sample size — sequences or individuals examined Crow & Kimura use n for number of loci in places; Fisher 1930 p. 77 uses n for generations elapsed. Never leave n unlabelled
k do not use bare At least four meanings: gametes per individual (Crow & Kimura §7.6.2), substitution rate (Kimura 1969), lineages in a coalescent sample (Hartl & Clark p. 305), and a selection coefficient (Haldane 1927)
Vk variance in offspring/gamete number Crow & Kimura eq. 7.6.2.13, p. 349
F inbreeding coefficient Collides with FST, the fixation index, and with Fisher's other uses. Subscript always

2.2 Alleles and variation

Symbol This site means Also in the literature
p, q allele frequencies, p + q = 1 Kimura 1962 p. 714 lets p be the mutant; Crow & Kimura p. 183 lets q be the deleterious allele; Hartl & Clark p. 223 lets p be the favoured allele. State it every time
H heterozygosity Hobs (counted) and Hexp = 2pq (Hardy-Weinberg) are different measurements, and F is built from their ratio. For haploids the same quantity is "gene diversity"
π nucleotide diversity, per site Sometimes reported per kilobase, sometimes as a raw count of differences. Always say "per site"
θ 4·Ne·μ for autosomal diploids Two meanings in one textbook. Hartl & Clark p. 57 use θ for Watterson's estimator; elsewhere it is the parameter. And Watterson's and Tajima's θ is per sequence, not per site
D do not use bare Three meanings: the linkage-disequilibrium coefficient, Tajima's D, and Haldane's D (the cost of natural selection)
r² squared correlation between allele states at two loci Not the same as D′. Collides with r for recombination

2.3 Selection and fitness

Symbol This site means Also in the literature
W, w W absolute fitness, w relative Ewens p. 12: only ratios matter, "it is therefore possible to scale the wij in any way convenient"
m Malthusian parameter, continuous growth rate Fisher's unit is the year (1930 p. 26; his worked value p. 28 is "+0·01231 per annum"). Mixing per-year m with per-generation w is a silent error of a factor of the generation time
s on the homozygote, h separate shet (per copy): Fisher 1930 p. 76, Kimura 1962 eq. (7) p. 715, Haldane 1927. shom: Crow & Kimura p. 183, Kimura 1962 eq. (12) p. 716, Kimura & Ohta 1969 p. 766, Hartl & Clark p. 223, Ewens eq. (1.25b) p. 13
h dominance of the mutant; 0 = recessive mutant Ewens eq. (1.25b) makes h = 1 the recessive case, because his h is on the favourable allele. hEwens = 1 − hCrow
L genetic load, (wmax − w̄)/wmax The reference differs by load type. Also L = number of loci in many papers
α do not use bare Three unrelated meanings: the diffusion parameter 2Ns (Ewens eq. 1.61), the proportion of substitutions fixed by positive selection (McDonald-Kreitman), and an intermediate in Crow & Kimura eq. 7.6.3.26

2.4 Mutation, recombination, time

Symbol This site means Also in the literature
μ per haploid site copy per generation Written u by Wright, Crow & Kimura and Ewens, and ν by Ohta. u collides head-on with u(p), the fixation probability: Kimura 1968 p. 625 writes u(p) for fixation probability on the same page as a per-gamete mutation rate4
U deleterious mutations per diploid genome per generation Some papers report per haploid gamete. Kimura 1968 states his own figure both ways: "roughly two per gamete and four per zygote"4
c recombination fraction per meiosis, 0-0.5 Collides with Fisher 1930 p. 76, where c is the offspring-number ratio, c = 1 + s
r recombination rate per bp per generation Also the correlation coefficient. Also the intrinsic rate of increase in ecology, which is Fisher's m
t time in generations Hartl & Clark p. 229 use t as a selection coefficient and apologise for it in print
τ coalescent time, usually scaled in units of 2Ne or 4Ne Also population divergence time in demographic-inference papers. A published τ is not in generations until somebody multiplies

3. The factor-of-2 audit

3.1 Everything that changes with the s convention

Quantity Genic (s = shet) Homozygote (s = shom, h = ½) Source
Genotype fitnesses 1 : 1+s : 1+2s 1 : 1+½s : 1+s Ewens eq. (1.25b), p. 13
Mean change per generation Δp = s p(1−p) Δp = ½ s p(1−p) Kimura 1962 eq. (7), p. 715
Diffusion scaling parameter 4Nes 2Nes Kimura 1962 p. 715; Ewens eq. (1.61), p. 24
Fixation probability, one new copy ≈ 2s·(Ne/N) ≈ s·(Ne/N) Fisher 1930 p. 76; Ewens eq. (1.82), p. 29
Deterministic sweep time (2/s)·ln 2N (4/s)·ln 2N both = 1,981 generations at N = 10⁴, shet = 0.01
Drift barrier, strict reading |s| ≲ 1/(4Ne) |s| ≲ 1/(2Ne)
Equilibrium frequency, deleterious q̂ = u/s q̂ = u/(hs) = 2u/s Crow & Kimura eq. 6.12.2.4, p. 300

The line that ties it together:  the exponent in every fixation formula is 4·Ne × (the advantage carried by one copy).  Kimura's 4Ns and Ewens' 2Ns are the same number.  Ewens says it out loud on p. 25:  his s equals "Fisher's 2a and Wright's 2s."

A writer who checks that one product will not go wrong.  This site therefore prints the master parameter rather than a threshold.

3.2 Everything that changes with ploidy

Quantity Diploid autosome Haploid (mtDNA, Y, bacteria)
Gene copies 2N N
Neutral fixation probability 1/(2N) 1/N
Neutral mean fixation time 4Ne 2Ne
Mean coalescence time of two copies 2Ne Ne
θ 4Neμ 2Neμ
Drift variance per generation p(1−p)/(2Ne) p(1−p)/Ne

The compartment table, which the mitochondrial-Eve and Y-chromosome articles cite:  at an even sex ratio, autosome : X : Y : mtDNA carry 4 : 3 : 1 : 1 gene copies, so expected diversity and expected coalescent time both scale as 1 : 0.75 : 0.25 : 0.25, and θ is 4Neμ : 3Neμ : 1Neμ : 1Neμ on one autosomal Ne.  The mtDNA and Y compartments are more often written 2Neμ, which is the same quantity on a haploid head count of transmitting individuals — half the autosomal Ne.  Say which Ne before using either form.

A correction to an earlier note in this project's own research files:  carrying an autosomal θ = 4Neμ into a mitochondrial calculation is out by a factor of four, not eight.

3.3 Places a factor of 2 is real biology, not notation

These must not be "corrected":

  • Minimising family-size variance doubles Ne. With Vk = 0, Ne = 2N − 1. Crow & Kimura p. 349: "the effective size is approximately twice as large as the actual size."
  • The mutation load is 2u per locus unless the allele is fully recessive, when it is u. The 2 is because a non-recessive deleterious allele is removed one copy at a time.
  • The 2 in Haldane's 2s is from the Poisson branching process, not from ploidy. Each new mutant copy leaves a Poisson number of copies with mean 1 + sh, so the survival probability is ≈ 2sh.
  • Neutral fixation time is 4Ne but neutral coalescence time of two copies is 2Ne. Both right; different questions.

3.4 The lethal-equivalent factor of two

Worth its own section because it has produced a published error inside this project's own research files, in a dossier that named the trap and then committed it forty lines earlier.

Morton, Crow and Muller defined B per gamete in 1956, and said so in words:  "the total genetic damage here measured is 1.5-2.5 lethal equivalents per gamete, or 3-5 per zygote."5  Conservation genetics quotes the zygote figure — Ralls' 3.14, O'Grady's 12, Grueber's 16 are all diploid.

excess mortality = 1 - exp( -(B_diploid / 2) * F )

Frankham's own published arithmetic settles it three separate ways:6

Frankham's published statement with B/2 with B
12 diploid LE at F = 5% → "decline in total fitness of 26%" 25.9% 45.1%
16 diploid LE at F = 5% → "a decline of 33%" 33.0% 55.1%
12 diploid LE, Ne = 142 for 5 generations → retains 90% 10.0% lost 19.0% lost

And Ralls, Ballou and Templeton's headline — "mortality was 33% higher in offspring of such matings than in offspring of unrelated parents" — reproduces at F = 0.25 from their median B as 32.5% under the halving and 54.4% without.

One further correction while checking:  Ralls et al. never print "3.14."  They report a median B of +1.57 per gamete and say the per-individual figure is twice that, "with a mean of 4.6 and a median of 3.1."  The 3.14 in the secondary literature is 2 × 1.57.  Their sample is 40 captive mammalian populations across 38 species, B ranges from −0.68 to +15.16 per gamete — equivalently −1.4 to 30.3 per individual — and only 9 of the 40 slopes (23%) were significantly greater than zero.7

3.5 Haigh's ratchet is the one haploid formula 

n0 = N * e^(-U/s)

Haigh 1978's model is asexual and effectively haploid:  N is the number of asexual individuals, U is deleterious mutations per haploid genome per generation, and s is the cost of one mutation with no dominance.  Read directly, the paper never uses the words diploid, haploid, heterozygote, or 2N — so any factor of two in the secondary literature came from a later author's diploid mapping, not from Haigh.8

Feeding it this site's homozygous s where a haploid per-mutation cost belongs doubles the exponent, and the difference decides whether the ratchet turns.  Note that a diploid U and a homozygous s cancel exactly — U/s is unchanged — so it is one substitution at a time that does the damage:

N = 1e6, U = 1, s = 0.02:   haploid reading 1.9e-16    diploid h=1/2  3.7e-38
N = 1e6, U = 1, s = 0.1:    haploid reading 45.4       diploid h=1/2  0.0021

4. The checklist

Nine questions to run against a draft.  Any "don't know" is a bug.

  1. Every s — homozygote or one copy? Is that stated in the sentence, not in a footnote?
  2. Every h — does h = 0 mean recessive here? Whose convention is the cited source using?
  3. Every N in a formula — census N (which sets the starting frequency 1/(2N)) or Ne (which sets the drift variance)? Kimura 1962 collapses them; Kimura & Ohta 1969 p. 763 separates them explicitly.
  4. Every ploidy-sensitive quantity — diploid, haploid, or uniparentally inherited? Is the 2 present or absent accordingly?
  5. Every mutation rate — per site or per genome? Per gamete or per zygote? Per generation or per year?
  6. Every θ — parameter (4Neμ) or estimator (Watterson's S/an)? Autosomal or organellar? Per site or per sequence?
  7. Every π — per site, and stated as such?
  8. Every rate expressed per year — converted with a stated generation time? Print the number.
  9. Every number produced by a tool — which module, and what does that module's s mean?

Question 9 is not hypothetical.  Before this site's calculator was consolidated, two of its modules answered the same question a factor of two apart — fixationProbNewMutant(0.01, 1e4) returned 1.98 × 10-2 while fixationProb(0.01, 1e4, 5e-5, 0.5) returned 9.95 × 10-3.  Both were correct and both were documented.  That is exactly what makes it dangerous.  There is now a single implementation of each formula and a cross-module consistency test that fails if any two paths disagree.

5. Two things worth knowing about the shortcuts

"2s" is an approximation, and the exact number is published.  Fisher's Table 2, p. 76 gives 0.0197 for a 1% advantage — "just less than 2 per cent."  Two hundred thousand Wright-Fisher runs at N = 10⁴, shom = 0.02 give 0.01981.  Print the exact number, then mention the shortcut.

The rule of thumb (2/s)·ln(2N) for sweep time runs about 50% high.  At N = 10⁴ and shet = 0.01 it gives 1,981 generations;  the diffusion answer conditional on fixation is 1,312;  200,000 Wright-Fisher runs give 1,322 and a separate 60,000-replicate run gives 1,327 ± 5.  The rule of thumb charges the mutation full deterministic time to escape the rare phase, but a mutation conditioned on fixing escaped it fast.

6. Two textbook slips found while checking this page

Neither matters scientifically.  Both matter as evidence that a reference article should show its arithmetic rather than assert it.

  • Hartl & Clark 1997 give the neutral fixation time as "4N generations" on p. 173 and "4Ne generations" on p. 318. The inconsistency is a direct cost of deferring the definition of Ne to p. 289.
  • Ewens 2004 p. 30 prints (0.99)²⁰⁰ ≈ 0.14. The value is 0.1340.

Sources

Entries without an inline marker are the sources of the tables above;  each is named in its row or caption.

  1. 1.

    Ewens, Warren J.  Mathematical Population Genetics I:  Theoretical Introduction, 2nd edition.  Springer, 2004.

    1. 1a.

      Page 13, introducing the three fitness parameterisations (1.25a), (1.25b) and (1.25c):  "It should be emphasized that nothing is involved here other than convenience of notation." ↩

  2. 2.

    Hartl, Daniel L. & Andrew G. Clark.  Principles of Population Genetics, 3rd edition.  Sunderland:  Sinauer, 1997.

    1. 2a.

      Page 229:  "This formulation is proposed at the risk of some confusion because t is now the selection coefficient against aa rather than the time in generations." ↩

  3. 3.

    Wright, Sewall.  "Evolution in Mendelian Populations."  Genetics 16(2):97-159.  1931.

    1. 3a.

      Page 107, correcting Fisher's 1922 rate of loss of heterozygosis:  it "comes out 1/4N... This is just half the rate indicated by the method referred to above." ↩

  4. 4.

    Kimura, Motoo.  "Evolutionary Rate at the Molecular Level."  Nature 217:624-626.  1968.  Page 625 carries both "u(p) is the probability of fixation given by" and "the mutation rate per generation for neutral mutations amounts to roughly two per gamete and four per zygote." ↩ ↩

  5. 5.

    Morton, N. E., J. F. Crow & H. J. Muller.  "An estimate of the mutational damage in man from data on consanguineous marriages."  PNAS 42(11):855-863.  1956.

    1. 5a.

      Page 859:  "the total genetic damage here measured is 1.5-2.5 lethal equivalents per gamete, or 3-5 per zygote." ↩

  6. 6.

    Frankham, Richard, Corey J. A. Bradshaw & Barry W. Brook.  "Genetics in conservation management:  Revised recommendations for the 50/500 rules, Red List criteria and population viability analyses."  Biological Conservation 170:56-63.  2014.  All three reproduced statements are on p. 58. ↩

  7. 7.

    Ralls, Katherine, Jonathan D. Ballou & Alan Templeton.  "Estimates of Lethal Equivalents and the Cost of Inbreeding in Mammals."  Conservation Biology 2(2):185-193.  1988.  The paper reports a median B of +1.57 per gamete and a per-individual mean of 4.6 and median of 3.1;  the widely quoted "3.14" is 2 × 1.57 and appears in the secondary literature rather than in Ralls et al.  Sample:  40 captive mammalian populations, 38 species;  B from −0.68 to +15.16 per gamete (−1.4 to 30.3 per individual);  9 of 40 slopes significantly greater than zero. ↩

  8. 8.

    Haigh, John.  "The accumulation of deleterious genes in a population — Muller's Ratchet."  Theoretical Population Biology 14(2):251-267.  1978.  The model is asexual;  the paper does not use the words diploid, haploid, heterozygote or 2N. ↩

  9. 9.

    Kimura, Motoo.  "On the Probability of Fixation of Mutant Genes in a Population."  Genetics 47(6):713-719.  1962.

  10. 10.

    Kimura, Motoo & Tomoko Ohta.  "The Average Number of Generations until Fixation of a Mutant Gene in a Finite Population."  Genetics 61(3):763-771.  1969.  Page 763 distinguishes census N from variance effective number Ne explicitly.

  11. 11.

    Crow, James F. & Motoo Kimura.  An Introduction to Population Genetics Theory.  New York:  Harper & Row, 1970.

  12. 12.

    Fisher, R. A.  The Genetical Theory of Natural Selection.  Oxford:  Clarendon Press, 1930.  Table 2, p. 76;  the Malthusian parameter defined p. 26 and worked p. 28.