What makes one ecosystem more diverse than another? The number of species seems to be an obvious answer. But is a forest with a thousand oaks and a single birch really as diverse as one with five hundred of each? And is a forest with a hundred different plant species as diverse as one with fifty plant and fifty animal species? These questions illustrate a fundamental challenge in measuring diversity: we must first decide what we mean by it.
Imagine you have been given the job of surveying the trees in a forest. Even if you could count every tree in a site, how would you turn your observations into a number that captures the site's diversity? Suppose you catalogue 8 oaks and 2 birches at a site. You can calculate the relative abundance of each species: the fraction of trees that belong to it. The composition of this site is $\mathbf{p}=(0.8, 0.2)$.
A common measure of diversity is species richness — the number of species — which in this case is 2. However, this feels slightly disingenuous. Yes, there are 2 species, but the oaks are much more prominent. Intuitively, diversity is not only about the presence of species, but also how evenly abundant they are. How do you account for (un)evenness?
The Simpson concentration of a site is the probability that two independently chosen trees are of the same species: $\sum_i p_i^2$. The more uneven the abundances, the higher the concentration. Inverse Simpson concentration — $1/{\sum_i p_i^2}$ — is a diversity index, which for your site is 1.47. This feels more honest: it conveys that there are multiple species, but they are not equally abundant. In fact, if there had been equal numbers of oaks and birches, the same formula would have given you 2! This hints at an interpretation for the value of the diversity index: the site is as diverse as a hypothetical one with 1.47 equally abundant species. Of course, there can never literally be 1.47 species. Rather, the number places the site on a scale between one dominant species and two equally abundant ones.
For good measure, you also calculate the Berger-Parker index — $1/\max_i p_i$ — the inverse of the population share of the most abundant species. In your site, it is 1.25. Once again, if oaks and birches had been equally abundant, this number would have been 2.
Each of these is a valid and useful way to measure diversity, but they yield different answers. This means that when comparing two sites, the judgement of which is more diverse may depend on the choice of the diversity index. This apparent ambiguity led ecologist Stuart Hurlbert to despair and call diversity a "nonconcept" (Hurlbert 1971).
In response, Mark Hill, while proposing the diversity indices that have come to be known after him, said that "...when carefully defined according to an appropriate notation, diversity can be as unequivocal as any other ecological parameter" (Hill 1973). The Hill numbers are a family of diversity indices indexed by a parameter $\alpha \geq 0$. Each choice of $\alpha$ yields a diversity index that can be interpreted as the effective number of species. In fact, all three of the indices we discussed are Hill numbers!
The Hill number of order $\alpha$ is the function
$\mathcal{D}_\alpha(\mathbf{p}) = \left( \sum_i p_i^{\alpha} \right)^\frac{1}{1-\alpha}$ for $\alpha \neq 1,\infty$, with
$\mathcal{D}_1 (\mathbf{p}) = \exp \left( - \sum_i p_i \ln p_i \right)$ and
$\mathcal{D}_\infty (\mathbf{p}) = \frac{1}{\max_i p_i}.$
In each of these equations, $i$ ranges over the species with non-zero abundance. Species richness is $\mathcal{D}_0$, inverse Simpson concentration is $\mathcal{D}_2$, and the Berger-Parker index is $\mathcal{D}_\infty$. $\mathcal{D}_1$ is the exponential of Shannon entropy, yet another famous diversity index. Irrespective of $\alpha$, the Hill numbers guarantee that the diversity of a community of $n$ equally abundant species is $n$.
There is no universally 'correct' choice of $\alpha$. Unless the ecological context privileges one, ecologists recommend comparing communities using their diversity profiles: the graph of $\mathcal{D}_\alpha (\mathbf{p})$ against $\alpha$. The benefit of viewing all these indices as members of a family lies in how we can interpret $\alpha$. So what does $\alpha$ mean?
The next morning, as you walk past the site, you are stunned to see that right there, nestled amongst the 8 oaks and 2 birches, is a pine! You had missed a tree, and you scramble to update your results for the new composition $\mathbf{p} = \left(8/11, 2/11, 1/11 \right)$. How much does each index change?
Diversity index | Before (8 oaks, 2 birches) | After (+1 pine) | Change | Percentage change |
Species richness ($\mathcal{D}_0$) | 2 | 3 | + 1 | + 50% |
Inverse Simpson concentration ($\mathcal{D}_2$) | 1.47 | 1.75 | + 0.28 | + 19% |
Berger-Parker index ($\mathcal{D}_\infty$) | 1.25 | 1.375 | + 0.125 | + 10% |
Diversity has increased according to all three indices. Interestingly, the higher the $\alpha$, the lower the increase. This reveals a useful interpretation of $\alpha$: increasing it makes $\mathcal{D}_\alpha$ less sensitive to rare species. $\mathcal{D}_0$ is extremely sensitive to them: observe just one individual of a new species and it jumps up by $1$. At the other extreme, $\mathcal{D}_\infty$ depends only on the dominance of the most abundant species.
So far, we have considered how many individuals belong to each species without paying much attention to the species themselves. That is going to have to change soon. As a reward for your excellent work, you have been given the opportunity to expand your study to include insects! You survey two sites. At the first, you catalogue 5 oaks, 4 birches, and 1 monarch butterfly. At the second, you see 5 oaks, 1 birch, and 4 monarch butterflies.
$\mathbf{p^1} = (0.5, 0.4, 0.1)$ and $\mathbf{p^2} = (0.5, 0.1, 0.4)$
You are now a deft hand at diversity analysis and instead of making an arbitrary choice of $\alpha$, you compute the diversity profiles of both sites. You discover that for each value of $\alpha$, they are identical! This is because the Hill numbers, like many other conventional diversity indices, depend only on the abundances of species and not on which species they are. In other words, they are built on the assumption that each species is completely and equally distinct from every other. This can lead to strange results. For instance, to quote Leinster and Cobbold (2012), "Nonspecialists are amazed to learn that a community of six dramatically different species is said to be no more diverse than a community of six species of barnacle."
In an evolutionary sense, oaks and birches are more similar to each other than they are to butterflies. Accounting for this, we might reasonably regard the second site, in which the abundances are more evenly split between trees and insects, as more diverse. To model this nuance, we use a species similarity matrix $\mathbf{Z}$, where we fill in a number between 0 and 1 for each pair of species. We can use $\mathbf{Z}$ to model any notion of similarity. For instance, it could reflect phylogeny (proximity in an evolutionary sense), physical traits, or roles in the ecosystem.
Suppose you use
$\mathbf{Z} = \begin{pmatrix} 1 & 0.8 & 0 \\0.8 & 1 & 0 \\0 & 0 & 1\end{pmatrix}$
to model oaks and birches being more similar to each other than they are to the monarch butterfly (the number 0.8 is purely illustrative). Instead of the abundance of a species, we now think of its ordinariness: its mean similarity to an individual chosen at random. The ordinariness of the $i$th species is $(Zp)_i = \sum_j Z_{ij}p_j$. A species can be quite ordinary despite being rare if it is similar to species that are common. For instance, in the second site, while birch only has a relative abundance of 0.1, it is very similar to the highly abundant oak. Its ordinariness is
$ 1 \times 0.1 + 0.8 \times 0.5 + 0 \times 0.4 = 0.5 $,
making it more ordinary than the butterfly, whose ordinariness and abundance are both 0.4.
To incorporate species similarity into Simpson concentration, we calculate the expected similarity of two random individuals — $\sum_{ij}Z_{ij}p_ip_j$ — instead of the probability that they are of the same species. Similarity-aware inverse Simpson concentration is $\mathcal{D}^\mathbf{Z}_2 (\mathbf{p}) = 1/\sum_{ij}Z_{ij}p_ip_j$. For your two sites, $\mathcal{D}^\mathbf{Z}_2 (\mathbf{p^1}) \approx 1.35$ and $\mathcal{D}^\mathbf{Z}_2 (\mathbf{p^2})=2$, reflecting the idea that all other things being equal, sites with more even shares of trees and butterflies are more diverse than those with mainly different species of tree.
Likewise, we can modify the other Hill numbers to define the similarity-aware Hill numbers (Leinster and Cobbold (2012)).
$\mathcal{D}^\mathbf{Z}_\alpha(\mathbf{p}) = \left( \sum_i p_i (Zp)_i^{\alpha-1} \right)^\frac{1}{1-\alpha}$ for $\alpha \neq 1,\infty$, with
$\mathcal{D}^\mathbf{Z}_1 (\mathbf{p}) = \exp \left( - \sum_i p_i \ln (Zp)_i \right)$ and
$\mathcal{D}^\mathbf{Z}_\infty (\mathbf{p}) = \frac{1}{\max_i (Zp)_i}.$
To go back to the simpler model of completely distinct species, we can set $\mathbf{Z}$ to the identity matrix: 1 on the diagonal and 0 elsewhere. This illustrates that there is nothing particularly special about it and that using it is a deliberate modelling choice.
The importance of measuring diversity extends far beyond ecology. Whether we are studying species in a forest, occupations in a city, or opinions in a population, we must make decisions about the importance of rare categories and the similarity between categories. These decisions need to be made carefully, and we must remember that mathematical methods that appear to not require them are simply making them for us.
Rohit Sahasrabuddhe is a postgraduate student in Oxford Mathematics.
References
Hurlbert, S. H. (1971). The nonconcept of species diversity: A critique and alternative parameters. Ecology, 52(4), 577-586. https://doi.org/10.2307/1934145
Hill, M. O. (1973). Diversity and evenness: A unifying notation and its consequences. Ecology, 54(2), 427-432. https://doi.org/10.2307/1934352
Leinster, T. and Cobbold, C. A. (2012). Measuring diversity: The importance of species similarity. Ecology, 93(3), 477-489. https://doi.org/10.1890/10-2402.1
Leinster, T. (2021). Entropy and diversity: The axiomatic approach. Cambridge University Press. https://doi.org/10.1017/9781108963558, https://arxiv.org/abs/2012.02113