Heterogeneity in meta-analysis: Q, τ², I² and prediction intervals

Heterogeneity is the variation in the true effects across the studies of a meta-analysis, over and above what chance alone would produce. It matters because it determines how far a single pooled effect can be read as "the" effect. Whether studies differ because of their populations, interventions, measures, or designs is exactly what a meta-analysis can help to uncover.

Observed variation is not the same as true variation

The effect sizes you see in a forest plot vary for two reasons: the true effects differ between studies, and each study estimates its own effect with error. If all studies shared the same true effect, the observed effects would still not be identical. Quantifying heterogeneity therefore means separating the true variation from the within-study error (Borenstein et al., 2021). The statistics below do this in different ways, and answer different questions.

The main statistics

• Q (Cochran's Q) answers: "Is there more variation than chance would explain?" It is a test of the hypothesis that all studies share one true effect. It has low power when studies are few or small, so a non-significant result must not be taken as evidence of no heterogeneity (Higgins et al., 2019). Conversely, its p-value depends on the number of studies, so with many studies even modest excess dispersion can look significant (Borenstein et al., 2021).
• τ² (tau-squared) and τ answer: "How much do the true effects vary?" τ² is the estimated variance of the true effects, in the squared units of the effect size; τ is its square root, the estimated standard deviation of the true effects (Higgins et al., 2019). It is an absolute measure.
• I² answers: "What proportion of the observed variation reflects true differences?" Introduced by Higgins and Thompson (2002), it is a relative measure: the share of the total observed variance that is not sampling error. It is easy to read, which is why it became the most widely reported heterogeneity statistic (Higgins et al., 2003).
• The prediction interval answers: "Where might the true effect of a new, similar study fall?" It describes the spread of the true effects, while the confidence interval around the pooled effect only describes the uncertainty about its mean (Higgins et al., 2009; Higgins et al., 2019).

For the formulas of Q and I², see Methods and formulas; for the step-by-step context, see Assessing heterogeneity.

What I² does not tell you

I² is often read as "how much heterogeneity there is". That is a misreading: because it is a proportion, it does not tell you how much the true effects vary in absolute terms (Borenstein et al., 2017, 2021).

• The common labels of about 25%, 50%, and 75% for "low", "moderate", and "high" heterogeneity cannot be used to classify the amount of dispersion. Borenstein and colleagues show examples where an analysis with a smaller I² has the wider range of true effects, because I² also depends on the precision of the studies.
• The Cochrane Handbook gives rough bands for I² in trials but warns that thresholds can be misleading, because the importance of inconsistency depends on several factors (Higgins et al., 2019).
• When the studies are very precise, I² can be high even if the true effects differ only slightly; when they are imprecise, I² can be low even if the true effects differ a lot.

What I² does show is the relationship between the observed and the true variation: if it is zero, all the observed variance is attributable to sampling error. To describe the absolute spread, report τ (or τ²) and a prediction interval, in the units of the effect size.

Few studies, uncertain estimates

All of these statistics are estimated with uncertainty, and the uncertainty is large when studies are few (Higgins et al., 2019). Report I² with an interval where possible (Higgins & Thompson, 2002), avoid simple thresholds to "diagnose" heterogeneity, and treat prediction intervals with caution: with few studies they can look spuriously wide or narrow. The Handbook encourages them when the number of studies is reasonable (for example more than ten) and there is no clear funnel plot asymmetry. See also How many studies do you need?.

What to do about heterogeneity

• Describe it: report Q, τ or τ², I² (with its limits in mind), and a prediction interval, not only the pooled effect and its confidence interval.
• Do not treat the model as a cure: a random-effects model accommodates heterogeneity but does not make it disappear; it is always preferable to explore its causes, when there are enough studies (Higgins et al., 2019).
• Explore it with moderators: subgroup analysis and meta-regression, ideally pre-specified before looking at the results.
• Judge whether pooling makes sense: substantial heterogeneity can be informative rather than a defect, but it changes how the pooled effect should be interpreted.

Learn this in depth

Choosing how to estimate τ², reading prediction intervals, deciding when heterogeneity is worth explaining, and reporting it correctly involve trade-offs that go beyond this overview. They are covered step by step in the ActiLab courses and manual, currently in preparation. Join the newsletter to be notified.

References

Article Borenstein, M., Higgins, J. P. T., Hedges, L. V., & Rothstein, H. R. (2017). Basics of meta-analysis: I2 is not an absolute measure of heterogeneity. Research Synthesis Methods, 8(1), 5–18. https://doi.org/10.1002/jrsm.1230

Book Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021). Introduction to meta-analysis (2nd ed.). Wiley.

Article Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558. https://doi.org/10.1002/sim.1186

Article Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327(7414), 557–560. https://doi.org/10.1136/bmj.327.7414.557

Article Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009). A re-evaluation of random-effects meta-analysis. Journal of the Royal Statistical Society: Series A (Statistics in Society), 172(1), 137–159. https://doi.org/10.1111/j.1467-985X.2008.00552.x

Book Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (Eds.). (2019). Cochrane handbook for systematic reviews of interventions (2nd ed.). John Wiley & Sons. Handbook updated online available at https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current