Meta-analysis glossary

This is a glossary of the most important terms used in meta-analysis, reported in alphabetical order.

A

Aggregation of effect sizes: Combination of several dependent effect sizes from the same study into a single one, with rules defined in advance.
Availability, citation, cost and familiarity bias: Distortions in the retrieval of studies: those that are easily accessible (availability), highly cited (citation), free (cost) or written by authors known to the researchers (familiarity) are more likely to be found.

B

Bayesian meta-analysis: Meta-analysis that combines a prior distribution of the parameters with the data of the studies to obtain a posterior distribution. It allows results to be expressed with credible intervals and Bayes factors.

C

Coding: Procedure that consists in extracting from primary studies information necessary to perform a meta-analysis. See Coding primary studies.
Coding protocol: Document that lists the variables to extract from the primary studies and the rules for coding them. It ensures consistency across studies and coders. See Coding primary studies.
Cohen's d: Effect size for the difference between two means, expressed in units of the pooled standard deviation. See Methods and formulas.
Cohen's Kappa: Statistical measure of the degree of agreement between two raters/judges, corrected for the agreement expected by chance (for instance, it can be used to evaluate the agreement about the inclusion or exclusion of studies from a meta-analysis).
Confidence interval (CI): Range of values that are plausible for the true effect size at a given confidence level; with a 95% level, 95% of the intervals computed in repeated sampling would contain the true value. Usually a 95% CI is computed for each study effect size and for the overall (combined) effect size.
Cumulative meta-analysis: A meta-analysis that is performed first with one study, then with two studies, then with three studies, and so on, until all studies have been included (usually in chronological order), to see how the estimate changes as studies are added.

D

Deduplication: Removal of bibliographic records that refer to the same study and appear more than once because they were found in different databases. See Selecting primary studies.
Dependence between effect sizes: Correlation between effect sizes that come from the same participants, the same control group or the same laboratory. Ignoring it leads to standard errors that are too small. See Multilevel models.

E

Effect size: A measure of the magnitude of a relationship or a difference between groups. It can be based on means (raw unstandardized mean difference, standardized mean difference: Cohen's d or Hedges's g, etc.); binary data (risk ratio, odds ratio, risk difference, etc.); correlations (r); survival data (hazard ratio). Standardized measures are comparable across studies that use different scales. See Computing effect sizes.
Egger's test: Test of funnel plot asymmetry based on the regression of the standardised effect size on its precision. Begg and Mazumdar's test uses rank correlation instead. See Methods and formulas.
Exclusion criteria: Criteria, established before the search, that define which studies should be excluded from a meta-analysis. See Defining inclusion and exclusion criteria.

F

Fail-safe N: Number of unretrieved studies with a null effect that would have to be added to bring the overall effect size to non-significance. It is now discouraged as the sole indicator of publication bias. See Methods and formulas.
Fisher's Z: Transformation of the correlation r, used to analyze correlations in meta-analysis. See Methods and formulas.
Fixed-effect model: Statistical model used to combine study effect sizes. According to this model (opposed to the random-effects model), there is a true effect size common to all studies. In assigning a weight to each study, it takes into account only one source of variance: the within-study variance. See Aggregating effect sizes.
Forest plot: Plot of effect sizes (with confidence intervals) of all the studies included in the meta-analysis, together with the overall effect size.
Funnel plot: Plot of the study effect sizes against their precision; its asymmetry may indicate publication bias.

G

Gray (grey) literature: Literature produced on all levels of government, academia, business, and industry in print and electronic formats, but which is not controlled by commercial publishers (for instance theses, reports and conference proceedings). The term is also used to refer to literature that has limited dissemination; being difficult to retrieve, it is easily missed, but searching for it helps to limit publication bias. See Searching the literature.

H

HARKing: Formulating hypotheses after seeing the results (Hypothesizing After the Results are Known) and presenting them as if they had been predicted in advance.
Hazard ratio: Effect size for survival data: ratio between the instantaneous rate at which an event occurs in one group and in the other.
Hedges's g: Cohen's d with a correction for small samples. See Methods and formulas.
Heterogeneity: Variation among the true effects of the studies included in the meta-analysis, beyond what is expected from sampling error. See Heterogeneity in meta-analysis.
Homogeneity test: Test based on Cochran's Q, with the null hypothesis of a single true effect common to all studies. It has little power with few studies. See Heterogeneity in meta-analysis.

I

I²: Proportion of the observed variation among studies that is true variation rather than sampling error; a relative, not absolute, measure. See Heterogeneity in meta-analysis.
Inclusion criteria: Criteria, established before the search, that define which studies should be included in a meta-analysis. See Defining inclusion and exclusion criteria.
Individual participant data (IPD) meta-analysis: Meta-analysis based on the data of single participants rather than on the aggregated results of each study.
Inter-coder agreement: Degree to which two or more coders, working independently, assign the same codes to the same studies. It is usually assessed with Cohen's kappa or the percentage of agreement. See Coding primary studies.
Intraclass correlation: Share of the total variance attributable to differences between clusters (for example between studies) rather than within clusters. It indicates how similar the effect sizes of the same cluster are.

L

Language bias: Distortion due to searching only certain languages: studies with significant results are more often published in English, the others in local languages.
Limit meta-analysis: Method that estimates the overall effect size expected for studies of infinite precision, by extrapolating the relationship between effect size and precision.
Living review: Review that is continuously updated as new studies are published.

M

MARS: American Psychological Association guidelines for reporting a meta-analysis (Meta-Analysis Reporting Standards).
MASEM: Meta-analytic structural equation modeling: it pools the correlation matrices of the studies and fits a structural equation model to them.
Mean difference: Effect size for the comparison of two means, expressed in the original unit of the scale. It cannot be compared across studies that use different scales.
Meta-analysis: Quantitative (statistical) synthesis of the results of several studies addressing the same question. See What is a systematic review / meta-analysis?.
Meta-regression: Statistical analysis that relates the effect size to one or more moderators; it is typically used to test the effect of a continuous moderator. See Moderators: subgroup analysis vs meta-regression.
Moderator: Variable that might explain differences in the effect sizes. A categorical moderator is usually tested by a subgroup analysis, a continuous moderator by a meta-regression. See Moderators: subgroup analysis vs meta-regression.
MOOSE: Guidelines for reporting meta-analyses of observational studies (Meta-analysis Of Observational Studies in Epidemiology).
Multilevel model: Model for dependent effect sizes, for instance when a study provides more than one. See Multilevel models.
Multiple meta-regression: Meta-regression with several moderators entered at the same time, to estimate the association of each with the effect size net of the others.
Multiple outcomes, comparisons and time points: Situations in which a study provides several effect sizes because it measures several outcomes, compares several conditions with the same control group or measures the same outcome at several times. They create dependence. See Multilevel models.
Multiple publication bias: Distortion due to the same study being published more than once and risking entering the meta-analysis more than once.
Multivariate meta-analysis: Meta-analysis that analyses several correlated outcomes together, taking into account the correlation between effect sizes of the same study.

N

Network meta-analysis: Meta-analysis that compares more than two interventions using both direct and indirect comparisons. It requires transitivity and consistency. See Network meta-analysis.

O

Odds ratio: Effect size for binary data, the ratio of the odds of an event in two groups. See Methods and formulas.
Outcome reporting bias: Distortion due to reporting only the outcomes with significant or favourable results.
Outlier: Study whose effect size differs markedly from that of the others. It is called influential if excluding it substantially changes the results.

P

p-curve: Distribution of the significant p-values of a set of studies. If the effect is real the curve is right-skewed (many very small p-values); if the results come from p-hacking it tends to flatten.
p-hacking: Practice of analysing the data in different ways until a significant result is obtained. It inflates effect sizes and the false-positive rate.
PET-PEESE: Pair of meta-regressions of the effect size on the standard error (PET) or on the variance (PEESE). The intercept estimates the effect of a study with infinite precision, free of small-study bias.
Prediction interval: Interval in which the true effect of a new, similar study is expected to fall. See Heterogeneity in meta-analysis.
Primary study: An individual empirical study that is included in a meta-analysis. See Selecting primary studies.
PRISMA: Reporting guideline for systematic reviews, with a checklist and a flow diagram; the current version is PRISMA 2020. See PRISMA flow diagram and reporting guideline.
Publication bias: Tendency for studies with statistically significant results to be published more easily, so that the available literature may not represent all the studies conducted; published studies thus differ systematically from unpublished ones (gray literature). Different methods for evaluating publication bias are available (e.g., funnel plot; Rosenthal's Fail-safe N; Orwin's Fail-safe N; Duval and Tweedie's Trim and Fill method; etc.). See Evaluating publication bias.

Q

Q (Cochran's Q): Test of the hypothesis that all studies share the same true effect. See Heterogeneity in meta-analysis.
Questionable research practices: Research practices that are not outright fraud but distort results, such as p-hacking, HARKing and selective outcome reporting.

R

Radial plot: Alternative to the funnel plot that shows the standardised effect size against precision, so that the studies line up along a straight line.
Random-effects model: Statistical model used to combine study effect sizes. According to this model (opposed to the fixed-effect model), the true effects in the studies are assumed to have been sampled from a distribution of effects. In assigning a weight to each study, it takes into account two sources of variance: within-study variance and between-studies variance. See Aggregating effect sizes.
Registered report: Publication format in which the protocol is peer-reviewed before data are collected and analysed; the journal commits to publishing the work regardless of the results.
Reliability generalization: Meta-analysis of the reliability coefficients (usually Cronbach's alpha) of a scale, to evaluate how much they vary across studies.
Residual heterogeneity: Heterogeneity that remains between studies after accounting for the moderators included in the model.
Risk difference: Effect size for binary data, equal to the difference between the proportion of events in the experimental group and in the control group.
Risk ratio: Effect size for binary data, the ratio of the risks of an event in two groups. See Methods and formulas.
Robust variance estimation (RVE): Method that computes reliable standard errors even when the dependence structure between effect sizes is not known precisely, using an estimator called sandwich. See Multilevel models.

S

Sandwich estimator: Estimator of the variance of the coefficients that remains valid even if the dependence model is only approximately specified.
Search strategy: Explicit and reproducible plan for identifying the primary studies to be included in a meta-analysis: databases, search strings and other sources. See Searching the literature.
Selection model: Model that estimates the overall effect size while accounting for a process that selects studies according to their result (for example their significance). Variants include Copas's model, step-function models and the three-parameter model.
Sensitivity analysis: Analysis that repeats the meta-analysis changing one choice (for example excluding a study or changing the model) to check how much the results depend on it.
Small-study effects: Tendency of small studies to report effect sizes different from (usually larger than) those of large studies. It may be due to publication bias, but also to heterogeneity or methodological differences.
Snowballing: Search technique that starts from the studies already found and examines their reference lists (backward) and the citations they received (forward). See Searching the literature.
Standard error: Standard deviation of the sampling distribution of an estimate, for example of an effect size. It measures the precision of the estimate and is the square root of its variance.
Statistical power: Probability that the analysis detects an effect that is actually present. In meta-analysis it depends on the number and precision of the studies, the size of the effect and the heterogeneity. See How many studies do you need?.
Subgroup analysis: Statistical analysis used to test the effect of a categorical moderator. See Moderators: subgroup analysis vs meta-regression.
Systematic review: Review that uses systematic and explicit methods to identify, select and appraise the studies relevant to a question. See What is a systematic review / meta-analysis?.

T

Tau (τ): Standard deviation of the true effects across studies, that is the square root of τ². It is expressed in the same unit as the effect size. See Heterogeneity in meta-analysis.
Tau squared (τ²): Estimated variance of the true effects across studies; τ is the corresponding standard deviation. See Heterogeneity in meta-analysis.
Tau-squared estimators: Methods for estimating the between-study variance in the random-effects model, including DerSimonian-Laird, maximum likelihood (ML), restricted maximum likelihood (REML), Paule-Mandel, Sidik-Jonkman, Hunter-Schmidt, Hedges and Empirical Bayes. They give different estimates, especially with few studies.
Three-level model: Multilevel model that distinguishes the sampling variance (level 1), the variance between effect sizes within the same study (level 2) and the variance between studies (level 3). See Multilevel models.
Time-lag bias: Distortion whereby studies with significant or favourable results are published earlier than the others, so the effect appears to shrink over time.
Trial sequential analysis: Sequential analysis of the meta-analysis that assesses whether the accumulated evidence is sufficient to draw conclusions, controlling the risk of false positives due to repeated testing.
Trim and fill: Method that estimates the number of studies missing because of funnel plot asymmetry, adds them and recalculates the corrected overall effect size.

U

Unit of analysis: Level at which an independent effect size is counted in the analysis, for example the study, the subgroup or the outcome. Choosing the right unit avoids counting the same participants more than once.

V

Validity generalization: Hunter and Schmidt's approach, which estimates how much of the variability between studies is due to statistical artefacts, to check whether the validity of a test generalises across settings.
Variance of the effect size: Measure of the uncertainty of a study's estimate; it determines the study's weight in the analysis. See Methods and formulas.
Vote counting: Method that, differently from meta-analysis, does not provide a quantitative summary of the results of primary studies, but is used to draw conclusions on a specific area of study by comparing the number of studies with significant results against the number of studies reporting non-significant or negative results. Since it ignores effect size and sample size, it is discouraged.

W

Weight: In meta-analysis, when study effect sizes are combined, a weight is assigned to each study. The weight assigned is usually the inverse of the variance, and its value depends on the model (fixed-effect vs. random-effects) chosen for combining the effect sizes.