A re-analysis of the Cochrane Library data: the dangers of unobserved heterogeneity in meta-analyses
Kontopantelis, E., Springate, D. A. & Reeves, D.
Heterogeneity has a key role in meta-analysis methods and can greatly affect conclusions. However, true levels of heterogeneity are unknown and often researchers assume homogeneity. We aim to: a) investigate the prevalence of unobserved heterogeneity and the validity of the assumption of homogeneity; b) assess the performance of various meta-analysis methods; c) apply the findings to published meta-analyses. Methods and findings We accessed 57,397 meta-analyses, available in the Cochrane Library in August 2012. Using simulated data we assessed the performance of various meta-analysis methods in different scenarios. The prevalence of a zero heterogeneity estimate in the simulated scenarios was compared with that in the Cochrane data, to estimate the degree of unobserved heterogeneity in the latter. We re-analysed all meta-analyses using all methods and assessed the sensitivity of the statistical conclusions. Levels of unobserved heterogeneity in the Cochrane data appeared to be high, especially for small meta-analyses. A bootstrapped version of the standard DerSimonian-Laird approach performed better both in detecting heterogeneity and providing more accurate overall effect estimates. Re-analysing all meta-analyses with this method we found that 17-20% of the statistical conclusions changed, when heterogeneity was detected with the standard model and ignored. The rates were much lower when the standard method did not detect heterogeneity or took it into account, between 1% and 3%. Conclusions: When evidence for heterogeneity is lacking, standard practice is to assume homogeneity and apply a simpler fixed-effect meta-analysis. We find that assuming homogeneity often results in a misleading analysis, since heterogeneity is very likely present but undetected. Our new method represents a small improvement but the problem largely remains, especially for very small meta-analyses. One solution is to test the sensitivity of the meta-analysis conclusions to assumed moderate and large degrees of heterogeneity. Equally, whenever heterogeneity is detected, it should not be ignored. Keywords: Meta-analysis; heterogeneity; homogeneity; DerSimonian-Laird; bootstrap; Cochrane library; Cochrane Database of Systematic Reviews; sensitivity analysis.