Criteria for the Quality Assessment of Meta-Analyses on Soil Organic Carbon

Meta-analyses are increasingly used to quantitively synthesise primary research in soil and agricultural sciences. Why we, as the soil scientific community, need to look more closely into this method and must acknowledge that not all available meta-analyses are trustworthy, is explained down below.

Importance of Meta-Analyses
Meta-analysis was first defined by Glass in 1976 as “the statistical analysis of a large collection of results for the purpose of integrating these findings”1. It not only supports the structuring of the increasing amount of information2 but also offers tools to process information with increased precision and reliability2,3. Meta-analysis was developed to facilitate quantitative synthesis and provide evidence based information4 to relevant stakeholders and decision-makers.

Why standardization is necessary
Scientific literature is often used for environmental management decision making5. Especially documents that synthesize the results of multiple studies and peer-reviewed publications are primary sources of information6. Although science is not the only factor which is influencing policy decisions, there have been cases in which scientific findings have had crucial impacts on policy changes7. Therefore, researchers are obligated to ensure that their evidence reviews (such as meta-analyses) accurately reflect the primary evidence base and are reliable and transparent8

Why our study was needed
Since 2000, the number of articles published on soil organic carbon (SOC) has increased yearly. Climate change is pushing the scientific community to search for mitigation and adaption opportunities in numerous ways, such as through agronomic practices. Carbon sequestration in soils has gained increased resonance on the EU political agenda - especially since the launch of “4 per mille initiative” at COP21.

Simultaneously, meta-analyses on overall agricultural issues are getting more and more popular. In the last 10 years, between 2011 and 2021, the number of agricultural meta-analyses published per year increased by more than 600% (17 and 124 meta-analyses, respectively). Previous reviews by Philibert et al. (2012), Beillouin et al. (2019) and Krupnik et al. (2019) show that many of these agronomic meta-analyses are of insufficient quality. This is due to the lack of experience with the meta-analytical methodology, which has only become more prevalent in this field in the last 10-20 years. Moreover, there are no specific guidelines available to facilitate standardized meta-analyses in soil and agricultural research.

All this provides reason to assume that core criteria, necessary in producing meta-analyses, are not clear to many researchers in the field of agricultural and soil sciences. In consequence, poor quality meta-analyses are being published. Moreover, the interest in SOC sequestration and subsequent increase in related publications raises the question whether there are meta-analyses synthesising this knowledge. If so, does their quality show similar trends to agricultural meta-analyses reviewed in the past by Philibert et al. (2012), Beillouin et al. (2019) and Krupnik et al. (2019).

Scope and Method
We conducted a study that aimed to quantitatively analyse the quality of available meta-analyses, studying the effects of different management practices on SOC. We found 31 studies, published between the years 2005-2020, which fit our inclusion and exclusion criteria. We further compiled a quality criteria-set, suitable for soil and agricultural sciences, by adapting existing meta-analytical guidelines from other disciplines. The criteria-set consists of 17 quality criteria, which are structured according to three groups, namely “Literature search and inclusion/exclusion criteria”, “Meta-analysis” and “Results and database presentation”. The set is supported by a scoring scheme, which allows a quantitative analysis. The 31 retrieved meta-analyses were structured according to 11 management categories, as e.g. tillage, cover crops, residue, biochar etc., which enabled us to assess the state-of-knowledge on these categories.

Major deficiencies were found in literature search, description of moderators, the use of standard metrics for effect size calculation, independence of effect sizes, standard deviation extraction for each study, weighting by the inverse of variance and display of results and full database. Only one out of 31 SOC meta-analyses, which studied the effects of no-till/reduced tillage compared to conventional tillage, was found to be of high quality. Therefore, improved meta-analyses on the effects of e.g., organic agriculture, biochar, fertilization or crop diversification on SOC are urgently needed.

Figure 1 Results of the quality assessment of 31 meta-analyses, studying the effects of different agricultural management practices on SOC. For each group, two important criteria and the corresponding results are displayed. Pie charts show the percentage of studies that fulfilled the criterium.

Quality is insufficient
We conclude that, despite the efforts over the last 15 years, the quality of meta-analyses on SOC research is still insufficient. In order for the scientific community to provide high quality synthesis work and to make advancements in the sustainable management of agricultural soils, we need to adapt rigorous methodologies of meta-analysis. As the information presented in synthesis research articles is frequently used by decision makers, this might also have negative impacts on policymaking. It is high time that the agricultural and soil scientific community adapts rigorous meta-analytical methodologies and improves the quality of its output.

Related and upcoming work
The complete review article, comprising the detailed quality criteria-set and all results, is currently under review. Please contact Julia Fohrafellner ( for updates. Moreover, the deliverable D7.4 will include a section on meta-analyses in soil research, including some findings of the manuscript and information on what to look out for when conducting a meta-analysis in soil and agricultural research.

More details on the study can be accessed

Further reading

Gurevitch, J., Koricheva, J., Nakagawa, S., Stewart, G., 2018. Meta-analysis and the science of research synthesis. Nature 555, 175–182.

O'Dea, R.E., Lagisz, M., Jennions, M.D., Koricheva, J., Noble, D.W., Parker, T.H., Gurevitch, J., Page, M.J., Stewart, G., Moher, D. and Nakagawa, S. (2021), Preferred reporting items for systematic reviews and meta-analyses in ecology and evolutionary biology: a PRISMA extension. Biol Rev. doi:10.1111/brv.12721

Koricheva, J., Gurevitch, J., Mengersen, K. (Eds.), Handbook of Meta-Analysis in Ecology and Evolution. Princton University Press, Princeton
Free online trainings:

About systematic reviews, which incorporate meta-analysis:

Collaboration for Environmental Evidence. 2018. Guidelines and Standards for Evidence synthesis in Environmental Management. Version 5.0 (AS Pullin, GK Frampton, B Livoreil & G Petrokofsky, Eds)


  1. Glass, G. V: Primary, Secondary, and Meta-Analysis of Research, Educ. Res., 5, 3–8,, 1976.
  2. Koricheva, J. and Gurevitch, J.: Uses and misuses of meta-analysis in plant ecology, J. Ecol., 102, 828–844,, 2014.
  3. Nakagawa, S. and Cuthill, I. C.: Effect size, confidence interval and statistical significance: A practical guide for biologists, Biol. Rev., 82, 591–605,, 2007.
  4. Stewart, G.: Meta-analysis in applied ecology, Biol. Lett., 6, 78–81,, 2010.
  5. Dicks, L. v., Walsh, J. C., and Sutherland, W. J.: Organising evidence for environmental management decisions: A “4S” hierarchy, Trends Ecol. Evol., Vol. 29, Issue 11, Elsevier Ltd., pp. 607–613,, 1 November 2014.
  6. Seavy, N. E. and Howell, C. A.: How can we improve information delivery to support conservation and restoration decisions?, Biodivers. Conserv., 19, 1261–1267,, 2010.
  7. Pullin, A. S. and Knight, T. M.: Science informing Policy - A health warning for the environment, Environ. Evid., 1,, 19 December 2012.
  8. O’Leary, B. C., Kvist, K., Bayliss, H. R., Derroire, G., Healey, J. R., Hughes, K., Kleinschroth, F., Sciberras, M., Woodcock, P., and Pullin, A. S.: The reliability of evidence review methodology in environmental science and conservation, Environ. Policy Gov., 64, 75–82,, 2016.
  9. Philibert, A., Loyce, C., and Makowski, D.: Assessment of the quality of meta-analysis in agronomy, Agric. Ecosyst. Environ., 148, 72–82,, 2012.
  10. Beillouin, D., Ben-Ari, T., and Makowski, D.: Evidence map of crop diversification strategies at the global scale, Environ. Res. Lett.,, 10 January 2019.
  11. Krupnik, T. J., Andersson, J. A., Rusinamhodzi, L., Corbeels, M., Shennan, C., and Gérard, B.: Does size matter? a critical review of meta-analysis in agronomy, Exp. Agric., 55, 200–229,, 2019.