Criteria for the Quality Assessment of Meta-Analyses on Soil Organic Carbon

Importance of Meta-Analyses
Meta-analysis was first defined by Glass in 1976 as “the statistical analysis of a large collection of results for the purpose of integrating these findings”¹. It not only supports the structuring of the increasing amount of information² but also offers tools to process information with increased precision and reliability^2,3. Meta-analysis was developed to facilitate quantitative synthesis and provide evidence based information⁴ to relevant stakeholders and decision-makers.

Why standardization is necessary
Scientific literature is often used for environmental management decision making⁵. Especially documents that synthesize the results of multiple studies and peer-reviewed publications are primary sources of information⁶. Although science is not the only factor which is influencing policy decisions, there have been cases in which scientific findings have had crucial impacts on policy changes⁷. Therefore, researchers are obligated to ensure that their evidence reviews (such as meta-analyses) accurately reflect the primary evidence base and are reliable and transparent⁸. 

Why our study was needed
Since 2000, the number of articles published on soil organic carbon (SOC) has increased yearly. Climate change is pushing the scientific community to search for mitigation and adaption opportunities in numerous ways, such as through agronomic practices. Carbon sequestration in soils has gained increased resonance on the EU political agenda - especially since the launch of “4 per mille initiative” at COP21.

Simultaneously, meta-analyses on overall agricultural issues are getting more and more popular. In the last 10 years, between 2011 and 2021, the number of agricultural meta-analyses published per year increased by more than 600% (17 and 124 meta-analyses, respectively). Previous reviews by Philibert et al. (2012), Beillouin et al. (2019) and Krupnik et al. (2019) show that many of these agronomic meta-analyses are of insufficient quality. This is due to the lack of experience with the meta-analytical methodology, which has only become more prevalent in this field in the last 10-20 years. Moreover, there are no specific guidelines available to facilitate standardized meta-analyses in soil and agricultural research.

All this provides reason to assume that core criteria, necessary in producing meta-analyses, are not clear to many researchers in the field of agricultural and soil sciences. In consequence, poor quality meta-analyses are being published. Moreover, the interest in SOC sequestration and subsequent increase in related publications raises the question whether there are meta-analyses synthesising this knowledge. If so, does their quality show similar trends to agricultural meta-analyses reviewed in the past by Philibert et al. (2012), Beillouin et al. (2019) and Krupnik et al. (2019).

Scope and Method
We conducted a study that aimed to quantitatively analyse the quality of available meta-analyses, studying the effects of different management practices on SOC. We found 31 studies, published between the years 2005-2020, which fit our inclusion and exclusion criteria. We further compiled a quality criteria-set, suitable for soil and agricultural sciences, by adapting existing meta-analytical guidelines from other disciplines. The criteria-set consists of 17 quality criteria, which are structured according to three groups, namely “Literature search and inclusion/exclusion criteria”, “Meta-analysis” and “Results and database presentation”. The set is supported by a scoring scheme, which allows a quantitative analysis. The 31 retrieved meta-analyses were structured according to 11 management categories, as e.g. tillage, cover crops, residue, biochar etc., which enabled us to assess the state-of-knowledge on these categories.

Results
Major deficiencies were found in literature search, description of moderators, the use of standard metrics for effect size calculation, independence of effect sizes, standard deviation extraction for each study, weighting by the inverse of variance and display of results and full database. Only one out of 31 SOC meta-analyses, which studied the effects of no-till/reduced tillage compared to conventional tillage, was found to be of high quality. Therefore, improved meta-analyses on the effects of e.g., organic agriculture, biochar, fertilization or crop diversification on SOC are urgently needed.

Figure 1 Results of the quality assessment of 31 meta-analyses, studying the effects of different agricultural management practices on SOC. For each group, two important criteria and the corresponding results are displayed. Pie charts show the percentage of studies that fulfilled the criterium.

Quality is insufficient
We conclude that, despite the efforts over the last 15 years, the quality of meta-analyses on SOC research is still insufficient. In order for the scientific community to provide high quality synthesis work and to make advancements in the sustainable management of agricultural soils, we need to adapt rigorous methodologies of meta-analysis. As the information presented in synthesis research articles is frequently used by decision makers, this might also have negative impacts on policymaking. It is high time that the agricultural and soil scientific community adapts rigorous meta-analytical methodologies and improves the quality of its output.

Related and upcoming work
The complete review article, comprising the detailed quality criteria-set and all results, is currently under review. Please contact Julia Fohrafellner (Julia.fohrafellner@boku.ac.at) for updates. Moreover, the deliverable D7.4 will include a section on meta-analyses in soil research, including some findings of the manuscript and information on what to look out for when conducting a meta-analysis in soil and agricultural research.