Commentary
Information Management During A Complex Meta-Analysis Project: A Practical Guide for Organizing Data Extraction

Matthew Goldsmith¹, Svetlana Komarova¹
Published online: June 2025

¹Faculty of Dental Medicine and Oral Health Sciences, McGill University, Montreal, Quebec, Canada
Corresponding Author: Svetlana Komarova. Email Address: svetlana.komarova@mcgill.ca

DOI: 10.26443/mjm.v22i1.1040

Introduction

Conducting a systematic review and meta-analysis is a complex process that requires significant organizational efforts (1-3). Many steps of this process are described in detail and clearly summarized in the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) checklist (4), including steps for data extraction and coding (3) for which several specific protocols, such as TIDieR (5) and DECiMAL (6), have been developed. It is generally advised to specify data extraction items before the full text reviews (3). However, this requires a prior knowledge of all the relevant data types and dimensions which is not always possible, especially in projects with significant variability in individual study protocols, specific parameters that measure the outcomes, or methodology used in different papers. In our experience, such projects involved studies analyzing data from a long time period with historical changes in experimental approaches (7-11), as well as projects aimed at summarizing the findings from basic science studies that use a variety of study designs aimed to understand the same or similar outcomes (12-14). Importantly, this variability often only becomes apparent after the full text inclusion is complete, making it difficult to prepare for this challenge. Having completed multiple meta-analytic projects on a diverse number of subjects (8, 10, 12-14), we report the protocol we developed to methodically screen the full text studies and update the coding table for extracting parameters for meta-analysis.

The Place of this Protocol in the Meta-Analysis Project

In meta-analytical projects, the authors expect that quantitative data are available for their research question. After screening the studies retrieved from the relevant databases, the researchers arrive at a set of full-text papers that can be small (7), reasonable with <studies (13-15), or quite large with 50-200 studies (8-10, 12). Even when the number of papers is small, each paper may contain multiple independent datasets of interest, such as measurements in different age, sex, or disease populations (13, 15). Further, the research question may be answered with a variety of different outcomes (7, 9, 15), dramatically increasing the complexity of information management. While general guidelines suggest that the next step is data extraction into the pre-defined coding tables, in many complex projects it is time-intensive, frustrating, and premature to engage in extraction of outcome measures at this stage. Instead, we suggest employing the following strategy that allows to update/develop coding tables focusing on the information most important for the meta-analysis project, while identifying incompatible studies prior to extraction of outcomes. This approach is important to prevent potential confirmation bias; that is, removing data that do not fit the hypothesis.

STEP 1: Record key study characteristics

With selected full texts retrieved, the key identifying characteristics of each study should be recorded such as: the author, year it was conducted, country where it was conducted, sample size, study type, population characteristics, etc. If PICO framework (P: Patient, problem or population, I: Intervention, C: Comparison, control or comparator; O: Outcome) (16) is used, this step focuses on the P of PICO. Assessment of study quality should be initiated as planned, in general using one of the established quality assessment tools that addresses potential biases relevant to their research question (2, 17). In addition, potential shortcomings of reporting specific to the current project may become apparent and can be included in the updated quality assessment checklist. The table resulting from this stage of analysis provides a general overview of the available information. Table 1 generated from our study of bone health in spacefaring rodents (9) illustrates how details important for the research project, such as the space mission, the year it was conducted, mission duration, the number of samples, and the bones measured were summarized. Please note, that while the coding table designed prior to the start of the project would always include general information (author, year, space mission), other details (such as the fact that different bone were analyzed in different studies) may not be known prior to full text analysis and are included at this stage.

STEP 2: Identify studies reporting duplicate/overlapping information

In a systematic review, the reviewers may encounter several articles that report the outcomes of the same studies/populations that may not be apparent without careful analysis of full text articles. This may happen when the same results are reported together with different aspects of the whole study (8),when the updates for the same population are published during continuous recruitment (8), or when the same dataset is used for normalizing in the studies focused on different outcomes (18). While some authors clearly identify when the same findings are reported in subsequent papers, more often publications are treated as self-contained, and the reviewer needs to ensure that the datasets included in the meta-analysis are independent. Analyzing the table produced in the previous step helps to identify potentially overlapping studies that were conducted in the same year or location, have the same sample size, report identical measurement outcomes, etc. In our example (Table 1), the results of three space missions, STS-131, Bion-M1 and CRS10, were reported in multiple manuscripts. Comparison of the reported measurements demonstrates that Maupin et al. 2019 (19) and Dadwal et al. 2019 (20) report the outcomes in the same set of bones, while other studies report the parameters for different bones from the animals that went on the same missions. When two articles report overlapping data, one of the articles should be excluded from the meta-analysis based on a pre-defined criteria (for example, the one that includes less information, exhibits lower reporting quality, or achieves a lower quality score).

STEP 3: Select parameters for analysis

Now we can determine which parameters have been reported in enough studies for meta-analysis and which measurements are compatible. If the dataset contains many different parameters, we suggest generating a new table where each parameter has its own column (Table 2). The same parameters can be given different names in different studies (due to different conventions in different groups or historical changes in nomenclature). Similarly, the same name can be given to different parameters. The same or similar parameters can also be measured using different methods. Keeping track of parameter definitions, measurement units and techniques allows to distinguish between different effect sizes. The table generated at this step allows to determine the number of instances the study-level outcomes are reported for each parameter, and thus the feasibility of a meta-analysis for different parameters. From the example in the Table 2, there is enough studies for meta-analysis of bone volume fraction in humerus, tibia, and femur and of marrow area in femur. However, there are only two datasets reporting cortical porosity in different bones. Thus, cortical porosity can be removed from the list of parameters to extract for analysis.

After candidate parameters for meta-analysis with at least three study-level outcomes are selected, we can determine whether measurements reported in different studies are compatible by comparing their measurement units and techniques. Ideally, the outcomes in all studies were obtained using the same methodology and reported on the same scale. However, in practice this may not be so, since meta-analyses often include papers performed at different times and places. Researcher must exhibit autonomous judgement for excluding certain studies as not compatible or choosing to combine the data while keeping track of implied assumptions and looking for opportunities to test them. For example, the relative amount of bone tissue (Bone volume/tissue volume, BV/TV %) can be obtained from 2-dimentional histological measurements or from 3-dimentional computed tomography. The studies reporting these outcomes could be designated as non-compatible, or the researchers can decide to combine them and explore in subgroup analysis if this introduces significant heterogeneity (7). At the end of this step, the set of parameters that will be included in meta-analysis is determined. All articles originally selected that do not report the chosen parameters or report incompatible outcomes (such as Tavella et al. 2012 in Table 2) will be excluded.

STEP 4: Define important covariates

With a good idea about the content of selected studies, now the reviewer should update the coding table for the covariates that can potentially affect the outcome. While key parameters can generally be identified before the start of the project, now, when the differences between the included studies are recorded, additional covariates may come into focus. Such covariates may be relevant to the specifics of included studies, such as genetic variants, surgery, treatment, measurement techniques, etc. Some covariates may make the data incompatible with the rest of the dataset, for example when something had happened in a small subset of studies and is biologically plausible to create significant difference in the outcome. The reviewer may decide to remove such articles as incompatible. The extraction table should be updated to include all the important covariates, which then can be analyzed during the meta-analytic study. The quality assessment checklist can also be updated for the quality of reporting of additional covariates.

STEP 5: Extract relevant information, assess compatibility, and remove inconsistent reporting

Finally, the researcher has a template for extracting information regarding participants (step 1-2), outcomes (step 3) and covariates (step 4). It is a good practice to test this form by two independent reviewers, who later discuss disagreements and then adjust the form if necessary. Using the finalized template, the information can be extracted from all the papers, now together with the quantitative data for the study level outcomes and associated variance. Next, when the quantitative study-level outcomes are available and necessary data transformation is performed to ensure that units of measurements are consistent, we recommend checking for drastically different values. While it is wrong to exclude the study based on the difference in reported values, this may also be an indication of data extraction error. If no typing error was found, one should also check for potential discrepancy between the description and values given in the original paper, as mistakes in reporting do happen (we have encountered a study reporting a parameter as “normal”, yet the value was at least 1000-fold higher than normal). Attempts should be made to contact the author(s) and clarify these discrepancies. If efforts to contact the original authors are unsuccessful, it is safer to exclude the measured outcome or study than to make additional assumptions regarding the values. Final additions to the quality assessment checklist can now be made for the quality of the outcome reporting. Completing this step will result in having a dataset ready for quantitative synthesis using the meta-analytic model of your choice.

Conclusion

We describe a protocol for information management during meta-analysis projects addressing research questions that involve variability in individual study approaches or methodologies that measure the outcomes. Such questions are more likely to arise in meta-analysis of basic science studies (2), which often employ exploratory methods rather than standardized clinical outcomes, and thus require a more open approach for managing the information to be included in the meta-analysis. Because of an open information management, the reviewers should at every step be cognizant of potential bias in their assessment, and carefully record the reasons for the exclusion of an individual study from the meta-analysis. Our protocol provides methodical guidelines for such an assessment, highlights potential difficulties that the reviewers may encounter, and suggests ways to address these difficulties in a clear unbiased way.

Declaration of Conflicting Interests

The Authors declare that there is no conflict of interest.

References

1. Borenstein M, Higgins JP. Meta-analysis and subgroups. Prevention science : the official journal of the Society for Prevention Research. 2013;14(2):134-43. DOI:10.1007/s11121-013-0377-7
2. Mikolajewicz N, Komarova SV. Meta-Analytic Methodology for Basic Research: A Practical Guide. Front Physiol. 2019;10:203. DOI:10.3389/fphys.2019.00203
3. Johnson BT, Hennessy EA. Systematic reviews and meta-analyses in the health sciences: Best practice methods for research syntheses. Social science & medicine (1982). 2019;233:237-51. DOI:10.1016/j.socscimed.2019.05.035
4. Page MJ, Moher D, McKenzie JE. Introduction to preferred reporting items for systematic reviews and meta-analyses 2020 and implications for research synthesis methodologists. Research synthesis methods. 2022;13(2):156-63. DOI:10.1002/jrsm.1535
5. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ (Clinical research ed). 2014;348:g1687. DOI:10.1136/bmj.g1687
6. Pedder H, Sarri G, Keeney E, Nunes V, Dias S. Data extraction for complex meta-analysis (DECiMAL) guide. Systematic reviews. 2016;5(1):212. DOI:10.1186/s13643-016-0368-4
7. Steer K, Stavnichuk M, Morris M, Komarova SV. Bone Health in Patients With Hematopoietic Disorders of Bone Marrow Origin: Systematic Review and Meta- Analysis. J Bone Miner Res. 2017;32(4):731-42. DOI:10.1002/jbmr.3026
8. Stavnichuk M, Mikolajewicz N, Corlett T, Morris M, Komarova SV. A systematic review and meta-analysis of bone loss in space travelers. NPJ Microgravity. 2020;6:13. DOI:10.1038/s41526-020-0103-2
9. Fu J, Goldsmith M, Crooks SD, Condon SF, Morris M, Komarova SV. Bone health in spacefaring rodents and primates: systematic review and meta-analysis. NPJ Microgravity. 2021;7(1):19. DOI:10.1038/s41526-021-00147-7
10. Goldsmith M, Crooks SD, Condon SF, Willie BM, Komarova SV. Bone strength and composition in spacefaring rodents: systematic review and meta-analysis. NPJ Microgravity. 2022;8(1):10. DOI:10.1038/s41526-022-00195-7
11. Moussa MS, Goldsmith M, Komarova SV. Craniofacial Bones and Teeth in Spacefarers: Systematic Review and Meta-analysis. JDR Clin Trans Res. 2022:23800844221084985. DOI:10.1177/23800844221084985
12. Mikolajewicz N, Mohammed A, Morris M, Komarova SV. Mechanically stimulated ATP release from mammalian cells: systematic review and meta-analysis. J Cell Sci. 2018;131(22). DOI:10.1242/jcs.223354
13. McKee TJ, Perlman G, Morris M, Komarova SV. Extracellular matrix composition of connective tissues: a systematic review and meta-analysis. Sci Rep. 2019;9(1):10542. DOI:10.1038/s41598-019-46896-0
14. Dsouza C, Komarova SV. Characterization of Potency of the P2Y13 Receptor Agonists: A Meta-Analysis. Int J Mol Sci. 2021;22(7). DOI:10.3390/ijms22073468
15. Mikolajewicz N, Bishop N, Burghardt AJ, Folkestad L, Hall A, Kozloff KM, et al. HR-pQCT Measures of Bone Microarchitecture Predict Fracture: Systematic Review and Meta-Analysis. J Bone Miner Res. 2020;35(3):446-59. DOI:10.1002/jbmr.3901
16. Schardt C, Adams MB, Owens T, Keitz S, Fontelo P. Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Med Inform Decis Mak. 2007;7:16. DOI:10.1186/1472-6947-7-16
17. Buccheri RK, Sharifi C. Critical Appraisal Tools and Reporting Guidelines for Evidence-Based Practice. Worldviews Evid Based Nurs. 2017;14(6):463-72. DOI:10.1111/wvn.12258
18. Mikolajewicz N, Sehayek S, Wiseman PW, Komarova SV. Transmission of Mechanical Information by Purinergic Signaling. Biophys J. 2019;116(10):2009-22. DOI:10.1016/j.bpj.2019.04.012
19. Maupin KA, Childress P, Brinker A, Khan F, Abeysekera I, Aguilar IN, et al. Skeletal adaptations in young male mice after 4 weeks aboard the International Space Station. npj Microgravity. 2019;5(1):21. DOI:10.1038/s41526-019-0081-4
20. Dadwal UC, Maupin KA, Zamarioli A, Tucker A, Harris JS, Fischer JP, et al. The effects of spaceflight and fracture healing on distant skeletal sites. Scientific Reports. 2019;9(1):11419. DOI:10.1038/s41598-019-47695-3
21. Lloyd SA, Morony SE, Ferguson VL, Simske SJ, Stodieck LS, Warmington KS, et al. Osteoprotegerin is an effective countermeasure for spaceflight-induced bone loss in mice. Bone. 2015;81:562-72. DOI:10.1016/j.bone.2015.08.021
22. Ortega AM, Bateman TA, Livingston EW, Paietta RC, Gonzalez SM, Stodieck LS, et al. Spaceflight Related Changes in Structure and Strength of Mouse Trabecular and Cortical Bone From the STS-118 Space Shuttle Mission. Proceedings of ASME 2013 Summer Bioengineering Conference; Sunriver, Oregon, USA. V01AT08A0052013.
23. Tavella S, Ruggiu A, Giuliani A, Brun F, Canciani B, Manescu A, et al. Bone turnover in wild type and pleiotrophin-transgenic mice housed for three months in the International Space Station (ISS). PLoS One. 2012;7(3):e33179. DOI:10.1371/journal.pone.0033179
24. Blaber EA, Dvorochkin N, Lee C, Alwood JS, Yousuf R, Pianetta P, et al. Microgravity induces pelvic bone loss through osteoclastic activity, osteocytic osteolysis, and osteoblastic cell cycle inhibition by CDKN1a/p21. PLoS One. 2013;8(4):e61372. DOI:10.1371/journal.pone.0061372
25. Blaber EA, Dvorochkin N, Torres ML, Yousuf R, Burns BP, Globus RK, et al. Mechanical unloading of bone in microgravity reduces mesenchymal and hematopoietic stem cell-mediated tissue regeneration. Stem Cell Research. 2014;13(2):181-201. DOI:10.1016/j.scr.2014.05.005
26. Berg-Johansen B, Liebenberg EC, Li A, Macias BR, Hargens AR, Lotz JC. Spaceflight-induced bone loss alters failure mode and reduces bending strength in murine spinal segments. J Orthop Res. 2016;34(1):48-57. DOI:10.1002/jor.23029
27. Gerbaix M, Gnyubkin V, Farlay D, Olivier C, Ammann P, Courbon G, et al. One-month spaceflight compromises the bone microstructure, tissue-level mechanical properties, osteocyte survival and lacunae volume in mature mice skeletons. Sci Rep. 2017;7(1):2659. DOI:10.1115/SBC2013-14785
28. Gerbaix M, White H, Courbon G, Shenkman B, Gauquelin-Koch G, Vico L. Eight Days of Earth Reambulation Worsen Bone Loss Induced by 1-Month Spaceflight in the Major Weight-Bearing Ankle Bones of Mature Mice. Front Physiol. 2018;9:746. DOI:10.3389/fphys.2018.00746
29. Shiba D, Mizuno H, Yumoto A, Shimomura M, Kobayashi H, Morita H, et al. Development of new experimental platform 'MARS'-Multiple Artificial-gravity Research System-to elucidate the impacts of micro/partial gravity on mice. Scientific reports. 2017;7(1):10837. DOI:10.1038/s41598-017-10998-4
30. Tominari T, Ichimaru R, Taniguchi K, Yumoto A, Shirakawa M, Matsumoto C, et al. Hypergravity and microgravity exhibited reversal effects on the bone and muscle mass in mice. Sci Rep. 2019;9(1):6614. DOI:10.1038/s41598-019-42829-z

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.