Systematic Review & Evidence Integration

• Simple Definition
• Advanced Definition
• Expanded Definition

Simple Definition

Systematic Review and Evidence Integration are tools that are more frequently being employed to conduct a Weight of Evidence (WoE) analysis of existing literature to address research questions.

Systematic Review uses explicit, pre-specified methods to identify, select, assess, and summarize findings of similar but separate studies to answer a focused research question. The systematic review process is undertaken to identify all relevant studies on the agent of interest, to evaluate the studies identified, and to provide a qualitative and, where possible, a quantitative synthesis (aka meta-analysis) of the identified studies.

Evidence Integration largely focuses on hazard identification. In this step of the process, the evidence for human studies and non-human animal studies, which have remained separate through the previous steps of the systematic review, are integrated along with other relevant data. Hazard identification conclusions are reached by integrating the highest level-of-evidence conclusion for a health effect(s) from the human and the animal evidence.

The five hazard identification conclusion categories are often characterized as:

• Known to be a hazard to humans
• Presumed to be a hazard to humans
• Suspected to be a hazard to humans
• Not classifiable as a hazard to humans
• Not identified to be a hazard to humans.

Advanced Definition

On an outcome basis, this approach applies to whether the data support a health effect conclusion, or if there is evidence of no health effect. For cases where a rigorous assessment is needed to address a question, a systematic review can help focus the evaluation and maximize transparency in both how the assessment was conducted as well as how it was used to draw conclusions. To help ensure that the evidence is selected and evaluated in an objective and consistent manner, a systematic review requires carefully crafting the research question and planning in advance which methods will be used for screening and analyzing the scientific literature to answer the question.

Compared to traditional narrative reviews, the systematic review framework is aimed at minimizing subjectivity and enhancing transparency, rigor, and consistency in the way reviews are conducted and reported. Transparency is enhanced through the drafting and posting of a protocol prior to commencing the review. The protocol specifies the research question; the literature search strategy; the inclusion/exclusion criteria for identifying relevant studies returned in the literature search; the framework for judging the quality of included studies; and the plan for data analysis, synthesis and presentation of findings. In this way, clear criteria for conducting the review are developed and specified in advance.

Methods for conducting systematic reviews of the comparative effectiveness of clinical interventions are well established (e.g. Cochrane Collaboration [Higgins and Green 2011] and IOM [2011]), and the methods have been adapted (e.g., Woodruff and Sutton 2014; NTP 2015) and used to answer environmental health questions. Rooney et al (2014) describe a seven-step process for conducting Systematic Reviews and Evidence Integration that the NTP is using to answer research questions.

Two National Research Council reports (2011, 2014) have made recommendations that EPA’s Integrated Risk Information System (IRIS) program use more transparent and consistent methods for conducting its toxicological evaluations. Systematic review methods were identified as having the necessary elements to support those types of assessments. EPA has also begun incorporating systematic review approaches into its chemical evaluations.

The European Food Safety Authority (EFSA) has been using systematic review approaches for a few years to fulfill its mandates. EFSA has also authorized the creation of 23 systematic reviews on topics including pesticides, nutrition, feed, animal health, plant health, contaminants, biological hazards, genetically modified organisms, and methodologies.

More recently, EFSA began what it calls the PROmoting METHods for Evidence Use in Science (Prometheus) project to further enhance the scientific rigor of the methodological approaches used in dealing with evidence. The project was based on the recognition that evidence is needed in all assessments, and the process for collecting, appraising, and analyzing it should be the same regardless of the objectives of the assessment or who conducts it. Assessments focused on efficacy, safety, and risk should all follow the same process.

Another rationale for the Prometheus project is to address the issues presented when evidence is not available or there is insufficient time for applying extensive or complex approaches. EFSA recently published a report on the resulting methodological framework (EFSA, 2015).

Expanded Definition

On an outcome basis, this approach applies to whether the data support a health effect conclusion, or if there is evidence of no health effect. For cases where a rigorous assessment is needed to address a question, a systematic review can help focus the evaluation and maximize transparency in both how the assessment was conducted as well as how it was used to draw conclusions. To help ensure that the evidence is selected and evaluated in an objective and consistent manner, a systematic review requires carefully crafting the research question and planning in advance which methods will be used for screening and analyzing the scientific literature to answer the question.

Compared to traditional narrative reviews, the systematic review framework is aimed at minimizing subjectivity and enhancing transparency, rigor, and consistency in the way reviews are conducted and reported. Transparency is enhanced through the drafting and posting of a protocol prior to commencing the review. The protocol specifies the research question; the literature search strategy; the inclusion/exclusion criteria for identifying relevant studies returned in the literature search; the framework for judging the quality of included studies; and the plan for data analysis, synthesis and presentation of findings. In this way, clear criteria for conducting the review are developed and specified in advance.

Methods for conducting systematic reviews of the comparative effectiveness of clinical interventions are well established (e.g. Cochrane Collaboration [Higgins and Green 2011] and IOM [2011]), and the methods have been adapted (e.g., Woodruff and Sutton 2014; NTP 2015) and used to answer environmental health questions. Rooney et al (2014) describe a seven-step process for conducting Systematic Reviews and Evidence Integration that the NTP is using to answer research questions.

Two National Research Council reports (2011, 2014) have made recommendations that EPA’s Integrated Risk Information System (IRIS) program use more transparent and consistent methods for conducting its toxicological evaluations. Systematic review methods were identified as having the necessary elements to support those types of assessments. EPA has also begun incorporating systematic review approaches into its chemical evaluations.

The European Food Safety Authority (EFSA) has been using systematic review approaches for a few years to fulfill its mandates. EFSA has also authorized the creation of 23 systematic reviews on topics including pesticides, nutrition, feed, animal health, plant health, contaminants, biological hazards, genetically modified organisms, and methodologies.

More recently, EFSA began what it calls the PROmoting METHods for Evidence Use in Science (Prometheus) project to further enhance the scientific rigor of the methodological approaches used in dealing with evidence. The project was based on the recognition that evidence is needed in all assessments, and the process for collecting, appraising, and analyzing it should be the same regardless of the objectives of the assessment or who conducts it. Assessments focused on efficacy, safety, and risk should all follow the same process.

Another rationale for the Prometheus project is to address the issues presented when evidence is not available or there is insufficient time for applying extensive or complex approaches. EFSA recently published a report on the resulting methodological framework (EFSA, 2015).

References

EFSA (European Food Safety Authority) 2015 Principles and process for dealing with data and evidence in scientific assessments. EFSA J . 13, 4121–4157.

Higgins, J.P.T., and S. Green, eds. 2011. Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0 (updated March 2011). The Cochrane Collaboration [online]. Available: www.handbook.cochrane.org [accessed March 8, 2017].

IOM (Institute of Medicine). 2011. Finding What Works in Health Care: Standards for Systematic Review. Washington, DC: The National Academies Press.

Ionnidis J.P.A. 2016.  The mass production of redundant, misleading, and conflicted Systematic Reviews and Meta-analyses. The Milbank Quarterly 94:3;485–514 DOI: 10.1111/1468-0009.12210

NRC. 2011. Review of the Environmental Protection Agency’s Draft IRIS Assessment of Formaldehyde. Washington, DC: The National Academies Press.

NRC. 2014 Review of EPA’s Integrated Risk Information System (IRIS) Process. Washington, DC: The National Academies Press.

NTP. 2015. Handbook for Conducting a Literature-Based Health Assessment Using OHAT Approach for Systematic Review and Evidence Integration. Office of Health Assessment and Translation, Division, National ToxicologyProgram, National Institute of Environmental Health Sciences. January 9, 2015 [online]. Available: http://ntp.niehs.nih.gov/ntp/ohat/pubs/handbookjan2015_508.pdf [accessed July 25, 2017].

Rooney, A.A., A.L. Boyles, M.S. Wolfe, J.R. Bucher, and K.A. Thayer. 2014. Systematic review and evidence integration for literature-based environmental health science assessments. Environ. Health Perspect. 122(7):711-718.

Stephens M.L., Betts K., Beck N.A., et al 2016. The emergence of systematic review in toxicology. Toxicol Sci 152 (1): 10-16. DOI: https://doi.org/10.1093/toxsci/kfw059

Woodruff, T.J., and P. Sutton. 2014. The Navigation Guide systematic review methodology: A rigorous and transparent method for translating environmental health science into better health outcomes. Environ. Health Perspect. 122(10):1007-1014.