Skip to main content

Systematic Reviews and Evidence Synthesis Guide

About

Evidence Synthesis (and Systematic Reviews)

Evidence synthesis is a form of research that includes dozens of types of reviews that all share a set of core methodologies, and that (in addition to systematic reviews) also includes umbrella reviews, clinical guidelines, rapid reviews, scoping reviews, meta-analyses, and more. The purpose of a evidence synthesis is to take all the available evidence on a topic and draw conclusions based on the amount and strength of the evidence available.

How Evidence Syntheses Differ from Literature Reviews

There’s a lot of confusion surrounding evidence synthesis, with the biggest source of confusion being how evidence syntheses differ from literature reviews. In short, though, literature reviews are scans of the literature that place a research project in context. They discuss, argue, highlight and support the hypothesis upon which the project is based.  Literature reviews make an argument for the project; evidence syntheses are the project and contain literature reviews that argue for them and place them in context.

 

Evidence Synthesis

Literature Review

Goal

 Answers a specific research question, identifies evidence gaps, evaluates research in a field, or otherwise examines a highly specific research area in order to draw conclusions or inform practice / further research. Summarizes a topic that may be either broad or narrow

Search Strategy

Uses a pre-determined systematic, comprehensive and replicable search strategy.

Often includes searches of grey literature sources to reduce the risk of publication bias affecting the results.

 Searching may be done organically and in a non-prescribed manner.

Components

Selection and inclusion criteria

Systematic search strategy

Appraisal of existing research

Synthesis of findings

Summary of the literature

Context for the paper

Hypotheses or thesis statements

 

Librarian Involvement

Role of Librarians in Evidence Syntheses

Most books and guidelines on writing evidence syntheses recommend the participation of a librarian at the searching stage.  This is because the style and depth of searching used in evidence synthesis requires specific training and a thorough understanding of the systematic approach.  At UNB Libraries, there are currently two librarians who provide evidence synthesis support (Richelle Witherspoon and Alex Goudreau) and we provide three levels of support in our roles: instruction & resources, consultation and collaboration.

Instruction & Resources

This service provides training and resources on the different steps involved in conducting evidence synthesis projects.  Synchronous and asynchronous instruction by trained librarians is available, as are librarian-curated resources (like this research guide).

This service is available to all UNB affiliated faculty, staff, and students wanting to learn how to conduct evidence syntheses.

This service includes:

  • This Research Guide, which provides information and resources on how to conduct evidence syntheses.  Included in this guide are video tutorials on advanced searching, links to guiding documents, and more.
  • Systematic Review Bootcamp, which is a series of 5 connected workshops that is typically offered in the Spring.  Asynchronous videos from the 2021 bootcamp are available in this guide.
  • Customize group or individual instruction by trained librarians.  This instruction can take place in a group or one-on-one, online or in-person, and may cover a variety of evidence synthesis-related topics.  Contact Richelle Witherspoon or Alex Goudreau to discuss options and availability.

Consultation

This service provides up to 5-hours of one-on-one support for individuals conducting evidence synthesis projects. It is intended as a way to supplement existing or developing researcher expertise by providing guidance, support and instruction in specific areas of systematic and scoping review methodology.

This service is targeted towards UNB students who are completing a evidence synthesis for a course, thesis or dissertation, but may also be used by faculty and staff who feel it is the right fit for their needs.

When acting as a consultant for an evidence synthesis project, the librarian’s role is to provide guidance, support, and feedback. As part of that role, the librarian may provide:

  • Guidance on database selection and search design
  • Suggestions about organization and record keep to support compliance with PRISMA-2020 and PRISMA-S
  • Database support specific to evidence synthesis
  • Advice on selecting screening and citation management tools
  • Training on obtaining full-text articles

Collaboration

This service embeds a librarian as a member of the research team. The librarian becomes responsible for search strategy development, implementation, and reporting, in consultation with other team members. At the completion of a project in which a librarian works in a collaborative, embedded role, they should be offered co-authorship in the final publication (as per the recommendations defining authors from ICMJE).

All UNB faculty and staff working on evidence synthesis projects may request librarian collaboration for their team. Submitting a request for collaboration does not guarantee that it will be given, as librarians may use their own discretion in joining research teams.

When serving as a collaborator on evidence synthesis projects, the librarian will:

  • Work with the team to develop an appropriate research question
  • Provide guidance on database selection
  • Check for existing or registered SRs on the same topic
  • Design and execute the systematic search strategy
  • Advise the team on grey literature searching and may assist with running the searches on a case-by-case basis )
  • Export search results in RIS format to the researcher’s citation or review tool of choice
  • Document search methodology, export search histories, and provide search details as recommended by PRISMA 2020 and PRISMA-S
  • Write the source and search methodology sections of protocols and final manuscripts; provide search details for the appendix
  • As a co-author, review and approve submitted manuscripts, and respond to reviewer feedback as needed

Note: Consultations and collaboration are available on a first-come, first served basis. Librarian availability to assist with evidence synthesis project may vary based on the time of year and the number of reviews being requested. Please be aware that requests may be deferred, and/or may be referred to another librarian.

Searching For Evidence Syntheses

Finding Existing Systematic Reviews

Before starting a new systematic review you must show that there is not already an existing systematic review - or systematic review protocol - on your topic.  If there is an existing review, you must be able to justify the need for a new one based on the age of the review, the applicability to your question, and/or the manner in which the previous review was performed.

Check Subject Databases

Do a thorough search of any databases that you consider relevant to your topic. In addition to using keywords to bring up the topic you are interested in, include search terms like "systematic review", "scoping review", "umbrella review", and "meta analysis" to narrow your search down to the article types you are interested in.

Check Systematic Review Providers & Registries

Campbell Systematic Reviews

Campbell Systematic Reviews is an open access journal prepared under the editorial control of the Campbell Collaboration. The journal publishes systematic reviews, evidence and gap maps, and methods research papers.

Open Science Framework Registries

Open Science Framework Registries is a multidisciplinary registry for evidence synthesis protocols.

Cochrane Library

The Cochrane Collaboration is a group that solicits and creates systematic reviews on clinical topics.  Their database contains a wide range of resources, but is most well-known for its extensive collection of systematic reviews.

  • Cochrane Library
    The Cochrane Library contains high-quality, independent evidence to inform healthcare decision-making. This database includes the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects, the Cochrane Central Register of Controlled Trials, the Cochrane Database of Methodology Reviews, the The Cochrane Methodology Register, the Health Technology Assessment Database, and the NHS Economic Evaluation Database.
    Access to the Cochrane Library in New Brunswick is through a province-wide license made possible by a partnership between UNB and the eight Regional Hospital Authority libraries.
    Unlimited simultaneous users.

PROSPERO

PROSPERO is a systematic review registry. This means that it is a place where researchers signal their intent to write a systematic review on a given topic.  Searching PROSPERO lets you know whether a review on your topic is already in progress and might be completed before yours.

It is also a good idea to register your own review in PROSPERO.

Joanna Briggs Institute

The Joanna Briggs Institute (JBI) is another solicitor / creator of systematic review that focuses specifically on nursing and allied health related topics.  To find out if there are any JBI reviews completed or pending on your topic, you must go to two different places:

  1. JBI's Registered Titles to find out if there is a systematic review that is currently in progress
  2. CINAHL or PubMed to find out if there is a completed systematic review on your topic

Since it is likely that you have already searched CINAHL or PubMed for systematic reviews when checking your subject specific databases, you probably don't need to do a special search for this. However, if you have a specific need to know if there is a JBI review on a topic, you can do your regular keyword search and then append the following to your search:

CINAHL:   JN("JBI database of systematic reviews & implementation reports" OR "JBI library of systematic reviews")

PubMed:   ("JBI database system rev implement rep" OR "JBI Libr Syst Rev" OR "JBI Evid Synth")

Appending the above phrase to your search will direct the database to only bring forward JBI reviews and will exclude all other database content.

Parts of a Synthesis

Parts of an Evidence Synthesis

While the contents of an evidence synthesis vary based on the type of synthesis being done (see ES Types), most good evidence syntheses do share a number of core components:

Introduction

A brief background on the topic that describes its context and importance

Review Question

The specific question or evidence gap to be explored by the evidence synthesis.

Inclusion Criteria

The inclusion criteria (which can also be framed as exclusion criteria in some ESs) are the guidelines that reviewers follow when determining whether a specific item will be considered in the evidence synthesis.  These criteria should be rigidly adhered to and constitute a promise to the readers by the reviewers that only the items that fit all the criteria were included in the review.

Participants

The participants section includes a description of the population(s) of interest in the review.

Phenomenon / Exposure / Intervention

The phenomenon / exposure / intervention usually describes the main issue of interest in the evidence synthesis.

Types of Studies

The types of studies section typically lists the types of research that were considered for inclusion.  The specific contents of this section vary widely between the types of syntheses and even within types as it is influenced by the amount of research available and by the types of research that have been performed.  Additionally, this is the one section where the criteria may be a little more flexible, as it may include a statement like "other types of studies were considered for inclusion if they were deemed to be highly relevant and of good quality."

Methods

The methods section includes a detailed description of how the evidence synthesis was carried out.  Because evidence syntheses are done based on a prescribed set of rules and criteria, most methods sections contain the same types of information.

Search strategy

The search strategy section describes how the search how search terms were generated and how the search protocol was developed, what databases were used, the date that the search was performed, and whether any grey literature sources were searched.

Note that while the search strategy / protocol is often very complex and reflects a similar approach when it is being used in traditional article databases, grey literature searching in systematic reviews does not follow the same criteria.  Few grey literature sources are large or sophisticated enough to make the use of advanced searching strategies practical, but are often included in the search strategy as a way to mitigate the effects of publication bias.

Study selection

The study selection section describes the process by which the two (or more) reviewers selected items for inclusion in the review.  It includes such information as the number of reviewers, the appraisal ratings that were considered acceptable and how they were obtained, and how differences of assessment were resolved between reviewers.

Data extraction

The data extraction section (which can go by other names) describes how information from each included article / item was recorded and used.  In the case of quantitative ESs, for example, specific information is usually extracted and recorded in table to make comparisons between studies (then arranged in rows) easier.

Results

The results section describes the outcomes of each of the processes implemented in the methods.  It provides information about the number of studies included, the appraisals of the studies, and a brief description of the studies.

Study inclusion (and PRISMA diagram)

The study inclusion section details the number of studies that were retrieved, the number of duplicates removed, the number that were considered relevant, the number that passed the appraisal criteria and the total number of studies that were finally considered for review.  This information is almost always provided in the form of a short paragraph, and is sometimes also provided in the form of a PRISMA Flow Diagram.  There may also be a summary of the most common reasons that articles were excluded from the review.

Methodological quality

The methodological quality section provides a summary of the overall appraised quality of the included articles. This typically includes a brief paragraph and a more detailed table that summarizes the appraisals for each article.

Characteristics of included studies

This section provides a quick overview of the specific details of the studies that were included, such as study type, number of participants, and geographical location.  It does not typically include mention of the outcomes of the studies.

Review Findings

This section describes the overall findings of the evidence synthesis.

Discussion

This section discusses the review findings in the context of the literature and in practice.

Appendix - Search strategy or strategies

This section contains a copy of the exact search that was performed in at least one of the databases (some reviews include copies of all the searches performed).  It does not include results of those searches, but instead provides insight into the keywords, subject terms, proximity searches, Boolean operators, etc. that were used to retrieve the items for review.

Synthesis Types

Types of Syntheses

There are many types of syntheses, and each differs slightly in its approach to the literature and its ultimate goal. 

Below are some of the most common types of systematic reviews:

SR of quantitative evidence

A review of the quantitative evidence on a topic. SRs of quantitative evidence focus on using studies (like RCTs) to determine whether an intervention is effective and/or the degree of effectiveness of the intervention.

SR of qualitative evidence

A review of the qualitative evidence on a topic. SRs of qualitative evidence involve analyzing the findings and themes of multiple qualitative studies on a topic (of various types and approaches) and using the extracted information to obtain a wholistic perspective on the phenomenon.

Umbrella Review

A reviews that assesses the existing systematic reviews (or clinical guidelines) on a topic when there is more than one available.  Umbrella reviews evaluate the thoroughness and quality of the procedures implemented in the systematic reviews they consider and synthesize the findings to arrive at a single conclusion.

Scoping Review

A review that addresses the literature on a topic in its entirety. Scoping reviews tend to consider slightly broader questions than other SRs and assess all the literature available on a topic; they tend to be performed on topics that are emerging or not heavily researched.

Mixed Methods SR

A review that includes both qualitative and quantitative methods.  Mixed methods SRs provide a combination of effectiveness and experiential information on a topic that results in a powerfully wholistic review.

Rapid Review

Rapid reviews are a systematic survey of literature on a topic or question of interest. Compared to a systematic review of literature, in a rapid review, several design decisions and practical steps are undertaken to reduce the time it takes to identify, aggregate and answer the question of interest.

Realist Review

Realist evaluation is a type of theory-driven evaluation method used in evaluating social programmes. It is based on the epistemological foundations of critical realism.

Clinical Guideline

Clinical guidelines are significant evidence syntheses that may involve more than one question and search.  Clinical guidelines are often compilations of multiple reviews and are intended to guide/inform all aspects of practice in a specific area.

 

Sample Syntheses

Sample Evidence Syntheses

Physical education teacher educators: A 25-year scoping review of literature

McEvoy, E., MacPhail, A., & Heikinaro-Johansson, P.

Teaching and Teacher Education

The Effects of Training, Innovation and New Technology on African Smallholder Farmers' Economic Outcomes and Food Security: A Systematic Review

Stewart, R., Langer, L., Rebelo Da Silva, N., Muchiri, E., Zaranyika, H., Erasmus, Y., Randall, N., Rafferty, S., Korth, M., Madinga, N., & de Wet, T.

Campbell Systematic Reviews

Vocational and Business Training to Improve Women's Labour Market Outcomes in Low- and Middle-Income Countries: A Systematic Review

Chinen, M., de Hoop, T., Alcazar, L., Balarin, M., Sennett, J.

Campbell Systematic Reviews

Prevalence and Reasons of Self-Medication in Pregnant Women: A Systematic Review and Meta-Analysis

Mohseni, M., Azami-Aghdash, S., Sheyklo, S. G., Moosavi, A., Nakhaee, M., Pournaghi-Azar, F., & Rezapour, A.

International Journal of Community Based Nursing & Midwifery

Participation in environmental enhancement and conservation activities for health and well‐being in adults: a review of quantitative and qualitative evidence

Husk  K., Lovell  R., Cooper  C., Stahl‐Timmins  W., & Garside  R.

Cochrane Database of Systematic Reviews

Systematic review : Craniocervical posture and craniofacial morphology.

Gomes, L., Horta, K., Gonçalves, J., & Santos-Pinto, A.

European Journal of Orthodontics

Systematic Review: Process of Forming Academic Service Partnerships to Reform Clinical Education.

Nabavi, F. H., Vanaki, Z., & Mohammadi, E.

Western Journal of Nursing Research

Effectiveness of patient simulation manikins in teaching clinical reasoning skills to undergraduate nursing students: A systematic review

Lapkin, S., Levett-Jones, T., Bellchambers, H., & Fernandez, R.

Clinical Simulation in Nursing

 

More ES Resources

Evidence Synthesis by Discipline

While the core traits of evidence synthesis remain the same across disciplines, different disciplines may prefer different guidelines, have different registries, and/or use different searching tools.  We have summarized some of the preferred resources for the disciplines which use ES most often in the documents below.

Database Tip Sheets

These database tip sheets do not cover general usage of any database, but instead focus on select complicated tools that are commonly used in evidence synthesis and that even advanced searchers might not reasonably be expected to know.  For more general database tips and tricks, please refer to the Library's Help Videos.

ES Software and Tools

There are 4 popular software options that support the entire systematic review process:

  • JBI SUMARI: By subscription and mostly used for doing JBI reviews. SUMARI supports the entire review process, including drafting your protocol, study selection, critical appraisal, data extraction and synthesis. 
  • Distiller SR: Also supports the entire process and costs money. Offers special pricing for students. 
  • Covidence: Supports the whole process & costs money. Recommended by Cochrane. 
  • RAYYAN: Supports the whole process. FREE to use. A little clunky but works fairly well. Users often recommend using the web-version as the app can be slow and a bit unreliable.

There are also other tools that are available that don’t support the whole process but guide or help with parts of it. Things like: 

  • Citation management software like Zotero, Endnote, Mendeley, etc. 
  • Templates for protocols and reviews like the ones available from JBI 
  • Reporting templates and guidelines like those available through PRISMA 

Select ES Guidelines

Guidelines provide a description of how a high quality evidence synthesis should be performed.  Three most popular of these are the ones produced by the Campbell Collaboration, the Cochrane Library, and the Joanna Briggs Institute (the top three guidelines linked below), however there are a number of guidelines available and the extensive detail in those top three guidelines should be used with consideration for the needs indicated in more discipline specific resources (linked both below and in the discipline links above).

Reporting Standards

Reporting standards offer guidance on what information should be shared in the review regarding its process and methodology.  By far the most popular of the standards is produced by PRISMA, which produces several guidelines that differ by synthesis type and methodological aspect.

Protocol Registries

Because of the significance of undertaking a quality evidence synthesis, there are protocol registries that allow researchers to share their intent to publish on a topic prior to completion of the review.  Protocol registries should be used both to notify the research community of your intent to publish, and to ensure that no one else has already done so on your topic.

Search Filters

Search filters, also called hedges, are pre-assembled search strings for specific topics.  While search filters vary in quality and design, they offer an excellent basis for new searches and - in some cases - may replace one component of a search entirely. For instance, the Cochrane RCT search filter is excellent and can easily replace any other such narrowing option.  When using a search filter (either in part or in whole) or when using a substantive portion of a search from another review, it is expected that the source be cited and the creator given credit.

There are many places you may look for search filters, like:

Critical Appraisal

Critical appraisal (also called 'risk of bias assessment') is a step used in some kinds of evidence synthesis in which each article selected for inclusion in the final review is appraised for quality and likelihood of bias. There are many critical appraisal tools available, with some tools being preferred by (or more appropriate in) certain disciplines.  A selection of the more common tools is below:

  • Joanna Briggs Critical Appraisal Tools  
    • Typically only used for JBI reviews  
    • 12 checklists – one each for 12 different kinds of studies  
    • Their “Checklist for text and opinions” can be used for grey literature 

Other

This section contains one-off content that has been created to provide comprehensive answers to common questions.

 

Videos

SR Bootcamp

This section includes links to videos about how to complete a systematic review.  These videos are recordings from the Systematic Reviews Bootcamp 2021 that was offered by UNB Libraries and cover the entire process of performing a systematic review.

Part 1: Getting Started with your SR

Part 2: Systematic Searching (Part 1)

Part 3: Systematic Searching (Part 2)

Part 4: Article Screening & Grey Literature

Part 5: Critical Appraisal and Data Extraction

 

Just Systematic Searching

This section includes links to a video series that walks you through the process of developing a systematic search step by step. These videos focus solely on the search development and execution portion of the knowledge synthesis process. Videos should be watched in order, and each video has a task that should be completed before the next video is watched.

*** It may be worthwhile to watch video 16 early in your searching process, as it offers tips that will be useful throughout. ***

1. About This Video Series (workbook: 0.%20Workbook.docx)

2. Forming a Research Question 

3. Choosing Your Databases 

4. Choose Resource Types 

5. Generating Synonyms 

6. Controlled Vocabulary 

7. Search Commands 

8. Proximity Searching 

9. Applying Limiters 

10. Boolean Operators 

11. Testing the Search 

12. Record, Save, Export and Report

13. Reviewing your Search

14. Translating the Search

15. Hand Searching

16. A Few Other Tips***

 

 

More Information More Information

  • Richelle Witherspoon (she/her/hers)
  • If you can't find the information you need, I am here to help. I provide support over email and Teams meetings.
  • Information Services Librarian
  • UNB Fredericton
  • r.witherspoon@unb.ca

Subject Specialties:
Nursing, kinesiology, and psychology.