Evidence Synthesis (and Systematic Reviews)
Evidence synthesis is a form of research that includes dozens of types of reviews that all share a set of core methodologies, and that (in addition to systematic reviews) also includes umbrella reviews, clinical guidelines, rapid reviews, scoping reviews, meta-analyses, and more. The purpose of a evidence synthesis is to take all the available evidence on a topic and draw conclusions based on the amount and strength of the evidence available.
How Evidence Syntheses Differ from Literature Reviews
There’s a lot of confusion surrounding evidence synthesis, with the biggest source of confusion being how evidence syntheses differ from literature reviews. In short, though, literature reviews are scans of the literature that place a research project in context. They discuss, argue, highlight and support the hypothesis upon which the project is based. Literature reviews make an argument for the project; evidence syntheses are the project and contain literature reviews that argue for them and place them in context.
|Answers a specific research question, identifies evidence gaps, evaluates research in a field, or otherwise examines a highly specific research area in order to draw conclusions or inform practice / further research.||Summarizes a topic that may be either broad or narrow|
Uses a pre-determined systematic, comprehensive and replicable search strategy.
Often includes searches of grey literature sources to reduce the risk of publication bias affecting the results.
|Searching may be done organically and in a non-prescribed manner.|
Selection and inclusion criteria
Systematic search strategy
Appraisal of existing research
Synthesis of findings
Summary of the literature
Context for the paper
Hypotheses or thesis statements
Role of Librarians in Evidence Syntheses
Most books and guidelines on writing evidence syntheses recommend the participation of a librarian at the searching stage. This is because the style and depth of searching used in evidence synthesis requires specific training and a thorough understanding of the systematic approach. At UNB Libraries, there are currently two librarians who provide evidence synthesis support (Richelle Witherspoon and Alex Goudreau) and we provide three levels of support in our roles: instruction & resources, consultation and collaboration.
Instruction & Resources
This service provides training and resources on the different steps involved in conducting evidence synthesis projects. Synchronous and asynchronous instruction by trained librarians is available, as are librarian-curated resources (like this research guide).
This service is available to all UNB affiliated faculty, staff, and students wanting to learn how to conduct evidence syntheses.
This service includes:
- This Research Guide, which provides information and resources on how to conduct evidence syntheses. Included in this guide are video tutorials on advanced searching, links to guiding documents, and more.
- Systematic Review Bootcamp, which is a series of 5 connected workshops that is typically offered in the Spring. Asynchronous videos from the 2021 bootcamp are available in this guide.
- Customize group or individual instruction by trained librarians. This instruction can take place in a group or one-on-one, online or in-person, and may cover a variety of evidence synthesis-related topics. Contact Richelle Witherspoon or Alex Goudreau to discuss options and availability.
This service provides up to 5-hours of one-on-one support for individuals conducting evidence synthesis projects. It is intended as a way to supplement existing or developing researcher expertise by providing guidance, support and instruction in specific areas of systematic and scoping review methodology.
This service is targeted towards UNB students who are completing a evidence synthesis for a course, thesis or dissertation, but may also be used by faculty and staff who feel it is the right fit for their needs.
When acting as a consultant for an evidence synthesis project, the librarian’s role is to provide guidance, support, and feedback. As part of that role, the librarian may provide:
- Guidance on database selection and search design
- Suggestions about organization and record keep to support compliance with PRISMA-2020 and PRISMA-S
- Database support specific to evidence synthesis
- Advice on selecting screening and citation management tools
- Training on obtaining full-text articles
This service embeds a librarian as a member of the research team. The librarian becomes responsible for search strategy development, implementation, and reporting, in consultation with other team members. At the completion of a project in which a librarian works in a collaborative, embedded role, they should be offered co-authorship in the final publication (as per the recommendations defining authors from ICMJE).
All UNB faculty and staff working on evidence synthesis projects may request librarian collaboration for their team. Submitting a request for collaboration does not guarantee that it will be given, as librarians may use their own discretion in joining research teams.
When serving as a collaborator on evidence synthesis projects, the librarian will:
- Work with the team to develop an appropriate research question
- Provide guidance on database selection
- Check for existing or registered SRs on the same topic
- Design and execute the systematic search strategy
- Advise the team on grey literature searching and may assist with running the searches on a case-by-case basis )
- Export search results in RIS format to the researcher’s citation or review tool of choice
- Document search methodology, export search histories, and provide search details as recommended by PRISMA 2020 and PRISMA-S
- Write the source and search methodology sections of protocols and final manuscripts; provide search details for the appendix
- As a co-author, review and approve submitted manuscripts, and respond to reviewer feedback as needed
Note: Consultations and collaboration are available on a first-come, first served basis. Librarian availability to assist with evidence synthesis project may vary based on the time of year and the number of reviews being requested. Please be aware that requests may be deferred, and/or may be referred to another librarian.
Searching For Evidence Syntheses
Finding Existing Systematic Reviews
Before starting a new systematic review you must show that there is not already an existing systematic review - or systematic review protocol - on your topic. If there is an existing review, you must be able to justify the need for a new one based on the age of the review, the applicability to your question, and/or the manner in which the previous review was performed.
Check Subject Databases
Do a thorough search of any databases that you consider relevant to your topic. In addition to using keywords to bring up the topic you are interested in, include search terms like "systematic review", "scoping review", "umbrella review", and "meta analysis" to narrow your search down to the article types you are interested in.
Check Systematic Review Providers & Registries
Campbell Systematic Reviews
Campbell Systematic Reviews is an open access journal prepared under the editorial control of the Campbell Collaboration. The journal publishes systematic reviews, evidence and gap maps, and methods research papers.
Open Science Framework Registries
Open Science Framework Registries is a multidisciplinary registry for evidence synthesis protocols.
The Cochrane Collaboration is a group that solicits and creates systematic reviews on clinical topics. Their database contains a wide range of resources, but is most well-known for its extensive collection of systematic reviews.
- Cochrane Library The Cochrane Library contains high-quality, independent evidence to inform healthcare decision-making. This database includes the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects, the Cochrane Central Register of Controlled Trials, the Cochrane Database of Methodology Reviews, the The Cochrane Methodology Register, the Health Technology Assessment Database, and the NHS Economic Evaluation Database.
Access to the Cochrane Library in New Brunswick is through a province-wide license made possible by a partnership between UNB and the eight Regional Hospital Authority libraries.
Unlimited simultaneous users.
PROSPERO is a systematic review registry. This means that it is a place where researchers signal their intent to write a systematic review on a given topic. Searching PROSPERO lets you know whether a review on your topic is already in progress and might be completed before yours.
It is also a good idea to register your own review in PROSPERO.
Joanna Briggs Institute
The Joanna Briggs Institute (JBI) is another solicitor / creator of systematic review that focuses specifically on nursing and allied health related topics. To find out if there are any JBI reviews completed or pending on your topic, you must go to two different places:
- JBI's Registered Titles to find out if there is a systematic review that is currently in progress
- CINAHL or PubMed to find out if there is a completed systematic review on your topic
Since it is likely that you have already searched CINAHL or PubMed for systematic reviews when checking your subject specific databases, you probably don't need to do a special search for this. However, if you have a specific need to know if there is a JBI review on a topic, you can do your regular keyword search and then append the following to your search:
CINAHL: JN("JBI database of systematic reviews & implementation reports" OR "JBI library of systematic reviews")
PubMed: ("JBI database system rev implement rep" OR "JBI Libr Syst Rev" OR "JBI Evid Synth")
Appending the above phrase to your search will direct the database to only bring forward JBI reviews and will exclude all other database content.
Parts of a Synthesis
Parts of an Evidence Synthesis
While the contents of an evidence synthesis vary based on the type of synthesis being done (see ES Types), most good evidence syntheses do share a number of core components:
A brief background on the topic that describes its context and importance
The specific question or evidence gap to be explored by the evidence synthesis.
The inclusion criteria (which can also be framed as exclusion criteria in some ESs) are the guidelines that reviewers follow when determining whether a specific item will be considered in the evidence synthesis. These criteria should be rigidly adhered to and constitute a promise to the readers by the reviewers that only the items that fit all the criteria were included in the review.
The participants section includes a description of the population(s) of interest in the review.
Phenomenon / Exposure / Intervention
The phenomenon / exposure / intervention usually describes the main issue of interest in the evidence synthesis.
Types of Studies
The types of studies section typically lists the types of research that were considered for inclusion. The specific contents of this section vary widely between the types of syntheses and even within types as it is influenced by the amount of research available and by the types of research that have been performed. Additionally, this is the one section where the criteria may be a little more flexible, as it may include a statement like "other types of studies were considered for inclusion if they were deemed to be highly relevant and of good quality."
The methods section includes a detailed description of how the evidence synthesis was carried out. Because evidence syntheses are done based on a prescribed set of rules and criteria, most methods sections contain the same types of information.
The search strategy section describes how the search how search terms were generated and how the search protocol was developed, what databases were used, the date that the search was performed, and whether any grey literature sources were searched.
Note that while the search strategy / protocol is often very complex and reflects a similar approach when it is being used in traditional article databases, grey literature searching in systematic reviews does not follow the same criteria. Few grey literature sources are large or sophisticated enough to make the use of advanced searching strategies practical, but are often included in the search strategy as a way to mitigate the effects of publication bias.
The study selection section describes the process by which the two (or more) reviewers selected items for inclusion in the review. It includes such information as the number of reviewers, the appraisal ratings that were considered acceptable and how they were obtained, and how differences of assessment were resolved between reviewers.
The data extraction section (which can go by other names) describes how information from each included article / item was recorded and used. In the case of quantitative ESs, for example, specific information is usually extracted and recorded in table to make comparisons between studies (then arranged in rows) easier.
The results section describes the outcomes of each of the processes implemented in the methods. It provides information about the number of studies included, the appraisals of the studies, and a brief description of the studies.
Study inclusion (and PRISMA diagram)
The study inclusion section details the number of studies that were retrieved, the number of duplicates removed, the number that were considered relevant, the number that passed the appraisal criteria and the total number of studies that were finally considered for review. This information is almost always provided in the form of a short paragraph, and is sometimes also provided in the form of a PRISMA Flow Diagram. There may also be a summary of the most common reasons that articles were excluded from the review.
The methodological quality section provides a summary of the overall appraised quality of the included articles. This typically includes a brief paragraph and a more detailed table that summarizes the appraisals for each article.
Characteristics of included studies
This section provides a quick overview of the specific details of the studies that were included, such as study type, number of participants, and geographical location. It does not typically include mention of the outcomes of the studies.
This section describes the overall findings of the evidence synthesis.
This section discusses the review findings in the context of the literature and in practice.
Appendix - Search strategy or strategies
This section contains a copy of the exact search that was performed in at least one of the databases (some reviews include copies of all the searches performed). It does not include results of those searches, but instead provides insight into the keywords, subject terms, proximity searches, Boolean operators, etc. that were used to retrieve the items for review.
Types of Syntheses
There are many types of syntheses, and each differs slightly in its approach to the literature and its ultimate goal.
Below are some of the most common types of systematic reviews:
SR of quantitative evidence
A review of the quantitative evidence on a topic. SRs of quantitative evidence focus on using studies (like RCTs) to determine whether an intervention is effective and/or the degree of effectiveness of the intervention.
SR of qualitative evidence
A review of the qualitative evidence on a topic. SRs of qualitative evidence involve analyzing the findings and themes of multiple qualitative studies on a topic (of various types and approaches) and using the extracted information to obtain a wholistic perspective on the phenomenon.
A reviews that assesses the existing systematic reviews (or clinical guidelines) on a topic when there is more than one available. Umbrella reviews evaluate the thoroughness and quality of the procedures implemented in the systematic reviews they consider and synthesize the findings to arrive at a single conclusion.
A review that addresses the literature on a topic in its entirety. Scoping reviews tend to consider slightly broader questions than other SRs and assess all the literature available on a topic; they tend to be performed on topics that are emerging or not heavily researched.
Mixed Methods SR
A review that includes both qualitative and quantitative methods. Mixed methods SRs provide a combination of effectiveness and experiential information on a topic that results in a powerfully wholistic review.
Rapid reviews are a systematic survey of literature on a topic or question of interest. Compared to a systematic review of literature, in a rapid review, several design decisions and practical steps are undertaken to reduce the time it takes to identify, aggregate and answer the question of interest.
Realist evaluation is a type of theory-driven evaluation method used in evaluating social programmes. It is based on the epistemological foundations of critical realism.
Clinical guidelines are significant evidence syntheses that may involve more than one question and search. Clinical guidelines are often compilations of multiple reviews and are intended to guide/inform all aspects of practice in a specific area.
Sample Evidence Syntheses
McEvoy, E., MacPhail, A., & Heikinaro-Johansson, P.
Teaching and Teacher Education
Stewart, R., Langer, L., Rebelo Da Silva, N., Muchiri, E., Zaranyika, H., Erasmus, Y., Randall, N., Rafferty, S., Korth, M., Madinga, N., & de Wet, T.
Campbell Systematic Reviews
Chinen, M., de Hoop, T., Alcazar, L., Balarin, M., Sennett, J.
Campbell Systematic Reviews
Mohseni, M., Azami-Aghdash, S., Sheyklo, S. G., Moosavi, A., Nakhaee, M., Pournaghi-Azar, F., & Rezapour, A.
International Journal of Community Based Nursing & Midwifery
Husk K., Lovell R., Cooper C., Stahl‐Timmins W., & Garside R.
Cochrane Database of Systematic Reviews
Gomes, L., Horta, K., Gonçalves, J., & Santos-Pinto, A.
European Journal of Orthodontics
Nabavi, F. H., Vanaki, Z., & Mohammadi, E.
Western Journal of Nursing Research
Lapkin, S., Levett-Jones, T., Bellchambers, H., & Fernandez, R.
Clinical Simulation in Nursing
More ES Resources
Evidence Synthesis by Discipline
While the core traits of evidence synthesis remain the same across disciplines, different disciplines may prefer different guidelines, have different registries, and/or use different searching tools. We have summarized some of the preferred resources for the disciplines which use ES most often in the documents below.
Database Tip Sheets
These database tip sheets do not cover general usage of any database, but instead focus on select complicated tools that are commonly used in evidence synthesis and that even advanced searchers might not reasonably be expected to know. For more general database tips and tricks, please refer to the Library's Help Videos.
ES Software and Tools
There are 4 popular software options that support the entire systematic review process:
- JBI SUMARI: By subscription and mostly used for doing JBI reviews. SUMARI supports the entire review process, including drafting your protocol, study selection, critical appraisal, data extraction and synthesis.
- Distiller SR: Also supports the entire process and costs money. Offers special pricing for students.
- Covidence: Supports the whole process & costs money. Recommended by Cochrane.
- RAYYAN: Supports the whole process. FREE to use. A little clunky but works fairly well. Users often recommend using the web-version as the app can be slow and a bit unreliable.
There are also other tools that are available that don’t support the whole process but guide or help with parts of it. Things like:
- Citation management software like Zotero, Endnote, Mendeley, etc.
- Templates for protocols and reviews like the ones available from JBI
- Reporting templates and guidelines like those available through PRISMA
Select ES Guidelines
Guidelines provide a description of how a high quality evidence synthesis should be performed. Three most popular of these are the ones produced by the Campbell Collaboration, the Cochrane Library, and the Joanna Briggs Institute (the top three guidelines linked below), however there are a number of guidelines available and the extensive detail in those top three guidelines should be used with consideration for the needs indicated in more discipline specific resources (linked both below and in the discipline links above).
- Campbell Collaboration - a multidisciplinary guide to knowledge synthesis methodology
- Cochrane Handbooks – a collection of reference guides to support researchers conducting a Cochrane Review.
- JBI Manual for Evidence Synthesis – a comprehensive guide that supports 10 different types of evidence synthesis
- IOM Standards for Systematic Reviews – 21 standards for performing a quality systematic review
- NHS Centre for Reviews and Dissemination - A free guide on performing systematic reviews
- Systematic Reviews for Animals & Food - A collection of resources to support the steps of the review process
- The Collaboration for Environmental Evidence - guidelines for knowledge synthesis methodologies in environmental management
- Guidelines for systematic review in conservation and environmental management – paper published in Conservation Biology by A. Pullin (2006).
- Literature Review as a Research Methodology: An Overview and Guidelines – paper published in Journal of Business Research by H. H. Snyder (2019).
- Shades of Grey: Guidelines for Working with the Grey Literature in Systematic Reviews for Management and Organizational Studies – paper published in International Journal of Management Reviews by R.J. Adams (2017).
Reporting standards offer guidance on what information should be shared in the review regarding its process and methodology. By far the most popular of the standards is produced by PRISMA, which produces several guidelines that differ by synthesis type and methodological aspect.
- PRISMA - Preferred Reporting Items for Systematic Reviews and Meta-analysis - The most popular reporting standards used in systematic reviews. PRISMA also provides reporting standards for various other types of knowledge synthesis (including protocols).
- JARS Quant - Quantitative Meta-analysis Reporting Standards - The APA’s knowledge synthesis reporting guidelines.
- Systematic Review Checklist: A Standardized Technique for Assessing and Reporting Reviews of Life Cycle Assessment Data – based on the 2009 PRISMA guidelines, but specialized for LCA.
Because of the significance of undertaking a quality evidence synthesis, there are protocol registries that allow researchers to share their intent to publish on a topic prior to completion of the review. Protocol registries should be used both to notify the research community of your intent to publish, and to ensure that no one else has already done so on your topic.
- PROSPERO – a well-known registry for knowledge synthesis protocols on health-related topics
- JBI Protocols – a registry of published JBI protocols
- Open Science Framework Registries – a multidisciplinary registry for knowledge synthesis protocols
Search filters, also called hedges, are pre-assembled search strings for specific topics. While search filters vary in quality and design, they offer an excellent basis for new searches and - in some cases - may replace one component of a search entirely. For instance, the Cochrane RCT search filter is excellent and can easily replace any other such narrowing option. When using a search filter (either in part or in whole) or when using a substantive portion of a search from another review, it is expected that the source be cited and the creator given credit.
There are many places you may look for search filters, like:
- ES providers and publishers
- University ES Guides
- Search filters can also be published in articles and are searchable in databases by searching for "search filters" or "search hedges"
Critical appraisal (also called 'risk of bias assessment') is a step used in some kinds of evidence synthesis in which each article selected for inclusion in the final review is appraised for quality and likelihood of bias. There are many critical appraisal tools available, with some tools being preferred by (or more appropriate in) certain disciplines. A selection of the more common tools is below:
- Joanna Briggs Critical Appraisal Tools
- Critical Appraisal Skills Programme (CASP) Checklists
- 8 checklists
- No checklists for grey lit
- Centre for Evidence-Based Medicine (CEBM) Checklists
- 6 Checklists
- Some checklists available in multiple languages (including Chinese, German, Lithuanian, and Persian)
- Scottish Intercollegiate Guidelines Network (SIGN) Checklists
- Major producer of clinical practice guidelines
- 6 Checklists
This section contains one-off content that has been created to provide comprehensive answers to common questions.
This section includes links to videos about how to complete a systematic review. These videos are recordings from the Systematic Reviews Bootcamp 2021 that was offered by UNB Libraries and cover the entire process of performing a systematic review.
Just Systematic Searching
This section includes links to a video series that walks you through the process of developing a systematic search step by step. These videos focus solely on the search development and execution portion of the knowledge synthesis process. Videos should be watched in order, and each video has a task that should be completed before the next video is watched.
*** It may be worthwhile to watch video 16 early in your searching process, as it offers tips that will be useful throughout. ***
More Information More Information
- Richelle Witherspoon (she/her/hers)
- If you can't find the information you need, I am here to help. I provide support over email and Teams meetings.
- Information Services Librarian
- UNB Fredericton
Nursing, kinesiology, and psychology.