Our Wellington and Manawatū campuses are open, Auckland remains closed at AL3. More information.

Systematic Reviews: Getting Started

1. Before you get started

Determine whether a systematic review is the right format to answer your research question. There are many other kinds of reviews that share some of the same characteristics. 

What is a systematic review?

A systematic review goes beyond a literature review. A systematic review:

  • Uses strict methods for finding, appraising, and synthesising evidence to answer a specific research question. It requires an advanced literature search strategy and has rules for filtering results.
  • Uses methods that are explained and justified to control bias
  • Includes a peer review process for selection of articles
  • Covers a large body of published literature where multiple published studies point to contradictory or uncertain results or outcomes

Arrow What type of review is right for you? (from Cornell University Library)
Arrow What is a systematic review? (from the University of Sydney Library). Includes timelines for each stage.
Arrow Systematic searches (from Yale Library). The first of a series of tutorials about searching health literature systematically. A useful introductory video.
Arrow Systematic quantitative literature review (from Griffith University's School of Environment and Science). Aimed at postgraduate environmental science students, but useful for others too. Includes video tutorials.
Arrow Systematic reviews and meta-analysis in business/management (from the University of Calgary Library)

Has someone published a recent review on your topic?

See section 5 for links to protocol registries. You can check for published or in-process reviews.

2. Who is involved?

Teams of 2 or more people undertake systematic reviews. For example:

  • Lead reviewer (like a project lead - generally has systematic reviewing experience)
  • Reviewer(s)
  • Librarian
  • Statistician

Arrow Who should do a systematic review? (from Cochrane.org)

3. What's your research question?

You need to look at:

Methods for formulating your question

Methods for question formulation include:

  • PEO - Population or Problem, Exposure or Experience, Outcome
  • PICO - Population, Intervention, Comparison, Outcomes
  • PICo - Population, Interest, Context
  • PICOT - Population, Intervention/Exposure, Comparison, Outcome, Time
  • SPIDER - Sample, Phenomenon of Interest, Design, Evaluation, Research type
  • SPICE - Setting, Perspective, Intervention, Comparison, Evaluation

Arrow Using PICO or PICo (from Murdoch University). Also includes a section on SPICE and SPIDER. Provides examples and worksheets for qualitative and quantitative health studies; can be applied to other subjects.
Arrow Rationale for well-formulated questions (from Cochrane.org). Outlines criteria and discusses scope. Focuses on writing for health and Cochrane reviews.
Arrow Systematic and systematic-like review toolkit (from Deakin University). Advice on selecting, using, searching and guidelines for frameworks focusing on topics relating to exercise, sport, health & medicine. 
Arrow Ask; BestBETS for vets; EBVM toolkit; PICO.vet A selection of tools for veterinary medicine.

4. What will you include or exclude?

Here are some criteria for inclusion or exclusion:

  • funding sources
  • language
  • methodologies
  • populations studied
  • size of samples
  • types of data

Arrow Inclusion and exclusion criteria (from the University of Melbourne).  Brief examples of common criteria used. 
Arrow Defining the criteria for including studies and how they will be grouped for the synthesis (from Cochrane.org). Lists key points along with a more detailed explanation of criteria. Useful for Cochrane Reviews but could be applied in other contexts.

5. Do you need to register your review protocol?

If you are publishing your systematic review you must register a protocol. Your protocol should be developed and registered before you start your data extraction.

Arrow What is a protocol? (from PRISMA -  Preferred Reporting Items for Systematic Reviews and Meta-Analyses). PRISMA is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses.
Arrow Here are some protocol registries:

6. Advanced literature searching

Enhance your search strategy:

Arrow Searching: beyond the basics
Arrow Selecting databases to search
Arrow Receiving new research alerts

Potential sources:

  • Blogs
  • Book chapters
  • Clinical trials
  • Grey literature (grey literature is published informally or non-commercially)
  • Journal articles and conference papers
  • Nzresearch.org.nz
  • Open access repositories
  • Patents
  • Policy documents
  • Standards
  • Theses
  • Unpublished research papers
  • Working papers

7. Screen your studies and extract the data

Here are a few examples of data screening and managment tools:

Arrow Abstrackr Screening program for systematic reviewers. Free, but registration is required.
Arrow Covidence Platform for collaborative title and abstract screening, full-text review, risk-of bias assessment, and data extraction. Full access requires a paid subscription - check to see if your college has a subscription first.
Arrow Critical Appraisal Tools (Joanna Briggs Institute). Assess the trustworthiness, relevance, and results of published papers.
Arrow EndNote referencing software EndNote can manage duplicates, enable smart groups, rank and flag. Get the most out of EndNote - contact a Subject Librarian
Arrow MS Excel. The most basic tool for data screening and extraction. You can customise workbooks and spreadsheets to help screen data.
Arrow Nvivo Staff and students can install Nvivo on Massey-owned computers and on home computers. Using NVivo in systematic reviews (from the University of Queensland Library)
Arrow Rayyan A free system for collaborative citation screening and full-text selection.

8. Analyse and synthesise your evidence

Evaluate the quality of the research

Check for bias:

  • attrition/incomplete data
  • blinding
  • participant selection
  • selective reporting

Categorise your findings

Identify commonalities & areas of difference, then combine all evidence together and summarise the research that addresses your question. 

Arrow JBI (Joanna Briggs Institute) Levels of Evidence (PDF file). A grading approach to evaluating evidence.
Arrow The Level of Evidence (from EBVM). An overview of levels of evidence and how they can be applied to analyse literature within veterinary science.

Tools

Arrow Systematic Review Toolbox A database of tools and software to assist with a variety of evidence synthesis projects.
Arrow Massey's Software at home, including:
Arrow Nvivo Software for qualitative and mixed-methods research
Arrow SPSS Software for statistical analysis

9. Report your findings

Describe:

  • the steps you took and what you discovered
  • themes that emerged
  • gaps in the research
  • contradictions in your findings
  • recommendations
  • best practices that you can highlight

Arrow PRISMA Checklist (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)
Arrow Writing the Report (from Dalhousie University Libraries)
Arrow Reporting results (from University of Michigan Library)
Arrow Academic Support Massey writing consultants can also give you feedback on academic writing, including structure, focus, paragraph structure, flow, presentation, use of sources, and referencing

10. Questions and feedback

If you have any questions or would like further information, please contact a Subject or Māori Services librarian.

Did this information help you? Tell us how we could improve it - email Subjectlibrarians@massey.libanswers.com