Our Wellington and Manawatū campuses are open, Auckland remains closed at AL3. More information.
Systematic Reviews: Getting Started
1. Before you get started
Determine whether a systematic review is the right format to answer your research question. There are many other kinds of reviews that share some of the same characteristics.
What is a systematic review?
A systematic review goes beyond a literature review. A systematic review:
- Uses strict methods for finding, appraising, and synthesising evidence to answer a specific research question. It requires an advanced literature search strategy and has rules for filtering results.
- Uses methods that are explained and justified to control bias
- Includes a peer review process for selection of articles
- Covers a large body of published literature where multiple published studies point to contradictory or uncertain results or outcomes
What type of review is right for you? (from Cornell University Library)
What is a systematic review? (from the University of Sydney Library). Includes timelines for each stage.
Systematic searches (from Yale Library). The first of a series of tutorials about searching health literature systematically. A useful introductory video.
Systematic quantitative literature review (from Griffith University's School of Environment and Science). Aimed at postgraduate environmental science students, but useful for others too. Includes video tutorials.
Systematic reviews and meta-analysis in business/management (from the University of Calgary Library)
Has someone published a recent review on your topic?
See section 5 for links to protocol registries. You can check for published or in-process reviews.
2. Who is involved?
Teams of 2 or more people undertake systematic reviews. For example:
- Lead reviewer (like a project lead - generally has systematic reviewing experience)
Who should do a systematic review? (from Cochrane.org)
3. What's your research question?
You need to look at:
- what's been done already?
- is there enough research in this area to support a systematic review?
- if you want to explore the literature further, you can have a look at using your search results to find out more about your subject area
Methods for formulating your question
Methods for question formulation include:
- PEO - Population or Problem, Exposure or Experience, Outcome
- PICO - Population, Intervention, Comparison, Outcomes
- PICo - Population, Interest, Context
- PICOT - Population, Intervention/Exposure, Comparison, Outcome, Time
- SPIDER - Sample, Phenomenon of Interest, Design, Evaluation, Research type
- SPICE - Setting, Perspective, Intervention, Comparison, Evaluation
Using PICO or PICo (from Murdoch University). Also includes a section on SPICE and SPIDER. Provides examples and worksheets for qualitative and quantitative health studies; can be applied to other subjects.
Rationale for well-formulated questions (from Cochrane.org). Outlines criteria and discusses scope. Focuses on writing for health and Cochrane reviews.
Systematic and systematic-like review toolkit (from Deakin University). Advice on selecting, using, searching and guidelines for frameworks focusing on topics relating to exercise, sport, health & medicine.
Ask; BestBETS for vets; EBVM toolkit; PICO.vet A selection of tools for veterinary medicine.
4. What will you include or exclude?
Here are some criteria for inclusion or exclusion:
- funding sources
- populations studied
- size of samples
- types of data
Inclusion and exclusion criteria (from the University of Melbourne). Brief examples of common criteria used.
Defining the criteria for including studies and how they will be grouped for the synthesis (from Cochrane.org). Lists key points along with a more detailed explanation of criteria. Useful for Cochrane Reviews but could be applied in other contexts.
5. Do you need to register your review protocol?
If you are publishing your systematic review you must register a protocol. Your protocol should be developed and registered before you start your data extraction.
What is a protocol? (from PRISMA - Preferred Reporting Items for Systematic Reviews and Meta-Analyses). PRISMA is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses.
Here are some protocol registries:
- Campbell Collaboration Social science & business focus
- Open Science Framework (OSF) Sciences focus
- Proposing and Registering New Cochrane Reviews Health focus
- PROSPERO International prospective register of systematic reviews
- Research Registry Human research studies, not just systematic reviews
- Systematic Review Register (Joanna Briggs Institute) Health focus
6. Advanced literature searching
Enhance your search strategy:
- Book chapters
- Clinical trials
- Grey literature (grey literature is published informally or non-commercially)
- Journal articles and conference papers
- Open access repositories
- Policy documents
- Unpublished research papers
- Working papers
7. Screen your studies and extract the data
Here are a few examples of data screening and managment tools:
Abstrackr Screening program for systematic reviewers. Free, but registration is required.
Covidence Platform for collaborative title and abstract screening, full-text review, risk-of bias assessment, and data extraction. Full access requires a paid subscription - check to see if your college has a subscription first.
Critical Appraisal Tools (Joanna Briggs Institute). Assess the trustworthiness, relevance, and results of published papers.
EndNote referencing software EndNote can manage duplicates, enable smart groups, rank and flag. Get the most out of EndNote - contact a Subject Librarian
MS Excel. The most basic tool for data screening and extraction. You can customise workbooks and spreadsheets to help screen data.
Nvivo Staff and students can install Nvivo on Massey-owned computers and on home computers. Using NVivo in systematic reviews (from the University of Queensland Library)
Rayyan A free system for collaborative citation screening and full-text selection.
8. Analyse and synthesise your evidence
Evaluate the quality of the research
Check for bias:
- attrition/incomplete data
- participant selection
- selective reporting
Categorise your findings
Identify commonalities & areas of difference, then combine all evidence together and summarise the research that addresses your question.
JBI (Joanna Briggs Institute) Levels of Evidence (PDF file). A grading approach to evaluating evidence.
The Level of Evidence (from EBVM). An overview of levels of evidence and how they can be applied to analyse literature within veterinary science.
Systematic Review Toolbox A database of tools and software to assist with a variety of evidence synthesis projects.
Massey's Software at home, including:
Nvivo Software for qualitative and mixed-methods research
SPSS Software for statistical analysis
9. Report your findings
- the steps you took and what you discovered
- themes that emerged
- gaps in the research
- contradictions in your findings
- best practices that you can highlight
PRISMA Checklist (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)
Writing the Report (from Dalhousie University Libraries)
Reporting results (from University of Michigan Library)
Academic Support Massey writing consultants can also give you feedback on academic writing, including structure, focus, paragraph structure, flow, presentation, use of sources, and referencing
10. Questions and feedback
Page authorised by University Librarian
Last updated on Tuesday 25 May 2021