Therapy – Systematic Review

Clinical scenario: man with a stroke who gets admitted to stroke unit.

Are the results of this study valid?

Anytime we do a search, finding a systematic review should be our first goal. (Note: the critical appraisal of systematic reviews requires the skills for appraising individual studies and therefore you might want to review the section on critically appraising single therapy trials before tackling this one).

A systematic review is a summary of the medical literature that uses explicit methods to systematically search, critically appraise and synthesise the literature on a specific topic. Let’s start by returning to our clinical scenario from the question formulation tutorial:

You admit a 75 year old man with a stroke (left sided weakness) who is having trouble ambulating, feeding, bathing and dressing himself. He has hypertension but it is well controlled with a diuretic. He is otherwise well and now that he is medically stable you decide after discussion with him to transfer him to a stroke unit. His family asks to see you because they are concerned about this transfer. They live very close to the acute care hospital and wonder why he can’t stay on the general medical ward where he currently is. You arrange to meet with him and his family to discuss their concerns. In the meantime, you decide to review the evidence for the use of stroke units.

In the tutorial on clinical questions we formulated the following question: In an elderly man with a stroke, does admission to a stroke unit decrease the risk of death and dependency?

Our search of the literature to answer this question retrieved an article from the The Cochrane Library (Stroke Trialists’ Collaboration. Cochrane Library, Update Software, 1999, Issue 4).

How can we critically appraise this paper? We’ll start off by considering validity and the following list outlines the questions that we need to consider when deciding if a systematic review is valid.

  1. Is this a systematic review of randomised trials?

    In our discussion of single therapy trials, we reviewed how randomised trials are susceptible to less bias than nonrandomised trials. Systematic reviews of RCTs provide the highest level of evidence but systematic reviews of trials that are of lower quality can compound the problems of the individually misleading trials and produce a lower quality of evidence.

    The review we found is a Cochrane review and follows the methodology outlined by the Methods Group. It includes randomised and quasi-randomised trials comparing inpatient stroke unit care with conventional care.

  2. Does it include a methods section that describes: a) finding and including all the relevant trials, and b) assessing their individual validity?

    We need to review the study to determine how the authors identified the studies and if this process was sufficiently complete. For example, we’d like to see that they contacted authors looking for nonpublished studies, checked bibliographies of all the articles retrieved, handsearched journals and included articles from all languages and from drug company databases. Negative studies are less likely to get published and therefore if these are not searched for, it could lead to a false positive conclusion.
    We’d also want to see that the study includes information on how individual studies were appraised for validity. We’d also be reassured if multiple reviews were done and if there was good agreement amongst the reviewers.
    In the Cochrane review we identified, the authors searched the Cochrane trials registry, the reference lists of all articles and contacted researchers to identify published and nonpublished studies. Two reviewers independently assessed the quality of the trials and determined their eligibility for inclusion in the review. Although a formal scoring system was not used to record methodological quality, the authors did record the method of allocation concealment, the duration of follow-up, whether or not an intention to treat analysis was done and whether theirs was a blinded assessment of outcomes.

  3. Were the results consistent from study to study?

    Did all the studies show results in the same direction? The results shouldn’t be exactly the same for all studies but we should be concerned if some studies showed a significant beneficial effect and others significantly excluded any benefit or showed a significant harmful effect. The investigators should test to determine if this lack of consistency (or heterogeneity) was statistically significant. If indeed it was significant, we would like to see that they have explored the reasons why this may have been observed.
    The authors of the Cochrane review tested for heterogeneity and it was not significant.

A finer point:

  1. Were individual patient data (or aggregate data) used in the analysis?

    This is a finer point to consider when assessing validity of a systematic review. We’d feel more assured if this was done, especially with regard to reports of subgroups.If the review fails any of the above criteria, we need to decide if the flaw is significant and threatens the validity of the study. If this is the case, we’ll need to look for another study to answer our question, Returning to our clinical scenario, the review we found satisfies all of the above criteria and we will proceed to assessing it for importance.

Are the results of this study important?

What is the magnitude and precision of the treatment effect?

The results of systematic reviews are commonly presented as odds ratios (OR) or relative risks (RR). The ratio of the odds of having the outcome of interest in the experimental group relative to the odds of that same event occurring in the control group is termed the odds ratio. The ratio of the risk of the outcome of interest in the experimental group relative to the risk of the same outcome in the control group is called the relative risk.

Like the relative risk reduction, neither the odds ratio nor the relative risks preserves the baseline risk of the outcome event, and, they are difficult to translate at the point of care. This can be overcome by converting them to a number needed to treat (NNT).

To calculate the NNT/NNH for any OR or RR we can use the following equations:

$$text{For $RR$ < 1: }$$

$$mathit{NNT}=1/(1-mathit{RR})timesmathit{PEER}$$

$$text{For $RR$ > 1:}$$

$$mathit{NNT}=1/(mathit{RR}-1)timesmathit{PEER}$$

$$text{For $OR$ < 1:}$$

$$mathit{NNT}=frac{1-[mathit{PEER} times (1-mathit{OR})]}{(1-mathit{PEER}) mathit{(PEER)} times (1-mathit{OR})}$$

$$text{For $OR$ > 1:}$$

$$mathit{NNT} = frac{1 + [mathit{PEER}times (mathit{OR}-1)]}{(1-mathit{PEER})times(mathit{PEER}) times (mathit{OR}-1)}$$

Alternatively, (and more simply) we could use the tables below to obtain the NNT for specific OR and PEERs.

$$text{When $OR$ < 1:}$$

Adapted from John Geddes, 1999
For Odds Ratios LESS than 1
0.9 0.8 0.7 0.6 0.5 0.4 0.3
Patient Expected Event Rate (PEER) 0.05 209 104 69 52 41 34 29
0.10 110 54 36 27 21 18 15
0.20 61 30 20 14 11 10 8
0.30 46 22 14 10 8 7 5
0.40 40 19 12 9 7 6 4
0.50 38 18 11 8 6 5 4
0.70 44 20 13 9 6 5 4
0.90 101 46 27 18 12 9 4

$$text{When $OR$ } > 1:$$

Adapted from John Geddes, 1999
For Odds Ratios GREATER than 1
1.1 1.25 1.5 1.75 2 2.25 2.5
Patient Expected Event Rate (PEER) 0.05 212 86 44 30 23 18 16
0.10 113 46 24 16 13 10 9
0.20 64 27 14 10 8 7 6
0.30 50 21 11 8 7 6 5
0.40 44 19 10 8 6 5 5
0.50 42 18 10 8 6 5 4
0.70 51 23 13 10 9 8 7
0.90 121 55 33 25 22 19 18

Returning to our clinical scenario, we are interested in the outcome death and dependency. The OR for these events is 0.75 (95 % CI 0.65 to 0.87). From the review, we’ve obtained a PEER for death and dependency of 68% and using the table above, we can estimate that the NNT is approximately 20.

Where to go from here?

Now that we’ve decided our article is both valid and important, we need to decide if we can apply it to our patient.

Other options:

  • Do you want to consider the validity of a systematic review?
  • Do you want to see a ‘CAT’ for this paper? (not yet online)
  • Do you want to learn about critically appraising:
    • Diagnosis articles
    • Prognosis articles
    • Therapy articles – single trials
    • Harm articles
  • Do you want some practice critically appraising systematic reviews from other clinical specialties?
  • Do you want more reading about critically appraising systematic reviews?

Systematic Reviews from other clinical specialties

  • Child Health

    In children, can home visiting reduce the injury rate?

    Roberts, I., Kramer, M.S., and Suissa, S. Does home visiting prevent childhood injury? BMJ 312:29-33,1996.

  • Critical Care Medicine

    In critically ill patients, does the use of albumin compared to crystalloid for volume resuscitation increase mortality?

    Roberts I. Cochrane injuries group albumin reviewers.
    BMJ 1998;317:235-240.

    In critically ill patients requiring mechanical ventilation, does prophylactic antibiotics reduce the risk of respiratory tract infection?

    Effectiveness of antibiotic prophylaxis in critically ill adult patients: systematic review of randomised controlled trials. BMJ 1998;316:1275-85.

  • Gastroenterology and Hepatology

    In a patient with steroid induced osteoporosis, does treatment with bisphosphonates reduce bone pain, fracture rate and improve bone mineral density?

    Homik J, Cranney A, Shea B, et al. Bisphosphonates for steroid induced osteoporosis. CDSR 1999;issue 1.

  • General Practice

    Is corticosteroid injection or physiotherapy more effective in patients with rotator cuff tendinitis?
    Green S, Buchbinder R, Glazier R, Forbes A. Interventions for shoulder pain (Cochrane Review). In: The Cochrane Library, Issue 1, 1999. Oxford: Update Software.

  • Geriatric Medicine

    In a patient with a stroke, does admission to a stroke unit decrease the risk of death and dependency?

    Stroke unit trialists’ collaboration.
    Collaborative systematic review of the randomised trials of organised inpatient (stroke unit) care after stroke. BMJ 1997;314:1151-9.

  • Mental Health

    In an adult with dysthymia, are antidepressants effective in lifting mood?

    Lima M, Moncrieff J. Drugs versus placebo for dysthymia (Cochrane Review). In: The Cochrane Library, Issue 1, 1999. Oxford: Update Software.

  • Neonatal Medicine

    In a very premature infant, does indomethacin prophylaxis improve survival without severe disability?

    Fowlie PW. Prophylactic intravenous indomethacin in very low birthweight infants (Cochrane Review). In: The Cochrane Library, 1997. Oxford: Update Software.

  • Nursing

    Is a self-management programme that includes asthma education plus regular review by health professionals, effective in improving health outcomes for adults with asthma?

    Gibson PG, Coughlan J, Wilson AJ et al. Self-management education and regular practitioner review for adults with asthma (Cochrane Review, latest update 25 May 1998). The Cochrane Library. Oxford: Update Software.

  • Therapy Practice

    In a patient with COPD, does an in-patient pulmonary rehabilitation program improve strength, endurance, and quality of life?

    LacasseY, Wong E, Guyatt GH, King D, Cook DJ, Goldstein RS. Meta-analysis of respiratory rehabilitation in chronic obstructive pulmonary disease. Lancet 1996;348:1115-9.

  • Purchasing

    In routine resuscitation of critically ill patients, are colloids or crystalloids more effective?

    Schierhout G, Roberts I, Alderson P. Colloids compared to crystalloids in fluid resuscitation of critically ill patients (Cochrane Review). In: The Cochrane Library, 4, 1998. Oxford: Update Software.

Further reading on systematic reviews