Clinical scenario: Man with extrasystoles on sotalol.

Are the results of this study valid?

Evidence about harm can come from a number of different study types. Ideally we’d like to see a high quality systematic review of randomised trials but these aren’t easy to find because RCTs aren’t always feasible for issues of harm. As a result, we usually find evidence about harm in cohort studies (groups of patients who are and aren’t exposed to the treatment are followed up for the outcome of interest) and case-control studies (patients with the outcome of interest are matched with patients without the outcome and investigators look retrospectively to determine exposure). Case-control studies are useful when the outcome of interest is rare or when the required follow-up is long. The strength of inference that can be drawn from a case-control study is limited because they are more susceptible to bias.

Returning to our clinical scenario from the question formulation tutorial:

You see a 50 year old man who asks for a repeat prescription of sotalol which he has been taking for extrasystoles for several years. He has a remote history of an MI. You haven’t seen him previously and are concerned about the proarrhythmic properties of sotalol given what is known about other antiarrhythmics.

During the tutorial on clinical questions we formulated the question: In a man with extrasystoles and a remote history of MI, does treatment with sotalol increase his risk of death?

Searching the literature we found an RCT from the Lancet (1996;348:7-12).

How do we critically appraise this harm paper? We’ll start off by considering validity first and the following list outlines the questions that we need to consider when deciding if a harm paper is valid.

  1. Were there clearly defined groups of patients, similar in all important ways other than exposure to the treatment or other cause?

    Consider the following table:

    Adverse Event Totals
    Present (Case) Absent (Control)
    Exposure to treatment (RCT or cohort) a b a + b
    No exposure to treatment (RCT or cohort) c d c + d
    Totals a + c b + d a + b + c + d

    This first question is easy to answer if we’ve been able to find an RCT during our search. Randomisation should make the 2 groups of patients similar for all causes of the outcome that we are interested in. In an RCT, patients in the experimental treatment group would be in cells a or b in the table above and patients in the control group would be in cells c or d.

    Returning to our clinical scenario, we have been fortunate in our search and have managed to find an RCT and are satisfied that patients are similar in all important ways other than exposure to sotalol.

    However, there’s not always an RCT available to answer our questions and indeed more frequently we find cohort or case control studies to answer our questions about harm and etiology. In a cohort study, 2 groups of patients are followed – one group with the exposure to the treatment (a+b in the table) and one group without the exposure (c+d) – for the development of the outcome of interest (either a or c ). Because the decision about who receives treatment is not randomised, exposed patients may differ from nonexposed patients for important determinants of the outcome (these determinants are called confounders). Investigators should document characteristics of patients and either show that they are similar or adjust for the confounders that they identify. This is limited by the fact that investigators can only adjust for confounders that are known and that have been measured.

    In case control studies, people with the outcome of interest (cases = a+c) are identified along with those without it (controls = b+d). The proportion of each group who were exposed to the putative agent is assessed. Case control studies are susceptible to more bias than cohort studies because confounders that are transient or that lead to early death won’t get measured. We also need to ensure when reading a case control study, that people in the control group had the same opportunity for exposure as people in the case group. For example, if we found a case control study looking at the association between sotalol and sudden cardiac death and its investigators assembled people with sudden cardiac death as the cases but excluded patients with atrial fibrillation from the control group, we’d be concerned that the association found between sotalol and sudden cardiac death could be spurious.

  2. Were treatments/exposures and clinical outcomes measured in the same ways in both groups? (Was the assessment of outcomes either objective or blinded to exposure?)
    The application of explicit criteria for the outcomes of interest, a discussion of how they were applied and evidence that they were applied without knowledge of which group the patient was in is important. Blinding is crucial if any judgment is required to assess the outcome (in RCTs and cohorts studies) or the exposure (in case control studies). For example, an unblinded investigator may search more aggressively for outcomes in people with exposure to the putative agent. Similarly, people with the adverse outcome may be more likely to have brooded about their situation and may have greater incentive to recall possible exposure. Therefore we would want patients and interviewers to be blind to the study hypothesis.

    In the RCT that we retrieved, the outcome was death and was the same for both groups.

  3. Was the follow-up of the study patients sufficiently long (for the outcome to occur and complete)?

    If follow-up is short, it may be that too few study patients will have the outcome of interest, thus providing little information of use to a patient. For example, if investigators were looking at the association between cancer and a particular agent and the follow-up time was 1 month, this would be too short for the investigators to see a clinically important effect.

    The more people who are unavailable for follow-up, the less accurate the estimate of the risk of the outcome is. Losses may occur because patients are too ill (or too well) to be followed or may have died, and the failure to document these losses threatens the validity of the study.

    The RCT that we found was stopped early because an increased risk of death was noted.

  4. Do the results of the harm study fulfil some of the diagnostic tests for causation?

    1. Is it clear that the exposure preceded the onset of the outcome?
      We’d want to make sure that the exposure occurred before the outcome and that it wasn’t just a marker that the outcome was already underway. With an RCT, the exposure clearly precedes the outcome as with the trial that we found. If it’s a case control study, this question becomes more difficult to answer, and more important to ascertain.
    2. Is there a dose-response gradient?
      With larger doses of the agent, was there an increased risk of the outcome event? In the study we retrieved, this wasn’t tested since the investigators looked at one dose of sotalol.
    3. Is there any positive evidence from a ‘dechallenge-rechallenge’ study?
      This occurs when the outcome event disappears (or decreases in intensity) when the putative agent is withdrawn and reappears when it is reinstituted. This couldn’t be done in the RCT we found because the outcome was death.
    4. Is the association consistent from study to study?
      Or, is this the only study where the association has been identified? We would be happy to see that several studies have looked at this question and have come to the same conclusion (or even better, if there was a systematic review of the topic). Only 1 RCT has had sufficient power to look at the use of sotalol and the risk of death.
    5. Does the association make biological sense?
      If the association between outcome and exposure makes biological sense, a causal relationship is more plausible. The results of the sotalol RCT are consistent with findings from studies that have looked at other antiarrythmics (e.g. CAST).
      If the study fails any of the above criteria, we need to decide if the flaw is significant and threatens the validity of the study. If this is the case, we’ll need to look for another study. Returning to our clinical scenario, the paper we found satisfies all of the above criteria and we will proceed to assessing it for importance.

Are the results of this study important?

What is the magnitude and precision of the association between the exposure and the outcome?

Let’s begin by drawing a 2×2 table using the data from the RCT that we found.

Adverse Event Totals
Present (Case) Absent (Control)
Experimental group (d-solatol)
a + b
Control group (placebo)
c + d
a + c
b + d
a + b + c + d

For RCTs and cohort studies, we look at the risk of the event in the treatment group relative to the risk of the event in the untreated patient. This ‘relative risk’ is calculated as:

$$mathit{RR} =frac{[ a/(a+b) ]}{[ c/(c+d) ]}$$

Using the values in the table, the relative risk for death in patients receiving d-sotalol is:

mathit{RR}&= frac{ [ 78/1549 ]}{[ 48/1572 ]} \\
&= 1.65

Case control studies sample outcomes, not exposure and therefore we can’t calculate the relative risk. Instead, the strength of association is estimated indirectly using the odds ratio = ad/bc.

How big should the relative risk (RR) or odds ratio (OR) be for us to be impressed by it? OR and RR > 1 indicate that there is an increased risk of the adverse outcome with the exposure. Because cohort studies and case control studies are susceptible to many biases, we need to ensure that the OR/RR is greater than that which could occur from bias alone. We also need to look at the confidence interval around the OR and RR to see how precise the estimate is.

A more clinically useful measure than the OR and RR is the number of patients that we’d need to treat with the putative agent in order to cause 1 additional harmful event (number needed to harm or NNH). Using the OR, the NNH can be calculated as:

$$mathit{NNH} = frac{[ mathit{PEER} (mathit{OR}-1) + 1 ]}{[mathit{PEER} (mathit{OR}-1)times(1-mathit{PEER}) ]}$$

Where PEER = the patient’s expected event rate

Calculate the confidence level for the OR, RR, and NNH online. Try it now!

Alternatively, we can refer to the tables below for this information. We can see from these tables that for different PEER, the same OR can generate very different NNHs.

When OR < 1:

Adapted from John Geddes, 1999
For Odds Ratios LESS than 1
0.9 0.8 0.7 0.6 0.5 0.4 0.3
Patient Expected Event Rate (PEER) 0.05 209 104 69 52 41 34 29
0.10 110 54 36 27 21 18 15
0.20 61 30 20 14 11 10 8
0.30 46 22 14 10 8 7 5
0.40 40 19 12 9 7 6 4
0.50 38 18 11 8 6 5 4
0.70 44 20 13 9 6 5 4
0.90 101 46 27 18 12 9 4

When OR > 1:

Adapted from John Geddes, 1999
For Odds Ratios GREATER than 1
1.1 1.25 1.5 1.75 2 2.25 2.5
Patient Expected Event Rate (PEER) 0.05 212 86 44 30 23 18 16
0.10 113 46 24 16 13 10 9
0.20 64 27 14 10 8 7 6
0.30 50 21 11 8 7 6 5
0.40 44 19 10 8 6 5 5
0.50 42 18 10 8 6 5 4
0.70 51 23 13 10 9 8 7
0.90 121 55 33 25 22 19 18

We can also convert the RR to an NNT/NNH using the following equations:

$$text{For $RR$ < 1}$$ $$mathit{NNT} = 1/(1-mathit{RR}) times mathit{PEER}$$ $$text{For $RR$ > 1}$$

$$text{$NNT$ (or $NNH$)} = 1/(mathit{RR}-1) times mathit{PEER}$$

Using the PEER (3.1%) from the study we found and the RR (1.65) that we calculated, the NNH for death from d-sotalol in the study is:

mathit{NNH} &= 1/(1.65-1)times 0.031\\
&= 50

Therefore we would need to treat 50 people with d-sotalol to cause 1 additional death. We can also calculate the confidence interval around this estimate using the inverse of the confidence interval for the absolute risk increase.

Where to go from here?

Now that we’ve decided our article is both valid and important, we need to decide if we can apply it to our patient.

Other options:

  • Do you want to consider the validity of a harm article?
  • Do you want to see a ‘CAT’ for this paper?
  • Do you want to learn about critically appraising:
    • Diagnosis articles
    • Prognosis articles
    • Therapy articles – single trials
    • Therapy articles – systematic review
  • Do you want some practice critically appraising harm/etiology articles from other clinical specialties?
  • Do you want more reading about critically appraising harm/etiology papers?

Harm/Etiology articles from other clinical specialties

  • Child Health

    Do pregnant women receiving Vitamin A supplements have a higher risk of children having birth defects?

    Rothman KJ, Moore LL, Singer MR et al. Teratogenicity of high Vitamin A intake. NEJM 1995;333(21):1369-1373.

  • Gastroenterology and Hepatology

    In critically ill trauma patients with gastric erosions are sucralfate or ranitidine associated with fewer infectious complications?

    O’Keefe GE, Gentilello LM, and Maier RV. Incidence of infectious complications associated with the use of Histamine-2-receptor antagonists in critically ill trauma patients. Annals of Surgery 1998;227:120-125.

  • General Practice

    In patients with extrasystoles does sotalol increase the risk of sudden cardiac death?

    Pratt CM et al. Mortality in the Survival With ORal D-sotalol (SWORD) trial: why did patients die? Am-J-Cardiol 1998: 81:869-76.

    Waldo AL et al. Effect of d-sotalol on mortality in patients with left ventricular dysfunction after recent and remote myocardial infarction. The SWORD Investigators. Survival With Oral d-Sotalol. Lancet 1996; 348:7-12.

  • General Surgery

    Is radical lymphadenectomy more dangerous than conventional resection in patients with gastric cancer?

    Bonenkamp JJ et al. Randomised comparison after D1 and D2 dissection for gastric cancer in 1996 Dutch patients. Lancet 1995;345:758-8. Geriatric Medicine

    In patients taking calcium antagonists for hypertension, are they at increased risk of cancer?

    Pahor M et al. Calcium-channel blockade and incidence of cancer in aged populations. Lancet 1996;348: 493-97. (also see 487-9 and 541-2)

  • Mental Health

    In patients taking sertraline therapy what is the likelihood of an adverse outcome to pregancy?

    Kulin, NA, Pastuszak, A, Sage, AR et al. Pregnancy outcome following maternal use of the new selective serotonin reuptake inhibitors: a prospective controlled multicenter study. JAMA 279 (8):609-610,1998.

  • Neonatal Medicine

    In children who were born full-term with normal birth weight, is maternal infection a possible cause of congenital cerebral palsy?

    Grether JK., Nelson KB.Grether JK., Nelson KB.
    Maternal infection and cerebral palsy in infants of normal birth weight. JAMA 1997; 278:207-211.

  • Nursing

    What is the association between oral contraceptive use in women and the risk for cardiovascular disease?

    Mant J, Painter R, Vessey M. Risk of myocardial infarction, angina and stroke in users of oral contraceptives: an updated analysis of a cohort study. Br J Obstet Gynaecol 1998;105:890-6.

  • Purchasing

    What is the risk of death in women who are taking oral contraceptives?

    Beral V, Hermon C, Kay C, Hannaford P, Darby S, Reeves G. Mortality associated with oral contraceptive use: 25 year follow up of cohort of 46,000 women from Royal College of General Practitioners’ oral contraception study. BMJ 1999;318:96-100.

Further reading on harm