Background Several recent trials in chronic obstructive pulmonary disease (COPD) have assessed the effectiveness of the fluticasone–salmeterol combination inhaler in preventing COPD exacerbations, while finding an increased risk of pneumonia. The number needed to treat (NNT) is a simple measure to perform the comparative benefit–risk impact, but its calculation involving repeated outcome events such as COPD exacerbations has been incorrect. We describe the proper methods to calculate the NNT and, using data from published trials, apply them to evaluate the relative impact of fluticasone–salmeterol treatment on exacerbations and pneumonias in patients with COPD.
Methods We review the fundamental definition of NNT and quantify it for situations with varying follow-up times. We review the ‘event-based’ NNT, proposed and used for repeated event outcomes, show its inaccuracy, describe its proper use and provide an approximate formula for its application.
Results We show that a 1-year trial of the fluticasone–salmeterol combination versus salmeterol used the incorrect event-based approach to calculate the NNT as two patients that need to be treated for 1 year to prevent one COPD exacerbation, when the proper calculation results in a NNT of 14. In contrast, 20 patients need to be treated to induce one pneumonia case. For the TORCH trial, the NNT is 44 patients treated for 3 years with fluticasone–salmeterol versus salmeterol to prevent one exacerbation compared with 16 patients to induce one pneumonia case.
Conclusions The NNT is a useful measure of the effect of drugs, but its proper calculation is essential to prevent misleading clinical practice guidelines.
Statistics from Altmetric.com
What is the key question?
The number needed to treat (NNT) is a simple measure to assess the benefit-risk impact of a drug, but its calculation involving repeated outcome events such as chronic obstructive pulmonary disease (COPD) exacerbations has been inaccurate in several trials of the fluticasone–salmeterol combination.
What is the bottom line?
When properly computed, the NNT measure shows that pneumonias are just as important as exacerbations in the benefit–risk assessment of the fluticasone–salmeterol combination in COPD.
Why read on?
The correct computational approaches for the NNT described in the paper illustrate the discrepancies: the NNT for the fluticasone–salmeterol combination versus salmeterol in a recent trial is in fact 14 patients who need to be treated for 1 year to prevent one COPD exacerbation, not the reported 2 patients estimated by the inaccurate approach.
The TORCH (Towards a Revolution in Chronic obstructive pulmonary disease (COPD) Health) randomised controlled trial was the first to identify the increased risk of pneumonia associated with inhaled corticosteroid use in patients with COPD.1 It reported that patients receiving fluticasone propionate combined or not combined with salmeterol had a significant 52% increase (HR 1.52; 95% CI 1.32 to 1.76) in the incidence of pneumonia adverse events during the 3-year follow-up compared with patients receiving placebo or salmeterol alone.2 Subsequently several other randomised trials and meta-analyses confirmed this increase in risk.3–8
An issue that has been raised regarding this increase in the risk of pneumonia is its relative importance vis-à-vis the prevention of COPD exacerbations. The Investigating New Standards for Prophylaxis in Reducing Exacerbations (INSPIRE) trial, which compared the fluticasone–salmeterol combination with tiotropium in a 2-year trial conducted an extensive investigation of pneumonia adverse events.4 ,9 It identified 87 pneumonia reports from adverse event records compared with 2255 COPD exacerbations, concluding that ‘pneumonia is much less frequent than exacerbation in COPD’. This conclusion can be misunderstood to imply that the adverse impact of fluticasone on pneumonia is much less important than its benefit on COPD exacerbations.
Using the relative frequency of two outcomes, such as exacerbations and pneumonias, to judge the relative importance of the risk and benefit of a drug can be misleading since it does not account for the actual effect of the drug. Instead, the number needed to treat (NNT) is a simple and valid tool that permits such a comparative impact to be assessed.10 This measure has been advocated and used in several recent studies of treatment for COPD.11 However, its calculation when dealing with outcomes involving repeated events such as exacerbations has been criticised, leading to possibly distorted estimates of the NNT.12–14
In this paper, we review these methods to calculate the NNT and provide formulae for the proper techniques. We also apply these methods to assess the relative impact of fluticasone–salmeterol on exacerbations and pneumonias in patients with COPD using data from several recently published randomised trials.
The NNT measure
The NNT to assess the effectiveness of a drug treatment is measured from the difference between two groups of patients in the cumulative incidence of the outcome (CI) over a fixed follow-up time period. This difference CI0−CI1, where 0 represents the reference treatment or placebo and 1 the treatment under study, represents the proportion of patients for whom the outcome was prevented due to the drug. Inverting this difference (1/(CI0−CI1)) will produce the number of patients that need to be treated by the drug to prevent one patient with the outcome, in other words the NNT.10 For example, a drug trial that produces an incidence of the outcome after 1 year of 1/100 (0.01) in the drug-treated group compared with 3/100 (0.03) in the placebo group implies that the drug prevents two outcome events per 100 treated for 1 year, corresponding to an NNT of 50 (1/0.02). That is, 50 patients need to be treated continuously for 1 year to prevent one patient incurring the outcome event. The NNT is now used extensively in randomised trials and observational studies to provide an additional and user-friendly measure of the impact of a drug or treatment on a given disease outcome.10 ,15
While the calculation of the NNT as shown is arithmetically simple, this is only so when trials follow all patients up for the same time and observe a single outcome event per patient. In practice, however, most trials involve unequal follow-up times while some will study recurrent outcomes with multiple events, such as COPD exacerbations. The NNT calculation is then not as straightforward in these situations.12–14
Unequal follow-up and single event: patient-based NNT
When follow-up times vary between patients, the CI of an outcome event cannot be calculated as a proportion of subjects. It must instead be based on the Kaplan–Meier approach, which estimates the CI accounting for variable follow-up.16 The NNT can then be directly computed by inverting the difference in the CI of the outcome between the two groups at the desired time point of follow-up from the Kaplan–Meier curves, or alternatively, approximated from the HR.17 It will estimate the number of patients that need to be treated by the drug to prevent one patient with the outcome over the given follow-up time. An example is the Kaplan–Meier curve provided for time to pneumonia in the INSPIRE trial.9 From the curves, the 2-year CI of pneumonia is 0.094 and 0.049 for fluticasone–salmeterol combination and tiotropium respectively for NNT=1/(0.094–0.049)=22 patients for 2 years to induce a pneumonia. The curves also permit the NNT to be computed at any other time point during follow-up, for example at 1 year, NNT=1/(0.06–0.03)=34 patients for 1 year to induce a pneumonia.
Unequal follow-up and multiple events: event-based NNT
With outcomes involving an event that can recur multiple times during follow-up, such as COPD exacerbations, most trials use the incidence rate (IR) to quantify the frequency. It is computed as the total number of events divided by the total amount of person-time, to account for varying follow-up times. The effect of the drug treatment is then measured by the difference in IRs between two groups, namely IR0−IR1, where 0 represents the reference treatment or placebo and 1 the treatment under study. This difference represents the IR of prevented events per person-moment. Several studies have, in this case, used an alternative formula to compute the corresponding NNT as 1/(IR0−IR1). This NNT, based not on the CI but on the IR, was interpreted as the NNT to prevent one ‘event’ over a given time period.11
An example is the trial by Anzueto that randomised close to 800 patients with COPD to either the fluticasone–salmeterol combination or salmeterol alone, and followed them for 52 weeks.6 With varying follow-up times, the IR of moderate or severe exacerbations, which could occur more than once during a patient's follow-up, was 1.59 per patient-year in the salmeterol group and 1.10 in the fluticasone–salmeterol group. The authors thus computed the NNT as 1/(1.59–1.10)=2.04 rounded to 2, which was interpreted as ‘two subjects need to be treated for 52 weeks … in order to prevent one exacerbation per year’.
This alternative formula, called the event-based NNT, has been criticised since it is not based on patients but on person-time, making the interpretation inappropriate.12 Indeed, in an example devised by Aaron and Fergusson, this approach based on rates can produce illogical results such as 0.5 subjects need to be treated for 1 year to prevent one exacerbation. Clearly, treating a half of a patient appears nonsensical. Of course, the rates can be modified and the NNT interpreted as ‘1 subject needs to be treated for 1 year to prevent 2 exacerbations’, but then we are deviating from the meaning of the NNT, which relates to preventing one patient with the event.
Proper use of the event-based NNT
Because the event-based NNT calculation is based on the IR and not the CI, its use and interpretation must be consistent with the definition of a ‘rate’. It is fundamental to understand that the rate is an ‘instantaneous’ measure of the incidence of an event. Thus, even though the Anzueto trial described above reported IRs of COPD exacerbation of 1.10 and 1.59 per patient-year, its proper usage should be instantaneous. To accomplish this, one should use the smallest measurable time unit in such studies, such as the day, as the instantaneous measure of time. The fact that a patient will not have two exacerbations on the same day makes the day a sufficiently small time unit. Thus, the more proper presentation of the rates of 1.10 and 1.59 per patient-year should be 0.0030 and 0.0044 exacerbations per patient-day respectively. Of course, in studies with much shorter follow-up, such as studies of inpatients, a smaller time unit such as the hour can be used depending on the outcome.
By using the instantaneous version of the rate with the smallest measurable time unit, it is then appropriate to use this alternative formula for the NNT. However, this NNT can only be interpretable over the instant-defining time interval. For example, the TORCH study reported IRs of COPD exacerbation of 1.13 and 0.85 per patient-year in the placebo and fluticasone–salmeterol combination group respectively. The NNT was computed as 1/(1.13–0.85)=3.6 rounded to 4 and inaccurately interpreted as ‘NNT of four to prevent one exacerbation in 1 year’.1 Instead, the corresponding instantaneous rates are 0.0031 and 0.0023 per patient-day corresponding to an NNT of 1250 patients treated with fluticasone–salmeterol on a given day to prevent one exacerbation on that day. Table 1 shows the proper instantaneous event-based NNT for the three trials of fluticasone–salmeterol that computed the NNT using the yearly event-based NNT.
It is important to note that the instantaneous event-based NNT cannot be converted directly to a longer term NNT. Thus, for example, it is incorrect to take the NNT of 1250 patients treated with fluticasone–salmeterol on a given day to prevent one exacerbation from the TORCH trial, divide it by 30 and claim that the resulting value of 42 corresponds to the NNT for 1 continuous month of treatment to prevent one exacerbation. The next section provides formulae that permit the conversion to different time periods.
Converting event-based to patient-based NNT
As noted previously, the fundamental flaw with the event-based NNT as it is commonly used in many studies is that it is not based on the instantaneous rates, but rather on rates that have been extrapolated to a longer time span. It is simply incorrect to convert directly the event-based NNT to a patient-based NNT. However, by first converting the instantaneous rate into a CI function over time, it becomes possible to then compute the corresponding patient-based NNT. To do this, an approximation based on the relation between the Poisson and exponential distributions can be used with the following formula: where CI is the cumulative incidence of the outcome event up to time t and IR is the incidence rate of outcome events measured in the same time units as t.14 Note that the CI and IR are practically equal when the IR is rare over the desired time span (IR less than 0.10 or even 0.15 per patient), making this conversion unnecessary. However, with more common outcomes such as COPD exacerbations (eg, IR=1.13 per patient per year in the TORCH study's placebo group), this formula becomes essential in the absence of the preferred Kaplan–Meier CI. It is also important to note that this formula is only approximate as it depends on the strong assumption that the events follow a Poisson distribution, which is not often the case. In fact, we know that this is clearly not so for COPD exacerbations as they tend to cluster in time and become more frequent.18–20 In this case, the formula will provide somewhat biased estimates of the NNT, which can only be properly computed from the Kaplan–Meier CI estimates.
In the example of the Anzueto trial, the authors used the rates of exacerbations (per patient per year) of 1.10 and 1.59 for the fluticasone–salmeterol and salmeterol groups respectively, leading to a reported ‘NNT’ of two as noted above. Instead, the conversion of the daily instantaneous rates of 0.0030 and 0.0044 per patient into CIs would give corresponding 1-year CIs of 0.67 and 0.80, leading to a NNT of eight patients who need to be treated for 1 year to prevent one exacerbation. In fact, the paper actually provided Kaplan–Meir curves for the time to the first exacerbation, with 1-year CI estimates of 0.60 and 0.67 for the fluticasone–salmeterol and salmeterol groups respectively, corresponding to a NNT of 14 patients needing to be treated for 1 year to prevent one exacerbation. This study illustrates well the inappropriateness of the event-based NNT (2 instead of 14) and to a lesser extent the bias from the deviation from the Poisson assumption needed to convert the rate to the CI, giving an NNT of 8 instead of 14.
COPD exacerbations versus pneumonias
Table 2 displays the properly computed or approximated NNTs for five trials assessing the fluticasone–salmeterol combination against a long-acting bronchodilator. The NNT to prevent one COPD exacerbation and to induce one pneumonia case are presented side by side. While some studies suggest that around twice as many patients need to be treated to induce a pneumonia compared with preventing a COPD exacerbation, the longer term TORCH and INSPIRE trials suggest an opposite benefit–risk impact.
The NNT is a simple measure of the effect of a drug or treatment that is often incorporated in reporting study results. Its calculation, however, is not straightforward when studies involve recurrent outcomes with multiple events. Several drug trials in COPD have miscalculated the NNT to prevent one exacerbation. For example, a 1-year trial of the fluticasone–salmeterol combination versus salmeterol alone calculated the NNT as 2, when the proper calculation resulted in a NNT of 14 patients who need to be treated for 1 year to prevent one COPD exacerbation. Such miscalculations can have a major influence on population calculations or economic impact studies: an incorrectly calculated NNT of 2 patients instead of 14 would imply the treatment of 20 000 instead of 140 000 patients to prevent 10 000 exacerbations in any such study, which can seriously distort any population-based cost analysis.
We showed that event-based NNTs are inherently flawed and should simply never be used, except when based on the smallest time unit, such as the day rather than the year, which in itself is hardly ever relevant. While a formula was provided to convert event-based to patient-based NNT, it is only approximate. In essence, the Kaplan–Meier CI curve is the most accurate technique to estimate the NNT in any trial with variable follow-up times.
An important application that we presented for the NNT is in assessing the relative importance of the benefit and the risk of a drug treatment. We showed the NNT provides a simple tool to compare the benefit of the fluticasone–salmeterol combination in terms of preventing COPD exacerbations versus its risk in inducing pneumonias. The NNT avoids misleading conclusions based on the frequency of these events, such as in the 2-year INSPIRE trial, which compared the fluticasone–salmeterol combination with tiotropium, and found that the frequency of pneumonias (87 events) was much lower than the 2255 COPD exacerbations, suggesting a less important risk than benefit for fluticasone propionate. Instead, the NNT provides a more accurate and objective contrast of this benefit–risk assessment, with a quite different conclusion.
In all, the NNT is a useful measure of the effect of drugs, but its proper calculation is essential to avoid misleading clinical and public health decisions.
Competing interests The author has received research grants and/or participated in advisory meetings or as conference speaker for AstraZeneca, Boehringer-Ingelheim, Forest, GlaxoSmithKline, Merck, Novartis, Nycomed and Pfizer.
Provenance and peer review Not commissioned; externally peer reviewed.