Original Article
Number needed to treat is incorrect without proper time-related considerations

https://doi.org/10.1016/j.jclinepi.2011.04.009Get rights and content

Abstract

The number needed to treat (NNT) is a simple measure of a treatment's impact, increasingly reported in randomized trials and observational studies. Its calculation in studies involving varying follow-up times or recurrent outcomes has been criticized. We discuss the NNT in these contexts, illustrating using several published studies. The computation of the NNT is founded on the cumulative incidence of the outcome. Instead, several published studies use simple proportions that do not account for varying follow-up times, or use incidence rates per person-time. We show that these approaches can lead to erroneous values of the NNT and misleading interpretations. For example, after converting the incidence rate to a cumulative incidence, we show that a trial reporting a NNT of 4 “to prevent one exacerbation in 1 year” should have reported a NNT of 9. A survey of all papers reporting NNT, published in four major medical journals in 2009, found that 6 out of all 10 papers involving varying follow-up times did not correctly estimate the NNT. As the “number needed to treat” becomes increasingly used in complex studies and in the comparative effectiveness of therapies, its accurate estimation and interpretation become crucial to avoid erroneous clinical and public health decisions.

Introduction

What is new?

  • The correct calculation of the “number needed to treat” (NNT) from randomized trials and observational studies is necessarily based on the cumulative incidence of the outcome over time.

  • Some studies use instead simple proportions of patients that produce incorrect estimates because they do not account for varying follow-up times.

  • Some studies use incidence rates per person-time, with a resulting NNT that is not actually a number of persons needed to treat but rather an ill-defined person-moments needed to treat.

  • Such measures lead to erroneous values of the NNT and ambiguous interpretations.

  • With the NNT increasingly used in comparative effectiveness studies of therapies, its accurate estimation and correct interpretation are crucial.

The number needed to treat (NNT) is now used extensively in randomized trials and observational studies to provide an additional and user-friendly measure of the impact of a drug or treatment on a given disease outcome [1], [2]. Its interpretation is appealing: it represents the number of patients with the disease under study who need to be treated with the drug or intervention to prevent the disease outcome in one patient. Its computation and interpretation have traditionally been based on a single occurrence of a dichotomous outcome assessed using the ideal randomized trial, namely a trial with equal follow-up for all patients. However, such ideal trials are rare. In practice, studies involve unequal follow-up times, and certain trials study recurrent outcomes with multiple events, such as infections and exacerbations. The NNT has nonetheless been applied to these situations. Recently, its application to outcomes with repeated events has been criticized and its potential misuse with varying follow-up times highlighted, raising some questions about its validity and interpretation [3], [4].

To clarify these points, we expand on some issues raised in a recent letter [4], by examining the use of the NNT measure in a number of trials published in four major journals, including the New England Journal of Medicine, the Journal of the American Medical Association, The Lancet, and the British Medical Journal. In the context of these trials, all with varying follow-up times or outcomes with multiple events, we present the proper approaches to estimate, analyze, and interpret the corresponding NNT.

Section snippets

The NNT measure

To better understand the origin of the NNT, recall that the frequency of disease outcome is typically measured as a cumulative incidence of the outcome per number of patients followed over a given time period. This results in a proportion, whose value is written for example as 0.5/100, 2/100, or 5/100, where fixing the denominator at 100 makes the magnitudes of outcome frequencies easily comparable. Alternatively, the motivation behind the NNT measure is to first make the cumulative incidences

Patient-based NNT: varying follow-up

In practice, trials generally result in varying follow-up times between patients in which case the cumulative incidence of an outcome cannot be calculated simply as a proportion of subjects. It must instead use the Kaplan–Meier approach, which accounts for varying follow-up times and provides a curve for the cumulative incidence over time [5]. The NNT can then be directly computed by inverting the difference in the cumulative incidence of the outcome between the two groups at the desired time

Patient-time-based NNT

Rather than using the absolute number of patients with the outcome in computing the NNT in studies with varying follow-up times, as discussed earlier, trials may use the incidence rate of the outcome as a measure of outcome frequency. It is computed as the number of patients with the outcome divided by the total amount of person-time or patient-time, to account for varying follow-up times. In this case, the effect of a treatment is measured by the difference in the incidence rate of the outcome

Event-based NNT: multiple events

In some studies, the outcome involves a recurrent event that can occur more than once during the patient’s follow-up [26], [27]. In such studies, the frequency of disease outcome is also measured as an incidence rate computed as the total number of outcome events divided by the total amount of person-time. Here again, the corresponding NNT has been computed as 1/(IR0  IR1) with the interpretation that it represents the number of patients who need to be treated for a given time period to prevent

From event-based to patient-based NNT

Although the event-based NNT is valid and useful, albeit with a proper interpretation, one may also wish to quantify the corresponding patient-based NNT. To do so, one may of course directly compute the cumulative incidence of the first event. Alternatively, one may use the relation between the Poisson and exponential distributions to relate the incidence rate and the cumulative incidence in the following way:CI=1eIR×twhere CI is the cumulative incidence of the outcome event up to time t and

Recent review: 2009

We reviewed all articles published in 2009 in four major medical journals, namely the New England Journal of Medicine, the Journal of the American Medical Association, The Lancet, and the British Medical Journal that reported the NNT. We used MEDLINE to search the articles (term: “number needed to treat”) and excluded meta-analyses. We identified 19 such articles. We found that most articles (13/19) used this measure properly. However, 9 of the 19 studies involved simple designs with fixed

Conclusion

The NNT is a simple and intuitive measure of the impact of a drug or treatment that is increasingly added to the reporting of study results. However, a clear understanding of what is being counted is crucial in its interpretation. The original NNT version is patient-based and counts the number of patients who need to be treated to prevent the outcome in one patient over a given time period. As most studies result in varying follow-up times, this measure inherently requires that the cumulative

References (27)

  • S. Wiebe et al.

    A randomized, controlled trial of surgery for temporal-lobe epilepsy

    N Engl J Med

    (2001)
  • P.J. Meunier et al.

    The effects of strontium ranelate on the risk of vertebral fracture in women with postmenopausal osteoporosis

    N Engl J Med

    (2004)
  • P.W. Serruys et al.

    Percutaneous coronary intervention versus coronary-artery bypass grafting for severe coronary artery disease

    N Engl J Med

    (2009)
  • Cited by (60)

    • Methodological standards for conducting and reporting meta-analyses: Ensuring the replicability of meta-analyses of pharmacist-led medication review

      2022, Research in Social and Administrative Pharmacy
      Citation Excerpt :

      Despite their usefulness in assisting clinical decisions, NNTs are uninterpretable if incompletely reported, and they may be misleading if calculating methods are inadequate to study designs and variables under evaluation.12 A study evaluating RCTs published in medical journals found that NNTs were miscalculated in 60% of cases.49 Mendes et al. (2017)12 additionally showed that less than one-third of authors report the CI for NNT.

    • Perils of prolonged ovarian suppression and hypoestrogenism in the treatment of breast cancer: Is the risk of treatment worse than the risk of recurrence?

      2021, Molecular and Cellular Endocrinology
      Citation Excerpt :

      It is applicable to comparing two treatment arms within a single study (Mendes et al., 2017). Randomized trials with variable durations of treatment will influence the NNT calculation although the Kaplan-Meier approach can compensate for variation in follow up (Suissa, 2009; Suissa et al., 2012). Confidence intervals should be stated (Suissa, 2009; Mendes et al., 2017; Suissa et al., 2012; Pines et al., 2012).

    • The "1-year-death number needed to treat" for comparing the impact of distinct interventions on patient outcomes

      2019, CMAJ
      Citation Excerpt :

      This dichotomization of a continuous measure (i.e., the likelihood that differences seen exceed that arising by chance) will unfairly exclude some studies (such as those with a p value of 0.05) whose importance is not materially distinct from those presented here. Several potential limitations of the number needed to treat (NNT) will also apply to the 1-year-death NNT.38,39 First, both statistics will tend to increase extensively when baseline risks decrease.38

    View all citing articles on Scopus

    Funding: This research was funded by grants from the Canadian Institutes of Health Research.

    View full text