Concerns about the safety of long-acting antimuscarinic agents for the treatment of COPD,1 ,2 particularly the use of tiotropium through the Respimat device, led to the TIOSPIR study.3 This large well-conducted randomised study compared the use of tiotropium through the Handihaler (18 µg) with tiotropium at two doses (2.5 µg and 5.0 µg) through the Respimat in over 17 000 patients with COPD. It showed no difference in mortality or efficacy between the two delivery systems, nor even a trend for a mortality difference. It is now worth reviewing the data that led to these concerns and the lessons that may be learned.

Two systematic reviews played a key role in raising concerns over tiotropium. Antimuscarinic agents were generally considered safe and well tolerated until a meta-analysis of trials that included short-acting and long-acting antimuscarinic agents suggested an increase in cardiac events and mortality.1 This led to fears about safety of the whole class, although the analysis was criticised because it combined short-acting and long-acting antimuscarinic agents. Concerns about tiotropium via the Handihaler were addressed by the UPLIFT study4 a large randomised double-blind placebo-controlled trial which compared the addition of tiotropium or placebo to routine treatment for COPD over a 4-year period. Although the primary outcome was rate of decline of lung function, data on deaths and cardiac side effects were collected prospectively. There was a numerical reduction in mortality in the tiotropium-treated group, which in the prespecified statistical evaluation did not reach statistical significance. There was also evidence of a reduction in reported cardiac events in the actively treated group. This led to the view that tiotropium through the Handihaler did not cause an excess of cardiac adverse events. The results were reviewed by the Regulatory Agencies who were in agreement with this.5

Following this, a meta-analysis of studies using the Respimat device, which provides a soft mist through which tiotropium is delivered at a lower dose, suggested a 52% increase in mortality.2 More recently, this concern was heightened by a database study which suggested that patients with COPD treated with the Respimat had a 27% greater mortality rate compared with those treated with the Handihaler.6 So concerned were a number of experts about these reports, that they suggested that tiotropium via the Respimat should not be prescribed in the treatment of COPD.7

This sequence of events raises two key questions:

1. Why do the findings of TIOSPIR conflict with the Respimat meta-analysis?

When meta-analysis and systematic reviews were first introduced, the rationale was that pooling data from different studies in a formal analysis would give a more reliable estimate of the treatment effect size than selectively quoting individual studies.8 A frequently quoted example is the use of cumulative meta-analysis of β blockers in secondary prevention of myocardial infarction, which showed that by 1981 the effect size estimated in the meta-analysis was the same as in the large clinical trials performed after that date.8 ,9 The important point to recognise here is that the original purpose of these meta-analyses was to combine data from as many small and underpowered trials as possible to test whether the treatment worked. Over time, use of this methodology has evolved and it is now often used to investigate the effect size in less frequently occurring outcomes that were not the primary outcome variable of the original trials and may have been collected less systematically than the primary outcome. This is particularly important when discussing mortality, since these data are usually treated as adverse events not as outcomes; mortality data may not be collected rigorously, particularly in patients who withdraw. By way of contrast, the authors of the UPLIFT study4 went to great lengths to ascertain the survival status of every patient randomised. This difference in the conduct of the trials is usually ignored and the same strength of conclusion is applied to meta-analyses of secondary outcomes and adverse event data as to conclusions about the average effect of the primary outcome. It is important when meta-analyses are designed or evaluated that the quality of the data capture for each outcome is considered as part of the interpretation.

• 2. Why do the findings of TIOSPIR conflict with the database analysis?

Database studies were originally used to check prescribing patterns of medicines in clinical practice and assess whether effects seen in trials were reproduced in routine use.10 ,11 With improvements in electronic medical records and accessibility to researchers, database studies have evolved and, are now also being used as stand-alone evidence of clinical effectiveness (or harm) using a more generalisable group of patients than those enrolled in traditional trials. The Integrated Primary Care Information Study of the Respimat6 demonstrates some of the challenges of this type of study. It compared new and established users of the Handihaler with new Respimat users, but it collected data from the time when the recently introduced Respimat was licensed and compared it with outcomes in patients who used the Handihaler, which had been in routine clinical practice for many years. This difference is important, because it has been shown that adverse cardiac events fall from the time of first prescription of long-acting bronchodilators, including tiotropium.12 To avoid this potential bias, this study should only have included new users of Respimat and Handihaler and excluded established users of the latter. Designing studies of new medicines compared with an established treatment is also challenging, because the new medicine may be used differently from those established in routine practice. It may be prescribed to more severe patients in whom other treatments have failed and by a group of doctors called ‘early adopters’ who are recognised to begin treatments soon after registration making the identification of suitably matched patients with similar baseline risks difficult.

Examination of the characteristics of the patients treated with Respimat and Handihaler in Primary Care Information Study of the Respimat6 shows that those treated with the Respimat had more severe COPD and higher level of cardiovascular comorbidity prior to initiating treatment. The authors used multiple techniques to evaluate the increased risk observed with Respimat, including stratification by incident use or presence of cardiovascular comorbidity and then adjust the results using propensity scores. In the case where the analysis suggests potential confounding by severity or channelling of the new treatment to a sicker patient population, careful attention to the results of new users and stratified analyses is warranted. A new user design should minimise survival or selection bias and address the situation where those who perceived a benefit continued to take the treatment, resulting in prevalent users with a higher likelihood for a positive outcome. The analysis in Primary Care Information Study of the Respimat that was restricted to patients with baseline cardiovascular comorbidity showed a higher risk of mortality (HR 1.36, 95% CI 1.07 to1.73), however it is difficult to interpret the independent effect of the Respimat and the extent of confounding by severity without a detailed assessment of the underlying COPD risk within this diverse subgroup and the unmeasured reason for the physician's choice of Respimat versus Handihaler. Propensity scores, calculated as the probability of receiving one treatment versus another, theoretically do adjust for confounding, however strong predictors of mortality such as FEV₁ and health status were not part of the database. As a result, it is unclear how much adjustment was offered by the propensity score and how much residual confounding remained in the final estimates. The incident user propensity score model showed a similar magnitude of risk to the full cohort, but wider CI (HR 1.29, 95% CI 0.90 to 1.84).

Although we believe that the TIOSPIR study provides more robust evidence than either the meta-analysis or the database study for this specific question, we acknowledge the criticisms generally made of controlled trials. One is that they recruit patients who are not the same as those treated in routine clinical practice, however, the characteristics of the patients recruited to the TIOSPIR study were almost identical to those in the studies that contributed to the meta-analysis. In contrast, database studies include patients who would otherwise be excluded from trials, on grounds of safety, and can offer much larger sample sizes to evaluate risk of rare events. In the case of TIOSPIR some important exclusion criteria were: recent serious heart disease and moderate-severe renal impairment as determined by plasma creatine, the latter because of the renal excretion of tiotropium. Impaired, glomerular filtration may be present in 20% of patients with COPD with a normal plasma creatine,13 so it is likely that TIOSPIR did recruit some patients with hidden renal impairment, although it is likely to have under-represented the target patient population with comorbid severe active or unstable CV disease, who are prescribed the treatment in clinical practice.

Overall we think that the lessons to be learned from the tiotropium Respimat experience are first: that authors and readers of meta-analyses should be cautious about interpreting data outside of the primary outcome measure or carefully collected secondary outcome measures. Second, conclusions should be drawn cautiously from database studies in the immediate postmarketing period in which confounding by severity may not be fully adjusted and in which there may be variability in prescribing patterns between new and established therapies. When performed, meta-analysis and database studies are useful to increase precision of estimates and generate hypotheses about benefit: risk ratio, especially by evaluating rare events; however the results should be interpreted with the totality of the clinical and preclinical evidence. When there is equipoise, inconclusive evidence from meta-analyses and/or observational studies should be tested by randomised trials in which the effect of those confounding biases may be minimised. There is hope that we are not doomed to repeat history, rather we increasingly have opportunities to design pragmatic trials using electronic medical records data14 to increase generalisability with broader patient populations and improve capture of comorbidities and coprescribing while maintaining the rigour of randomised treatment assignment.