Article Text
PDF
Statistics from Altmetric.com
The review of placebo controlled trials of sodium cromoglycate (SCG) in childhood asthma by Tasche et al 1 concludes that “there is insufficient evidence that SCG has a beneficial effect as maintenance treatment in children with asthma”.
We do not believe this conclusion is justified. The principal criteria for assessing the efficacy of inhaled SCG are the mean differences in the effect on the symptoms of cough and wheeze between SCG and placebo across all trials included in the review. The 95% confidence intervals for these differences are 0.12 to 0.27 for cough and 0.11 to 0.26 for wheeze. As the differences are in favour of SCG and the confidence intervals do not include zero, the only conclusion that can be drawn is that there is strong statistical evidence for a beneficial effect of SCG in children with asthma—the reverse conclusion to that presented.
The review identified 24 randomised controlled trials and is claimed to include “all published randomised, placebo controlled trials”. Two published trials are not included (those by Bermanet al 2 and Mikawaet al 3) and the trial by Silverman et al,4 which is included, does not meet the author's criteria. Both excluded trials, which involved 472 children, reported statistically significant differences in favour of SCG compared with placebo. The study by Silverman et al compared the combination of inhaled SCG and isoprenaline with isoprenaline alone and did not include a placebo comparison.
The authors have calculated a tolerance distribution for study effects and we believe it is this distribution which has been fundamentally misinterpreted. They provide a tolerance interval for the results of future studies based on the between study variation seen. The fact that this interval contains zero does not mean that there is no evidence of a difference between treatments, as the authors imply, but that not all future studies can be predicted to give a difference numerically in favour of SCG. The tolerance interval quoted suggests that perhaps 90% of future trials will have an outcome in favour of SCG, a really positive finding. This is, of course, entirely consistent with the data from studies used in the review which indicates that, although some studies are directionally in favour of placebo, the vast majority are in favour of SCG. The fact that a few of the studies are in favour of placebo is not surprising since the authors reject the null hypothesis of homogeneity.
In the comment section of the paper the authors state that older studies were more likely to produce a positive effect of SCG treatment.
Inhaled SCG was originally developed as a capsule containing 20 mg of dry powder of SCG to be inhaled four times a day using a special inhalation device, the Spinhaler, and its efficacy was established in trials involving children aged 5–17 years. Twelve trials included in this review use this dose and inhaler system, of which 11 are described as positive and one as positive/equal. All were published between 1968 and 1980. Examination of the mean differences for severity scores shows that, in all 12 trials, the 95% confidence intervals do not include zero and inhaled SCG is therefore statistically (p<0.05) significantly better than placebo.
The remaining 12 studies were published between 1980 and 1997. All used different inhalation systems, dosage schedules and, in some cases, a different diagnosis from the earlier trials. In nine trials the drug was delivered as a nebulised solution at a dosing schedule of 20 mg 3–4 times daily, two used a 5 mg metered dose inhaler (MDI), and one used a 1 mg MDI. The age range of the children in these trials was from 0 to 6 years, with many under 1 year of age. Of the nine trials which used a nebuliser solution, four were in children with asthma, two in children with wheezy bronchitis, and one each in children with persistent wheezing, recurrent wheezing, and preterm babies with respiratory symptoms. Four were classed as positive (0–6 years), one as equal/positive (0–2 years), and four as equal (0–4 years).
The MDI formulations were introduced 10–15 years after the original capsule formulation. The one trial using the 1 mg MDI conducted in children aged 4–13 years was classed as positive/equal. Of the two trials which used the 5 mg MDI, one was conducted in patients for whom the drug is not indicated—recurrent respiratory symptoms in preterm babies at a dose of 5 mg three times daily using a coffee cup as a spacer— and the other was conducted by the authors of this paper in a general practice setting in children aged 1–4 years and it has already been pointed out that the delivery system used may not have provided an adequate dose.5 ,6 They also used less than the recommended dose (three times a day rather than four) and an incorrect dosing method (two puffs into the spacer for one inhalation). In view of the major flaws in this trial it is difficult to understand how it scored highest in the methodological assessment of the trials. One can only conclude that the outcome of this one trial has influenced the conclusion drawn by the authors and they have chosen to ignore the 16 positive and the three positive/equal trials of the 24 they reviewed.
This more detailed examination of the sequencing of the trials does show that the main trials that clearly demonstrate the efficacy of the drug in childhood asthma were conducted in children over the age of 5 years with the drug delivered as a dry powder using the Spinhaler at a dose of 20 mg four times a day. They were published before 1980. The trials published after 1980 were conducted mainly in children aged less than 5 years using either nebulised solution or metered dose aerosols. As the authors rightly state, “diagnosis and measurement of asthma in young children is difficult”. It is therefore not surprising that in these later trials it has proved more difficult to demonstrate consistently the efficacy of SCG.
It appears that the authors' conclusions of lack of beneficial effect are based mainly on their own trial and trials conducted in younger age groups using nebuliser solution. While it is possible that the drug is less effective in the younger age groups, it is more likely that the apparent lesser effect in these children is related to the difficulties of diagnosis, assessment, and drug delivery.
The authors identified 251 articles in their Medline search but only 18 met their inclusion criteria. Many of the excluded articles are supportive of the use of SCG in childhood asthma. They also refer to a number of other review articles but ignore the fact that these were all positive towards the drug, preferring their own conclusions.
Finally, the authors further conclude that they doubt whether the effect seen is of clinical relevance but also point out that mean symptom scores in childhood asthma studies are generally low due to dilution by symptom-free days. In these circumstances the estimated size of effect may well have clinical relevance.
The analysis as presented provides very strong evidence of statistically significant benefits in favour of SCG for the key symptoms of cough and wheeze; this would be strengthened further by the inclusion of all published studies and further still if only those studies which identified a clear asthmatic population were included. Despite the fact that we consider this review to have major flaws, we believe that the results have been misinterpreted, that it does show clearly and significantly that inhaled SCG is of benefit in childhood asthma, and it cannot be used to draw the conclusion of the authors.
authors' reply We thank Dr Edwards and colleagues for their reaction and for the opportunity they provide to further clarify our findings.
Our conclusion was based on three elements—the strong indication of publication bias, the tolerance interval including zero, and the small treatment effect. These three elements should be considered in combination as it is this combination of findings that has led us to conclude that there is insufficient evidence for a beneficial effect of maintenance treatment with SCG in children with asthma. We think that Dr Edwards and colleagues have misunderstood this conclusion as they only address the issue of the tolerance interval. We agree that, from the interval as we found it, one would predict some 90% of future studies to have a positive outcome (though not all significant). However, because of the strong suggestion of publication bias, with negative studies being underrepresented, the overall small effect size and the corresponding confidence intervals are probably biased—that is, overestimating the true effect size.
The pooled point estimates were found to be 0.19 both for cough and wheeze. This means that, compared with the placebo group, the children who used SCG had, on average, a 0.19 lower symptom score on a scale of 0–3 during the period they were studied. These are small effects indeed, of which the clinical relevance is questionable, especially as the bias caused by selective publication has probably increased the effect size. The true effect size will therefore be even smaller and hence irrelevant.
We excluded the studies by Berman et al 1-1 and Mikawa et al 1-2 because they did not fulfil our criteria—the first because the upper age limits are unclear and probably included patients above 18 years of age, and the second because it was not published in English. We do not think that the fact that they are key studies for registration is a valid argument to vary our predefined inclusion and exclusion criteria.
We agree with the authors that the study by Silvermanet al 1-3 should have been excluded because it did not include a placebo comparison. However, exclusion of this study does not affect the point estimates nor the confidence intervals.
We appreciate the help of the authors in explaining the differences and time trends. As we described in our paper, we performed a multivariate regression analysis with many study characteristics in the model, including method of administration of the medication, year of publication of the study, and age of the children. Only the year of publication of the study turned out to be a significant predictor of effect size in this analysis.
We will not react here to the “major flaws” that the authors have found in our own trial1-4 as we have dealt with these points eleswhere.1-5
The findings in young children are not unequivocal. Some studies found a positive effect1-6 1-7 while some found no effect.1-5 1-8 1-9
The argument that many other reviews published in the past were positive is not convincing as none of these reviews was systematic, none assessed the methodological quality of the studies, and none pooled outcome measures. Common pitfalls of narrative reviews are that they tend to repeat the success stories of medical history but ignore negative studies, that they are authority based instead of evidence based, and authors of this type of review may have strong relations with pharmaceutical companies.1-10-1-12 Our aim was not to give our personal opinion but to assess the benefits of SCG in asthmatic children in a systematic and unbiased way, without conflicts of interest.
In our review we concluded that there is insufficient evidence that SCG has a beneficial effect as maintenance treatment in asthmatic children. Our conclusion was based on three elements—the strong indication of publication bias, the tolerance interval including zero, and the small treatment effect. We believe that the arguments provided by Dr Edwards and colleagues are insufficient to modify this conclusion.