Statistics from Altmetric.com
McCaffery et al 1conclude that “antistaphylococcal treatment achieves sputum clearance of Staphylococcus aureus in patients with cystic fibrosis . . .” and that prophylactic treatment in young children is “. . . likely to be of clinical benefit”. These positive conclusions are based on the results of a study which has important methodological problems. Neither the introduction nor the methods section of this review state what hypotheses the review set out to test, the criteria used to decide whether a study was suitable for inclusion, outcomes to be studied in the review, or methods used to assess the methodological quality of included studies. Systematic reviews differ from narrative reviews in that they test hypotheses using a methodology which is well described.2 The authors have described their search strategy, which is based on that developed by the Cochrane Collaboration, to identify randomised controlled trials. The authors have, however, included a number of studies in their review which are not randomised controlled trials. It is not clear from the information provided whether their search strategy is sensitive enough to identify all possible relevant studies.
The authors base their conclusions on the results of just two randomised controlled trials, involving only 66 children, with a maximum follow up of two years.3 4 All of these children were under seven years of age (most under two years) and had upper respiratory samples taken, not sputum. Of the other studies described as randomised, one used alternate allocation (and so was not randomised)5 and one reported further outcomes in patients included in one of the randomised controlled trials.6 Only two randomised controlled trials actually reported the prevalence ofS aureus in respiratory secretions. The larger study by Weaver et al 3reported that the prevalence of S aureus was reduced with prophylaxis but “clearance” was not achieved from nose and throat swabs.
The important issues for cystic fibrosis patients and their families are not eradication of an organism but fewer symptoms, improved lung function, and prolonged survival. None of the studies described in the review addressed these issues nor, indeed, the concern that prophylactic antibiotics may encourage chronic pulmonary infection with organisms such as Pseudomonas aeruginosa or multiply resistant S aureus. A Cochrane review addressing this intervention is currently being undertaken by two of us (AS and SW) and this will have the advantage that it will be regularly updated to incorporate new studies. Only when further properly designed studies have been completed and included in an up to date systematic review will we be able to evaluate whether antistaphylococcal antibiotic prophylaxis is “. . . likely to be of clinical benefit”.
authors' reply Smyth et al have listed a number of features that they regard as being essential to systematic reviews. In particular, they suggest that systematic reviews should always test hypotheses. However, a clear understanding of the existing evidence is necessary for the generation of valid hypotheses and, in our view, this is best achieved by systematic reviewing. Indeed, many important systematic reviews published in major clinical journals do not specifically test hypotheses, but study the current evidence in order to identify the state of existing knowledge and to define areas for further research.1-1 1-2 This objective is consistent with the view of the authors of the Cochrane Collaboration Handbook who recognise that systematic reviews can have different motivations, one of which is the resolution of conflicting evidence.1-3 Indeed, it is probably difficult to define systematic reviews as formally as Smythet al (and others) have proposed as the science of systematic reviewing is undergoing continuous development. More systematic reviews are being performed now than ever before (a Medline search looking for “systematic review” in titles and abstracts presents 4158 citations in the last 10 years, 1538 (37%) of which are in the last two), with reviewers defining their methods according to the problem in question.
Again, because of the nature of the field being studied, we had purposely not defined stricter criteria for study selection or drawn up a preselected list of outcomes of interest. As the area under investigation was largely unknown, we felt such criteria could limit our search. Also, in the absence of any significant background information, we were uncertain if such a choice of outcomes could be made objectively. Indeed, if we had arbitrarily drawn up a list of outcomes that were of interest to us, we would have missed a number of outcomes that others had used and which could be of potential interest to readers when designing clinical trials in the future. We did not use quality scores because there is little objective evidence to support the use of quality scoring in systematic reviews.1-4 Many of the scoring systems have not been developed with sufficient rigour1-4 and could add the analyst's bias to the results.1-5 A recent review of a random sample of 240 meta-analyses showed that less than half assessed trial quality.1-6 However, we note that newer techniques such as meta-regression may provide better alternatives in the future.
As we were principally interested in randomised controlled trials (RCTs), we used a search strategy that has been well validated for the recall of such trials. However, as before,1-7 we wanted to present an analysis of outcomes of both RCTs and non-RCTs because we felt this would make our conclusions more objective. Again, this approach is supported by the authors of the Cochrane Collaboration Handbook.1-3 Smyth et al state quite rightly that the important issues for cystic fibrosis patients and their families are not eradication of an organism but fewer symptoms, improved lung function, and prolonged survival. However, this should not inhibit the use of laboratory based outcomes which could influence clinical decision making until appropriate clinical data are available. Indeed, given the high predictive value of oropharyngeal cultures in children for identifying pathogens in bronchoalveolar lavage fluid (sensitivity and specificity of 90%),1-8 we feel the evidence we have defined in support of clearance ofStaphylococcus aureus from the upper or lower respiratory tract with anti-staphylococcal antibiotics does suggest that this therapeutic intervention is likely to be of clinical benefit, although we strongly support their argument that properly designed studies are needed to confirm this hypothesis.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.