With pressure mounting from legislators, insurers and consumer advocates to drive ventilator-associated pneumonia (VAP) rates to zero, the article by Conway Morris and colleagues (see page 516) in this issue of Thorax is a timely reminder that VAP rates are uncertain estimates rather than concrete measures of patient morbidity.1 Conway Morris and colleagues show that the reported VAP rate of an Intensive Care Unit (ICU) is heavily dependent upon its favoured diagnostic technique. ICUs that exclusively use bronchoalveolar lavage (BAL) to diagnose VAP are liable to report VAP rates that are 76% lower than those that exclusively use endotracheal aspirates. This laxity in the VAP definition confers a risk that some well-intended initiatives may decrease VAP rates yet provide little benefit to patients and perhaps even put some at risk.

Conway Morris and colleagues base their estimate of varying VAP rates upon the comparative yield of simultaneous BAL and endotracheal aspirate cultures taken from 53 critically ill patients with clinical syndromes suggestive of pneumonia. All patients had radiographic infiltrates, pyrexia or an abnormal leucocyte count, and clinical signs of chest infection such as increased volume or purulence of sputum, crepitations and deterioration in oxygenation. In these patients, there was a clear step function in culture results: 89% of endotracheal aspirate qualitative cultures were positive (defined as any growth), 51% of endotracheal quantitative cultures were positive (defined as ⩾10⁶ colony-forming units (CFU)/ml), and 21% of BAL cultures were positive (defined as ⩾10⁴ CFU/ml).

Each of these culture thresholds qualifies as VAP using the definitions of the Hospitals in Europe Link for Infection Control through Surveillance (HELICS) system. Consequently, depending upon one’s choice of diagnostic technique, measured VAP rates can vary fourfold. HELICS does assign different VAP codes to patients depending upon which culture technique and threshold was used to establish the case, but these distinctions are seldom if ever cited when hospitals report their VAP rates. For the purposes of most public or government reporting, a VAP is a VAP regardless of diagnostic method.

Conway Morris et al modelled the impact of diagnostic technique on VAP rates and antibiotic usage by applying their data on differential culture yields to their medical–surgical ICU’s surveillance and dispensing data from the previous year. They estimated that exclusive use of endotracheal aspirates would have yielded a VAP rate of 25.5 cases per 1000 ventilator days whereas exclusive use of BAL would have yielded 6.1 cases per 1000 ventilator days. They further estimated that exclusive use of BAL would have led to a 30% reduction in antibiotic use since fewer patients would have had positive cultures.

Conway Morris and colleagues used these findings to catalyse a practice improvement initiative aimed at increasing the use of BAL in their ICU. They managed to increase the rate of bronchoscopy in patients with suspected VAP from 37% in the year prior to the intervention to 58% in the year following intervention. As predicted, their observed incidence of VAP dropped by 50% (p = 0.001) while antibiotic usage declined by 21% (p = 0.08). There was no apparent deleterious impact on patients caused by the change in practice. In fact, mortality decreased in the postintervention period, although the before–after design of this study and a significant difference in baseline APACHE (Acute Physiology and Chronic Health Evaluation) scores between the preintervention and postintervention populations preclude confidence in this finding.

The demonstration by Conway Morris et al that exclusive use of BAL might lower a hospital’s VAP rate by up to three-quarters will be very enticing to administrators striving to project their hospital in the best light possible. The study begs the question, however, of what constitutes truth in VAP surveillance. Is BAL-based surveillance a more accurate reflection of patients’ morbidity or is it merely gaming the VAP definition to make one’s VAP rates look as low as possible without meaningfully improving patient care?

In truth, there is no universally accepted reference standard for diagnosis of VAP. Conway Morris and colleagues implicitly endorse BAL by using it as the comparison standard to evaluate endotracheal aspirates. In addition, they used their findings to marshal a practice improvement initiative dedicated to increasing the use of BAL. There is no reason, however, to believe that BAL diagnosis is any closer to the truth than endotracheal aspirates. Studies assessing the sensitivity and specificity of BAL versus endotracheal aspirates relative to lung histology show little consistent difference. The reported sensitivity of BAL ranges from 11% to 77% while specificity ranges from 42% to 94%.2^–6 Likewise, the sensitivity of endotracheal aspirate is 56–69% while the specificity is 75–95%.3^–5 As such, both techniques miss many cases of VAP and inappropriately label some people as having the disease.

BAL’s lack of discriminating power is borne out in Conway Morris and colleagues’ own data. They found that patients suspected of having VAP had identical mortality rates regardless of whether their BAL culture was positive or negative. Likewise, their data on antibiotic usage indicates that clinicians take BAL results with a pinch of salt: they found that clinicians stopped antibiotics within 1.5 days following a negative endotrachael aspirate but continued treating patients for a mean of 5.8 days following a negative BAL culture.

One might be tempted to recommend that VAP benchmarking proceed albeit with a modified definition that explicitly stipulates quantitative BAL as the only diagnostic standard. This is unrealistic and dangerous. It is unrealistic because some patients are not candidates for bronchoscopy (typically those that require high levels of ventilator support). It is dangerous because bronchoscopy is an invasive procedure with small but real risks including pneumothorax, bleeding and oxygen desaturation.

A safer alternative might be to establish a weighting system to adjust a hospital’s VAP rate according to the diagnostic method used. VAPs diagnosed using endotracheal aspirate cultures, which tend to be positive more often than BAL, should count less towards an institution’s overall VAP rate than VAPs diagnosed using bronchoscopy. This method would be fairer than the current system, but is still flawed to the extent that it fails to take into account the interobserver variability and lack of specificity in the rest of the VAP definition.7 Patients are selected for microbiological sampling on the basis of clinical signs such as fever, abnormal leucocyte count, purulent sputum and radiographic infiltrates. These signs are notoriously non-specific and subject to substantial differences in interpretation between observers.8 Limiting the diagnosis of VAP to patients with positive BAL cultures or adjusting VAP rates according to sampling method will not obviate variability in selecting patients for investigation in the first place.

Conway Morris and colleagues argue that increased uptake of BAL is better care because it will lower antibiotic usage, which in turn will lower costs, limit antibiotic adverse effects and decrease selection for antibiotic-resistant organisms. In fairness to them, their study was begun at a time when best available evidence did suggest that preferential use of BAL could not only lower antibiotic usage but also decrease mortality.9 More recent studies, however, have not confirmed this finding.10 A large trial assessing clinical outcomes amongst patients with suspected VAP who were randomised to BAL with quantitative cultures versus endotracheal aspirates with non-quantitative cultures found no difference in patient mortality, length of stay in the ICU or hospital, or antibiotic usage.11 The trend towards lower antibiotic use and decreased mortality in the study of Conway Morris et al does not trump this randomised controlled trial given the before–after study design and lower APACHE scores in the postintervention group.

It is an irony of the quality improvement movement that the drive for zero hospital-acquired infections might occasionally do patients more harm than good. In the case of VAP, the looseness of the surveillance definition makes this a particular concern. Hospitals might be tempted to encourage an expensive and invasive diagnostic technique that has been largely discredited for routine practice by clinical trial data. This brings out a disturbing tension between hospital surveillance and clinical care in the case of VAP. The purpose of both surveillance and clinical care is to improve outcomes for patients. Subjectivity and inaccuracy in the VAP definition, however, allow room for hospitals to undertake practices that will markedly decrease their VAP rates and yet do little or nothing to improve patient outcomes.12

The implications are as follows: (1) VAP as currently defined is not a suitable measure for benchmarking hospitals or for external assessment of quality of care since the diagnosis is too inexact and liable to be gamed; (2) internal quality improvement initiatives to decrease VAP rates need to pay close attention to the diagnostic protocol to ensure consistency over time; and (3) we need a new, more objective and accurate measure for quality of care in ventilated patients that will reliably predict patients’ outcomes. Until then, quality improvement initiatives would do well to track directly patient outcomes, such as duration of ventilation, length of stay and mortality, to increase confidence that an observed change in VAP rates translates into improved patient outcomes. This is critical to avoid unintended consequences from well-meaning initiatives to drive our hospital-acquired infection rates to zero.