Community-acquired pneumonia (CAP) is a common reason for emergency adult admissions to hospital, and costs the UK’s National Health Service (NHS) at least £441 million/year.¹ Three well-controlled studies from North America have shown that care pathways can improve processes of care in CAP,^2–4 but only one showed an improvement in outcome³ and none assessed cost effectiveness. A systematic review of 235 studies of guideline implementation reported that only two studies estimated the costs of guideline development, only four studies included comprehensive data on the costs of implementation and no studies calculated cost effectiveness.⁵

A previous audit (n = 205) in our hospital (Ninewells Hospital) showed that less than two thirds (61%) of patients with severe CAP and half (52%) of all patients with CAP received antibiotics within 4 h of admission to hospital.⁶ The 2001 guidelines of the British Thoracic Society (BTS) recommend that all patients with severe CAP receive intravenous antibiotic treatment immediately after diagnosis.⁷ A national confidential inquiry of deaths from CAP in young adults in England and Wales showed that three quarters of patients received appropriate empirical antibiotic treatment and two thirds received antibiotics within 2 h of admission to hospital.⁸ The authors concluded that although many of the deaths were not preventable, appropriate antibiotics should be given as soon as possible after admission.

This conclusion is supported by large observational studies that have shown a marked association between improved clinical outcomes and both the time to administration^9,10 and selection of initial antibiotics.^11,12 In the US, the Medicare Quality Improvement Community now recommends a door-to-drug time of 4 h instead of their previous 8-h threshold, and states that “several thousand deaths could be prevented each year among hospitalised Medicare pneumonia patients if the initial dose of antibiotic was administered within 4 hours of hospital arrival”.¹³ The aim of our study was to evaluate from the hospital’s perspective the clinical- and cost-effectiveness of implementing a management pathway designed to improve the timing and selection of antibiotics for hospitalised patients with CAP.

METHODS

Study design

A controlled before-and-after study was designed to fulfil the quality criteria of the Cochrane Effective Practice and Organisation of Care Review Group checklist.¹⁴ Baseline data were collected from all patients with CAP admitted to an intervention and a control site between 1 November 2001 and 30 April 2002. The intervention was implemented in September and October 2002 and post-intervention data were collected between 1 November 2002 and 30 April 2003.

Intervention and control sites

The intervention was targeted at the accident and emergency (A&E) department and acute medical admissions unit (AMAU) of a 1000-bed teaching hospital. The control site was a 500-bed district general hospital within the same teaching hospital trust. Although junior doctors at both sites were on the same training rotations, we carefully designed the implementation and evaluation to minimise the risk of contamination. At baseline, both sites had comparable patient to staff ratios (table 1) and pathways of care for patients with CAP.

Methods used to understand the problem

A structured survey of 83 junior medical staff was conducted at the intervention site to identify barriers to implementation, supplemented by in-depth interview of eight doctors purposively sampled from the structured survey respondents (summarised in appendix A available online at http://www.thorax.bmjjournals.com/supplemental). These data were used to design the intervention and implementation strategy. Qualitative data were also collected in the post-implementation period to provide additional detail for use in the feedback process.

Intervention and implementation strategies

An experienced multidisciplinary steering committee developed a CAP management pathway (appendix B available online at http://www.thorax.bmjjournals.com/supplemental) based on the 2001 BTS guidelines, the results of baseline data analyses, and the quantitative and qualitative surveys described earlier. The pathway was implemented using a combination of information packs distributed to junior and senior medical staff and relevant senior nursing staff, interactive group educational sessions and reminders in the form of laminated colour posters (electronic versions were also sited on ward computers) of the management pathway in key clinical areas. In the post-implementation period, the proportion of patients receiving appropriate antibiotics within 4 h of admission to hospital at the intervention site was continuously audited and compared with baseline data. The results were fed back to key medical and nursing staff at the intervention site through a 6-weekly newsletter. The investigators did not provide clinical advice to the medical teams caring for included patients.

Patients

Patients were identified prospectively by review of admission records at the AMAUs at the intervention and control sites. Patients were included if they were receiving an antibiotic for a lower respiratory tract infection and had either a new infiltrate on the chest radiograph or a clinical diagnosis of CAP, made by a specialist registrar or consultant, documented in the case-notes. Patients were excluded if they were aged <16 years, taking immunosuppressive drugs or HIV positive or had neutropenia (neutrophil count <1.0×10⁹/l), aspiration or hypostatic pneumonia, or progressive malignancy, or if the diagnosis was changed before discharge or transfer from the AMAU.

Data collection and statistical analyses

Quantitative data were initially recorded on a pre-piloted data collection sheet and subsequently double entered onto an EPI-info V.6 database. The accuracy of data collection was audited for all patients at the point of discharge from hospital or death. Statistical analyses were carried out using SPSS V.10 for Windows. The primary statistical analysis was of the absolute change in the proportion of patients receiving appropriate antibiotics within 4 h of admission (percentage at intervention site post implementation minus percentage at control site post implementation), adjusted (using linear regression) for age, sex and BTS-recommended adverse prognostic indicators (respiratory rate ⩾30/min, systolic blood pressure <90 mm Hg or diastolic blood pressure ⩽60 mm Hg, urea >7.0 mmol/l, new confusion (Mental Test Score (MTS) ⩽8 or 2 point drop in MTS), age ⩾50 years, presence or absence of comorbidity, and arterial oxygen percent saturation <92% or arterial oxygen tension <8.0 kPa (on any fractional inspired oxygen)).⁷ Appropriate antibiotic treatment was defined according to the 2001 BTS guidelines.⁷ Secondary end points were length of hospital stay and 30-day post-admission mortality. Length of hospital stay was compared using Cox’s regression analyses, adjusted for age, sex and BTS adverse prognostic indicators (as defined earlier).

Before the study, we estimated that samples of 188 (intervention) and 47 (control) individuals in the post-implementation period would be 80% powered (p⩽0.05) to detect a difference of ⩾22.5% by two-sided significance tests in the proportion of patients receiving appropriate antibiotics within 4 h of admission to hospital, and that samples of 204 (intervention) and 51 (control) individuals would have 80% power to detect a 1-day (standard deviation (SD) 2.25) decrease in length of hospital stay.

Cost-effectiveness analyses

Cost-effectiveness analyses were performed from the hospital’s perspective using 2002 prices. Comparisons of antibiotic costs were limited to the cost of the first dose prescribed. Sensitivity analyses focused on the costs of development and evaluation of the intervention, and on the cost of maintaining the intervention over time. We did not include discounting of future costs in the sensitivity analyses because of the short time horizons used. Cost per death prevented was estimated by calculating the number of additional patients who needed to receive antibiotics within 4 h of admission to prevent one death (number needed to treat) using our baseline risk of 30-day mortality (19%) and the odds ratio (OR) of 30-day mortality for patients with CAP who received antibiotics within 4 h of admission compared with >4 h from the study by Houck et al⁹ (OR 0.85, 95% confidence interval (CI) 0.76 to 0.95). In these analyses, the cost savings associated with a mean reduction of 0.4 days in length of stay were included which was a secondary finding of this study.⁹

RESULTS

Of the 623 patients reviewed in the pre-implementation period, 39% were included (181 at the intervention site and 60 at the control site). In the post-implementation period, 770 patients were reviewed and 34% were included (209 at the intervention site and 53 at the control site). The reasons for exclusion are shown in appendix C (available online at http://www.thorax.bmjjournals.com/supplemental). Table 2 describes the demographic and clinical characteristics of the four cohorts. After transfer from the AMAU, 3% (n = 14) of patients had their diagnoses changed to non-pneumonia conditions. Of the 273 patients who had their chest radiographs reviewed by a consultant radiologist or chest physician, 93% were reported to have changes consistent with CAP.

Before implementation, 33% of patients at the intervention site and 32% at the control site received appropriate antibiotic treatment within 4 h of admission to hospital. After implementation, the proportions increased to 56% at the intervention site and 36% at the control site (table 3). The improvement at the intervention site was progressive over time (fig 1) and was mostly due to improving door-to-antibiotic time (from a median time of 4 h before to 2 h 18 min after at the intervention site compared with 3 h 59 min before and 3 h 30 min after at the control site). The difference between the sites in appropriate antibiotic prescribing at 4 h had disappeared by 8 h after admission (table 3). For patients admitted through the A&E department at the intervention site, the proportion of patients receiving appropriate antibiotic treatment within 4 h increased from 28% (16/58) before to 72% (34/47) after (unadjusted change 44%; χ² = 20.85, p<0.001) compared with 36% (44/123) before and 52% (84/162) after (unadjusted change 16%; χ² = 7.31, p = 0.007) for patients admitted directly to the AMAU (χ² = 1.19, p = 0.3 for the difference (8%) between the A&E and AMAU cohorts before implementation; χ² = 6.22, p = 0.01 for the difference (20%) between the A&E and AMAU cohorts after implementation).

Appropriate antibiotic prescribing improved at both sites, from 75% to 87% at the intervention site and from 62% to 77% at the control site (table 2). Antibiotic prescribing patterns changed more at the intervention site. For example, the prescribing of coamoxiclav in combination with a macrolide for patients with severe CAP increased by 65% (from 9% to 74%) at the intervention site compared with 19% (from 11% to 30%) at the control site (unadjusted difference in absolute change from baseline, 46%; unadjusted absolute change, 44%, Fisher’s exact test; p = 0.015). The prescribing of broad-spectrum antibiotic regimens increased (from 42.5% to 77.5%) at the intervention site and decreased (from 45% to 41.5%) at the control site (unadjusted difference in absolute change from baseline, 38.5%; unadjusted absolute change, 36%; χ² test = 26.21, p<0.001).

We found important differences in the severity of illness, however, between intervention and control sites. At baseline, a CURB65 (confusion, urea, respiratory, blood pressure, age over 65) score of ⩾3 (most severe) was present in 37% of patients at the intervention site versus 45% at the control site. In the post-implementation period, the proportions were 44% and 19%, respectively. After adjustment for the differences in severity, the absolute change for the primary outcome remained statistically significant (table 2). We found no significant change in mortality or length of stay at the intervention site or in absolute change between the sites (table 2).

Cost-effectiveness analyses

Funding of a full-time specialist registrar supported our intervention, which allowed for collection of prospective data during the baseline and intervention periods as well as the surveys of medical staff. The total cost of the project was £70 580. We estimate that a hospital could adapt the intervention with a more limited evaluation for £17 809, which includes 780 h of specialist registrar time (appendix D available online at http://www.thorax.bmjjournals.com/supplemental). The cost per additional patient receiving appropriate antibiotics within 4 h was £456, with the latter evaluation costs (table 4) or £132 with no evaluation. Cost efficiency improved markedly over time as most of the costs occur in the first year (table 4).

From the data of Houck et al,⁹ the number of additional patients who need to receive antibiotics within 4 h to prevent one death is 42 (95% CI 26 to 129). In the post-intervention period, 17% more patients received appropriate antibiotics within 4 h as a result of the intervention. The cost reduction as a result of an expected 0.4-day reduction in mean length of stay in these patients would be £2424. Using this cost reduction, the recalculated costs per additional patient receiving appropriate antibiotics within 4 h are £396 (95% CI £251 to £833) for a limited evaluation and £71.50 (95% CI £45 to £150.50) for no evaluation. The cost per death prevented therefore is £16 632 (range £6526 to £107 457) with a limited evaluation and £3003 (range £1170 to £19 414) with no evaluation.

Quantitative and qualitative surveys

The key findings can be summarised as follows:

1. Medical staff regarded CAP as less important than other common acute medical emergencies, such as myocardial infarction.

2. Medical staff had poor awareness of the BTS guidelines.

3. Medical staff had poor understanding of severity assessment in CAP and were unable to link this to an appropriate management strategy.

4. Patients who presented to the A&E department often did not receive antibiotics before transfer to the AMAU.

5. At the AMAU, timing of the first dose of antibiotics could be improved by simply prescribing an immediate once-only dose in the appropriate section of the prescription chart.

Appendix A available online at http://www.thorax.bmjjournals.com/supplemental provides further details.

DISCUSSION

The two most important lessons learnt were (1) patients admitted to the AMAU from the A&E department may experience delays in essential treatment unless this is initiated before transfer (this has led to a general review of care for all patients, not just for those with CAP); and (2) prescribing a once-only dose of antibiotics in the AMAU is important in reducing door-to-antibiotic time.

Our intervention markedly improved the proportion of patients who received appropriate antibiotics within 4 h of admission to hospital. This was mostly due to reducing door-to-antibiotic time. The improvement occurred throughout the post-intervention period, which suggests that feedback to clinicians is an important part of an ongoing quality improvement programme. The difference between the intervention and control sites after implementation in the proportion of patients receiving appropriate antibiotics within 4 h had disappeared at 8 h post admission. It is disappointing that one quarter of patients were still not receiving appropriate antibiotics within 8 h at the intervention site post implementation, although improvement may have been constrained by a ceiling effect (described later). There have been concerns about the long-term sustainability of improvements in processes of care, such as described in this study. To optimise the likelihood of sustained improvement, it is vital that interventions are embedded in routine clinical practice. Although this could be achieved with most of the components of our intervention, many hospitals in the UK do not have the required resources for high-quality long-term surveillance and feedback to healthcare staff.

Although the absolute change in appropriate antibiotic prescribing post intervention approached significance (p = 0.08), this analysis does not take account of the baseline differences between the intervention and control sites. The adjusted difference in absolute change from baseline, which takes account of baseline and severity differences, was +6%. This, along with the other antibiotic prescribing data presented, suggests that the intervention had more effect on appropriate antibiotic prescribing at the intervention site, albeit on a background of secular trends at both sites. The latter may have been due to increasing awareness of BTS antibiotic recommendations or local factors such as a drive to use coamoxiclav rather than cephalosporins at the control site, but not at the intervention site, to reduce Clostridium difficile-associated diarrhoea. The intervention site also had a higher baseline level of performance in appropriate antibiotic prescribing (75% v 62%), and was therefore likely to be closer to a ceiling effect, which may have limited the opportunity for change compared with the control site. From the qualitative data collected during the post-implementation period, we estimated the maximum ceiling effect for our primary outcome measure to be 74%. A worrying observation was the increase in broad-spectrum antibiotic prescribing at the intervention site post implementation, which may have caused adverse effects, such as C difficile-associated diarrhoea or the evolution of antibacterial resistance. Also, an unmeasured increase in costs and adverse effects may have occurred in non-pneumonia conditions (eg, chronic obstructive pulmonary disease) due to guideline overspill. Although such adverse effects were not measured in our study, we recommend that trials measuring the clinical or cost-effectiveness of antibiotic guideline implementation in the future should measure and report these data.

No other interventions occurred at either site during the study period to explain the changes in door-to-antibiotic time or antibiotic prescribing. Contamination from the intervention to control site is unlikely to have been the cause of the increase in appropriate antibiotic prescribing: if this were the case, improvement in door-to-antibiotic time should have occurred at both sites. Regression to the mean, which is a recognised threat to the validity of quasi-experimental observational studies, cannot be ruled out, but would seem unlikely as both hospitals had similar processes of care at baseline. Also, the achievement of processes of care in our previous audit⁶ was not markedly different from that in the pre-implementation cohort at the intervention site (ie, 52% v 50% for door to antibiotic time within 4 h, and 72% v 75% for appropriate antibiotics). In addition, there were no major changes in quality indicators that were not targeted by the intervention. For example, at the intervention site, procurement of a blood culture within 24 h of admission in patients with severe CAP was 71% before and remained 71% after implementation. Likewise, discharge within 24 h of switching to oral antibiotics was 22% before and 22% after implementation. Compared with our previous audit⁶ and the CURB65 derivation and validation study,¹⁵ mortality in this study was higher (15% and 10%, respectively v 19% in the pre-implementation and post-implementation cohorts at the intervention site). This can be explained by the higher proportion of patients with severe CAP in this study (24.5% in our previous audit and 29% in the CURB65 derivation and validation study v 37% and 44% in the pre-implementation and post-implementation cohorts at the intervention site).

Cost effectiveness was most sensitive to the level of evaluation performed. As evaluation is an integral component of the quality improvement cycle, it is likely that most hospitals implementing our intervention would want to undertake at least a limited evaluation. The costs of evaluation could be markedly reduced with the availability of electronic clinical and prescribing data for individual patients. As things stand, even the estimated minimum cost per additional patient receiving antibiotics within 4 h is high enough that hospitals will need to be convinced that there is a true association between the outcome in patients with CAP and door-to-antibiotic time. The most convincing evidence comes from two large observational studies on mortality from CAP in patients aged>65 years.^9,12 An additional study found that, in comparison with other patients with CAP, those with prolonged hospital stay had considerably longer door-to-needle time and were also less likely to have received appropriate antibiotics.¹⁰ However, two observational studies failed to show a significant relationship between door to needle time and time to clinical stability for hospitalised patients with CAP,^16,17 probably because they were not adequately powered to detect clinically important differences. A larger study (n = 28 661) showed that implementation of a treatment guideline was associated with reduced inpatient 30-day mortality in CAP.³ Detailed process measures were not reported, although the proportion of patients receiving appropriate antibiotics increased from 28% before to 56% after implementation.

Our results are consistent with those of a previous North American study,² which also showed that marked improvement in door-to-needle time did not lead to an improvement in mortality or a reduction in hospital stay. This study included 2087 patients, but, as with our study, was still not adequately powered to reliably detect important differences in clinical outcomes.² We therefore integrated our results with those of Houck et al⁹ to estimate the cost per death prevented. Presuming that the link between door-to-antibiotic time is real, the results of this analysis suggest that the intervention could be a cost-effective strategy to reduce mortality from CAP.

Three assumptions in our cost-effectiveness analyses are worth considering. Firstly, as our intervention was targeted at initial antibiotic prescribing, we included only the cost of the first dose of antibiotics. It is likely, however, that initial prescribing will affect subsequent antibiotic use. In the worse case scenario that the initial antibiotic regimen had been continued for a further 10 days, the cost effectiveness ratios would increase to £454 and £778 per additional patient receiving antibiotics within 4 h for no evaluation and a limited evaluation, respectively. Secondly, to calculate the cost per death prevented, we extrapolated data from a North American study to UK practice; the validity of this approach is open to debate. Thirdly, the study by Houck et al⁹ was of patients aged >65 years. Our study was of patients aged >16 years; the majority (71%), however, were aged >65 years.

The two most important lessons relate to processes of care (door to initiation of treatment time and once-off prescribing in the drug chart), which are relevant to all patients with serious infections and some non-infection conditions. It is likely to be more cost effective for hospitals to adapt our intervention to all patients with sepsis. It is debatable as to whether such an intervention requires a specialist registrar for development, implementation and evaluation, as other healthcare professionals (eg, a quality improvement nurse or pharmacist) are likely to have the required expertise. Using such personnel would also decrease costs.

Changing processes of care requires complex interventions, which raise particular challenges regarding generalisation of evidence about effectiveness.¹⁸ According to the Medical Research Council’s (MRC) proposed “continuum of evidence” we are now in Phase II, with a sound theoretical basis, observational data that allow modelling of the potential effect of care processes on outcome and evidence about successful implementation of change from two exploratory trials.¹⁹ The next step on the MRC’s continuum of evidence should be a definitive trial (Phase III), preferably with a cluster randomised design before progression to Phase IV (long-term implementation). However, an alternative approach to quality improvement is to define care bundles with several components that are likely to improve the outcome of care and to test the relationship between implementation and outcome in multiple hospitals.²⁰ In this approach, the early administration of antibiotics would be one component of a bundle, along with other key quality improvement targets. A review of the management of young patients who had died from CAP identified delayed antibiotic treatment as just one potentially important factor.⁸ The others were: (1) failure to identify early in the course of management the need for intensive care unit assessment or admission; (2) suboptimal oxygen therapy; and (3) fluid resuscitation. These are all components of the care pathway whose implementation in the US was associated with a marked reduction in mortality from CAP.³ We favour the quality improvement approach based on care bundles as, based on this study, the collection of definitive data about the effectiveness of individual components of a bundle is likely to be prohibitively expensive and difficult to justify ethically.

In summary, the costs of developing and evaluating a local CAP quality improvement initiative were considerable. The intervention predominantly improved door-to-antibiotic time and could be a cost-effective strategy to reduce mortality. This needs to be confirmed in a multihospital study and we believe that the most efficient design would be to test the effectiveness of implementing a bundle of care that includes early antibiotic administration. Although awaiting definitive evidence, hospitals can implement two simple and inexpensive strategies to improve the delivery of antibiotics: (1) give the first dose of antibiotics in the A&E department before transfer to the AMAU, and (2) for patients admitted directly to the AMAU, prescribe a first dose for immediate administration in the once-only section of the prescription chart.