METHODS

Setting and study design

The study was conducted at a 1000 bed tertiary care university hospital between June 1998 and May 2001, thus including three winter seasons. All consecutive and non-selected patients presenting with CAP were prospectively recorded. Pneumonia was defined as a new infiltrate together with symptoms and signs of a lower respiratory tract infection. Exclusion criteria were the presence of severe immunosuppression associated with a risk of opportunistic infections, pneumonia as an expected terminal event of a severe chronic disabling co-morbidity, and an alternative diagnosis during follow up.

The decision where to treat the patient with pneumonia was left to the discretion of the attending physician. No attempt was made to implement standardised criteria or rules for the assessment of pneumonia severity.

The study design was approved by the local ethical committee.

Data collection and evaluation

The patients were assessed on admission and during follow up using a standardised data sheet. The following parameters recorded on admission were evaluated: date of presentation (month, year), age, sex, alcohol habits, co-morbidity, residence in nursing home, duration of symptoms, clinical symptoms (body temperature, respiratory rate, heart rate, arterial systolic and diastolic blood pressure, pneumonia associated confusion—that is, disorientation with regard to person, place or time that is not known to be chronic, stupor, or coma), blood gas analysis (pH, Pao₂, Paco₂, Fio₂), chest radiograph (number of lobes affected, pleural effusion), and laboratory parameters (haematocrit, leucocyte count, band forms, serum creatinine, blood urea nitrogen, sodium, blood glucose). At the clinical end points (hospital discharge or death) the following parameters were recorded: ICU admission, mechanical ventilation, septic shock, renal failure, and 30 day in-hospital mortality.

Definitions

For comparison of the distribution of pneumonia severity with other populations, the PSI and CURB indexes were used. These indices are defined as follows:

• The pneumonia severity index (PSI) developed by Fine and coworkers is a validated index composed of the variables age, sex, co-morbidity, and vital sign abnormalities together with several laboratory, blood gas, and radiographic parameters. It results in a five class point scoring system reflecting an increasing risk of mortality.⁶

• The CURB index recently proposed by Lim and Macfarlane⁷ is simple and is derived from the four core severity criteria identified in the original BTS study⁵: confusion, urea nitrogen (>7 mmol/l), respiratory rate (⩾30/min), and blood pressure (diastolic pressure ⩽60 mm Hg). As an alternative to the diastolic blood pressure, systolic blood pressure of <90 mm Hg can be used. Again the risk of severity increases with the number of criteria met.

The following prediction rules were validated with regard to pneumonia severity and mortality:

• Modified ATS rule: this rule was met if at least two of three minor criteria assessed at admission (systolic blood pressure <90 mm Hg, multilobar (>2 lobes) involvement, Pao₂/Fio₂ <250) or one of two major criteria assessed at admission or during follow up (requirement for mechanical ventilation or septic shock) were present.³

• BTS rule I: this rule was met if at least two or three of the following criteria assessed at admission were present: respiratory rate ⩾30/min, diastolic blood pressure ⩽60 mm Hg, or blood urea nitrogen >7 mmol/l.

• BTS rule II: this rule modifies rule I, replacing the last criterion (blood urea nitrogen) by confusion.⁵

• Modified BTS rule as proposed by Neill and coworkers: this rule uses the same adverse prognostic factors in the presence of two or more of these criteria.⁸

• ATS 2001 guideline rules: these rules include the modified ATS rule or the following: two of six minor criteria or one of four major criteria according to the original ATS statement.⁴

• Sepsis, severe sepsis, and septic shock were defined according to Bone and coworkers. Thus, sepsis (systemic inflammatory response syndrome (SIRS) plus pneumonia) was present in cases with at least two of the following four criteria: body temperature >38°C or <36°C; heart rate >90 bpm; respiratory rate >20/min, or Paco₂ <32 mm Hg; or leucocyte count >12×10⁹/ml, <4×10⁹/ml, or >10% band forms. Severe sepsis was defined as sepsis associated with organ dysfunction together with perfusion abnormalities. One of the following criteria had to be met: pH <7.3, pneumonia associated confusion, acute renal failure, systolic blood pressure<90 mm Hg, Pao₂/Fio₂ <200. Septic shock was defined as severe sepsis despite adequate fluid resuscitation and the requirement for vasopressors more than every 4 hours.⁹

The predictive rules were also evaluated using the two risk class scoring systems. In addition, the CURB index was validated against the PSI index, and the CURB index was compared with an index composed of the number of sepsis criteria present.

Statistical analysis

All data were analysed and processed on SPSS Version 10.01 on a Windows 98 operating system. Statistics not provided by this software package were calculated using EpiInfo 6.0. Results are expressed as mean (SD) unless indicated otherwise.

For the calculation of the operative indices of predictive rules, the following definitions were applied:

Sensitivity: ratio of predicted severe to truly severe cases;

Specificity: ratio of predicted non-severe to truly non-severe cases;

Positive predictive value (PPV): ratio of predicted severe cases to number of cases meeting the criterion ( = incidence of criterion);

Negative predictive value (NPV): ratio of non-severe cases to number of cases meeting the criterion.

The overall accuracy was the ratio of correctly predicted severe and non-severe cases to the total number of cases. The exact binominal 95% confidence intervals were calculated for all operational characteristics (tables only).

Receiver-operator-characteristics (ROC) curves were calculated for the PSI and CURB scores. All other predictive rules were not suitable for this analysis. The area under the curve, standard error, and 95% confidence intervals are given and values were compared using the Student’s t test. The χ² test was used to compare proportions. The significance level of all analyses was set at 5%.

RESULTS

Patient population

A total of 731 patients were eligible to participate in the study. Fourteen patients who died with pneumonia as an expected terminal event or severe disabling co-morbidity and who were not admitted to the ICU were excluded. An additional 21 patients (five admitted to the ICU) had to be excluded because the treatment setting was not documented. Thus, 696 patients (66% male) of mean (SD) age 67.8 (17.1) years (range 18–101) were available for analysis. Of these, 481 (69%) had datasets for all risk scores under evaluation. Two thirds of the patient population (n = 464) were elderly (>65 years). 116 patients (16.7%) were admitted to the ICU (54 immediately on admission to hospital, 50 within the first 24 hours, four after 48 hours, and five after 72 hours). In the remaining three cases the time of ICU admission was not precisely documented. Sixty one patients received mechanical ventilation; 13 received non-invasive ventilation, six of whom were later intubated. The mortality rate in the whole group was 6.8% (47/481); in those admitted to the ICU the mortality rate was 40.5% (47/116).

Risk assessment

Risk classes according to the PSI could be calculated in 489 patients (70.3%), including 90 patients admitted to the ICU (18.4%); 34 (7%) were classified as risk class I, 50 (10%) as class II, 98 (20%) as class III, 194 (40%) as class IV, and 113 (23%) as class V. The rates of ICU admission and mortality were 0% for risk class I, 10% and 2% for class II, 10% and 3% for class III, 21% and 8% for class IV, and 31% and 18% for class V, respectively (a complete set of data is given in table S1 available on the Thorax website at www.thoraxjnl.com/supplemental). Compared with the original PORT (Pneumonia Patient Outcomes Research Team) cohort in which the PSI was validated, the proportions of risk classes and rates of mortality were comparable; however, the rates of ICU admission were twice as high in our population.

The CURB criteria could be calculated in 592 of 696 patients (85.1%); 229 of the 592 patients (39%) had no criteria, 231(39%) had one, 91 (15%) had two, 33 (6%) had three, and eight (1%) fulfilled all four criteria. The distribution of CURB criteria according to the PSI scoring system is shown for all patients in whom both scores could be calculated (482/692, 70%) in table S2 available on the Thorax website at www.thoraxjnl.com/supplemental. There was a narrow association between the number of CURB criteria and increasing risk in the PSI. When the CURB criteria were used to predict ICU admission, the probability of severe pneumonia increased with the number of criteria and was best reflected by summarising three CURB categories (3% for CURB 0, 21% for CURB 1–2, and 56% for CURB 3–4). The corresponding numbers for mortality were 1%, 8%, and 34%. A summary of the predictive potential is given in table 1.

The CURB criteria were superior to the PSI criteria with regard to predicting pneumonia severity (ROC 0.725 (0.025) v 0.665 (0.03), p = 0.023; fig 1).

Application of the sepsis score resulted in similar increasing risk patterns but inferior risk estimates compared with the CURB criteria. The risk for severe pneumonia was 4% (1/25) in the absence of sepsis criteria, 11% (39/348) in the presence of one or two criteria, and 24% (67/283) with three or four criteria. The corresponding numbers for mortality were 4% (1/25), 4% (13/347), and 10% (28/282). Overall, the presence of sepsis criteria only weakly differentiated severity according to the PSI score. Severity was reflected increasingly appropriately by the presence of severe sepsis and septic shock (table 2).

Validation of severity rules

The modified ATS rule achieved a sensitivity of 69%, specificity of 97%, PPV of 87%, and NPV of 94% in predicting severe pneumonia. The alternative ATS rule (two of six minor or one of four major criteria) was more sensitive but less specific. Both operative indices were superior to the predictive values of BTS rules I and II (table 3). A combination of these rules (severe pneumonia in the presence of the modified ATS rule or one of two or both BTS rules) did not improve the predictive power (sensitivity 79%, specificity 81%, PPV 51%, NPV 95%).

Of the 12 patients misclassified as severe by the modified ATS rule but not admitted to the ICU (false positive cases), nine (75%) were >65 years and only two were classified as low risk patients (Fine class 1 and 3, respectively). No patient died. The only non-elderly patient with Fine class 1 had severe asthma and acute respiratory failure not requiring ventilatory support, but no additional severity criteria.

Conversely, of the 36 patients misclassified as non-severe but admitted to the ICU (false negative cases), 20 (56%) were younger than 65 years and 18 (50%) were high risk patients (Fine class 4 and 5, respectively). Three patients died. Overall, 39% met one severity criterion of the original ATS statement, 36% two, 3% three, and 22% none. Of these, most had one criterion of acute respiratory failure (42%) and none had both criteria of acute respiratory failure and haemodynamic compromise.

The sensitivities of all the prediction rules for predicting ICU admission immediately or within 24 hours of admission to hospital roughly corresponded to the overall performance (modified ATS rule 67% and 67%; alternative ATS rule 89% and 87%; BTS I rule 40% and 37%; BTS II rule 48% and 49%; modified BTS rule 55% and 52%). Both ATS rules predicted 100% of ICU admissions later in the course of hospitalisation (48–72 hours; n = 9). The corresponding performances for the BTS rules were uniformly poor (22%, 11% and 22%, respectively). Accordingly, only two of nine patients (22%) meeting the modified ATS rule after 48–72 hours initially met “minor” criteria.

The assessment of the modified ATS rule according to the PSI risk class scoring system resulted in the following classifications: 3% risk class I, 6% class II, 6% class III, 18% class IV, and 29% class V (table 4). The BTS rules tended to overestimate severity, particularly in risk class V. Sepsis was a poor predictor of severity due to lack of specificity, and severe sepsis had inferior operative indices compared with the modified ATS rule. Septic shock was inferior to the modified ATS rule in terms of sensitivity.

Predictive potential of severity rules for mortality

The predictive potential of the modified ATS rule was very high, particularly the NPV which exceeded 99%. This rule was superior to both BTS rules also in terms of prediction of mortality (table 5). Mortality was 8%, 40%, and 69% in the presence of sepsis, severe sepsis, and septic shock, respectively. The predictive potential of septic shock was comparable to that of the modified ATS rule.

Missing values

Because of the relatively high proportion of cases with missing values, all calculations were repeated including only patients with complete sets of data. The results were very similar to those of the whole population (data not shown). In particular, sensitivities and specificities were all within the ranges calculated for the whole population.

DISCUSSION

The most important findings of this study are: (1) the modified ATS rule had excellent predictive indices for pneumonia severity of individual patients with a sensitivity of 69%, specificity of 98%, PPV 87%, and NPV 94% (these figures compared favourably with all three BTS rules); (2) the same was true for the prediction of mortality, with a sensitivity of 94%, specificity of 93%, PPV 49%, and NPV 99.5%; (3) with regard to risk class assessment of the population studied, the CURB criteria achieved predictions of pneumonia severity and mortality comparable to the PSI. It was not possible to collect complete sets of parameters for the calculation of the rules for the whole population, which suggests a limited workability of all rules.

Severe pneumonia remains difficult to define, regardless of the reference used when validating defined criteria. This is mainly due to structural differences across treatment settings with regard to the relative role of emergency departments, intermediate care facilities and ICUs, and ongoing changes in medical practice such as non-invasive ventilation which inherently modify concepts of severity. Since we used “ICU admission” in our hospital as a reference for severe pneumonia, the predictions presented here apply exclusively to hospitals with a similar structure. However, in hospitals with different structures, these criteria could still be used to identify patients who need increased attention and specific treatment.

Another crucial issue in the validation of a predictive rule for pneumonia severity is a detailed description of the population studied. Our population had four basic characteristics: (1) all were recruited from our tertiary care university hospital which specialises in the treatment of pneumonia; (2) two thirds of the population were men and were aged over 65 years; (3) the seasonal variability of the incidence of CAP was overcome by including three winter seasons in the study; (4) the distribution of proportions of pneumonia severity and mortality rates were comparable to the PORT population used for the validation of the PSI.⁶

However, in contrast to the PORT population, the rate of ICU admissions in our population was twice as high as the upper limit of reported rates of 10–18% in the literature.¹ This difference is most probably due to different medical attitudes to the treatment of patients with severity criteria but not requiring mechanical ventilation and/or vasopressors. In our view, these patients are at higher risk of adverse outcomes and, whether treated in the ICU or any facility comparable to intermediate care units, should be considered as severe CAP. This view is supported by the fact that our mortality rates were lower than predicted by PSI, which may mean that the ICU was used expectantly and to the benefit of the patients.

The modified ATS rule provided very favourable predictions of severity and mortality. Compared with the derivation study,³ the sensitivity was somewhat lower (69% v 78%). Predictions of immediate requirement for ICU admission within 24 hours did not differ from the overall performance of the rule. In fact, 50% of the 36 patients classified as false negative (forming 5% of the whole population) were high risk patients (Fine classes 4 and 5), indicating that the rule missed detecting patients with severity criteria. Three of these misclassified patients died. In most instances (42%) the only severity criterion present was acute respiratory failure reflected by a respiratory rate of >30/min or a Pao₂/Fio₂ <250. No patient was missed with both acute respiratory failure and haemodynamic compromise, which suggests that most of these patients had pneumonia of limited severity. This is in accordance with our previous study.³ Determination of the severity of patients with moderate acute respiratory failure (not requiring ventilatory support) remains difficult to standardise. It is important to determine whether these patients can be assessed more reliably using serial determinations of respiratory rate and blood gas tensions during the first 30–60 minutes after admission. In any case, the false negative predictions underline the fact that the decision to admit to the ICU is always a clinical one, with any predictive rule of severity representing an aid for decision making rather than an imperative algorithm.

The specificity of the rule was very close to the derivation study (94% v 95%). Of the false positive predictions, 75% were elderly and all except two were high risk patients according to the PSI. None of these patients died. It is therefore probable that these misclassified cases represent patients who would not necessarily have undergone intubation or vasopressor treatment in case of further deterioration due to advanced age and disabling co-morbidity.

Compared with the modified ATS rule, the alternative ATS rule and all the BTS rules had inferior predictive indices. The alternative ATS rule in particular was less specific (82%), resulting in a low PPV (49%). The BTS rules were originally designed for the prediction of mortality of hospitalised patients.^5,10 Several previous studies have confirmed the predictive potential of these rules,^8,11,12 although some more recent studies reported less impressive results both in general and elderly populations.^13–16 All three BTS rules had limited specificity (89%, 86%, and 83%) and low sensitivities (34%, 44%, and 48%, respectively). In accordance with all previous reports, the rules had high NPVs (87%, 89%, and 89%, respectively).^8,11–16 These differences from the modified ATS rule were mainly due to an overestimation of severity of all BTS rules in PSI risk class V, a finding which has been described previously.¹⁵

In the recent validation study by Angus et al¹⁷ the modified ATS criteria had the best overall discrimination as measured by ROC curves. However, the PPVs remained low for all ATS and BTS prediction rules (22% at best measured for the modified ATS criteria). The sensitivity of the modified ATS criteria was similar to that described in the first derivation study (71% v 78%). The specificity reached only 72%, but it is difficult to understand why 28% of patients met the severity criteria but were not admitted to the ICU. Clearly, a large number of these patients had “minor” baseline criteria since all patients requiring mechanical ventilation were admitted to the ICU and met the “major” criteria. Nevertheless, specificity for the second major criterion of septic shock was only 97%, implying that 3% were not admitted. Since patients meeting “minor” criteria met two of three criteria reflecting acute respiratory failure, severe sepsis or shock, or extension of radiographic infiltrates, these patients would have been ideal candidates for ICU admission. As the authors do not provide information about treatment restrictions and access to intermediate care facilities, the meaning of these figures remains difficult to interpret.

The criteria of sepsis, severe sepsis, and septic shock proposed by Bone and coworkers appropriately reflected increasing pneumonia severity. However, as expected, the criterion of sepsis remained poorly specific (24%), severe sepsis had only moderate predictive power (sensitivity 75%, specificity 74%), and septic shock was poorly sensitive (48%). Similar patterns applied for the prediction of mortality, although both severe sepsis and septic shock proved to be more sensitive in predicting death (89% and 81%). These trends could also be seen when the sepsis criteria were compared with the PSI scoring system. The modified ATS rule specific for pneumonia is therefore clearly superior to criteria of sepsis for the prediction of severe pneumonia.

The PSI scoring system has been shown to be a powerful tool for assigning the risk of death from pneumonia in different populations.^6,14,18,19 It is noteworthy that this scoring system was primarily designed to identify patients with a low mortality risk who could safely be treated as outpatients. This differs from the ATS rules which were derived to identify individual patients who should be treated in the ICU and the BTS rules which were established to predict mortality of individual hospitalised patients. Thus, although the PSI can be used to predict the probability of death for different groups at risk, it does not allow for individual predictions.

In this study we have confirmed the superior predictive power of the PSI scoring system. However, the CURB rating—which is much easier to calculate—has only a weak association with the PSI and has a higher predictive power with regard to pneumonia severity. Whereas the predictions for requiring admission to the ICU were 4%, 4%, 6%, 11%, and 17% for PSI risk classes I–V in the PORT population and 0, 10%, 10%, 21%, and 31% in our population, the figures for CURB classes 0, 1–2, and 3–4 were 3%, 21%, and 56% in our population. Similar patterns were also seen for predicting mortality (0.5%, 1%, 1%, 9%, and 27% for PSI in the PORT population; 0.5%, 1%, 1%, 9%, 27% for PSI in our population; 1%, 8%, and 34%, respectively, for CURB classes 0, 1–2, and 3–4). The CURB classification may therefore have the potential to replace the PSI scoring system for deciding which patients to treat as outpatients. CURB class 0 patients would have a very low risk of mortality comparable to patients with PSI class I–III and be candidates for ambulatory treatment. This is in accordance with similar suggestions made by the recent BTS guidelines,¹⁹ but our data do not support the use of a cut off of two or more CURB criteria for deciding whether to admit to the ICU, as described in these guidelines. In contrast to the findings of Lim and coworkers,⁷ we did not find clear cut offs for mortality risk between CURB classes 1 and 2 and CURB classes 3 and 4 (mortality rates 7% and 8%, and 39% and 13%, respectively). In our study a reasonable cut off for the decision to admit to the ICU would have been three or more CURB criteria. This issue clearly deserves further study in different populations.

In conclusion, our study confirms that the modified ATS rule is highly powered for predicting severe pneumonia in individual patients and may therefore be incorporated into current guidelines for the assessment of pneumonia severity. Although the PSI scoring system was a reliable tool for the validation of predictive rules in our study, it is tedious to calculate and may fail to gain widespread acceptance for identifying low risk patients in the non-expert routine medical setting. In this situation the CURB criteria may be a valuable alternative.