Community-acquired pneumonia (CAP) is a common disease which frequently leads to hospitalisation and, not rarely, to death.1 As a result, clinical prediction rules are often used to determine the prognosis. Patients who are assessed to have moderate and high risks of death are admitted to the general ward and the intensive care unit (ICU), respectively. Such prediction rules assist in the appropriate utilisation of limited hospital and ICU resources. Two commonly used rules are the Pneumonia Severity Index (PSI) and the CURB-65 (confusion, urea, respiratory rate, blood pressure and age) score.2 3 However, while they are useful in determining which low-risk patients may be discharged home from the emergency department, they—and most other rules to date—are less accurate in determining the patients at the highest risk of death who require ICU admission.2^–5 Indeed, how severe CAP should be defined remains unclear.

In response, the Infectious Disease Society of America (IDSA) and the American Thoracic Society (ATS) issued guidelines in 2007 which defined severe CAP—and recommended ICU admission—when one of two major criteria (the need for invasive mechanical ventilation or vasopressors) or three of nine minor criteria are fulfilled.1 The minor criteria included variables in the CURB-65 score except age, hypoxaemia, multilobar infiltrates, leucopenia, thrombocytopenia and hypothermia. In the first ever study of these IDSA/ATS guidelines, Liapikou and colleagues were able to validate the major but not the minor criteria.6 Yet, while fulfilment of the major criteria will obviously necessitate ICU admission, validation of the minor criteria—which, when present, often create a dilemma on where to place the patient—is arguably more important.

We therefore conducted this study with the aim of assessing the predictive characteristics of the IDSA/ATS minor criteria for hospital mortality and ICU admission in comparison with the PSI and the CURB-65 scores.

METHODS

Study population

From January 2004 to December 2007 we prospectively recorded all adult admissions into our 1000-bed university hospital for CAP onto an electronic database. CAP was defined as an acute infection of the pulmonary parenchyma associated with an acute infiltrate on the chest radiograph with two or more symptoms including fever (⩾38°C), hypothermia (<36°C), rigors, sweats, new cough or change in colour of respiratory secretions, chest discomfort or dyspnoea. Patients who were hospitalised within 14 days before the onset of symptoms or discharged home from the emergency department, immunocompromised patients (with HIV infection or haematological malignancies, undergoing chemotherapy, or on steroids equivalent to at least 10 mg/day prednisolone for a month) and patients who were subsequently diagnosed with tuberculosis were excluded. As the key focus of this study is the IDSA/ATS minor criteria, patients who fulfilled any IDSA/ATS major criteria for severe CAP on presentation at the emergency department (ie, the need for invasive ventilation or vasopressors) were also excluded.

Clinical management

During the study period, patients with CAP were admitted to a general ward under the care of respiratory physicians. After office hours the patients were clerked by the general medicine team on call. The nurse-to-patient ratio was one staff nurse and one assistant nurse to 12 patients, and monitoring of parameters was performed 1–6-hourly as needed. The 2001 ATS guidelines were applied and patients were discharged when they reached clinical stability and became afebrile.7 Admission to the ICU was left to the discretion of the attending physicians in the emergency department or the general ward. This was a closed medical ICU with a nurse-to-patient ratio of 1:2 and run 24 h by respiratory physicians who were also accredited as intensivists. Management was modelled after the Surviving Sepsis Campaign guidelines.8

Data collection

At the end of the study period, 12 research nurses under the close supervision of three investigators (JP, NWA, KCS) collected the following data via medical record review using a standardised data collection form: demographics (age, sex, nursing home residence), comorbid illnesses (heart failure, cerebrovascular, neoplastic, renal or liver diseases), the first available physical examination findings (mental status, pulse, respiratory rate, blood pressure, temperature), laboratory findings (blood urea nitrogen, glucose, sodium, haematocrit, white blood cell and platelet count, arterial oxygen tension (Pao₂) or oxygen saturation with the fraction of inspired oxygen (Fio₂), arterial pH) and radiological findings (pleural effusion, unilobar versus multilobar infiltrates) at the emergency department, and the use of invasive ventilation and vasopressors in the ICU. The principal investigator (JP) reviewed every form and medical record to ensure accuracy. These data were used to determine the IDSA/ATS minor criteria for severe CAP, the PSI and the CURB-65 score.

The primary and secondary outcome measures were all-cause hospital mortality and admission to the ICU, respectively.

Data analyses

Hospital mortality was classified by the number of IDSA/ATS minor criteria present. We calculated the sensitivity, specificity, positive (PPV) and negative (NPV) predictive values, relative risk and the discriminatory power using the areas under the receiver operating characteristic (ROC) curves of these criteria to predict hospital mortality. These characteristics were compared with those of the PSI and the CURB-65 score. High-risk patients were defined using commonly accepted classifications (IDSA/ATS minor criteria ⩾3, PSI classes IV and V and CURB-65 score ⩾3) and we evaluated the outcomes of patients with discordant classifications using these rules.

After excluding patients with orders to withhold life-sustaining treatment, the above analyses were repeated for ICU admission. Focusing on patients with three or more IDSA/ATS minor criteria, we compared the baseline characteristics, severity and hospital mortality of those with and without ICU admission using the χ² test and the Student t test. Patients who were first sent to the general ward before ICU transfer were defined as having a delayed ICU admission. We entered the presence of a delay into a logistic regression model to assess its effect on mortality, and included the number of PSI points into the model to account for the overall disease severity because it incorporates patient demographics, chronic conditions and acute parameters.

The Bonferroni correction was used for multiple comparisons and a p value of <0.05 was considered statistically significant. The statistical software SPSS Version 15.0 (SPSS, Chicago, Illinois, USA) was used for analyses.

RESULTS

Study population

During the study period 1310 patients were admitted to the hospital with CAP, of which 68 were excluded because they immediately fulfilled the IDSA/ATS major criteria for severe CAP (ie, they were invasively ventilated and/or started on vasopressors in the emergency department). The baseline characteristics of the remaining 1242 patients are shown in table 1.

Hospital mortality

Table 2 describes the predictive characteristics of the individual IDSA/ATS minor criteria for hospital mortality. While each criterion had relatively high specificity and NPV, hypoxaemia, multilobar infiltrates, confusion and a high blood urea nitrogen had higher sensitivity and specificity.

Overall, the hospital mortality was 14.7%. Mortality increased with the number of IDSA/ATS minor criteria present (table 3) and rose sharply from 0.9%, 1.5% and 10.8%, respectively, for patients with none, one and two minor criteria to 35.2% for patients with three minor criteria. Altogether, 330 (26.6%) patients had at least three minor criteria.

The area under the ROC curve (fig 1) for the prediction of hospital mortality using the IDSA/ATS minor criteria was 0.88 (95% confidence interval (CI) 0.86 to 0.91), which was higher than that of the PSI (0.86 (95% CI 0.83 to 0.88)) and the CURB-65 score (0.82 (95% CI 0.78 to 0.85), p<0.05 after Bonferroni correction for each pairwise comparison).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Receiver operating characteristic curves for hospital mortality for the Infectious Diseases Society of America/American Thoracic Society (IDSA/ATS) minor criteria, Pneumonia Severity Index (PSI) class and CURB-65 (confusion, urea nitrogen, respiratory rate, blood pressure) score. The areas under the receiver operating characteristic curves were 0.88 (95% CI 0.86 to 0.91) for the IDSA/ATS minor criteria, 0.86 (95% CI 0.83 to 0.88) for the PSI class and 0.82 (95% CI 0.78 to 0.85) for the CURB-65 score (p<0.05 after Bonferroni correction for each pairwise comparison).

The application of a threshold of three or more minor criteria as suggested by IDSA/ATS to define severe CAP resulted in a sensitivity, specificity, PPV and NPV of 81.4%, 82.9%, 45.2% and 96.3% for hospital mortality (table 4). The corresponding values for a PSI class of IV or higher were 96.2%, 57.9%, 28.3% and 98.9% respectively, while the corresponding values for a CURB-65 score of 3 or higher were 61.2%, 83.3%, 38.8% and 92.5%, respectively. The IDSA/ATS minor criteria, PSI class and CURB-65 score classified 26.6%, 50.1% and 23.3% as severe CAP, respectively.

Table 5 describes the outcomes of patients with discordant classifications into high-risk versus low-risk CAP using the various clinical prediction rules. The IDSA/ATS minor criteria classified as severe 99 patients who were defined as low-risk by the CURB-65 score (43.4% of these patients died). The PSI class classified as severe 324 patients who were defined as low-risk by the IDSA/ATS minor criteria (only 9.6% of these patients died). Using a threshold of two instead of three minor IDSA/ATS criteria to define severe CAP improved the concordance with the PSI class but also decreased the PPV of the criteria.

ICU admission

To evaluate the predictive characteristics of the IDSA/ATS minor criteria for ICU admission, 225 patients with orders to withhold life-sustaining treatment were excluded. In the remaining 1017 patients, the area under the ROC curve (fig 2) for the prediction of ICU admission using the IDSA/ATS minor criteria was 0.85 (95% CI 0.81 to 0.88), which was higher than that of the PSI class (0.75, 95% CI 0.71 to 0.79) and CURB-65 score (0.68, 95% CI 0.63 to 0.72; p<0.05 after Bonferroni correction for each pairwise comparison).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Receiver operating characteristic curves for admission to the intensive care unit for the Infectious Diseases Society of America/American Thoracic Society (IDSA/ATS) minor criteria, Pneumonia Severity Index (PSI) class and CURB-65 (confusion, urea nitrogen, respiratory rate, blood pressure) score. The areas under the receiver operating characteristic curves were 0.85 (95% CI 0.81 to 0.88) for the IDSA/ATS minor criteria, 0.75 (95% CI 0.71 to 0.79) for the PSI class and 0.68 (95% CI 0.63 to 0.72) for the CURB-65 score (p<0.05 after Bonferroni correction for each pairwise comparison).

A threshold of three or more IDSA/ATS minor criteria (172 patients) resulted in a sensitivity, specificity, PPV and NPV of 58.3%, 90.6%, 52.9% (91 patients) and 92.3% for ICU admission. The corresponding values for a PSI class of IV or higher were 72.4%, 65.3%, 27.4% and 92.9% respectively, while the corresponding values for a CURB-65 score of 3 or higher were 26.3%, 88.3%, 28.9% and 86.9%, respectively.

Among the 172 patients with three or more IDSA/ATS minor criteria, 91 were admitted to the ICU, of which 75 were subsequently invasively ventilated and 67 put on vasopressors. These patients were younger (63.4 vs 73.7 years, p<0.001), included fewer nursing home residents (1.1% vs 28.4%, p<0.001) and had higher PSI points (129 (32) vs 118 (34), p = 0.04) and mortality (45.1% vs 0%, p<0.001) than the remaining 81 non-ICU patients. Fifty-five patients with delayed ICU admission had similar PSI points (126 (27) vs 132 (39), p = 0.42) but higher mortality (54.5% vs 30.6%, p = 0.03) than 36 patients without delay. In a logistic regression model which incorporated this delay and the number of PSI points, delayed ICU admission was the sole independent predictor of mortality (odds ratio 3.13, 95% CI 1.23 to 7.87, p = 0.02). The median time spent in the ward before ICU transfer was 12.0 h (interquartile range 4.3–24.0) for the delayed group.

DISCUSSION

Our findings show that the 2007 IDSA/ATS minor criteria for severe CAP had greater discriminatory power, as reflected by the area under the ROC curve, for the prediction of both hospital mortality and ICU admission than the better established PSI and CURB-65 score. The IDSA/ATS minor criteria were markedly more specific than the PSI and markedly more sensitive than the CURB-65 score in these predictions.

The rationale for the introduction of the IDSA/ATS criteria for severe CAP is clear: no widely accepted definition of severe CAP exists, and previous clinical prediction rules were suboptimal predictors of both death and ICU admission.1 5 The PSI is cumbersome9^–11 and was originally designed to identify low-risk patients.2 Because it includes variables which do not reflect the current illness severity, otherwise well patients with stable co-morbidities are often assigned to PSI classes IV and V.4 The CURB-65 score is easier to use9 10 but lacks important variables such as oxygenation and is insensitive for predicting mortality.12 Consistent with previous studies that have compared the two rules,13^–17 we found that the PSI had a poor specificity of 57.9% while the CURB-65 score had a poor sensitivity of 61.2% for mortality.

In our study the IDSA/ATS minor criteria for severe CAP appeared to bridge the gap between the PSI and the CURB-65 score. Using ROC curve analysis, the optimal threshold for the prediction of hospital mortality was the fulfilment of three of nine minor criteria, just as suggested in the IDSA/ATS guidelines. This threshold resulted in a sensitivity of 81.4% and a specificity of 82.9% for hospital mortality. The area under the curve was higher for the IDSA/ATS minor criteria (0.88) than the PSI and the CURB-65 score for mortality prediction. When discordant classifications were compared, we found a considerable benefit of using the minor criteria over the CURB-65 score for predicting mortality, and only a marginal benefit of using the PSI over the minor criteria which is associated with over-labelling of many patients as severe.

The 2007 IDSA/ATS guidelines recommended that ICU admission be considered for patients who fulfilled three or more minor criteria.1 The fact that almost half (45.2%) of such patients died in our study lends support to this recommendation. However, although the minor criteria had a high NPV of 92.3% for predicting ICU admission, the PPV was lower at 52.9%, even when patients with orders to withhold life-sustaining treatment were excluded. This problem is consistently seen in other clinical prediction rules.5 12^–14 18 19 Nevertheless, our findings reveal that, just as in mortality, the IDSA/ATS minor criteria were more specific than the PSI, more sensitive than the CURB-65 score and had the greatest overall discriminatory power for predicting ICU admission.

What is the usefulness of a clinical prediction rule for severe CAP? A rule such as the IDSA/ATS criteria with a relatively high sensitivity and specificity for predicting mortality can guide clinical decisions on which patients require rapid escalation to resuscitation and management bundles for severe sepsis and/or respiratory failure,20 followed by direct admission to the ICU. Such decisions should not be taken lightly. Admission to the ICU for low-risk patients will drive up health care costs unnecessarily,5 while failure to or delay in admitting patients with truly severe CAP to the ICU may lead to increased mortality.21 The delay faced by 60% of our ICU patients who were labelled as severe CAP based on the IDSA/ATS minor criteria, but were first transferred from the emergency department to the general ward, was an independent predictor of mortality. Such a delay is not uncommon in many centres,19 and underscores the arbitrary nature of the decision for ICU admission in the absence of validated clinical prediction rules. This is especially true in situations where patients may not need emergency invasive ventilation or vasopressors (ie, the IDSA/ATS major criteria), but rather fulfil some of the minor criteria. Indeed, older studies have shown an extreme variation of ICU admission rates from 3% to 39%.22 23

In the only other published study on the IDSA/ATS criteria, Liapikou and colleagues evaluated 1924 patients after excluding 178 patients with the major criteria.6 The number of minor criteria was related to mortality, and a threshold of two or more minor criteria resulted in a sensitivity and specificity of 64% and 72%, respectively, for ICU admission. There are several differences between this study and ours. First, while we found that each minor criterion was predictive of mortality, Liapikou and colleagues used serum creatinine measurements in place of blood urea nitrogen and could not demonstrate an association between hypotension, thrombocytopenia and multilobar involvement and mortality. This is surprising as the IDSA/ATS guidelines had chosen these variables based on prior evidence.1 3 19 24 25 Second, patients with severe CAP who were not admitted to the ICU because of orders not to intubate were included in their analyses.26 This will adversely affect the predictive indices of the IDSA/ATS criteria for ICU admission. Third, while we emphasise that any delay in ICU admission when indicated may increase mortality, Liapikou and colleagues highlighted their findings that the number of minor criteria present did not predict any mortality reduction from ICU admission. These findings are difficult to interpret. Their patients were less sick than ours: 11.4% of their patients compared with 26.6% of ours fulfilled the IDSA/ATS minor criteria, with a hospital mortality of 10.5% versus 45.2%. The benefits of ICU admission are understandably diluted in less sick patients. Moreover, only 47 of their patients compared with 91 of ours who fulfilled the minor criteria were admitted to the ICU. Such small sample sizes limit the interpretation of any subgroup analyses on the impact of ICU admission on mortality.

The limitations of our study must be acknowledged. First, although we prospectively recorded all cases of CAP during the study period, the data collection was subsequently performed using medical record review. Nevertheless, we made all attempts to ensure data accuracy with standardised data forms and subsequent checks by the principal investigator (JP) for every case. Second, the hospital mortality of our cohort of 14.7% which, though similar to the mortality reported by some investigators,14 was higher than that reported by others.2 3 15 17 19 Several factors may account for this. Different forms of mortality were used in these studies. Given the logistical difficulties of ensuring close follow-up after hospital discharge, we chose hospital instead of 28–30-day mortality as the primary outcome. Since the former may be influenced by discharge practices, we followed recommended criteria for hospital discharge.7 The similar baseline characteristics of our patients to the original PSI and CURB-65 cohorts suggest that our findings may reasonably be extrapolated to most CAP populations.2 3 Even so, it is known that clinical prediction rules should be recalibrated to local settings.27 Hence, future multicentre studies should be performed to confirm the generalisability of the findings of our single-centre study. Third, several other clinical prediction rules which focus on severe CAP have emerged in recent years, including the PIRO score, the SMART-COP score and the SCAP prediction rule.12 16 28 29 Comparisons of these rules with the IDSA/ATS criteria will now be required.

In conclusion, our findings support the use of the IDSA/ATS minor criteria to predict hospital mortality and guide escalation of care with prompt ICU admission in inpatients with CAP who do not require emergency invasive mechanical ventilation or vasopressors. The IDSA/ATS minor criteria had a greater discriminatory power for mortality and ICU admission than the better established PSI and CURB-65 score, and help bridged the gap between the low specificity of the PSI and the low sensitivity of the CURB-65 score for these important clinical outcomes.