Development of the questionnaire

Six items were identified from textbooks, literature, and experts’ opinions as the most specific symptoms that characterise the respiratory condition in CAP. The items were the presence of dyspnoea (graded as presence of dyspnoea at rest, while walking around, washing and dressing, going for a walk, showering, or walking up stairs), severity of dyspnoea in general, coughing, coughing up sputum, coughing up sputum with ease, and colour of the sputum. To these respiratory symptoms we added three items to cover the general state of well being: the general state of health, fatigue, and fitness. The resulting questionnaire therefore contained nine items. Dyspnoea was rated using yes/no response options. Fatigue and fitness were measured using a visual analogue scale. All other items were rated using a Likert scale (see Appendix 1 available online at www.thoraxjnl.com/supplemental).

We tested the items for clarity and comprehensiveness in a pilot study of 18 patients and made minor changes in wording where necessary.

Patients

The psychometric properties of the questionnaire were evaluated in a subset of patients enrolled in a randomised, double blind, multicentre trial comparing two durations of treatment of CAP. Eight hospitals participated in the main study, but in this substudy we report data on four hospitals.

Eligibility was assessed according to the following criteria: temperature >38°C, clinical signs of pneumonia, a new infiltrate on the chest radiograph, and a pneumonia severity index (PSI) of <110.¹⁴ As the exclusion of afebrile patients may have excluded elderly patients, elderly patients who had evident clinical signs of pneumonia and chest radiograph abnormalities but a temperature of <38°C were also included. Patients with effective antibiotic treatment for more than 24 hours before admission or with another infection necessitating antibiotic treatment and patients with an inadequate cognitive state were excluded from the study. Consenting patients with CAP who met the inclusion criteria were treated with an intravenous β-lactam antibiotic. After 3 days patients with significant clinical improvement were randomised to receive placebo or oral amoxicillin for 5 days. All randomised patients were followed until 28 days after the beginning of antibiotic treatment. At the end of the follow up period we evaluated clinical cure, which was defined as complete recovery or lessening of pneumonia related symptoms and lack of progression of chest radiographic abnormalities. The study was approved by the medical ethical committees of the participating hospitals.

Collection of data

The questionnaire was completed at baseline and on days 3, 7, 10, 14 and 28 by seven different interviewers who were instructed in advance. The interviewers used the questionnaire in a face to face interview, except for day 14 when it was completed in a telephone interview.

At baseline the medical history was taken and a physical examination was performed by the treating physician who was also asked to indicate the presence or absence and the severity of respiratory symptoms (dyspnoea, coughing, coughing up sputum, and the colour of the sputum) using a separate standardised form. Body temperature, oxygen saturation (finger cuff), and respiratory rate were recorded. Blood was taken for measurement of white blood cell count (WBC), C-reactive protein (CRP), and erythrocyte sedimentation rate (ESR). A chest radiograph was also performed. These clinical and laboratory parameters were re-evaluated at days 3 (randomisation day), 7, 10, 14 and 28. The chest radiograph was repeated at day 10 and, if at that time complete resolution was lacking, repeated again at day 28.

Psychometric evaluation of the questionnaire

Based on data generated by the questionnaire, a single scale score was constructed. This total score (CAP score) was examined for the four psychometric properties—acceptability, reliability, validity, and responsiveness. These properties were tested using standardised procedures and instrument review criteria developed by the Scientific Advisory Committee of the Medical Outcomes Trust.¹⁵

Construction of single scale score

To combine multiple items into a single scale score, the items should be internally consistent. This was examined using three indicators of internal consistency—corrected item–total correlations, mean inter-item correlation, and Cronbach α coefficient.

Corrected item–total correlations indicate the extent to which each item relates to the construct measured by the total score. Correcting the total score by removing the item of interest prevents spuriously high values due to item overlap.¹⁶ A recommended minimum value is 0.40.¹⁷ Inter-item correlation indicates the mutual relation between individual items of a rating scale. It is recommended that the mean inter-item correlation should exceed 0.3.¹⁸ Internal consistency was further assessed using Cronbach α coefficient. The minimum recommended value is 0.7.¹⁹

The composite score was obtained using principal component analysis for categorical variables (PRINCALS). This procedure provides optimally scaled categories of the original items and subsequently calculates a composite score. This composite score was transformed to a 0–100 scale in order to allow manual assessment and calculation for practical use and is referred to as the CAP score.

Acceptability

Acceptability of the questionnaire was analysed from the number of missing data which should not exceed 5%. Floor and ceiling effects (percentage of patients with the lowest and highest scores) were also determined, indicating the potential to detect variances in the extremes. It is recommended that they do not exceed 20%.²⁰

Reliability

Reliability is the degree to which the instrument is free from random error. Internal consistency as described above is one indicator for reliability of the instrument, the other indicator for reliability being reproducibility or stability of an instrument over time. For this purpose the test-retest reliability was examined in a subset of 27 patients by determining intra-rater and inter-rater agreement between CAP scores on different occasions, to be reported as an intraclass correlation coefficient (ICC). The ICC should exceed 0.80.²¹

Validity

The validity of an instrument is defined as the degree to which the instrument measures what it is intended to measure. We examined three types of validity:

• Construct-related validity: evidence that supports a proposed interpretation of scores based on theoretical implications associated with the construct. This is assessed by obtaining evidence that a single construct is measured and that items can be combined to form a summary score. This was determined by calculating the internal consistency of the items as described above.

• Face validity: the extent to which the CAP score shows the supposed change during the clinical course of pneumonia. We assessed face validity by determining the improvement at all assessment points.

• Clinical validity: the extent to which evidence can be obtained that the scale is correlated with objective clinical measures. This was assessed by comparing the CAP score with external clinical criteria. We examined the correlation between the CAP score at baseline and the doctor’s judgement of the extent of the respiratory symptoms and the PSI on admission. We further examined over time the correlations between the CAP score and the objective clinical, laboratory and radiographic measures—that is, temperature, respiratory rate, oxygen saturation, WBC, CRP, ESR, and chest radiograph.

Responsiveness

Responsiveness is the ability of a scale to detect a clinically significant change over time under treatment. This can be determined by calculating effect sizes from admission to end of treatment. Standardised response means—mean change score divided by SD of change scores—are usually chosen for this purpose because these are the most relevant to clinical studies.²² Effect sizes for the CAP score were calculated from changes from the normal level to baseline (day 0) and from baseline to days 10 and 28 (end of follow up). Effect sizes are considered to be moderate when their absolute values exceed 0.50 and large when they exceed 0.80.²³

Patient characteristics

Baseline assessments were available for 67 patients enrolled in the CAP study between November 2000 and November 2002 (table 1⇓). At the time of the analysis the trial had recruited 45 men (66%) and 22 women (34%) of mean (SD) age 56.6 (17.8) years (range 21–96).

After 3 days five patients (7.5%) had not sufficiently improved to be randomised. After randomisation one patient was found to have another infection (endocarditis) at the time of inclusion and this patient was withdrawn from the study. Data on a total of 61 patients were therefore available for the analysis, of which 56 (84%) did not need an additive or alternative antibiotic treatment during follow up. The clinical status of five patients (7.5%) deteriorated while using the study drug and they were subsequently treated with an alternative antibiotic. These failures remained in the study for further analysis until the end of the follow up period. At day 28 all patients fulfilled the criteria for clinical cure.

The patients were interviewed by seven interviewers. It occurred often that one patient was interviewed by two or more interviewers during follow up. In total, the 67 patients were interviewed 352 times. The median number of interviews per interviewer was 36 (range 2–120).

Scale construction

One item (fatigue) showed substantial overlap with fitness and was therefore excluded. The remaining items had a high internal consistency with corrected item–total correlations ranging from 0.49 to 0.73, a mean inter-item correlation of 0.51, and a Cronbach α of 0.87. All recommended criteria for scale construction were therefore satisfied and the items were combined to form a composite score reflecting 59% of the variance in the original items. This composite score was further simplified for practical use into a CAP score. This simplified CAP score (see details in Appendices 1 and 2 available online at www.thoraxjnl.com/supplemental) showed perfect correlation with the original composite score (r = 0.99, p<0.01).

Acceptability

The number of missing data did not exceed the recommended 5%. The floor and ceiling effects of the CAP score were below the recommended maximum of 20% (table 2⇓).

Reliability

As shown in table 2⇑, the CAP score satisfied the recommended criteria for internal consistency and test-retest reproducibility.

Validity

• Construct validity: evidence of high internal consistency (table 2⇑) supported the construct validity of our CAP questionnaire.

• Face validity: the CAP score showed good improvement during follow up and also deterioration in the case of failure (fig 1A⇓). We divided the CAP score into a respiratory section—subsequently referred to as the respiratory score—which consists of the dyspnoea symptoms, cough and sputum (fig 1B⇓), and a well being section—subsequently referred to as the well being score—which consists of the items fitness and the general state of health (fig 1C⇓). The respiratory score showed an excellent improvement during follow up. At the end of the follow up period the respiratory score approached the normal level of the patients—that is, 1 month before the development of pneumonia. The well being score showed less improvement during follow up and at the end of the follow up period it still had not reached the normal level.

• Clinical validity: the CAP score at baseline was correlated with the doctors’ judgement on admission (r = 0.35, p = 0.04). Correlations of the CAP score with the objective clinical and laboratory criteria are shown in table 3⇓. Temperature, respiratory rate, oxygen saturation, WBC, CRP, and ESR were all moderately correlated with the CAP score. No association was found between the CAP score and the PSI on admission or the findings on the chest radiograph.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

 (A) The CAP score; (B) the respiratory score containing the items dyspnoea, coughing, and sputum; (C) the well being score containing the items fitness and the general state of health. Day −30 represents the pre-pneumonia level, day 0 is the day of starting antibiotic treatment, and day 28 is the end of the follow up period. The day of failure can occur between days 3 and 10; all failures (n = 5) remained in the study.

Responsiveness

Table 4⇓ shows effect sizes for the change in CAP score between the normal level and baseline (day 0) and between baseline and days 10 and 28 (end of follow up).

As expected, the CAP score showed a substantial decline between the normal level and baseline and a gradual improvement between baseline and days 10 and 28. Effect sizes were large at all assessment points. We further distinguished changes in the respiratory and the well being sections. Both showed large effect sizes. The respiratory score still showed improvement between days 10 and 28 while the well being score remained stable between these time points.