Article Text
PDF
Abstract
The inter-rater/test–retest reliability and construct validity of a palliative care needs assessment tool in interstitial lung disease (NAT:PD-ILD) were tested using NAT:PD-ILD-guided video-recorded consultations, and NAT:PD-ILD-guided consultations, and patient and carer-report outcomes (St George’s Respiratory Questionnaire (SGRQ)-ILD, Carer Strain Index (CSI)/Carer Support Needs Assessment Tool (CSNAT)). 11/16 items reached at least fair inter-rater agreement; 5 items reached at least moderate test–retest agreement. 4/6 patient constructs demonstrated agreement with SGRQ-I scores (Kendall’s tau-b, 0.24–20.36; P<0.05). 4/7 carer constructs agreed with the CSI/CSNAT items (kappa, 0.23–20.53). The NAT:PD-ILD is reliable and valid. Clinical effectiveness and implementation are to be evaluated.
- interstitial fibrosis
- palliative care
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Introduction
People with interstitial lung disease (ILD) are symptomatic,1 and have limited disease-modifying treatment options, poor prognosis and poor quality of life.2 Identification and management of patients’ and carers’ palliative care needs are rare despite policy directives promoting palliative care,3 and availability of palliative interventions.4
The validated needs assessment tool in cancer (NAT:PD-C) helps clinicians identify and address palliative needs in daily practice.5 We adapted, validated and tested the reliability of the NAT:PD-C for patients with ILD (NAT:PD-ILD) and explored implementation implications in practice. This single page guide prompts clinicians to assess holistic needs (priority prompts for specialised palliative care input, patients’ well-being, informal carers’ needs and information needs), triage ongoing care (‘directly managed’, ‘refer to other team member’, ‘refer to specialist palliative care’) and also acts as a referral form.
Methods
Summary design
The initial adaptation, face and content validation6 and implementation work7 are reported elsewhere. This study tested the psychometric properties of (1) inter-rater and test–retest reliability, and (2) construct validity.
Clinicians (doctors, nurses, physiotherapists), patients with ILD and their family carers were recruited from four ILD tertiary referral clinics.
Inter-rater and test–retest reliability
Video recordings were made of 10 patient-clinician consultations (range of disease severity and carer present/absent, with clinicians using the NAT:PD-ILD to guide assessment). Clinicians were trained to use the NAT:PD-ILD and rated at least one video consultation (group viewing or individual viewing via secure online service). Clinicians were asked to rerate the same video at least 2 weeks later. Weighted Fleiss’ kappa with quadratic weights was calculated for the ratings on 10 videos (kappa <0=poor agreement, 0–0.20=slight, 0.21–0.40=fair, 0.41–0.60=moderate, 0.61–0.80=substantial, 0.81–1=almost perfect agreement).8 Data simulations indicated that we required 60 paired assessment ratings to detect at least ‘substantial’ inter-rater agreement (kappa >0.6) for the item 4 (‘Is the patient experiencing unresolved physical symptoms?’) with 80% power.
Construct validation
Trained clinicians conducted a NAT:PD-ILD-guided clinic consultation. Patients completed the St George’s Respiratory Questionnaire (SGRQ-I) and carers completed the Carer Strain Index (CSI) and Carer Support Needs Assessment Tool (CSNAT). Kendall’s tau-b correlation coefficient was calculated to determine the correlation between the NAT:PD-ILD patient well-being items and a subset of SGRQ-I similar constructs identified a priori. The prevalence and bias-adjusted kappa (PABAK), Cohen’s kappa and observed percentage agreement were used to assess agreement between the NAT:PD-ILD carer items and appropriate CSI and CSNAT constructs identified a priori. Data simulations indicated that a sample size of 65 patients would allow estimation of the kappa statistic for agreement such that the CI would not extend beyond the neighbouring category.
All analyses were conducted using Stata V.13 (StataCorp, Stata Statistical Software: Release 13. College Station, TX: StataCorp, 2013).
Results
Reliability
Fifty-three clinicians (32 doctors, 18 physiotherapists, 2 clinical physiologists, 1 nurse) participated in 64 first views across 10 videos, with 21 test–retest observations on four videos.
Inter-rater reliability
Eleven (69%) NAT:PD-ILD items reached at least fair agreement (weighted kappa >0.2).
Test–retest reliability
Five items exhibited at least moderate agreement (weighted kappa >0.4) (table 1).
Inter-rater reliability test and test–retest results, weighted kappa
Construct validation
Nine clinicians (six doctors, three nurses) and 68 patients (mean age 66, SD 10.3; 62% men; 45 with a carer (28 participated); 35% oxygen therapy; 56% interstital pulmonary fibrosis ; 80% Medical Research Council breathlessness 3–5) were recruited (online supplementary eTable 1). The SGRQ-I mean summary score was 62.5 (SD 20.9): symptom component 67.4 (26.1); activities component 82.1 (23.2); impact component 51.2 (24.3). Scores for the carer comparator outcome measures are shown in online supplementary eTable 2. Agreement between NAT:PD-ILD concerns and comparator outcomes is shown in table 2. Items 2 (unresolved psychological symptoms/loss quality of life), 3 (problems with daily living activities), 5 (work, financial or legal concerns) and 6 (health beliefs, cultural or social factors) of the NAT:PD-ILD were significantly positively correlated with their comparator SGRQ-I scores (r range 0.24–0.36, P<0.05). PABAK values comparing the NAT:PD-ILD items with CSI/CSNAT items were mostly positive (0.04–0.57, minimum 52% agreement); however, items 11 and 13 have negative PABAK values (interpersonal relationships and grief topics).
Supplementary file 1
Relationship between NAT:PD-ILD items and SGRQ-ILD comparator items, and between the NAT:PD-ILD items relating to the carer and appropriate CSI and CSNAT items
Discussion
Items within the NAT:PD-ILD demonstrated acceptable inter-rater reliability and construct validation given the broad constructs assessed and the breadth of clinical experience. The constructs of patient-reported quality of life (SGRQ-I) and assessment of need are related but different, therefore the relatively small number of items rated as moderate or strong is unsurprising. Similarly, many carer-related items both on NAT:PD-ILD and within CSI and CSNAT capture areas of concern that overlap, but are not directly comparable. The NAT-PD-C, with similar psychometric properties, resulted in reduced patient and carer needs when applied in practice; the key factor in any clinical tool.5
The NAT:PD-ILD is best seen as a communication and decision tool where action is thereby triggered if more in-depth exploration is needed, rather than an outcome measurement. The challenges of a clinical assessment and diagnosis are recognised as an inexact science with variation between clinicians.9 10 Study clinicians had a wide range of clinical experience to increase generalisability. Some NAT:PD-ILD items with poor agreement (inter-rater and construct) are consistent with the clinician participants’ expressed lack of confidence, for example, spiritual and existential concerns and may rather reflect an educational need.7
The clinician participants rated the videos after 10–15 min of training, inter-rater reliability is likely to improve with practice. Clinicians using the tool in daily practice will gain more experience as they use the tool.
Although we reached our target sample size for the construct validity analysis, we have insufficient sample for the carer comparisons. Also, our sample was convenience, not consecutive, potentially affecting representativeness.
The tool is yet to be tested in a clinical trial to evaluate its use by clinicians in terms of impact on patient and carer experience.
Conclusions
The adapted NAT:PD-ILD has adequate reliability and construct validation. Effectiveness in clinical practice, and optimum implementation are yet to be evaluated and identified.
Acknowledgments
We would like to thank the patients, carers and clinicians who took part in this study for their time and insights. The tool is available through the NICE.
Footnotes
Contributors MJJ and DC conceived the study and design. MJJ, DCC, MB, JMB, JR, JY, GG and TP. MJJ, DCC and JY wrote the protocol. JR, AJ and JA led on reliability testing. JY led on construct. GG provided carer assessment expertise. JMB and CF conducted the analysis. All authors contributed to interpretation. MJJ wrote the first draft and all authors contributed to revisions and the final draft.
Funding This work was supported bythe Marie Curie Research Grants Scheme, grant (MCCC-RP-14-A16976)
Competing interests None declared.
Ethics approval National Research Ethics Service North East-Tyne & Wear South (14/NE/0127) and institutional approval was obtained.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement The corresponding author can be contacted regarding use of anonymised data.