• COVID-19

Dr Knight and colleagues report about a large-scale external validation of the 4C (ISARIC (International Severe Acute Respiratory and emerging Infections Consortium) Coronavirus Clinical Characterisation Consortium) prediction models to predict in-hospital outcomes in patients admitted to a hospital because of COVID-19.1 The models use commonly available information to predict the probability of in-hospital deterioration and mortality (https://isaric4c.net/risk/). While the models require quite a few variables, they do allow for missingness of some predictors, which facilitates their use in practice. The ISARIC4C consortium was able to compile data from 76 588 patients from 306 hospitals and nine NHS regions of England, Scotland and Wales. The high frequency of outcome events (37.4% deteriorated and 25.1% died) allowed for very precise estimates of discrimination and calibration in the entire study population, but also in subgroups. Another strength is the temporal validation. The validation showed good performance of both models, similar to the predictive performance in the development cohort that consisted of patients hospitalised during the first wave of the pandemic, where treatments and outcomes were inferior compared with the second wave. This provides some reassurance that the models can be used in the (near) future unless a disruptive change in the management of COVID-19 occurs.

Although discrimination and calibration are important metrics, they do not tell much about clinical utility.2 A clinical prediction model should only be used for risk-stratified prevention or treatment if using the model leads to better health outcomes than not using the model. Ultimately, the usefulness of a clinical prediction model should be evaluated in a cluster-randomised trial where hospitals or wards are randomly assigned to the risk-stratified approach or usual care. However, in the early phase of risk model development, a decision curve analysis,3 as discussed recently in detail elsewhere,4 is a useful first step to assess the potential net benefit of using a prediction model across the risk spectrum. The ISARIC4C investigators went beyond discrimination and calibration and performed such an analysis, which clearly showed the potential net clinical benefit of using the models compared with no risk stratification.

In addition, the authors referred to specific strategies to care for patients with COVID-19 admitted to a hospital, based on predictions from the 4C models: if the risk of in-hospital deterioration or death is low, the patient could be discharged to a hospital at home or home care, and if the risk is moderate, monitoring and treatment on a ward may be the best strategy. Patients at high risk of in-hospital deterioration or death may have to be treated more aggressively and be closely monitored for the need of intensive care unit admission. This strategy can be particularly relevant due to the significant pressure the pandemic has put on the inpatient capacity, and the benefit of such risk stratification can reach to patients with other conditions (eg, those on the wait list for an elective surgery).

Such risk stratification, however, invokes the question as to how to decide on the risk threshold for labelling a patient as low risk or high risk. To derive such a risk threshold, the balance of benefits and harms and sometimes also costs of alternative management decisions needs to be established, as, for example, has been done on the field of cardiovascular medicine or, more recently, for inhaled corticosteroids in patients with COPD.5 6 Once risk thresholds are defined, clinicians can use the validated 4C models to inform their decision whether to discharge patients from the hospital, keep them on a ward or provide as much care and monitoring as possible to minimise the risk of deterioration and death due to COVID-19.

The validation study by the ISARIC4C consortium represents a true advance for validating prediction models in respiratory medicine since it provides a large-scale external validation (including temporal validation and validation across different hospitals), a decision curve analysis as preliminary evidence for the net benefit of using the 4C models, and outlines a clear clinical decision-making context. This work also serves as call for the last mile to be walked, where specific risk thresholds need to be defined and the impact of using prediction models and a risk-stratified approach on health outcomes is tested through clinical trials.7