Article Text

Download PDFPDF

Validation of the 4C prediction models to inform care for patients with COVID-19: final steps towards clinical application
  1. Milo A Puhan1,
  2. Mohsen Sadatsafavi2
  1. 1 Department of Epidemiology, Epidemiology, Biostatistics and Prevention Institute, University of Zurich, Zurich, Switzerland
  2. 2 Department of Respiratory Evaluation Sciences Program, Collaboration for Outcomes Research and Evaluation, Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, British Columbia, Canada
  1. Correspondence to Dr Milo A Puhan, Epidemiology, University of Zurich, Zurich, ZH, Switzerland; miloalan.puhan{at}

Statistics from

Dr Knight and colleagues report about a large-scale external validation of the 4C (ISARIC (International Severe Acute Respiratory and emerging Infections Consortium) Coronavirus Clinical Characterisation Consortium) prediction models to predict in-hospital outcomes in patients admitted to a hospital because of COVID-19.1 The models use commonly available information to predict the probability of in-hospital deterioration and mortality ( While the models require quite a few variables, they do allow for missingness of some predictors, which facilitates their use in practice. The ISARIC4C consortium was able to compile data from 76 588 patients from 306 hospitals and nine NHS regions of England, Scotland and Wales. The high frequency of outcome events (37.4% deteriorated and 25.1% died) allowed for very precise estimates of discrimination and calibration in the entire study population, but also in subgroups. Another strength is the temporal validation. The validation showed good performance of both models, similar to the predictive performance in the development cohort that consisted of patients hospitalised during the first wave of the pandemic, where treatments and outcomes were inferior compared with the second wave. This provides some reassurance that the models can be used in the (near) future unless a disruptive change in the management of COVID-19 occurs.

Although discrimination and calibration are important metrics, they do not tell much about clinical utility.2 A clinical prediction model should only be used for risk-stratified prevention or treatment if using the model leads to better health outcomes than not using the model. Ultimately, the usefulness of a clinical prediction model should be evaluated in a cluster-randomised trial where hospitals or wards are randomly assigned to the risk-stratified approach or usual care. However, in the early phase of risk model development, a decision curve analysis,3 as discussed recently in detail elsewhere,4 is a useful first step to assess the potential net benefit of using a prediction model across the risk spectrum. The ISARIC4C investigators went beyond discrimination and calibration and performed such an analysis, which clearly showed the potential net clinical benefit of using the models compared with no risk stratification.

In addition, the authors referred to specific strategies to care for patients with COVID-19 admitted to a hospital, based on predictions from the 4C models: if the risk of in-hospital deterioration or death is low, the patient could be discharged to a hospital at home or home care, and if the risk is moderate, monitoring and treatment on a ward may be the best strategy. Patients at high risk of in-hospital deterioration or death may have to be treated more aggressively and be closely monitored for the need of intensive care unit admission. This strategy can be particularly relevant due to the significant pressure the pandemic has put on the inpatient capacity, and the benefit of such risk stratification can reach to patients with other conditions (eg, those on the wait list for an elective surgery).

Such risk stratification, however, invokes the question as to how to decide on the risk threshold for labelling a patient as low risk or high risk. To derive such a risk threshold, the balance of benefits and harms and sometimes also costs of alternative management decisions needs to be established, as, for example, has been done on the field of cardiovascular medicine or, more recently, for inhaled corticosteroids in patients with COPD.5 6 Once risk thresholds are defined, clinicians can use the validated 4C models to inform their decision whether to discharge patients from the hospital, keep them on a ward or provide as much care and monitoring as possible to minimise the risk of deterioration and death due to COVID-19.

The validation study by the ISARIC4C consortium represents a true advance for validating prediction models in respiratory medicine since it provides a large-scale external validation (including temporal validation and validation across different hospitals), a decision curve analysis as preliminary evidence for the net benefit of using the 4C models, and outlines a clear clinical decision-making context. This work also serves as call for the last mile to be walked, where specific risk thresholds need to be defined and the impact of using prediction models and a risk-stratified approach on health outcomes is tested through clinical trials.7

Ethics statements

Patient consent for publication



  • Contributors MP drafted the editorial, and MS reviewed and revised it. Both authors finalised the editorial.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Provenance and peer review Commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles

  • Respiratory research
    Stephen R Knight Rishi K Gupta Antonia Ho Riinu Pius Iain Buchan Gail Carson Thomas M Drake Jake Dunning Cameron J Fairfield Carrol Gamble Christopher A Green Sophie Halpin Hayley E Hardwick Karl A Holden Peter W Horby Clare Jackson Kenneth A Mclean Laura Merson Jonathan S Nguyen-Van-Tam Lisa Norman Piero L Olliaro Mark G Pritchard Clark D Russell Catherine A Shaw Aziz Sheikh Tom Solomon Cathie Sudlow Olivia V Swann Lance C W Turtle Peter J M Openshaw J Kenneth Baillie Annemarie Docherty Malcolm G Semple Mahdad Noursadeghi Ewen M Harrison J Kenneth Baillie Malcolm G Semple Peter JM Openshaw Gail Carson Beatrice Alex Benjamin Bach Wendy S Barclay Debby Bogaert Meera Chand Graham S Cooke Annemarie B Docherty Jake Dunning Ana da Silva Filipe Tom Fletcher Christopher A Green Ewen M Harrison Julian A Hiscox Antonia YingWai Ho Peter W Horby Samreen Ijaz Saye Khoo Paul Klenerman Andrew Law Wei Shen Lim J Mentzer Alexander Laura Merson Alison M Meynert Mahdad Noursadeghi Shona C Moore Massimo Palmarini William A Paxton Georgios Pollakis Nicholas Price Andrew Rambaut David L Robertson Clark D Russell Vanessa Sancho-Shimizu Janet T Scott Louise Sigfrid Tom Solomon Shiranee Sriskandan David Stuart Charlotte Summers Richard S Tedder Emma C Thomson Ryan S Thwaites Lance CW Turtle Maria Zambon Hayley Hardwick Chloe Donohue Jane Ewins Wilna Oosthuyzen Fiona Griffiths Lisa Norman Riinu Pius Tom M Drake Cameron J Fairfield Stephen R Knight Kenneth A Mclean Derek Murphy Catherine A Shaw Jo Dalton Michelle Girvan Egle Saviciute Stephanie Roberts Janet Harrison Laura Marsh Marie Connor Sophie Halpin Clare Jackson Carrol Gamble Gary Leeming Andrew Law Ross Hendry William Greenhalf Victoria Shaw Sarah McDonald Katie A Ahmed Jane A Armstrong Milton Ashworth Innocent G Asiimwe Siddharth Bakshi Samantha L Barlow Laura Booth Benjamin Brennan Katie Bullock Benjamin WA Catterall Jordan J Clark Emily A Clarke Sarah Cole Louise Cooper Helen Cox Christopher Davis Oslem Dincarslan Chris Dunn Philip Dyer Angela Elliott Anthony Evans Lewis WS Fisher Terry Foster Isabel Garcia-Dorival Willliam Greenhalf Philip Gunning Catherine Hartley Antonia Ho Rebecca L Jensen Christopher B Jones Trevor R Jones Shadia Khandaker Katharine King Robyn T Kiy Chrysa Koukorava Annette Lake Suzannah Lant Diane Latawiec L Lavelle-Langham Daniella Lefteri Lauren Lett Lucia A Livoti Maria Mancini Sarah McDonald Laurence McEvoy John McLauchlan Soeren Metelmann Nahida S Miah Joanna Middleton Joyce Mitchell Shona C Moore Ellen G Murphy Rebekah Penrice-Randal Jack Pilgrim Tessa Prince Will Reynolds P Matthew Ridley Debby Sales Victoria E Shaw Rebecca K Shears Benjamin Small Krishanthi S Subramaniam Agnieska Szemiel Aislynn Taggart Jolanta Tanianis-Hughes Jordan Thomas Erwan Trochu Libby van Tonder Eve Wilcock J Eunice Zhang Kayode Adeniji Daniel Agranoff Ken Agwuh Dhiraj Ail Ana Alegria Brian Angus Abdul Ashish Dougal Atkinson Shahedal Bari Gavin Barlow Stella Barnass Nicholas Barrett Christopher Bassford David Baxter Michael Beadsworth Jolanta Bernatoniene John Berridge Nicola Best Pieter Bothma David Brealey Robin Brittain-Long Naomi Bulteel Tom Burden Andrew Burtenshaw Vikki Caruth David Chadwick Duncan Chambler Nigel Chee Jenny Child Srikanth Chukkambotla Tom Clark Paul Collini Catherine Cosgrove Jason Cupitt Maria-Teresa Cutino-Moguel Paul Dark Chris Dawson Samir Dervisevic Phil Donnison Sam Douthwaite Ingrid DuRand Ahilanadan Dushianthan Tristan Dyer Cariad Evans Chi Eziefula Chrisopher Fegan Adam Finn Duncan Fullerton Sanjeev Garg Sanjeev Garg Atul Garg Jo Godden Arthur Goldsmith Clive Graham Elaine Hardy Stuart Hartshorn Daniel Harvey Peter Havalda Daniel B Hawcutt Maria Hobrok Luke Hodgson Anita Holme Anil Hormis Michael Jacobs Susan Jain Paul Jennings Agilan Kaliappan Vidya Kasipandian Stephen Kegg Michael Kelsey Jason Kendall Caroline Kerrison Ian Kerslake Oliver Koch Gouri Koduri George Koshy Shondipon Laha Susan Larkin Tamas Leiner Patrick Lillie James Limb Vanessa Linnett Jeff Little Michael MacMahon Emily MacNaughton Ravish Mankregod Huw Masson Elijah Matovu Katherine McCullough Ruth McEwen Manjula Meda Gary Mills Jane Minton Mariyam Mirfenderesky Kavya Mohandas Quen Mok James Moon Elinoor Moore Patrick Morgan Craig Morris Katherine Mortimore Samuel Moses Mbiye Mpenge Rohinton Mulla Michael Murphy Megan Nagel Thapas Nagarajan Mark Nelson Igor Otahal Mark Pais Selva Panchatsharam Hassan Paraiso Brij Patel Justin Pepperell Mark Peters Mandeep Phull Stefania Pintus Jagtur Singh Pooni Frank Post David Price Rachel Prout Nikolas Rae Henrik Reschreiter Tim Reynolds Neil Richardson Mark Roberts Devender Roberts Alistair Rose Guy Rousseau Brendan Ryan Taranprit Saluja Aarti Shah Prad Shanmuga Anil Sharma Anna Shawcross Jeremy Sizer Richard Smith Catherine Snelson Nick Spittle Nikki Staines Tom Stambach Richard Stewart Pradeep Subudhi Tamas Szakmany Kate Tatham Jo Thomas Chris Thompson Robert Thompson Ascanio Tridente Darell Tupper-Carey Mary Twagira Andrew Ustianowski Nick Vallotton Lisa Vincent-Smith Shico Visuvanathan Alan Vuylsteke Sam Waddy Rachel Wake Andrew Walden Ingeborg Welters Tony Whitehouse Paul Whittaker Ashley Whittington Meme Wijesinghe Martin Williams Lawrence Wilson Sarah Wilson Stephen Winchester Martin Wiselka Adam Wolverson Daniel G Wooton Andrew Workman Bryan Yates Peter Young