Table 2

Recalibrating and model revision methods considered for logistic regression models for a prognostic index (PI) with n predictors (ie, methods 1–5) and extension methods considering k new predictors; not included in the original model (ie, methods 6–8)14 30

NoUpdating methodDescriptionInterpretation
1No adjustmentFit the original model in exactly the same way it was constructed in the derivation data set. No parameters are estimated.This method makes no revisions to the model but evaluates the performance of the original scores when applied to an independent cohort.
Recalibration2 α Correct the ‘calibration in the large’ by updating the intercept α .If a lack of calibration is noted during the external validation process (ie, Embedded Image , Embedded Image ), an initial ‘re-calibration step’ is performed, in which α more or less Embedded Image can be updated without changing the PI itself. Updating only the intercept (method 2) addresses problems arising from overly high or low predicted probabilities in the validation data set through only a change in the intercept (Embedded Image or Embedded Image ).
Beyond updating the intercept, in method 3, the calibration slope can also be updated, in which all original regression coefficients can be multiplied by a constant. This addresses the problem of coefficients in the original model being too large: Embedded Image (or too small: Embedded Image ).
3 Embedded Image Re-estimate both calibration slope Embedded Image and intercept α . This step is called ‘logistic calibration’.
Model revision4 Embedded Image Building on method 3, each covariate from the original model is included in a forward stepwise procedure using Akaike information criterion (AIC), including only those that lower (improve) the AIC. Their coefficients Embedded Image (for i in 1 to n) are then estimated. Of note, their updated regression coefficients Embedded Image are then: Embedded Image .In the ‘re-calibration step’ above, the PI is revised globally and equally for all regression coefficients estimated in the derivation data set. In method 4, covariates are included one by one if they improve the relative goodness of the fit of the model (ie, lower the AIC), which identifies covariates having a significantly different effect in the validation versus derivation data sets and then to recalibrate the PI by increasing (Embedded Image ) or decreasing (Embedded Image ) their weights in the model. Notably, in clinical practice, this procedure can account for big changes in covariates over time by recalibrating the model, for example.
In method 5, all coefficients are re-estimated in the validation data set. This step also highlights covariates that have a different influence in derivation and validation data sets by comparing α and Embedded Image with the original ones. With this method, the original PI is not preserved.
5 Embedded Image Re-estimate coefficients of all the covariates: Embedded Image included in the original model and the intercept α in the validation data set without any selection.
Model extension6 Embedded Image Re-estimate the intercept α , the slope on the PI: Embedded Image and the coefficients of the covariates included in the model: Embedded Image , as well as coefficients of new covariates: Embedded Image in a stepwise manner (as in method 4).In the ‘model revision’ step, only the original covariates were used. In the ‘model extension’ step, new covariates not originally included in the model are added.
Method 6 is similar to method 4 but adds new covariates only if they increase the relative goodness of the fit of the model (ie, lower AIC). For example, in clinical practice, this process enables the incorporation of new covariates into the model should they become available.
Method 7 approximates method 5 but, as in method 6, adds new covariates only if they decrease the AIC of the model.
Finally, in method 8, we create a completely new model without considering the original one by estimating all the regression coefficients (for covariates originally included as well as new covariates).
7 Embedded Image Fit the exact same model as in the original one; re-estimating all coefficients of the original covariates: Embedded Image , without selection process, and adding new covariates Embedded Image only if they improve model performance in the validation data set.
8 Embedded Image Fit a model which allows the estimation of all the n+k coefficients: Embedded Image and the intercept α without selection process.