Calibration of clinical prediction rules does not just assess bias

Objectives: Calibration is often thought to assess the bias of a clinical prediction rule. In particular, if the rule is based on a linear logistic model, it is often assumed that an overestimation of all coefficients results in a calibration slope less than 1 and an underestimation in a slope larger than 1.

Study design and setting: We investigate the relation of the bias and the residual variation of clinical prediction rules with the typical behavior of calibration plots and calibration slopes, using some artificial examples.

Results: Calibration is not only sensitive to the bias of the clinical prediction rule but also to the residual variation. In some circumstances, the effects may cancel out, resulting in a misleading perfect calibration.

Conclusion: Poor calibration is a clear indication of limited usefulness of a clinical prediction rule. However, a perfect calibration should be interpreted with care as this may happen even for a biased prediction rule.