Abstract
The use of modelling in economic evaluation is widespread, and it most often involves synthesising data from a number of sources. However, even when economic evaluations are conducted alongside clinical trials, some form of modelling is usually essential. The aim of this article is to review the handling of uncertainty in the cost-effectiveness results that are generated by the use of decision-analytic-type modelling. The modelling process is split into a number of stages: (i) a set of methods to be employed in a study are defined, which should include a ‘reference case’ of agreed methods to enhance the comparability of results; (ii) the clinical and demographic characteristics of the patients the model relates to should be specified as carefully as in any experimental study; and (iii) the data requirements of the model should be estimated using the principles of Bayesian statistics, such that prior distributions are specified for unknown model parameters. Monte Carlo simulation can then be employed to sample from these prior distributions to obtain a distribution of the cost effectiveness of the intervention. Such probabilistic analyses are related to parameter uncertainty. In addition, modelling uncertainty is likely to add a further layer of uncertainty to the results of particular analyses.
Similar content being viewed by others
Notes
1 Note that in this article, cost effectiveness is used generically to describe results of economic evaluations that are commonly compared in league tables (i.e. those reporting results in terms of cost per life-year and cost per qualityadjusted life-year. Studies reporting results in so-called ‘natural’ units, such as cases detected or ‘symptom-free days avoided’, are also known as cost-effectiveness studies. However, since these natural units cannot be compared between different disease areas, the usefulness of these studies for allocating resources in the health sector is limited.
For UK analysts, this issue has become all the more important recently because of the updated HM Treasury guidelines,[16]which the Department of Health has interpreted as recommending differential discounting for health outcomes in cost-effectiveness analysis (by removing the component of discounting assumed by the Treasury to relate to the combination of annual growth of income and the marginal utility of income).
These costs are often referred to as indirect costs; however, this term is avoided in this article since Drummond and colleagues24 have argued that this terminology can cause confusion through the use of the same term in accountancy to mean overhead costs.
Another margin that might be identified relates to ‘returns to scale’. Standard economic textbooks emphasise falling costs in the presence of returns to scale. However, in terms of the exposition given here, the issue of returns to scale is considered to be a technical efficiency problem. Similarly, where capital equipment results in a discontinuous cost function, it is assumed that the location of such equipment is organised to maximise throughput and therefore minimise overall cost.
In any case, these rates are taken from large national studies and their variation can be assumed to be trivial compared with the other model parameters.
Note that the same individual simulation approach could be used for decision tree-type models, since a given individual can only pass down 1 branch of the tree at a given chance node.
One might be tempted to calculate the individual-level cost-effectiveness ratios and average across the patients. Stinnett and Paltiel36 examined the implications of calculating ‘mean ratios’ versus the ‘ratio of means’ and demonstrated that the analysis of ‘mean ratios’ is inappropriate.
Of course, at the design stage of a clinical trial the sample size for an evaluation is under the control of the design team, and is subject to cost and other logistical constraints. However, it is clear that in a Monte Carlo simulation situation there are no such constraints and the effective sample size (the number of simulations) can be set at the analysis stage.
Note that this causes problems for the dedicated decision analysis software used to generate the individual simulation results.49 Although it is possible to run only a first-order analysis, the second-order uncertainty can only be included as an adjunct to the first-order uncertainty. This problem should be rectified in future releases of the software.
Uncertainty intervals are employed here as a generic term rather than employing the frequentist ‘confidence interval’ or the Bayesian equivalent ‘credible interval’.
The intervals presented in the illustrative example are valid as all the simulated resultswere in the positive quadrant of the cost-effectiveness plane.
It is worth noting that this interpretation of cost-effectiveness acceptability curves as showing the probability that an intervention is cost effective (probability of the hypothesis given the data) necessarily requires a Bayesian interpretation.57
References
Posnett J, Jan S. Indirect cost in economic evaluation: the opportunity cost of unpaid inputs. Health Econ 1996; 5 (1): 13–23
Buxton MJ, Drummond MF, van Hout BA, et al. Modelling in economic evaluation: an unavoidable fact of life. Health Econ 1997; 6: 217–27
Association of British Pharmaceutical Industries. Pharmaceutical industry and Department of Health agree guidelines for the economic analysis of medicines [press release]. London: Association of British Pharmaceutical Industries, 1994
Australian Commonwealth Department of Human Services and Health. Guidelines for the pharmaceutical industry on preparation of submissions to the Pharmaceutical Benefits Advisory Committee: including major submissions involving economic analysis. Canberra: Australian Government Publishing Service, 1995
Drummond MF, Jefferson TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. BMJ 1996; 313: 275–83
Canadian Coordinating Office for Health Technology Assessment. Guidelines for the economic evaluation of pharmaceuticals: Canada. 2nd ed. Ottawa: Canadian Coordinating Office for Health Technology Assessment (CCOHTA), 1997
Manning WG, Fryback DG, Weinstein MC. Reflecting uncertainty in cost-effectiveness analysis. In: Gold MR, Siegel JE, Russell LB, et al., editors. Cost-effectiveness in health and medicine. New York (NY): Oxford University Press, 1996: 247–75
Drummond M, Brandt A, Luce B, et al. Standardizing methodologies for economic evaluation in health care: practice, problems, and potential. Int J Technol Assess Health Care 1993; 9 (1): 26–36
Lipscomb J. Time preference for health in cost-effectiveness analysis. Med Care 1989; 27 (3 Suppl.): S233–53
Parsonage M, Neuburger H. Discounting and health benefits. Health Econ 1992; 1 (1): 71–6
Cairns J. Discounting and health benefits: another perspective [comment]. Health Econ 1992; 1 (1): 76–9
Coyle D, Tolley K. Discounting of health benefits in the pharmacoeconomic analysis of drug therapies. Pharmacoeconomics 1992; 2 (2): 153–62
Katz DA, Welch HG. Discounting in cost-effectiveness analysis of healthcare programmes. Pharmacoeconomics 1993; 3 (4): 276–85
Lipscomb J. The proper role for discounting: search in progress. Med Care 1996; 34 (12 Suppl.): DS119-23
Lipscomb J, Weinstein M, Torrance, et al. Time preference. In: Gold MR, Siegel JE, Russell LB, et al., editors. Cost-effectiveness in health and medicine. New York (NY): Oxford University Press, 1996: 214–46
HM Treasury. Appraisal and evaluation in central government. London: H.M. Stationery Office, 1997
Donaldson C. Willingness to pay for publicly-provided goods: a possible measure of benefit. J Health Econ 1990; 9: 103–18
Llewellyn TH, Sutherland HJ, Tibshirani R, et al. The measurement of patients’ values in medicine. Med Decis Making 1982; 2 (4): 449–62
Mehrez A, Gafni A. Quality-adjusted life years, utility theory, and healthy-years equivalents. Med Decis Making 1989; 9: 142–9
Torrance GW. Measurement of health state utilities for economic appraisal: a review. J Health Econ 1986; 5: 1–30
Conroy RM, O’Brien E, O’Malley K, et al. Measurement error in the Hawksley random zero sphygmomanometer: what damage has been done and what can we learn? BMJ 1993; 306 (6888): 1319–22
Gold MR, Siegel JE, Russell LB, et al. Cost-effectiveness in health and medicine. New York (NY): Oxford University Press, 1996
Meltzer David, Accounting for future costs in medical cost-effectiveness analysis, J. Health Econ 1997; 16 (1): 33–64
Drummond MF, O’Brien B, Stoddart GL, et al. Methods for the economic evaluation of health care programmes. 2nd ed. Oxford: Oxford University Press, 1997
Koopmanschap MA, Rutten FFH. Indirect costs: the consequence of production loss or increased costs of production. Med Care 1996; 34 (12 Suppl.): DS59–68
Koopmanschap MA, Rutten FFH, Van Ineveld BM, et al. The friction cost method for measuring indirect costs of disease. J Health Econ 1995; 14: 171–89
Russell LB. Is prevention better than cure? Washington, DC: The Brookings Institution, 1986
Briggs AH, Gray AM. Handling uncertainty when performing economic evaluation of health care interventions. Health Technol Assess 1999; 3 (2): 1–134
Anderson MH, Camm AJ. Implications for present and future applications of the implantable cardioverter-defibrillator resulting from the use of a simple model of cost efficacy. Br Heart J 1993; 69 (1): 83–92
Pharoah PD, Hollingworth W. Cost effectiveness of lowering cholesterol concentration with statins in patients with and without pre-existing coronary heart disease: life table method applied to health authority population. BMJ 1996; 312 (7044): 1443–8
Eddy DM. Screening for cervical cancer. Ann Intern Med 1990; 113 (3): 214–26
Johannesson M, Weinstein MC. On the decision rules of cost-effectiveness analysis. J Health Econ 1993; 12: 459–67
Weinstein M, Fineberg HV. Clinical decision analysis. Philadelphia (PA): WB Saunders Company, 1980
Williams A. The economics of coronary artery bypass grafting. BMJ 1985; 291: 326–9
Pauker SG, Kassirer J. Decision analysis. N Engl J Med 1987; 316 (5): 250–8
Stinnett AA, Paltiel AD. Estimating CE ratios under second order uncertainty: the mean ratio versus the ratio of means. Med Decis Making 1997; 17 (4): 483–9
Eddy DM, Hasselblad V, Shachter R. A Bayesian method for synthesizing evidence: the Confidence Profile Method. Int J Technol Assess Health Care 1990; 6 (1): 31–55
Eddy DM, Hasselblad V, Shachter R. An introduction to a Bayesian method for meta-analysis: the confidence profile method. Med Decis Making 1990; 10 (1): 15–23
Zhou XH, Melfi CA, Hui SL. Methods for comparison of cost data. Ann Intern Med 1997; 127 (8 Pt 2): 752–6
Briggs A, Gray A. The distribution of health care costs and their statistical analysis for economic evaluation. J Health Serv Res Policy 1998; 3: 233–45
Gelman A, Carlin J, Stern H, et al. Bayesian data analysis. London: Chapman & Hall, 1995
Wennberg J, Gittelsohn A. Variations in medical care in small areas. Sci Am 1982; 4: 120–4
Andersen TF, Mooney G. The challenge of medical practice variations. Basingstoke: Macmillan, 1990
Cleary PD, Greenfield S, Mulley AG, et al. Variations in length of stay and outcomes for six medical and surgical conditions in Massachusetts and California. JAMA 1991; 266 (1): 73–9
Draper D. Assessment and propagation of model uncertainty. J R Stat Soc Br 1995; 57 (1): 45–97
Briggs A, Sculpher M. An introduction to Markov models for economic evaluation. Pharmacoeconomics 1998; 13 (4): 397–409
Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Med Decis Making 1993; 13: 322–38
SMLTREE [computer program]. 2.9. Hollenberg J, 1989
Decision Analysis by TreeAge (DATA) [computer program]. Tree Age Software Inc. v3.5. Williamstown (MA): Tree Age, 1998
Briggs AH, Fenn P. Confidence intervals or surfaces? Uncertainty on the cost-effectiveness plane. Health Econ 1998: 7 (8): 723–40
Sharples LD, Briggs A, Caine N, et al. A model for analyzing the cost of main clinical events after cardiac transplantation. Transplantation 1996; 62 (5): 615–21
Hunink MG, Bult JR, de Vries J, et al. Uncertainty in decision models analyzing cost-effectiveness: the joint distribution of incremental costs and effectiveness evaluated with a nonparametric bootstrap method. Med Decis Making 1998; 18 (3): 337– 46
Pratt JW, Raiffa H, Schlaifer R. Introduction to statistical decision theory. Cambridge (MA): MIT Press, 1995
Stinnett AA, Mullahy J. The negative side of cost-effectiveness analysis [letter]. JAMA 1997; 277 (24): 1931–2
Stinnett AA, Mullahy J. Net health benefits: a new framework for the analysis of uncertainty in cost-effectiveness analysis. Med Decis Making 1998; 18 (2 Suppl.): S65–80
van Hout BA, Al MJ, Gordon GS, et al. Costs, effects and C/Eratios alongside a clinical trial. Health Econ 1994; 3 (5): 309–19
Briggs AH. A Bayesian approach to stochastic cost-effectiveness analysis. Health Econ 1999; 8 (3): 257–61
Doubilet P, Begg CB, Weinstein MC, et al. Probabilistic sensitivity analysis using Monte Carlo simulation: a practical approach. Med Decis Making 1985; 5: 157–77
Critchfield GC, Willard KE, Connelly DP. Probabilistic sensitivity analysis methods for general decision models. Comput Biomed Res 1986; 19: 254–65
Berwick DM, Cretin S, Keeler E. Cholesterol, children, and heart disease: an analysis of alternatives. Pediatrics 1981; 68 (5): 721–30
Hornberger JC. The hemodialysis prescription and cost effectiveness: Renal Physicians Association Working Committee on Clinical Guidelines. J Am Soc Nephrol 1993; 4 (4): 1021–7
Gabriel SE, Campion ME, O’Fallon WM. A cost-utility analysis of misoprostol prophylaxis for rheumatoid arthritis patients receiving nonsteroidal antiinflammatory drugs. Arthritis Rheum 1994; 37 (3): 333–41
Fiscella K, Franks P. Cost-effectiveness of the transdermal nicotine patch as an adjunct to physicians’ smoking cessation counseling. JAMA 1996; 275 (16): 1247–51
Oh PI, Maerov P, Pritchard D, et al. A cost-utility analysis of second-line antibiotics in the treatment of acute otitis media in children. Clin Ther 1996; 18 (1): 160–82
O’Brien BJ, Drummond MF, Labelle RJ, et al. In search of power and significance: issues in the design and analysis of stochastic cost-effectiveness studies in health care. Med Care 1994; 32 (2): 150–63
Sheldon TA. Problems of using modelling in the economic evaluation of health care [editorial]. Health Econ 1996; 5 (1): 1–11
Sculpher MJ, Drummond MF, Buxton MJ. The iterative use of economic evaluation as part of the process of health technology assessment. J Health Services Res Policy 1997; 2: 26–30
Sonnenberg FA, Roberts MS, Tsevat J, et al. Toward a peer review process for medical decision analysis models. Med Care 1994; 32 (7 Suppl.): JS52–64
Claxton K, Posnett J. An economic approach to clinical trial design and research priority-setting. Health Econ 1996; 5 (6): 513–24
Felli JC, Hazen GB. Sensitivity analysis and the expected value of perfect information. Med Decis Making 1998; 18 (1): 95–109
Fenwick E, Claxton K, Sculpher M, et al. Improving the efficiency and relevance of health technology assessment: the role of decision analytic modelling. Health Economists’ Study Group Conference; 1999 Jan 6–8; Birmingham
Acknowledgements
I am grateful to Dr Alastair Gray for comments and suggestions on earlier versions of this work, the participants of the Expert Workshop on ‘Validating Cost-Effectiveness Models’, 22 to 23 April 1999, held in Sheffield, England, and particularly to Dr Dennis Fryback for his insightful comments on a previous draft of this article, and finally to 2 anonymous referees for their comments. Of course, the responsibility for errors and inaccuracies in this article is all my own.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
The illustrative model used in this paper will be made available for download from http://www.ihs.ox.ac.uk/herc/.
Rights and permissions
About this article
Cite this article
Briggs, A.H. Handling Uncertainty in Cost-Effectiveness Models. Pharmacoeconomics 17, 479–500 (2000). https://doi.org/10.2165/00019053-200017050-00006
Published:
Issue Date:
DOI: https://doi.org/10.2165/00019053-200017050-00006