Header

UZH-Logo

Maintenance Infos

Validating and updating a risk model for pneumonia - a case study


Held, Ulrike; Bové, Daniel Sabanes; Steurer, Johann; Held, Leonhard (2012). Validating and updating a risk model for pneumonia - a case study. BMC Medical Research Methodology, 12:99.

Abstract

BACKGROUND: The development of risk prediction models is of increasing importance in medical research - their use in practice, however, is rare. Among other reasons this might be due to the fact that thorough validation is often lacking. This study focuses on two Bayesian approaches of how to validate a prediction rule for the diagnosis of pneumonia, and compares them with established validation methods. METHODS: Expert knowledge was used to derive a risk prediction model for pneumonia. Data on more than 600 patients presenting with cough and fever at a general practitioner's practice in Switzerland were collected in order to validate the expert model and to examine the predictive performance of it. Additionally, four modifications of the original model including shrinkage of the regression coefficients, and two Bayesian approaches with the expert model used as prior mean and different weights for the prior covariance matrix were fitted. We quantify the predictive performance of the different methods with respect to calibration and discrimination, using cross-validation. RESULTS: The predictive performance of the unshrinked regression coefficients was poor when applied to the Swiss cohort. Shrinkage improved the results, but a Bayesian model formulation with unspecified weight of the informative prior lead to large AUC and small Brier score, naïve and after cross-validation. The advantage of this approach is the flexibility in case of a prior-data conflict. CONCLUSIONS: Published risk prediction rules in clinical research need to be validated externally before they can be used in new settings. We propose to use a Bayesian model formulation with the original risk prediction rule as prior. The posterior means of the coefficients, given the validation data showed best predictive performance with respect to cross-validated calibration and discriminative ability.

Abstract

BACKGROUND: The development of risk prediction models is of increasing importance in medical research - their use in practice, however, is rare. Among other reasons this might be due to the fact that thorough validation is often lacking. This study focuses on two Bayesian approaches of how to validate a prediction rule for the diagnosis of pneumonia, and compares them with established validation methods. METHODS: Expert knowledge was used to derive a risk prediction model for pneumonia. Data on more than 600 patients presenting with cough and fever at a general practitioner's practice in Switzerland were collected in order to validate the expert model and to examine the predictive performance of it. Additionally, four modifications of the original model including shrinkage of the regression coefficients, and two Bayesian approaches with the expert model used as prior mean and different weights for the prior covariance matrix were fitted. We quantify the predictive performance of the different methods with respect to calibration and discrimination, using cross-validation. RESULTS: The predictive performance of the unshrinked regression coefficients was poor when applied to the Swiss cohort. Shrinkage improved the results, but a Bayesian model formulation with unspecified weight of the informative prior lead to large AUC and small Brier score, naïve and after cross-validation. The advantage of this approach is the flexibility in case of a prior-data conflict. CONCLUSIONS: Published risk prediction rules in clinical research need to be validated externally before they can be used in new settings. We propose to use a Bayesian model formulation with the original risk prediction rule as prior. The posterior means of the coefficients, given the validation data showed best predictive performance with respect to cross-validated calibration and discriminative ability.

Statistics

Citations

3 citations in Web of Science®
3 citations in Scopus®
Google Scholar™

Altmetrics

Downloads

24 downloads since deposited on 13 Feb 2013
2 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:04 Faculty of Medicine > Epidemiology, Biostatistics and Prevention Institute (EBPI)
04 Faculty of Medicine > University Hospital Zurich > Clinic and Policlinic for Internal Medicine
Dewey Decimal Classification:610 Medicine & health
Language:English
Date:2012
Deposited On:13 Feb 2013 14:16
Last Modified:07 Aug 2017 04:48
Publisher:BioMed Central
ISSN:1471-2288
Free access at:PubMed ID. An embargo period may apply.
Publisher DOI:https://doi.org/10.1186/1471-2288-12-99
PubMed ID:22817850

Download

Download PDF  'Validating and updating a risk model for pneumonia - a case study'.
Preview
Content: Published Version
Filetype: PDF
Size: 383kB
View at publisher
Licence: Creative Commons: Attribution 2.0 Generic (CC BY 2.0)