Transferring clinical prediction models across hospitals and electronic health record systems

Abstract

Recent years have seen a surge in studies developing clinical prediction models based on electronic health records (EHRs) as a result of advances in machine learning techniques and data availability. Yet, validation and implementation of such models in practice are rare, in part because EHR-based clinical prediction models are more difficult to apply to new data sets than results of classical clinical studies due to less controlled clinical environments. In this paper we propose to use the theoretical framework of domain adaptation to analyze the problem of transferring machine-learning-based clinical prediction models across different hospitals and EHR systems. Using the model of Thoral et al. [12] predicting patient-level risk of readmission and mortality after intensive care unit discharge as a case study, we discuss, apply and compare multiple domain adaptation methods. We transfer the model from the original source data set to two new target data sets. We find that, while model performance deteriorates substantially when applying a model developed for one data set to another directly, updating models with training data from the target set and using methods that explicitly model differences in data sets always improves model performance. In a simulation experiment, we show that having access to data or model parameters from another hospital can substantially reduce the amount of data required to build an accurate prediction model for a new hospital. We also show that these performance gains diminish with increasing availability of data from the target hospital.