Predicting Readmission, Death After Discharge From ICU

Many machine learning (ML) models have been developed for use in the ICU, but their effectiveness in new settings is uncertain due to a lack of external validation. A study showed that less than one-third of Food and Drug Administration (FDA) approved ML models have undergone multisite assessment. In addition, only 11% of  ICU prediction models have been externally validated. 

The ICU is a popular area of application for ML models due to the large volume of continuously monitored patient data. ML models have been developed in the ICU to predict sepsis onset, COVID-19 disease progression, and mortality and readmission. Clinicians are increasingly being presented with ML vendors who claim their technology will revolutionise clinical workflows, environments, and patient outcomes. To ensure that these models are effective, clinicians must understand the necessary quality assessment steps that must be taken before implementation in their local setting. 

A study was conducted to evaluate the performance of a decision support tool based on an ML model for predicting readmission or death within seven days after ICU discharge. The tool was tested before, during, and after retraining and recalibration to assess its effectiveness in new settings. The study was conducted in two ICUs in tertiary care centres in the Netherlands.

This study describes the development and validation of a gradient-boosted ML model using electronic health record data from 2004 to 2021. An independent validation of the model was also conducted using data from 2011 to 2019 from a different tertiary care centre.

Findings show moderate discrimination (AUC of 0.72) in the external validation, which improved after retraining and recalibration (AUC of 0.79). The study found that the machine learning model had difficulty achieving generalisability despite similarities in patient population, healthcare context, and model specification. However, retraining focused on disease severity monitoring, and ICU specialty improved the model's predictive performance.

These findings emphasise the importance of external validation and retraining before using ML models in new settings. The results highlight that these steps should be considered by clinicians and decision-makers when evaluating and implementing new ML models in their local healthcare setting.