Improving early recognition of sepsis in the Neonatal Intensive Care Unit using machine learning models and electronic health record data.
Neonatal Sepsis - Incidence and Outcomes
Despite advances in knowledge and medical care, sepsis remains a major cause of morbidity and mortality in infants worldwide, claiming the lives of one million newborn infants each year according to the World Health Organization (Liu et al. 2015;Vogel 2017). By definition, sepsis involves the immune response to invading pathogens, and is characterised by presence of a bloodstream infection accompanied by multi-organ system dysfunction. Although sepsis affects relatively few healthy, term infants, the incidence is significantly higher (200-fold) in those born prematurely or chronically hospitalised (Zea-Vera and Ochoa 2016; Liu et al. 2015).
Prematurely born infants experience the highest mortality, and among survivors, 30-50% incur major long term impairments including prolonged hospitalisation, chronic lung disease and neurodevelopmental disabilities (Stoll et al. 2004; Stoll et al. 2010). To date, despite increased understanding of the pathophysiology of sepsis and sophistication of neonatal intensive care strategies, there have been only modest improvements in outcomes (Wynn 2016).
Challenges in Neonatal Sepsis Recognition and Treatment
Early detection of sepsis, followed by timely intervention, is key to reducing neonatal morbidity and mortality. However, delays in recognition and treatment are common (Castellanos-Ortega et al. 2013). Infants frequently demonstrate subtle, ambiguous clinical signs, which overlap with other neonatal disease processes. Multiple diagnostic biomarkers have been studied, but none have yet achieved sufficient accuracy to be employed in clinical practice (Reinhart et al. 2012; Ng et al. 2018). In a retrospective review of infants in our level IV Neonatal Intensive Care Unit (NICU) at the Children’s Hospital of Philadelphia (CHOP) who underwent sepsis evaluations with subsequent positive blood cultures, recognition was delayed more than 3 hours in 30% and a significant proportion progressed to severe sepsis and multi-organ system dysfunction. These findings reflect the challenge of interpreting non-specific clinical signs in the face of complex underlying conditions and support the need for improved methods for sepsis detection in infants.
As detailed in the Surviving Sepsis Campaign, early treatment such as timely antibiotic administration is associated with decreased sepsis mortality (Dellinger et al. 2013). Recent studies of infected adults and children demonstrate significantly increased risk of mortality and prolonged organ dysfunction when antimicrobial therapy was delayed (Seymour et al. 2017; Weiss et al. 2014; Evans et al. 2018). However, there is little evidence regarding optimal timing and consequences of delayed antibiotic administration in infants with sepsis. Recent work using our neonatal sepsis registry (see below) has demonstrated that prolonged time to antibiotic initiation was associated with significantly increased morbidity and mortality in infants with sepsis, highlighting the importance of rapid recognition of sepsis in the NICU (Schmatz et al. 2019).
To avoid adverse outcomes of delayed antibiotic administration while recognising the heterogeneous, complex nature of sepsis and the immune inflammatory response, empiric antibiotics are widely administered despite the modest prevalence of culture proven sepsis (Schlapbach et al. 2018) and the potential for overtreatment of non-infected infants (Squire et al. 1979). Infants with suspected sepsis are often managed conservatively and receive weeks of antibiotic therapy, often despite negative cultures (Gonsalves et al. 2009; Connel et al. 2007). Recent studies demonstrate that unnecessary antibiotic exposure in non-infected infants may worsen clinical outcomes and contribute to the development of antibiotic resistance (Ting et al. 2016; Cotten et al. 2009; Kuppala et al. 2011). These findings underscore the importance of developing novel, improved methods for sepsis detection in infants with potentially life threatening illness while minimising the overtreatment of non-infected infants.
Neonatal Sepsis Registry at Children’s Hospital of Philadelphia
In 2014, we established a sepsis registry in the CHOP NICU which provides automated identification and data abstraction from the electronic health record (EHR) of all infants less than one year of age who are evaluated for sepsis (EHR–Epic Systems Inc. Verona, WI). The CHOP NICU is a 100 bed quaternary unit that admits and treats roughly 1300 infants annually including outborn infants with complex medical conditions as well as inborn infants with surgical and other anomalies delivered in the Special Delivery Unit at CHOP. Infants are enrolled into the registry when clinical concern prompts the collection of a blood culture and initiation of intravenous antibiotics. The registry captures EHR data for variables including patient demographics, laboratory and vital sign data, medication administration records, respiratory and inotropic support, NICU length of stay and mortality. Comorbid conditions are identified based on EHR ICD-9/ICD-10 codes. Infants are then further classified when results of blood and other systemic cultures are known. Electronically abstracted data are intermittently evaluated by manual chart review to ensure accuracy. The registry currently includes data from 1,868 infants who experienced 3,384 episodes of sepsis evaluation. Of these evaluations, 336 (10%) resulted in positive cultures for bacterial pathogens. There were an additional 682 evaluations (20%), of “clinical sepsis” where clinicians nevertheless chose to treat with antibiotics for at least 5 days despite the inability to identify a bacterial pathogen.
Models to Predict Infant Sepsis
We used readily available EHR data for infants in our registry to develop prediction models that may be useful to improve the early recognition of sepsis (Masino et al. 2019). We demonstrated that several machine learning algorithms could achieve good performance to differentiate infected (either culture proven or clinical sepsis) from non-infected infants 4 hours prior to the time of clinical recognition (i.e. the time when sepsis evaluation was initiated). Six of the algorithms we evaluated achieved an area under the receiver operating characteristic (AUROC) > 0.8, with the best performing algorithm (gradient boosting) achieving an AUROC of 0.87 [95% confidence interval (CI): 0.82, 0.92]. At a pre-specified sensitivity of 0.8, the gradient boosting algorithm had a specificity of 0.74 [95% CI 0.63, 0.84]. Our results compare favourably with the few recent studies that have attempted to predict sepsis in advance of clinical recognition (Desautels et al. 2016; Fairchild et al. 2017; Shashikumar et al. 2017; Nemati et al. 2017). Only one of these studies was performed in infants, and that study required the use of high frequency vital sign data from bedside monitors, which is not readily available in most EHRs (Fairchild et al. 2017).
Path Forward to Precision Medicine Using Sepsis Prediction Models
Despite the promise of prediction models that have excellent test characteristics for discriminating infected from non-infected patients in advance of current recognition, there remain important barriers to translation into clinical practice. For conditions such as sepsis, where delayed recognition and treatment results in significant mortality, implementers typically favour sensitive alerts at the expense of specificity. However, even algorithms that achieve high levels of specificity will typically have low positive predictive values (PPV) in real-world clinical environments. To address this concern, two-phase sepsis alerts that use a highly sensitive initial alert to recommend additional evaluation, sometimes known as a “sepsis huddle,” followed by a more specific secondary assessment have been used successfully in paediatric emergency departments (ED) (Balamuth et al. 2017). In these settings there is a specific moment in time, typically during patient triage, where the ED team decides whether or not to proceed with a sepsis evaluation.
In contrast to the ED setting, there is no single evaluation moment in an intensive care unit (ICU) setting, rather there is continuous patient monitoring and evaluation for sepsis. The frequent evaluations produced by a predictive model in an ICU setting may further compound the problem of low PPV, as it may lead to high false alarm rates and alarm fatigue which markedly decreases the likelihood that clinicians will respond to an alarm, especially when those alarms occur repeatedly for the same patient (Ancker et al. 2017). It is, unfortunately, not obvious how best to extend a two-phase approach that is effective in the ED to the ICU setting. An obvious alternative is to require models with both high sensitivity and PPV. However, this is a daunting challenge for rare event prediction; consider for example the difficulty of accurately identifying fraudulent credit card transactions despite the availability of huge amounts of data and resources (Fu et al. 2016). Clinicians are trained to view decision-making as a task that occurs at particular moments in time. They arrive at the patient’s bedside with a collection of practice guidelines, decision rules, heuristics and instincts to establish a treatment plan. However, given the challenges above, it may be more useful to think of sepsis prediction in the ICU as “weather forecasting” rather than as the familiar concept of “alarm systems” that have been used to support clinical decision-making for decades. Additional understanding of how clinical teams approach decisions related to sepsis is required before new approaches such as continuously available long- and short-range forecasts of sepsis probability estimates can be introduced in clinical settings. New approaches to estimating and reporting the uncertainty that is inherent in prediction models must also be developed. Clinical teams are also unlikely to accept “black box” model predictions without a way of understanding the key patient features that are driving a particular risk estimate. Our team’s future work will focus on these challenges of determining how to best support clinical teams with imperfect forecasts of sepsis probability that are available continuously at the bedside.
Machine learning models can identify infants with sepsis in the NICU hours prior to clinical recognition and may be valuable as a clinical decision support tool. As discussed above, we anticipate significant challenges in translating retrospective sepsis decision support models into effective clinical tools. Nonetheless, given the significance of neonatal sepsis and the consequences of delayed recognition and treatment, we are committed to the performance of clinical trials to identify infants at highest risk of sepsis and provide clinicians and nurses with the decision support needed to improve the health and safety of these infants.
- Neonates and infants are uniquely susceptible to infection and experience high morbidity and mortality from this disease.
- Rapid recognition and treatment are crucial to improve sepsis outcomes.
- Prediction models using EHR data may be useful in early recognition of infants with sepsis.
- Results support the future implementation of novel decision support tools in clinical trials to improve clinical decision making in infants with sepsis.
Conflict of Interest
The authors report no conflict of interest.
Ancker JS et al. (2017) Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak, 17(1):36.
Balamuth F et al. (2017) Improving recognition of pediatric severe sepsis in the emergency department: contributions of a vital sign–based electronic alert and bedside clinician identification. Ann Emerg Med, 70(6):759-769.
Castellanos-Ortega A et al. (2010) Impact of the surviving sepsis campaign protocols on hospital length of stay and mortality in septic shock patients: results of a three-year follow-up quasi-experimental study. Crit Care Med, 38(4):1036-1043.
Connell TG et al. (2007) How reliable is a negative blood culture result? Volume of Blood Submitted for Culture in Routine Practice in a Children’s Hospital. Pediatrics, 119(5):891-896.
Cotten MC et al. (2009) Prolonged duration of initial empirical antibiotic treatment is associated with increased rates of necrotizing enterocolitis and death for extremely low birth weight infants. Pediatrics, 123(1):58-66.
Dellinger RP et al. (2013) Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med, 4(2):580-637.
Desautels T et al. (2016) Prediction of sepsis in the intensive care unit with minimal electronic health record data: a machine learning approach. JMIR Med informatics, 4(3):e28.
Evans IVR et al. (2018) Association between the New York sepsis care mandate and in-hospital mortality for pediatric sepsis. JAMA, 320:358–367.
Fairchild KD et al. (2017) Vital signs and their cross-correlation in sepsis and NEC: a study of 1,065 very-low-birth-weight infants in two NICUs. Pediatr Res, 81(2):315-321.
Fu K et al. (2016) Credit card fraud detection using convolutional neural networks. In: International Conference on Neural Information Processing. Springer Cham, 483-490.
Gonsalves WI et al. (2009) Effects of volume and site of blood draw on blood culture results. J Clin Microbiol, 47(11):3482-3485.
Kuppala VS et al. (2011) Prolonged initial empirical antibiotic treatment is associated with adverse outcomes in premature infants. J Pediatr, 159(5):720-725.
Liu L et al. (2015) Global, regional, and national causes of child mortality in 2000-13, with projections to inform post-2015 priorities: an updated systematic analysis. Lancet, 385(9966): 430-440.
Masino AJ et al. (2019). Machine learning models for early sepsis recognition in the neonatal intensive care unit using readily available electronic health record data. Plos One, 14(2):e0212665.
Nemati S et al. (2017) An interpretable machine learning model for accurate prediction of sepsis in the ICU. Crit Care Med.
Ng S et al. (2018) Precision medicine for neonatal sepsis. Frontiers in Molecular Biosciences, 5:70.
Reinhart K et al. (2012) New approaches to sepsis: molecular diagnostics and biomarkers. Clinical Microbiology Reviews, 25(4):609-634.
Schlapbach LJ et al. (2019) Reducing collateral damage from mandates for time to antibiotics in pediatric sepsis—primum non nocere. JAMA Pediatrics, 173(5):409-410.
Schmatz M et al. (2019) Surviving sepsis in the neonatal intensive care unit: association between time to antibiotic administration and in-hospital outcomes. To be published in J Pediatr.
Seymour CW et al. (2017) Time to treatment and mortality during mandated emergency care for sepsis. N Engl J Med, 376(23): 2235-2244.
Shashikumar SP et al. (2017) Multiscale network representation of physiological time series for early prediction of sepsis. Physiol Meas, 38(12):2235-2248.
Squire E et al. (1980) Diagnosis of neonatal bacterial infection: hematologic and pathologic findings in fatal and nonfatal cases. Obstet Gynecol Surv, 35(7):448-450.
Stoll BJ et al. (2004) Neurodevelopmental and growth impairment among extremely low-birth-weight infants with neonatal infection. J Am Med Assoc, 292(19):2357-2365.
Stoll BJ et al. (2010) Neonatal outcomes of extremely preterm infants from the NICHD neonatal research network. Pediatrics, 126(3):443-456.
Ting JY et al. (2016) Association between antibiotic use and neonatal mortality and morbidities in very low-birth-weight infants without culture-proven sepsis or necrotizing enterocolitis. JAMA Pediatr, 170(12):1181-1187.
Vogel L (2017) Sepsis kills one million newborns a year: WHO. CMAJ, 189(40):E1272.
Weiss SL et al. (2014) Delayed antimicrobial therapy increases mortality and organ dysfunction duration in pediatric sepsis. Crit Care Med, 15(9):798-805.
Wynn JL (2016) Defining neonatal sepsis. Curr Opin Pediatr, 28(2):135-140.
Zea-Vera A, Ochoa TJ (2015) Challenges in the diagnosis and management of neonatal sepsis. J Trop Pediatr, 61(1):1-13.