Stat Fails collects documented cases where a statistical error had real consequences, with the reasoning worked through. References at the end.
Sepsis kills fast, so an early-warning model that flags it before a clinician does could save lives. One such model, built into Epic, the electronic record system running much of American healthcare, was switched on in hundreds of hospitals. It came with a headline number.
An area under the ROC curve of 0.76 to 0.83.
In 2021 a team at the University of Michigan checked that number against their own patients, about 38,000 hospital stays. What they found is the reason this is a Stat Fails.
0.63the AUC they actually measured
0.76-0.83the AUC on the vendor's slides
67%of sepsis cases it missed
18%of all patients it alerted on
An AUC of 0.63 is barely above 0.5, the score of a coin. And a number on a slide is not a bedside: at the alert threshold the hospital used, the model missed two of every three sepsis cases while firing an alarm on nearly one in five patients, most of them false.
That combination is the worst of both worlds. Miss most of the disease, and bury the staff in false alarms until they stop trusting the alarm at all. Alert fatigue is not a metaphor: when four alerts in five are wrong, the fifth gets ignored too.
The mechanism in one line: the AUC on the vendor's data was never the AUC in your hospital, and no amount of marketing closes that gap. Only an independent test on local patients can.
This is the ROC curve and AUC meeting the real world. If those terms are fuzzy, the illustrated lesson builds them from scratch, and the playground lets you feel how a threshold trades catching disease for false alarms:
When someone sells you a model by its AUC, ask two questions before you trust it: measured on whose patients, and at what threshold. A high number on the vendor's population, read off no particular cutoff, tells you almost nothing about what happens on your ward.
The full breakdown, in plain English
The quick version is enough to make you distrust a vendor's AUC. This is the same story, slowly, with the numbers drawn out.
What 0.63 means
Read AUC as one plain sentence: the chance the model gives a patient who will become septic a higher score than one who will not. At 1.0 it never gets a pair wrong; at 0.5 it is guessing. The vendor sold a band from 0.76 to 0.83. The independent measurement landed at 0.63, closer to the coin than to the box.
The AUC scale from a coin flip (0.5) to perfect (1.0). The vendor's claimed band sits well to the right; the independently measured 0.63 sits just above guessing.
The numbers in patients
An AUC hides what a threshold does, so put it in people. Sepsis showed up in about 66 of every 1,000 hospital stays, so take 1,000 admissions.
• About 66 of the 1,000 develop sepsis. • The model fires an alert on about 180 of the 1,000. • Of those 180 alerts, roughly 22 are real sepsis and 158 are false alarms. • And 44 of the 66 septic patients are never flagged at all. A nurse minding the alerts sees 180 warnings, 158 of them wrong, while two-thirds of the sepsis walks past unflagged.
Per 1,000 admissions. Of 66 sepsis cases the model catches 22 and misses 44 (red). Of 180 alerts, only 22 are real and 158 are false (orange).
Why the vendor's number was higher: three reasons
Three separate mechanisms pushed the vendor's AUC up and the field-measured AUC down. Any one would have been enough to matter; all three were in play.
1. Domain shift
The model learned its patterns on a specific set of hospitals. When it was switched on somewhere else, several things quietly diverged.
Equipment. Lab analysers calibrate to different reference ranges. A creatinine of 1.4 mg/dL on one machine is not always the same physiological signal as 1.4 on another.
Documentation habits. Nurses on one ward record vital signs hourly; on another, every four hours. The model sees a different data density for the same underlying patient.
Patient mix. An academic centre and a community hospital see different populations: ages, comorbidities, referral patterns. The base rate of sepsis and everything correlated with it shifts.
2. Late scoring
An early-warning model is only useful before someone else has noticed. When the Michigan team scored the model strictly on predictions made before sepsis was clinically apparent, the AUC was 0.63. When they broadened the window to include predictions made up to three hours after sepsis onset, the AUC jumped to 0.80.
Sepsis three hours in is a different clinical scene. Blood pressure is dropping, lactate is rising, the team is already reaching for antibiotics. Almost any model, including a very simple one, can tell you the patient is septic once the sepsis is obvious.
3. Data leakage
Some of the model's inputs were events like antibiotic orders. Antibiotics are prescribed because someone already suspects sepsis. So the feature the model was reading was, in effect, a summary of what the clinical team had already concluded.
This is called label leakage: an input feature that sits causally downstream of the outcome the model is trying to predict. It makes evaluation numbers look better than the model's real utility, because on the retrospective data used to score the model those features were already in place. In production, on a patient whose sepsis has not been suspected yet (the patient the model was supposed to catch), the feature is not there.
How this happened
Every time I come back to this story, it surprises me. The United States has the FDA. It has a culture around drug approval where a failed phase III trial ends a compound. The bar for a new molecule is high, and applied consistently. Yet this model was live in hundreds of hospitals for years, on real patients, before anyone independently checked its numbers.
The specific gap is regulatory. Software built into the electronic health record and marketed as clinical decision support fell into an exemption category: if the software supports rather than replaces a clinician, no FDA review was required. The 21st Century Cures Act (2016) formalised that carve-out. The Epic sepsis model was sold, deployed and turned on inside that exemption.
A sense of the scale
Epic runs the electronic record for roughly 40% of U.S. acute-care hospitals (KLAS 2022). Sepsis affects around 1.7 million U.S. adults per year and contributes to approximately 270,000 deaths (CDC). At the deployment threshold, the model missed roughly two thirds of septic patients and fired an alert on almost one in five admissions, most of the alerts false. Multiple sepsis studies estimate that each hour of delay in appropriate antibiotics is associated with a 4 to 8% increase in mortality (Kumar et al. 2006 and follow-up work).
Financially: U.S. sepsis care runs on the order of 62 billion dollars per year (AHRQ). A working early-warning system that identified even 20% more true positives at an actionable threshold would translate into billions in avoided ICU days and shorter courses of care. Deploying an under-performing model instead added the operating costs of running the model and the cost of clinician time spent on false alarms, with no offsetting gain.
What to carry out of this
An AUC without a population and a threshold is a slogan, not a measurement. Before a model touches your patients, ask for its performance on patients like yours, at the cutoff you would actually use, and count the false alarms in people, not in decimals.
Sources
Wong, A., Otles, E., Donnelly, J. P., et al. (2021). External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients. JAMA Internal Medicine, 181(8), 1065–1070. https://doi.org/10.1001/jamainternmed.2021.2626
Habib, A. R., Lin, A. L., & Grant, R. W. (2021). The Epic Sepsis Model Falls Short—The Importance of External Validation. JAMA Internal Medicine, 181(8), 1040–1041. https://doi.org/10.1001/jamainternmed.2021.3333
Kapoor, S., & Narayanan, A. (2022). Leakage and the Reproducibility Crisis in ML-based Science. arXiv preprint arXiv:2207.07048. https://arxiv.org/abs/2207.07048
Stat Exam Pro drills ROC, AUC, sensitivity and specificity, and the rest of medical statistics, as exam questions with worked answers, in English and French.
