The 82% vs 44% that meant nothing
Stat Fails collects documented cases where a statistical error had real consequences, with the reasoning worked through. References at the end.
In 2007, campaigning for the US presidency, Rudy Giuliani ran a radio ad that used his own prostate cancer to argue American medicine was better than Britain's socialized system. It rested on one comparison.
My chance of surviving prostate cancer in the United States? Eighty-two percent. My chances of surviving prostate cancer in England? Only forty-four percent under socialized medicine.Rudy Giuliani campaign ad, October 2007
The two survival figures were real. The conclusion drawn from them was not. Both numbers are five-year survival, and five-year survival is the wrong measuring stick for whether a health system keeps men from dying of prostate cancer.
82%US five-year survival
44%UK five-year survival
~26 vs ~27deaths per 100,000 men per year - nearly identical
That is the whole case. If American care were twice as good at keeping men alive, American men would die of prostate cancer far less often. They did not: the death rates were within a whisker of each other. Two countries with nearly the same mortality cannot differ that much in real survival. The 82-versus-44 gap was not medicine working; it was the statistics being counted differently, because the two countries screened differently.
The mechanism in one line: five-year survival is measured from diagnosis, and heavy screening moves diagnosis earlier and sweeps in harmless cancers - so it inflates survival without saving a single life.
This is lead-time bias, plus its cousin overdiagnosis, at national scale. We built an interactive page where you move one diagnosis earlier and watch five-year survival climb while the date of death never budges:
See it: lead-time bias and overdiagnosisWhat to remember
To compare how well two systems treat a cancer, compare mortality, not five-year survival. Survival rewards diagnosing earlier and diagnosing more, whether or not anyone lives longer. When a survival gap is not matched by a mortality gap, the survival gap is the artefact.
The full breakdown, in plain English
The quick version is enough to distrust a survival comparison. This is the same story, slowly, with the two biases separated out.
Bias one: the clock starts earlier
Five-year survival is the share of diagnosed patients still alive five years after their diagnosis. In the US, aggressive PSA testing found prostate cancers years before they caused any symptoms. In the UK, more men were diagnosed only once symptoms appeared. Move the starting line earlier and the same man clears the five-year bar who would have missed it - not because he lived any longer, but because his clock started sooner. This is lead-time bias, and on its own it can open a large survival gap between two populations dying at the same rate.
Bias two: the harmless cancers flood in
Prostate cancer is the textbook case of overdiagnosis. Autopsy studies find that a large share of older men who died of something else were carrying a prostate cancer that never troubled them. PSA screening detects many of these slow, harmless tumours. Every one of them becomes a diagnosed patient who does not die of the disease - a guaranteed five-year survivor added to the numerator. Diagnose enough indolent cancers and survival climbs mechanically, even as the number of actual deaths stays flat. Lead time stretches each patient's clock; overdiagnosis packs the denominator with people who were never in danger. Both push survival up; neither postpones a single death.
Why mortality is the honest number
Mortality counts deaths against the whole population, not against the diagnosed. It does not care when the diagnosis was made, and it is not moved by finding extra harmless cancers - a man who was never going to die of prostate cancer does not change the death count whether or not you label him. That is exactly why it is the endpoint that screening trials are judged on. On mortality, the US and UK were level: roughly 26 versus 27 deaths per 100,000 men per year at the time. The system that diagnosed more cancers earlier did not keep more men alive; it just produced better-looking survival statistics.
What to carry out of this
A survival statistic is not a measure of how well a disease is treated until you know it is free of lead-time and overdiagnosis bias. Between populations that screen differently, it never is. Ask for mortality, and be suspicious of any survival gap that no mortality gap backs up.
Sources
- Gigerenzer, G., Gaissmaier, W., Kurz-Milcke, E., Schwartz, L. M., & Woloshin, S. (2007). Helping Doctors and Patients Make Sense of Health Statistics. Psychological science in the public interest, 8(2), 53–96. https://doi.org/10.1111/j.1539-6053.2008.00033.x
- Welch, H. G., & Black, W. C. (2010). Overdiagnosis in cancer. Journal of the National Cancer Institute, 102(9), 605–613. https://doi.org/10.1093/jnci/djq099
Stat Exam Pro drills survival, screening and the rest of medical statistics as exam-style questions with worked answers, in English and French.
Get it on iPhone