ENFR

ROC curve: the sensitivity vs specificity tradeoff

A playground: move one threshold and watch a test trade catching the sick for raising false alarms.

New to ROC and AUC? Read the illustrated lessonTrain with us

Sensitivity (Se)

Of the diseased, the fraction the test flags positive. Raise the bar for calling a result positive and more sick people slip through.

Specificity (Sp)

Of the healthy, the fraction correctly called negative. Lower the bar and you catch more disease, but at the cost of more false alarms.

AUC (area under the ROC)

One number for the whole test, whatever threshold you pick: the chance it ranks a random sick person above a random healthy one. 0.5 is a coin flip, 1.0 is perfect.

The situation

Most diagnostic tests measure something on a scale: a hormone level, a density, a risk score. You pick a cutoff. Above it, the test says positive; below, negative. The two humps below are the healthy and the diseased populations, and where they overlap, no cutoff can separate them cleanly. Every place you put the threshold is a choice about which mistake to make more of: missing the sick, or alarming the well.

How to use

The top chart shows the two populations and your threshold. The ROC curve plots every possible threshold at once; the dot is the one you have chosen now. Move the threshold slider to slide the dot along the curve. Move the test-quality slider to pull the two humps apart, which bows the ROC toward the top-left corner and raises the AUC.

Sensitivity
Specificity

The two populations and your cutoff

ROC curve

You cannot maximize both. Slide the threshold left and sensitivity climbs while specificity falls; slide it right and the reverse happens. There is no cutoff that is best for everyone, only a cutoff that is best for a purpose: a screen that must not miss cancers sits far to one side, a test that confirms a diagnosis before surgery sits far to the other.

One more thing the ROC curve hides. AUC and the whole curve describe the test, and they do not move when the disease gets rarer. Whether a positive result is actually trustworthy is a different question, and it does depend on how common the disease is. See it on the PPV playground

Here is the reasoning in words, then the formulas.

What the curve is made of

At any threshold, sensitivity is the true positive rate (of the sick, how many test positive) and specificity is the true negative rate. Its complement, the false positive rate, is 1 minus specificity. The ROC curve is just those two numbers plotted against each other as the threshold sweeps from strict to lenient: the false positive rate on the x-axis, the true positive rate on the y-axis.

$$\text{Se} = \text{TPR} \qquad \text{Sp} = \text{TNR} \qquad \text{FPR} = 1 - \text{Sp}$$

For the tidy case of two equal-width bell curves separated by a distance d, the whole curve has a closed form and the area under it is simply the normal CDF of d over root two.

$$\text{AUC} = \Phi\!\left(\frac{d}{\sqrt{2}}\right)$$

That area has a plain meaning worth memorizing: it is the probability that the test gives a random sick person a higher score than a random healthy one. An AUC of 0.9 means that in 90 out of 100 such pairings, the sick person scores higher. It is the same quantity as the Mann-Whitney statistic, and in machine learning it is the standard summary of a binary classifier.

One test, two jobs

Problem

A lab reports that a blood test for a cancer has an AUC of 0.92. A colleague says the test is 95% sensitive; another says it is 98% specific. Both quote the same test. How can both be right?

This one is best solved on the playground, not on paper. Set the test-quality slider until it reads AUC 0.92, then move the threshold: at one position the panel shows 95% sensitivity, and further along it shows 98% specificity. One curve, two operating points.

Because sensitivity and specificity are not fixed properties of the test. They are a single point you chose on the ROC curve, and the AUC of 0.92 fixes the whole curve, not the point.

• For a screening program that must not miss cancers, set a low cutoff: sensitivity 95%, specificity 64%. About a third of healthy people are called back, and that is accepted because a positive screen leads to a confirmatory test, not a diagnosis.
• To confirm the cancer before major surgery, set a high cutoff: specificity 98%, sensitivity 48%. A positive now means something, at the price of missing more than half the cancers.
• Same test, same AUC of 0.92. Two operating points, chosen for two different costs of being wrong.
So the right question is never just how good is this test, but where should its threshold sit for this job. Press the two presets above, Screening and Confirmation, to watch the same curve read out as two very different tests.

Common questions

What does AUC mean?

AUC is the probability that the test gives a randomly chosen sick patient a higher score than a randomly chosen healthy one. 0.5 is a coin flip and 1.0 is perfect separation.

What is a good AUC value?

0.5 is useless, no better than chance, and most useful clinical tests land between about 0.75 and 0.95. Higher is better, but the right threshold to use still depends on the clinical cost of missed cases versus false alarms.

What is the difference between sensitivity and specificity?

Sensitivity is the share of diseased people the test flags positive; specificity is the share of healthy people it correctly clears. Moving the decision threshold trades one for the other, which is exactly what the ROC curve traces.

This is one intuition trap. The Paradoxes & Intuition Traps section of Stat Exam Pro is a whole set of them - exam-style questions with worked answers, in English and French.

Get it on iPhone