ROC curves and AUC explained for research studies

An ROC curve shows how well a continuous score separates two groups across every possible cut-off, and the area under it (AUC) condenses that into one number. The AUC answers a narrow question: how well the score ranks members of one group above the other. It does not tell you where to put a threshold, and a study that reports a good AUC has not yet shown that any particular cut-off works.

The worked example below is one seeded, synthetic research study: a laboratory marker measured once on 240 participants whose status (case or control) is known from a reference standard. It shows the AUC with its interval, the trade-off at three cut-offs, and what happens when the “best” cut-off is applied to new participants. The page does not draw the curve; it gives three engine-computed points on it instead. Nothing here concerns any individual person.

What an ROC curve plots

An ROC (receiver operating characteristic) curve starts from a continuous score, here a marker value, and a binary truth, here case or control. Pick a cut-off, call every participant at or above it positive, and two numbers follow: sensitivity, the share of cases called positive, and specificity, the share of controls called negative.

Move the cut-off and both numbers change. A lower cut-off calls more participants positive, so sensitivity rises and specificity falls. The ROC curve plots sensitivity on the vertical axis against 1 minus specificity (the false positive rate) on the horizontal axis, one point per cut-off, from the highest cut-off at the bottom left to the lowest at the top right (Fawcett 2006).

A score that carries no information gives points scattered around the diagonal from (0, 0) to (1, 1). A score that separates the groups perfectly reaches the top left corner, where sensitivity and specificity are both 1. Real markers sit between the two, and the shape of the curve shows where along the range of the score the separation is good or poor.

The curve itself has no cut-off values on its axes. Two curves can have the same shape while the cut-offs that produce each point are quite different, so a report that gives a point on the curve should also give the marker value that produced it.

What the AUC measures

The area under the ROC curve has a direct meaning. Take one case and one control at random: the AUC is the probability that the case has the higher marker value, with ties counted as half (Hanley and McNeil 1982). It is the same quantity as the Mann-Whitney statistic scaled to run from 0 to 1.

An AUC of 0.5 means the marker ranks a random case above a random control no more often than a coin would. An AUC of 1 means every case scores above every control. Values clearly below 0.5, beyond sampling noise, mean the marker runs the other way from what was assumed.

Direction matters for that reason. Before running the analysis, state which way the marker points: higher values indicate the condition, or lower values do. If the direction is instead chosen from the same data, the AUC is pushed toward or above 0.5, which flatters a weak marker. That happens even for a marker that ranks worse than chance in the stated direction. In the worked example the direction is fixed in advance as “higher values indicate a case”.

The AUC is a ranking measure. It does not depend on any cut-off, and it does not change if the marker goes through a strictly increasing transformation, such as taking logarithms. That is its strength as a summary of separation, and also why it cannot tell you which cut-off to use.

The worked example: one marker against known status

The example is seeded synthetic data generated for this post, not data from any study. Each of the 240 participants has one marker value and one reference status. Cases have a higher average marker value than controls, with overlap between the two groups, as with most real markers. Participants are independent, there is one measurement per participant, and there is no censoring.

The sample has 73 cases and 167 controls. Those counts matter more than the total: every precision statement below depends on the size of each group, and especially on the smaller one.

The AUC with its interval

The engine reports an AUC of 0.7703 with a 95% confidence interval of 0.7060 to 0.8346, using DeLong’s nonparametric method for the interval (DeLong et al. 1988). In words, a randomly chosen case has a higher marker value than a randomly chosen control about 77% of the time in this sample (ties counted as half), and the data are compatible with a population value anywhere from about 0.71 to 0.83.

The interval is as important as the estimate. With 73 cases and 167 controls the interval is this wide, and a report that gives 0.77 alone hides that the lower end of the interval describes a weaker marker than the upper end.

As with any confidence interval, the 95% describes the procedure: over repeated studies of the same design, intervals built this way contain the true AUC about 95% of the time. It is not a 95% probability that the true AUC lies in this particular interval. The post on p-values and confidence intervals covers the wording.

The engine’s assumption check here is a sample size check, which passed. The DeLong interval does not assume the marker is normally distributed in either group, but it does assume independent participants. Repeated measurements from the same person, or several samples per participant, need a method that accounts for the clustering.

The threshold trade-off at three cut-offs

To show the trade-off, the same 240 participants were read at three cut-offs. Two were set in the run specifications before those runs and not changed after seeing their results, a low one at 10.0 and a high one at 13.5. The third, 11.75, is the cut-off the engine found by maximising Youden’s index, sensitivity plus specificity minus 1 (Fluss et al. 2005). Each row is one point on the same ROC curve.

The intervals for the two fixed cut-offs are exact binomial intervals from the 2x2 table at each cut-off. The Youden row has no interval, because the engine’s ROC analysis reports the cut-off and its two proportions as found, and an ordinary binomial interval would be too narrow, and centred on an optimistic value, for a cut-off chosen as the best one in the same data.

At the low cut-off, 64 of the 73 cases are called positive, but so are 88 of the 167 controls. At the high cut-off, only 9 controls are called positive, but 51 cases are missed. Neither is wrong. They answer different study questions.

A low cut-off suits a research setting where missing a case is the costly error, for example a first-stage screen in a study where every positive participant will receive the full reference assessment anyway. A high cut-off suits a setting where a false positive is costly, for example selecting participants for an intensive second phase where each place is expensive. The Youden cut-off weights sensitivity and specificity equally, which is not the same as weighing the two errors by what they cost in a given study.

Why AUC alone does not justify a cut-off

The AUC of 0.7703 is the same whichever row of the table above a study picks. That single fact is the main reason the AUC cannot justify a cut-off: it summarises the whole curve, and a cut-off is one point on it.

Four further problems follow.

The AUC averages over cut-offs nobody would use. Part of the area comes from regions of the curve with very low sensitivity or very low specificity. Two markers with the same AUC can perform quite differently in the narrow region that matters for a given purpose, for example at the high specificity a second-phase selection needs.

The Youden cut-off fixes the weights for you. Maximising sensitivity plus specificity gives the two proportions equal weight. That matches equal costs per error only when cases and controls are equally common; otherwise it implies a cost ratio set by the prevalence, not by the study’s purpose. If the real costs differ, the best cut-off moves, and the right one depends on the purpose and on how common cases are in the population of interest. Decision curve analysis is one way to bring those costs in explicitly (Vickers and Elkin 2006).

A cut-off chosen on the data looks better than it is. Picking the best point of a noisy curve selects for the cut-off where chance happened to favour it. Sensitivity and specificity reported at a data-driven cut-off are biased upwards, most strongly in small samples (Leeflang et al. 2008).

Predictive values need prevalence. Sensitivity and specificity at a cut-off do not say what share of positives are cases. That depends on how common cases are, which in a case-control sample is set by the design. The sensitivity vs specificity post shows how much the predictive values move when only that proportion changes.

The same cut-off in a new sample

To check the cut-off on participants it was not chosen from, the Youden cut-off of 11.75 was carried over unchanged to a fresh seeded sample of 240 participants drawn from the same generator, with a different seed fixed before the run. The new sample has 83 cases (47 + 36) and 157 controls (24 + 133).

Sensitivity fell from 0.6438 to 0.5663 and specificity rose from 0.7784 to 0.8471, while their sum barely moved. So this draw shows the pair shifting, not the overall optimism that the literature describes on average; one new sample is sampling variation, and another draw could differ. What it does show is that the pair found in the first sample is only an estimate for that cut-off. The new sensitivity interval, 0.4529 to 0.6747, is wide even with 83 cases, and it contains the first sample’s 0.6438.

The practical rule follows. Either fix the cut-off before looking at the data, with a reason tied to the study’s purpose, or choose it in one sample and estimate its sensitivity and specificity in another. Cross-validation or a bootstrap within a single sample is a fallback when no second sample exists.

Computing it in R

The pROC package (Robin et al. 2011) computes the curve, the AUC, the DeLong interval and the Youden cut-off. With a data frame holding status and marker:

library(pROC)

r <- roc(data$status, data$marker, levels = c("control", "case"), direction = "<")
auc(r)
ci.auc(r, method = "delong")
coords(r, "best", best.method = "youden",
       ret = c("threshold", "sensitivity", "specificity"))

# accuracy at a cut-off fixed in advance, as a 2x2 table
table(test = data$marker >= 10, status = data$status)

levels puts the controls first and the cases second, and direction = "<" states that controls have lower values. Writing both out keeps the analysis from depending on how the software orders the labels or guesses the direction.

The worked example’s own script is shown below this post, exactly as the engine generated it, with the seed and its results summary; it reads the 240 participants’ data and recomputes the AUC, its interval and the Youden cut-off with its sensitivity and specificity. The results summary gives the AUC and its interval only. Running the script prints a list that also holds the Youden cut-off of 11.75 and its sensitivity and specificity (as youden_cutoff, sensitivity and specificity). The rows for the two fixed cut-offs and the new sample come from separate seeded runs whose scripts are not shown on this page.

In Celeus, the same analysis is the ROC curve / AUC method: choose the reference column and the continuous score, say which value of the reference column means the condition is present, and set the direction to “Higher scores indicate the condition” or “Lower scores indicate the condition” rather than leaving it to be chosen from the data. The report draws the ROC curve and gives the AUC with its DeLong interval and the Youden cut-off. Accuracy at a fixed cut-off is the separate diagnostic accuracy method, which takes the test result as a yes/no column, as the two fixed cut-offs above were run.

Reporting an ROC analysis

The STARD 2015 checklist (Bossuyt et al. 2015) covers diagnostic-accuracy reports in general. For an ROC analysis, give:

• the number of cases and controls, how participants were sampled, and how the reference status was established;
• which direction of the marker indicates the condition, stated before the analysis;
• the AUC with its confidence interval and the interval method;
• for any cut-off reported, its sensitivity and specificity with intervals, and whether it was fixed in advance or chosen from the data;
• if chosen from the data, its performance in a separate sample or by cross-validation.

For example: “In 73 cases and 167 controls, the marker had an AUC of 0.7703 (95% CI 0.7060 to 0.8346, DeLong). At a pre-specified cut-off of 13.5, sensitivity was 0.3014 (95% CI 0.1994 to 0.4200) and specificity 0.9461 (95% CI 0.9002 to 0.9751).”

Reporting several cut-offs is fine when they are labelled as descriptive points on one curve. If the study then picks one of them as its result, say so, say how many were considered, and do not present the chosen pair as if it had been fixed in advance.

When cases are rare in the intended population, the ROC curve can look reassuring while most positives are false. A precision-recall curve, which plots positive predictive value against sensitivity, makes that visible (Saito and Rehmsmeier 2015), as long as it is computed at the prevalence of the intended population rather than the sample’s.

What these numbers do and do not say

The AUC and the sensitivity and specificity in this post describe how one synthetic marker separated synthetic cases from controls in the samples generated for it. They are research summaries with sampling uncertainty, shown by the intervals. They are not a statement about any individual’s status, and this post is not guidance on testing or diagnosing any person.

For agreement between two numeric measurements of the same quantity, rather than a score against a yes/no truth, see the Bland-Altman guide. To combine several predictors into one score before an ROC analysis, logistic regression is the usual starting point. The guide to choosing a statistical test sets out the other options, and the method itself is described on the ROC curve and AUC reference page and the diagnostic accuracy reference page.

# Results

- Test: ROC curve / AUC
- n: 240
- statistic: 0.7703
- p-value: NA
- effect size: area under the ROC curve = 0.7703 (95% CI 0.706 to 0.8346)
- 95% CI: [0.706, 0.8346]
- seed: 20261101

## Assumption checks
- sample size: passed=TRUE (adequate n)

## Power
- Power is not calculated for this test.

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: ROC curve / AUC  |  engine 0.41.7  |  catalog c40-dc08
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261101)
.data_path <- "data/roc-curve-auc.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("status", "marker")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- { .roc <- pROC::roc(as.character(data[["status"]]), data[["marker"]], levels = c("control", "case"), direction = "<", quiet = TRUE); .auc_ci <- function(r) { a <- as.numeric(pROC::auc(r)); if (min(length(r$controls), length(r$cases)) < 2 || a %in% c(0, 1)) c(NA_real_, NA_real_) else as.numeric(pROC::ci.auc(r, method = "delong"))[c(1, 3)] }; .cut <- pROC::coords(.roc, "best", best.method = "youden", ret = c("threshold", "sensitivity", "specificity"), transpose = FALSE)[1, ]; list(auc = as.numeric(pROC::auc(.roc)), auc_ci = .auc_ci(.roc), youden_cutoff = unname(.cut[["threshold"]]), sensitivity = unname(.cut[["sensitivity"]]), specificity = unname(.cut[["specificity"]]), direction = .roc$direction) }
print(result)