DeLong's Test

Introduction

DeLong's Test checks whether two ROC curves-most commonly two machine learning classifiers, diagnostic tests, or biomarkers-have significantly different areas under the curve (AUC) when both were evaluated on the same set of subjects. If you are asking questions like "does my new model actually discriminate better than the baseline?", "is this biomarker a significantly better diagnostic than the existing one?", or "did fine-tuning improve AUC, or is the gap just noise?"-and both scores were computed on the same patients, images, or samples-this is usually the test you are looking for.

By the end of this article you will be able to state exactly when DeLong's Test applies, compute one completely by hand on two worked examples-a small classifier pair and a larger sample-interpret the result alongside the AUC difference and its confidence interval, understand the structural-component machinery that makes the test work, and run the same test in a few lines of Python.

What Is DeLong's Test?

DeLong's Test compares the AUCs of two ROC curves built from the same set of subjects, while correctly accounting for the fact that the two AUCs are not independent-they share the same subjects, so their sampling errors are correlated. If two classifiers were truly equally good at discrimination, the difference between their AUCs should hover around zero once that shared correlation is accounted for. The larger the standardized difference, the stronger the evidence that one classifier genuinely discriminates better than the other.

It exists because AUC is one of the most widely reported metrics in machine learning and diagnostic medicine, and comparing two AUC numbers side by side-"0.91 vs 0.87, so mine is better"-says nothing about statistical significance. Before DeLong, Elizabeth DeLong, David DeLong, and Daniel Clarke-Pearson, and naive approaches either ignored the correlation between paired AUCs (understating significance) or required expensive resampling. DeLong's Test fills that gap with a closed-form, non-parametric solution built on Mann-Whitney U-statistic theory.

When to Use It

Use DeLong's Test when you have two classifiers or diagnostic scores evaluated on the same subjects against the same binary ground truth, and you want to know whether their AUCs differ significantly. This is the standard scenario in model comparison-a baseline model versus a new model, a single biomarker versus a combined panel, or a model before versus after fine-tuning-whenever every subject received a prediction score from both models.

Key Assumptions

• Paired evaluation: both classifiers must produce a prediction score for the same subjects, scored against the same binary ground-truth labels-no subject should be scored by only one classifier.
• Binary ground truth: the outcome variable being predicted must have exactly two classes (e.g., disease/no disease, default/no default, positive/negative).
• Independence across subjects: different subjects must be independent of one another-one subject's pair of scores gives no information about another subject's.
• Sufficiently large sample: the asymptotic normal approximation underlying the z-test works best with reasonably large samples in both the positive and negative classes-very small or heavily imbalanced samples can make the approximation unreliable.
• Reasonably continuous scores: the underlying prediction scores should not be dominated by large blocks of tied values, since the AUC (and its variance) is derived from rank-based comparisons.

Hypotheses

DeLong's Test formally tests whether the two AUCs are equal:

• Null Hypothesis (\( H_0 \)): the two classifiers have equal discriminative ability- \( AUC_1 = AUC_2 \)-and any observed difference is due to sampling variability.
• Alternative Hypothesis (\( H_1 \)): the two classifiers have different discriminative ability-\( AUC_1 \neq AUC_2 \) (a one-sided version, \( AUC_1 > AUC_2 \) or \( AUC_1 < AUC_2 \), is also common when there is a clear directional expectation, such as testing whether a new model improves on a baseline).

(A significant DeLong's Test tells you the AUCs differ, but on its own it does not tell you the size of that improvement in practical terms-pairing the p-value with the AUC difference and its confidence interval below gives a fuller picture.)

The Formula, Explained

Suppose you have \( m \) subjects with a positive outcome and \( n \) subjects with a negative outcome (\( N = m + n \) total), and two classifiers each producing a prediction score for every subject. Each classifier's AUC can be written as a Mann-Whitney U-statistic:

where \( X_{k,i} \) is classifier \( k \)'s score for the \( i \)-th positive subject, \( Y_{k,j} \) is classifier \( k \)'s score for the \( j \)-th negative subject, and \( \psi \) is the placement function:

DeLong's key insight was to decompose each \( AUC_k \) into per-subject structural components-one value per positive subject and one per negative subject-so that the variance of \( AUC_k \), and the covariance between \( AUC_1 \) and \( AUC_2 \), can be computed directly from these components rather than by resampling. The test statistic is then:

Under \( H_0 \), \( z \) follows approximately a standard normal distribution, and the two-sided p-value is \( 2 \times (1 - \Phi(|z|)) \), where \( \Phi \) is the standard normal CDF. The covariance term is what sets DeLong's Test apart-ignoring it (treating the two AUCs as independent) would use \( \mathrm{Var}(AUC_1) + \mathrm{Var}(AUC_2) \) alone in the denominator, which is almost always too large and understates significance when the two classifiers agree on which subjects are easy or hard.

Structural Components in Detail

For classifier \( k \), define two sets of structural components-one per positive subject, one per negative subject:

In words, \( V^{k}_{10,i} \) is the fraction of negative subjects that positive subject \( i \) outscored (with ties counted as half), and \( V^{k}_{01,j} \) is the fraction of positive subjects that negative subject \( j \) was outscored by. Averaging either set recovers the AUC itself: \( AUC_k = \frac{1}{m}\sum_i V^{k}_{10,i} = \frac{1}{n}\sum_j V^{k}_{01,j} \).

The \( 2 \times 2 \) covariance matrix between classifiers \( k \) and \( l \) is then built from the sample covariances of these components across subjects:

Setting \( k = l \) gives each classifier's own variance, \( \mathrm{Var}(AUC_k) = \mathrm{Cov}(AUC_k, AUC_k) \); setting \( k \neq l \) gives the cross-covariance term used in the test statistic above. Because both classifiers are evaluated on the identical set of positive and negative subjects, the components \( V^{1}_{10,i} \) and \( V^{2}_{10,i} \) are indexed by the same subject \( i \), which is exactly what makes this covariance computable-and exactly what a naive independent-samples comparison would throw away.

Worked Example 1: Two Small Classifiers by Hand

Suppose a small validation set has \( m = 3 \) positive subjects and \( n = 3 \) negative subjects. Two classifiers each produce a score for every subject:

Step 1: Compute each classifier's AUC. Classifier 1 places every positive score above every negative score except P3 (0.65) vs N3 (0.55), which it still gets right, so all 9 positive-negative pairs are concordant: \( AUC_1 = 9/9 = 1.000 \). Classifier 2 is messier: P3 (0.70) beats all three negatives, P1 (0.60) beats N3 (0.40) and N1 (0.50, tie counts 0.5) but loses to N2 (0.65), and P2 (0.55) beats N3 (0.40) but loses to N1 (0.50, tie) and N2 (0.65). Counting concordant pairs as 1, ties as 0.5, and discordant pairs as 0 gives 6.5 out of 9 credit: \( AUC_2 = 6.5/9 \approx 0.722 \).

Step 2: Compute structural components \( V^{k}_{10,i} \) for each positive subject (the fraction of the 3 negatives each positive score beats, ties = 0.5):

Step 3: Compute the sample covariance \( S^{10} \) across the three positive subjects. Classifier 1's components are constant at 1.000, so \( S^{10}_{11} = 0 \) and, because any covariance with a zero-variance series is also zero, \( S^{10}_{12} = 0 \) as well. For Classifier 2 alone, \( \bar{V}^{2}_{10} = 0.556 \), giving \( S^{10}_{22} \approx 0.0817 \). An identical component computation on the negative side yields \( S^{01} \). Working through the full \( 2\times2 \) covariance matrix (as the Python example below verifies numerically) gives \( \mathrm{Var}(AUC_1) = 0 \), \( \mathrm{Var}(AUC_2) \approx 0.0206 \), and \( \mathrm{Cov}(AUC_1, AUC_2) = 0 \)-Classifier 1's perfect, constant discrimination on this tiny sample leaves it with zero estimated variance and zero covariance with Classifier 2.

Step 4: Compute the z-statistic.

This gives a two-sided p-value of approximately \( p \approx 0.053 \)-just short of the conventional \( \alpha = 0.05 \) threshold. This tiny, illustrative sample makes the mechanics of the covariance computation transparent, but with only 3 positive and 3 negative subjects it also demonstrates exactly why DeLong's Test needs a reasonably large sample in practice, as Worked Example 2 below shows.

Worked Example 2: A Larger Comparison

Now suppose a diagnostic imaging study evaluates two radiomics models-a single-sequence model and a multi-sequence model-on the same cohort of 200 patients (90 disease-positive, 110 disease-negative). The single-sequence model achieves \( AUC_1 = 0.780 \) and the multi-sequence model achieves \( AUC_2 = 0.845 \), both computed on the identical 200 patients. Working through the full structural- component covariance calculation (shown numerically in the Python section below) yields \( \mathrm{Var}(AUC_1) \approx 0.000922 \), \( \mathrm{Var}(AUC_2) \approx 0.000658 \), and \( \mathrm{Cov}(AUC_1, AUC_2) \approx 0.000401 \)-a positive covariance, as expected, since both models tend to find the same patients easy or hard to classify.

This gives a two-sided p-value of approximately \( p \approx 0.0198 \), below \( \alpha = 0.05 \): the multi-sequence model's AUC is significantly higher than the single-sequence model's on this cohort. Notice that ignoring the covariance term-treating the two AUCs as independent-would inflate the denominator to \( \sqrt{0.000922 + 0.000658} \approx 0.0398 \), giving a smaller \( z \approx 1.633 \) and a non-significant \( p \approx 0.1025 \). Properly modeling the shared-subject correlation is what recovers the statistically significant result here.

Effect Size: AUC Difference & Confidence Interval

Like the z-statistic elsewhere on this site, DeLong's test statistic grows with sample size even for a modest true difference, so it should be paired with an effect size. The natural choice here is simply the AUC difference itself, together with its confidence interval:

For Worked Example 2:

The multi-sequence model's AUC is about 6.5 percentage points higher, with a 95% confidence interval that excludes zero-consistent with the significant p-value above. Reporting the interval alongside the p-value lets readers judge both statistical and practical significance: a significant DeLong's Test with a confidence interval of \( [0.001,\ 0.003] \) is statistically real but may be practically negligible, while one like \( [0.010,\ 0.120] \) suggests a meaningfully better classifier.

Python Example

Scikit-learn does not ship DeLong's Test directly, but it is straightforward to implement from sklearn.metrics.roc_auc_score plus the structural-component covariance shown above. A compact, widely-used implementation looks like this:

import numpy as np
from scipy import stats

def _structural_components(scores, is_positive):
    """Per-subject V10 (positives) and V01 (negatives) placement values."""
    pos = scores[is_positive]
    neg = scores[~is_positive]
    m, n = len(pos), len(neg)
    # psi(x, y): 1 if x>y, 0.5 if x==y, 0 if x 0, 1.0, np.where(diff == 0, 0.5, 0.0))
    v10 = psi.mean(axis=1)   # length m
    v01 = psi.mean(axis=0)   # length n
    auc = v10.mean()
    return auc, v10, v01

def delong_test(y_true, scores_1, scores_2):
    y_true = np.asarray(y_true).astype(bool)
    auc1, v10_1, v01_1 = _structural_components(np.asarray(scores_1), y_true)
    auc2, v10_2, v01_2 = _structural_components(np.asarray(scores_2), y_true)

    m, n = v10_1.shape[0], v01_1.shape[0]

    s10 = np.cov(np.vstack([v10_1, v10_2]), ddof=1)  # 2x2
    s01 = np.cov(np.vstack([v01_1, v01_2]), ddof=1)  # 2x2
    cov = s10 / m + s01 / n                          # 2x2 AUC covariance matrix

    var1, var2, covariance = cov[0, 0], cov[1, 1], cov[0, 1]
    z = (auc1 - auc2) / np.sqrt(var1 + var2 - 2 * covariance)
    p_value = 2 * (1 - stats.norm.cdf(abs(z)))
    return auc1, auc2, z, p_value

# Worked Example 2: 200 patients, single-sequence vs multi-sequence radiomics
rng = np.random.default_rng(42)
y_true = np.array([1] * 90 + [0] * 110)
scores_1 = np.concatenate([rng.normal(0.62, 0.20, 90), rng.normal(0.38, 0.20, 110)])
scores_2 = np.concatenate([rng.normal(0.68, 0.18, 90), rng.normal(0.32, 0.18, 110)])

auc1, auc2, z_stat, p_value = delong_test(y_true, scores_1, scores_2)
print(f"AUC (single-sequence): {auc1:.3f}")
print(f"AUC (multi-sequence):  {auc2:.3f}")
print(f"z-statistic: {z_stat:.3f}")
print(f"p-value: {p_value:.4f}")

Output (illustrative, seeded for reproducibility):

AUC (single-sequence): 0.781
AUC (multi-sequence):  0.847
z-statistic: 2.318
p-value: 0.0205

These figures closely track Worked Example 2's hand-calculated values. You can cross-check either AUC independently with sklearn.metrics.roc_auc_score:

from sklearn.metrics import roc_auc_score

print(f"sklearn AUC 1: {roc_auc_score(y_true, scores_1):.3f}")
print(f"sklearn AUC 2: {roc_auc_score(y_true, scores_2):.3f}")

sklearn AUC 1: 0.781
sklearn AUC 2: 0.847

confirming the structural-component AUCs match the standard Mann-Whitney-based AUC computation exactly, while the covariance-aware z-test above gives the significance test that a bare AUC comparison cannot.

How to Interpret Results

A significant DeLong's Test tells you that the two AUCs differ by more than sampling variability would plausibly produce-one classifier discriminates significantly better than the other on this data. It does not, by itself, tell you the practical size of that improvement; report the AUC difference and its confidence interval (as in the effect size section above) alongside the p-value so readers can judge both statistical and practical significance.

It is also worth being explicit about direction: a significant result only tells you the AUCs differ, not automatically which one is better in a clinically or operationally meaningful way-always report which classifier had the higher AUC alongside the test result, exactly as both worked examples above do.

Checking Assumptions in Practice

Before running DeLong's Test, confirm that both classifiers really were scored on the identical set of subjects against the identical ground-truth labels- applying DeLong's Test to two classifiers evaluated on different test sets will produce a covariance estimate that has no real meaning, since the whole method depends on the pairing.

Next, check your class balance and sample size. Very small samples, or samples with only a handful of positive or negative subjects, make the asymptotic normal approximation shaky-consider the tiny sample used in Worked Example 1 above as a cautionary illustration rather than a template. Finally, check for large blocks of tied scores, which can happen with low-resolution predicted probabilities or heavily quantized biomarker values; heavy ties reduce the effective information content the test statistic relies on.

Advantages

• Purpose-built for paired AUC comparison, correctly modeling the correlation between two classifiers evaluated on the same subjects instead of treating them as independent.
• Closed-form and non-parametric-derived directly from Mann-Whitney U-statistic theory, so it does not require bootstrapping or resampling to estimate variance and covariance.
• Computationally efficient, especially with the vectorized structural-component formulation, making it practical even for large evaluation sets with thousands of subjects.
• Pairs naturally with an intuitive effect size (the AUC difference and its confidence interval) for reporting practical significance alongside the p-value.

Limitations

• Only applies to a strictly binary ground truth-comparing AUCs for a multi-class or ordinal outcome requires a different, multi-class extension of ROC analysis.
• Relies on an asymptotic normal approximation that can be unreliable with very small samples or heavily imbalanced classes-Worked Example 1 above illustrates the borderline behavior that can result.
• Compares exactly two curves at a time-extending to three or more classifiers requires either pairwise testing with a multiple-comparisons correction or a multivariate extension of the covariance approach.
• Like other significance tests, a significant result establishes a statistically real AUC difference but not its clinical or operational magnitude, making an accompanying effect size such as the AUC difference essential.

When NOT to Use It

• Independent-samples AUC test: use instead when the two ROC curves come from independent subjects-such as two separate test cohorts-rather than the same subjects scored twice.
• Bootstrap AUC comparison: a reasonable alternative when you want an empirical distribution of the AUC difference without relying on the asymptotic normal approximation, at the cost of extra computation.
• McNemar's Test: use instead when you only care about comparing two classifiers' binary decisions at a fixed threshold, rather than their full-curve discrimination measured by AUC.
• Chi-Square Test of Independence: use instead when you are testing association between two categorical variables rather than comparing continuous-score discrimination performance.

DeLong's Test vs Bootstrap vs McNemar's vs Independent AUC Tests

17.1 DeLong's Test vs Bootstrap AUC Comparison

17.2 DeLong's Test vs McNemar's Test

17.3 DeLong's Test vs Independent-Samples AUC Test

Common Misconceptions

• "A higher AUC number automatically means a significantly better classifier." No-a numerically higher AUC could easily fall within the range of sampling noise; only a formal test like DeLong's, with an accompanying p-value and confidence interval, establishes statistical significance.
• "I can just run two independent t-tests or z-tests on the two AUCs." Not correctly- doing so ignores the covariance between the two AUCs that arises from evaluating both classifiers on the same subjects, which typically understates statistical significance, as shown explicitly in Worked Example 2 above.
• "DeLong's Test requires bootstrapping to work." No-this is precisely what distinguishes it from the bootstrap approach to AUC comparison; DeLong's method is closed-form, derived analytically from U-statistic theory.
• "A significant DeLong's result proves my new model is clinically useful." No-the test only establishes that the AUCs differ statistically; clinical or operational usefulness depends on the size of that difference, the deployment context, and metrics beyond AUC alone.
• "DeLong's Test works fine with any sample size." It relies on an asymptotic approximation that becomes less reliable with very small or heavily imbalanced samples-see Worked Example 1's borderline result as a caution.

Interview Questions

1. Walk through why DeLong's Test needs a covariance term between the two AUCs, and what would happen to the test statistic if that term were dropped.
2. Explain what a structural component \( V^{k}_{10,i} \) represents, and how averaging it recovers the AUC.
3. Why does DeLong's Test require both classifiers to be evaluated on the same set of subjects?
4. What does it mean for two classifiers to have a positive covariance between their AUCs, and why is that typically the case in practice?
5. Under what circumstances would you prefer a bootstrap-based AUC comparison over DeLong's Test?
6. How would you extend DeLong's Test to compare three or more correlated ROC curves at once?
7. Explain the relationship between the AUC and the Mann-Whitney U-statistic, and why that relationship is central to DeLong's derivation.
8. Why can a significant DeLong's Test result not, by itself, be used to claim a model is ready for deployment?
9. What role do ties in the prediction scores play in the structural-component computation?
10. How would you decide whether a McNemar's Test or a DeLong's Test is the more appropriate way to compare two classifiers?

Frequently Asked Questions

• What is DeLong's Test used for? +
  DeLong's Test is used to determine whether two ROC curves-most often two classifiers, diagnostic tests, or biomarkers evaluated on the same set of subjects-have significantly different areas under the curve (AUC), while correctly accounting for the correlation between the two AUCs that arises from sharing the same evaluation data.
• What is the formula for DeLong's Test? +
  Each classifier's AUC is re-expressed as a sum of per-subject structural components tied to the Mann-Whitney U statistic. From these components you compute the variance of each AUC and the covariance between the two AUCs. The test statistic is z = (AUC_1 - AUC_2) / sqrt(Var(AUC_1) + Var(AUC_2) - 2*Cov(AUC_1, AUC_2)), which follows approximately a standard normal distribution under the null hypothesis that the two AUCs are equal.
• What is the difference between DeLong's Test and just comparing two AUC numbers? +
  A raw AUC difference tells you nothing about whether that gap is statistically meaningful, and naively treating the two AUCs as independent ignores the correlation induced by scoring the same subjects with both classifiers. DeLong's Test resolves both problems by explicitly computing the covariance between the two AUCs and folding it into the variance of their difference.
• What are the assumptions of DeLong's Test? +
  Both classifiers (or diagnostic tests) must be scored on the same subjects with the same binary ground-truth labels-a paired design-the outcome must be binary, the sample should be large enough for the asymptotic normal approximation of AUC to hold, and the underlying prediction scores should be continuous or near-continuous rather than dominated by ties.
• How do you interpret a DeLong's Test p-value? +
  If the p-value is below your chosen significance level (commonly alpha = 0.05), you reject the null hypothesis and conclude the two AUCs differ significantly-one classifier discriminates better than the other on this data. If the p-value is at or above alpha, there is not enough evidence that the two AUCs are different.
• What should I use instead of DeLong's Test for unpaired classifiers? +
  When the two ROC curves come from independent samples of subjects rather than the same subjects, use an unpaired (independent-samples) AUC comparison test instead, which uses Var(AUC_1) + Var(AUC_2) with no covariance term, since there is no shared subject-level correlation to account for.
• Does DeLong's Test require bootstrapping? +
  No. DeLong's Test is a closed-form, non-parametric procedure based on U-statistic theory-it does not require resampling. This is one of its main advantages over the bootstrap approach to comparing AUCs, which needs thousands of resamples to estimate the same covariance structure.
• Can DeLong's Test be used for more than two classifiers? +
  The classic DeLong's Test compares exactly two correlated ROC curves at a time. To compare three or more classifiers, you either run DeLong's Test pairwise with a multiple-comparisons correction such as Bonferroni or Holm, or use a multivariate extension of the same covariance-matrix approach that tests all curves simultaneously.

Key Takeaways

• DeLong's Test checks whether two correlated ROC AUCs-typically from two classifiers evaluated on the same subjects-differ significantly, by explicitly modeling the covariance between them.
• It is ideal for model comparison, biomarker evaluation, and any paired classifier or diagnostic-test comparison-where treating the two AUCs as independent would understate statistical significance.
• Core assumptions: paired evaluation on identical subjects and ground truth, a strictly binary outcome, independence across subjects, a sufficiently large sample for the asymptotic approximation, and reasonably continuous prediction scores.
• The test is closed-form-built from per-subject structural components derived from the Mann-Whitney U-statistic-so it needs no bootstrapping, and standard implementations pair it with the AUC difference and confidence interval for a complete effect size picture.
• If \( p < 0.05 \), the two AUCs differ significantly; if \( p \geq 0.05 \), there isn't enough evidence of a real difference-but sample size and class balance, not just the p-value, determine how much to trust the underlying normal approximation.
• Always report both AUCs, their difference, and its confidence interval alongside the p-value, and treat very small or heavily imbalanced samples with caution, as demonstrated in the worked examples above.

DeLong's Test earns its place as the standard tool for comparing two classifiers' AUCs precisely because so much real-world model evaluation-machine learning benchmarks, diagnostic biomarker studies, radiomics model comparisons-scores multiple candidate models on the very same evaluation set rather than on independent cohorts. Its logic is refreshingly principled: decompose each AUC into per-subject structural components, use those components to compute how strongly the two AUCs' errors move together, and measure how large the AUC gap is relative to that jointly-estimated uncertainty.

The two worked examples above, a tiny 6-subject illustration and a larger 200-patient radiomics comparison, show both ends of this test's range, and a useful lesson in between: ignoring the covariance term-treating the two AUCs as independent-can turn a genuinely significant result into a non-significant one, exactly as Worked Example 2 demonstrates. Reporting the z-statistic and p-value alongside the AUC difference and its confidence interval, and treating small or imbalanced samples with the same caution shown in Worked Example 1, gives a complete and honest picture that a bare AUC comparison by itself cannot.