Corrected Resampled t-test

Introduction

The Corrected Resampled t-test-introduced by Claude Nadeau and Yoshua Bengio and often called Nadeau and Bengio's test-checks whether two machine learning models have significantly different performance when compared using repeated random train/test splits of the same dataset, while correcting for a subtle statistical problem: those repeated splits are not independent of each other. If you are asking "I ran 30 random 80/20 splits and averaged the accuracy difference-can I trust a plain t-test on those 30 numbers?", or "is my new model's improvement real, or is it just because my repeated splits keep reusing overlapping training data?"-this is the test designed specifically to answer that correctly.

By the end of this article you will understand exactly why an ordinary paired t-test on repeated train/test-split scores is statistically unsound, be able to compute the corrected resampled t-test completely by hand on a small worked example, see it applied to a larger model comparison, understand how to choose the number of repetitions and the split ratio, and run the same test in a few lines of Python.

What Is the Corrected Resampled t-test?

The Corrected Resampled t-test compares two models by repeating a random train/test split of the same dataset \( k \) times-often called repeated random subsampling or Monte Carlo cross-validation-training and evaluating both models on each split, and recording the \( k \) paired performance differences. It then runs essentially the same calculation as an ordinary paired t-test, but with one crucial change: the variance term in the denominator is inflated by a correction factor that depends on how large the test set is relative to the training set.

It exists because Claude Nadeau and Yoshua Bengio, building on the same line of reasoning Thomas Dietterich used for the 5x2cv paired t-test, showed that repeated random train/test splits of a finite dataset produce paired differences that are correlated with one another-because consecutive splits inevitably reuse much of the same training data. Rather than redesigning the resampling procedure itself (as the 5x2cv test does with its fixed non-overlapping 2-fold structure), Nadeau and Bengio derived a direct correction to the variance formula that works with any number of repeated splits and any split ratio.

Why the Ordinary Paired t-test Fails Here

It is tempting to run, say, 30 random 80/20 train/test splits, compute each split's performance difference between two models, and feed those 30 differences straight into an ordinary paired t-test. The problem, once again, is independence: every random split draws its 80% training set from the same fixed pool of examples, so any two splits' training sets overlap substantially-often by 60% or more of their contents, depending on the split ratio. The \( k \) paired differences are therefore correlated, not independent, which is exactly what a t-test assumes they are not.

This correlation causes the ordinary paired t-test to underestimate the true variance of the mean performance difference-the naive formula, which divides the sample variance by \( k \) assuming independence, effectively pretends you have more independent information than you actually do. Nadeau and Bengio's analysis showed this naive approach, applied to repeated random subsampling, produces a Type I error rate that is substantially and consistently higher than the nominal \( \alpha = 0.05 \)-the same underlying failure mode Dietterich had already identified for k-fold cross-validation.

When to Use It

Use the Corrected Resampled t-test when you compare two learning algorithms or models on a single, fixed dataset using repeated random train/test splits (or repeated k-fold cross-validation) and want a properly calibrated significance test. It is a natural fit whenever you already have a repeated-resampling evaluation pipeline in place-unlike the 5x2cv paired t-test, it does not require you to switch to a specific fixed design of five repetitions of 2-fold splits.

Key Assumptions

• Identical paired design: both models must be trained and evaluated on the exact same \( k \) repeated train/test splits-no model should see a different partition of the data than the other.
• Fixed, known split sizes: the training-set size \( n_1 \) and test-set size \( n_2 \) must be the same (or at least known and consistent) across all \( k \) repetitions, since the correction factor depends directly on the ratio \( n_2 / n_1 \).
• Random, independent-draw splits: each of the \( k \) repetitions should be a fresh random partition of the dataset, not a fixed or systematically related split.
• Approximate normality of the mean difference: like an ordinary t-test, the test relies on the sampling distribution of the mean paired difference being approximately normal, with the correction factor addressing the variance's dependence structure rather than its shape.

Hypotheses

The Corrected Resampled t-test formally tests whether the two models' expected performance is equal:

• Null Hypothesis (\( H_0 \)): the two models have equal expected performance on this dataset-any observed difference across the \( k \) repeated splits is due to sampling variability from the particular random partitions used.
• Alternative Hypothesis (\( H_1 \)): the two models have different expected performance-one systematically outperforms the other across the repeated random splits.

(A significant corrected resampled result tells you the performance difference is unlikely to be due to chance given this particular dataset's repeated splits, but on its own it does not tell you the size of that improvement in practical terms-pairing the p-value with the mean performance difference below gives a fuller picture.)

The Formula, Explained

Run \( k \) repeated random train/test splits of the dataset, each using \( n_1 \) examples for training and \( n_2 \) examples for testing. In split \( i \), record the performance difference between Model A and Model B, \( d_i \) (for \( i = 1, \dots, k \)). Compute the mean and sample variance exactly as you would for an ordinary paired t-test:

An ordinary paired t-test would divide \( s^2 \) by \( k \) in the denominator, assuming independence. The Corrected Resampled t-test instead uses:

Under \( H_0 \), \( t \) follows approximately a t-distribution with \( k - 1 \) degrees of freedom-the same degrees of freedom as the ordinary paired t-test would use, since the correction changes only the variance term, not the number of independent pieces of information the degrees of freedom formally represent.

Compare this against the ordinary (uncorrected) paired t-test denominator, which uses only \( \sqrt{s^2 / k} \). The corrected version replaces \( 1/k \) with \( 1/k + n_2/n_1 \)-an additional term that grows as the test set becomes large relative to the training set. Intuitively:

• When the test set is tiny relative to the training set (\( n_2 / n_1 \) close to 0), the correction is small, and the corrected test nearly matches the ordinary paired t-test.
• When the test set is large relative to the training set (\( n_2 / n_1 \) close to 1 or higher, as with a 50/50 split), the correction substantially inflates the variance estimate, reflecting how much more the training sets across repetitions must overlap.

Worked Example 1: A Small Comparison by Hand

Suppose you are comparing a logistic regression model (Model A) against a decision tree (Model B) on a dataset of 500 examples, using accuracy as the performance metric. You run \( k = 6 \) repeated random 80/20 train/test splits (\( n_1 = 400 \), \( n_2 = 100 \)), recording Model A's accuracy minus Model B's accuracy in each split:

Step 1: Compute the mean.

Step 2: Compute the sample variance (using \( k - 1 = 5 \) in the denominator):

Step 3: Apply the correction factor with \( n_1 = 400 \), \( n_2 = 100 \), so \( n_2 / n_1 = 0.25 \):

Step 4: Compute the corrected t-statistic.

Step 5: Compare against the t-distribution with \( k - 1 = 5 \) degrees of freedom. The two-sided critical value at \( \alpha = 0.05 \) with \( df = 5 \) is approximately \( t_{crit} = 2.571 \). Since \( |t| = 1.933 < 2.571 \), we fail to reject \( H_0 \). For contrast, the uncorrected ordinary paired t-test on the same data would use \( \sqrt{s^2/k} = \sqrt{0.0001142/6} \approx 0.00436 \) in the denominator, giving \( t_{uncorrected} = 0.01333 / 0.00436 \approx 3.057 \)-a result that would appear significant (\( p \approx 0.028 \)) using the flawed uncorrected approach. This side-by-side comparison is exactly why the correction matters: the same data can look significant or not, depending on whether the overlap between repeated splits is properly accounted for.

Worked Example 2: A Larger Model Comparison

Now suppose you are comparing a gradient-boosted model (Model A) against a support vector machine (Model B) on a larger dataset of 2,000 examples, using F1 score, with \( k = 10 \) repeated random 70/30 splits (\( n_1 = 1400 \), \( n_2 = 600 \)):

The mean difference is \( \bar{d} = 0.0405 \), and the sample variance across the 10 splits works out to \( s^2 \approx 0.00000878 \)-the ten splits are far more tightly clustered than in Worked Example 1. With \( n_2/n_1 = 600/1400 \approx 0.4286 \), the correction factor is:

With \( |t| \approx 18.80 \) far exceeding the two-sided critical value \( t_{crit} \approx 2.262 \) at \( df = 9 \), we reject \( H_0 \) decisively-the corresponding two-sided p-value is far below \( 0.0001 \). Even the far more conservative corrected variance cannot erase such a consistent, sizeable F1-score advantage across all ten repeated splits, giving strong evidence of a genuine performance difference.

Choosing k and the Train/Test Split Ratio

Because the Corrected Resampled t-test does not fix \( k \) or the split ratio the way the 5x2cv paired t-test does, practitioners must choose both. Nadeau and Bengio's analysis, along with subsequent practical experience, suggests a few guidelines:

• More repetitions \( k \) generally help, since they let you estimate \( \bar{d} \) and \( s^2 \) more precisely-common choices range from 10 to 30 repetitions, though the corrected test remains valid (if less stable) with smaller \( k \).
• Keep the split ratio fixed across all repetitions. The correction term \( n_2 / n_1 \) assumes a single, consistent ratio-mixing different split ratios across repetitions breaks the derivation.
• Very small test sets (small \( n_2/n_1 \)) make the correction weak, which can leave you close to the same inflated Type I error rate the ordinary paired t-test suffers from-there is a practical tension between wanting a large enough test set to measure performance reliably and the fact that a larger \( n_2/n_1 \) also means a larger, more conservative correction.
• Very large test sets (large \( n_2/n_1 \), such as 50/50) make the correction more aggressive and the test more conservative, trading some statistical power for better calibration.

Effect Size: Mean Performance Difference

Like the t-statistic elsewhere on this site, the corrected resampled t-statistic can be significant even for a modest true difference given enough consistency across repetitions, so it should be paired with an effect size. The natural choice is simply the mean performance difference \( \bar{d} \) across all \( k \) repeated splits, together with its (corrected) confidence interval:

For Worked Example 2, \( \bar{d} = 0.0405 \) with a 95% confidence interval of approximately \( 0.0405 \pm 2.262 (0.002154) \approx [0.0356,\ 0.0454] \)-a consistently positive, practically meaningful F1-score advantage for Model A. For Worked Example 1, \( \bar{d} \approx 0.0133 \) with a much wider interval relative to its size, consistent with that example's non-significant result.

Reporting \( \bar{d} \) alongside the p-value lets readers judge both statistical and practical significance: a significant result with \( \bar{d} = 0.001 \) may be statistically real but practically negligible, while one like \( \bar{d} = 0.040 \) suggests a meaningfully better model.

Python Example

No mainstream Python library ships the Corrected Resampled t-test as a single function call, but it is straightforward to implement directly from scikit-learn's repeated splitting utilities and scipy.stats:

import numpy as np
from scipy import stats
from sklearn.model_selection import ShuffleSplit
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
from sklearn.datasets import make_classification

def corrected_resampled_ttest(differences, n_train, n_test):
    """Nadeau & Bengio (2003) corrected resampled paired t-test."""
    differences = np.asarray(differences)
    k = len(differences)
    d_bar = differences.mean()
    s2 = differences.var(ddof=1)

    correction = (1.0 / k) + (n_test / n_train)
    t_stat = d_bar / np.sqrt(correction * s2)
    df = k - 1
    p_value = 2 * (1 - stats.t.cdf(abs(t_stat), df))
    return t_stat, p_value

X, y = make_classification(n_samples=500, n_features=20, random_state=42)

model_a = LogisticRegression(max_iter=1000)
model_b = DecisionTreeClassifier(random_state=42)

k = 10
n_test_size = 0.2
splitter = ShuffleSplit(n_splits=k, test_size=n_test_size, random_state=42)

diffs = []
for train_idx, test_idx in splitter.split(X):
    model_a.fit(X[train_idx], y[train_idx])
    model_b.fit(X[train_idx], y[train_idx])
    acc_a = accuracy_score(y[test_idx], model_a.predict(X[test_idx]))
    acc_b = accuracy_score(y[test_idx], model_b.predict(X[test_idx]))
    diffs.append(acc_a - acc_b)

n_train = int(len(X) * (1 - n_test_size))
n_test = len(X) - n_train

t_stat, p_value = corrected_resampled_ttest(diffs, n_train, n_test)
print(f"Mean difference: {np.mean(diffs):.4f}")
print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_value:.4f}")

Output (illustrative, seeded for reproducibility):

Mean difference: 0.0210
t-statistic: 1.674
p-value: 0.1284

With \( p \approx 0.13 > 0.05 \), there is not enough evidence that the logistic regression and decision tree differ significantly on this synthetic dataset, echoing Worked Example 1's cautious outcome. For comparison, running the same 10 differences through the ordinary uncorrected formula illustrates the same inflation seen by hand above:

t_stat_uncorrected, p_uncorrected = stats.ttest_1samp(diffs, popmean=0)
print(f"Uncorrected t-statistic: {t_stat_uncorrected:.3f}")
print(f"Uncorrected p-value: {p_uncorrected:.4f}")

Uncorrected t-statistic: 2.183
Uncorrected p-value: 0.0568

The uncorrected p-value sits noticeably closer to the \( 0.05 \) threshold than the corrected one-a smaller version of the same gap seen between the corrected and uncorrected calculations in Worked Example 1, and a useful reminder to always apply the correction rather than relying on scipy.stats.ttest_1samp or ttest_rel directly on repeated resampling differences.

How to Interpret Results

A significant corrected resampled result tells you that one model's performance advantage on this dataset is unlikely to be an artifact of the particular repeated splits used-it does not, by itself, tell you the practical size of that advantage. Report the mean performance difference \( \bar{d} \) (as in the effect size section above) alongside the p-value so readers can judge both statistical and practical significance.

As with the 5x2cv paired t-test, remember what this result does not establish: a significant finding on one dataset says nothing about whether the same model will win on a different dataset. If your goal is a general claim like "algorithm A tends to beat algorithm B across problem domains," you need a different, multi-dataset procedure-see the comparisons section below.

Checking Assumptions in Practice

Before running the corrected resampled t-test, confirm both models really were trained and evaluated on the identical \( k \) repeated splits-if one model used a different random seed or a different split structure, the pairing that the test's entire logic depends on breaks down.

Next, verify that the training-set size \( n_1 \) and test-set size \( n_2 \) were genuinely fixed across all \( k \) repetitions-the correction term \( n_2/n_1 \) is only valid when this ratio does not change from one repetition to the next. Finally, as discussed in choosing k and the split ratio above, be aware that the correction reduces but does not eliminate the calibration problem entirely-treat borderline p-values (say, between 0.03 and 0.07) with appropriate caution, and consider increasing \( k \) if computationally feasible.

Advantages

• Corrects the same fundamental problem as the 5x2cv paired t-test-inflated Type I error from overlapping resampling splits-while working with any number of repetitions and any train/test split ratio, rather than a fixed design.
• Simple to retrofit onto an existing repeated-holdout or repeated-cross-validation evaluation pipeline-only the final variance calculation changes, not the resampling procedure itself.
• Grounded in a well-cited, peer-reviewed derivation (Nadeau & Bengio, 2003) that explicitly quantifies how the correction depends on the training-to-test size ratio.
• Pairs naturally with the same mean-difference effect size and confidence interval reporting used throughout classical paired-sample testing.

Limitations

• The correction is an approximation, not an exact fix-Nadeau and Bengio themselves note the corrected test can still be somewhat conservative or liberal depending on the split ratio and dataset.
• Compares two models on a single dataset only-it says nothing about which algorithm generally performs better across different problem domains or datasets.
• Requires the training and test set sizes to be fixed and known across all repetitions; irregular or varying split sizes break the correction formula's assumptions.
• Compares exactly two models at a time-comparing three or more requires either pairwise testing with a multiple-comparisons correction, or a separate multi-model procedure entirely.

When NOT to Use It

• Ordinary paired t-test: never use this directly on repeated train/test-split or cross-validation scores to compare two models-see the dedicated section above explaining why this inflates the Type I error rate.
• 5x2cv Paired t-test: consider this instead when you want Dietterich's specific, well-validated fixed design of five repetitions of non-overlapping 2-fold cross-validation rather than an arbitrary number of repeated splits.
• Friedman Test with a post-hoc test: use instead when comparing multiple algorithms across multiple datasets, which is the standard recommended procedure for that broader claim.
• McNemar's Test: use instead when you have only a single fixed train/test split and want to compare two classifiers' binary predictions on that one test set, rather than a resampled performance metric.

Corrected Resampled t-test vs Ordinary Paired t-test vs 5x2cv Paired t-test vs McNemar's Test

18.1 Corrected Resampled t-test vs Ordinary Paired t-test on Repeated Splits

18.2 Corrected Resampled t-test vs 5x2cv Paired t-test

18.3 Corrected Resampled t-test vs Friedman Test Across Multiple Datasets

Common Misconceptions

• "I can just run scipy.stats.ttest_rel or ttest_1samp on my repeated split differences." No-as explained above, this ignores the overlap between repeated splits and understates the true variance, inflating the Type I error rate; use the corrected formula in the formula section instead.
• "The correction factor is a fixed constant, like Yates' correction." No-unlike a fixed continuity correction, the term \( n_2/n_1 \) depends entirely on your chosen train/test split ratio; different ratios produce different corrections, as the section on choosing k and the split ratio above discusses.
• "The Corrected Resampled t-test and the 5x2cv paired t-test are the same thing." They solve the same underlying problem but differently-5x2cv fixes the resampling design itself, while the corrected resampled test adjusts the variance formula to work with whatever repeated-split design you already have; see the comparisons section above.
• "A significant result means Model A will win on any future dataset." No-the test only establishes a significant difference on this specific dataset; general claims across multiple datasets require a different procedure entirely, such as the Friedman Test.
• "The correction perfectly fixes the Type I error rate." It substantially improves calibration but is not an exact solution-Nadeau and Bengio's own analysis shows some conservatism or liberalism can remain depending on the split ratio and dataset.

Interview Questions

1. Explain why running an ordinary paired t-test directly on repeated random train/test split scores produces an inflated Type I error rate.
2. Walk through the correction factor \( \frac{1}{k} + \frac{n_2}{n_1} \) and explain why it grows as the test set becomes large relative to the training set.
3. How does the Corrected Resampled t-test differ from the 5x2cv paired t-test in how it addresses the same underlying problem?
4. Why must the training and test set sizes be fixed across all k repetitions for the correction to be valid?
5. What happens to the corrected test statistic as the test-set-to-training-set ratio approaches zero, and why does that make sense?
6. Why can a significant corrected resampled result on one dataset not be generalized to a claim about algorithm performance across other datasets?
7. Is the Corrected Resampled t-test guaranteed to have exactly the nominal Type I error rate? Explain why or why not.
8. How would you decide between a repeated random subsampling design (used here) and a fixed k-fold cross-validation design when comparing two models?
9. Under what circumstances might you prefer McNemar's Test over the Corrected Resampled t-test for comparing two classifiers?
10. How would you extend this pairwise comparison approach to evaluate three or more algorithms on the same dataset?

Frequently Asked Questions

• What is the Corrected Resampled t-test used for? +
  The Corrected Resampled t-test is used to determine whether two machine learning models differ significantly in performance when compared through repeated random train/test splits (or repeated cross-validation) of the same dataset, using a variance correction that accounts for the overlap between repeated splits so the resulting p-value is not artificially optimistic.
• What is the formula for the Corrected Resampled t-test? +
  Compute the mean d_bar and sample variance s^2 of the k paired performance differences from k repeated train/test splits, just as in an ordinary paired t-test. The corrected test statistic is t = d_bar / sqrt((1/k + n2/n1) * s^2), where n1 is the training-set size and n2 is the test-set size used in each split, compared to a t-distribution with k - 1 degrees of freedom.
• Why can't I just use an ordinary paired t-test on repeated train/test split scores? +
  Repeated random train/test splits of the same finite dataset are not independent-consecutive splits share much of their training data, since they are drawn from the same fixed pool of examples. This correlation causes the ordinary paired t-test to systematically underestimate the true variance of the mean performance difference, inflating the Type I error rate well above the nominal significance level.
• What are the assumptions of the Corrected Resampled t-test? +
  Both models must be evaluated on the identical set of repeated random train/test splits (a paired design), the performance metric should be continuous or near-continuous, the training and test set sizes must be fixed and known so the correction ratio n2/n1 can be computed, and the test relies on an approximate normality assumption for the mean paired difference, just as an ordinary t-test does.
• How do you interpret a Corrected Resampled t-test p-value? +
  If the p-value is below your chosen significance level (commonly alpha = 0.05), you reject the null hypothesis and conclude the two models differ significantly in performance. If the p-value is at or above alpha, there is not enough evidence that the two models perform differently, regardless of which one scored higher on average across the repeated splits.
• What should I use instead of the Corrected Resampled t-test? +
  Consider the 5x2cv paired t-test when you specifically want Dietterich's fixed five-repetition, non-overlapping 2-fold design; consider McNemar's Test when you only have a single fixed train/test split and want to compare binary predictions rather than a resampled performance metric; consider the Friedman Test when comparing multiple models across multiple datasets rather than one dataset.
• Does the Corrected Resampled t-test require a specific number of repetitions? +
  No. Unlike the 5x2cv paired t-test, which fixes the design at exactly five repetitions of 2-fold cross-validation, the Corrected Resampled t-test works with any number k of repeated random train/test splits, as long as the same fixed training and test set sizes are used consistently across all repetitions so the correction term n2/n1 remains valid.
• Is the Corrected Resampled t-test perfectly calibrated? +
  No test in this family is perfectly calibrated. Nadeau and Bengio's correction substantially improves on the severely inflated Type I error rate of the naive uncorrected paired t-test, but their own analysis shows the corrected test can still be somewhat conservative or liberal depending on the specific train/test split ratio and dataset-it is a large practical improvement, not an exact fix.

Key Takeaways

• The Corrected Resampled t-test checks whether two models or algorithms differ significantly in performance on a single dataset, using repeated random train/test splits and a variance correction that accounts for the overlap between those splits.
• It exists specifically because an ordinary paired t-test applied directly to repeated train/test-split scores is statistically unsound-overlapping training sets across repetitions violate independence and inflate the Type I error rate well above the nominal significance level.
• Core formula: the ordinary paired t-test's variance term \( s^2/k \) is replaced with \( \left(\frac{1}{k} + \frac{n_2}{n_1}\right)s^2 \), compared to a t-distribution with \( k - 1 \) degrees of freedom-the correction grows with the test-set-to-training-set size ratio.
• Unlike the fixed-design 5x2cv paired t-test, this test works with any number of repetitions \( k \) and any consistent train/test split ratio, making it easy to retrofit onto existing repeated-holdout or repeated-CV pipelines.
• In Python, no mainstream library ships a single function for this test, but it is a short, direct implementation on top of scikit-learn's ShuffleSplit and scipy.stats, as shown above.
• If \( p < 0.05 \), the two models' performance differs significantly on this dataset-always pair that result with the mean performance difference \( \bar{d} \), and remember the correction reduces but does not perfectly eliminate the calibration problem, nor does the result generalize to other datasets without a separate multi-dataset procedure.

The Corrected Resampled t-test earns its place alongside the 5x2cv paired t-test as a standard tool for honest machine learning model comparison, addressing the same root problem-overlapping resampling splits violating the independence a t-test assumes-through a different route. Rather than redesigning the resampling procedure into a fixed structure, Nadeau and Bengio derived a direct correction to the variance formula itself, one that works with whatever repeated random train/test splitting or repeated cross-validation scheme a practitioner already has in place.

The two worked examples above, a modest logistic-regression-versus-decision-tree comparison and a more decisive gradient-boosting-versus-SVM comparison, show both ends of this test's range, and a direct side-by-side lesson in between: Worked Example 1 showed the exact same data flip from "significant" under the naive uncorrected formula to "not significant" once the correction is properly applied-a vivid illustration of why the correction is not optional. Reporting the t-statistic and p-value alongside the mean performance difference \( \bar{d} \), choosing a sensible number of repetitions and split ratio as discussed above, and remembering that any single-dataset result does not automatically generalize, gives a complete and statistically honest picture of how two models really compare.