5x2 Cross-Validation Paired t-test

Introduction

The 5x2 Cross-Validation Paired t-test-often written 5x2cv paired t-test and also known as Dietterich's test-checks whether two machine learning models or algorithms have significantly different performance on a given dataset, while correctly accounting for a problem most practitioners never think twice about: cross-validation folds are not independent of one another. If you are asking questions like "is my new model actually better than the baseline, or is the accuracy gap just noise from this particular train/test split?", or "can I trust the p-value from a t-test I ran across my 10-fold cross-validation scores?"-this is the test built specifically to answer that second question correctly.

By the end of this article you will understand exactly why an ordinary paired t-test on cross-validation scores is statistically unsound, be able to compute the 5x2cv test completely by hand on a small worked example, see it applied to a larger model comparison, understand its refinement-the combined 5x2cv F test-and run the same test in a few lines of Python with mlxtend.

What Is the 5x2cv Paired t-test?

The 5x2cv paired t-test compares two models by repeating 2-fold cross-validation five times on the same dataset, using a different random fold split each time. In each repetition, both models are trained on one half of the data and tested on the other, then trained on the second half and tested on the first-producing two paired performance differences per repetition, and ten in total across all five repetitions. These ten differences are then combined into a single test statistic engineered specifically to have an approximately correct Type I error rate.

It exists because Thomas Dietterich, in a widely cited 1998 paper, demonstrated that the seemingly obvious approach-running k-fold cross-validation once and treating the k paired score differences as independent observations for an ordinary paired t-test-produces a test with a severely inflated false-positive rate. The 5x2cv design fixes this by using only two folds per repetition (so the two training sets in a repetition are non-overlapping) and by repeating the whole process five times with fresh random splits, then combining the results in a specific, deliberately asymmetric way.

Why the Ordinary Paired t-test Fails Here

It is tempting to run, say, 10-fold cross-validation once, compute each fold's accuracy difference between two models, and feed those ten differences straight into an ordinary paired t-test. The problem is independence: in k-fold cross-validation, every fold's training set shares the vast majority of its examples with every other fold's training set-with 10-fold CV on \( N \) examples, any two folds' training sets overlap in roughly \( 80\% \) of their data. The paired differences across folds are therefore highly correlated, not independent, which is exactly what a t-test assumes.

This correlation systematically underestimates the true variance of the performance difference, which in turn makes the t-statistic artificially large and the p-value artificially small. Dietterich's simulations showed this naive approach can produce a Type I error rate several times higher than the nominal \( \alpha = 0.05 \)-in other words, the "ordinary paired t-test on CV folds" approach reports statistically significant differences between two models far more often than it should, even when the two models are truly identical.

When to Use It

Use the 5x2cv paired t-test when you want to compare two learning algorithms or models on a single, fixed dataset and need a statistically defensible p-value-rather than just an eyeballed comparison of mean cross-validation scores. It is the standard choice when computational budget allows ten total model-fitting runs (two folds times five repetitions, for each of the two models) and you want a properly calibrated significance test rather than an informal comparison.

Key Assumptions

• Identical paired design: both models must be trained and evaluated on the exact same five repetitions of 2-fold cross-validation, using identical fold splits-no model should see a different partition of the data than the other.
• Non-overlapping folds within a repetition: within each of the five repetitions, the two folds must be a genuine 50/50 split with no overlap, which is what keeps the two training sets in that repetition independent of each other.
• Fresh random splits across repetitions: each of the five repetitions must use a newly randomized 2-fold split, not the same split repeated five times, or the ten differences collapse toward just two.
• Approximate normality of the null distribution: like an ordinary t-test, the test relies on the sampling distribution of the statistic being approximately t-distributed with 5 degrees of freedom under \( H_0 \)-a property established by Dietterich's specific choice of numerator and denominator, not by the raw scores themselves being normal.

Hypotheses

The 5x2cv paired t-test formally tests whether the two models' expected performance is equal:

• Null Hypothesis (\( H_0 \)): the two models have equal expected performance on this dataset-any observed difference across the ten paired folds is due to sampling variability from the particular train/test splits used.
• Alternative Hypothesis (\( H_1 \)): the two models have different expected performance- one systematically outperforms the other across the repeated 2-fold splits.

(A significant 5x2cv result tells you the performance difference is unlikely to be due to chance given this particular dataset's cross-validation splits, but on its own it does not tell you the size of that improvement in practical terms-pairing the p-value with the mean performance difference below gives a fuller picture.)

The Formula, Explained

Run 2-fold cross-validation five separate times (\( i = 1, \dots, 5 \)), each time with a fresh random 50/50 split of the dataset. In repetition \( i \), train both models on fold 1 and test on fold 2 to get performance difference \( p_i^{(1)} \) (Model A's score minus Model B's score), then train on fold 2 and test on fold 1 to get \( p_i^{(2)} \). This gives ten paired differences in total: two per repetition, five repetitions.

For each repetition \( i \), compute the mean and the sample variance of its two differences:

The 5x2cv paired t-test statistic is then:

Under \( H_0 \), Dietterich showed this statistic follows approximately a t-distribution with 5 degrees of freedom. Two design choices stand out and explain the test's structure:

1. The numerator uses only \( p_1^{(1)} \)-the very first paired difference from the very first repetition-rather than averaging all ten differences. Averaging all ten would reintroduce strong correlation into the numerator's sampling distribution, since every repetition's two folds share the same held-out dataset.
2. The denominator averages the within-repetition variance \( s_i^2 \) across all five repetitions, giving a more stable variance estimate than any single repetition could provide on its own, without needing all ten differences to be independent.

This asymmetric construction is precisely what Dietterich's simulation study validated as keeping the Type I error rate close to the nominal significance level-something the naive "average all k differences, run a standard paired t-test" approach fails to do.

Worked Example 1: A Small Comparison by Hand

Suppose you are comparing a logistic regression model (Model A) against a decision tree (Model B) on a dataset, using accuracy as the performance metric. You run 2-fold cross-validation five times, each with a fresh 50/50 split, and record Model A's accuracy minus Model B's accuracy in each fold:

Step 1: Verify a sample variance calculation. For repetition 3, \( \bar{p}_3 = (0.030 + 0.000)/2 = 0.015 \), so \( s_3^2 = (0.030 - 0.015)^2 + (0.000 - 0.015)^2 = 0.015^2 + (-0.015)^2 = 0.000225 + 0.000225 = 0.00045 \), matching the table.

Step 2: Average the five within-repetition variances.

Step 3: Compute the t-statistic using only the first repetition's first fold difference, \( p_1^{(1)} = 0.020 \):

Step 4: Compare against the t-distribution with 5 degrees of freedom. The critical value for a two-sided test at \( \alpha = 0.05 \) with \( df = 5 \) is approximately \( t_{crit} = 2.571 \). Since \( |t| = 1.754 < 2.571 \), we fail to reject \( H_0 \). Despite Model A outperforming Model B in 9 of the 10 folds, the 5x2cv test correctly recognizes that the underlying variance across repetitions is large enough (driven largely by repetition 3's noisy split) that this apparent edge is not yet statistically decisive-a good illustration of how conservative and appropriately cautious this test is compared to a naive average-and-t-test approach.

Worked Example 2: A Larger Model Comparison

Now suppose you are comparing a gradient-boosted model (Model A) against a support vector machine (Model B) on a larger tabular dataset, using F1 score. The five repetitions of 2-fold cross-validation give:

Here the five repetitions are far more consistent than in Worked Example 1-every repetition centers close to a \( 0.040 \) F1-score advantage for Model A, with much smaller within-repetition variance.

With \( |t| \approx 5.906 \) far exceeding the two-sided critical value \( t_{crit} \approx 2.571 \) at \( df = 5 \), we reject \( H_0 \)-the corresponding two-sided p-value is approximately \( p \approx 0.002 \). Model A's F1-score advantage is both consistent across repetitions and large relative to the estimated variance, giving strong evidence of a genuine performance difference rather than an artifact of any single train/test split.

The Combined 5x2cv F Test

A well-known critique of the 5x2cv paired t-test is that it uses only one of the ten paired differences, \( p_1^{(1)} \), in its numerator-discarding information from the other nine. Ethem Alpaydin proposed a refinement, the combined 5x2cv F test, which uses all ten differences symmetrically:

Under \( H_0 \), \( f \) follows approximately an F distribution with 10 and 5 degrees of freedom. Because it pools information from all ten paired differences rather than a single one, the combined F test generally has somewhat better statistical power and stability than the original 5x2cv paired t-test, while relying on essentially the same repeated 2-fold cross-validation design and the same underlying rationale for why five repetitions and non-overlapping folds are needed in the first place.

Effect Size: Mean Performance Difference

Like the t-statistic elsewhere on this site, the 5x2cv test statistic can be significant even for a modest true difference given enough consistency across repetitions, so it should be paired with an effect size. The natural choice is the mean performance difference across all ten paired folds:

For Worked Example 2, every repetition's mean was exactly \( 0.040 \), so \( \bar{p} = 0.040 \)-Model A's F1 score is, on average, 4 percentage points higher than Model B's across the repeated splits. For Worked Example 1, \( \bar{p} = 0.013 \)-a much smaller average edge, consistent with that example's non-significant result.

Reporting \( \bar{p} \) alongside the p-value lets readers judge both statistical and practical significance: a significant 5x2cv result with \( \bar{p} = 0.001 \) may be statistically real but practically negligible, while one like \( \bar{p} = 0.040 \) suggests a meaningfully better model.

Python Example

The most widely used Python implementation is mlxtend.evaluate.paired_ttest_5x2cv, which handles the repeated 2-fold splitting, training, and scoring automatically:

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from mlxtend.evaluate import paired_ttest_5x2cv

X, y = make_classification(n_samples=500, n_features=20, random_state=42)

model_a = LogisticRegression(max_iter=1000)
model_b = DecisionTreeClassifier(random_state=42)

t_stat, p_value = paired_ttest_5x2cv(
    estimator1=model_a,
    estimator2=model_b,
    X=X, y=y,
    scoring="accuracy",
    random_seed=42
)

print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_value:.4f}")

Output (illustrative, seeded for reproducibility):

t-statistic: 1.812
p-value: 0.1298

With \( p \approx 0.13 > 0.05 \), there is not enough evidence that the logistic regression and decision tree differ significantly on this synthetic dataset-mirroring the cautious, non-significant outcome from Worked Example 1. For the combined F test variant, discussed in the previous section:

from mlxtend.evaluate import combined_ttest_5x2cv

f_stat, p_value_f = combined_ttest_5x2cv(
    estimator1=model_a,
    estimator2=model_b,
    X=X, y=y,
    scoring="accuracy",
    random_seed=42
)

print(f"f-statistic: {f_stat:.3f}")
print(f"p-value: {p_value_f:.4f}")

f-statistic: 3.104
p-value: 0.1147

Both variants agree qualitatively here. Note that random_seed controls the five random 2-fold splits-changing it will change the specific numbers, since the whole point of the 5x2cv design is to be robust to any single split rather than to reproduce identical results from a single fixed partition.

How to Interpret Results

A significant 5x2cv result tells you that one model's performance advantage on this dataset is unlikely to be an artifact of the particular cross-validation splits used-it does not, by itself, tell you the practical size of that advantage. Report the mean performance difference \( \bar{p} \) (as in the effect size section above) alongside the p-value so readers can judge both statistical and practical significance.

It is also important to remember what the test does not establish: a significant result on one dataset says nothing about whether the same model will win on a different dataset. If your goal is a general claim like "algorithm A tends to beat algorithm B across problem domains," you need a different, multi-dataset procedure-see the comparisons section below.

Checking Assumptions in Practice

Before running the 5x2cv test, confirm both models really were trained and evaluated on the identical five repetitions of 2-fold splits-if one model used a different random seed or a different fold structure, the pairing that the test's entire logic depends on breaks down.

Next, make sure each repetition's two folds are a genuine, non-overlapping 50/50 split-not, say, a 90/10 split relabeled as "2-fold." The whole reason 2-fold (rather than 5-fold or 10-fold) cross-validation is used here is that with only two folds, each repetition's training sets share zero data, keeping that repetition's two paired differences as independent of each other as the design allows. Finally, remember that reusing the same random seed across all five repetitions defeats the purpose-verify your five repetitions genuinely use five different random partitions, as the Python example above does via its random_seed argument driving five internally-varied splits.

Advantages

• Purpose-built to fix the inflated Type I error rate that a naive paired t-test on ordinary cross-validation scores suffers from, giving a properly calibrated significance test.
• Requires only ten total model-fitting runs per model (five repetitions times two folds), making it computationally modest compared to more elaborate repeated k-fold schemes.
• Has a well-studied theoretical foundation from Dietterich's original simulation study, plus a documented refinement (the combined 5x2cv F test) for users who want slightly more statistical power.
• Widely implemented in accessible libraries like mlxtend, making correct usage straightforward without hand-rolling the repeated cross-validation loop.

Limitations

• Compares two models on a single dataset only-it says nothing about which algorithm generally performs better across different problem domains or datasets.
• Uses only one of the ten paired differences in its numerator (the original t-test variant), discarding information that the combined F test makes use of instead.
• With only 2 folds per repetition, each individual fold's training set is smaller (50% of the data) than in typical 5-fold or 10-fold cross-validation, which can matter for small datasets or data-hungry models.
• Compares exactly two models at a time-comparing three or more requires either pairwise testing with a multiple-comparisons correction, or a separate multi-model procedure entirely.

When NOT to Use It

• Ordinary paired t-test: never use this directly on k-fold cross-validation scores to compare two models-see the dedicated section above explaining why this inflates the Type I error rate.
• Friedman Test with a post-hoc test: use instead when comparing multiple algorithms across multiple datasets, which is the standard recommended procedure for that broader claim.
• McNemar's Test: use instead when you have only a single fixed train/test split and want to compare two classifiers' binary predictions on that one test set, rather than a cross-validated performance metric.
• Combined 5x2cv F test: consider using this refinement instead of the original 5x2cv paired t-test whenever you want slightly better statistical power from the same repeated-split design-see the section above.

5x2cv Paired t-test vs Ordinary Paired t-test vs 10-Fold CV t-test vs McNemar's Test

18.1 5x2cv Paired t-test vs Ordinary Paired t-test on CV Folds

18.2 5x2cv Paired t-test vs Combined 5x2cv F Test

18.3 5x2cv Paired t-test vs Friedman Test Across Multiple Datasets

Common Misconceptions

• "I can just run a paired t-test on my 10-fold cross-validation scores." No-as explained above, overlapping cross-validation folds violate the independence assumption a t-test relies on and inflate the Type I error rate; the 5x2cv design exists specifically to fix this.
• "The 5x2cv test wastes information by only using one of the ten differences." The numerator does use only \( p_1^{(1)} \), but this is a deliberate design choice, not an oversight-it keeps the null distribution close to a genuine t-distribution; readers who want to use all ten differences symmetrically should reach for the combined 5x2cv F test instead.
• "A significant 5x2cv result means Model A will win on any future dataset." No-the test only establishes a significant difference on this specific dataset; general claims across multiple datasets require a different procedure entirely, such as the Friedman Test.
• "Any 2-fold, 5-repetition split will do; the randomness doesn't matter much." It does- each repetition must use a genuinely fresh random 50/50 partition; reusing the same split, or using unbalanced folds, breaks the theoretical guarantees the test relies on.
• "5x2cv is only for accuracy." No-the same procedure works for any scalar performance metric, including F1 score, AUC, RMSE, or a custom business metric, as long as it can be computed per fold and paired between the two models.

Interview Questions

1. Explain why running an ordinary paired t-test directly on k-fold cross-validation scores produces an inflated Type I error rate.
2. Walk through why the 5x2cv test's numerator uses only \( p_1^{(1)} \) instead of averaging all ten paired differences.
3. Why does the 5x2cv design specifically use 2-fold cross-validation rather than 5-fold or 10-fold within each repetition?
4. What is the difference between the 5x2cv paired t-test and the combined 5x2cv F test, and when would you prefer one over the other?
5. Explain what it means for two folds within a single repetition to be "non-overlapping," and why that property matters here.
6. Why can a significant 5x2cv result on one dataset not be generalized to a claim about algorithm performance across other datasets?
7. How would you compare three or more algorithms using an approach built on the same underlying logic as the 5x2cv test?
8. What role does the number of degrees of freedom (5) play in the 5x2cv test statistic's null distribution?
9. Under what circumstances might you prefer McNemar's Test over the 5x2cv paired t-test for comparing two classifiers?
10. Why does reusing the same random seed for all five repetitions defeat the purpose of the 5x2cv design?

Frequently Asked Questions

• What is the 5x2 cross-validation paired t-test used for? +
  The 5x2cv paired t-test is used to determine whether two machine learning models or algorithms differ significantly in performance on a given dataset. It repeats 2-fold cross-validation five times, pairs the two models' scores within every fold, and combines those paired differences into a single test statistic that is properly calibrated for the correlation that ordinary cross-validation introduces.
• What is the formula for the 5x2cv paired t-test? +
  Run 2-fold cross-validation five separate times. In repetition i, compute the performance difference between the two models in fold 1 (p_i^(1)) and fold 2 (p_i^(2)), and their sample variance s_i^2 = (p_i^(1) - mean_i)^2 + (p_i^(2) - mean_i)^2. The test statistic is t = p_1^(1) / sqrt((1/5) * sum of s_i^2 across all five repetitions), compared to a t-distribution with 5 degrees of freedom.
• Why can't I just use an ordinary paired t-test on cross-validation fold scores to compare two classifiers? +
  Cross-validation folds are not independent-every fold's training set overlaps heavily with every other fold's training set, since they are drawn from the same finite dataset. This violates the independence assumption an ordinary paired t-test relies on, which shrinks the estimated variance and substantially inflates the Type I error rate, often well beyond the nominal 5% level.
• What are the assumptions of the 5x2cv paired t-test? +
  Both models must be evaluated on the identical five repetitions of 2-fold cross-validation using the same dataset and the same fold splits (paired design), the performance metric should be continuous or near-continuous, and the test statistic's null distribution relies on an approximate normality assumption for the small set of ten paired differences it produces.
• How do you interpret a 5x2cv paired t-test p-value? +
  If the p-value is below your chosen significance level (commonly alpha = 0.05), you reject the null hypothesis and conclude the two models differ significantly in performance. If the p-value is at or above alpha, there is not enough evidence that the two models perform differently on this dataset, regardless of which one scored higher on average.
• What should I use instead of the 5x2cv paired t-test? +
  Consider the combined 5x2cv F test, proposed by Alpaydin as a refinement with generally better statistical properties, when you want a slightly more powerful and stable variant of the same idea; consider McNemar's Test when you only have a single train/test split and want to compare classifiers' binary predictions rather than a cross-validated performance metric.
• Does the 5x2cv paired t-test use all ten cross-validation scores equally? +
  No. The numerator of the test statistic uses only the paired difference from the very first fold of the very first repetition, while the denominator averages the variance across all five repetitions-a deliberate asymmetry Dietterich introduced to keep the null distribution close to a genuine t-distribution with 5 degrees of freedom.
• Can the 5x2cv paired t-test compare more than two models at once? +
  The standard 5x2cv paired t-test compares exactly two models. Comparing three or more models pairwise with the 5x2cv test requires a multiple-comparisons correction such as Bonferroni or Holm, or a dedicated multi-model procedure such as the Friedman test with a post-hoc test across multiple datasets.

Key Takeaways

• The 5x2cv paired t-test checks whether two models or algorithms differ significantly in performance on a single dataset, by repeating 2-fold cross-validation five times and combining the resulting paired differences into a properly calibrated test statistic.
• It exists specifically because an ordinary paired t-test applied directly to k-fold cross-validation scores is statistically unsound-overlapping folds violate independence and inflate the Type I error rate well above the nominal significance level.
• Core design: five repetitions of non-overlapping 2-fold splits, a deliberately asymmetric test statistic using only \( p_1^{(1)} \) in the numerator and the averaged within-repetition variance in the denominator, compared to a t-distribution with 5 degrees of freedom.
• The combined 5x2cv F test is a well-established refinement that uses all ten paired differences symmetrically and generally offers slightly better statistical power.
• In Python, mlxtend.evaluate.paired_ttest_5x2cv (and its combined_ttest_5x2cv counterpart) automate the repeated splitting, training, and scoring in a single function call.
• If \( p < 0.05 \), the two models' performance differs significantly on this dataset-always pair that result with the mean performance difference \( \bar{p} \), and remember the result does not generalize to other datasets without a separate multi-dataset procedure.

The 5x2cv paired t-test earns its place as a standard tool in machine learning model comparison precisely because the obvious shortcut-running an ordinary paired t-test on ordinary cross-validation fold scores- turns out to be statistically broken. Dietterich's insight was to redesign the cross-validation procedure itself around the requirements of the significance test, using non-overlapping 2-fold splits repeated five times, rather than trying to patch a standard t-test onto a resampling scheme it was never suited for.

The two worked examples above, a modest logistic-regression-versus-decision-tree comparison and a more decisive gradient-boosting-versus-SVM comparison, show both ends of this test's range, and a useful lesson in between: the deliberately asymmetric formula-using only \( p_1^{(1)} \) in the numerator-is not a shortcut or an oversight but the specific mechanism that keeps the test properly calibrated, with the combined 5x2cv F test available as a refinement when you want to use all ten paired differences instead. Reporting the t-statistic and p-value alongside the mean performance difference \( \bar{p} \), and remembering that any single-dataset result does not automatically generalize, gives a complete and statistically honest picture of how two models really compare.