Wilcoxon Test for Model Comparison

Introduction

The Wilcoxon Test for Model Comparison applies the Wilcoxon Signed-Rank Test to a very specific and very common machine learning question: does Model A actually outperform Model B, once you have evaluated both across a whole collection of datasets or repeated cross-validation folds-not just one? If you have ever averaged accuracy scores across a benchmark suite of datasets and wondered whether a plain paired t-test on those numbers is trustworthy, this is the test built for exactly that situation.

By the end of this article you will understand exactly why treating repeated benchmark scores as raw numbers for a t-test is risky, be able to compute this test completely by hand on two worked examples-one where a model's edge does not hold up, one where it clearly does-interpret the result alongside a proper effect size, and run the same test in a few lines of Python with scipy.stats.wilcoxon.

What Is the Wilcoxon Test for Model Comparison?

Mechanically, this is nothing more than the ordinary Wilcoxon Signed-Rank Test applied to a specific kind of paired data: one performance difference per dataset (or per cross-validation fold), rather than one before-and-after measurement per subject. For each of \( n \) datasets, you compute \( d_i = \) Model A's score minus Model B's score, then rank the absolute values of those differences and sum the ranks separately by sign.

It exists because Josef Demsar's influential 2006 paper on comparing classifiers showed that the intuitive approach-averaging a performance metric across many heterogeneous datasets and running a paired t-test on the result-rests on assumptions that rarely hold in practice. Working with ranks instead of raw score differences sidesteps most of those problems, which is why the Wilcoxon test became the recommended default for comparing two classifiers across a benchmark suite.

Why an Ordinary Paired t-test Struggles Here

It is tempting to compute Model A's score minus Model B's score on each of, say, 15 benchmark datasets and run those 15 numbers straight through an ordinary paired t-test. Three practical problems make this risky. First, a typical benchmark comparison uses far fewer datasets than the large-sample regime a t-test's normality assumption is forgiving about-15 to 30 datasets is common, and normality of the differences is hard to justify or even check with so few points. Second, performance metrics are rarely commensurable across datasets: a 2-percentage-point accuracy gain on an easy, near-saturated dataset does not represent the same underlying advantage as a 2-point gain on a hard one, yet a t-test treats both identically in its mean and variance. Third, benchmark suites routinely contain one or two idiosyncratic datasets where a model behaves very differently from its typical pattern-and because the t-test's variance term squares each deviation, a single such dataset can dominate the result.

The Wilcoxon test's rank-based construction limits how much damage any single dataset can do: no matter how extreme one dataset's raw performance difference is, it can only occupy the largest rank position-shifting the test statistic by at most one unit, rather than by an amount proportional to its raw size. This is the same robustness property that makes the plain Wilcoxon Signed-Rank Test attractive for skewed or outlier-prone paired data, applied here to the specific setting of cross-dataset model comparison.

When to Use It

Use the Wilcoxon Test for Model Comparison when you have evaluated two models or algorithms on the same collection of datasets (or the same repeated cross-validation folds within one dataset), and want to know whether one systematically outperforms the other, without assuming the per-dataset performance differences are normally distributed.

Key Assumptions

• Paired design: both models must be evaluated on the exact same set of datasets or the exact same cross-validation folds-no dataset should appear for one model but not the other.
• At least ordinal performance measure. Accuracy, F1, AUC, and similar metrics all qualify, since only their relative ranking across datasets is used, not their raw scale.
• Independence across datasets (or folds). One dataset's performance difference should not be mechanically tied to another's-a reasonable assumption across genuinely separate datasets, but a caveat worth remembering when the pairs are cross-validation folds from a single dataset, which are not fully independent of one another.
• Symmetry of the differences. The distribution of the per-dataset differences should be roughly symmetric about the median so the test is properly interpreted as testing that median.
• No requirement of normality. This is the key relaxation relative to a paired t-test-the per-dataset differences need not follow any particular parametric distribution.

Hypotheses

The Wilcoxon Test for Model Comparison formally tests the median of the per-dataset performance differences:

• Null Hypothesis (\( H_0 \)): the median performance difference between Model A and Model B across the tested datasets is zero-neither model systematically outperforms the other.
• Alternative Hypothesis (\( H_1 \)): the median performance difference is not zero-one model systematically outperforms the other across the tested datasets.

(For a one-tailed test, \( H_1 \) can instead state the median difference is specifically greater than or less than zero-for example, if you only care whether a new model improves on a baseline, not the reverse.)

The Formula, Explained

For \( n \) datasets (or folds), let \( d_i = \text{score}_{A,i} - \text{score}_{B,i} \) be Model A's performance minus Model B's performance on dataset \( i \). The procedure is identical to the general Wilcoxon Signed-Rank Test:

1. Discard any datasets where \( d_i = 0 \) (a tie), reducing the effective sample size to \( n' \).
2. Rank the remaining \( |d_i| \) values from smallest (rank 1) to largest (rank \( n' \)), assigning the average rank to any ties.
3. Attach the original sign of \( d_i \) to each rank, producing signed ranks.
4. Sum the positive signed ranks to obtain \( W_+ \), and sum the absolute value of the negative signed ranks to obtain \( W_- \). Note \( W_+ + W_- = \dfrac{n'(n'+1)}{2} \).

The test statistic is usually taken as:

For small \( n' \) (roughly \( n' \leq 25 \)), \( W \) is compared directly against exact critical values from Wilcoxon signed-rank tables. For larger \( n' \), a normal approximation is used, based on the known mean and variance of \( W_+ \) under \( H_0 \):

This \( z \) value (with a continuity correction of 0.5, and a tie correction to \( \sigma_W \) when ties are present) is compared against the standard normal distribution to obtain a p-value. Because only ranks are used, the test makes no assumption about the shape of the underlying score distributions-only that the per-dataset differences are roughly symmetric.

Worked Example 1: Two Models Across 8 Datasets

Suppose you compare a gradient-boosted tree model (Model A) against a regularized logistic regression model (Model B) on 8 benchmark datasets, recording the difference in macro-F1 score (Model A minus Model B) on each:

Step 1: Rank the absolute differences from smallest to largest (no ties, no zeros, so \( n' = 8 \)):

Step 2: Re-attach the signs and sum by direction. Dataset 1 (\( d = 0.010 \)) gets rank 5, positive; Dataset 2 (\( d = -0.006 \)) gets rank 3, negative; and so on for all eight:

Check: \( W_+ + W_- = 23 + 13 = 36 = \dfrac{8 \times 9}{2} \). ✓

Step 3: Compute the test statistic.

Step 4: Compare against the exact critical value. For \( n' = 8 \), the exact two-sided critical value at \( \alpha = 0.05 \) is \( W_{crit} = 3 \) (reject \( H_0 \) only if \( W \leq 3 \)). Since \( W = 13 > 3 \), we fail to reject \( H_0 \). The exact p-value works out to \( p \approx 0.547 \)-even though Model A won on 5 of the 8 datasets, that record is not consistent or large enough to rule out chance at this sample size.

Worked Example 2: A Decisive 12-Dataset Comparison

Now suppose a newer architecture (Model A) is compared against a strong baseline (Model B) on a larger suite of 12 datasets, again recording the macro-F1 difference:

All 12 differences are non-zero, so \( n' = 12 \). Ranking the absolute values and re-attaching signs, only Dataset 7 (\( d = -0.005 \), the smallest absolute difference) is negative-every other dataset favors Model A:

Check: \( W_+ + W_- = 77 + 1 = 78 = \dfrac{12 \times 13}{2} \). ✓

For \( n' = 12 \), the exact two-sided critical value at \( \alpha = 0.05 \) is \( W_{crit} = 13 \). Since \( W = 1 \leq 13 \), we reject \( H_0 \) decisively. The exact p-value is \( p \approx 0.00098 \)-Model A's advantage is consistent across 11 of the 12 datasets and far too one-sided to be explained by chance.

Effect Size: Rank-Biserial Correlation & Win/Tie/Loss

A significant Wilcoxon result only says the observed rank imbalance is unlikely under \( H_0 \)-it does not say how large or how consistent the advantage is. Two complementary effect sizes fill that gap. The first, the matched-pairs rank-biserial correlation, uses the same signed ranks already computed:

For Worked Example 1: \( r_{rb} = (23 - 13)/36 \approx 0.278 \)-a small-to-medium, unconvincing effect, consistent with the non-significant result. For Worked Example 2: \( r_{rb} = (77 - 1)/78 \approx 0.974 \)-a very large effect, consistent with the decisive rejection of \( H_0 \).

The second, more ML-native effect size is simply the win/tie/loss count: how many datasets each model won outright. Worked Example 1 was 5 wins / 0 ties / 3 losses for Model A-a modest edge in raw count that the test correctly flags as statistically inconclusive. Worked Example 2 was 11 wins / 0 ties / 1 loss-the kind of lopsided record the rank-biserial correlation and the p-value both agree is a real effect.

Confidence Interval via the Hodges-Lehmann Estimator

Because this test targets the median difference rather than the mean, its natural confidence interval is not centered on \( \bar{d} \) the way a paired t-test's would be. The standard approach uses the Hodges-Lehmann estimator: compute the pairwise averages (Walsh averages) of all \( d_i \) values, including each value averaged with itself, and take the median of those averages as the point estimate of the location shift; a distribution-free interval is then built from the ordered Walsh averages.

For Worked Example 2, the Hodges-Lehmann estimate works out to \( \approx 0.030 \), with an approximate 95% confidence interval of [0.020, 0.037]-entirely above zero, matching the decisive rejection above. For Worked Example 1, the estimate is only \( \approx 0.003 \), with an interval of roughly [-0.007, 0.012]-spanning zero, consistent with that example's failure to reject \( H_0 \).

Python Example

scipy.stats.wilcoxon runs this test in one line once you have one performance score per dataset for each model:

import numpy as np
from scipy import stats

datasets = ["breast_cancer", "wine", "diabetes", "heart", "credit-g", "sonar",
            "ionosphere", "vehicle", "segment", "spambase", "adult", "bank-marketing"]

model_a = np.array([0.844, 0.762, 0.696, 0.808, 0.736, 0.847,
                     0.753, 0.728, 0.841, 0.711, 0.765, 0.733])
model_b = np.array([0.812, 0.734, 0.655, 0.789, 0.701, 0.823,
                     0.758, 0.690, 0.812, 0.667, 0.744, 0.700])

w_stat, p_value = stats.wilcoxon(model_a, model_b)

diffs = model_a - model_b
n = len(diffs)
w_max = n * (n + 1) / 2
r_rank_biserial = (w_max - 2 * w_stat) / w_max

wins   = int((diffs > 0).sum())
losses = int((diffs < 0).sum())
ties   = int((diffs == 0).sum())

print(f"W-statistic: {w_stat:.3f}")
print(f"p-value: {p_value:.5f}")
print(f"Median difference: {np.median(diffs):.4f}")
print(f"Rank-biserial correlation: {r_rank_biserial:.3f}")
print(f"Win/Tie/Loss (Model A vs Model B): {wins}/{ties}/{losses}")

Output:

W-statistic: 1.000
p-value: 0.00098
Median difference: 0.0305
Rank-biserial correlation: 0.974
Win/Tie/Loss (Model A vs Model B): 11/0/1

Every number matches the hand calculation in Worked Example 2 exactly-wilcoxon(model_a, model_b) internally computes \( d_i = \) model_a\(_i\) − model_b\(_i\) and applies precisely the ranking procedure from the Formula section. For comparison, running an ordinary paired t-test on the same 12 differences:

t_stat, p_ttest = stats.ttest_rel(model_a, model_b)
print(f"Paired t-test statistic: {t_stat:.3f}")
print(f"Paired t-test p-value: {p_ttest:.6f}")

Paired t-test statistic: 7.546
Paired t-test p-value: 0.000011

Here both tests agree-the advantage is large and remarkably consistent across 11 of 12 datasets, so even a mean-based test detects it easily. That agreement is expected when an effect is this strong; with fewer datasets, a noisier benchmark, or a single outlier dataset (as discussed in the section above), the two tests can diverge, which is precisely why Demsar (2006) recommends defaulting to the rank-based test rather than assuming the paired t-test's normality assumption holds.

How to Interpret Results

Checking Assumptions in Practice

• Symmetry of differences: a histogram or boxplot of the \( d_i \) values-strong skew complicates interpreting the result strictly as a test of the median.
• Ties and zeros: check how many datasets produced \( d_i = 0 \) (they are dropped, reducing power) and how many tied absolute differences remain (they receive averaged ranks).
• Genuine pairing: confirm both models were evaluated on identical datasets or identical folds-mismatched pairs invalidate the entire procedure.
• Independence caveat for CV folds: if the pairs are cross-validation folds from a single dataset rather than separate datasets, treat the independence assumption as an approximation rather than an exact match to the test's requirements.
• Sample size and exact vs. approximate p-values: for small \( n' \) (roughly \( n' \leq 25 \)), use the exact distribution, as both worked examples above did; for larger \( n' \), the normal approximation is generally reliable.

Advantages

• Does not require per-dataset performance differences to be normally distributed, which is realistic given how few datasets most benchmark comparisons actually use.
• Robust to datasets that are not on a comparable performance scale, since only the rank of each difference matters, not its raw magnitude.
• Much less sensitive to a single outlier dataset than a mean-based test-an extreme difference can only shift its rank by one position.
• Explicitly recommended by Demsar (2006) as the default statistical test for comparing two classifiers across multiple datasets, giving it broad acceptance in ML research and reviewing.
• A single line of code with scipy.stats.wilcoxon, with exact p-values available for the small sample sizes typical of benchmark suites.

Limitations

• Compares exactly two models at a time-comparing three or more requires the Friedman Test with a post-hoc procedure instead.
• Reduces each dataset's performance difference to a rank, discarding information about exactly how large the difference was.
• Has low statistical power with very few datasets (roughly fewer than 6-8), where even a consistent edge may not reach significance-as Worked Example 1 illustrates.
• A significant result is specific to the tested collection of datasets and does not, by itself, license a general claim that one algorithm is "better" across all problem domains.
• When the pairs are cross-validation folds rather than separate datasets, the independence assumption is only approximately satisfied.

When NOT to Use It

• Corrected Resampled t-test or 5x2cv Paired t-test: use these instead when you have a single dataset and want to properly account for overlap between repeated train/test splits or cross-validation folds.
• McNemar's Test: use instead when you have a single fixed train/test split and want to compare two classifiers' binary predictions on that one test set.
• Friedman Test with a post-hoc test: use instead when comparing three or more models across multiple datasets.
• Ordinary paired t-test: only reasonable when you have many datasets, a metric that is genuinely comparable across them, and good evidence the differences are approximately normal-conditions that are the exception rather than the rule in typical ML benchmarking.

Wilcoxon Test vs Paired t-test vs Corrected Resampled t-test vs Friedman Test

18.1 Wilcoxon Test for Model Comparison vs Ordinary Paired t-test (Across Datasets)

18.2 Wilcoxon Test for Model Comparison vs Corrected Resampled / 5x2cv Paired t-test

18.3 Wilcoxon Test for Model Comparison vs Friedman Test

Common Misconceptions

• "This test can compare any number of models at once." No-it is strictly pairwise; use the Friedman Test with a post-hoc test for three or more models.
• "It has no assumptions at all." It still requires a genuine paired design, independence across datasets, and roughly symmetric differences-what it relaxes is specifically the normality requirement.
• "This is the same tool as the Corrected Resampled t-test." They solve different problems-one handles multiple separate datasets, the other handles repeated resampling of a single dataset; see the comparisons section above.
• "A significant result means Model A will win on any future dataset." No-the test only establishes a systematic advantage on the specific datasets tested, not a general guarantee across all problem domains.
• "More wins always means significance." Not necessarily-Worked Example 1 shows Model A won 5 of 8 datasets yet the test still failed to reach significance, because the margins and consistency of those wins mattered too.

Interview Questions

1. Why is a paired t-test often a poor choice for comparing two models across a suite of benchmark datasets, even when it might be reasonable for a single dataset's repeated CV folds?
2. Walk through how the Wilcoxon Test for Model Comparison limits the influence of a single outlier dataset compared to a mean-based test.
3. Explain the difference between comparing two models across multiple datasets (this test) and comparing two models via repeated resampling of one dataset (the Corrected Resampled t-test).
4. Why does Demsar (2006) recommend the Wilcoxon Signed-Rank Test as the default for comparing two classifiers over multiple datasets?
5. What happens to this test's independence assumption when the paired observations are cross-validation folds from a single dataset rather than separate datasets?
6. How would you compute and interpret the rank-biserial correlation for this test, and why is a p-value alone insufficient?
7. Given 8 datasets where a new model wins on 5 and loses on 3, explain why the test might still fail to reject the null hypothesis.
8. How would you extend this pairwise approach to compare four different model architectures across the same benchmark suite?
9. Why does scipy.stats.wilcoxon(model_a, model_b) require the two arrays to be the same length and in matched dataset order?
10. Under what circumstances might a paired t-test and the Wilcoxon test on the same benchmark data reach different conclusions, and which would you trust more?

Frequently Asked Questions

• What is the Wilcoxon Test for Model Comparison used for? +
  It is used to determine whether two machine learning models differ significantly in performance across a collection of datasets or repeated cross-validation folds, by ranking the paired per-dataset performance differences rather than relying on their raw magnitudes-avoiding the normality assumption a paired t-test needs.
• What is the formula for the Wilcoxon Test for Model Comparison? +
  Compute d_i, the performance difference between Model A and Model B on each of n datasets. Discard zero differences, rank the remaining absolute values (averaging tied ranks), reattach the sign of each d_i, then sum the positive signed ranks to get W+ and the negative signed ranks to get W-. The test statistic W is typically taken as min(W+, W-), evaluated against exact tables for small n or a normal approximation for larger n.
• Why not just use a paired t-test to compare model performance across datasets? +
  A paired t-test compares the mean of the per-dataset differences and requires them to be approximately normal-an assumption that is difficult to justify with the small number of datasets typical of ML benchmarking (often 10-30), where performance scores are not on a comparable scale across problems and a single unusual dataset can dominate the mean and standard deviation. The Wilcoxon test's rank-based approach is much more robust to these issues.
• How many datasets or folds are needed for this test? +
  There is no hard minimum, but power is very low below roughly 6-8 datasets. Demsar (2006) recommends this test as the standard approach for comparing two classifiers over a typical benchmark suite, and most published comparisons use somewhere between 10 and 40 datasets or folds.
• What are the assumptions of the Wilcoxon Test for Model Comparison? +
  Both models must be evaluated on the identical set of datasets or folds (a paired design), the performance differences should be independent of one another, the distribution of the differences should be roughly symmetric about the median, and the performance measure only needs to be at least ordinal-no normality is required.
• How do you interpret the p-value from this test? +
  If the p-value is below your chosen significance level (commonly alpha = 0.05), you reject the null hypothesis and conclude the two models differ significantly across the tested datasets. If the p-value is at or above alpha, there is not enough evidence that one model systematically outperforms the other on this collection of datasets.
• Is this the same test as the plain Wilcoxon Signed-Rank Test? +
  The underlying mathematics is identical to the general Wilcoxon Signed-Rank Test-what differs is the application: here, each paired observation is one dataset's (or fold's) performance difference between two models, rather than a before-and-after measurement on a single subject.
• What should I use if I am comparing more than two models across multiple datasets? +
  Use the Friedman Test to test whether any of the models differ overall, followed by a post-hoc procedure such as the Nemenyi test or Dunn's test to pinpoint which specific pairs differ-the standard multi-model extension of the logic used here.

Key Takeaways

• The Wilcoxon Test for Model Comparison applies the Wilcoxon Signed-Rank Test to one performance difference per dataset (or per CV fold) between two models-the standard recommended non-parametric approach for comparing two classifiers over multiple datasets.
• It exists because averaging performance differences across a small, heterogeneous benchmark suite and running an ordinary paired t-test rests on a normality assumption that is difficult to justify and is fragile to any single unusual dataset.
• Core procedure: rank the absolute per-dataset differences, re-attach their signs, and compare the sum of positive ranks against the sum of negative ranks-\( W = \min(W_+, W_-) \).
• Always pair the p-value with an effect size-the rank-biserial correlation and the win/tie/loss count-since a lopsided win count is not automatically statistically significant, as Worked Example 1 shows.
• In Python, scipy.stats.wilcoxon(model_a, model_b) gets you the W-statistic and p-value in one call, with exact p-values available at the small sample sizes typical of ML benchmark suites.
• If you are resampling a single dataset rather than comparing across multiple datasets, use the Corrected Resampled t-test or 5x2cv paired t-test instead; if comparing more than two models, use the Friedman Test with a post-hoc test.

The Wilcoxon Test for Model Comparison earns its place as the standard non-parametric tool for a question machine learning practitioners face constantly: does this model actually beat that one, once you look across more than a single dataset? Its logic mirrors the general Wilcoxon Signed-Rank Test-collapse paired differences into signed ranks, then ask whether the positive and negative ranks are too imbalanced to be chance-while specifically addressing the practical realities of ML benchmarking: few datasets, scores that are not commensurable across problems, and the ever-present risk of one idiosyncratic dataset skewing a mean-based comparison.

The two worked examples above, an inconclusive 8-dataset comparison and a decisive 12-dataset comparison built on the same underlying logic, show both ends of what this test can tell you-and why a raw win count alone is not enough to call a result significant. Reporting the p-value alongside the rank-biserial correlation and win/tie/loss record, as demonstrated throughout this guide, and remembering that any result applies specifically to the datasets tested rather than to algorithms in general, gives a complete and honest picture of how two models really compare.