Nemenyi Post-Hoc Test: When, Why, and How to Use It

Introduction

The Nemenyi Post-Hoc Test answers the question a significant Friedman Test deliberately leaves open: which specific pair of related conditions actually differs? The Friedman Test is an omnibus test-it tells you a difference exists somewhere among three or more related conditions, but not where. The Nemenyi Test picks up exactly where the Friedman Test leaves off, using the same within-subject ranks to compare every pair of conditions directly.

By the end of this article you will be able to state exactly when the Nemenyi Test applies, compute one completely by hand on two worked examples that continue directly from the Friedman Test guide, read a critical difference diagram, and run the same comparison in a few lines of Python with scikit-posthocs.

What Is the Nemenyi Post-Hoc Test?

The Nemenyi Post-Hoc Test compares every possible pair of conditions from a repeated-measures design-the same one analyzed by the Friedman Test-using the average rank each condition received across all subjects. Rather than running a separate, uncorrected test for each pair (which would inflate the chance of a false positive as the number of comparisons grows), it uses a single critical difference derived from the studentized range distribution, so that all pairwise comparisons share one consistent threshold controlling the overall family-wise error rate.

It exists because a significant Friedman Test, on its own, is an incomplete answer. Knowing that "the conditions are not all the same" is rarely the question a researcher actually cares about-the real question is almost always "which conditions differ, and from which others?" The Nemenyi Test converts the Friedman Test's omnibus signal into a concrete, pair-by-pair verdict.

When to Use It

Use the Nemenyi Post-Hoc Test immediately after a Friedman Test returns a significant result (\( p < 0.05 \)) on three or more related conditions, and you need to know specifically which conditions are responsible for that significance-not just that a difference exists somewhere. It is the natural next step whenever the Friedman Test itself was the right choice: the same subjects measured repeatedly, normality not assumed, and at least ordinal data.

Key Assumptions

• Prior Friedman Test structure. The Nemenyi Test operates on the same block design as the Friedman Test-every subject measured under every one of the \( k \) conditions being compared.
• At least ordinal data. The values must be at least rankable, inherited directly from the ranking step already performed for the Friedman Test.
• Independence across subjects. One subject's ranks should not influence another subject's ranks.
• \( k \geq 3 \) related conditions. With exactly two conditions there is only one possible pairwise comparison, so a dedicated post-hoc procedure is unnecessary-use the Wilcoxon Signed-Rank Test directly.
• Conventionally follows a significant omnibus result. Most applied guidance runs the Nemenyi Test only after the Friedman Test itself rejects \( H_0 \), to avoid stacking an unnecessary extra layer of multiple-comparison correction on top of a null result.

Hypotheses

Unlike the single omnibus hypothesis pair tested by the Friedman Test, the Nemenyi Test evaluates a separate hypothesis pair for every combination of two conditions:

• Null Hypothesis (\( H_0^{(a,b)} \)): conditions \( a \) and \( b \) have the same underlying distribution of ranks-there is no real difference between this specific pair.
• Alternative Hypothesis (\( H_1^{(a,b)} \)): conditions \( a \) and \( b \) differ-one systematically tends to rank higher than the other.

(With \( k \) conditions there are \( \binom{k}{2} \) such pairs to test. The Nemenyi critical difference is constructed so that the probability of making at least one false-positive claim across all of these simultaneous comparisons stays at the nominal \( \alpha \) level.)

The Formula, Explained

Starting from the same \( n \) subjects and \( k \) conditions used by the Friedman Test, with rank sums \( R_j \) already computed for each condition \( j \):

1. Convert each rank sum to an average rank: \( \bar{R}_j = R_j / n \).
2. Compute a single critical difference (CD) for the entire set of pairwise comparisons.
3. Compare the absolute difference in average ranks for every pair of conditions against that one CD.

The critical difference is:

where \( q_\alpha \) is a critical value of the studentized range statistic, divided by \( \sqrt{2} \)-this convention is what most published Nemenyi tables and software already report directly, so \( q_\alpha \) can typically be read straight from a table indexed by \( k \). Two conditions \( a \) and \( b \) are declared significantly different whenever:

Notice that \( CD \) does not depend on which specific pair of conditions is being compared-it is a single number computed once for the whole design, which is what makes the test simple to apply by hand across many pairs at once.

Worked Example 1: A Small Numerical Example by Hand

This continues Worked Example 1 from the Friedman Test guide: grip strength (in kg) for 5 patients at baseline, week 4, and week 8, where the Friedman Test already returned \( Q = 8.4 \), \( p \approx 0.015 \)-a significant result that justifies running the Nemenyi Test as a follow-up.

Recall the rank sums computed there: \( R_{\text{Week 0}} = 6 \), \( R_{\text{Week 4}} = 9 \), \( R_{\text{Week 8}} = 15 \), with \( n = 5 \) patients.

Step 1: Convert to average ranks

Step 2: Compute the critical difference

With \( k = 3 \), \( q_{0.05} = 2.343 \), and \( n = 5 \):

Step 3: Compare every pair

Worked Example 2: A Training Program (Six Students, Three Checkpoints)

This continues Worked Example 2 from the Friedman Test guide: 6 students tested pre-training, at week 2, and at week 4, where the Friedman Test returned \( Q \approx 8.33 \), \( p \approx 0.0156 \).

Recall the rank sums: \( R_{\text{Pre}} = 7 \), \( R_{\text{Wk2}} = 12 \), \( R_{\text{Wk4}} = 17 \), with \( n = 6 \) students.

Step 1: Convert to average ranks

Step 2: Compute the critical difference

With \( k = 3 \), \( q_{0.05} = 2.343 \), and \( n = 6 \):

Step 3: Compare every pair

This matches exactly what scikit-posthocs reports for this dataset in the Python section below: an exact pairwise p-value of about 0.011 for Pre-training vs Week 4 (below 0.05), and about 0.32 for the other two pairs (above 0.05)-the hand calculation using the critical-difference shortcut and the software's exact p-value agree on every conclusion.

Reading a Critical Difference Diagram

Because the Nemenyi Test produces one shared critical difference for the entire set of comparisons, its results are often summarized visually as a critical difference (CD) diagram: a number line with each condition's average rank plotted as a point, and a horizontal bar of length \( CD \) drawn to show which points are close enough together to be considered statistically indistinguishable.

For Worked Example 2, the three average ranks are 1.17 (Pre-training), 2.00 (Week 2), and 2.83 (Week 4), with \( CD \approx 1.353 \). A connecting bar would span Pre-training through Week 2, and separately Week 2 through Week 4-but not Pre-training all the way through Week 4, since that gap (1.67) exceeds the critical difference. Conditions joined by the same bar are not significantly different from one another; conditions with no bar connecting them are.

Python Example

The Friedman Test itself runs through scipy.stats.friedmanchisquare, and the Nemenyi post-hoc comparison runs through scikit-posthocs:

from scipy import stats
import numpy as np
import pandas as pd
import scikit_posthocs as sp

pre  = [62, 74, 58, 81, 69, 70]
wk2  = [68, 79, 65, 80, 75, 77]
wk4  = [71, 78, 69, 83, 77, 79]

# Step 1: confirm the omnibus Friedman Test is significant
q_stat, p_value = stats.friedmanchisquare(pre, wk2, wk4)
print(f"Friedman Q: {q_stat:.3f}, p-value: {p_value:.4f}")

# Step 2: run the Nemenyi post-hoc test on the same block design
data = np.array([pre, wk2, wk4]).T  # rows = subjects, columns = conditions
df = pd.DataFrame(data, columns=["Pre", "Wk2", "Wk4"])

nemenyi_result = sp.posthoc_nemenyi_friedman(df)
print(nemenyi_result)

Output:

Friedman Q: 8.333, p-value: 0.0156
          Pre       Wk2       Wk4
Pre  1.000000  0.318645  0.010865
Wk2  0.318645  1.000000  0.318645
Wk4  0.010865  0.318645  1.000000

Each off-diagonal cell is the pairwise p-value for that comparison. Reading the Pre/Wk4 cell, \( p \approx 0.011 \) is below 0.05, matching the "significantly different" conclusion from the critical-difference calculation in Worked Example 2. The Pre/Wk2 and Wk2/Wk4 cells both sit around \( p \approx 0.32 \), confirming those two pairs are not significantly different.

Visualizing Results with a Significance Plot

import matplotlib.pyplot as plt

# scikit-posthocs includes a built-in heatmap-style significance plot
sp.sign_plot(nemenyi_result)
plt.title("Nemenyi Post-Hoc Test: Pairwise Significance")
plt.show()

This produces a heatmap where each cell is shaded by significance level, giving the same information as the critical difference diagram above in a different visual format-useful when comparing more than three or four conditions, where a CD diagram can get visually crowded.

How to Interpret Results

The significance level \( \alpha = 0.05 \) is the standard threshold used to decide whether a specific pairwise comparison is "statistically significant," applied consistently across every pair in the design.

Checking Assumptions in Practice

• Confirm the Friedman Test was significant first: running Nemenyi comparisons after a non-significant omnibus result is generally discouraged, since the Nemenyi Test's own correction is calibrated assuming it follows a signal worth chasing.
• Complete blocks: the same requirement as the Friedman Test-every subject must have a value for every condition being compared.
• Number of conditions: as \( k \) grows, the number of pairwise comparisons \( \binom{k}{2} \) grows quickly, and the critical difference grows too (via \( q_\alpha \)), making individual pairs progressively harder to distinguish-consider whether all pairs are truly of interest, or only a subset.
• Sample size: small \( n \) inflates the critical difference, as seen in comparing Worked Example 1 (\( n = 5 \), \( CD \approx 1.48 \)) against Worked Example 2 (\( n = 6 \), \( CD \approx 1.35 \))-more subjects generally sharpen the test's ability to detect real pairwise differences.

Advantages

• Directly answers the natural follow-up question after a significant Friedman Test: which conditions specifically differ.
• Uses a single critical difference for the whole design, making it simple to apply by hand across many pairs at once, and easy to visualize with a critical difference diagram.
• Controls the family-wise error rate across all pairwise comparisons simultaneously, avoiding the inflated false-positive risk of running repeated uncorrected tests.
• Reuses the ranks already computed for the Friedman Test, requiring no additional data collection or re-ranking.

Limitations

• Conservative by design-the family-wise error correction that protects against false positives also makes it harder to detect real but moderate pairwise differences, especially with few subjects, as seen in Worked Example 1.
• Only meaningful as a follow-up to a prior Friedman Test on the same block design-it is not a general-purpose pairwise comparison tool for arbitrary related samples.
• Statistical power drops as the number of conditions \( k \) increases, since more comparisons raise the critical-difference threshold.
• Like the Friedman Test itself, requires a complete design-subjects with missing conditions are typically excluded entirely.

When NOT to Use It

• Wilcoxon Signed-Rank Test: use instead when there are only two related conditions to begin with-there is no pairwise ambiguity to resolve.
• Bonferroni-corrected Wilcoxon comparisons: consider instead when you only care about a small, pre-specified subset of pairwise comparisons rather than all \( \binom{k}{2} \) pairs, since Bonferroni correction over fewer comparisons can be less conservative.
• Conover's post-hoc test: consider instead when you want a less conservative, higher-power alternative that still follows a significant Friedman Test, at some cost in strictness of error-rate control.
• Tukey's HSD (after repeated measures ANOVA): use instead when the parametric assumptions hold and you are following up a significant repeated measures ANOVA rather than a Friedman Test.

Nemenyi vs Wilcoxon vs Bonferroni vs Conover

16.1 Nemenyi Test vs Repeated Wilcoxon Signed-Rank Tests

16.2 Nemenyi Test vs Bonferroni-Corrected Comparisons

16.3 Nemenyi Test vs Conover's Post-Hoc Test

Common Misconceptions

• "A significant Friedman Test means every pair of conditions differs." Not necessarily- see Worked Example 1 and Worked Example 2, where the omnibus test was significant but only one of the three possible pairs cleared the Nemenyi critical difference in each case.
• "The Nemenyi Test can be run on its own, without a Friedman Test." It is designed as a post-hoc procedure that reuses the Friedman Test's rank structure and is conventionally interpreted alongside a prior significant omnibus result, not as a standalone test.
• "A non-significant pairwise comparison proves those two conditions are identical." Failing to clear the critical difference only means there was not enough evidence to distinguish that pair at this sample size-not proof that no real difference exists.
• "The Nemenyi Test and a plain t-test comparison are interchangeable." They are not-the Nemenyi Test's critical difference is specifically calibrated for simultaneous, rank-based comparisons across a repeated-measures design, controlling error across all pairs at once.
• "More conditions always mean more significant pairwise findings." The opposite is often true-as \( k \) grows, \( q_\alpha \) grows too, raising the critical difference and making individual pairwise comparisons harder to distinguish, even with the same underlying effect size.

Interview Questions

1. Why is a post-hoc test like the Nemenyi Test necessary after a significant Friedman Test, rather than just inspecting which condition has the highest rank sum?
2. Walk through how the critical difference formula is derived, and explain the role of the studentized range distribution in it.
3. Why does the Nemenyi Test use a single critical difference for all pairwise comparisons instead of a separate threshold for each pair?
4. What happens to the critical difference as the number of conditions \( k \) increases, holding sample size constant, and why?
5. How would you read and explain a critical difference diagram to someone unfamiliar with the Nemenyi Test?
6. Compare the conservativeness of the Nemenyi Test to running repeated Wilcoxon Signed-Rank Tests with no correction at all.
7. Under what circumstances might you prefer Conover's post-hoc test over the Nemenyi Test?
8. Is it appropriate to run the Nemenyi Test when the Friedman Test itself was not significant? Why or why not?
9. Explain what each off-diagonal cell in scikit-posthocs's posthoc_nemenyi_friedman output represents.
10. How does statistical power for a specific pairwise comparison change as the total number of subjects \( n \) increases, all else held equal?

Frequently Asked Questions

• What is the Nemenyi Post-Hoc Test used for? +
  The Nemenyi Post-Hoc Test is used after a significant Friedman Test to pin down exactly which pairs of three or more related conditions differ from one another. The Friedman Test alone only tells you that a difference exists somewhere among the conditions-it does not say which specific pair is responsible.
• What is the formula for the Nemenyi Post-Hoc Test? +
  Starting from the same within-subject ranks used by the Friedman Test, compute the average rank for each condition (rank sum divided by number of subjects, n). For every pair of conditions, compare the absolute difference in average ranks against a single critical difference, CD = q_alpha · sqrt(k(k+1) / (6n)), where q_alpha comes from the studentized range distribution for k conditions at your chosen significance level. Any pair whose average-rank difference exceeds CD is declared significantly different.
• What is the difference between the Nemenyi Test and the Friedman Test? +
  The Friedman Test is an omnibus test-like a one-way ANOVA's F-test-that only establishes whether at least one condition differs from the others across the whole set of k related conditions. The Nemenyi Test is the pairwise follow-up: it reuses the same rank sums to test every pair of conditions individually while controlling the overall false-positive rate across all those comparisons.
• What are the assumptions of the Nemenyi Post-Hoc Test? +
  Since it operates on the same ranks as the Friedman Test, it inherits that test's assumptions: independent subjects (blocks), a complete design where every subject has a value for every condition, and at least ordinal data. It is conventionally run only after the Friedman Test itself returns a significant p-value, though some analysts run it regardless and rely on its own correction for multiple comparisons.
• How do you interpret Nemenyi Post-Hoc Test results? +
  Each pairwise comparison is judged independently: if the absolute difference in average ranks exceeds the critical difference (equivalently, if the pairwise p-value is below your significance level), that pair of conditions is significantly different. It is entirely possible-and common-for the omnibus Friedman Test to be significant while several individual pairs are not, as shown in the worked examples on this page.
• Is the Nemenyi Test conservative? +
  Yes, noticeably so. Because it controls the family-wise error rate across all C(k,2) pairwise comparisons simultaneously using the studentized range distribution, it demands a larger average-rank gap to declare significance than an uncorrected series of Wilcoxon Signed-Rank Tests would-appropriate given how many comparisons are being made, but it does mean real, smaller differences can be missed, especially with few subjects.
• Can the Nemenyi Test be used without running the Friedman Test first? +
  It is designed as a post-hoc procedure for repeated-measures (blocked) designs and its critical difference formula assumes the same rank-sum structure the Friedman Test produces, so in practice it is always paired with a prior Friedman Test-typically run only when that omnibus test is significant, to avoid an extra, unnecessary layer of multiple-comparison correction.

Key Takeaways

• The Nemenyi Post-Hoc Test is the standard follow-up to a significant Friedman Test, identifying exactly which pairs of related conditions differ rather than just confirming a difference exists somewhere.
• It compares every pair of average ranks against a single critical difference, \( CD = q_\alpha \sqrt{k(k+1)/(6n)} \), controlling the family-wise error rate across all comparisons at once.
• Core assumptions are inherited directly from the Friedman Test: a complete block design and independence across subjects-no normality is required.
• In Python, scikit_posthocs.posthoc_nemenyi_friedman(...) returns a full matrix of pairwise p-values that matches the hand-calculated critical-difference conclusions exactly, as shown in both worked examples above.
• A significant omnibus Friedman result does not imply every pair differs-both worked examples on this page found only one of three possible pairs significant under the Nemenyi Test, underscoring why this step matters.
• The test is intentionally conservative; if you only need a single, pre-specified pairwise comparison rather than all pairs, the Wilcoxon Signed-Rank Test alone may be more appropriate and more powerful.

The Nemenyi Post-Hoc Test completes the story the Friedman Test starts. An omnibus test answering only "does something differ?" is rarely the end goal of a repeated-measures analysis-researchers and analysts almost always want to know specifically which conditions moved, and by how confidently. The Nemenyi Test supplies that answer using the same rank-based machinery already built for the Friedman Test, adding a single, principled critical difference that keeps the overall risk of a false pairwise claim under control even as the number of comparisons grows.

The two worked examples above, continuing directly from the same patients and students used throughout the Friedman Test guide, show a consistent pattern: a significant omnibus result driven by one clear endpoint-to-endpoint difference, while the intermediate, smaller gaps did not individually clear the stricter pairwise bar. Reporting the Friedman \( Q \) and p-value alongside the full Nemenyi pairwise comparison table, as demonstrated throughout this guide, gives a complete and honest account of a repeated-measures result that an omnibus p-value alone cannot provide.