Dunn's Post-Hoc Test

Introduction

Dunn's Post-Hoc Test is a non-parametric procedure for pairwise comparison of independent groups following a significant Kruskal-Wallis Test. The Kruskal-Wallis Test is an omnibus test — it establishes that at least one group differs among three or more, but does not identify which pair. Dunn's Test resolves that ambiguity by converting each pairwise mean-rank difference into a z-statistic and adjusting the resulting p-values for the number of simultaneous comparisons.

This article covers the conditions for use, key assumptions, the complete formula, Bonferroni vs. Holm adjustment, two fully worked numerical examples, and a Python walkthrough with scikit-posthocs.

What Is Dunn's Post-Hoc Test?

Proposed by Olive Jean Dunn (1964), the test operates on the combined rank sums already computed during the Kruskal-Wallis Test. For each pair of groups, the difference in mean ranks is standardized into a z-statistic and converted to a two-tailed p-value; those p-values are then corrected — typically by Bonferroni or Holm — to bound the family-wise error rate across all \( \binom{k}{2} \) comparisons.

Its role is to disaggregate the omnibus result: a significant Kruskal-Wallis Test establishes that the group distributions are not all equal, but cannot attribute that difference to any specific pair. Dunn's Test produces a pair-by-pair verdict while maintaining a controlled overall false-positive rate.

When to Use It

Apply Dunn's Test when a Kruskal-Wallis Test on three or more independent groups yields a significant result (\( p < 0.05 \)) and the specific pairs responsible for that result must be identified. It is appropriate under exactly the same conditions as the Kruskal-Wallis Test: independent groups, at least ordinal data, no normality assumption.

Key Assumptions

• Independent-groups design. The same structure required by the Kruskal-Wallis Test: three or more groups, each observation belonging to exactly one group, with the combined ranking step already completed.
• Independence of observations. No subject appears in more than one group; observations within a group do not influence one another.
• At least ordinal measurement. Values must support meaningful rank ordering.
• Significant prior omnibus result (conventional). The test is conventionally applied only after the Kruskal-Wallis Test rejects \( H_0 \), avoiding a spurious pairwise search on a non-significant omnibus result.

Hypotheses

A separate hypothesis pair is stated for every combination of two groups. For groups \( i \) and \( j \):

• Null hypothesis \( H_0 \): groups \( i \) and \( j \) are drawn from the same distribution; any observed difference in mean ranks is attributable to chance.
• Alternative hypothesis \( H_1 \): groups \( i \) and \( j \) are drawn from different distributions; their mean ranks differ systematically.

With \( k \) groups, \( \binom{k}{2} \) such pairs are tested simultaneously — 3 comparisons for \( k = 3 \), 6 for \( k = 4 \). Testing each at the uncorrected \( \alpha = 0.05 \) level would allow the family-wise false-positive rate to exceed 5% substantially, motivating the p-value adjustment step.

The Formula, Explained

Given rank sums \( R_j \) for each group \( j \) of size \( n_j \), computed over a pooled sample of \( N \) observations (as in the preceding Kruskal-Wallis Test):

1. Compute the mean rank for each group: \( \bar{R}_j = R_j / n_j \).
2. For every pair \( (i, j) \), compute a z-statistic from the mean-rank difference.
3. Convert each z-statistic to a two-tailed p-value; apply a multiple-comparison correction across all pairs.

The z-statistic for groups \( i \) and \( j \) is:

\( z_{ij} \) follows a standard normal distribution; the two-tailed p-value is obtained directly from \( \Phi \). When ties span multiple groups, the denominator is adjusted by the standard tie-correction factor (the same applied in the Kruskal-Wallis \( H \) statistic). Raw p-values are never reported directly against \( \alpha \) — a multiple-comparison adjustment is always applied first.

P-Value Adjustment: Bonferroni vs Holm

With \( m = \binom{k}{2} \) simultaneous comparisons, comparing each raw p-value directly against \( \alpha = 0.05 \) allows the family-wise error rate (FWER) to exceed 5% substantially. Two corrections are standard:

• Bonferroni. Multiply each raw p-value by \( m \) (capped at 1.0), or equivalently compare each raw p-value against \( \alpha / m \). Guarantees FWER \( \leq \alpha \) but is conservative — every comparison receives the same, maximum-possible penalty regardless of the individual evidence.
• Holm (step-down). Sort the \( m \) raw p-values from smallest to largest; compare the \( r \)-th smallest against \( \alpha / (m - r + 1) \), stopping at the first non-rejection (all larger p-values are then declared non-significant). Controls the same FWER as Bonferroni but is uniformly more powerful — it never rejects fewer hypotheses, and often rejects more.

Worked Example 1: A Small Numerical Example by Hand

Continuing from Worked Example 1 in the Kruskal-Wallis Test guide: weight loss (kg) across three diets, 5 subjects per diet. The Kruskal-Wallis Test returned \( H \approx 6.85 \), \( p \approx 0.033 \), warranting a post-hoc follow-up.

Rank sums from the omnibus test: \( R_A = 58.5 \), \( R_B = 40 \), \( R_C = 21.5 \); \( n_A = n_B = n_C = 5 \), \( N = 15 \).

Step 1: Convert to mean ranks

Step 2: Compute the z-statistic for each pair

Equal group sizes (\( n_i = n_j = 5 \)) yield a common standard error across all three pairs: \( \sqrt{\frac{15 \times 16}{12} \cdot \frac{2}{5}} = \sqrt{8} \approx 2.828 \).

Step 3: Convert to raw p-values, then adjust

With \( m = 3 \) pairwise comparisons, Bonferroni multiplies each raw p-value by 3:

Worked Example 2: Comparing Three Training Methods

Continuing from Worked Example 2 in the Kruskal-Wallis Test guide: performance scores for 7 employees (new method), 6 (old method), and 6 (self-guided). The Kruskal-Wallis Test returned \( H \approx 15.64 \), \( p \approx 0.0004 \).

Rank sums: \( R_{\text{New}} = 112 \), \( R_{\text{Old}} = 56 \), \( R_{\text{SG}} = 22 \); \( n_{\text{New}} = 7 \), \( n_{\text{Old}} = n_{\text{SG}} = 6 \), \( N = 19 \).

Step 1: Convert to mean ranks

Step 2: Compute the z-statistic for each pair

Group sizes are unequal, so the standard error is recomputed for each pair. The shared factor \( \sqrt{N(N+1)/12} = \sqrt{19 \times 20/12} \approx 5.627 \) carries through all three:

Step 3: Convert to raw p-values, then adjust

With \( m = 3 \) pairwise comparisons, Bonferroni multiplies each raw p-value by 3:

The data in this example are used verbatim in the Python walkthrough below, enabling direct verification of the hand calculations against scikit-posthocs.

Python Example

The omnibus test runs via scipy.stats.kruskal; pairwise comparisons via scikit-posthocs:

from scipy import stats
import pandas as pd
import scikit_posthocs as sp

new_method  = [85, 90, 88, 92, 84, 91, 89]
old_method  = [78, 82, 75, 80, 79, 77]
self_guided = [70, 74, 68, 72, 76, 71]

# Step 1: confirm the omnibus Kruskal-Wallis Test is significant
h_stat, p_value = stats.kruskal(new_method, old_method, self_guided)
print(f"Kruskal-Wallis H: {h_stat:.3f}, p-value: {p_value:.4f}")

# Step 2: build a long-format DataFrame for the post-hoc test
data = (
    [("New", v) for v in new_method]
    + [("Old", v) for v in old_method]
    + [("Self-Guided", v) for v in self_guided]
)
df = pd.DataFrame(data, columns=["group", "score"])

# Step 3: run Dunn's Test with Bonferroni correction
dunn_bonf = sp.posthoc_dunn(df, val_col="score", group_col="group", p_adjust="bonferroni")
print(dunn_bonf)

# Step 4: compare against Holm correction
dunn_holm = sp.posthoc_dunn(df, val_col="score", group_col="group", p_adjust="holm")
print(dunn_holm)

Output (Bonferroni-adjusted):

Kruskal-Wallis H: 15.639, p-value: 0.0004
                 New       Old  Self-Guided
New          1.00000  0.099660     0.000245
Old          0.09966  1.000000     0.243392
Self-Guided  0.00025  0.243392     1.000000

Each off-diagonal cell is the Bonferroni-adjusted pairwise p-value. New/Self-Guided (\( p \approx 0.0002 \)) is significant; New/Old (\( p \approx 0.10 \)) is not — consistent with the hand calculation in Worked Example 2. Switching to p_adjust="holm" yields marginally lower adjusted p-values for all non-extreme pairs but does not change the significance conclusions in this dataset.

Visualizing Results with a Significance Plot

import matplotlib.pyplot as plt

sp.sign_plot(dunn_bonf)
plt.title("Dunn's Test: Pairwise Significance (Bonferroni-Adjusted)")
plt.show()

sign_plot renders a heatmap shaded by significance level — particularly useful when \( k > 4 \), where a plain p-value matrix becomes difficult to scan.

How to Interpret Results

All decisions are made on the adjusted p-value for each pair, not the raw z-derived value. The conventional threshold is \( \alpha = 0.05 \).

Checking Assumptions in Practice

• Significant omnibus result. Confirm the Kruskal-Wallis Test rejected \( H_0 \) before proceeding; the correction is calibrated for a post-hoc context.
• Independent groups. No observation may appear in more than one group.
• Scale of \( k \). As \( k \) increases, \( \binom{k}{2} \) grows rapidly and Bonferroni in particular becomes increasingly conservative. Restrict comparisons to pairs of a priori interest where feasible.
• Correction choice declared in advance. Selecting Bonferroni vs. Holm after inspecting results invalidates the correction's FWER guarantee.
• Ties. Most software applies the standard tie correction to the denominator automatically; verify this is active when tied values span multiple groups.

Advantages

• Provides a direct pair-by-pair verdict after the Kruskal-Wallis Test's omnibus finding.
• Reuses the rank sums already computed for the Kruskal-Wallis Test — no additional ranking required.
• Handles unequal group sizes naturally through the per-pair standard error term.
• Supports multiple adjustment methods (Bonferroni, Holm, Benjamini–Hochberg, etc.), allowing the analyst to select the appropriate FWER or FDR trade-off.

Limitations

• Conservative by design: the correction that controls false positives simultaneously reduces power for detecting real moderate differences, as illustrated in both worked examples.
• Valid only as a post-hoc complement to the Kruskal-Wallis Test on the same independent-groups data; it is not a general-purpose pairwise comparison procedure.
• Power decreases as \( k \) increases, since the adjusted significance threshold tightens with each additional comparison.
• Bonferroni is noticeably conservative with many groups; Holm's correction is uniformly preferable when software supports it.

When NOT to Use It

• Two groups only. Use the Mann-Whitney U Test directly; no post-hoc disambiguation is needed.
• Related (repeated-measures) groups. Use the Nemenyi Test as a follow-up to a significant Friedman Test.
• Parametric assumptions satisfied. Use Tukey's HSD Test after a significant one-way ANOVA.
• Single pre-specified comparison. A single Mann-Whitney U Test is more powerful than applying Dunn's correction across all pairs when only one pair was of interest a priori.

Dunn's Test vs Nemenyi vs Tukey's HSD vs Bonferroni

16.1 Dunn's Test vs Nemenyi Test

16.2 Dunn's Test vs Tukey's HSD Test

16.3 Dunn's Test vs Plain Bonferroni-Corrected Mann-Whitney Tests

Common Misconceptions

• "A significant Kruskal-Wallis result means every pair differs." Incorrect. Both worked examples here show a significant omnibus result in which only one of three possible pairs clears Dunn's adjusted threshold.
• "Dunn's Test can be used without a prior Kruskal-Wallis Test." The test is designed as a post-hoc procedure and is conventionally interpreted alongside a prior significant omnibus result. Its z-statistic reuses the Kruskal-Wallis rank structure directly.
• "A non-significant pairwise result proves the groups are equal." Failure to reject the pairwise null hypothesis establishes only insufficient evidence at the given sample size — not the absence of a true difference.
• "Bonferroni and Holm always agree." Any pair significant under Bonferroni is also significant under Holm, but Holm may additionally reject pairs that Bonferroni retains, since it applies a less uniform penalty.
• "More groups yield more significant findings." The opposite commonly occurs: as \( k \) grows, so does \( \binom{k}{2} \), tightening the adjusted threshold and reducing power per pair even at a fixed effect size.

Interview Questions

1. Why is a post-hoc test like Dunn's Test necessary after a significant Kruskal-Wallis Test, rather than just inspecting which group has the highest mean rank?
2. Walk through how the z-statistic for a single pairwise comparison is derived, and explain the role of each term in the standard error.
3. Why does Dunn's Test require a p-value adjustment across all pairs, rather than comparing each raw p-value directly to 0.05?
4. Explain the difference between Bonferroni and Holm correction, and why Holm is generally preferred when available.
5. How does an unequal group size affect the standard error used in a specific pairwise z-statistic, compared to the equal-group-size case?
6. Compare Dunn's Test to simply running repeated Mann-Whitney U Tests with no correction at all.
7. Under what circumstances might you prefer the Nemenyi Test over Dunn's Test for a multi-group comparison?
8. Is it appropriate to run Dunn's Test when the Kruskal-Wallis Test itself was not significant? Why or why not?
9. Explain what each off-diagonal cell in scikit-posthocs's posthoc_dunn output represents.
10. How does statistical power for a specific pairwise comparison change as the total number of groups \( k \) increases, holding each group's sample size constant?

Frequently Asked Questions

• What is Dunn's Post-Hoc Test used for? +
  Dunn's Post-Hoc Test is used after a significant Kruskal-Wallis Test to pin down exactly which pairs of three or more independent groups differ from one another. The Kruskal-Wallis Test alone only tells you that a difference exists somewhere among the groups-it does not say which specific pair is responsible.
• What is the formula for Dunn's Post-Hoc Test? +
  Starting from the same combined ranks used by the Kruskal-Wallis Test, compute the mean rank for each group (rank sum divided by group size, R̄ⱼ = Rⱼ / nⱼ). For every pair of groups i and j, compute z = (R̄ᵢ − R̄ⱼ) / sqrt[(N(N+1)/12)(1/nᵢ + 1/nⱼ)], convert z to a two-tailed p-value from the standard normal distribution, then adjust that p-value for the total number of comparisons made, typically with a Bonferroni or Holm correction.
• What is the difference between Dunn's Test and the Kruskal-Wallis Test? +
  The Kruskal-Wallis Test is an omnibus test-like a one-way ANOVA's F-test-that only establishes whether at least one group differs from the others across the whole set of k independent groups. Dunn's Test is the pairwise follow-up: it reuses the same combined rank sums to test every pair of groups individually while adjusting for the overall false-positive risk across all those comparisons.
• What are the assumptions of Dunn's Post-Hoc Test? +
  Since it operates on the same ranks as the Kruskal-Wallis Test, it inherits that test's assumptions: independent groups, independent observations within each group, and at least ordinal data. It is conventionally run only after the Kruskal-Wallis Test itself returns a significant p-value, though some analysts run it regardless and rely on its own correction for multiple comparisons.
• How do you interpret Dunn's Post-Hoc Test results? +
  Each pairwise comparison is judged independently: if the adjusted p-value falls below your significance level, that pair of groups is significantly different. It is entirely possible-and common-for the omnibus Kruskal-Wallis Test to be significant while several individual pairs are not, as shown in the worked examples on this page.
• Should I use Bonferroni or Holm correction with Dunn's Test? +
  Holm's correction is generally preferred over Bonferroni whenever you have a choice-it controls the same family-wise error rate but is uniformly more powerful (never more conservative), since it adjusts each p-value based on its rank among all the comparisons rather than dividing every p-value by the same fixed number.
• Can Dunn's Test be used without running the Kruskal-Wallis Test first? +
  It is designed as a post-hoc procedure for independent-groups designs and its z-statistic reuses the same combined rank structure the Kruskal-Wallis Test produces, so in practice it is typically paired with a prior Kruskal-Wallis Test-conventionally run only when that omnibus test is significant, to avoid an extra, unnecessary layer of multiple-comparison correction.

Key Takeaways

• Dunn's Post-Hoc Test is the standard pairwise follow-up to a significant Kruskal-Wallis Test, identifying which specific group pairs account for the omnibus result.
• The z-statistic \( z_{ij} = (\bar{R}_i - \bar{R}_j) / \sqrt{(N(N+1)/12)(1/n_i + 1/n_j)} \) is derived from the combined rank sums; p-values are adjusted — typically by Bonferroni or Holm — to control the family-wise error rate.
• Assumptions (independent groups, at least ordinal data, no normality requirement) are inherited directly from the Kruskal-Wallis Test.
• scikit_posthocs.posthoc_dunn() returns a complete matrix of adjusted pairwise p-values consistent with the hand calculations in the worked examples above.
• A significant Kruskal-Wallis result does not imply all pairs differ — both examples here produced only one significant pair from three possible comparisons.
• Holm's correction is uniformly at least as powerful as Bonferroni and is preferred when software supports both.

Dunn's Post-Hoc Test converts the Kruskal-Wallis Test's omnibus finding into a pair-by-pair verdict. It reuses the same combined rank structure, applies an explicit multiple-comparison correction, and returns an adjusted p-value for every group pair — making the specific locus of a significant result identifiable without inflating the family-wise false-positive rate. Both worked examples demonstrate the essential point: a highly significant omnibus result and a conservative pairwise threshold can coexist, and only the full pairwise table reveals which comparisons are individually defensible.

Complete reporting requires stating the Kruskal-Wallis \( H \) and p-value alongside the full Dunn's Test pairwise comparison table, as shown throughout this article. An omnibus p-value alone is an incomplete account of an independent-groups result.