Tukey's HSD Test: When, Why, and How to Use It

Introduction

Tukey's Honest Significant Difference (HSD) Test answers the question a significant one-way ANOVA deliberately leaves open: which specific pair of independent groups actually differs? The ANOVA's F-test is an omnibus test-it tells you a difference exists somewhere among three or more group means, but not where. Tukey's HSD picks up exactly where the ANOVA leaves off, comparing every pair of group means directly while controlling the overall risk of a false-positive claim.

By the end of this article you will be able to state exactly when Tukey's HSD applies, compute one completely by hand on two worked examples, build a confidence interval for each pairwise comparison, and run the same test in a few lines of Python with statsmodels.stats.multicomp.pairwise_tukeyhsd.

What Is Tukey's HSD Test?

Tukey's HSD Test compares every possible pair of group means from a one-way ANOVA on three or more independent groups. Rather than running a separate, uncorrected t-test for each pair (which would inflate the chance of a false positive as the number of comparisons grows), it uses a single honest significant difference derived from the studentized range distribution, so that all pairwise comparisons share one consistent threshold controlling the overall family-wise error rate.

It exists because a significant one-way ANOVA, on its own, is an incomplete answer. Knowing that "the group means are not all the same" is rarely the question a researcher actually cares about-the real question is almost always "which groups differ, and from which others?" Tukey's HSD converts the ANOVA's omnibus signal into a concrete, pair-by-pair verdict, along with a confidence interval for the size of each difference.

When to Use It

Use Tukey's HSD Test immediately after a one-way ANOVA returns a significant result (\( p < 0.05 \)) on three or more independent groups, and you need to know specifically which groups are responsible for that significance-not just that a difference exists somewhere. It is the natural next step whenever a one-way ANOVA was the right choice: independent groups, approximately normal residuals, and roughly equal variances.

Key Assumptions

• Prior one-way ANOVA structure. Tukey's HSD operates on the same independent-groups design as the one-way ANOVA-three or more groups, with observations in each group unrelated to observations in any other group.
• Independence. Observations must be independent both within each group and across groups-no subject contributes to more than one group.
• Normality. The residuals (each observation minus its group mean) should be approximately normally distributed, inherited directly from the ANOVA it follows.
• Homogeneity of variance. The groups should have roughly equal variances-Tukey's HSD uses a single pooled mean squared error (MSE) across all groups, which assumes that pooling is appropriate.
• Conventionally follows a significant omnibus result. Most applied guidance runs Tukey's HSD only after the one-way ANOVA itself rejects \( H_0 \), though the test's own correction remains valid even when applied more liberally.

Hypotheses

Like the Nemenyi Test, Tukey's HSD evaluates a separate hypothesis pair for every combination of two groups, rather than a single omnibus hypothesis pair:

• Null Hypothesis (\( H_0^{(a,b)} \)): groups \( a \) and \( b \) have the same population mean-\( \mu_a = \mu_b \)-there is no real difference between this specific pair.
• Alternative Hypothesis (\( H_1^{(a,b)} \)): groups \( a \) and \( b \) have different population means-\( \mu_a \neq \mu_b \).

(With \( k \) groups there are \( \binom{k}{2} \) such pairs to test. The Tukey HSD threshold is constructed so that the probability of making at least one false-positive claim across all of these simultaneous comparisons stays at the nominal \( \alpha \) level.)

The Formula, Explained

Starting from a completed one-way ANOVA on \( k \) independent groups, each with \( n \) observations (a balanced design), with mean squared error \( MSE \) and within-groups degrees of freedom \( df_w = N - k \) (where \( N \) is the total sample size):

1. Compute the group means \( \bar{x}_1, \dots, \bar{x}_k \).
2. Look up the critical value \( q_\alpha \) from the studentized range distribution, indexed by \( k \) and \( df_w \).
3. Compute a single honest significant difference for the entire set of pairwise comparisons.
4. Compare the absolute difference in group means for every pair against that one HSD.

The honest significant difference is:

Two groups \( a \) and \( b \) are declared significantly different whenever:

For unbalanced designs (unequal group sizes), the Tukey-Kramer adjustment replaces \( n \) in the formula with the harmonic mean of the two group sizes being compared:

Notice that \( HSD \) does not depend on which specific pair of groups is being compared (in a balanced design)-it is a single number computed once for the whole design, which is what makes the test simple to apply by hand across many pairs at once.

Worked Example 1: A Small Numerical Example by Hand

Suppose an agricultural researcher measures plant growth (in cm) after 6 weeks under three different fertilizer treatments, with 5 independent plants per treatment (15 plants total, none shared between groups-unlike the repeated-measures designs in the Friedman Test guide, each plant receives exactly one treatment).

Step 1: Confirm the one-way ANOVA is significant

The group means are \( \bar{x}_A = 20.0 \), \( \bar{x}_B = 25.0 \), \( \bar{x}_C = 20.0 \). Running a one-way ANOVA on this data gives \( F = 16.67 \) with \( df_b = 2 \) and \( df_w = 12 \), for \( p \approx 0.0003 \)-well below 0.05, so a Tukey HSD follow-up is warranted.

Step 2: Compute the mean squared error

Summing squared deviations from each group's own mean gives \( SSW = 30.0 \) across \( df_w = 15 - 3 = 12 \) degrees of freedom:

Step 3: Compute the honest significant difference

With \( k = 3 \), \( df_w = 12 \), \( q_{0.05} \approx 3.773 \), and \( n = 5 \) per group:

Step 4: Compare every pair

Worked Example 2: Three Teaching Methods (Independent Groups)

Suppose 18 students are randomly assigned to one of three independent teaching methods (6 students per method, no student appearing in more than one group) and then take the same test at the end of the program-an independent-groups counterpart to the repeated-measures training data used in the Friedman Test guide, letting you compare how the two designs are analyzed differently.

Step 1: Confirm the one-way ANOVA is significant

The group means are \( \bar{x}_A \approx 61.17 \), \( \bar{x}_B \approx 75.17 \), \( \bar{x}_C = 67.5 \). The one-way ANOVA gives \( F \approx 44.61 \) with \( df_b = 2 \) and \( df_w = 15 \), for \( p \approx 0.0000005 \)-strongly significant.

Step 2: Compute the mean squared error

Step 3: Compute the honest significant difference

With \( k = 3 \), \( df_w = 15 \), \( q_{0.05} \approx 3.672 \), and \( n = 6 \) per group:

Step 4: Compare every pair

Unlike Worked Example 1, every pairwise comparison here clears the HSD threshold-Method B outperforms Method C, which in turn outperforms Method A, and every gap is large enough relative to the within-group variability to be distinguished from chance. This matches what statsmodels reports for this dataset in the Python section below.

Confidence Intervals for Each Pair

Beyond a significant/not-significant verdict, Tukey's HSD naturally produces a simultaneous confidence interval for the true difference between each pair of group means, using the same HSD value as the margin of error:

For Worked Example 2, the Method A vs Method B comparison has an observed difference of \( -14.0 \) (B minus A), giving a 95% simultaneous confidence interval of approximately \( [-14.0 - 3.854,\ -14.0 + 3.854] = [-17.85, -10.15] \). Because this interval excludes zero entirely, it confirms the pairwise significance found above-and it additionally quantifies the effect: Method B's true advantage over Method A is estimated to be somewhere between roughly 10 and 18 points.

Python Example

The one-way ANOVA runs through scipy.stats.f_oneway, and Tukey's HSD post-hoc comparison runs through statsmodels.stats.multicomp.pairwise_tukeyhsd:

import numpy as np
from scipy import stats
from statsmodels.stats.multicomp import pairwise_tukeyhsd

method_a = [62, 58, 65, 60, 63, 59]
method_b = [74, 79, 71, 76, 78, 73]
method_c = [68, 65, 70, 67, 69, 66]

# Step 1: confirm the omnibus one-way ANOVA is significant
f_stat, p_value = stats.f_oneway(method_a, method_b, method_c)
print(f"ANOVA F: {f_stat:.3f}, p-value: {p_value:.6f}")

# Step 2: run Tukey's HSD on the same groups
all_scores = np.array(method_a + method_b + method_c)
labels = np.array(["A"] * 6 + ["B"] * 6 + ["C"] * 6)

tukey_result = pairwise_tukeyhsd(all_scores, labels, alpha=0.05)
print(tukey_result)

Output:

ANOVA F: 44.605, p-value: 0.000000

 Multiple Comparison of Means - Tukey HSD, FWER=0.05
=====================================================
group1 group2 meandiff p-adj   lower    upper  reject
-----------------------------------------------------
     A      B     14.0    0.0  10.1441 17.8559   True
     A      C   6.3333 0.0018   2.4774 10.1892   True
     B      C  -7.6667 0.0003 -11.5226 -3.8108   True
-----------------------------------------------------

Every number here matches the hand calculation in Worked Example 2 exactly-the reported lower and upper columns are precisely the confidence interval computed in the previous section, and reject: True marks every pair as significant, consistent with all three comparisons exceeding the HSD threshold above.

How to Interpret Results

The significance level \( \alpha = 0.05 \) is the standard threshold used to decide whether a specific pairwise comparison is "statistically significant," applied consistently across every pair in the design.

Checking Assumptions in Practice

• Confirm the ANOVA was significant first: running Tukey HSD comparisons after a non-significant omnibus result is generally discouraged, since it risks chasing pairwise differences the overall test found no evidence for.
• Normality of residuals: a Q-Q plot or Shapiro-Wilk test on the ANOVA residuals- moderate departures are usually tolerated, especially with roughly equal group sizes.
• Homogeneity of variance: Levene's test or a simple visual comparison of each group's spread-if variances differ substantially, consider the Games-Howell post-hoc test instead, which does not assume equal variances.
• Balanced vs unbalanced design: confirm whether group sizes are equal; if not, ensure your software applies the Tukey-Kramer adjustment (most modern implementations, including statsmodels, do this automatically).

Advantages

• Directly answers the natural follow-up question after a significant one-way ANOVA: which groups specifically differ.
• Uses a single honest significant difference for the whole design (in balanced designs), making it simple to apply by hand across many pairs at once.
• Controls the family-wise error rate across all pairwise comparisons simultaneously, avoiding the inflated false-positive risk of running repeated uncorrected t-tests.
• Produces simultaneous confidence intervals for every pairwise difference, not just a significant/ not-significant label, giving a sense of effect size alongside significance.
• Generally more statistically powerful than a Bonferroni correction for this specific comparison structure (all pairs from one ANOVA), since it is calibrated exactly for that scenario.

Limitations

• Requires the same normality and equal-variance assumptions as the one-way ANOVA it follows-if those are badly violated, consider Games-Howell (unequal variances) or a non-parametric alternative instead.
• Only applies to independent groups-not repeated-measures or matched designs, which need the Friedman Test and Nemenyi Test instead.
• Statistical power drops as the number of groups \( k \) increases, since more comparisons raise \( q_\alpha \) and therefore the HSD threshold.
• The classic formula assumes equal group sizes; unbalanced designs require the Tukey-Kramer adjustment, which is only an approximation rather than an exact solution.

When NOT to Use It

• Nemenyi Test: use instead when your design is repeated-measures (the same subjects across all conditions) rather than independent groups-see the Friedman Test and Nemenyi Test guides.
• Games-Howell Test: use instead when group variances are clearly unequal, since Tukey's HSD assumes homogeneity of variance via its pooled MSE.
• Dunnett's Test: use instead when you only want to compare every group against a single control group, rather than every possible pair-Dunnett's Test is more powerful for that narrower question.
• Kruskal-Wallis Test with Dunn's post-hoc: use instead when the normality assumption cannot be met even approximately, for independent-groups data that should be ranked rather than analyzed on raw values.

Tukey HSD vs Bonferroni vs Scheffe vs Nemenyi

16.1 Tukey's HSD vs Bonferroni-Corrected t-tests

16.2 Tukey's HSD vs Scheffe's Test

16.3 Tukey's HSD vs the Nemenyi Test

Common Misconceptions

• "A significant ANOVA means every pair of groups differs." Not necessarily-see Worked Example 1, where the omnibus ANOVA was significant but only two of the three possible pairs cleared the HSD threshold; Fertilizer A and C were statistically indistinguishable.
• "Tukey's HSD can be run on its own, without an ANOVA." It is designed as a post-hoc procedure that reuses the ANOVA's pooled MSE and is conventionally interpreted alongside a prior significant omnibus result, not as a standalone test.
• "A non-significant pairwise comparison proves those two groups are identical." Failing to clear the HSD only means there was not enough evidence to distinguish that pair at this sample size-not proof that no real difference exists.
• "Tukey's HSD works for repeated-measures data too." It does not-it assumes independent groups. For the same subjects measured under multiple related conditions, use the Friedman Test and Nemenyi Test instead.
• "More groups always mean more significant pairwise findings." The opposite is often true-as \( k \) grows, \( q_\alpha \) grows too, raising the HSD threshold and making individual pairwise comparisons harder to distinguish, even with the same underlying effect size.

Interview Questions

1. Why is a post-hoc test like Tukey's HSD necessary after a significant one-way ANOVA, rather than just inspecting which group has the highest mean?
2. Walk through how the HSD formula is derived, and explain the role of the studentized range distribution in it.
3. Why does Tukey's HSD use a single threshold for all pairwise comparisons instead of a separate t-test for each pair?
4. What happens to the HSD as the number of groups \( k \) increases, holding sample size constant, and why?
5. How does the Tukey-Kramer adjustment modify the HSD formula for unbalanced designs?
6. Compare the statistical power of Tukey's HSD to a Bonferroni correction for the same set of pairwise comparisons.
7. Under what circumstances would you prefer the Games-Howell test over Tukey's HSD?
8. Is it appropriate to run Tukey's HSD when the one-way ANOVA itself was not significant? Why or why not?
9. Explain how a simultaneous confidence interval from Tukey's HSD relates to its significant/ not-significant verdict for the same pair.
10. Why can't Tukey's HSD be applied directly to the output of a Friedman Test?

Frequently Asked Questions

• What is Tukey's HSD Test used for? +
  Tukey's Honest Significant Difference Test is used after a significant one-way ANOVA to pin down exactly which pairs of three or more independent groups differ from one another. The ANOVA alone only tells you that a difference exists somewhere among the group means-it does not say which specific pair is responsible.
• What is the formula for Tukey's HSD Test? +
  Starting from the same mean squared error (MSE) and group means used by the one-way ANOVA, compute HSD = q_alpha * sqrt(MSE / n), where q_alpha comes from the studentized range distribution for k groups and the ANOVA's within-group degrees of freedom, and n is the sample size per group (or, for unequal group sizes, the Tukey-Kramer adjustment uses the harmonic mean of each pair's sizes). Any pair of group means whose absolute difference exceeds the HSD is declared significantly different.
• What is the difference between Tukey's HSD Test and the Nemenyi Test? +
  Both are post-hoc tests built on the studentized range distribution and use a single critical threshold for all pairwise comparisons, but Tukey's HSD is the parametric follow-up to a one-way ANOVA on independent groups, comparing raw group means and assuming normality and equal variances, while the Nemenyi Test is the non-parametric follow-up to a Friedman Test on related, repeated-measures data, comparing average ranks instead of raw means.
• What are the assumptions of Tukey's HSD Test? +
  Since it operates on the same ANOVA model, it inherits that test's assumptions: independent observations within and across groups, approximately normally distributed residuals, and homogeneity of variance across groups. It is conventionally run only after the one-way ANOVA itself returns a significant p-value.
• How do you interpret Tukey's HSD Test results? +
  Each pairwise comparison is judged independently: if the absolute difference in group means exceeds the honest significant difference (equivalently, if the pairwise adjusted p-value is below your significance level), that pair of groups is significantly different. It is entirely possible for the omnibus ANOVA to be significant while some individual pairs are not, though in balanced designs with a clear ordering it is also common for every pair to be significant, as shown in one of the worked examples on this page.
• Does Tukey's HSD Test require equal sample sizes? +
  The original Tukey HSD formula assumes a balanced design with equal sample sizes per group. When group sizes differ, the Tukey-Kramer adjustment replaces n in the formula with the harmonic mean of the two group sizes being compared, and this adjusted version is what most modern statistical software, including statsmodels, implements by default regardless of whether the design is balanced.
• How is Tukey's HSD different from a Bonferroni-corrected t-test? +
  Bonferroni correction divides the significance level by the number of comparisons made, which becomes increasingly conservative as the number of groups grows. Tukey's HSD instead uses the studentized range distribution, which is specifically calibrated for the exact set of all pairwise comparisons among group means from one ANOVA, generally giving it more statistical power than an equivalent Bonferroni correction for this particular comparison structure.

Key Takeaways

• Tukey's HSD Test is the standard follow-up to a significant one-way ANOVA, identifying exactly which pairs of independent groups differ rather than just confirming a difference exists somewhere.
• It compares every pair of group means against a single honest significant difference, \( HSD = q_\alpha \sqrt{MSE/n} \), controlling the family-wise error rate across all comparisons at once.
• Core assumptions are inherited directly from the one-way ANOVA: independence, normality, and homogeneity of variance across groups.
• In Python, statsmodels.stats.multicomp.pairwise_tukeyhsd(...) returns adjusted p-values and simultaneous confidence intervals for every pair, matching the hand-calculated HSD conclusions exactly, as shown in both worked examples above.
• A significant omnibus ANOVA does not automatically mean every pair differs-Worked Example 1 found only two of three possible pairs significant, while Worked Example 2 found all three, illustrating why the pairwise step matters in either direction.
• For repeated-measures designs rather than independent groups, use the Friedman Test followed by the Nemenyi Test instead.

Tukey's HSD Test completes the story a one-way ANOVA starts. An omnibus F-test answering only "does something differ?" is rarely the end goal of a comparative analysis-researchers and analysts almost always want to know specifically which groups moved, by how much, and how confidently. Tukey's HSD supplies that answer using the same pooled mean squared error already computed for the ANOVA, adding a single, principled threshold-and a matching confidence interval-that keeps the overall risk of a false pairwise claim under control even as the number of groups grows.

The two worked examples above show both sides of that logic in practice-a fertilizer study where only one treatment stood apart from an otherwise indistinguishable pair, and a teaching-methods comparison where every pairwise gap was large enough to be conclusive. Reporting the ANOVA's \( F \) and p-value alongside the full Tukey HSD pairwise comparison table and confidence intervals, as demonstrated throughout this guide, gives a complete and honest account of a multi-group result that an omnibus p-value alone cannot provide.