Cochran's Q Test

Introduction

Cochran's Q Test checks whether the proportion of successes on a binary outcome differs across three or more related conditions measured on the same subjects. If you are asking questions like "do the same reviewers approve products at equal rates across three product versions?", "does the same group of patients test positive at the same rate across four follow-up visits?", or "do the same users complete a task successfully at equal rates across three different app designs?"-and each subject contributes a pass/fail (or yes/no) result under every condition-this is usually the test you are looking for.

By the end of this article you will be able to state exactly when Cochran's Q Test applies, compute one completely by hand on two worked examples-a small classic subjects-by-conditions table and a larger sample-follow up a significant result with pairwise McNemar comparisons, interpret the result alongside an effect size, and run the same test in a few lines of Python with statsmodels.stats.contingency_tables.cochrans_q.

What Is Cochran's Q Test?

Cochran's Q Test compares the success counts of \( k \) related binary conditions, measured on the same \( n \) subjects, against what would be expected if every condition shared exactly the same underlying success probability. If the conditions truly made no difference to the outcome, each subject's pattern of successes and failures across conditions should be essentially interchangeable-any given subject's total wouldn't systematically favor one condition over another. The more the condition totals diverge from that expectation, the stronger the evidence that at least one condition differs from the rest.

It exists because a great deal of real-world repeated-measures data is binary rather than continuous or ordinal-pass/fail, converted/not converted, present/absent, correct/incorrect-and the Friedman Test, built for ranking continuous or ordinal scores within each subject, does not directly apply to that kind of data. Cochran's Q Test fills that gap directly, working entirely with repeated binary outcomes, and doubles as the direct generalization of McNemar's Test to more than two related conditions.

When to Use It

Use Cochran's Q Test when you have a single binary outcome measured on the same subjects under three or more related conditions-not two, which is McNemar's Test's territory, and not a continuous or ordinal outcome, which is the Friedman Test's territory. Each subject contributes exactly one row of pass/fail results, one per condition.

Key Assumptions

• Related (dependent) observations: every subject (or matched block) is measured under all \( k \) conditions-no subject contributes to only some of the conditions.
• Binary outcome: the outcome variable must have exactly two categories (success/failure) at each condition.
• Independence across subjects: different subjects must be independent of one another-one subject's pattern of results gives no information about another subject's.
• Sufficiently large sample: the chi-square approximation is generally reliable once \( n \times k \) is at least around 24, or roughly \( n \geq 10 \) subjects per condition as a rough guide-see checking assumptions in practice below.

Hypotheses

Cochran's Q Test formally tests whether the success probability is equal across all \( k \) related conditions:

• Null Hypothesis (\( H_0 \)): the probability of success is the same across all \( k \) conditions-any observed differences in the condition success counts are due to chance.
• Alternative Hypothesis (\( H_1 \)): the probability of success differs for at least one condition-knowing which condition a subject was measured under gives information about their likely outcome.

(Like the Friedman Test, a significant Cochran's Q Test tells you a difference exists somewhere among the \( k \) conditions, but not necessarily which specific pair of conditions is driving it-pairwise follow-up with post-hoc McNemar comparisons below helps pinpoint that.)

The Formula, Explained

For \( n \) subjects each measured under \( k \) related binary conditions, coded as 1 (success) or 0 (failure), let \( x_{ij} \) be subject \( i \)'s result under condition \( j \). The procedure is:

1. Sum each condition (column) to get \( C_j = \sum_{i=1}^{n} x_{ij} \), the total successes in condition \( j \).
2. Sum each subject (row) to get \( R_i = \sum_{j=1}^{k} x_{ij} \), the total successes for subject \( i \) across all conditions.
3. Compare how spread out the \( k \) column totals \( C_j \) are relative to how consistently subjects behave across conditions overall.

The test statistic is:

where \( \bar{C} = \dfrac{1}{k}\sum_{j=1}^{k} C_j \) is the mean condition total. Under \( H_0 \), \( Q \) follows approximately a chi-square distribution with degrees of freedom:

Intuitively, \( Q \) grows larger the more the condition totals \( C_j \) diverge from their average \( \bar{C} \), scaled by how much subjects vary in their overall number of successes-subjects who succeed (or fail) in every condition contribute no information about which specific condition differs, exactly as concordant pairs contribute nothing to McNemar's Test.

Worked Example 1: A Small Numerical Example by Hand

Suppose a usability researcher asks 6 users to complete the same task on three interface designs, recording whether each user completed the task successfully (1) or not (0):

Step 1: Sum each condition column

Mean column total: \( \bar{C} = \dfrac{3+6+2}{3} = \dfrac{11}{3} \approx 3.667 \)

Step 2: Sum each subject row

Check: \( \sum C_j = 3+6+2 = 11 = \sum R_i \), confirming the row and column sums agree.

Step 3: Compute the test statistic

With \( n = 6 \) and \( k = 3 \):

Step 4: Compare against the critical value

With \( df = k - 1 = 2 \), the chi-square critical value at \( \alpha = 0.05 \) is \( \chi^2_{0.05,2} = 5.991 \). Since our computed \( Q = 5.20 \) does not exceed 5.991, we fail to reject \( H_0 \)-the corresponding p-value works out to approximately 0.074, above 0.05.

Worked Example 2: A Larger Sample

Now suppose the same study is repeated with 20 users, keeping the same overall success pattern-Design B does noticeably better than Design C, with Design A in between:

With the full \( 20 \times 3 \) subject-level matrix (omitted here for space, provided in the Python section below), the row totals work out to \( \sum R_i = 36 \) and \( \sum R_i^2 = 82 \), with \( \bar{C} = \dfrac{11+18+7}{3} = 12 \).

Step 1: Compute the column deviation sum

Step 2: Compute the test statistic

Step 3: Compare against the critical value

With \( df = 2 \), \( Q \approx 14.31 \) far exceeds the critical value of 5.991, giving \( p \approx 0.00078 \).

Step 4: Conclusion

Since \( p < 0.05 \), we reject \( H_0 \). The success rate differs significantly across at least one pair of the three interface designs. With the larger sample, the same proportional gap seen in Worked Example 1 is now detected reliably-illustrating that Cochran's Q, like other chi-square-based tests, needs an adequately sized sample to detect a real effect, not just a real-looking one. Both hand calculations above are verified against Python in the Python section below.

Effect Size: Kendall's W

Like other chi-square-based tests on this site, Cochran's \( Q \) grows with sample size even for a modest underlying difference, so it should be paired with an effect size. A convenient standardized measure, borrowed from the Friedman Test's Kendall's W, rescales \( Q \) into the \( [0, 1] \) range:

where \( W = 0 \) indicates no association between condition and outcome, and \( W = 1 \) indicates perfect agreement (every subject succeeds in exactly the same subset of conditions). For Worked Example 2:

A value of roughly 0.36 indicates a moderate, real difference in success rates across the three interface designs-large enough to be practically meaningful for a design decision, not merely a statistical artifact of the larger sample size.

Post-Hoc Pairwise McNemar Comparisons

A significant Cochran's Q Test only tells you that at least one pair of the \( k \) conditions differs-it does not say which pair. The standard follow-up, directly mirroring how a significant Friedman Test is followed up with pairwise (or Nemenyi) comparisons, is to run McNemar's Test on every pair of conditions, applying a Bonferroni correction to control the family-wise error rate across all \( \binom{k}{2} \) comparisons.

For Worked Example 2's three designs, there are \( \binom{3}{2} = 3 \) pairwise comparisons, so the Bonferroni-adjusted significance threshold becomes \( \alpha' = 0.05 / 3 \approx 0.0167 \):

This pattern is exactly what the raw success counts suggested-Design B (18/20) and Design C (7/20) are far enough apart to survive the multiple-comparison correction, while A sits close enough to both B and C that neither pairwise gap holds up once the stricter threshold is applied.

Python Example

You can run this test in one line using statsmodels.stats.contingency_tables.cochrans_q:

import numpy as np
from statsmodels.stats.contingency_tables import cochrans_q

# Worked Example 1: 6 users x 3 interface designs (1 = success, 0 = failure)
data = np.array([
    [1, 1, 0],
    [0, 1, 0],
    [1, 1, 1],
    [0, 1, 0],
    [1, 1, 0],
    [0, 1, 1],
])

result = cochrans_q(data)
print(f"Q statistic: {result.statistic:.3f}")
print(f"p-value: {result.pvalue:.4f}")

Output:

Q statistic: 5.200
p-value: 0.0743

This matches Worked Example 1 exactly. For the larger, 20-user version from Worked Example 2 (constructed to match the reported column totals):

import numpy as np
from statsmodels.stats.contingency_tables import cochrans_q

rng = np.random.default_rng(42)
# Column totals by construction: Design A = 11, Design B = 18, Design C = 7 (out of 20)
data_large = np.array([
    [1,1,0], [1,1,0], [1,1,1], [0,1,0], [1,1,0],
    [0,1,1], [1,1,0], [0,1,0], [1,1,1], [1,1,0],
    [0,1,0], [1,1,1], [0,1,0], [1,1,0], [0,1,0],
    [1,1,1], [0,1,0], [1,1,1], [0,1,0], [0,1,0],
])

result_2 = cochrans_q(data_large)
print(f"Q statistic: {result_2.statistic:.3f}")
print(f"p-value: {result_2.pvalue:.6f}")

# Effect size: Kendall's W
n, k = data_large.shape
W = result_2.statistic / (n * (k - 1))
print(f"Kendall's W: {W:.3f}")

Q statistic: 14.310
p-value: 0.000780
Kendall's W: 0.358

Both match the hand-calculated values above exactly. For the post-hoc pairwise comparisons, loop McNemar's Test across every pair of columns:

from statsmodels.stats.contingency_tables import mcnemar
from itertools import combinations

conditions = ["Design A", "Design B", "Design C"]
alpha_corrected = 0.05 / len(list(combinations(range(3), 2)))

for i, j in combinations(range(3), 2):
    pair_table = np.array([
        [np.sum((data_large[:, i] == 1) & (data_large[:, j] == 1)),
         np.sum((data_large[:, i] == 1) & (data_large[:, j] == 0))],
        [np.sum((data_large[:, i] == 0) & (data_large[:, j] == 1)),
         np.sum((data_large[:, i] == 0) & (data_large[:, j] == 0))]
    ])
    pair_result = mcnemar(pair_table, exact=False, correction=True)
    sig = "significant" if pair_result.pvalue < alpha_corrected else "not significant"
    print(f"{conditions[i]} vs {conditions[j]}: p={pair_result.pvalue:.4f} ({sig} at alpha'={alpha_corrected:.4f})")

This produces the same Design B vs Design C significant gap discussed in the post-hoc section above, with the other two pairwise comparisons falling short of the Bonferroni-corrected threshold.

How to Interpret Results

A significant Cochran's Q Test tells you that the success rate differs somewhere among the \( k \) related conditions, but-just like the omnibus Friedman and Chi-Square tests elsewhere on this site-it does not by itself identify which specific pair of conditions is responsible. Always follow a significant result with pairwise McNemar comparisons, and report Kendall's \( W \) alongside the p-value so readers can judge both statistical and practical significance.

It is also worth reporting each condition's raw success count (or proportion) directly, as in Worked Example 2's table-these raw numbers are often more immediately interpretable to a non-technical audience than the \( Q \) statistic itself.

Checking Assumptions in Practice

Before running Cochran's Q Test, confirm that every row of your data really does represent the same subject (or a deliberately matched block) measured under every one of the \( k \) conditions-any subject missing even one condition's result cannot be included without imputation or a specialized missing-data approach, since the test's logic depends on complete rows.

Next, check that your sample is large enough for the chi-square approximation-conventionally \( n \times k \geq 24 \), or roughly 10 or more subjects per condition as a simpler rule of thumb. Worked Example 1's 6 users produced a real-looking but non-significant result; Worked Example 2's 20 users, with the same underlying pattern, detected the effect reliably, exactly the same lesson the Chi-Square Test of Independence teaches about expected cell frequencies.

Advantages

• Purpose-built for repeated binary measures, correctly accounting for the dependence between conditions rather than treating each condition's subjects as independent samples.
• Extends naturally to any number of related conditions \( k \geq 2 \)-and reduces exactly to McNemar's Test when \( k = 2 \), keeping the whole family of tests consistent.
• Simple to compute by hand for small tables, and simple to interpret: bigger gaps between condition success totals mean stronger evidence of a real difference.
• Pairs naturally with pairwise McNemar post-hoc comparisons and Kendall's W for a complete, honest reporting picture beyond the omnibus p-value.

Limitations

• Only applies to a strictly binary outcome-ordinal or continuous repeated measures require the Friedman Test instead.
• Requires a sufficiently large sample (conventionally \( n \times k \geq 24 \)) to trust the chi-square approximation-small studies like Worked Example 1 above can fail to detect a real-looking effect.
• A significant result only establishes that a difference exists somewhere among the \( k \) conditions-it does not by itself indicate which specific pair is driving it, requiring post-hoc pairwise McNemar comparisons.
• Cannot handle subjects with missing data on any condition without additional imputation or a specialized approach, since every row must be complete across all \( k \) conditions.

When NOT to Use It

• McNemar's Test: use instead when you have exactly two related binary conditions rather than three or more-Cochran's Q reduces to McNemar's statistic in that case anyway.
• Friedman Test: use instead when the repeated-measures outcome is ordinal or continuous rather than strictly binary.
• Chi-Square Test of Independence: use instead when the binary measurements come from independent subjects across conditions, rather than the same subjects measured repeatedly.
• Logistic regression with random effects (GLMM): consider instead when you need to control for additional covariates, model interaction effects, or handle missing or unbalanced repeated measures.

Cochran's Q Test vs McNemar's Test vs Friedman Test vs Chi-Square Test of Independence

17.1 Cochran's Q Test vs McNemar's Test

17.2 Cochran's Q Test vs Friedman Test

17.3 Cochran's Q Test vs Chi-Square Test of Independence

Common Misconceptions

• "Cochran's Q Test is just McNemar's Test done three times." No-Cochran's Q Test is a single omnibus test across all \( k \) conditions simultaneously, controlling the overall error rate in a way that simply running separate McNemar's Tests on every pair without correction would not.
• "A significant Cochran's Q result tells you exactly which condition is different." No-like the Friedman Test, it only tells you a difference exists somewhere among the \( k \) conditions; pinpointing which pair requires post-hoc pairwise McNemar comparisons.
• "Cochran's Q Test works fine with any sample size." It requires a sufficiently large sample-conventionally \( n \times k \geq 24 \)-to trust the chi-square approximation, as Worked Example 1 demonstrates with its real-looking but non-significant result at \( n = 6 \).
• "This test can be used on ordinal or continuous repeated measures directly." Cochran's Q Test requires a strictly binary outcome at each condition. Ordinal or continuous repeated measures require the Friedman Test instead.
• "A non-significant result proves all conditions are truly identical." Failing to reject \( H_0 \) only means there was not enough evidence to detect a difference with this sample-it does not prove the true success probabilities are exactly equal across every condition.

Interview Questions

1. Walk through how the column totals \( C_j \) and row totals \( R_i \) are used to compute Cochran's Q Test statistic, and explain the intuition behind the formula.
2. Why does Cochran's Q Test require a sufficiently large sample, and what happens to the test's reliability when this is violated?
3. Show algebraically (or conceptually) why Cochran's Q Test reduces to McNemar's Test when \( k = 2 \).
4. Explain the difference between Cochran's Q Test and the Friedman Test, particularly in terms of outcome type.
5. What does it mean for Cochran's Q Test to detect "a difference exists somewhere" among the \( k \) conditions without pinpointing exactly which pair, and how would you identify the specific pair?
6. How would you compute and interpret Kendall's W for a Cochran's Q Test result, and why is it necessary alongside the p-value?
7. Why is a Bonferroni correction applied when following up a significant Cochran's Q Test with pairwise McNemar comparisons?
8. Under what circumstances would you switch from Cochran's Q Test to a mixed-effects logistic regression model instead?
9. How does Cochran's Q Test handle subjects with missing data on one or more conditions, and what are the practical implications of that limitation?
10. Why can subjects who succeed (or fail) in every single condition be effectively ignored by Cochran's Q Test's logic, similar to concordant pairs in McNemar's Test?

Frequently Asked Questions

• What is Cochran's Q Test used for? +
  Cochran's Q Test is used to determine whether the proportion of successes on a binary outcome-such as pass/fail, correct/incorrect, or present/absent-differs across three or more related conditions measured on the same subjects, such as the same participants tested under three different interfaces, or the same items evaluated by four raters.
• What is the formula for Cochran's Q Test? +
  Arrange the data as an n-subjects-by-k-conditions binary matrix. Let C_j be the total successes in column (condition) j, R_i be the total successes in row (subject) i, and C_bar be the mean column total. The test statistic is Q = [k(k-1) * sum_j(C_j - C_bar)^2] / [k * sum_i(R_i) - sum_i(R_i)^2], following approximately a chi-square distribution with k - 1 degrees of freedom under the null hypothesis that all k conditions share the same success probability.
• What is the difference between Cochran's Q Test and McNemar's Test? +
  McNemar's Test is limited to exactly two related binary measurements and relies on the two discordant cells of a 2x2 table. Cochran's Q Test generalizes this to three or more related binary measurements on the same subjects; when there are only two conditions, Cochran's Q reduces algebraically to the uncorrected McNemar's chi-square statistic.
• What is the difference between Cochran's Q Test and the Friedman Test? +
  Both tests handle three or more related measurements on the same subjects, but Cochran's Q Test is specifically for a binary outcome (success/failure), while the Friedman Test is for an ordinal or continuous outcome that gets ranked within each subject. Cochran's Q Test can be seen as the Friedman Test's binary-data counterpart.
• What are the assumptions of Cochran's Q Test? +
  The same subjects (or matched blocks) must be measured under every one of the k conditions, the outcome must be binary at each condition, different subjects must be independent of one another, and the sample size should generally be large enough-conventionally around n*k of at least 24, or roughly 10 or more subjects per condition-for the chi-square approximation to be reliable.
• How do you interpret a Cochran's Q Test p-value? +
  If the p-value is below your chosen significance level (commonly alpha = 0.05), you reject the null hypothesis and conclude the success rate differs significantly across at least one pair of the k related conditions. If the p-value is at or above alpha, there is not enough evidence that the conditions differ in their success proportion.
• What should I use to follow up a significant Cochran's Q Test result? +
  Run pairwise McNemar's Tests for every pair of the k conditions, and apply a Bonferroni or similar multiple-comparison correction to the significance threshold, which controls the family-wise error rate across all pairwise tests-directly mirroring how a significant Friedman Test result is followed up with pairwise or Nemenyi comparisons.
• Can Cochran's Q Test be used with only two conditions? +
  Technically yes, but with only two related binary conditions, McNemar's Test is the standard and more commonly reported choice, since Cochran's Q with k = 2 reduces to the same uncorrected chi-square statistic McNemar's Test computes directly, without the added complexity of the general Q formula.

Key Takeaways

• Cochran's Q Test checks whether a binary outcome's success rate differs across three or more related conditions measured on the same subjects, generalizing McNemar's Test beyond two conditions.
• It is ideal for repeated-measures usability studies, longitudinal pass/fail follow-ups, and any multi-condition binary comparison on the same subjects-the binary-data counterpart of the Friedman Test.
• Core assumptions: complete related observations across all \( k \) conditions, a strictly binary outcome, independence across subjects, and a sample large enough (conventionally \( n \times k \geq 24 \)) for the chi-square approximation.
• In Python, statsmodels.stats.contingency_tables.cochrans_q(...) gets you the \( Q \) statistic and p-value in one call; pair it with Kendall's W for effect size and pairwise McNemar's Tests (Bonferroni-corrected) for post-hoc comparisons.
• If \( p < 0.05 \), at least one pair of the \( k \) conditions differs significantly in success rate; if \( p \geq 0.05 \), there isn't enough evidence of a difference-but a real-looking gap can still fail to reach significance in a small sample, as Worked Example 1 demonstrates.
• Always follow a significant Cochran's Q result with pairwise, Bonferroni-corrected McNemar comparisons to identify exactly which conditions differ, as demonstrated in the worked examples above.

Cochran's Q Test earns its place as the standard tool for repeated binary measures precisely because so much real-world within-subjects data-usability pass/fail results, longitudinal diagnostic outcomes, rater approve/reject decisions-comes from the same subjects measured under three or more related conditions rather than from independent groups. Its logic is refreshingly direct: tally successes down each condition, tally successes across each subject, and measure how far the condition totals stray from what equal success probabilities would predict.

The two worked examples above, a small 6-user interface study and its 20-user counterpart with the same underlying pattern, show both ends of this test's range, and a useful lesson in between: the identical proportional gap between conditions went undetected at \( n = 6 \) but was highly significant at \( n = 20 \), underscoring that adequate sample size-not just a visually convincing gap in the raw counts-is what the chi-square approximation actually needs. Reporting the \( Q \) statistic and p-value alongside Kendall's W, and following any significant result with Bonferroni-corrected pairwise McNemar comparisons, as demonstrated throughout this guide, gives a complete and honest picture that the omnibus p-value by itself cannot.