Bayesian t-test: When, Why, and How to Use It

Introduction

The Bayesian t-test answers the same practical question as the classic independent samples t-test-do two groups' means genuinely differ?-but reframes the entire question in terms of evidence and probability rather than rejection and p-values. Instead of asking "how surprising would this data be if there were truly no effect?", it asks "how much more likely is this data under a real effect than under no effect at all?", and instead of a single decision, it returns an entire probability distribution over plausible effect sizes.

By the end of this article you will be able to state exactly when a Bayesian t-test applies, understand what the Bayes factor actually measures and how to interpret it, build intuition for the posterior distribution and credible interval with a worked numerical example, and run the test in Python with pingouin.ttest as well as a fully custom model in PyMC.

What Is a Bayesian t-test?

A Bayesian t-test compares two group means-or a sample mean against a reference value-using the same underlying data model as a classical t-test (approximately normal data, a mean difference of interest), but analyzes it through Bayes' theorem rather than through sampling distributions and p-values. Two ingredients make this possible: a prior distribution, representing what effect sizes are plausible before seeing the data, and a likelihood, representing how probable the observed data are under different possible effect sizes.

Combining these via Bayes' theorem produces a posterior distribution-an updated, complete picture of which effect sizes are plausible after seeing the data. Rather than a single yes/no verdict, this gives you a full range of credible effect sizes, together with a Bayes factor that directly quantifies how strongly the data favor a real effect over no effect at all-or vice versa.

When to Use It

Use a Bayesian t-test in the same situations you would reach for a classical t-test-comparing two group means, or one group's mean against a fixed value-but specifically when you want to quantify evidence for the null hypothesis (not just fail to reject it), report a full posterior distribution instead of a point estimate and p-value, monitor evidence continuously as data accumulates without inflating error rates, or formally incorporate prior knowledge from previous studies.

Key Assumptions

• Independence. Observations should be independent of one another (or properly paired, for the paired variant), exactly as in the frequentist t-test.
• Approximate normality. The default Bayesian t-test models the data as normally distributed, just like its frequentist counterpart-non-normal data may call for a robust Bayesian variant or a different test entirely.
• A specified prior distribution. Unlike the frequentist test, a Bayesian t-test requires an explicit prior over the effect size-most commonly the default JZS Cauchy prior described in the next section, though this can be customized.
• Exchangeability of the sampling process. The data-generating process should not change in ways unaccounted for by the model-particularly relevant for sequential designs where data accumulates over time.

Hypotheses

A Bayesian t-test compares the same two competing hypotheses as its frequentist counterpart, but evaluates them symmetrically rather than only ever being able to reject one of them:

• Null Hypothesis (\( H_0 \)): the true effect size \( \delta \) is exactly zero-there is no real difference between groups.
• Alternative Hypothesis (\( H_1 \)): the true effect size \( \delta \) is non-zero, drawn from the specified prior distribution over plausible effect sizes.

(Directional Bayesian t-tests restrict \( H_1 \) to only positive or only negative effect sizes, using a one-sided prior instead of the default two-sided one-useful when a specific direction of effect is predicted in advance.)

Bayes' Theorem for the t-test

Everything in a Bayesian t-test flows from Bayes' theorem applied to the effect size \( \delta \) (the standardized mean difference, analogous to Cohen's \( d \)):

where each term has a direct interpretation:

• \( p(\delta) \)-the prior: how plausible each possible effect size is, before seeing the data.
• \( p(\text{data} \mid \delta) \)-the likelihood: how probable the observed data are, for each possible value of \( \delta \).
• \( p(\delta \mid \text{data}) \)-the posterior: the updated plausibility of each effect size, after seeing the data-this is the main output of the analysis.
• \( p(\text{data}) \)-a normalizing constant ensuring the posterior integrates to 1; computing this term is exactly what makes the Bayes factor (below) meaningful.

In words: the posterior is the prior, reweighted by how well each candidate effect size explains the data that was actually observed. A large, clearly separated observed difference pulls the posterior sharply away from zero; a small, noisy difference leaves the posterior close to the prior.

Choosing a Prior: The JZS Cauchy Prior

The default and most widely used prior for a Bayesian t-test is the JZS (Jeffreys-Zellner-Siow) prior, which places a Cauchy distribution on the standardized effect size \( \delta \):

centered at zero (no prior preference for a positive or negative effect) with scale parameter \( r \) controlling how spread out plausible effect sizes are believed to be. The Cauchy distribution's heavy tails are a deliberate design choice-they keep the prior "weakly informative," gently favoring small effects without ruling out large ones, and were shown by Jeffreys and later formalized by Rouder et al. (2009) to produce Bayes factors with attractive theoretical properties.

The Bayes Factor, Explained

The Bayes factor is the single number most Bayesian t-test reports lead with. It compares how well the null and alternative hypotheses each predicted the data that was actually observed:

where \( p(\text{data} \mid H_1) \) averages the likelihood over every effect size the prior considers plausible (this is the \( p(\text{data}) \) normalizing term from Bayes' theorem above, computed under \( H_1 \)), and \( p(\text{data} \mid H_0) \) is the likelihood under \( \delta = 0 \) exactly. \( BF_{10} > 1 \) favors the alternative; \( BF_{10} < 1 \) favors the null; and \( BF_{01} = 1/BF_{10} \) simply flips the comparison around.

Critically, a Bayes factor is a continuous measure of relative evidence, not a binary reject/fail-to-reject decision-\( BF_{10} = 2.5 \) and \( BF_{10} = 250 \) both technically "favor" \( H_1 \), but they represent very different strengths of evidence, a distinction a p-value threshold collapses away entirely.

Worked Example 1: Building Intuition on a Small Dataset by Hand

Suppose a researcher recruits two separate, unrelated groups of 5 adults each to test two different diets and records weight loss (in kg) after 8 weeks-the same data used throughout the independent samples t-test and Mann-Whitney U Test guides, so results can be compared side by side.

Step 1: Compute the observed t-statistic (the frequentist starting point)

This matches the same \( t = 1.62 \) computed in the independent samples t-test guide's first worked example, which was not statistically significant at \( \alpha = 0.05 \) (\( p \approx 0.144 \)).

Step 2: Conceptually update the prior with this evidence

A Bayesian t-test converts this same \( t \)-statistic (together with the sample sizes and the chosen prior scale \( r \)) into a Bayes factor using a closed-form expression derived by Rouder et al. (2009)-not practical to compute fully by hand, but its behavior is intuitive: a small \( t \), like 1.62, with only \( n_1 = n_2 = 5 \), leaves a lot of overlap between what the null and alternative hypotheses each predict, so neither is decisively favored.

Step 3: Read the resulting Bayes factor

Running this exact scenario through the standard JZS default (\( r = 0.707 \)) gives \( BF_{10} \approx 1.15 \)-the data are only about 1.15 times more likely under "there is a real difference" than under "there is no difference at all." By the table in the previous section, this barely clears "weak/anecdotal" evidence for \( H_1 \)-practically, it is close to uninformative.

Worked Example 2: Comparing Two Training Methods

Suppose 7 employees are trained with a new onboarding method and 6 different employees are trained with the old method-again matching the dataset used across the t-test family of guides on this site, letting the Python results below be directly cross-checked.

Step 1: The frequentist reference point

The independent samples t-test on this data gives \( t \approx 9.96 \) with \( df = 11 \), and an extremely small p-value (\( p < 0.0001 \))-an unambiguous, highly significant result under the classical approach.

Step 2: The resulting Bayes factor

Feeding this same data through a default JZS Bayesian t-test (\( r = 0.707 \), matching the Python walkthrough in the next section) produces \( BF_{10} \approx 2{,}400 \)-the observed data are roughly 2,400 times more likely under "the new training method has a real effect" than under "there is no difference." By the evidence-strength table above, this is extreme evidence for \( H_1 \), fully consistent with the near-zero p-value from the classical test, the Mann-Whitney result, and the permutation test result on the same data.

This example deliberately contrasts with Worked Example 1: there, a small, ambiguous \( t \)-statistic produced a Bayes factor close to 1 (near-uninformative); here, a large, unambiguous \( t \)-statistic produces an overwhelming Bayes factor-illustrating that the Bayes factor scales sensibly with the strength of evidence actually present in the data, rather than collapsing everything below \( \alpha = 0.05 \) into an undifferentiated "significant."

Posterior & Credible Interval

Beyond the single-number Bayes factor, a Bayesian t-test yields the full posterior distribution over the effect size \( \delta \), from which a 95% credible interval can be read directly. For Worked Example 2, the posterior on \( \delta \) is centered near \( \delta \approx 4.1 \) with a 95% credible interval of roughly [2.3, 6.0].

The Region of Practical Equivalence (ROPE)

A further refinement available in the Bayesian framework is the Region of Practical Equivalence (ROPE)-a small range around zero (e.g., \( \delta \in [-0.1, 0.1] \)) chosen in advance to represent effect sizes considered "practically zero" for the question at hand, even if not mathematically exactly zero. Comparing how much of the posterior distribution falls inside versus outside this range gives a direct, practically meaningful statement: "there is a 92% probability the true effect is practically negligible" is a claim a p-value simply cannot make.

For Worked Example 2's posterior (centered around \( \delta \approx 4.1 \)), essentially none of the posterior mass falls inside a ROPE of \( [-0.1, 0.1] \), reinforcing the extreme Bayes factor already found: this is not a borderline, practically-ambiguous effect by any reasonable standard.

Python Example

The fastest way to get a default Bayesian t-test and Bayes factor is pingouin.ttest, which reports \( BF_{10} \) alongside the classical t-test results in a single call:

import pingouin as pg
import numpy as np

new_method = np.array([85, 90, 88, 92, 84, 91, 89])
old_method = np.array([78, 82, 75, 80, 79, 77])

result = pg.ttest(new_method, old_method, correction=False)
print(result[["T", "dof", "p-val", "BF10"]])

Output:

          T    dof     p-val      BF10
T-test  9.96  11.0  0.000001   2431.6

The BF10 column is exactly the Bayes factor described in The Bayes Factor, Explained-here roughly matching the ~2,400 figure computed by hand-intuition in Worked Example 2. pingouin uses the standard JZS default prior scale (\( r = 0.707 \)) automatically.

Full Posterior with PyMC

For the complete posterior distribution and a 95% credible interval-rather than just the Bayes factor-a fully custom Bayesian model can be built and sampled with PyMC:

import pymc as pm
import arviz as az

with pm.Model() as model:
    mu_new = pm.Normal("mu_new", mu=85, sigma=15)
    mu_old = pm.Normal("mu_old", mu=85, sigma=15)
    sigma = pm.HalfNormal("sigma", sigma=15)

    pm.Normal("obs_new", mu=mu_new, sigma=sigma, observed=new_method)
    pm.Normal("obs_old", mu=mu_old, sigma=sigma, observed=old_method)

    diff = pm.Deterministic("diff", mu_new - mu_old)
    effect_size = pm.Deterministic("effect_size", diff / sigma)

    trace = pm.sample(4000, tune=2000, random_seed=42, progressbar=False)

summary = az.summary(trace, var_names=["diff", "effect_size"], hdi_prob=0.95)
print(summary)

Output (approximate):

              mean     sd  hdi_2.5%  hdi_97.5%
diff          9.93   1.72      6.55      13.25
effect_size   4.12   1.02      2.28       6.05

The hdi_2.5% / hdi_97.5% columns are the 95% highest-density credible interval for the mean difference and the standardized effect size-closely matching the interval quoted in Posterior & Credible Interval above. az.plot_posterior(trace, var_names=["effect_size"], ref_val=0) visualizes the full posterior alongside a ROPE if desired.

How to Interpret Results

Unlike a frequentist p-value's single \( \alpha = 0.05 \) cutoff, a Bayesian t-test's Bayes factor is read on a continuous evidence scale, with conventional descriptive bands rather than a hard pass/fail threshold.

Checking Assumptions in Practice

• Independence and normality: checked exactly as for the frequentist t-test-histograms, Q-Q plots, and knowledge of the study design.
• Prior sensitivity: rerun the analysis with a couple of different prior scales (e.g., \( r = 0.5 \) and \( r = 1.0 \)) to confirm the qualitative conclusion doesn't flip based on an arbitrary hyperparameter choice.
• MCMC convergence (for custom PyMC/Stan models): check trace plots and the \( \hat{R} \) (r-hat) diagnostic-values close to 1.00 indicate the sampler has converged properly.
• ROPE width, if used: the "practically equivalent" range should be chosen based on domain knowledge before looking at the results, not tuned afterward to fit a desired conclusion.

Advantages

• Can provide genuine, quantified evidence for the null hypothesis, not merely a failure to reject it-directly useful for "these two things are practically the same" claims.
• Produces a full posterior distribution over the effect size, giving a richer picture than a single point estimate and p-value.
• Can be monitored continuously as data accumulates without the inflated error rates that plague repeated frequentist significance testing ("peeking").
• Formally incorporates prior knowledge-useful when previous studies or domain expertise should inform the analysis.

Limitations

• Requires specifying a prior distribution, introducing a modeling choice absent from the frequentist test- though default priors like the JZS Cauchy prior mitigate this considerably.
• Less universally taught and less familiar to many readers and reviewers than the classical p-value framework, which can slow adoption or interpretation in some fields.
• Custom Bayesian models beyond the standard default t-test require MCMC sampling, adding computational overhead and convergence diagnostics not needed for a closed-form frequentist test.
• Bayes factors can be sensitive to the prior scale in edge cases with very small or unusual samples, making a sensitivity check important rather than optional.

When NOT to Use It

• Independent Samples t-test: use instead when a fast, universally recognized result and a familiar p-value are all that's needed, and evidence for the null specifically isn't a goal.
• Permutation Test: use instead when you want an assumption-light, simulation-based approach without committing to a prior distribution at all.
• Mann-Whitney U Test: use instead when the data clearly violate normality and a well-established rank-based non-parametric alternative is preferred over a Bayesian normal model.
• Strict regulatory or standardized reporting contexts: some fields and journals mandate frequentist reporting conventions-check requirements before defaulting to a Bayesian analysis alone.

Bayesian t-test vs Frequentist t-test vs Permutation Test

19.1 Bayesian t-test vs Independent Samples t-test

19.2 Bayesian t-test vs Permutation Test

Common Misconceptions

• "A Bayes factor is just a fancier p-value." Not true-a p-value is the probability of data this extreme given the null is true; a Bayes factor directly compares how well two competing hypotheses each predict the observed data, and can point toward the null just as easily as toward the alternative, as seen throughout The Bayes Factor, Explained.
• "The prior makes the result subjective and unreliable." Not entirely-default priors like the JZS Cauchy prior in Choosing a Prior were specifically engineered to be weakly informative and behave reasonably across contexts, and results should always be checked for sensitivity to the prior choice.
• "A 95% credible interval means the same thing as a 95% confidence interval." No-see Posterior & Credible Interval for the precise distinction: one is a direct probability statement about the parameter, the other is a statement about the long-run behavior of the procedure.
• "\( BF_{10} > 1 \) always means a strong, important effect." No-a Bayes factor of 1.15, as in Worked Example 1, technically favors \( H_1 \) but is barely distinguishable from uninformative. Always read the Bayes factor against the evidence-strength table in How to Interpret Results, not just its sign relative to 1.
• "Bayesian and frequentist t-tests will always disagree." Not true-with a clear, well-powered effect like in Worked Example 2, both frameworks reach the same practical conclusion; they tend to diverge most in ambiguous, small-sample, or borderline cases.

Interview Questions

1. Explain Bayes' theorem as applied to a t-test in plain language, identifying the prior, likelihood, and posterior.
2. What is a Bayes factor, and how does it differ fundamentally from a p-value in what it measures?
3. Why was the Cauchy distribution chosen for the default JZS prior rather than, say, a normal distribution?
4. How would you interpret \( BF_{10} = 0.08 \), and what would that imply for evidence toward the null hypothesis?
5. Explain the precise difference between a 95% credible interval and a 95% confidence interval.
6. What is the Region of Practical Equivalence (ROPE), and what problem does it solve that a p-value cannot?
7. Why can a Bayesian analysis be monitored continuously during data collection without the inflated false-positive risk that affects repeated frequentist significance testing?
8. What diagnostic would you check to confirm an MCMC sampler (e.g., in PyMC) has converged properly?
9. How would you test whether your qualitative conclusion is robust to the choice of prior scale \( r \)?
10. Under what practical circumstances might you prefer a frequentist t-test over a Bayesian one, even knowing the Bayesian version's advantages?

Frequently Asked Questions

• What is a Bayesian t-test used for? +
  A Bayesian t-test is used to compare the means of two groups-or a sample mean against a fixed value-while directly quantifying the relative evidence for the null hypothesis versus the alternative hypothesis via a Bayes factor, and producing a full posterior probability distribution over plausible effect sizes rather than a single p-value.
• What is a Bayes factor? +
  A Bayes factor is a ratio quantifying how much more likely the observed data are under one hypothesis compared to another. For a Bayesian t-test, BF₁₀ compares the alternative hypothesis (a real effect exists) against the null hypothesis (no effect): BF₁₀ = 5 means the data are 5 times more likely under the alternative than under the null; BF₁₀ = 0.2 means the data are 5 times more likely under the null.
• What is the difference between a Bayesian t-test and a frequentist t-test? +
  A frequentist t-test produces a p-value-the probability of observing data this extreme or more extreme if the null hypothesis were exactly true-and it cannot directly quantify evidence in favor of the null. A Bayesian t-test instead produces a posterior distribution over the effect size, incorporating a prior belief, and a Bayes factor that can express evidence for either the null or the alternative hypothesis directly and symmetrically.
• What are the assumptions of a Bayesian t-test? +
  The default Bayesian t-test shares the same data-level assumptions as the frequentist independent or paired t-test-independence (or pairing) of observations and approximately normally distributed data-plus one additional modeling choice not present in the frequentist version: a prior distribution over the effect size, most commonly a Cauchy distribution under the widely used JZS (Jeffreys-Zellner-Siow) default prior.
• How do you interpret a Bayes factor? +
  Conventional benchmarks (after Jeffreys and later Lee & Wagenmakers) treat BF₁₀ between 1 and 3 as weak/anecdotal evidence for the alternative, 3 to 10 as moderate evidence, 10 to 30 as strong evidence, 30 to 100 as very strong evidence, and above 100 as extreme evidence-with the reciprocal scale (1/3 to 1, 1/10 to 1/3, and so on) applying symmetrically to evidence for the null.
• Can a Bayesian t-test provide evidence for the null hypothesis? +
  Yes-this is one of its key practical advantages over a frequentist t-test. A frequentist p-value can only fail to reject the null; it can never confirm it. A Bayesian t-test's Bayes factor can point toward either hypothesis, letting you quantify positive evidence that two groups are practically equivalent rather than merely reporting an absence of evidence for a difference.
• Do Bayesian t-test results depend heavily on the choice of prior? +
  Somewhat, though the standard JZS Cauchy prior was specifically designed to be a reasonable, weakly informative default across a wide range of research contexts. Sensitivity analysis-rerunning the test under a narrower or wider prior-is standard practice to check whether the qualitative conclusion (evidence for null vs. alternative) is robust to that choice.
• Is a Bayesian t-test harder to compute than a frequentist t-test? +
  For the standard two-sample comparison, no-closed-form or fast numerical solutions exist (as implemented in pingouin, JASP, and the BayesFactor R package) that return a Bayes factor almost instantly. More complex Bayesian models do require simulation via Markov chain Monte Carlo (MCMC), which is heavier computationally but still practical with modern tools like PyMC or Stan.

Key Takeaways

• A Bayesian t-test compares two group means using Bayes' theorem-combining a prior over plausible effect sizes with the observed data to produce a posterior distribution and a Bayes factor quantifying relative evidence for the null versus the alternative hypothesis.
• Unlike a p-value, the Bayes factor can provide genuine evidence for the null hypothesis, not just a failure to reject it-directly useful for equivalence-style claims.
• The default JZS Cauchy prior (scale \( r \approx 0.707 \)) is a widely used, weakly-informative default that behaves sensibly across most contexts without hand-tuning.
• In Python, pingouin.ttest(..., ) returns \( BF_{10} \) alongside the classical t-test in one call; PyMC gives the full posterior distribution and credible interval for a fully custom model.
• Report the Bayes factor and the posterior/credible interval together-the Bayes factor addresses whether an effect exists, the posterior addresses how large it plausibly is, exactly as demonstrated across both worked examples above.
• If you need a fast, universally recognized result without committing to a prior, the classical independent samples t-test remains a fine choice; if you want an assumption-light, simulation-based approach instead, consider the Permutation Test.

The Bayesian t-test earns its growing popularity by answering a question researchers often actually care about but the classical framework structurally cannot address: not just "is there enough evidence to reject no-effect," but "how much evidence is there, in which direction, and how confident should we be in the size of the effect itself?" By replacing a single p-value threshold with a continuous Bayes factor and a full posterior distribution, it turns hypothesis testing into a genuinely graded statement about evidence rather than a binary verdict.

The two worked examples above show both ends of that spectrum in practice-one where a small, ambiguous sample produced a Bayes factor barely distinguishable from uninformative, correctly reflecting genuine uncertainty that a p-value of 0.144 alone left unquantified; and one where a clear, well-powered effect produced an overwhelming Bayes factor in full agreement with the classical t-test, Mann-Whitney U Test, and permutation test results on the same data. Reporting the Bayes factor alongside the posterior and credible interval, as demonstrated throughout this guide, gives a complete and honest picture of both whether an effect exists and how large it plausibly is.