Kolmogorov-Smirnov Test

Introduction

The Kolmogorov-Smirnov Test (often just called the "KS test") checks whether a sample of data comes from a specific theoretical distribution, or whether two independent samples come from the same underlying distribution. Unlike tests built for a single named distribution, it works by comparing cumulative distribution functions directly, which makes it flexible enough to test against any continuous distribution-not just normal.

By the end of this article you will be able to state exactly when a one-sample or two-sample KS test applies, compute the D statistic completely by hand on two contrasting worked examples, understand why estimating distribution parameters from the same sample you are testing invalidates the standard p-value, know when to reach for the Shapiro-Wilk Test instead, and run both versions of the test in Python with scipy.stats.

What Is the Kolmogorov-Smirnov Test?

Developed by Andrey Kolmogorov in 1933 and extended to the two-sample case by Nikolai Smirnov in 1939, the KS test is one of the oldest and most general non-parametric tests for comparing distributions. Its central idea is the empirical distribution function (ECDF): for a sorted sample, the ECDF is a step function that jumps by \( 1/n \) at each observed value, tracing out the sample's cumulative proportion at every point.

The test statistic, \( D \), is simply the largest vertical distance between two such curves-either the sample's ECDF against a specified theoretical CDF (one-sample version), or two samples' ECDFs against each other (two-sample version). A large \( D \) means the curves diverge substantially somewhere along the way; a small \( D \) means they track each other closely across their entire range.

When to Use It

Use the KS test whenever you need to check whether continuous data matches a specific theoretical distribution, or whether two independent samples share the same underlying distribution-and you do not want to restrict the comparison to normality alone, in which case the Shapiro-Wilk Test may be a more powerful, narrower tool.

The key requirement: continuous data, and either a fully specified theoretical distribution (one-sample) or a second independent sample to compare against (two-sample)-if the theoretical distribution parameters must be estimated from the same sample, see Common Pitfalls before trusting the standard p-value.

Key Assumptions

• Continuous data. The test assumes a continuous underlying distribution; heavily tied or discrete data can distort the comparison of cumulative distribution functions.
• Independent observations. Both the one-sample and two-sample versions assume observations are independent of one another.
• Fully specified theoretical distribution (one-sample version). The distribution and its parameters must be stated in advance, not estimated from the same sample being tested-see Common Pitfalls for what happens when this is violated.
• Independent samples (two-sample version). The two samples being compared must be drawn independently of each other, not paired or matched observations.
• Reasonably sized samples. Very small samples limit the test's power to detect real differences, while very large samples can make trivial differences statistically significant.

Hypotheses

One-sample version:

• Null Hypothesis (\( H_0 \)): the sample was drawn from the specified theoretical distribution.
• Alternative Hypothesis (\( H_1 \)): the sample was not drawn from that distribution.

Two-sample version:

• Null Hypothesis (\( H_0 \)): both samples were drawn from the same underlying distribution.
• Alternative Hypothesis (\( H_1 \)): the samples were drawn from different underlying distributions.

(As with most goodness-of-fit tests, failing to reject \( H_0 \) never proves a match-it only means the test did not find enough evidence, in this sample, to rule one out.)

The Formula, Explained

One-sample statistic. Let \( F_n(x) \) be the empirical distribution function of a sample of size \( n \), and \( F(x) \) the specified theoretical CDF. The KS statistic is:

Because \( F_n(x) \) is a step function that jumps at each sorted observation \( x_{(i)} \), this supremum can only occur at those jump points. For each sorted value \( x_{(i)} \) (\( i = 1, \ldots, n \)), the relevant comparisons are:

Two-sample statistic. With two samples of sizes \( n_1 \) and \( n_2 \) and empirical distribution functions \( F_{n_1}(x) \) and \( F_{n_2}(x) \):

computed by evaluating both ECDFs at every distinct value across the combined samples and taking the largest absolute difference. In both versions, larger \( D \) values are less likely under \( H_0 \), and software computes the associated p-value from the known sampling distribution of \( D \) (asymptotic for large samples, exact for small samples).

Worked Example 1: One-Sample Test Against a Normal

Suppose 8 measurements are collected and we want to check whether they plausibly came from a \( N(5.3, 0.5^2) \) distribution, specified in advance:

With \( n = 8 \), compute \( F(x_{(i)}) \) from the specified normal, along with \( (i-1)/n \) and \( i/n \):

The largest value across every entry in the last two columns is \( 0.1587 \), occurring at \( i = 1 \) (comparing \( F(4.8) = 0.1587 \) against the left-hand step value of \( 0 \)):

For \( n = 8 \), a \( D \) of \( 0.1587 \) is far below the critical value needed for significance at \( \alpha = 0.05 \), giving \( p \approx 0.969 \). We fail to reject \( H_0 \)-the sample is entirely consistent with having been drawn from the specified \( N(5.3, 0.5^2) \) distribution.

Worked Example 2: Two-Sample Comparison

Now suppose two independent samples of size 6 are collected, and we want to check whether they come from the same underlying distribution:

Combining and sorting all distinct values across both samples, then evaluating each sample's ECDF at every one of those values:

The largest gap is \( 0.833 \), occurring at both \( x = 9 \) and \( x = 11 \):

For two samples of size 6 each, \( D \approx 0.8333 \) gives \( p \approx 0.026 \)-below the conventional \( 0.05 \) threshold. We reject \( H_0 \): the two samples show a statistically significant difference in their underlying distributions, consistent with how the data was constructed (Sample A occupies a distinctly higher range than Sample B, with only a small overlap).

Python Examples

One-sample test against a specified normal distribution, using scipy.stats.kstest:

from scipy import stats

data = [4.8, 4.9, 5.1, 5.2, 5.3, 5.6, 5.7, 6.0]

result = stats.kstest(data, 'norm', args=(5.3, 0.5))

print(f"D statistic: {result.statistic:.4f}")
print(f"p-value: {result.pvalue:.4f}")

Output:

D statistic: 0.1587
p-value: 0.9688

This matches Worked Example 1 exactly.

Two-sample test, using scipy.stats.ks_2samp:

from scipy import stats

sample_A = [10, 12, 14, 16, 18, 20]
sample_B = [1, 3, 5, 7, 9, 11]

result = stats.ks_2samp(sample_A, sample_B)

print(f"D statistic: {result.statistic:.4f}")
print(f"p-value: {result.pvalue:.4f}")

Output:

D statistic: 0.8333
p-value: 0.0260

This matches Worked Example 2 exactly.

How to Interpret Results

Always pair the p-value with a plot of the empirical distribution functions-the KS test tells you whether the curves differ, not where along the range the biggest discrepancy occurs, and that location often matters for understanding what is actually driving the result.

Visualizing the ECDF Comparison

Because the KS test is built entirely from cumulative distribution functions, plotting them is the most direct way to see what is driving a given \( D \) value. Step functions for each sample (or a sample versus a theoretical CDF) make the largest gap visually obvious:

import numpy as np
import matplotlib.pyplot as plt

sample_A = [10, 12, 14, 16, 18, 20]
sample_B = [1, 3, 5, 7, 9, 11]

for sample, label in [(sample_A, "Sample A"), (sample_B, "Sample B")]:
    x = np.sort(sample)
    y = np.arange(1, len(x) + 1) / len(x)
    plt.step(x, y, where="post", label=label)

plt.xlabel("Value")
plt.ylabel("Cumulative Probability")
plt.legend()
plt.title("ECDF Comparison")
plt.show()

Whenever the KS test comes back significant, a quick ECDF plot next to the p-value usually makes the result far more interpretable to a non-technical audience than the D statistic alone.

One-Sample vs Two-Sample: Choosing the Right Version

Common Pitfalls and How to Avoid Them

• Estimating distribution parameters from the same sample being tested. If you use the sample mean and standard deviation as the normal distribution's parameters and then run a standard one-sample KS test against that fitted normal, the resulting p-value is invalid-the test becomes conservative and understates significance. Use a Lilliefors-corrected test in that situation instead.
• Treating a non-significant result as proof of a match. A large p-value means insufficient evidence of a difference was found, not that the distributions are provably identical.
• Applying the test to discrete or heavily tied data. The KS test assumes continuity; large numbers of repeated values can distort the comparison of cumulative distribution functions.
• Over-relying on the p-value at very large sample sizes. As with most goodness-of-fit tests, very large samples can make trivial, practically unimportant differences statistically significant.
• Forgetting the test says nothing about the type or location of the discrepancy. A significant \( D \) does not say whether the difference is in location, spread, or shape-inspect an ECDF plot to see where the gap occurs.

Advantages

• Fully non-parametric and distribution-free-works with any continuous theoretical distribution, not just normal.
• Naturally extends to comparing two samples directly, without assuming either follows a named distribution.
• Based on the full cumulative distribution function, so it is sensitive to differences in location, spread, and shape all at once.
• Simple to compute, widely implemented, and pairs naturally with an ECDF plot for intuitive interpretation.
• Exact p-values are available for small samples, not just asymptotic approximations.

Limitations

• Invalid p-value if parameters are estimated from the same sample. The standard one-sample test requires a fully specified distribution in advance-see Common Pitfalls.
• Less powerful than specialized tests for a specific distribution. When testing specifically for normality, the Shapiro-Wilk Test is generally more powerful.
• More sensitive near the center of the distribution than in the tails. The D statistic can under-detect differences that are concentrated in the extreme tails; the Anderson-Darling test gives more weight to tail discrepancies.
• Requires continuous data. Discrete or heavily rounded data can distort results.
• No quantified effect size beyond \( D \) itself. \( D \) has a clear geometric meaning but no direct real-world interpretation the way a mean difference or hazard ratio does.

When NOT to Use It

• When testing specifically for normality and power matters-use the Shapiro-Wilk Test instead.
• When distribution parameters had to be estimated from the same sample being tested, without applying a Lilliefors correction.
• When the data is discrete, heavily rounded, or contains many tied values.
• When the primary interest is tail behavior specifically-the Anderson-Darling test is more sensitive there.
• When more than two groups need to be compared simultaneously in one test.

KS Test vs Shapiro-Wilk vs Anderson-Darling vs Chi-Square

Common Misconceptions

• "The KS test can be used to check normality after fitting a normal to the same data." Only with a correction-fitting the distribution's parameters from the sample and then running the standard test invalidates the p-value; see Common Pitfalls.
• "A non-significant KS test proves the distributions match." It only means no significant difference was detected at the given sample size, not that the distributions are provably identical.
• "The KS test is always more powerful than a distribution-specific test." Not true-for normality specifically, the Shapiro-Wilk Test is generally more powerful, especially in small samples.
• "D tells you how different the distributions are, like an effect size." D is a geometric quantity (the largest CDF gap) rather than an interpretable effect size in the original units of the data.
• "The two-sample KS test requires equal sample sizes." It does not-the two samples can have different sizes, and the formula and software both handle this directly.

Interview Questions

1. Explain in your own words what the Kolmogorov-Smirnov statistic measures, and why it is based on cumulative distribution functions rather than densities or histograms.
2. Walk through how the one-sample D statistic is computed from a sorted sample and a specified theoretical CDF.
3. Why does the standard one-sample KS test become invalid when distribution parameters are estimated from the same sample being tested, and what correction addresses this?
4. How does the two-sample KS test differ from the one-sample version, and what question does each one answer?
5. When would you prefer the Kolmogorov-Smirnov Test over the Shapiro-Wilk Test, and vice versa?
6. Why is the KS test generally less sensitive to differences concentrated in the tails of a distribution, and what alternative addresses that weakness?
7. What does a non-significant two-sample KS test result actually tell you about two groups?
8. How would you visually communicate a significant KS test result to a non-technical stakeholder?
9. Can the KS test be used to compare more than two groups at once? Why or why not, and what would you do instead?
10. Explain why the KS test requires continuous data and what can go wrong if it is applied to data with many tied values.

Frequently Asked Questions

• What is the Kolmogorov-Smirnov Test used for? +
  The Kolmogorov-Smirnov (KS) Test is a non-parametric test with two main uses: checking whether a sample plausibly comes from a specific, fully specified theoretical distribution (the one-sample version), and checking whether two independent samples plausibly come from the same underlying distribution (the two-sample version). Because it compares cumulative distribution functions directly, it works with any continuous distribution, not only the normal distribution.
• What is the formula for the Kolmogorov-Smirnov Test statistic? +
  The D statistic is defined as the largest absolute vertical gap between two cumulative distribution functions, evaluated across every possible value. In the one-sample version, this compares the sample empirical distribution function against the specified theoretical CDF; in the two-sample version, it compares the empirical distribution functions of the two samples against each other. Larger gaps indicate a bigger discrepancy, and the sampling distribution of D under the null hypothesis is used to compute a p-value.
• How do you interpret the Kolmogorov-Smirnov Test result? +
  A small p-value, conventionally below 0.05, indicates the sample significantly departs from the specified distribution (one-sample case) or that the two samples significantly differ (two-sample case). A large p-value means the test did not find sufficient evidence of a difference; this does not prove the distributions match exactly, only that no statistically significant departure was detected given the available data.
• What is the difference between the Kolmogorov-Smirnov Test and the Shapiro-Wilk Test? +
  The Shapiro-Wilk Test is purpose-built to detect departures from normality specifically, and it is generally the more powerful choice for that narrow job, particularly with small samples. The Kolmogorov-Smirnov Test is a more general-purpose goodness-of-fit test: its one-sample version can compare a sample against any fully specified continuous distribution, not just normal, and its two-sample version can compare two samples directly against each other without assuming either one follows a particular distribution at all.
• What assumptions does the Kolmogorov-Smirnov Test make? +
  The test assumes the data is continuous and the observations are independent. A critical requirement for the one-sample version is that the theoretical distribution and its parameters must be fully specified in advance, not estimated from the same sample being tested-estimating parameters from the data (for example, using the sample mean and standard deviation as the normal distribution's parameters) changes the sampling distribution of D and makes the standard p-value invalid, a pitfall usually addressed with a Lilliefors correction.
• Can the Kolmogorov-Smirnov Test compare two samples directly? +
  Yes. The two-sample Kolmogorov-Smirnov Test compares the empirical distribution functions of two independent samples directly and tests the null hypothesis that they were drawn from the same underlying distribution, without requiring either sample to match a named theoretical distribution such as normal. This makes it useful for comparing, for example, two experimental groups whose distributions are not assumed to be normal.

Key Takeaways

• The Kolmogorov-Smirnov Test compares cumulative distribution functions-either a sample against a specified theoretical distribution (one-sample), or two samples against each other (two-sample)-and reports the largest vertical gap, \( D \), between the curves.
• The statistic \( D = \sup_x |F_n(x) - F(x)| \) (one-sample) or \( D = \sup_x |F_{n_1}(x) - F_{n_2}(x)| \) (two-sample) is bounded between 0 and 1, with smaller values indicating closer agreement.
• It is fully non-parametric and works with any continuous distribution, but the one-sample version requires the theoretical distribution to be fully specified in advance-not fitted from the same sample-or a Lilliefors correction is needed.
• A significant result only says the curves differ-it does not say where along the range the biggest discrepancy sits; pair the test with an ECDF plot.
• In Python, scipy.stats.kstest handles the one-sample version and scipy.stats.ks_2samp handles the two-sample version.
• For normality testing specifically, the Shapiro-Wilk Test is generally more powerful; use the KS test when you need a more general-purpose or two-sample comparison.
• A non-significant result never proves the distributions match-it only means no significant departure was detected given the sample size and data at hand.

The Kolmogorov-Smirnov Test earns its place as one of the most flexible tools in the goodness-of-fit toolbox because it asks a very general question-do these cumulative distribution curves genuinely differ-without committing to any particular distributional shape. That generality is also its main trade-off: for a specific, narrow question like normality, a purpose-built test such as Shapiro-Wilk will usually detect a real departure with fewer observations.

The two worked examples above show how directly the test reads from the data-comparing a sample's stepped cumulative curve against a theoretical one, or against a second sample's own stepped curve, and reporting the single largest gap between them. Watch for the parameter-estimation pitfall in the one-sample version, pair any significant result with an ECDF plot to see where the discrepancy actually lives, and reach for the Shapiro-Wilk Test when normality specifically, rather than a general distributional match, is the real question.