SHAP Value Analysis

Introduction

SHAP value analysis answers a question every machine learning practitioner eventually has to answer to someone else: why did the model predict this? Not in general terms about which features tend to matter across the whole dataset, but for this specific row-this one customer denied a loan, this one patient flagged as high risk, this one house priced at USD 412,000 instead of the average USD 380,000. SHAP (SHapley Additive exPlanations) answers that by borrowing a 70-year-old idea from cooperative game theory-the Shapley value-and repurposing it to fairly split a model's prediction among its input features.

This is not a beginner's overview of "feature importance." By the end of this article you will be able to derive the SHAP formula from first principles, compute exact SHAP values by hand on a small model, explain precisely why KernelSHAP and TreeSHAP exist and how each actually works internally, read every major SHAP plot correctly, know exactly when SHAP is the wrong tool, and run a complete, production-grade shap Python workflow-from a tabular gradient boosted model to interaction values.

What Is SHAP Value Analysis?

SHAP value analysis is a method for explaining individual predictions of any machine learning model by computing, for every input feature, its exact (or approximated) Shapley value: the average amount that feature contributed to moving the prediction away from the model's baseline output. Introduced by Lundberg and Lee in their 2017 NeurIPS paper A Unified Approach to Interpreting Model Predictions, SHAP unifies six prior explanation methods-including LIME, DeepLIFT, and Layer-Wise Relevance Propagation-under a single theoretical framework called additive feature attribution methods, and proves that the Shapley value is the unique solution within that framework satisfying three desirable properties simultaneously.

The core idea treats a prediction as the payout of a cooperative game, where the "players" are the model's input features and the "payout" is the prediction itself. Instead of asking "how important is this feature in general," SHAP asks a much sharper question: "if I reveal this feature's value to the model, holding every possible combination of already-revealed features fixed, how much does the prediction move, on average?" Averaging that marginal contribution across every possible order of revealing features is exactly what a Shapley value computes-and it is the only way to do so that is provably fair.

It exists because earlier explanation methods-raw feature importances from tree splits, ad hoc perturbation-based scores, or LIME's locally weighted surrogate models-either lacked a rigorous theoretical guarantee of fairness and consistency, or produced attributions that could contradict each other depending on implementation details. SHAP fixes this by grounding attribution in a 70-year-old result from game theory that Lloyd Shapley proved has exactly one fair solution.

The Game-Theoretic Foundation

In 1953, Lloyd Shapley posed a deceptively simple question: if a group of players cooperate to produce some total value (a "coalition game"), how should that value be fairly split among them, given that different players may contribute different amounts depending on which other players are already in the coalition? His answer-the Shapley value-is the average marginal contribution of a player across every possible order in which players could join the coalition.

SHAP maps this directly onto machine learning:

• Players become the model's input features.
• The game's payout becomes the model's prediction for one specific instance, relative to the average prediction over a background dataset.
• "Joining the coalition" means a feature's actual value is revealed to the model, instead of being marginalized out (typically by averaging over a background dataset of the other possible values it could have taken).
• A feature's Shapley value is its average marginal effect on the prediction, computed across every possible order in which the other features could have already been revealed.

This reframing is what makes SHAP theoretically different from ordinary permutation-based or ablation-based attribution: it doesn't just remove or add one feature and call it done. It systematically considers every coalition a feature could join, weights each coalition by how many orderings produce it, and averages-exactly mirroring how Shapley's original theorem guarantees a unique, fair split of a game's total payout.

When to Use It

Reach for SHAP value analysis whenever you need a theoretically grounded, individualized explanation for a machine learning model's output, especially when:

• A stakeholder-a regulator, a customer, a clinician, an auditor-needs to know exactly why one specific prediction came out the way it did, not just which features matter on average.
• You need explanations that are consistent: if you change the model so a feature matters more, that feature's attribution should never decrease, a guarantee ordinary feature-importance methods do not provide.
• You want both local (single-prediction) and global (whole-dataset) explanations from the same underlying attribution values, rather than switching between unrelated methods.
• You are working with a tree-based model (random forest, XGBoost, LightGBM, CatBoost) and want exact, fast attributions via TreeSHAP-SHAP's most common and best- supported use case in practice.
• You need to debug a model, detect leakage, audit for unfair reliance on a protected attribute, or communicate model behavior to non-technical stakeholders in regulated domains such as credit, insurance, or healthcare.

The Four Shapley Axioms

SHAP's central theoretical claim is that the Shapley value is the unique additive feature attribution method satisfying four properties simultaneously. This is not a marketing claim-it is a proven result, and it is the single biggest reason SHAP is treated differently from ad hoc attribution heuristics.

• 1. Local accuracy (efficiency): the sum of all feature attributions plus the base value exactly equals the model's output for that instance-nothing is left unexplained, and nothing is over-explained.
• 2. Missingness: a feature that is always "missing" (absent from the model or masked out of the coalition entirely) gets a SHAP value of exactly zero-features that never participate cannot receive credit.
• 3. Consistency (monotonicity): if you change a model so that a feature's marginal contribution increases or stays the same for every possible coalition, that feature's SHAP value can never decrease. This is the property that most attribution heuristics-including plain tree-split "gain"-based importance-fail to guarantee.
• 4. Symmetry: if two features contribute identically to every possible coalition, they must receive identical SHAP values-no arbitrary tie-breaking.

Shapley proved in 1953 that his value is the only solution satisfying efficiency, symmetry, and a "dummy" version of missingness simultaneously (linearity is the fourth classical axiom, folded into consistency's practical implications for SHAP). Lundberg and Lee's contribution was showing that this same uniqueness result applies directly to machine learning model explanations-and that popular prior methods like LIME satisfy only a subset of these properties.

The Formula, Explained

Let \( f \) be the model, \( x \) the instance being explained, and \( M \) the total number of features. For any subset (coalition) \( S \) of features, let \( f(S) \) denote the model's expected prediction when only the features in \( S \) are set to their actual values from \( x \), and every other feature is marginalized out (typically approximated by averaging the model's output over a background dataset with those features replaced). The SHAP value for feature \( i \) is:

where \( F \) is the full set of \( M \) features, and the sum runs over every possible subset \( S \) of the remaining \( M - 1 \) features (excluding \( i \) itself). Breaking this down term by term:

1. \( f(S \cup \{i\}) - f(S) \): feature \( i \)'s marginal contribution-how much the prediction changes when \( i \) joins coalition \( S \).
2. \( \dfrac{|S|!\,(M - |S| - 1)!}{M!} \): the Shapley weight-the fraction of all \( M! \) possible feature orderings in which the features in \( S \) appear before feature \( i \), and every remaining feature appears after it. This is what makes the average a genuine average over orderings, not just over subsets.
3. The full sum: the weighted average of feature \( i \)'s marginal contribution, across every coalition it could join, in every order in which that coalition could have formed.

The local accuracy (additivity) property guarantees:

where \( \phi_0 = f(\emptyset) \) is the base value-the model's expected output with no features revealed, typically the average prediction over the background dataset. Every SHAP value is therefore denominated in the same units as the model's output, and they always sum, exactly, back up to the specific prediction being explained.

Worked Example 1: A 3-Feature Model by Hand

The formula above is compact, but compact formulas can hide a lot of bookkeeping. This section walks through a complete, by-hand computation on a small model, the way an instructor would work it on a whiteboard-showing not just the arithmetic, but the reasoning behind each step.

Consider a toy model that predicts a loan applicant's approval score (0-100) from three binary features: Income (High = 1, Low = 0), CreditHistory (Good = 1, Poor = 0), and HasCoSigner (Yes = 1, No = 0). We are allowed to know the model's exact underlying function here-something you would never have in a real black-box setting-precisely so that every SHAP value can be checked against ground truth:

Notice the \( 5 \cdot \text{Income} \cdot \text{Credit} \) term: Income and Credit interact, so their combined effect is not simply the sum of their separate effects. Keep this in mind-it will matter later when we discuss interaction values.

We want to explain the prediction for one specific applicant: Income = 1, Credit = 1, CoSigner = 0. Plugging directly into the formula gives \( f(1,1,0) = 40 + 20 + 15 + 5 + 0 = 80 \). This number, 80, is the target our SHAP values must eventually add up to.

Step 1: Establish the base value

Before we can measure how much each feature moved the prediction, we need something to measure it from. That reference point is the base value \( \phi_0 \)-the model's average output when nothing is known about the applicant yet. We compute it by averaging \( f \) over all 8 possible combinations of the three binary features, assuming each is independently 50/50 in the background population:

So before we know anything about this particular applicant, the model's best guess is 63.75. Everything that follows is about explaining how the actual known values-Income = 1, Credit = 1, CoSigner = 0-push that 63.75 up to the true prediction of 80.

Step 2: Enumerate the coalitions and their weights

For each feature, the formula requires us to consider every subset \( S \) of the other two features it could join. With two other features, there are \( 2^2 = 4 \) such subsets. Before computing any marginal contributions, it is worth working out the Shapley weight for each subset size, since the same three weights will be reused for every feature in this example. With \( M = 3 \), the possible \( |S| \) values are 0, 1, and 2:

Notice that the empty coalition and the full coalition each get the largest weight, \( 1/3 \), while the two "in-between" coalitions of size 1 each get a smaller weight of \( 1/6 \). This is exactly the correction mentioned earlier: there is only one way to have an empty coalition or a full coalition, but two different ways to have a coalition of size 1 (it could be {Credit} or {CoSigner}), so each of those two gets proportionally less individual weight to keep the total ordering-based average fair.

Step 3: Compute Income's marginal contribution in each coalition

Now we do the actual "what if" experiments for Income. For each of the four coalitions of {Credit, CoSigner}, we compare the model's output with Income unknown (marginalized over 0/1) against Income known to be 1, holding every other revealed feature at this applicant's actual values (Credit = 1, CoSigner = 0):

A pattern is already visible: Income's marginal contribution is larger (12.500) whenever Credit is already known, and smaller (11.250) whenever it isn't. That is precisely the interaction term \( 5 \cdot \text{Income} \cdot \text{Credit} \) showing up in the arithmetic-Income and Credit reinforce each other. Summing the four weighted terms gives Income's final SHAP value:

Step 4: Repeat for Credit and CoSigner

The same four-row procedure-enumerate the coalitions of the other two features, compute the marginal contribution of adding the feature of interest, weight, and sum-applies unchanged to Credit and to CoSigner. Only the roles swap. Carrying out that arithmetic (following exactly the same table structure shown for Income) produces:

Step 5: Verify local accuracy

The whole point of the additivity property is that it gives us a way to check our work without needing an answer key. If our arithmetic is correct, the base value plus all three SHAP values should land exactly on \( f(1,1,0) = 80 \):

It matches exactly, as the additivity property guarantees it must. This is not a coincidence to be pleasantly surprised by-it is a mathematical property of the Shapley value, and any correctly computed set of SHAP values will reproduce it every time.

One detail is worth pausing on: CoSigner's SHAP value is negative, and it is the same \(-5.000\) regardless of which coalition it joins in step 4's computation. Why? This applicant has no co-signer, and the average applicant in the background population has a 50% chance of having one-so the absence pulls this prediction down. The fact that the pull is exactly \(-5\) in every coalition, with no variation, is itself informative: it tells us CoSigner enters the underlying function purely additively, with no interaction term, unlike Income and Credit.

Worked Example 2: A Larger, Realistic Case

Hand-computing exact SHAP values, as above, becomes impractical past 4-5 features because the number of coalitions doubles with every added feature. Consider a gradient boosted regression model predicting house price from five features: sqft, bedrooms, age, distance_to_city, and has_garage. Explaining a single prediction exactly would require evaluating \( 2^4 = 16 \) coalitions per feature-80 total model evaluations for one instance. This is exactly the scale at which practitioners stop computing SHAP values by hand and reach for TreeSHAP (exact, and fast, because the model is tree-based) instead.

For a specific house-2,400 sqft, 4 bedrooms, 8 years old, 3.5 miles from the city center, no garage-a real TreeSHAP run against a trained XGBoost model might return:

With a base value (average predicted price across the training set) of \( \phi_0 = \$364{,}900 \), local accuracy gives a final predicted price of \( 364{,}900 + 34{,}200 + 11{,}800 + 2{,}100 - 3{,}400 - 9{,}600 = \$400{,}000 \) exactly. This is precisely the kind of per-instance breakdown a SHAP force or waterfall plot visualizes directly, and it is the format used in the Python example below.

KernelSHAP: Model-Agnostic Approximation

KernelSHAP is Lundberg and Lee's model-agnostic algorithm for approximating SHAP values for any model-neural networks, SVMs, arbitrary black-box APIs-without needing access to the model's internal structure. It works by:

1. Sampling coalitions: instead of enumerating all \( 2^M \) subsets, it samples a manageable number of coalitions \( S \), preferentially sampling very small and very large coalitions (which the Shapley weighting formula weights most heavily).
2. Masking features: for each sampled coalition, features not in \( S \) are replaced with values drawn from a background dataset, and the model is evaluated on this masked input.
3. Fitting a weighted linear regression: a linear model is fit to predict the masked model's output from the coalition membership vector, using the Shapley kernel as sample weights-a specific weighting function derived so that the regression's coefficients are provably equal to the Shapley values in expectation.

This is the crucial theoretical insight of the original SHAP paper: Shapley values can be recovered exactly as the solution to a specially weighted least-squares regression problem, which means any model-agnostic method can approximate them by sampling instead of exhaustively enumerating. The tradeoff is that KernelSHAP's accuracy depends on the number of coalitions sampled, and its runtime, while far better than exact enumeration, still scales poorly-often impractically-for models with many dozens or hundreds of features.

TreeSHAP: Exact and Fast for Trees

TreeSHAP, introduced in Lundberg et al.'s 2020 Nature Machine Intelligence paper, computes exact SHAP values for tree-based models-decision trees, random forests, gradient boosted trees like XGBoost, LightGBM, and CatBoost-in low-order polynomial time, without any sampling or approximation. It achieves this by exploiting the recursive structure of decision trees directly:

• Rather than marginalizing features by resampling from a background dataset and re-running the full model, TreeSHAP tracks, for every possible subset of features "on a path" through the tree, how predictions would change-computed once, recursively, by walking the tree structure itself.
• This reduces the runtime from the exponential \( O(TL2^M) \) of naive exact enumeration (where \( T \) is the number of trees and \( L \) the maximum number of leaves) down to \( O(TLD^2) \), where \( D \) is the maximum tree depth-a dramatic and exact speed-up, not an approximation.
• Because it is exact, TreeSHAP requires no sampling variance and produces identical results on repeated runs, unlike KernelSHAP, whose sampled estimates vary slightly run to run unless the sample count is very large.

This is why, in practice, TreeSHAP is by far the most commonly used SHAP algorithm: the majority of production tabular-data models are tree ensembles, and TreeSHAP makes exact, dataset-wide SHAP analysis computationally routine rather than prohibitively expensive.

Other SHAP Explainers

Beyond KernelSHAP and TreeSHAP, the shap library provides several specialized explainers, each trading generality for speed by exploiting a specific model architecture:

• DeepExplainer: approximates SHAP values for deep neural networks using a connection to DeepLIFT, propagating attributions backward through network layers rather than sampling coalitions directly.
• GradientExplainer: combines ideas from integrated gradients and SHAP, using the model's gradients to approximate SHAP values for differentiable models, especially useful for image and tabular deep learning models.
• LinearExplainer: computes exact SHAP values in closed form for linear models, exploiting the fact that a linear model's marginal contributions do not depend on coalition order at all-the fastest possible case.
• PermutationExplainer: a general-purpose, model-agnostic explainer that approximates Shapley values via random feature-orderings and permutation sampling, useful as a robust default when a more specialized explainer isn't available.

SHAP Interaction Values

Standard SHAP values attribute a prediction to individual features, but they can obscure cases where two features only matter together-like Income and CreditHistory's interaction term in Worked Example 1 above. SHAP interaction values, based on the Shapley interaction index from game theory, decompose each pairwise SHAP value further:

where \( \delta_{ij}(S) = f(S \cup \{i,j\}) - f(S \cup \{i\}) - f(S \cup \{j\}) + f(S) \) is the pairwise marginal interaction effect. The main (non-interaction) SHAP value for feature \( i \) is then \( \phi_i = \phi_{i,i} - \sum_{j \neq i} \phi_{i,j} \), splitting the original \( \phi_i \) into a pure main effect and a set of pairwise interaction effects that together still sum back to the original prediction. TreeSHAP computes these efficiently for tree ensembles via shap_interaction_values(), revealing synergies (like Income and CreditHistory reinforcing each other) that a plain SHAP summary plot would miss.

Python Example

The shap library is the reference implementation, built directly by SHAP's original authors. Here is a complete workflow on a gradient boosted model, mirroring Worked Example 2:

import shap
import xgboost
import pandas as pd
from sklearn.model_selection import train_test_split

# Load data and train a gradient boosted model
X, y = shap.datasets.california(n_points=2000)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = xgboost.XGBRegressor(n_estimators=200, max_depth=4, random_state=42)
model.fit(X_train, y_train)

# TreeSHAP: exact, fast SHAP values for tree-based models
explainer = shap.TreeExplainer(model)
shap_values = explainer(X_test)

print(f"Base value: {shap_values.base_values[0]:.3f}")
print(f"First prediction's SHAP values:\n{shap_values.values[0]}")

# Local accuracy check: base value + sum(SHAP values) == model prediction
reconstructed = shap_values.base_values[0] + shap_values.values[0].sum()
actual = model.predict(X_test.iloc[[0]])[0]
print(f"Reconstructed: {reconstructed:.3f}  |  Actual prediction: {actual:.3f}")

Output:

Base value: 2.073
First prediction's SHAP values:
[ 0.412 -0.086  0.037  0.021 -0.009  0.058  0.114 -0.203]
Reconstructed: 2.417  |  Actual prediction: 2.417

The reconstructed value matching the actual prediction exactly is the local accuracy axiom in action-not a coincidence, but a mathematical guarantee. For a single explained instance as a waterfall plot, and for a full dataset-wide summary:

import matplotlib.pyplot as plt

# Local explanation: one prediction, one waterfall plot
shap.plots.waterfall(shap_values[0])

# Global explanation: mean absolute SHAP value per feature, across all test instances
shap.plots.bar(shap_values)

# Distributional view: every instance's SHAP value per feature, colored by feature value
shap.plots.beeswarm(shap_values)

For SHAP interaction values, exposing pairwise synergies between features (see SHAP Interaction Values above):

# Pairwise SHAP interaction values (TreeSHAP only)
interaction_values = explainer.shap_interaction_values(X_test)
print(f"Interaction tensor shape: {interaction_values.shape}")  # (n_samples, n_features, n_features)

shap.summary_plot(interaction_values, X_test)

And for a model-agnostic approach when TreeSHAP doesn't apply-say, an sklearn ensemble or arbitrary prediction function-KernelSHAP works identically to any black-box callable:

from sklearn.ensemble import RandomForestRegressor

rf_model = RandomForestRegressor(n_estimators=100, random_state=42).fit(X_train, y_train)

# Model-agnostic: KernelExplainer needs a representative background sample
background = shap.sample(X_train, 100)
kernel_explainer = shap.KernelExplainer(rf_model.predict, background)

kernel_shap_values = kernel_explainer.shap_values(X_test.iloc[:5], nsamples=200)
print(kernel_shap_values.shape)  # (5, n_features)

Reading SHAP Plots

• Waterfall plot: explains one prediction. Starts at the base value on the left, then shows each feature's SHAP value as a bar pushing the running total up (red, positive) or down (blue, negative), ending exactly at the model's predicted output-visually identical to the arithmetic in Worked Example 1.
• Force plot: the same additive story as a waterfall, compressed into a single horizontal bar-useful for embedding many explanations compactly, such as one per row in a report.
• Bar plot (global): ranks features by their mean absolute SHAP value across every instance in a dataset, giving a single, theoretically consistent global feature-importance ranking.
• Beeswarm plot: shows every instance's SHAP value for every feature simultaneously, one row per feature, points colored by the feature's own value (typically red = high, blue = low)-this is the single most information-dense SHAP plot, revealing both importance ranking and the direction/shape of each feature's effect at once.
• Dependence plot: plots one feature's actual value on the x-axis against its SHAP value on the y-axis for every instance, often colored by a second, automatically-selected interacting feature- this is where interaction effects become visually obvious.

How to Interpret Results

Always report SHAP values in the model's actual output units (dollars, probability, log-odds), never as unitless "importance scores" alone-this is what makes SHAP values directly interpretable rather than merely a ranking. State explicitly, for any single explanation, which direction "positive" and "negative" point (typically toward and away from the base value), since this is a frequent source of miscommunication when sharing plots with non-technical audiences.

For classification models, be precise about which output space the SHAP values are computed on-raw log-odds/margin output versus post-sigmoid probability-because SHAP values are additive in the model's raw output space by default, and the additivity guarantee does not carry over cleanly after a nonlinear transformation like the sigmoid without extra care (the shap library exposes both, but they are not interchangeable).

Practical Considerations & Pitfalls

• Correlated features split credit unintuitively. When two features are highly correlated, SHAP (using the standard, interventional definition of \( f(S) \)) can split their combined effect between them in ways that look arbitrary feature-by-feature, even though the total combined attribution remains meaningful and correct.
• Background dataset choice matters. The base value and every downstream SHAP value depend on which background dataset is used to represent "no information"-a background sample that isn't representative of the true population will produce systematically skewed base values and attributions.
• KernelSHAP sample count trades off speed for precision. Too few sampled coalitions produce noisy, unstable SHAP value estimates that can even change sign on repeated runs-always check for stability by increasing nsamples and confirming the values converge.
• SHAP explains the model, not the world. A feature receiving a large SHAP value means the model relied heavily on it-this is a statement about the model's learned function, not a causal claim about the underlying data-generating process.

Advantages

• The only additive feature attribution method with a proven uniqueness guarantee, satisfying local accuracy, missingness, and consistency simultaneously-a mathematical guarantee competing methods like plain LIME or ad hoc perturbation scores do not offer.
• Unifies local (single-prediction) and global (dataset-wide) explanations under one consistent framework-no need to switch methods when zooming from one row to the whole dataset.
• TreeSHAP makes exact, fast attribution routine for the most common production model class (tree ensembles), removing the sampling-variance tradeoff entirely for that case.
• Extends cleanly to interaction values, revealing pairwise synergies that single-feature attribution methods cannot expose.

Limitations

• Exact computation is exponential in the number of features-practical use depends entirely on either algorithmic shortcuts (TreeSHAP) or sampling approximations (KernelSHAP), each with its own caveats.
• Highly correlated features can produce attributions that are individually counter-intuitive, even though their combined effect remains correct and meaningful.
• SHAP values explain the model's learned behavior, not causal relationships in the underlying data-a common and consequential misinterpretation when presenting results to stakeholders.
• Choice of background dataset is a real modeling decision with real consequences for the resulting values, and is easy to get wrong silently (e.g., using an unrepresentative or too-small sample).

When NOT to Use It

• Permutation feature importance: use instead when you only need a quick, global ranking of feature importance and don't need per-instance explanations or SHAP's additivity guarantee.
• Partial dependence plots (PDP): use instead (or alongside SHAP dependence plots) when you want to see a feature's average marginal effect on predictions across its whole range, without needing individualized, instance-level attribution.
• Intrinsically interpretable models (a short decision tree, sparse linear regression, or a scorecard): when the model itself is already transparent, a post-hoc explanation layer like SHAP adds computational cost without adding clarity.
• Causal inference tasks: use dedicated causal methods (e.g., instrumental variables, causal graphical models, randomized experiments) instead-SHAP explains model behavior, not cause-and-effect in the real world.

SHAP vs LIME vs Permutation Importance vs Partial Dependence

20.1 SHAP vs LIME

20.2 SHAP vs Permutation Feature Importance

20.3 SHAP vs Partial Dependence Plots

Common Misconceptions

• "A high SHAP value means the feature causes the outcome." No-SHAP explains what the model learned to rely on, not a causal relationship in the real world; correlation baked into training data becomes attribution, not causation.
• "SHAP values are just a fancier feature importance ranking." No-unlike plain importance scores, SHAP values are signed, additive, and sum exactly to each individual prediction (see local accuracy), giving both direction and per-instance detail that a ranking alone cannot.
• "KernelSHAP and TreeSHAP give the exact same numbers." Not necessarily-TreeSHAP is exact by construction for tree models, while KernelSHAP is a sampling-based approximation; they converge to the same values as sample size grows, but will differ slightly for any finite sample.
• "SHAP works instantly on any model." Exact SHAP computation is exponential in the number of features; only TreeSHAP (tree models) and a handful of other specialized explainers achieve practical speed-KernelSHAP on an arbitrary black-box model can still be slow for large feature counts.
• "A feature with a near-zero average SHAP value is unimportant." Not always-a feature can have large positive SHAP values for some instances and large negative values for others, canceling out in a naive average; always inspect the beeswarm plot's spread, not just the mean absolute value or a raw mean.

Interview Questions

1. Derive the SHAP value formula from the Shapley value in cooperative game theory, and explain what each term in the formula represents.
2. State and explain the four Shapley axioms, and explain why the Shapley value is the unique solution satisfying them.
3. Why is exact SHAP computation exponential in the number of features, and how do KernelSHAP and TreeSHAP each address this differently?
4. Walk through how TreeSHAP achieves exact SHAP values in polynomial time by exploiting tree structure, without resorting to sampling.
5. Explain the local accuracy (additivity) property and why it makes SHAP values fundamentally different from ordinary feature-importance scores.
6. How do SHAP interaction values decompose a standard SHAP value, and what do they reveal that a plain SHAP summary plot cannot?
7. What is the practical difference between interventional and tree path-dependent feature perturbation in TreeSHAP, and when would you choose one over the other?
8. Why can two highly correlated features produce individually counter-intuitive SHAP values, even though their combined attribution remains correct?
9. Explain why SHAP values computed in a model's raw output space cannot simply be summed after applying a sigmoid transformation for a classification model.
10. Compare SHAP and LIME as members of the "additive feature attribution methods" family, and explain specifically which theoretical guarantees LIME lacks that SHAP provides.

Frequently Asked Questions

• What is a SHAP value? +
  A SHAP value is the average marginal contribution of one feature to a single model prediction, calculated by averaging that feature's effect across every possible coalition (subset) of the other features, using the Shapley value from cooperative game theory. It answers the question: how much credit does this feature deserve for pushing this specific prediction away from the model's average output?
• What is the formula for SHAP values? +
  For feature i among M total features, phi_i = sum over subsets S not containing i of [ |S|!(M-|S|-1)! / M! ] * [ f(S union {i}) - f(S) ]. The weighting term |S|!(M-|S|-1)!/M! is exactly the Shapley weight, which accounts for every ordering in which feature i could be added to a coalition of size |S|, and f(S) is the model's expected prediction conditioned only on the features in S.
• How do you interpret SHAP values? +
  A positive SHAP value pushes a prediction above the model's base (average) value; a negative SHAP value pushes it below. By the additivity property, base value plus the sum of every feature's SHAP value for an instance equals that instance's exact predicted output, so SHAP values behave like signed, additive credits rather than an abstract importance score.
• What is the difference between KernelSHAP and TreeSHAP? +
  KernelSHAP is a model-agnostic method that approximates SHAP values by sampling coalitions and fitting a specially weighted linear regression; it works with any model but its runtime grows exponentially with the number of features unless sampled. TreeSHAP computes exact SHAP values in low-order polynomial time specifically for tree-based models by exploiting the tree's recursive structure, making it both exact and dramatically faster than KernelSHAP for random forests and gradient boosted trees.
• What is the difference between SHAP and LIME? +
  LIME fits a local, ad hoc weighted linear surrogate model around one instance with a heuristic kernel and neighborhood-sampling scheme, and does not guarantee a unique or theoretically grounded solution. SHAP is the unique additive feature attribution method satisfying local accuracy, missingness, and consistency simultaneously, derived directly from Shapley values in cooperative game theory-Lundberg and Lee's 2017 paper shows LIME is a special case of the broader additive feature attribution family that SHAP formalizes and unifies.
• Do SHAP values sum to the prediction? +
  Yes. This is the local accuracy (also called additivity or efficiency) property inherited directly from Shapley values: for any single prediction, the base value (the model's average output over a background dataset) plus the sum of that instance's SHAP values across all features exactly equals the model's predicted output for that instance, with zero unexplained residual.
• Why is computing exact SHAP values expensive? +
  The exact formula requires evaluating the model on every possible coalition (subset) of the remaining M-1 features for each feature you want to explain, which means 2^(M-1) model evaluations per feature. This becomes computationally intractable for even moderately sized feature sets, which is exactly why approximations like KernelSHAP (sampling) and exact-but-specialized algorithms like TreeSHAP (structure exploitation) exist.
• Can SHAP values be used for global feature importance, not just individual predictions? +
  Yes-averaging the absolute SHAP value of a feature across every instance in a dataset produces a global importance ranking, and this is exactly what a SHAP summary or bar plot displays. This lets SHAP unify local, per-prediction explanations and global, dataset-wide feature importance in a single, theoretically consistent framework.

Key Takeaways

• SHAP value analysis explains individual model predictions by computing each feature's exact (or approximated) Shapley value-its average marginal contribution across every possible coalition of the other features, borrowed directly from 1953 cooperative game theory.
• SHAP is the unique additive feature attribution method satisfying local accuracy, missingness, and consistency simultaneously-a proven guarantee, not a heuristic claim.
• TreeSHAP computes exact SHAP values in polynomial time for tree-based models (random forests, XGBoost, LightGBM); KernelSHAP approximates them for any model via a specially weighted local linear regression.
• By local accuracy, the base value plus every feature's SHAP value for an instance sums exactly to that instance's prediction-verified directly in both worked examples above.
• SHAP unifies local (per-prediction) and global (dataset-wide) explanations, extends to pairwise interaction values, and is best read through waterfall, force, bar, beeswarm, and dependence plots-each surfacing a different layer of the same underlying attribution.
• SHAP explains what a model learned to rely on, not what is causally true about the world-keep that distinction explicit whenever reporting results, especially to non-technical stakeholders.

SHAP value analysis earns its place as the standard for model explanation not because it is popular, but because it is provably the only additive attribution method with a fair, unique, and consistent solution-a guarantee inherited directly from Lloyd Shapley's 1953 game-theoretic result, seven decades before it found its most consequential application. Where earlier explanation methods offered plausible heuristics, SHAP offers a proof: local accuracy, missingness, and consistency, together, uniquely determine the Shapley value, and nothing else satisfies all three.

The two worked examples above-a fully hand-computed 3-feature model and a TreeSHAP-scale 5-feature case-show both ends of SHAP's practical range, and the lesson connecting them: exact computation is tractable by hand only for a handful of features, which is exactly why TreeSHAP's algorithmic shortcut and KernelSHAP's sampling approximation exist as the two practical paths to the same underlying quantity. Reporting SHAP values in the model's native output units, checking the local accuracy identity explicitly, reading the beeswarm and dependence plots for direction and interaction-not just the bar plot for ranking-and always keeping "the model relied on this" separate from "this causes that," gives a complete and honest explanation that a raw feature-importance score never could.