Demystifying SHAP: How Game Theory Explains Machine Learning Predictions

Machine learning models predict customer attrition, project property valuations, and detect financial anomalies with high precision. But a probability score of 0.82 or a predicted revenue of $115,000 is useless if you cannot explain why the model arrived at that number.

In production environments, black-box predictions create operational friction. Risk teams reject opaque credit decisions. HR managers cannot act on unexplained turnover alerts. Engineers cannot debug unexpected model outputs without tracing feature contributions.

This is where explainable AI (XAI) becomes essential.

In this post, we cover what explainable AI is, how SHAP (Shapley Additive exPlanations) computes feature attributions from first principles, and how to read local waterfall plots and global directional beeswarm diagrams.

What is explainable AI?

Explainable AI refers to methods and techniques that allow human operators to understand and audit the outputs generated by machine learning algorithms.

Traditional tabular models like linear regression offered natural interpretability through model coefficients. Modern deep learning architectures, gradient boosted trees, and tabular foundation models like TabICL capture complex non-linear feature interactions, but lose direct coefficient interpretability.

Explainable AI bridges this gap post-hoc. Instead of restricting yourself to simplistic linear models, you use high-capacity models and apply post-hoc attribution algorithms to compute the exact marginal contribution of every input column.

How does SHAP work from first principles?

Introduced in Scott Lundberg and Su-In Lee’s seminal 2017 paper, SHAP (Shapley Additive exPlanations) is the gold standard for explainable machine learning. For API references and python implementation details, see the official SHAP documentation.

SHAP adapts cooperative game theory developed by Nobel laureate Lloyd Shapley in 1953.

Imagine a game where a team of players collaborates to achieve a final payout. Some players contribute more than others; some players only add value when paired with specific teammates. How do you divide the payout fairly among the players?

Lloyd Shapley proved that there is exactly one payout distribution that satisfies four fundamental axioms of fairness: efficiency, symmetry, dummy player neutrality, and additivity.

In tabular machine learning:

• The game is the prediction task for a specific row.
• The players are the feature columns (Age, MonthlyIncome, OverTime).
• The payout is the difference between the model’s actual prediction $f(x)$ and the expected base prediction $\text{baseValue}$ across the training dataset.

To calculate the Shapley value $\phi_i$ for feature $i$, the algorithm measures how the prediction changes when feature $i$ is added to every possible subset (coalition) $S$ of remaining features $F$:

$$\phi_i = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|!(|F| - |S| - 1)!}{|F|!} \left[ f(S \cup \{i\}) - f(S) \right]$$

The core property: Additive feature attribution

Because Shapley values satisfy the efficiency axiom, the sum of all feature attributions equals the exact distance between the average dataset expectation and the model output:

$$\text{baseValue} + \sum_{i=1}^{M} \phi_i = f(x)$$

Every feature receives a positive or negative score in the exact units of the prediction output.

Global feature ranking: The SHAP summary bar plot

Before diving into individual rows or directional effects, teams often want a high-level ranking of overall feature importance across the entire dataset.

The SHAP global summary bar plot calculates the average absolute SHAP value for each feature across all reference samples:

$$\text{mean } |\text{SHAP value}| = \frac{1}{N} \sum_{j=1}^{N} |\phi_i^{(j)}|$$

Features are sorted from top to bottom by their overall magnitude of influence on the model’s predictions.

Here is how to read Figure 1:

• Magnitude ranking. Features are ordered from top to bottom by mean absolute impact. OverTime has the largest overall effect on model predictions ($\text{mean } |\text{SHAP}| = 0.18$), followed by MonthlyIncome ($0.15$).
• Simple macro summary. The bar plot provides a clean, executive summary when you want to communicate which input variables matter most without overwhelming non-technical stakeholders with individual sample points.

Reading local attributions: The SHAP waterfall plot

A SHAP waterfall plot explains a single prediction for one specific row in your dataset.

It shows how the model starts at the expected baseline value $\text{baseValue}$ (the average prediction across the reference dataset) and pushes step-by-step through each feature contribution until reaching the final prediction $f(x)$.

Here is how to read Figure 2:

1. Baseline value ($\text{baseValue} = 0.35$). The average attrition risk across the entire reference dataset is 35%.
2. Positive pushes (red bars). OverTime = Yes increases attrition probability by +24%. MonthlyIncome = $2,400 adds another +15%. StockOptionLevel = 0 adds +7%.
3. Negative pushes (blue bars). TotalWorkingYears = 2 reduces attrition risk by -11%.
4. Final prediction ($f(x) = 0.70$). Summing the baseline and individual attributions yields a final attrition probability of 70%.

Waterfall plots tell you precisely why an individual prediction was made.

Reading global attributions: The SHAP directional beeswarm plot

While a waterfall plot explains a single row, a SHAP directional beeswarm plot combines feature importance with feature effects across your entire dataset.

A standard bar chart only shows mean absolute importance $|\phi|$. It tells you that OverTime matters, but it does not tell you whether working overtime increases or decreases risk.

The directional beeswarm plot solves this by plotting every sample row as a distinct colored point.

Here is how to read Figure 3:

• Vertical ordering (Y-axis). Features are ordered by global importance from top to bottom. OverTime is the single most influential feature across the dataset.
• Horizontal position (X-axis). Points on the right of the center line ($>0$) pushed the prediction higher. Points on the left ($<0$) pulled the prediction lower.
• Color gradient. Deep blue represents a low feature value (e.g. OverTime = No or low salary). Red represents a high feature value (e.g. OverTime = Yes or high salary).
• Vertical point jitter. When multiple sample points have identical SHAP values, they stack vertically to show density.

Key dataset insights revealed by Figure 3

• OverTime effect. High values (red points, OverTime = Yes) cluster far to the right ($+0.25$). Low values (blue points, OverTime = No) cluster to the left ($-0.12$). Overtime is a strong positive driver of attrition.
• Monthly income effect. High monthly income (red points) clusters on the left ($-0.25$). Low monthly income (blue points) clusters on the right ($+0.15$). Higher compensation strongly reduces churn risk.

Directional beeswarm plots reveal subtle, non-linear relationships that single scalar importance metrics completely obscure.

Executing SHAP locally inside Google Sheets with Carla

Traditionally, generating SHAP explanations required export scripts, Python notebooks, or sending sensitive customer data to third-party cloud analytics services.

Carla brings instant, interactive SHAP explanations directly into your Google Sheets workflow:

• 100% On-Device Privacy. Model explanations are calculated entirely inside your browser. Your raw spreadsheet data never leaves your computer or passes through an external server.
• Instant Interactive Explanations. As soon as you select a model or row, Carla generates local waterfall attributions and global feature importance charts right inside the Google Sheets sidepanel.
• Smooth & Responsive Workflow. Heavy calculations run seamlessly in the background, keeping your spreadsheet fast, responsive, and easy to navigate.

You get complete transparency into every model prediction—without compromising data privacy or leaving your spreadsheet.

Try Carla in private beta

Carla brings privacy-first tabular AI and explainability directly into Google Sheets.

If you want to train models and inspect SHAP waterfall and beeswarm diagrams on your business data, join our private beta waitlist.

Join the Carla private beta to get early access, or explore our documentation to see how Carla keeps your tabular data private.