A quiet assumption in every compliance document
Article 13 of the EU AI Act requires high-risk AI systems to provide deployers with information sufficient to interpret the system's output. In practice, for a gradient-boosted fraud model, that means SHAP values — a feature attribution method that has become the default answer to “why did the model flag this transaction.”
Most compliance documentation treats this as settled. The model produces a prediction, SHAP explains it, a human reviews the explanation, done. What that picture quietly assumes is that the explanation is a stable, reliable witness to what the model actually did. My earlier post on this thesis covered what happens to the predictionunder adversarial attack. This one is about a narrower and, I'd argue, more consequential question: what happens to the explanation, and does it still satisfy Article 13 once the input has been adversarially perturbed?
The answer draws on a body of research most fraud teams haven't encountered: Slack et al. (2020)showed that post-hoc explanation methods including SHAP and LIME can be deliberately gamed — it's possible to construct a model that produces innocuous, trustworthy-looking explanations for an auditor while behaving completely differently on the inputs that matter. That result is the foundation this thesis builds on, and it changes the compliance question from “does the model produce an explanation” to “can the explanation be trusted under exactly the conditions where trust matters most.”
Why SHAP specifically, and why it should be more resistant than this
SHAP isn't an ad-hoc heuristic. Lundberg and Lee's original formulation grounds it in cooperative game theory's Shapley value, which satisfies four provable axioms — efficiency, symmetry, dummy, and linearity — that together guarantee a unique, fair attribution of a prediction across its input features. The TreeExplainer algorithm makes this tractable for tree ensembles like XGBoost by computing exact Shapley values in polynomial time, rather than the exponential-time brute force the naive formulation would require.
None of that theoretical rigor is in question. What the earlier post's data showed is that the rigor of the attribution methoddoesn't protect you from the instability of the input it's attributing over. SHAP will always give you a mathematically well-defined answer. Under adversarial perturbation, that well-defined answer can point in a mathematically well-defined but practically backwards direction — a Spearman correlation of −0.558 between the clean and adversarial feature rankings, which is close to a perfect inversion rather than a modest degradation.
Cross-checking with a second explanation method
To sanity-check whether this instability was an artifact of SHAP specifically, the thesis ran a side-by-side comparison against LIME, Ribeiro et al.'s model-agnostic method that fits a local linear surrogate around each explained instance rather than computing global Shapley values.
SHAP vs LIME agreement on top-3 features across clean fraud cases
On clean data, the two methods — built on genuinely different theoretical foundations — agreed on the top-3 most important features in 77.78% of fraud cases, with amount_to_balance_ratio identified as the primary driver by both methods in every fraud case tested. That's a meaningful cross-validation result: it means the clean-data explanations aren't an artifact of SHAP's specific mathematics, since an unrelated method built differently converges on the same story. It also means the instability documented under attack isn't a SHAP quirk either — Slack et al.'s original finding was that bothSHAP and LIME are manipulable, and the mechanism is the same in both cases: the explanation is only as stable as the decision boundary geometry it's describing, and adversarial perturbation is specifically engineered to move a point across that geometry.
The compliance implication nobody's written down
Here's the part of the thesis I think is genuinely underappreciated outside academic XAI circles. The EU AI Act treats Article 9 (risk management), Article 13 (transparency), and Article 15 (accuracy, robustness, and cybersecurity) as separate obligations, assessed independently. The empirical results say they aren't independent at all.
Article 13/15 dependency — how adversarial robustness and explanation stability are coupled
A model can pass Article 15 testing — demonstrating strong accuracy and even reasonable adversarial resilience on its predictions — while still failing Article 13, because its explanations become unreliable under exactly the adversarial pressure Article 15's testing regime is designed to probe. Put differently: satisfying Article 13(2)'s explanation-stability requirement, in practice, requires the adversarial training defence that Article 15(3) mandates. A conformity assessment that tests these articles in isolation — which is how most compliance checklists are structured today — can wave through a system that would fail the moment both pressures are applied at once.
This also reframes a debate that's been running in the XAI compliance literature about whether explaining PCA-transformed features counts as “meaningful information” under Article 13(1). It technically produces a valid SHAP value. It does not produce something a fraud analyst or auditor can act on, because “component 7 contributed +0.3” carries no domain meaning. The thesis's choice to use interpretable engineered features (transaction amount, balance change, account drainage pattern) rather than anonymized components isn't just a methodological preference — it's closer to what Article 13(1) is actually asking for.
Where this sits relative to prior work
Related-work positioning — how this thesis extends prior adversarial XAI research
Cartella et al. demonstrated near-perfect attack success rates against gradient-boosted fraud classifiers but didn't touch explanation stability. Slack et al. and Baviskar showed SHAP and LIME can be destabilized, but neither combined that with a black-box threat model, a measured defence, and a regulatory mapping. As far as this thesis's literature review could establish, no existing empirical study connects adversarial robustness, SHAP/LIME stability, and EU AI Act compliance mapping in one pipeline on financial tabular data — which is the specific gap this work fills.
What this means practically
If you're deploying SHAP or LIME explanations in a regulated financial context, the takeaway isn't “don't use post-hoc explanations.” It's narrower and more actionable than that:
- ›Test explanation stability as its own metric, separate from prediction accuracy — Spearman correlation between clean and adversarial rankings is a cheap, informative one to start with.
- ›Don't treat Article 13 and Article 15 conformity as independent checkboxes.The evidence suggests they're coupled, and a system tested for each in isolation can still fail when both pressures hit at once.
- ›Prefer interpretable engineered features over black-box dimensionality reduction if the explanation is meant to be substantively actionable rather than just technically present.
- ›Cross-validate with a second explanation method where practical.Agreement between SHAP and LIME on clean data doesn't guarantee both survive adversarial conditions, but persistent disagreement on clean data is itself an early warning sign worth investigating before an attacker ever gets involved.