When your model declines someone, you may owe them a specific reason, and a regulator may later ask you to stand behind it. That reason can be an artefact of which of two overlapping inputs the model happened to credit, and it can change next month while the decision itself stays identical. No dashboard will flag it.

What a feature attribution actually claims

When a model produces a decision, a feature attribution method assigns each input a share of the responsibility for that particular output. The result is the ranked list your team shows you: income counted for this much, account age for that much, postcode for a little.

The important word is assigns. An attribution is a reconstruction, not a recording. The model did not decide “because of income” and write that down. Attribution methods work backwards, asking what the prediction would have been without each input, across many combinations, then dividing the difference between them. That is an inference about the model, produced after the fact, by a separate piece of software. If your team names the tools they use for this, the two you are most likely to hear are SHAP and LIME.

Which means explainability has its own accuracy, separate from the model’s. NIST makes this explicit in its four principles of explainable artificial intelligence: an explanation has to be meaningful to the person receiving it and accurate to the process it describes, and that second property is measured against the model’s actual behaviour rather than against reality. Your model can be right and your explanation wrong. Those are two measurements, and most organizations only take the first.

Why unreliable explanations land on your desk

This is not a data science quality issue that stays in the data science team. It arrives at your desk for three reasons.

The explanation is a regulated deliverable. In United States consumer lending, a creditor must give specific principal reasons when it takes an adverse action, and the Consumer Financial Protection Bureau has stated directly that complex algorithms are no defence: a lender cannot decline someone and then plead that the model is too complicated to explain. The EU AI Act takes a parallel position, giving people subject to certain high-risk decisions a right to an explanation of that decision. Once an explanation is something you owe rather than something you produce for yourself, its reliability is a compliance property.

It is the evidence in your own defence. When a decision is challenged, months or years later, the explanation is the artefact you produce. If your attribution would come out differently against a retrained model, you cannot reproduce the reason you originally gave. You are left asserting a number you can no longer generate, which is a poor position in front of a regulator and a worse one in front of a court.

It quietly corrupts your fairness testing. If an attribution reports that a sensitive proxy barely influenced anything, the natural conclusion is that you do not have a proxy problem. That conclusion is only as good as the attribution underneath it. NIST’s work on identifying and managing bias in AI is clear that bias arrives from institutional practice reflected in the data, not only from unrepresentative sampling, and that kind of bias is exactly what overlapping inputs carry.

It is worth separating this from the failure covered elsewhere in this series. AI observability answers the question “is this model still working?”. Here the model is working. Accuracy holds, drift detectors stay quiet, and the decisions are the ones you want. There is no degradation for monitoring to catch, because nothing has degraded. Only the account of why has come loose.

How attribution breaks when features overlap

The mechanism is simple enough to hold in your head.

Suppose two inputs carry substantially the same information. Postcode and the average income of that postcode. Account age and total transactions. Salary in one currency and the same salary converted. The model can reach the same prediction by leaning on either one, or by splitting between them in any proportion. Every one of those weightings fits the data about equally well, so the model has no principled basis for choosing, and it settles on one arbitrarily.

Retrain it on slightly different data and it settles somewhere else.

  Retrain #1            Retrain #2            Retrain #3

  postcode    0.31      postcode    0.04      postcode    0.22
  income      0.05      income      0.32      income      0.14
  ...............      ...............      ...............
  combined    0.36      combined    0.36      combined    0.36

        same data, same accuracy, same decision for this applicant
                  three different reasons on the notice

The combined influence is stable and correct. The split between the two is noise presented as insight. If your adverse action notice names the top contributing factor, that notice named postcode in one version of the model and income in the next, for the same applicant and the same outcome.

The name for the underlying condition is multicollinearity, and it is measurable. The standard check is the variance inflation factor, which takes each input in turn and asks how well the remaining inputs predict it. The more redundant the input, the higher the number. A value of one means the input stands alone; conventional practice treats values above five as worth investigating and above ten as worth acting on. It is a routine diagnostic, and the useful question for you is not what the numbers are but whether anyone is running it before explanations reach a customer.

The available remedies each cost something, which is why this is a decision rather than a fix. You can drop one of the overlapping inputs, which is clean but throws away information. You can combine them into a single stated factor, which is honest but changes what your notice can say. You can apply techniques that constrain how much weight the model gives any one input, which stabilises the split without removing the overlap. Or you can mathematically rebuild the inputs into non-overlapping components, which solves the statistics completely and leaves you with factors that have no business meaning at all, and therefore nothing you could put on a notice. That last option cures the condition by removing the organ.

Where explanation integrity goes wrong

The failure modes are organizational rather than mathematical, and each one turns into something you have to answer for.

  • Nobody checks for overlap before explanations go out. The redundancy diagnostic is cheap and well understood, and it is frequently not part of the release checklist, so the first time anyone asks whether the stated reasons are stable is after somebody disputes one.
  • Explanations are not versioned with the model. The decision is logged, the model version may be logged, and the explanation is regenerated on demand against whatever model is current. Reproducing what you told someone in March then becomes impossible.
  • The people producing the explanation own the performance metrics. When the same team is measured on model accuracy and also supplies the account of how the model reasons, nobody independent is asking whether the second thing is true.
  • Attribution is treated as ground truth in fairness testing. A low attribution score for a sensitive proxy is taken as evidence of fairness, when it may only be evidence that the credit landed on a correlated input instead.
  • A vendor supplies the explanation and the method is opaque. You are then passing on an account of a decision that you cannot inspect, while remaining the party who owes the explanation.
  • The remedy is chosen without anyone naming the trade-off. A team under pressure to make the diagnostic numbers look good can pick the option that destroys interpretability, which technically resolves the finding and quietly removes your ability to explain anything at all.

Questions to ask about your model explanations

You will not be reading attribution output yourself, and you do not need to. A short list separates an explanation you could defend from one you are hoping about.

  • For any model that makes decisions about people, do we check our inputs for redundancy before release, and who sees that result?
  • When we tell someone why they were declined, could we reproduce that same explanation in twelve months, against the model version that actually made the decision?
  • Are our explanations stored with the decision, or regenerated later against whatever model is live?
  • If we retrained tomorrow, would the stated top factor for a given decision stay the same? Has anyone ever tested that?
  • Who independently checks that our explanations describe the model accurately, and is that a different person from the one accountable for its accuracy?
  • Where a vendor supplies both the model and its explanations, what method produces them, and can we inspect it?
  • When we conclude from an attribution that a sensitive proxy is not influencing outcomes, what else supports that conclusion?
  • If someone challenged one of our automated decisions today, what exactly would we hand over, and who has read it recently?

Crisp answers mean you have explanations you can stand behind. Shrugs mean you have a regulated deliverable that nobody is measuring.

A reason you cannot reproduce is not a reason. It is a story that happened to fit.