Feature attribution

Feature attribution is how the ranked list of reasons behind a model decision gets produced: income counted for this much, account age for that much, postcode for a little. It is the mechanism most organizations are relying on when they say a model is explainable, and the thing that ends up printed on a notice sent to a customer.

The word doing the work is assigns. A model does not decide “because of income” and record that fact. Attribution methods work backwards from the output, asking what the prediction would have been without each input, across many combinations of the others, then dividing the difference between them. Some methods explain a single prediction, others aggregate across many predictions to rank inputs overall. All of them are producing an inference about the model, after the fact, using separate software. That is a reconstruction, and it can be a good one or a poor one.

Which means it carries its own error, independent of the model’s. NIST’s AI Risk Management Framework treats being explainable and interpretable as a characteristic of a trustworthy system in its own right, alongside accuracy, rather than as something accuracy delivers for free. In a linear regression the coefficients are simple enough that the model largely explains itself. In a neural network there is nothing readable to inspect, so an attribution method is the only account you have, and its trustworthiness becomes the trustworthiness of your explanation.

The failure worth knowing about is what happens when two inputs carry overlapping information. The method has no principled basis for splitting the credit between them, so it splits arbitrarily, and the split moves when the model is retrained. The combined influence stays right while the breakdown becomes noise. See multicollinearity. For a security leader the practical question is not which method your team uses but whether anyone checks that its output is stable before it reaches a regulator or a customer.