Explainability

Explainability answers one question: how did this system arrive at this result? It is worth separating from transparency, which it is often confused with. Transparency is disclosure about the system: what it is, what it was trained on, what it is for, where its limits are. Explainability is about a particular output. You can be entirely transparent about a model and still unable to say why it declined a specific applicant.

The reason it belongs to a security leader rather than only to a data science team is that explainability is load bearing for everything downstream of a decision. You cannot contest a decision you cannot explain, prove it was not discriminatory, demonstrate a control worked, or investigate it after the fact. Where a system is opaque enough that its outputs cannot be accounted for, the usual phrase is the black box problem, and its practical consequence is that errors and bias become hard to find, compliance becomes hard to demonstrate, and every detective control that depends on understanding the output gets weaker.

The part most organizations miss is that an explanation has its own accuracy, and it is not the same measurement as the model’s. NIST sets out four principles of explainable artificial intelligence: a system should provide an explanation, that explanation should be meaningful to its audience, it should be accurate to the process the system actually used, and the system should operate only within the conditions it was designed for. The third of those is the one that catches people. A model can be right while the stated reasons for any individual output are unreliable, and no accuracy metric will reveal it. See feature attribution for how those reasons are generated, and multicollinearity for the most common way they come loose.

Under the EU AI Act, explanation stops being a good practice and becomes a deliverable: people subject to certain high risk decisions have a right to an explanation of them. At that point the reliability of your explanations is a compliance property, and someone has to own it.