Multicollinearity

Postcode and the average income of that postcode. Account age and total transactions. A salary recorded twice in different currencies. When inputs overlap like this, a model can reach the same prediction by leaning on either one or by splitting between them in almost any proportion, and every one of those weightings fits the data about equally well. Having no principled way to choose, it settles on one arbitrarily, and settles somewhere else when it is retrained on slightly different data.

The consequence is narrower than it first sounds, and worse. Predictive accuracy is largely unaffected: the combined influence of the overlapping inputs stays stable and correct, which is why performance metrics, test results and drift monitoring all stay quiet. What becomes unreliable is the account of why. Feature attribution splits credit between the overlapping inputs and the split moves between model versions, so the top stated factor behind a decision can change while the decision itself does not. Coefficients can also swing hard between retrains, which makes model behaviour harder to compare across versions than it appears.

This is the caveat on a claim that otherwise holds: the appeal of a linear regression is that you can read its coefficients and argue with them. That is true while the inputs are independent, and stops being reliable when they are not. The standard check is the variance inflation factor, which takes each input in turn and measures how well the remaining inputs predict it. A value of one means the input stands alone; conventional practice investigates above five and acts above ten.

There is a second consequence that reaches fairness work. Removing a protected attribute does not remove its influence when correlated inputs still carry it, and NIST’s work on identifying and managing bias in AI is clear that bias reflecting institutional practice survives in ordinary looking data. Remedies exist, and each costs something: drop one input and lose information, merge them and change what your explanation can say, constrain how much weight any single input receives, or rebuild the inputs as non overlapping components and lose interpretability altogether. That last option resolves the statistics and leaves you nothing you could put in front of a regulator, which makes it the wrong answer whenever explainability was the reason you cared.