Why Explainability is Not the Problem in Machine Learning Risk

Explainability has become one of the most debated topics in modern risk management. Regulators expect it, institutions invest heavily in it, and model risk teams continue to refine techniques designed to make complex machine learning outputs interpretable. Yet according to Joe Breeden, Chief Executive Officer at Deep Future Analytics, the industry may be focusing on the wrong challenge.

“We don’t have a problem with inability to explain,” he says. “We simply have regions that are not predictive, and if the prediction is not usable, then there’s no point trying to explain it.”

This distinction is critical. The prevailing assumption across financial services is that every model output should be explainable, particularly in high-stakes decisions such as credit approval or fraud detection. However, Breeden argues that explainability is only meaningful when the underlying prediction is reliable.

“If you don’t have a usable forecast, you won’t be able to explain that forecast,” he explains. “If there’s no good forecast, there’s no good explanation.”

At the heart of the issue is a lack of focus on predictive uncertainty. Traditional models often imply a level of confidence based on historical data and relatively stable relationships. Machine learning models, by contrast, operate in more dynamic environments where predictive accuracy can vary significantly depending on the data.

Despite this, most institutions do not explicitly measure how confident they should be in a given prediction. “I don’t know any operational systems that regularly compute forecast uncertainty,” Breeden says. “They have a score, but what’s the confidence interval around that score?”

Without that context, organizations risk placing undue confidence in model outputs and the explanations that accompany them. This can lead to a false sense of assurance, particularly when widely used explainability techniques are applied indiscriminately.