Banks Race to Validate the Machines They Can't Explain

Generative and agentic AI models are reshaping how banks operate - creating code, summarizing risk data, and even generating credit decisions. 

But their arrival has exposed one of the most urgent challenges in model risk management: how to validate systems that think in ways their creators cannot fully explain.

In traditional model validation, practitioners rely on interpretability. They check assumptions, verify parameters, and re-perform calculations. None of that applies neatly to large language models (LLMs) or multi-agent systems. 

These architectures are not deterministic. They respond to context, data prompts, and internal embeddings rather than static equations.

Regulators have begun to take notice. The U.S. Federal Reserve’s supervisory letters, the EU’s forthcoming AI Act, and the UK’s PRA SS1/23 guidance all demand that firms maintain ‘appropriate validation standards’ for AI systems. 

But what that means in practice remains unsettled. A 2025 Risk.net survey found that over 60% of banks now classify GenAI applications as models under SR 11-7 - but admit they lack the tools to perform independent validation.

Institutions are therefore embracing outcome-based testing. Rather than inspecting the logic of an AI model, they measure whether its decisions remain consistent, fair, and reproducible under stress. 

Perturbation analysis - slightly altering inputs to observe the stability of outputs - is one emerging approach. Another is red-team testing, where specialist teams intentionally provoke hallucinations or bias to understand model behavior under pressure.

Documentation has become a frontline defense. Firms are mapping dependencies, logging prompts, and tracking version control of foundation models to satisfy auditors. 

According to the Bank for International Settlements, effective AI governance now requires “model lineage visibility,” ensuring that all data transformations and fine-tuning steps are recorded for accountability.