As generative AI enters financial services, traditional validation approaches are falling short. Experts call for a new paradigm - one grounded in risk, transparency, and human oversight - to ensure AI systems remain accountable and aligned with their int
As generative AI finds its way into
customer communications, decision support tools, and internal workflows,
financial institutions are being forced to rethink what effective model
validation really means.
At a recent CeFPro industry panel,
senior practitioners made clear that while the technology may be new, the
underlying challenge is familiar: how to ensure models behave as intended,
remain within acceptable risk tolerances, and can be governed in a way that
stands up to regulatory and internal scrutiny.
The Model and AI governance executive
at a major European insurance company argued that validation must start with
clarity of purpose.
GenAI systems are often capable of
answering an almost unlimited range of questions, but that theoretical
flexibility is precisely what creates risk.
According to him, institutions must
explicitly define what a model is allowed to do, under what conditions, and
with what tolerance for error. He emphasized that this is especially critical
for externally facing use cases, where even small inaccuracies or so called
hallucinations can undermine customer trust.
“You have to decide where your
hallucination threshold sits and be very deliberate about it,” the executive
said, noting that acceptable tolerance levels may differ sharply between
internal productivity tools and customer facing applications.
Quantifying hallucinations, however,
is far from straightforward. The speaker explained that outputs can vary
significantly depending on prompt phrasing, retrieval methods, and the
structure of underlying documents.
Policies written with complex
language or double negatives can confuse embedding models, forcing institutions
to revisit not just the AI system but the content it consumes.
In this sense, validation becomes a
feedback loop, aimed not only at checking performance but also at improving
inputs so the model can function more reliably.
A fellow panellist stressed that this
mirrors earlier generations of model risk management.
Validation has always required
understanding what a model is supposed to do, identifying key risk drivers, and
testing how it behaves when those drivers are stressed.
In the GenAI context, those risks
include bias, explainability limits, and failure under edge cases. Even if the
core large language model cannot be opened or replicated, components around it
can still be tested in isolation to understand vulnerabilities.
Breaking the system into parts
emerged as a central theme. The governance executive explained that effective
validation involves examining the retrieval augmented generation architecture,
the embeddings, the prompt logic, the data sources, and the outputs separately.
“You may not be able to touch the
core model, but you can still challenge how the system behaves,” the speaker
said.
