A new dual backtesting framework developed by Krishan Kumar Sharma aims to solve a longstanding weakness in bank capital modeling by isolating scenario-driven forecasting errors from downstream model inaccuracies. The methodology could improve capital eff
A new
diagnostic construction extends existing backtesting literature into the domain
of scenario-driven regulatory capital models - with the potential, according to
the author, to improve the efficiency of regulatory capital across the US
banking system.
In the
following article, we explore a methodological development that addresses a
challenge identified by supervisors in large-bank capital modeling: the
difficulty of separating, measuring, and governing errors that compound as
macroeconomic scenarios cascade through downstream revenue and loss models.
The framework,
developed by Krishan Kumar Sharma, a model risk management leader and senior
vice president at Citigroup in New York, and published in the Journal of
Risk Model Validation, introduces a dual backtesting framework - a
diagnostic construction that extends existing backtesting literature - and
applies it to the problem of disentangling forecasting error from downstream
model error in regulatory capital exercises.
The
Problem: A Persistent Blind Spot in Scenario-Driven Capital Modeling
Every major US
capital regime - CCAR, CECL, IFRS 9, and ICAAP - rests on a multi-stage
forecasting cascade.
External
macroeconomic scenarios (from Blue Chip, Consensus Economics, or regulatory
narratives) are expanded into regional and sector-specific macroeconomic
variables, transformed (logs, quarter-over-quarter growth rates, lagged
specifications), and finally fed into downstream Pre-Provision Net Revenue
(PPNR) and Credit Loss Forecasting (CLF) models that drive reserves and
regulatory capital.
The industry
has long understood, in the abstract, that errors introduced at any stage do
not simply add up - they may compound nonlinearly as they propagate through
convex PD curves, lifetime loss projections, and capital adequacy calculations.
What has been
less developed is a practical and widely implemented method to isolate which
layer of the cascade contributes to which portion of the error.
Traditional
validation practice, as documented in supervisory guidance and in the academic
literature, typically treats each model in isolation. Validators backtest
downstream PPNR and CLF models against actual macroeconomic outcomes and may
not consistently challenge the scenario-generation and MEV-expansion layers
upstream.
The
consequence is a structural limitation: institutions may not be able to
determine whether a model is miscalibrated or whether it is being influenced by
a flawed scenario.
The industry’s
response has often included the use of overlays, qualitative reserves, and
capital buffers to absorb uncertainty that existing validation approaches have
not been designed to isolate.
The cost of
this conservatism, borne by shareholders and potentially affecting lending
capacity, can be difficult to quantify precisely.
The
Innovation: A Two-Stage Diagnostic Construction
Sharma’s
contribution is a unified two-stage backtesting protocol that, taken together,
enables the scenario-layer component of nested model error to be estimated
separately:
Single-Blind
Backtesting (SBBT). The production PPNR and CLF models are run on actual,
realized historical macroeconomic paths - not on forecasts.
This isolates
the intrinsic accuracy of the downstream models themselves, excluding noise
contributed by the scenario layer. It addresses the question: if the bank had
perfect foresight into macroeconomic conditions, how accurate would its models
be?
Double-Blind
Backtesting (DBBT). The full production environment is reconstructed as of a
past point in time - using only the information, scenario outputs, and model
versions that were in place - and the entire cascade is executed end-to-end
against realized outcomes.
This captures
the full chain of nested errors from scenario generation through to reserves
and capital.
The framework
derives an estimate of scenario-driven error by comparing the results of DBBT
and SBBT. This enables a decomposition of what would otherwise appear as a
combined error term into two components: scenario-related and model-related.
This
decomposition is consistent with the “use test” principle articulated in the
Federal Reserve’s SR 26-2 Supervisory
Guidance on Model Risk Management, which requires models to be validated in the
context of their intended business use.
In the
peer-reviewed paper, the framework is demonstrated through a full
implementation example with quantified error metrics, including Cumulative
Percentage Error, Mean Absolute Percentage Error, and Percentage Error at each
stage.
