This article examines a new diagnostic framework designed to improve transparency and accuracy within regulatory capital forecasting models. Based on work by Krishan Kumar Sharma, the framework separates scenario-generation errors from downstream model er
A new diagnostic framework is extending existing backtesting practices into scenario-driven regulatory capital models, offering a structured approach to measuring how errors propagate through forecasting systems and potentially improving capital efficiency across the banking sector.
The framework, developed by Krishan Kumar Sharma and published in the Journal of Risk Model Validation, addresses a persistent challenge identified by supervisors: the difficulty of isolating and governing errors that accumulate as macroeconomic scenarios flow through downstream revenue and loss models.
Modern capital regimes, including CCAR, CECL, IFRS 9, and ICAAP, rely on multi-stage forecasting processes. Macroeconomic scenarios are expanded into granular variables and fed into models that determine revenues, losses, reserves, and ultimately regulatory capital.
While it is widely understood that errors can compound through this cascade, institutions have historically lacked a practical way to identify which stage is responsible for specific inaccuracies.
Traditional validation approaches tend to assess models individually, often focusing on downstream outputs without challenging upstream scenario generation.
This creates a structural limitation, where institutions cannot easily distinguish between model miscalibration and weaknesses in the underlying scenarios.
In response, firms have often relied on conservative overlays and buffers to manage uncertainty, though the precise cost of this approach has been difficult to quantify.
The proposed framework introduces a two-stage diagnostic process designed to separate these effects. Single-Blind Backtesting evaluates downstream models using actual historical macroeconomic outcomes, isolating their intrinsic accuracy.
Double-Blind Backtesting reconstructs the full production environment at a prior point in time and runs the entire forecasting chain using only information available at that moment, capturing the combined impact of all model layers.
By comparing the results of these two approaches, institutions can estimate the portion of error attributable specifically to scenario generation. This creates a clearer basis for targeted model improvements, rather than relying on broad adjustments applied across the system.
The methodology aligns with supervisory expectations, including the “use test” principle outlined in Federal Reserve guidance, which requires models to be validated in the context of their intended application.
