The Federal Reserve and the OCC define model risk as “the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports.”[1] Statistical models are the core of stress testing and credit analysis, but banks are increasingly using them in strategic planning. And the more banks integrate model outputs into their decision making, the greater their exposure to model risk.
Regulators have singled out model risk for supervisory attention;[2] managers who have primary responsibility for their bank’s model development and implementation processes should be no less vigilant. This article summarizes the principles and procedures we follow to mitigate model risk on behalf of our clients.
The first source of model risk is basing decisions on incorrect output. Sound judgment in the design stage and procedural discipline in the development phase are the best defenses against this eventuality. The key steps in designing a model to meet a given business need are determining the approach, settling on the model structure, and articulating the assumptions.
- Selecting the approach means choosing the optimal level of granularity (for example, should the model be built at the loan or segment level).
- Deciding on the structure means identifying the most suitable quantitative techniques (for example, should a decision tree, multinomial logistic, or deep learning model be used).
- Stating the assumptions means describing both those that are related to the model structure (for instance, distribution of error terms) and those pertaining to the methodology (such as default expectations and the persistence of historical relationships over the forecast horizon).
Once the model is defined, the developers can progressively refine the model, critically subjecting it to rounds of robust testing both in and out of sample. They will make further adjustments until the model reliably produces plausible results.
Additionally, independent model validation teams provide a second opinion on the efficacy of the model. Further model refinement might be required. This helps to reduce the risk of confirmation bias on the part of the model developer.
This iterative design, development, and validation process reduces the first kind of risk by improving the likelihood that the final version will give decision makers solid information.
The second kind of model risk, misusing the outputs, can be addressed in the implementation phase. Risk managers learned the hard way in the financial crisis of 2007-2008 that it is vitally important for decision makers to understand—not just intellectually but viscerally—that mathematical modeling is an art and models are subject to limitations. The future may be unlike the past. Understanding the limitations can help reduce the “unknown unknowns” and inhibit the misuse of model outputs.
Being aware of the potential for model risk is the first step. Acting to reduce it is the second. What hedges can you put in place to mitigate the risk?
First, design, develop, and test models in an open environment which welcomes objective opinions and rewards critical thinking. Give yourself enough time to complete multiple cycles of the process to refine the model.
Second, describe each model’s inherent limitations, as well as the underlying assumptions and design choices, in plain language that makes sense to business executives and risk managers who may not be quantitatively or technologically sophisticated.
Finally, consider engaging an independent third party with the expertise to review your model documentation, audit your modeling process, and validate your models.
For information on how FRG can help you defend your firm against model risk, please click here.
[1] Federal Reserve and OCC, “Supervisory Guidance on Model Risk Management,” Attachment to SR Letter 11-07 (April 4, 2011), page 3. Emphasis added.
[2] See for example the Federal Reserve’s SR letters 15-8 and 12-17.