Insights into cracking model risk for prepayment models

Within the field of financial risk management, professionals strive to develop models to tackle the complexities in the financial domain. However, due to the ever-changing nature of financial variables, models only capture reality to a certain extent. Therefore, model risk - the potential loss a business could suffer due to an inaccurate model or incorrect use of a model - is a pressing concern. This article explores model risk in prepayment models, analyzing various approaches to quantify and forecast this risk. 

There are numerous examples where model risk has not been properly accounted for, resulting in significant losses. For example, Long-Term Capital Management was a hedge fund that went bankrupt in the late 1990s because its model was never stress-tested for extreme market conditions. Similarly, in 2012, JP Morgan experienced a $6 billion loss and $920 million in fines due to flaws in its new value-at-risk model known as the ‘London Whale Trade’.  

Despite these prominent failures, and the requirements of CRD IV Article 85 for institutions to develop policies and processes for managing model risk,¹ the quantification and forecasting of model risk has not been extensively covered in academic literature. This leaves a significant gap in the general understanding and ability to manage this risk. Adequate model risk management allows for optimized capital allocation, reduced risk-related losses, and a strengthened risk culture.  

This article delves into model risk in prepayment models, examining different methods to quantify and predict this risk. The objective is to compare different approaches, highlighting their strengths and weaknesses.  

Definition of Model Risk

Generally, model risk can be assessed using a bottom-up approach by analyzing individual model components, assumptions, and inputs for errors, or by using a top-down approach by evaluating the overall impact of model inaccuracies on broader financial outcomes. In the context of prepayments, this article adopts a bottom-up approach by using model error as a proxy for model risk, allowing for a quantifiable measure of this risk. Model error is the difference between the modelled prepayment rate and the actual prepayment rate. Model error occurs at an individual level when a prepayment model predicts a prepayment that does not happen, and vice versa. However, banks are more interested in model error at the portfolio level. A statistic often used by banks is the Single Monthly Mortality (SMM). The SMM is the monthly percentage of prepayments and can be calculated by dividing the amount of prepayments for a given month by the total amount of mortgages outstanding. 

Using the SMM, we can define and calculate the model error as the difference between the predicted SMM and the actual SMM: 

The European Banking Authority (EBA) requires financial institutions when calculating valuation model risk to set aside enough funds to be 90% confident that they can exit a position at the time of the assessment. Consequently, banks are concerned with the top 5% and lowest 5% of the model risk distribution (EBA, 2016, 2015). ² Thus, banks are interested in the distribution of the model error as defined above, aiming to ensure they allocate the capital optimally for model risk in prepayment models.  

Approaches to Forecasting Model Risk 

By using model error as a proxy for model risk, we can leverage historical model errors to forecast future errors through time-series modelling. In this article, we explore three methods: the simple approach, the auto-regressive approach, and the machine learning challenger model approach.

Simple Approach

The first method proposed to forecast the expected value, and the variance of the model errors is the simple approach. It is the most straightforward way to quantify and predict model risk by analyzing the mean and standard deviation of the model errors. The model itself causes minimal uncertainty, as there are just two parameters which have to be estimated, namely the intercept and the standard deviation.

The disadvantage of the simple approach is that it is time-invariant. Consequently, even in extreme conditions, the expected value and the variance of model errors remain constant over time.

Auto-Regressive Approach

The second approach to forecast the model errors of a prepayment model is the auto-regressive approach. Specifically, this approach utilizes an AR(1) model, which forecasts the model errors by leveraging their lagged values. The advantage of the auto-regressive approach is that it takes into account the dynamics of the historical model errors when forecasting them, making it more advanced than the simple approach.

The disadvantage of the auto-regressive approach is that it always lags and that it does not take into account the current status of the economy. For example, an increase in the interest rate by 200 basis points is expected to lead to a higher model error, while the auto-regressive approach is likely to forecast this increase in model error one month later.

Machine Learning Challenger Model Approach                           

The third approach to forecast the model errors involves incorporating a Machine Learning (ML) challenger model. In this article, we use an Artificial Neural Network (ANN). This ML challenger model can be more sophisticated than the production model, as its primary focus is on predictive accuracy rather than interpretability. This approach uses risk measures to compare the production model with a more advanced challenger model. A new variable is defined as the difference between the production model and the challenger model.

Similar to the above approaches, the expected value of the model errors is forecasted by estimating the intercept, the parameter of the new variable, and the standard deviation. A forecast can be made and the difference between the production model and ML challenger model can be used as a proxy for future model risk.

The advantage of using the ML challenger model approach is that it is forward looking. This forward-looking method allows for reasonable estimates under both normal and extreme conditions, making it a reliable proxy for future model risk. In addition, when there are complex non-linear relationships between an independent variable and the prepayment rate, an ML challenger can be more accurate. Its complexity allows it to predict significant impacts better than a simpler, more interpretable production model. Consequently, employing an ML challenger model approach could effectively estimate model risk during substantial market changes.

A disadvantage of the machine learning approach is its complexity and lack of interpretability. Additionally, developing and maintaining these models often requires significant time, computational resources, and specialized expertise.

Conclusion 

The various methods to estimate model risk are compared in a simulation study. The ML challenger model approach stands out as the most effective method for predicting model errors, offering increased accuracy in both normal and extreme conditions. Both the simple and challenger model approach effectively predicts the variability of model errors, but the challenger model approach achieves a smaller standard deviation. In scenarios involving extreme interest rate changes, only the challenger model approach delivers reasonable estimates, highlighting its robustness. Therefore, the challenger model approach is the preferred choice for predicting model error under both normal and extreme conditions.

Ultimately, the optimal approach should align with the bank’s risk appetite, operational capabilities, and overall risk management framework. Zanders, with its extensive expertise in financial risk management, including multiple high-profile projects related to prepayments at G-SIBs as well as mid-size banks, can provide comprehensive support in navigating these challenges. See our expertise here.

Ready to take your IRRBB strategy to the next level?

Zanders is an expert on IRRBB-related topics. We enable banks to achieve both regulatory compliance and strategic risk goals by offering support from strategy to implementation. This includes risk identification, formulating a risk strategy, setting up an IRRBB governance and framework, and policy or risk appetite statements. Moreover, we have an extensive track record in IRRBB [EU1] and behavioral models such as prepayment models, hedging strategies, and calculating risk metrics, both from model development and model validation perspectives.

Contact our experts today to discover how Zanders can help you transform risk management into a competitive advantage. Reach out to: Jaap Karelse, Erik Vijlbrief, Petra van Meel, or Martijn Wycisk to start your journey toward financial resilience.