This blog series explores the 10 steps of the Model Development Lifecycle (MDLC) so that you understand the components of model development and maintenance as well as where potential sources of model risk can reside. Read the first post here for an introduction to the series and why we are likening this process to the creation of a cookie recipe as a “real-world” example.
In this blog about the MDLC the focus is on: Model Validation and Deficiency Assessment.
What is the difference between model validation and model evaluation?
While the two terms are similar in nature, they both serve different functions. In this context, model evaluation is performed by the model developer with the purpose of measuring how well the model performs. Model validation is performed by the model validator with the purpose of ensuring the model is conceptually sound and aligns with business use.
Cookie’s Story: A Second Set of Eyes
“Well, I’ll be …,” whispered I. M. Gates, Lead Recipe Validator, to the computer screen. An email from Cookie, with a recipe attached, had just landed in his inbox. It was rare for her to send a recipe directly to him. Most times they came through her assistants. This implied urgency. He opened the email and read.
When he finished reading, Gates looked up at his whiteboard. It listed other cookie recipes he and his team were testing, along with their status. The Pfeffernusse cookies for Germany were being evaluated for tenderness. The Pizzelles for Italy were being evaluated for flavor after freezing. And, of course, the recipe for Kuih Bangkit for Malaysia was being reviewed for completeness. All those needed to be finalized soon so Cookie’s Bakery could push into new markets. That meant that he’d be the one testing Cookie’s recipe.
Gates straightened his apron and baker’s cap. “Alrighty, let’s do this.”
He printed the email’s attachment—recipe and notes—and then headed to his test kitchen. He spent the rest of the day, and much of the next, testing the recipe.
The first thing he did was carefully review the recipe and notes. He wanted to make sure the recipe and its purpose aligned and that there were no conceptual flaws. Satisfied, he then made a batch following the recipe instructions exactly as they were written. When the cookies finished baking, he sampled a couple and determined they tasted as expected.
Next, Gates addressed the two important assumptions in the recipe: time of baking and temperature. He let the cookies bake at the same temperature but extended the time by 30- second increments until he had added an additional 5 minutes. That last test resulted in cookies more akin to coal than baked goods.
He did similar tests with the temperatures, increasing or decreasing in 1-degree increments until he had added or reduced the temperature by 10 degrees. Anything beyond the 1-degree change resulted in either severely undercooked or overbaked cookies. He jotted these results down in his notebook. He suspected it was the Irish butter—the higher percent of butterfat contributed to a richer flavor but played a bit with the cookies’ spread.
Gates clicked his pen against his teeth. Everything Cookie had wanted from the recipe worked as intended. It was a solid. Well, he couldn’t use that word in the report. He’d have to change it to robust or sound. Regardless, though, she had pulled it off.
He pushed himself away from the counter he was leaning against, hung up his apron and baker’s cap, then washed his hands. However, things will go sideways, he thought, if the oven temperatures aren’t accurate. She needs to make sure the bakeries keep their ovens calibrated.
He hummed the “C is for Cookie” Cookie Monster song as walked to his office to type up his report.
Relating it to the MDLC
The 5th and 6th stages of the MDLC involve an independent review of the models and then communicating any deficiencies.
Why is model validation critical?
Model Validation is one of the key steps in reducing model risk which is defined as “the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports” (SR Letter 11-7).
The model validation process is focused on conceptual soundness. It includes work like:
- reviewing documentation and code
- evaluating the reasons used to select the model methodology and variables
- searching for theoretical errors
- checking the model controls and assumptions.
During the validation process, if any deficiencies are found they are summarized in a report and to decide the next steps. If the deficiencies are severe, the model is rejected and the entire model build process will need to restart.
Gates’ job was to evaluate the recipe and make sure it performed as expected. Actions he took to do this were:
- Reviewed the recipe and notes Cookie provided (i.e., documentation and code review).
- Baked the recipe (i.e., analysis of data, model methodology, output).
- Tested baking temperature and bake time (i.e., evaluated model controls and assumptions)
When Gates finished his validation, he typed up his notes (i.e., validation report) which would include what he did and his concern about the temperature (i.e., model limitations) and how to address it.
After the model has been tested and evaluated for soundness, it is ready to implement into production. Our next blog will focus on how to integrate the model logic on to your platform and monitor its performance.
Jonathan Leonardelli, FRM, Director of Business Analytics for FRG, leads the group responsible for business analytics, statistical modeling and machine learning development, documentation, and training. He has more than 20 years’ experience in the area of financial risk.
RELATED:
What is the Model Development Lifecycle, or, What’s Baking at FRG?
The Model Development Lifecycle (MDLC): Defining the Business and Model Objectives
The Model Development Lifecycle (MDLC): Data Assessment
The Model Development Lifecycle (MDLC): Building and Testing the Model