This blog series explores the 10 steps of the Model Development Lifecycle so that you understand the components of model development and maintenance as well as where potential sources of model risk can reside. Read the first post here for an introduction to the series and why we are likening this process to the creation of a cookie recipe as a “real-world” example.
We are coming to the end of the Model Development Lifecycle. In this post about the MDLC the focus is on steps seven, eight, and nine: Model Implementation, Model Monitoring, and Scheduled Reviews.
Cookie’s Story: The Recipe Goes Live
Cookie sat back in her chair, clasped her hands, and tapped her index fingers together. One would think pushing a new cookie recipe out to the bakeries would be easy. That’s what the bakeries did—they baked cookies. Yet…
This was a critical recipe to have in place. It ensured Cookie’s Bakery would continue to produce the quality product they were known for despite changes in the quality of the underlying ingredients. While this recipe was only intended to be used during supply chain disruptions, it needed to be implemented into their process now so it would be available.
That meant the cookie production line required retooling. To begin, the dough mixer parameters needed to be set for the new recipe. Also, given the consistency of the dough was softer than normal, the dough feeder and dough-forming machines needed tweaks. And, of course, the gas-fired ovens. The recipe specifications would need to be programmed into them. The downstream equipment—the cooling conveyors, stackers, and packing table—wouldn’t require changes.
So, the technicians had some work to do to implement this new recipe. Cookie wrote a description of the necessary changes and included Gates’s requirement on oven calibration. She then emailed it to her lead technical writer so he could do his magic and make it a useful guide. Nothing half-baked here.
She stood, walked over to her window, and reflected. Quality. Despite one’s best efforts, quality invariably waned. She believed there were more forces in the world that dragged quality down than buoyed it up. Which was why her company’s motto was: quality ain’t easy.
But she could only control what was at her company. Cookie’s Bakery depended on suppliers who in turn depended on farmers and other producers. That meant for her business the quality of the ingredients she received could decline without her knowing. Not just the high-quality ingredients, but also the low-quality ones. Would the new recipe still perform as expected if low-quality ingredients somehow became even lower quality?
What she needed to do was what she did with the other recipes the company used. She needed to monitor the recipe output. Periodically, the head bakers would make cookies using their current inventory and compare it to a baseline.
And that’s where the cookies she stored in the freezer would come in. With those cookies, she would be able to create metrics to which she could compare future cookies. Significant deviations in the metrics would indicate a decrease in the quality of the ingredients her bakeries received.
Cookie walked over to her desk and jotted a note to herself: set up quarterly meetings with head bakers to share analysis. Then she turned and headed into her kitchen. She needed to create the baseline metrics so she could monitor ingredient deterioration.
Which metrics would be relevant? Moisture content, for one. Color, for another. Then there was spread. She should look at tenderness too…
Relating it to the MDLC
The 7th, 8th, and 9th stages of the MDLC involve implementing the model and monitoring its performance.
At its core, model implementation involves integrating the model’s logic into a production system or platform. This is done using software developers who follow the software development lifecycle (SDLC). Once implemented, the model runs on new data and produces output.
What is the Software Development Lifecycle?
The Software Development Lifecycle is a process used to create high-quality software and usually contains the following steps:
- Plan
- Design
- Build
- Test
- Deploy
- Maintain
Model monitoring, which can be paired with scheduled reviews, evaluates how the model continues to perform. Typically, models are run on monthly or quarterly data and their output is compared to what happened in the same period. The model monitoring team uses metrics to determine performance and compares the metrics to thresholds to identify deterioration. Commonly the thresholds used for the model monitoring metrics are defined as part of the model development process.
To make sure her models were valid, Cookie instructed her team to take these actions:
- The bakery technicians would incorporate the recipe into the cookie production line (i.e., model implementation).
- The head bakers would be required to bake the cookies once a quarter (i.e., model monitoring) and then meet to discuss the results (i.e., scheduled reviews). The meetings would cover how the recipes performed based on metrics such as moisture content, color, and spread. Baseline values for the metrics would be derived from the cookies baked during development.
Even the most accurate models eventually become less effective; nothing lasts forever. Our next, and final, post in this series about the Model Development Lifecycle will explain why and when financial institutions need to decommission their financial models to mitigate their model risk.
Jonathan Leonardelli, FRM, Director of Business Analytics for FRG, leads the group responsible for business analytics, statistical modeling and machine learning development, documentation, and training. He has more than 20 years’ experience in the area of financial risk.
RELATED:
What is the Model Development Lifecycle, or, What’s Baking at FRG?
The Model Development Lifecycle (MDLC): Defining the Business and Model Objectives
The Model Development Lifecycle (MDLC): Data Assessment
The Model Development Lifecycle (MDLC): Building and Testing the Model