In this blog of FRG’s blog series on coding best practices to reduce model risk during model development, we focus on the Honed element of MATHS.
To hone a process, skill, or ability, means to make it “perfect or completely suitable for its purpose.” That is the focus of the Honed best practice: making sure the code a model developer creates is completely suitable for the purpose.
For FRG modelers, a honed module is:
- Tested.
- Complete.
- Efficient.
Before discussing these characteristics in more detail, one should first understand when to hone code. Honing should happen after initial development. This is because honing takes time. It’s a rabbit hole that will take you into the land of “What Ifs” and “How Tos”. You don’t want to be landscaping there when a deadline is darkening the horizon.
How does honed code reduce model risk?
Having code that is honed decreases model risk because it:
- Is tested.
- Becomes the “go to” code.
The first effect, as stated in our previous best practice blogs, reduces the chance of using flawed code. The second effect minimizes the need for the model developer to write new code. This, also stated in our previous blogs, reduces the risk of writing conceptually flawed code in the future[2].
Steps in the Honing Process
As mentioned above, a honed module has three characteristics. Updating the code to incorporate these characteristics can happen in any order.
Tested
The most important action to take in the honing process is testing. We discussed the testing of modules in the previous blog but didn’t mention when one should test. Ideally, any time code is modified the model developer should test it (this is a benefit of having a suite of unit tests archived in a code repository). We can summarize the ideal code development process as:
- After the initial build, the developer archives the module in a code repository.
- The code is then tested, fixed if bugs are found, then archived to ensure the fixed code is the most recent version.
- During the honing process, the model developer pulls the most recent version of the code, updates it, then archives it. The updated code then goes through its own testing process.
This is the ideal development process because the model developer archives the code after they modify it. In reality, the code may be archived after testing. Regardless, testing is a key action in this process.
Complete
When first developing a module, a modeler should have a clear idea of its single purpose. For example: “I want this module to pull data” or “I want this module to impose the prepayment business rule.” So, scripts always start with a want.
But sometimes that want grows. One day you may find yourself returning to a piece of code to tweak it to do something else with its main purpose. Then a little while later you return to the code again to add another tweak. If you find yourself doing this then you are performing the second quality of the Honed best practice: making it complete.
This might seem like we are Texas Two Stepping ourselves around the modularized best practice, but we aren’t. We are still embracing the “single purpose” concept; we are just augmenting the module with necessary functionality.
Consider, for example, a module whose purpose is to produce a graph of two variables to show their relationship over time. The “making it complete” process may look like:
Efficient
Efficient code is code that quickly executes.
The model developer will not need to tweak all scripts to make them run faster. Some modules, given their purpose, will run quickly. For example, a script that creates a graph or one that produces a correlation value should finish within milliseconds after you trigger its execution. These are straightforward processes that should not require much data manipulation or calculations.
Other scripts might run slower than expected but don’t warrant code changes for efficiency given the small gain. For example, a script that takes 3 seconds to run might be irritating to wait for, but is it worth the development time to get it down to less than 1 second? Most likely the time invested in speeding up the process will take longer than the efficiency gains one would get over the life of the module.
Then there are the other modules, those developed and tested but chug along like they haven’t a care in the world. In the process of running, these modules not only deplete your time but also your patience. These are the ones that a model developer needs to review for efficiency.
Making a module efficient requires a deep understanding of the programming language the model developer is using. If the code queries databases, making a module efficient will also require knowledge of the database. Some items to consider when reviewing your code to improve efficiency:
- Remove repetitive calculations and operations.
- Keep only necessary variables.
- Check your data join logic.
- Optimize loops and other conditional logic.
There are many more actions a person can take to make their code efficient. These are just a few to get you started. A search online, along with the programming language you are using, should surface multiple tips.
To summarize, the Honed best practice is about moving your module from merely functional to one that satisfies your needs completely. At FRG, a module has been honed when it has been tested, does everything it needs to do, and is efficient.
Jonathan Leonardelli, FRM, Director of Business Analytics for FRG, leads the group responsible for business analytics, statistical modeling and machine learning development, documentation, and training. He has more than 20 years’ experience in the area of financial risk.
[2] There is always a risk, even after testing, that a conceptual error exists in the code. As a result, not having to write new code eliminates the chance the model developer incorporates new “black swan bugs” into the model development process.
RELATED:
Model Risk Management: Reduce your Model Risk through Coding Best Practices
Reducing Model Risk Through Modular Programming