Simulation-based Credit Analytics: Background and the impact of uncertainty on credit data

Updated: Oct 11

Adapting to data limited uncertainty - An alternative approach to Mortgage and Consumer Lending credit loss and performance forecasting

There are times when historical data is not sufficiently predictive. Certainly, the pandemic is an example of a sudden change causing historic credit data to lack predictability. Also, future trends like Environmental, Social, and Governance ("ESG") are expected to have significant impact on lending volumes, credit risk, and profitability. Yet no one knows exactly how and when ESG will impact Mortgage and Consumer Lending businesses. Changes like these are reverberating through our economy with many unknown effects. Change is only accelerating. This uncertainty tests those responsible for forecasting the future. The uncertainty environment challenges those responsible for credit loss estimation and providing accurate inputs to the Allowance for Credit Loss (ACL), loan pricing, loan decisions, and a wide variety of financial services analytical needs.

This is Part 1 of a 3 part series:

((Please click the boxes for the other articles.)

The uncertainty related to sudden and unexpected events have been shown, in part, to be the result of a failure of imagination.

Pearl Harbor (1941)

“We were so busy thinking through some ‘obvious’ Japanese moves that we neglected to hedge against the choice that they actually made. There is a tendency in our planning to confuse the unfamiliar with the improbable.”

Thomas Schelling forward on a book about the Pearl Harbor attack (1)

9/11 (2001)

"….the most important failure [concerning the 9/11 attacks] was one of imagination."

- 9/11 Commission (2)

Financial Crisis (2007)

The 2008 Financial Crisis was caused by “high risk, complex financial products; undisclosed conflicts of interest; the failure of regulators, the credit rating agencies, and the market itself to rein in the excesses of Wall Street.”

– U.S. Senate (3)

Pandemic (2020)

"The coronavirus pandemic is a failure of imagination...It’s not just policy failures that got us to this point. It’s an inability to conceive of planetary threats."

- Stephen Kinzer is a senior fellow at the Watson Institute for International and Public Affairs at Brown University. (4)

Not all events born from uncertainty are the same. For example, the Financial Crisis is different than Pearl Harbor, 9/11, and the Pandemic. Unlike the other unexpected events, the Financial Crisis was caused, in part, by the banking system itself, specifically the Mortgage Finance system. In the run-up to the financial crisis, the ability to anticipate the issues related to poor lending standards was clearly available. However, these lending issues were ignored and the Mortgage Finance system became destructively dysfunctional. I note this because the political will to “bail out the bankers” was clearly much higher during the Pandemic than it was during the Financial Crisis. In fact, the banking system itself was mobilized as a part of the Federal Government's aid distribution system. This resulted from the CARES Act and related fiscal policies.

Be that as it may, all these unexpected events share some common ground. They are all characterized by an anticipatory failure of imagination and all share an outcome marked with significant uncertainty. Could we have anticipated the credit impact of an unexpected event? Possibly – most events born from uncertainty give rise to overlooked prophets. For example:

  • Prior to the Pandemic, Bill Gates described some of the growing risks associated with disease. (5)

  • Prior to the Financial Crisis, Raghuram Rajan, then a senior IMF official, warned of the growing risks in the financial system and proposed risk reduction policies. (6)

However, following an uncertain event, this is mostly hindsight. Ideally, Financial Services companies forecast their credit losses in anticipation of an uncertain environment. For ESG, the need for organizational planning is clear. While there is uncertainty related to how and when ESG changes will impact financial services, there is clear indication that ESG impacts WILL ultimately occur. The ESG train has left the station.

In respect to uncertain events, credit loss estimation is in a challenging phase environment. In a very real sense, it is in a massive phase transition. Metaphorically, this transition can be related to physics, much like we think about matter phase change from ice to water, as heat is added. Think of uncertain events, like ESG or the pandemic, as heat that is driving an abrupt phase change, significantly impacting our credit loss understanding.

Credit loss estimation is also challenged by the nuanced but significant difference between risk and uncertainty. In my experience, credit risk policy and analytics require a VERY high reliance on historical data. Credit models are driven from historical information that, at its essence, describes human behavior. Simply put, the models describe the causal connection between HOW people behaved in the past and WHY they sometimes default on a loan in the future. But this data-rich risk world is periodically upended as a result of a sudden phase change.

The comfortable, data rich risk world is periodically replaced by the data limited uncertainty world. The uncertainty world is where historical data significantly lacks predictive power.

To be fair, the new uncertainty world is knowable, it is just not knowable in advance of the change. There will be some similarities to the old world but pretending the old credit models act as a loss point estimate foundation is a fallacy. Making a standard “On Top Adjustment (OTA)” to existing models should be met with skepticism. (7)

As an example of the credit modeling challenges, especially the use of Machine Learning (ML) and Artificial Intelligence (AI), follow this link to our article, The Inertia-On-Inertia Paradox. This paradox was especially evident in the last financial crisis. Model based inertia was a driver of the Financial Crisis.

The next section presents an approach to simulate the uncertainty world. The new approach a) seeks to adjust the standard thinking of those involved in the credit forecasting process, b) appropriately utilizes existing data and c) engages the experience of multiple expert “bettors.”

Please follow the link at the top of the article for additional sections.

For more information, please contact Definitive Business Solutions, Inc.:

John Sammarco, President |

Jeff Hulett, Executive Vice President |


(1) Roberta Wohlstetter, Pearl Harbor: Warning and Decision, June, 1962

(2) The 9/11 Commission Report at, July 2004

(3) Wall Street and the Financial Crisis: Anatomy of a Financial Collapse, United States Senate Permanent Subcommittee on Investigations, April 2011

(4) Stephen Kinzer, The coronavirus pandemic is a failure of imagination, The Boston Globe, March 2020

(5) Bill Gates, Who Has Warned About Pandemics For Years, On The U.S. Response So Far, NPR, Heidi Glenn, April 2020.

(6) In 2005, Dr. Rajan said the financial system was at risk “of a catastrophic meltdown.”

(7) The subtle difference between risk and uncertainty is manifest when considering OTAs. Generally, making an OTA relates to potential qualitative events that may impact the risk-based expected loss, specific to its standard deviation. Uncertainty, on the other hand, is focused on tail risk, or kurtosis. Thicker tails may cause wildly different loss results, as per the Financial Crisis. Please see our article The numbers don't lie - anticipating risk, managing uncertainty, and making quality decisions for more information.

(8) In general, using independent participants grounded approach to creating simulations is known as the "common task method." Independent participants, utilizing common data sources compete to develop accurate outcomes. While this is not as common in credit loss estimation, it is common in computer science in the development of machine learning algorithms for standard tests or photograph interpretations. Other examples include (a) In 1980, Robert Axelrod, professor of political science at the University of Michigan, held a tournament of various strategies for the prisoner's dilemma. He invited a number of well-known game theorists to submit strategies to be run by computers. (spoiler alert: a very simple "Tit For Tat" strategy won). b) In 2020, after a long-term, longitudinal study about "Fragile Families" was completed, a competition of over 160 teams was held to find the most predictive algorithms to answer key questions concerning teenage children at risk. Interestingly enough, consistent with the game theory example, the simple models did quite well, almost as good as the more complex Machine Learning models. Given the danger of unrevealed overfit, one can argue simple is always better, especially when different algorithms have similar explanatory power. (To wit: Occam’s Razor)

(9) This quote is from the book Noise: A Flaw in Human Judgment by Daniel Kahneman, Olivier Sibony, Cass R. Sunstein.

(10) Troy Segal, The Five C's Of Credit, March, 2020

(11) Thomas L. Saaty (1982) Decision Making for Leaders: The Analytic Hierarchy Process for Decisions in a Complex World

(12) Definitive Business Solutions, Inc. designed and integrated a bespoke solver engine, utilizing a number of solver programing code sets and data transmission capabilities, uniquely needed for the AHP and cloud environment.

34 views0 comments