Nate Silver, The Signal and the Noise
“Statistical techniques, however, are subject to diminishing returns. Hurricanes are not exactly
rare, but severe storms hit the United States
perhaps once every year on average. Whenever
you have a large number of candidate variables
applied to a rarely occurring phenomenon,
there is the risk of overfitting your model and
mistaking the noise in the past data for a signal.
There is an alternative, however, when you
have some knowledge of the structure behind
the system. This second type of model essentially creates a simulation of the physical
mechanics of some portion of the universe. It
takes much more work to build than a purely
statistical method and requires a more solid
understanding of the root causes of the phenomenon.”
Jeff’s thinking….
Thinking on statistical models v simulation.
This is the kind of model I built for forecasting the appropriate risk premia used in loan acceptance and pricing decisions.
The model simulated the loan business, with statistical based inputs for consumer behavior. The traditional statistical approach was still used, but used for simulation inputs.
This model was harder and more complex, but also simple in that it was easy to understand the real world context of the model and the dynamics between the model components. Also, I'm sure there were plenty of SIFs assumptions that hopefully ere irrelevant or minimally irrelevant.
My biggest concern about simulation is, 1) how do you know you got the model right and 2) how do you account for the uncertainty that can require a new simulation because the world has changed? Especially in the case of economics.
1) can be handled through challenge but 2) is different. It may suffer from the same "inertia on inertia paradox" as do statistical models.
At least in the physical sciences, there are physical laws that can be counted on not to change. The last financial crisis demonstrated that economics does not have the same sort of stable laws. Especially in uncertain (wild) environments defined by high inertia, high entropy ...where independence is harder to achieve and kurtosis is >3.
