All Models Are Wrong
Covid-19 has put “models” in the daily headlines worldwide. Ten different models for the virus were recently cited to project the number of cases and fatalities. There was a wide range to their estimates. It’s reasonable to ask why are there so many different models with such different findings.
We use models for almost everything. Besides the coronavirus, we model all manner of things, such as hurricanes, economic outlook, unemployment levels, investment portfolios, population growth, elections, global warming, and sea level rise. Given our reliance on models, it’s important to understand that models are not reality. They can be useful, but in the end, we all need to remember that they are models – and may be very different from the real world.
A good example of the limitation of models that should be familiar, is human life expectancy. If you have life insurance, you have probably had an estimate for how long you will live. Based on your current age, sex, the age your biologic parents died, where you live, your height and weight, whether you smoke, and your health history, any insurance company will have an estimate for how long you will live. Let’s assume you bought the term life policy when you were forty years old and it said you are projected to live to 76 years of age. The model is giving a statistical probability, but is not saying you will live that long. You could die the day after the model was run or live to be over a hundred. In fact, it’s very unlikely that you would die at exactly 76. Other insurance companies might have their own models coming up with a different estimate. If you were planning your savings and investments for retirement, you might consider the estimate, but you would be foolish to count on it being correct.
Models may be millions of lines of computer code, or they can be a simple Excel type spreadsheet. They are based on assumptions of relationships between data sets. They attempt to convert highly complex multi-variable situations into math. Weather and climate are the most demanding systems to model, with thousands of variables and millions of lines of “code” that run on supercomputers. Those who create models are constantly tweaking them, changing variables, fine tuning formulas, adding another new parameter to consider, or inserting a new row of data to incorporate. The value of models is to see how dependent variables cascade and compound over time. By changing the factors in a model some really useful understandings can emerge that we might not intuitively consider.
In my area of focus – rising sea level – I come across all kinds of models for future ocean height and flooding as the planet continues to warm. Considering many factors, the various models try to estimate the melt rate of the giant ice sheets and glaciers on Antarctica and Greenland, the primary source of future sea level rise. Given that those two places have enough ice potentially to raise global sea level about two hundred feet, it’s an important calculation to have. For example, the most widely regarded climate study says that in a worst case scenario by the year 2100, the upper bound for global average temperature increase is 8.6 degrees Fahrenheit (4.8 Celsius) with global mean sea level being as much as 32 inches (82 cm) higher. Careful reading of the fine print however, explains that the upper end projection includes only nine inches of sea level increase due to melting of the Greenland ice sheet and about six inches from Antarctica. The problem is that no glaciologist or climate expert that I respect believes that if the planet warms that much in the next eighty years, that sea level will only be thirty two inches higher as a worst case scenario. Most would put the figure much higher, perhaps eight to ten feet. The models are good at looking at incremental change, but unable to project abrupt change, such as an ice sheet collapsing.
Many who create complex models are fully aware of their limitations and clearly explain them. But human nature being what it is, we often take the result, without remembering the conditions that go with it. After hundreds of hours of work trying to improve or “perfect” a model, even their creators can be prone to thinking the models represent reality.
When using models let’s be aware that they are striving to calculate highly complex systems that we do not yet fully understand. Where multiple models are available, look at them all to see the range, the zone of convergence, as well as the “outliers” as we do when looking at hurricane forecasts.
The late George Box, a highly respected British statistician, may have said it best:
“All models are wrong, but some are useful”
As we all are persuaded by more and more models on all manner of things, including life and death, let’s remember they are just models. Nonetheless they serve an important purpose to help us peer into the future.