One of the challenges actuaries regularly face come from management or customers who seek an answer – not just any answer, mind you, but The One True and Infallible Answer, a particular number that one’s hat can be hung on.
That sort of demand makes it a good thing that recent developments in computing technology and modeling algorithms make it possible to generate a number, with quite a bit of impressive-looking analysis and number-crunching behind it.
There is a problem, however. The One True Answer is rarely just a number. It’s a range, couched in caveats on potential sources of uncertainty, and subject to the chance that assumptions made in modeling and number-crunching could be wrong.
That uncertainty doesn’t necessarily damn the model’s output, mind you. However, it’s important to remember that the model provides just one piece of information. The possible other outcomes need to be considered. And, when conventional wisdom or common sense contradict model output, there’s a healthy opportunity to review what we know, what the shortcomings of limited knowledge might be.
Sometimes it’s OK to discount, or even discard, the output of the black box when there’s reason to believe it’s not quite right. Just document the phenomenon, periodically review the deviations to ensure they’re honest and not a reflection of bias, and attempt to retrain the tool given what we’ve learned about its shortcomings.
Sadly, in management’s quest for The One True Answer, such deviation and retraining can be forgotten. Managers like their firm, seemingly infallible numbers.
A taste of that phenomenon appeared yesterday in a Wall Street Journal article discussing criticism of property/casualty catastrophe modeling (subscriber link):
Perhaps the most prominent critic to surface is Karen Clark, an economist who founded one of the first cat-modeling firms two decades ago. Today, she warns about the programs’ misapplication.
After Katrina, she attended insurance-company meetings to discuss "what went wrong" and concluded that there were more problems with how insurers were using the models than with the models themselves.
Companies that rely too heavily on cat-model data "are subjecting their businesses and their customers to the volatility of computer models," says Ms. Clark, who now runs a Boston cat-model consulting business. "The models are being used as if they produce definitive answers rather than uncertain estimates." Ms. Clark says she advises clients to use them in conjunction with other factors, such as broad historical data.
To be sure, insurers themselves are facing higher rates from the reinsurance companies that backstop their claims. The reinsurers, and the financial ratings agencies that assess the health of carriers, are also using the controversial newer models.
It’s thinking about debates like this which makes my job fun. 
And, before anyone thinks otherwise…no, this doesn’t mean Florida politicians are correct. Providing cat wind cover in Florida is risky and expensive even before model outputs are grossed up for possible short-term climate phenomena and regardless of anyone’s overreliance on the models.
