Nate Silver and the Problem of Prediction

This desire for certainty, for prediction, is everywhere in modern society for the observant.

Consistency is a temptress.
Consistency is a temptress. Slaven Vlasic/Getty Images

Wall Street always overflows with Holy Grails—those predictions, strategies, secret formulas, and genius interpretations that promise otherworldly knowledge and riches if you just trust. They are most often delivered in the investment world through a black box—a closed system where the inputs and outputs are known, but the internal analytical workings are left top secret, only for the high priests’ consumption. Black box positioning goes far beyond markets, however. It is not surprising in a modern, interconnected age that when you take a very smart guy, rows of computers, proprietary formulas, and code that only the one smart guy can see, and then add a string of successful forecasts, boom—you end up with a nerdy, made-for-social-media superstar who suddenly makes prediction cool for the proletariat.

Sign Up For Our Daily Newsletter

By clicking submit, you agree to our <a href="http://observermedia.com/terms">terms of service</a> and acknowledge we may use your information to send you emails, product samples, and promotions on this website and other properties. You can opt out anytime.

See all of our newsletters

Nate Silver is that guy. Consider:

  • He successfully called the outcomes in 49 of the 50 states in the 2008 U.S. presidential election.
  • He successfully called the outcomes in 50 of the 50 states in the 2012 U.S. presidential election.

Silver became the go-to smart numbers guy overnight. He went from baseball stats expert to political stats expert. His mathematical model for elections beat political journalists and commentators at their own game—so goes his Moneyball-for-politics narrative. In short order his followers on Twitter surpassed 1 million, his book became a bestseller, and FiveThirtyEight.com became ubiquitous—even offering investing insights such as, “Worried about The Stock Market? Whatever You Do, Don’t Sell.”

Yet for anyone who followed the 2016 presidential race, Silver’s political predictions hit the wall. Over the course of 2016 Silver posted daily election odds that jumped around like a cat on fire—not surprising if your new guru status forces you to offer forecasts every day of an election cycle for over a year.

I acknowledge that for us mere mortals, consistency is a temptress—even for the highly educated and affluent. An associate of mine, an accomplished businessman, and I had lunch a few years back. During our conversation he mentioned that he was caught up in a very well known hedge fund scam (not Madoff). I asked him, before he could explain much, if his returns were positive every month. He replied, “Every month.” Later, I Googled him and found out that he indeed had lost several million dollars.

This desire for certainty, for prediction, is everywhere in modern society for the observant. Sports too? You bet. During a Monday Night Football game, one of the announcers, Ron Jaworski, put numbers and odds in perspective for playing the game of football: “Play calling is about probability, not certainty.”

That sounds so easy to digest. But it’s not. That lack of certainty is so often a trigger for our deepest fears. David Rock had a great piece in Psychology Today nailing the conundrum:

“Some parts of accounting and consulting make their money by helping executives experience a perception of increasing certainty, through strategic planning and ‘forecasting’. While the financial markets of 2008 showed once again that the future is inherently uncertain, the one thing that’s certain is that people will pay lots of money to at least feel less uncertain. That’s because uncertainty feels, to the brain, like a threat to your life.

That’s why Silver became so popular—he eliminated the fear—or at least that was the narrative. But even the guy with all the numbers went away from the numbers. He admitted in his laborious mea culpa, “How I Acted Like a Pundit and Screwed Up on Donald Trump” to not using his statistical models on Donald Trump’s candidacy. He instead used educated guesses, which blew his statistical-model-made-me-so-famous-you-can-now-trust-I-am-not-a-typical-pundit-with-built-in-biases story line out of the water. Worse yet, the way Silver had made his predictions, he could essentially say he was right no matter what happened. For Silver’s followers, his 2016 hedged forecasts, his arguable mathturbation, doesn’t matter:

“Look at Nate’s record. Trump was an outlier.

“There was not enough historical data.”

“He gets most of them right.”

“Nate’s winning percentage is so high.

Those weak retorts illustrate a faulty foundation in Silver’s approach. Michael Mauboussin, one of the great thinkers in the field of decision-making in the face of uncertainty, educates: “What is striking is that the leading thinkers across varied fields—including horse betting, casino gambling, and investing—all emphasize the same point. We call it the Babe Ruth effect: even though Ruth struck out a lot, he was one of baseball’s greatest hitters.”

Said another way: You want to think in terms of expected value. Yet that is not a valuable or useful way to think for a one off election where both sides are literally imagining life and death. But there is no certainty, no guarantee—ever. The world is about surprise.

And Silver leaves out the surprises, the unusual and unexpected. He has no way of predicting or accounting for those. On Trump’s win, Silver said as much, “It’s the most shocking political development of my lifetime.” But no one can predict outliers, so if someone like Silver pretends he can—watch out. Spyros Makridakis, in his famed 1979 paper “Accuracy of Forecasting: An Empirical Investigation,” showed that simple beats complicated and that moving averages beat tortuous econometric routines. Would moving averages also have predicted the two elections that made Silver a household name?

Nonetheless, Silver defenders come back to the 49 out of 50 and 50 out of 50 in 2008 and 2012—proof, they say as they ignore 2016. But in 2008 and 2012, what percentage of those were hard calls? For example, California is blue no matter what. So there is always a huge risk in relying on strategy that gets the easy calls right and punts on the hard ones.

These are not terribly new inconsistencies to ponder. The efficient market theory—the strategy that runs the world’s money—also has no solution for surprise (i.e., black swans). Silver simply illustrates the conundrum across a different discipline, but with the same pressing problem. In summer 2016, Nassim Taleb, the father of the black swan concept, launched a detailed criticism of Silver into the public sphere: “@FiveThirtyEight is showing us a textbook case on how to be totally clueless about probability yet make a business in it.”

The intellectual cage match: Nassim Taleb versus Nate Silver.

Pick your side carefully. It’s life or death.

Michael Covel is the author of five books: including the international bestseller, Trend Following, and his investigative narrative, The Complete TurtleTrader. Michael is also the voice behind Trend Following Radio, the underground alternative hit that has been as high as #2 on iTunes with 5 million listens.

Nate Silver and the Problem of Prediction