For years, we’ve been told that polling is sacrosanct. Polling predicts — until it doesn’t. What we saw happen this election cycle is the limitation of polling. The media and pollsters all predicted Hillary Clinton would win. Donald Trump won. On Election Day, The New York Times placed Clinton with an 85 percent chance of winning. Even Steve Schmidt, Republican strategist, predicted a Clinton outcome three weeks before the election. “I think she is trending over 400 [Electoral Votes],” he said on Morning Joe.
Then it all shifted in an overnight instant. Trump emerged the winner, shocking nearly everyone. What happened?
Let’s first take a look back to the year 2008, which was the most accurate presidential election for polling in decades, with 92 percent of pollsters correctly picking the winner according to fivethirtyeight.com. It’s been a mere eight years since the “good old days” when polling was accurate, and social media was benign.
Barack Obama won in 2008 around the slogan of “Change we can believe in.” He embodied something different for the country, which needed change. Then, after eight years, he became the norm and people once again wanted something different. There are several reasons for this. The simplest is that many feel they didn’t get to participate in the “change” that occurred—they were left out. This election, like 2008, was not about policy or experience. Obama had no executive experience in 2008, similar to how Trump has no formal political experience. The 2008 election, like 2016, was about feelings. Voters chose who they felt reflected their own ideals. Trump saw this and leveraged it. He positioned himself as the outsider who will enact change.
According to Henry De Sio, chief operating officer for Obama’s successful 2008 campaign, a big part of the issue in this election was the failure of traditional systems to account for the new national identity based on specific values, a powerful driver of voting behavior:
“Back when there were some two-dozen hopefuls at the start of the 2016 presidential primaries, I believed the edge would go to the candidate who most closely identified as the ‘changemaker’ in the field. That’s because changemaker qualities—an innovative mind, a service heart, an entrepreneurial spirit, and a collaborative outlook—embody the new bedrock values running through the whole electorate. We see these attributes in ourselves, and they are central to the national identity that was validated with Barack Obama’s election in ’08,” De Sio noted. “The 2016 contest was confused around this point. Pollsters didn’t account for the changemaker effect in their surveys…Trump did run an unconventional campaign, however, which decisively differentiated him in one regard from all the other candidates. This likely ticked a box for many voters searching for the changemaker in the race. But the conventional measurements simply did not pick up on these undercurrents.”
To learn a valuable lesson about polling and predictions from the November election, the first thing to examine is where errors can occur in polling:
Margin of Error Every poll has a margin of error. This is combination of two factors. First is sampling error, which provides a number representing how close an estimate should be to the real value, based upon the size of the sample. Second is non-sampling error, a combination of factors that could also influence the accuracy of an estimate. The sampling error is easy to compute, but the non-sampling error is tricky to estimate because so many factors can affect the accuracy of an estimate. As a result, most pollsters simply report the sampling error as the “margin of error,” ignoring all of the other sources of error, and leading to a high likelihood that unusual factors will have an impact on voting.
The sampling error for political polling tends to be around plus or minus 3 percent. The typical sample size for political surveys is about 1000 voters. In a normal predictive environment, this has some accuracy. However, given how rabid our election season was, many people simply didn’t want to acknowledge their support for Trump it publicly. These individuals may have been undecided voters who voted for Trump at the end, people who wanted to vote for him all around and be discreet about it, or people who would have normally voted Democrat, but didn’t show up to vote, in protest or in apathy. Voter turnout is difficult to predict. In a normal election, people are more likely to follow patterns. This election was emotionally charged, and people broke the expected frame. Voter sentiments that were not captured in polling increased the non-sampling error and were ultimately revealed only at the voting booths.
Likelihood of Voting Every poll response receives a weight reflecting the likelihood that the person responding will vote. Since the best predictor of future behavior is usually past behavior, people who voted in the last Presidential election were usually weighted as being more likely to vote than those who did not. But this year, the “enthusiasm gap” resulted in a high number of disaffected, potential Clinton supporters not voting—especially across the rust belt—while a larger number of people who did not vote in the last election made the effort to vote this year.
As odd as this may sound, people heavily responded to positive messaging framed to them personally. Trump did this and he got the turnout where he needed it most. Trump won in the states that needed hope. Trump, with all of his flaws and antics, offered hope in the easy-to-grasp slogan “Make America Great Again,” and hammered the point home to people who needed inspiration. His main message was simple and positive. Clinton, meanwhile, did not inspire with her messaging. She used more inclusive language, “Stronger together,” but her data to back it up was akin to a continuation of a program that did not deliver hope to these same people over the last four years.
Voter Sampling The method and nature of the sample is important. Some surveys use mail, others internet, landlines, or cell phones. Mixed-mode or mixed channel surveys yield a broader set of results that can be weighed and combined. For example, telephone polls made over landlines disproportionately favor older voters, while younger voters are more likely to respond to mobile phone polls. Balancing the results of multiple poll methods will have a better outcome than any one by itself. To understand what the results really mean, it is important to match the voter pool sampled with demographics, and then balance the pool to reflect the likelihood of turnout by demographic segment. This often is not disclosed in polling results—mainly because it is complicated and detailed math, and statistics don’t make for good television. One survey can inform the population, which can be skewed as a result of who it was targeting.
At the end of the day, data-driven tools are superior forecasting tools than even the most enlightened gut-feeling. There are startup research companies that use innovative approaches to yield more accurate opinion gathering and predictions. Let’s look at several that beat the media pundits at their own game, and how we can learn from the experience.
In February, the Silicon Valley firm 1World used a Triangulation Method to predict—better than any other group in the country—the Republican primary results in South Carolina. 1World correctly picked the order of candidate placement as well as the exact percent of vote for the winner, Trump.
Additionally, 1World predicted the general election popular vote within a 0.5 percent for both Clinton (47.7 percent predicted, 48.1 percent actual) and Trump (46.1 percent predicted, 46.4 percent actual); again what is notable here is that no other traditional polling firm in the world got closer on this prediction.
So how did 1World do it?
The 1World triangulation method started with analysis of polling data published by other organizations. About a dozen polls for each election were entered into a database that weighted the results according to the sample size, date of the poll and other factors, yielding a combined estimate for each candidate. These results typically reflected a number of undecided voters.
“A key component of 1World’s accuracy is that our polls consider a wider variety of factors than most pollsters normally consider,” said Brad Kayton, COO and GM of Data Analytics of 1World. “Our triangulation method includes the key ability to do spot opinion gathering across a network of digital properties and segment the data based on location and sentiment of where the user came from.”
Another startup takes a completely different view. Instead of transforming polling, they look to monitor and “listen” to social media. “We found that social media was telling a powerful story that was different than what mainstream media reported in the polls. We crawled the top 2,000 videos of the 2016 Election across both camps and analyzed over 2.5M comments,” said Dan Goikhman, CEO of Unreel.me. “It was obvious that Trump was great at getting people engaged, regardless of whether the conversation was positive or negative. Trump videos had 20X the reactions to Clinton videos. Trump owned people’s attention. The lesson to be learned from the election is to look for share of voice.”
A third startup’s approach is to eliminate the opinion of polling and social media altogether, and instead to follow the trail of money. Maxim Lott and John Stossel launched ElectionBettingOdds.com, which analyzes the cash-money betting odds at online betting sites to predict elections. “People should look at betting instead, since it’s the best predictor,” said Lott. If there are any better predictors of the future, you could make money betting based on them.”
Betting markets failed to predict Trump and Brexit initially, but then got it right before networks did; bettors gave Trump a 90 percent chance at 11 p.m. on November 8. Historically, studies have found bettors’ predictions to be more accurate than polls; an example of a better prediction is how bettors correctly predicted Clinton’s win over Sanders in the primary, consistently putting his odds very low, at less than 20 percent.
Another powerful technique is running a counter-poll. A counter-poll approach is a psychological technique to guide people to acknowledge what they are thinking indirectly. This is a great way to validate assumptions. For example, in a situation where voters are likely to hide information, ask respondents who they think their neighbors are voting for and why. This will often lead to transference where voters will project their own ideas onto their neighbors. In a situation where a lot of voters didn’t want to acknowledge they were voting for Trump, this can be telling. Make sure the results of this counter-poll match your statistically analyzed, self-reported opinion poll. If they match, stand by the winner.
Kellyanne Conway, Trump’s campaign manager and a long-time professional pollster, used this indirect approach to get at the ability to win the rust belt states, and achieve “the inside flush” to get above 270 electoral college votes, while other pollsters missed it. Most of the media was actually calling that the Trump campaign was going to lose in the closing days.
How can you predict the future? Utilize multiple systems with checks and balances.
Start with making sure your polling is across multiple channels, and that your demographics reach an accurate sample of the population. Match your channel of collection (i.e, cell phones, internet, mail) with historical turnout data broken down by demographics. This way you can accurately measure who is likely to turn out by demographics in each location. This is more valuable than generalized data. Next, pay attention to what is going on in social media and the nature of the conversation. Look at whose comments are most engaged. This will be telling to see whose message is resonating. Finally run a quick counter-poll, and check-in at your local bookmaker. Then blend what you learn and make a judgment. If they all match, you have a winner.
The predictions sector will change over the next election cycle more than we’ve seen in the last 50 years. If 2008 was the good old days of polling and predicting, today’s world of interfacing with the electorate takes you down the dark and windy path of social media ranting, fake news, and the gambler’s den to grope for answers. The traditional system is broken, and research methodologies haven’t kept up with current times. Technology, including harnessing social media channels, and innovative analytical approaches, can offer us the salvation from the polling issues we just saw in the 2016 election. Within every problem lies an opportunity, and with so much at stake the one thing you can surely predict is that smart minds will be at work to provide better and effective solutions to opinion gathering. Expect to see a rush of entrepreneurs entering the space with new methods, technologies and tools, and ultimately making a claim that they can predict the future.
Disclosure: Donald Trump is the father-in-law of Jared Kushner, the publisher of Observer Media, and the author is an advisor to 1world and Unreel.
Richard Hecker is a global entrepreneur, political organizer and investor. He is the CEO of Traction + Scale, an incubator and investment firm that builds businesses to make people’s lives easier. He has lectured at Parsons, Columbia, and NYU on entrepreneurship and technology.
