Over the past year, rarely has a month gone by without someone asking me if polling is broken. That’s a question that, until recently, I had not been asked for most of my two decades working as a pollster. But it is not at all surprising this question has come up now, as so many recently-released public polls have been flat-out wrong.
The answer is straightforward. The problem is largely isolated to a specific low-cost polling methodology that has proliferated in recent years: robo polls, also called automated or IVR (interactive voice response), which use a recorded voice to ask respondents to answer questions by punching a number on their phone.
The polling industry’s gold standard methodology (professional interviewers calling voters on landlines and cellphones) continues to produce highly accurate results.
Polling grades: A’s and F’s
A review of the three off-year statewide elections held in 2017 (gubernatorial elections in Virginia and New Jersey, and the special Senate election in Alabama) clearly illustrates this dynamic.
Of all public polls conducted within one month of each of these elections, most robo polls showed the ultimate losing candidate ahead, while almost all polls using the gold standard methodology showed the ultimate winner ahead.
Of polls conducted within one month of each election, fully 86% of gold standard polls showed the ultimate winner leading. Just 39% of robo polls did so.
While we can’t say for sure why robo polls performed so much worse than live interviewer polls, the most likely reason is they are prohibited by law from calling cellphones. More than half (52%) of American households only have cellphones, which means automated polls miss them entirely. Add in the fact that cell-only households tend to be younger, less affluent, and more diverse (all factors that tie closely to voting behavior) and you begin to see some of the major obstacles for robo polls.
The Alabama special Senate election brings this point home. There were 13 automated polls conducted within one month of the election, and just three – less than a quarter – showed eventual winner Doug Jones with a lead. Meanwhile, three of the four gold standard polls conducted in Alabama showed Jones leading, and the fourth showed a tie. Exit poll results suggest Jones performed particularly well among younger voters and blacks – exactly the voters most likely to live in cell-only households that robo polls are prohibited from calling.
Anderson Robbins polling with Fox News
Anderson Robbins is the Democratic partner on a bipartisan polling team for Fox News Channel. Along with our Republican partner, Daron Shaw, we had the opportunity to poll all three 2017 races. All of our polling utilizes gold standard methodologies and all of our 2017 polls showed the eventual winner leading.
Our New Jersey poll showed Democrat Phil Murphy leading by the exact 14-point margin he won with. In Virginia, our poll had Democrat Ralph Northam leading by five points. He won by nine (just two other polls were closer to the final margin).
Alabama was a unique challenge. There were no comparable past races to help inform turnout expectations. It was a special election held in December with a Republican candidate who was already extremely polarizing before the accusations of sexual misconduct.
Perhaps as a result of these challenges, hardly any major news outlets polled the Alabama race. Fox News took the plunge and released two likely voter polls in the month prior to the election. Both showed Democrat Doug Jones leading with 50% of the vote, the exact number he achieved on Election Day.
In our final poll, Republican Roy Moore’s support was just 40%, while 2% said they’d write in a candidate, and 8% were undecided. The demographics of the undecided voters in our poll suggested they would not be voting for Jones, so the question was: Would they come out for Moore or stay home?
The election result (Jones 50%, Moore 48%, other 2%) suggests late deciders broke decisively for Moore. So, while our Alabama poll didn’t nail the margin, our results did reflect the reality of the closing five days of the race; Jones voters had made up their mind and were not wavering, but some Moore voters were unsure until the last minute that they could support a uniquely damaged Republican candidate.
A Look Ahead
Assessing polls’ accuracy by looking only at how well they predicted wins and losses is admittedly simplistic, but concerns about predictions are also the reason why people are now asking if polling is broken. When polls lead people to think the wrong candidate is going to win, that’s what sticks with them after the election: the polls said one thing; voters said another.
The widespread misunderstanding of polling in the 2016 presidential race – in which the national polls were actually highly accurate in predicting a Hillary Clinton popular vote win – compounds the issue.
However, the good news from our 2017 polling review is that traditional, live-interviewer polling remains highly accurate and is very much alive and well, even though the prevalence of automated polls probably led some to conclude otherwise.
While there’s a time and a place for robo polls, 2017 was another reminder that for political polling there’s no substitute for high-quality data paired with careful, thoughtful analysis. As we head into what will surely be an eventful 2018, we look forward to providing our clients the reliable data that is the bedrock of good strategic decision-making.