A new, highly anticipated report from the leading association of pollsters confirms just how wrong the 2020 election polls were. But nine months after that closer-than-expected contest, the people asking why are still looking for answers.
National surveys of the 2020 presidential contest were the least accurate in 40 years, while the state polls were the worst in at least two decades, according to the new, comprehensive report from the American Association for Public Opinion Research.
But unlike 2016, when pollsters could pinpoint factors like the education divide as reasons they underestimated Donald Trump and offer specific recommendations to fix the problem, the authors of the new American Association for Public Opinion Research report couldn’t put their finger on the exact problem they face now. Instead, they’ve stuck to rejecting the idea that they made the same mistakes as before, while pointing to possible new reasons for inaccuracy.
“We could rule some things out, but it’s hard to prove beyond a certainty what happened,” said Josh Clinton, a professor at Vanderbilt University and the chair of the association’s 2020 election task force. “Based on what we know about polling, what we know about politics, we have some good prime suspects as to what may be going on.”
Those “prime suspects” will hardly be comforting to pollsters and those who depend on them, from political campaigns to the news media. The most likely — if far from certain — culprit for off-kilter polling results is that key groups of people don’t answer polls in the first place.
Decreasing response rates have been a major source of concern for pollsters for more than a decade. But the politicization of polling during the Trump era — including the feedback loop from the former president, who has falsely decried poll results he doesn’t like as “fake” or deliberately aimed at suppressing enthusiasm for answering polls among GOP voters — appears to be skewing the results, with some segment of Republicans refusing to participate in surveys.
But pollsters say they can’t be sure that’s the main reason, because you never know exactly whom you’re not talking to.
That makes the problems with polling a lot harder to fix than the diagnosis four years ago, which mostly focused on adjusting surveys to account for Trump’s popularity with voters who haven’t earned college degrees and his corresponding weakness with college degree-holders.
“It seems plausible to the task force that, perhaps, the Republicans who are participating in our polls are different from those who are supporting Republican candidates who aren’t participating in our polls,” Clinton said. “But how do you prove that?”
The task force’s first job was to evaluate the performance of the 2020 public election polls. On that measure, polling earned a failing grade. While the national polls were the worst in four decades, the state-level polls of the presidential, Senate and gubernatorial races were as bad as they’ve been as far back as there are records (20 years).
According to the report, national polls of the presidential race conducted in the final two weeks of the election were off by an average of 4.5 percentage points, while the state polls were off by just over 5 points. Most of the error was in one direction: Looking at the vote margin, the national polls were too favorable to now-President Joe Biden by 3.9 points, and the state polls were 4.3 points too favorable for Biden.
Most of the error came from underestimating Trump’s support, as opposed to overestimating Biden’s. Comparing the final election results to the poll numbers for each candidate, Trump’s support was understated by a whopping 3.3 points on average, while Biden’s was overstated by a point — turning what looked like a solid Biden lead into a closer if still decisive, race.
It wasn’t just a Trump effect, either. The polls of Senate and governor’s races were off by an even greater margin: 6 points on average.
“Within the same state, polling error was often larger in senatorial contests than the presidential contest,” the AAPOR report reads. “Whether the candidates were running for president, senator, or governor, poll margins overall suggested that Democratic candidates would do better and Republican candidates would do worse relative to the final certified vote.”
No one methodology performed head-and-shoulders above the others. According to the report, there were only “minor differences” whether polls were conducted on the phone, over the internet or using a mixed methodology, including texts and smartphone apps — or whether they contacted voters randomly versus off a list of registered voters. “Every mode of interviewing and every mode of sampling overstated the Democratic-Republican margin relative to the final certified vote margin,” the report said.
After the 2016 election, AAPOR’s autopsy blamed that year’s polling errors on a number of different factors. First, the organization said, a larger-than-usual number of undecided voters measured in polls flocked toward Trump disproportionately at the very end of the race, giving him an advantage that would be impossible to measure beforehand.
But 2020’s error can’t be blamed on late deciders: Only 4 percent of voters weren’t behind one of the two major candidates in state polls conducted over the final two weeks, and exit polls suggest late-deciding voters split roughly evenly between Biden and Trump.
Another of the 2016 problems — the failure of many pollsters to weight by education — wasn’t to blame last year, either, the report said. Four years earlier, many pollsters adjusted their results to get the right mix of voters by race and gender. But that missed a key, emerging dynamic in the electorate: Increasingly, white voters with college degrees have supported Democrats, while those who didn’t graduate from college rapidly flocked toward Republicans. Studies show voters without college degrees are less likely to participate in polls.
In 2020, however, the majority of state polls made adjustments to get more non-college voters in their polls. But they were still wrong.
Other 2016-style factors were also dismissed: Voters weren’t lying to pollsters about whom they’d support because of some kind of “shy Trump” theory (otherwise the errors wouldn’t be larger in downballot races). It wasn’t that one candidate’s backers didn’t show up to vote (as evidenced by the record-breaking turnout in last year’s race). And estimating the number of voters who would cast early ballots versus show up on Election Day also wasn’t to blame (the polls mostly nailed that split).
The report is clear on what didn’t cause the 2020 polling miss. But it says “identifying conclusively why polls overstated the Democratic-Republican margin relative to the certified vote appears to be impossible with the available data.”
The most plausible — yet still unproven — theory is that the voters the polls are reaching are fundamentally different from those they are not. And Trump’s rantings about the polls being “fake” or rigged only exacerbate that problem.
“If the voters most supportive of Trump were least likely to participate in polls then the polling error may be explained as follows: Self-identified Republicans who choose to respond to polls are more likely to support Democrats and those who choose not to respond to polls are more likely to support Republicans,” the report reads. “Even if the correct percentage of self-identified Republicans were polled, differences in the Republicans who did and did not respond could produce the observed polling error.”
AAPOR isn’t the only organization struggling to nail down where things went wrong. A collaborative report conducted by five of the largest Democratic campaign polling firms, released this spring, said “no consensus on a solution has emerged” to fix the 2020 errors.
While explanations remain elusive, pollsters and their clients are hard at work on changes to methodologies. Soliciting poll respondents via text messages — or text surveys entirely — are increasingly popular as fewer Americans are willing to take a 15-minute phone poll. Online polling continues to grow as well.
Public polls commissioned by the media are also changing. NBC News and The Wall Street Journal terminated their more-than-30-year-long polling partnership late last year, a Wall Street Journal spokesperson confirmed to POLITICO. The two news organizations had long worked with a bipartisan pair of major polling firms on regular phone surveys.
Without definitive answers about the causes of the 2020 miss, however, pollsters aren’t sure they’ll be able to get it right in 2022, 2024 or beyond.
“Even seven months after the fact, you’d think you’d be able to know exactly what happened,” Clinton said.
“How certain are we that we can fix this in the future? Well, it’s unclear,” Clinton added. “We’ll have to wait and see what happens — which isn’t a particularly reassuring position. But I think that’s the honest answer.”
Princeton’s Sam Wang had to eat his words (and a cricket) in 2016. He talks about the impacts of the pandemic and QAnon on public-opinion tallies in 2020
By Gloria Dickie, November 13, 2020
In the weeks leading up to the November 2016 election, polls across the country predicted an easy sweep for Democratic nominee Hillary Clinton. From Vanuatu to Timbuktu, everyone knows what happened. Media outlets and pollsters took the heat for failing to project a victory for Donald Trump. The polls were ultimately right about the popular vote. But they missed the mark in key swing states that tilted the Electoral College toward Trump.
This time, prognosticators made assurances that such mistakes were so 2016. But as votes were tabulated on November 3, nervous viewers and pollsters began to experience a sense of déjà vu. Once again, more ballots were ticking toward President Trump than the polls had projected. Though the voter surveys ultimately pointed in the wrong direction for only two states—North Carolina and Florida, both of which had signaled a win for Joe Biden—they incorrectly gauged just how much of the overall vote would go to Trump in both red and blue states. In states where polls had favored Biden, the vote margin went to Trump by a median of 2.6 additional percentage points. And in Republican states, Trump did even better than the polls had indicated—by a whopping 6.4 points.
Four years ago, Sam Wang, a neuroscience professor at Princeton University and co-founder of the blog Princeton Election Consortium, which analyzes election polling, called the race for Clinton. He was so confident that he made a bet to eat an insect if Trump won more than 240 electoral votes—and ended up downing a cricket live on CNN. Wang is coy about any plans for arthropod consumption in 2020, but his predictions were again optimistic: he pegged Biden at 342 electoral votes and projected that the Democrats would have 53 Senate seats and a 4.6 percent gain in the House of Representatives.
Scientific American recently spoke with Wang about what may have gone wrong with the polls this time around—and what bugs remain to be sorted out.
[An edited transcript of the interview follows.]
How did the polling errors for the 2020 election compare with those we saw in the 2016 contest?
Broadly, there was a polling error of about 2.5 percentage points across the board in close states and blue states for the presidential race. This was similar in size to the polling error in 2016, but it mattered less this time because the race wasn’t as close.
The main thing that has changed since 2016 is not the polling but the political situation. I would say that worrying about polling is, in some sense, worrying about the 2016 problem. And the 2020 problem is ensuring there is a full and fair count and ensuring a smooth transition.
Still, there were significant errors. What may have driven some of those discrepancies?
The big polling errors in red states are the easiest to explain because there’s a precedent: in states that are historically not very close for the presidency, the winning candidate usually overperforms. It’s long been known turnout is lower in states that aren’t competitive for the presidency because of our weird Electoral College mechanism. That effect—the winner’s bonus—might be enhanced in very red states by the pandemic. If you’re in a very red state, and you’re a Democratic voter who knows your vote doesn’t affect the outcome of the presidential race, you might be slightly less motivated to turn out during a pandemic.
That’s one kind of polling error that I don’t think we need to be concerned about. But the error we probably should be concerned about is this 2.5-percentage-point error in close states. That error happened in swing states but also in Democratic-trending states. For people who watch politics closely, the expectation was that we had a couple of roads we could have gone down [on election night]. Some states count and report votes on election night, and other states take days to report. The polls beforehand pointed toward the possibility of North Carolina and Florida coming out for Biden. That would have effectively ended the presidential race right there. But the races were close enough that there was also the possibility that things would continue. In the end, that’s what happened: we were watching more counting happen in Pennsylvania, Michigan, Wisconsin, Arizona and Nevada.
How did polling on the presidential race compare with the errors we saw with Senate races this year?
The Senate errors were a bigger deal. There were seven Senate races where the polling showed the races within three points in either direction. Roughly speaking, that meant a range of outcomes for between 49 and 56 Democratic seats. A small polling miss had a pretty consequential outcome because every percentage point missed would lead to, on average, another Senate seat going one way or the other. Missing a few points in the presidential race was not a big deal this year, but missing by a few points in Senate races mattered.
What would more accurate polling have meant for the Senate races?
The real reason polling matters is to help people determine where to put their energy. If we had a more accurate view of where the races were going to end up, it would have suggested political activists put more energy into the Georgia and North Carolina Senate races.
And it’s a weird error that the Senate polls were off by more than the presidential polls. One possible explanation would be that voters were paying less attention to Senate races than presidential races and therefore were unaware of their own preference. Very few Americans lack awareness of whether they prefer Trump or Biden. But maybe more people would be unaware of their own mental processes for say, [Republican incumbent] Thom Tillis versus [Democratic challenger] Cal Cunningham [in North Carolina’s Senate race]. Because American politics have been extremely polarized for the past 25 years, people tend to [end up] voting [a] straight ticket for their own party.
Considering that most of the polls overestimated Biden’s lead, is it possible pollsters were simply not adequately reaching Trump supporters by phone?
David Shor, a data analyst [who was formerly head of political data science at the company Civis Analytics], recently pointed out the possibility that people who respond to polls are not a representative sample. They’re pretty weird in the sense that they’re willing to pick up the phone and stay on the phone with a pollster. He gave evidence that people are more likely to pick up the phone if they’re Democrats, more likely to pick up under the conditions of a pandemic and more likely to pick up the phone if they score high in the domain of social trust. It’s fascinating. The idea is that poll respondents score higher on social trust than the general population, and because of that, they’re not a representative sample of the population. That could be skewing the results.
This is also related to the idea that states with more QAnon followers experienced more inaccurate polling. The QAnon belief system is certainly correlated with lower social trust. And those might be people who are simply not going to pick up the phone. If you believe in a monstrous conspiracy of sex abuse involving one of the major political parties of the U.S., then you might be paranoid. One could not rule out the possibility that paranoid people would also be disinclined to answer opinion polls.
In Florida’s Miami-Dade County, we saw a surprising surge of Hispanic voters turning out for Trump. How might the polls have failed to take into account members of that demographic backing Trump?
Pollsters know Hispanic voters to be a difficult-to-reach demographic. In addition, Hispanics are also not a monolithic population. If you look at some of the exit polling, it looks like Hispanics were more favorable to Trump than they were to Clinton four years ago. It’s certainly possible Hispanic support was missed by pollsters this time around.
Given that the presidential polls have been off for the past two elections, how much attention should people pay to polls?
I think polling is critically important because it is a way by which we can measure public sentiment more rigorously than any other method. Polling plays a critical role in our society. One thing we shouldn’t do is convert polling data into probabilities. That obscures the fact that polls can be a few points off. And it’s better to leave the reported data in units of opinion [as a percentage favoring a candidate] rather than try to convert it to a probability.ADVERTISEMENT
It’s best not to force too much meaning out of a poll. If a race looks like it’s within three or four points in either direction, we should simply say it’s a close race and not force the data to say something they can’t. I think pollsters will take this inaccuracy and try to do better. But at some level, we should stop expecting too much out of the polling data.
ABOUT THE AUTHOR
Gloria Dickie is an environmental journalist based in British Columbia, Canada. Her work appears in National Geographic, Wired, the New York Times, and elsewhere.
By Byron York
As the last presidential election approached, one could sense that something was wrong with the polls. The race felt tighter than the polls indicated, and specifically, Joe Biden’s solid lead in the polls felt less solid than the surveys suggested.
As they had in 2016, pollsters appeared on their way to underestimating the vote for Donald Trump. That did not mean Trump was actually leading the race, or that he would win — and in the end, he did not — but that the contest was closer than the polls said.
Now it turns out that feeling was right. And the polls were not only wrong, but more wrong than they had been in decades, including their dismal performance in 2016. A new report from the American Association for Public Opinion Research has found that national election polls overstated Biden’s lead over Trump by 3.9 percentage points. And state-level polls were even worse, overstating Biden’s lead by 4.3 points.
The polls’ performances in statewide races — campaigns for senator and governor — were worse still. There, they overstated the Democratic candidates’ lead by an average of 6 percentage points.
Notice something? The errors went in one direction: Exaggerating support for Democratic candidates. “Whether the candidates were running for president, senator or governor, poll margins overall suggested that Democratic candidates would do better and Republican candidates would do worse relative to the final certified vote,” the report says.
So all those Republicans who said the polls were slanted against Trump and the GOP — they were right.
Now, the question is why. Many Republicans will answer that it’s simple — that the polls were just biased. Indeed, don’t ever discount the possibility that pure bias is at work. But a panel of political scientists and polling experts appointed by the AAPOR found that pollsters had managed to avoid repeating the mistake they made in 2016 — undercounting voters without a college degree who voted for Trump. So if that wasn’t the problem in 2020, what was?
The analysts do not think the issue was specifically Trump, or pollsters would have done better on the senators and governors races. They also couldn’t find any specific group of voters that they missed. Instead, they focused on the theory that many Republicans distrust the media and pollsters so much that they would not respond to polling requests. Focusing only on those Republicans who were receptive to pollsters gave the polls a skewed view of GOP attitudes.
“That the polls overstated Biden’s support in whiter, more rural and less densely populated states is suggestive (but not conclusive) that the polling error resulted from too few Trump supporters responding to polls,” the report says. “A larger polling error was found in states with more Trump supporters.”
In the end, though, the panel says it cannot definitively determine where pollsters made their mistake. “Identifying conclusively why polls overstated the Democratic-Republican margin … appears to be impossible with the available data,” the report concludes.
The bottom line: The polls were wrong, and they were consistently wrong in one direction, overstating Democratic support and understating Republican support. And the problem wasn’t only about Trump.
That is an extremely troubling conclusion for the millions of Americans who follow politics. Media coverage of big races is often based on a general understanding of who is leading and who is trailing. And that understanding comes from public opinion polls. If a candidate has a 1-point lead over an opponent, for example, the coverage will present the race as virtually tied and highly competitive. If a candidate has a 6-point lead over an opponent, the coverage will present the leader as the dominant figure in the race. That can discourage supporters of the trailing candidate. And it all comes from polls.
And what if the polls are wrong — not just a little wrong, but a lot wrong? Either the polls will be fixed, or we will have to find a different way to cover politics.