How polls are a threat to US democracy

The US media and public’s shared obsession with polling fuelled an entire election season that virtually ignored policy – and perhaps helped Trump win

Six days before the US election, Donald Trump gave a crowd in Miami some unusual instructions:

Maybe the then Republican nominee had been reading up on political science research. In the fall of 2014, two academics published a study asking a question that seems very relevant now: could believing that one candidate is going to lose increase their chances of winning?

The paper, by Todd Rogers and Don A Moore, looked at emails sent during the 2012 US presidential campaign. Based on their analysis of more than a million observations, the researchers concluded that messages emphasizing that a candidate was “barely losing” raised 55% more money than emails emphasizing that a candidate was “barely winning”. The phenomenon has been studied before in political science – it is known as the underdog effect.

In the lead-up to the 2016 election, polls and political forecasts repeatedly told US voters that Trump was losing to Hillary Clinton, albeit barely. The Real Clear Politics average of polls showed Trump’s support careening in the final months of the election, at times being less than one percentage point behind Clinton, at times being as much as seven percentage points behind. But rarely was he show to be ahead.

Journalism’s contribution to democracy was never about predicting public behavior; it was about informing it. But forecasting sites such as Nate Silver’s FiveThirtyEight and the New York Times’ Upshot used polling data to give readers probabilities of a Trump win. Again, the numbers fluctuated over the course of the election, but by voting day, Silver claimed Trump had a 28% chance of becoming the next US president and the Times put his chances at 15%.

The audience reading those numbers was trying to understand who was the underdog and who was the favorite – and it was a large audience. According to ESPN, 16.5 million unique users visited FiveThirtyEight on 8 November alone, and millions more would have read those probabilities when they were quoted by countless other publications in the weeks up to voting day.

Andrew Therriault was certainly one of those readers. Therriault was director of data science for the Democratic National Committee for two years before leaving in June 2016. Speaking on the phone, he said those forecasts might have helped Trump win: “When it’s this close, anything that could even plausibly have an effect could be a reason why the outcome happened. I don’t think it had a sizable effect. But there’s a possibility that a not-so sizable effect was sizable enough.”

Therriault’s reluctance to endorse the underdog effect theory as the only explanation makes sense. There are other equally compelling explanations – like bandwagon effect theory, which would argue that forecasts showing Clinton would win helped voters rally around her rather than Trump.

Seeming to hedge his bets, Trump’s rhetoric during the election made use of both appeals to voters; the candidate would routinely claim that he was worth supporting because he was a winner and because he could lose.

Trump’s strategy could be understood using a cruder political science theory: “ass-covering”. A similar theory could be used to understand Silver’s final election prediction, in which he said that Clinton would probably win – but that she could also lose to Trump, or win by a landslide. Similarly, once Trump’s victory was confirmed, Silver described the result as shocking, but not surprising.

After 8 November, Silver seems very, very unwilling to say that FiveThirtyEight was wrong, insisting instead that the site’s forecast was more accurate than others. There’s some truth to that claim. In predicting the popular vote, Silver had projected that Clinton would win 49% and Trump 45% (in the end, Hillary Clinton won 48% and Trump won 46%). And, when looking at state-level results, Silver’s model guessed better than eight other forecasts.

But many readers weren’t interested in those statistical nuances – they simply had a question: “Who will be the next US president?” and believed, based on forecasts, that the answer was Hillary Clinton.

But polling is inaccurate, and it’s getting worse – something even political forecasters admit.

Speaking on WNYC days after the result, Silver told the host: “To me, it’s a miracle that the polls aren’t off by more.” The Upshot had a similar line in its postmortem, writing: “It was the biggest polling miss in a presidential election in decades. Yet in many ways, it wasn’t wholly out of the ordinary.”

The Americans who did cast a ballot on election day were not simply black or white, male or female, wealthy or poor. They included middle-class Native American women and low-income college-educated black men – it is very hard to capture that demographic complexity in the 1,000 respondents you are contacting in a poll. American society is only becoming more diverse, and fewer Americans today are willing to talk to a pollster for up to 30 minutes for nothing in return.

In the 1980s, around 60% of those contacted to take part in a poll would do so. By 2012, response rates had fallen to 9%, and by 2016, Therriault told me that when he was with the DNC, “I would have killed to get 9%.” Those falling response rates are problematic. They make conducting polls an expensive business because more slammed phones mean more people are needed to make those calls.

Some elections pollsters used a panel instead, contacting the same thousand or so individuals every week to reduce those costs and get a more accurate picture of how their opinion was changing over the course of the election.

But pollsters also found that after a candidate had received negative media coverage (eg for having a private email server or for being accused of sexual assault for the 12th time), respondents simply wouldn’t pick up the phone to avoid being questioned about the scandals and whether they would continue to support their candidates. That made it hard for pollsters to know what those people would do in voting booths.

Forecasts rely heavily on those polls, but they also use historical data about factors such as turnout and voter behavior. In the past, they have been a good guide. The 2012 presidential election wasn’t so dramatically inconsistent with the 2008 election – which is one reason why Silver and others correctly predicted the outcome. But, to state the obvious, 2016 is no ordinary year. And, in the absence of using the past as a guide, it’s important to look around at the present. And the present can look very different depending on who you are in America.

In that same WNYC interview, aired on 11 November, Silver said of Trump:

That view may not have been shared by non-white polling analysts, of which there are few in the American media. In the year and a half that I worked at FiveThirtyEight, I was the only non-white staff writer there. Unlike for Silver, the fact that racism was alive and well in America was no surprise to me. That affected my skepticism when reading polling numbers – I was convinced that they were off and underestimated Trump’s support (although exit polling data suggests that the poorest Americans might not have voted for Trump, as I thought at the time).

The inaccurate polling numbers did not just affect what forecasters wrote. Based on her campaign’s own internal data, Clinton decided not to visit Maine, Wisconsin or Minnesota, and it was only in the last week of the race that she campaigned in Michigan. In the end, she lost Wisconsin and Michigan, and only narrowly won Maine and Minnesota.

Political polling has repeatedly proved unreliable recently – in Britain’s Brexit vote, the 2015 UK election and the Israeli election the same year. But even when the numbers are correct, are polling calculations healthy for democracy? In the US, the media and the public’s shared obsession with polling fuelled an entire election season, including three televised candidate debates, that virtually ignored policy. Headlines focused instead on scandals and their potential effect on the horserace numbers.

Midterms are two years away, so there might be a pause before forecasting dominates the US media again in quite the same way. But polling isn’t going anywhere – if Trump’s candidacy is any guide, Trump as president will obsessively quote his approval ratings and use polling numbers to craft populist policies.

It will take months to gather the voter file data necessary to accurately understand the inaccuracy of the polls in 2016. But it’s not too early to question whether polls threaten democracy.