Why Polling On The Issues Probably Isn’t As Broken As Horse-Race Polling

Posted On March 12, 2021

Welcome to Pollapalooza, our weekly polling roundup.

How many Americans support legalizing marijuana? Sixty-eight percent, per Gallup. And according to Morning Consult/Politico, 60 percent think the minimum wage should be increased to $15 an hour by 2025. But how much stock should we really be putting into polls like this, especially given the polling error we saw in the 2016 and 2020 elections?

Turns out, there are a number of reasons why we shouldn’t be too worried, according to a new Pew Research Center study, which analyzed nearly a year’s worth of its polls asking Americans how they felt about a variety of issues. (Pew examined both the actual survey results and what the results would have looked like if they had factored in a person’s presidential vote choice.) Of course, this doesn’t mean these polls are totally immune from the type of polling error we’ve seen before, so here are the big things you should keep in mind when looking at public opinion polling in 2021 and beyond.

First, there is error in issue polls. It’s just that the error is, on average, smaller than the error observed in horse-race polls. That is, the majority of questions Pew looked at didn’t change all that much when a person’s presidential vote choice was factored in — less than one percentage point on average. Or put another way, adjusting for whether someone voted for Joe Biden or Donald Trump didn’t significantly alter their overall response on how they felt about, say, climate change. Just three out of the 48 questions Pew looked at would have changed by 3 points.¹

This suggests that issue polls are just less affected by sampling problems than election polls. But why? Many issues just aren’t as partisan as, well, parties are. Factors that might lead to Republican nonresponse in election polls might have less of an effect on issue support, since support for issues is at least somewhat more bipartisan than support for candidates. (In other words, not all Democrats support issues typically aligned with the Democratic Party, and not all Republicans oppose them.)

Another reason: One of the biggest problems in election polling is modeling voter turnout — and, given the pandemic, that could have been a particularly big factor in 2020 — but that simply is not a factor in issue polling, which is more about taking the temperature of how Americans feel than about predicting any one outcome.

And finally, as Stanford University political scientist Doug Rivers wrote in an email to FiveThirtyEight, issue polling, for the most part, just isn’t as divided as horse-race polling: “I think the real difference is that 2-3 point errors on a typical issue item rarely changes the conclusion, while being off 2 points on a candidate’s vote share [is] considered catastrophic.” Take Pew’s study. Most of the issues they looked at had a big enough majority that if a polling error had occurred on the scale of what we saw in 2020, that still wouldn’t be enough to swing the outcome. Alternatively, a big enough polling error in the election could have meant a Trump victory, as was the case in 2016.

That said, issue polls face a number of their own challenges. For one, the accuracy of issue polls isn’t really ever tested. Or at least not in the way horse-race polls are scrutinized. That’s in part because every election cycle ends in an actual election, whereas there isn’t really the same kind of end point for an issue poll. (Pollsters do test how well they do on issue polling, often using benchmarks from higher-quality government surveys, but those tests aren’t conducted as frequently, and even with higher-quality data, it could still be wrong.)

There’s another caveat specific to the Pew study. It assumes that adjusting for how someone voted fixes any underlying problems with issue polls, but that assumes that how someone feels about an issue is perfectly correlated with vote choice — which we know isn’t a 1:1 relationship — or that the source of survey nonresponse is presidential vote choice. But it’s possible that something else entirely is driving the error in issue polls. As Robert Griffin, research director of the Democracy Fund Voter Study Group, told me, even if vote choice is a source of error, Republicans who are responding to the survey might have different views on an issue than those who don’t respond, so upweighting those who did respond won’t fix the problem either.

Finally, while issue polling is not yet as partisan as horse-race polling, that might not be as true moving forward. The three questions on which Pew found the biggest differences when presidential vote preference was factored in: 1) presidential job approval, 2) whether the bigger problem is people not seeing existing racism, or people seeing nonexistent racism, and 3) whether the U.S. had controlled the coronavirus outbreak “as much as it could have.” Partisanship might not yet determine everything, but the more it does, the more that will affect issue polling, too.

Other polling bites

Several polls this week have found widespread support for Biden’s $1.9 trillion COVID-19 relief package: 70 percent of Americans overall, according to Pew, with particularly strong support among low-income Americans (82 percent). In fact, while Democrats of all income groups were very supportive of the package, a majority of low-income Republicans (63 percent) were in favor, too. The Economist/YouGov found 64 percent supported the bill overall, Data for Progress found 69 percent in support, and Morning Consult/Politico, which listed the bill’s major provisions in the question, found 75 percent in support. (Both The Economist/YouGov and Data for Progress also found broad support across the bill’s individual provisions.)
Despite the plethora of research showing the disproportionate effect the pandemic has had on women’s participation in the workforce (not to mention their workload at home), just 6 percent of Americans think it will have lasting negative effects on women’s equality, including just 5 percent of women. In contrast, 50 percent think things will go back to how they were before.
There is a marked difference between Brits’ and Americans’ reactions to Oprah Winfrey’s interview of the Duke and Duchess of Sussex on Sunday. While 44 percent of Americans think Harry and Meghan’s interview was appropriate, just 21 percent of Brits agree. Far more were likely to say it was inappropriate (47 percent) compared with just 20 percent of Americans. And among Brits, views of the interview differed drastically by respondents’ politics: 30 percent of Labour supporters said they thought it was inappropriate, compared with 74 percent of Conservative supporters.
With German regional elections in Baden-Württemberg and Rhineland-Palatinate fast approaching on March 14, the center-right Christian Democratic Union has recently lost support in both states, leaving the Green Party in the lead in Baden-Württemberg and CDU and Social Democrats essentially tied in Rhineland-Palatinate. It remains to be seen whether this bodes ill for the CDU in the general election this fall, likely the first election without its former leader, Chancellor Angela Merkel, in over 15 years. (And that’s not the only big shakeup in German leadership: National soccer coach Joachim Löw has announced that he is retiring this summer, after the European Championship. His term will have ended up overlapping almost perfectly with Merkel’s.)
Meanwhile, in a very early poll of the New York City mayoral race — the primaries aren’t until June and the general election not until November — former 2020 Democratic presidential primary contender Andrew Yang is already a front-runner with 32 percent support of likely Democratic voters, although it looks to be a very crowded field.
Texans are closely divided on Gov. Greg Abbott’s decision to repeal the statewide mask mandate: 48 percent of registered Texas voters are in favor, and the same share is opposed. However, this masks (heh) big racial and urban-rural disparities, with white and rural Texans more likely to support his decision and non-white and urban Texans more likely to oppose it.

Biden approval

According to FiveThirtyEight’s presidential approval tracker,² 53.2 percent of Americans approve of the job Biden is doing as president, while 40.2 percent disapprove (a net approval rating of +13 points). At this time last week, 52.7 percent approved and 38.6 percent disapproved (a net approval rating of +14.2 points). One month ago, Biden had an approval rating of 54.3 percent and a disapproval rating of 37.1 percent, for a net approval rating of +17.1 points.

Source: https://fivethirtyeight.com/features/why-polling-on-the-issues-probably-isnt-as-broken-as-horse-race-polling/