Political polls are appallingly unreliable. As reported by news media, polling data led to erroneous claims in the run up to elections in 2016, 2018, 2020 and 2022.
New York Times journalists have argued that reporting on political polls in 2020 was flawed because “traditional nonpartisan pollsters” conducted fewer polls. (They don’t fall off trees. Someone has to pay for them.) The NYT journalists also asserted (without evidence) that those polls “largely reflected reality.”
To fill the insatiable 24×7 news hole, profit-oriented news businesses (my characterization) turn to partisan polls, which are atrociously inaccurate. The NYT journalists included no mea culpa for having given them attention although they chide FOX News.
The result: click-and-attention getting headlines. Jay Rosen has characterized this news treatment as a focus on “the odds” that a candidate might win or lose rather than the “stakes” of their success. Such reporting does a disservice to voters and democracy, according to Rosen.
An ex-TV news journalist friend shrugged matter of factly last night when we discussed how polls get reported: newsrooms have to turn a profit, she said.
To illustrate the scope of the problem, my Bing search of the past month yielded 30,400 results for the term ‘Biden Trump poll’. That’s more than 1,000 a day for a 30-day month.
A call to action
Don’t watch, listen or read news stories about the latest political poll. It’s not only that they are unreliable (especially 14 months before a presidential election). They also divert time and attention from substantive analysis. That’s what Rosen has dubbed the “stakes” resulting from who makes it into elected office.
When we stop rewarding clickbait, maybe — just maybe — we will make it less profitable.
If you are so inclined, tell news organizations your concerns with their reliance on polls via a traditional letter to the editor or a digital reply on Facebook or Twitter (the company that insists on calling itself X).
That’s reasons one and two: unreliable and diversionary.
8 more reasons to ignore current political polling news
3. Innumeracy (theirs and ours)
Journalists (or their copy editors) are notoriously innumerate. That innumeracy — a poor understanding of numbers, statistics, and probability (math) — carrys over to horse race political reporting.
For example, poll-focused stories usually fail to report or contextualize the margin of error, even as an after-thought at the story’s close. From the NYT in 2016:
As anyone who follows election polling can tell you, when you survey 1,000 people, the margin of error is plus or minus three percentage points. This roughly means that 95 percent of the time, the survey estimate should be within three percentage points of the true answer.
As anyone who follows election polling can tell you, when you survey 1,000 people, the margin of error is plus or minus three percentage points. This roughly means that 95 percent of the time, the survey estimate should be within three percentage points of the true answer.
If 54 percent of people support Hillary Clinton, the survey estimate might be as high as 57 percent or as low as 51 percent, but it is unlikely to be 49 percent. This truism of modern polling, heralded as one of the great success stories of statistics, is included in textbooks and taught in college classes, including our own.
But the real-world margin of error of election polls is not three percentage points. It is about twice as big.
Did you know the chance that Donald Trump might win in 2016 was within the margin of error of most national polls?
A 95% confidence interval [most poll results] with a 4 percent margin of error means that your statistic will be within 4 percentage points [+/- 4 points or an 8-point bracket] of the real population value 95% of the time.
If you do read a story touting poll results, check the margin of error. If you add to one candidate’s estimated support, you must subtract from the other one. Revisit the example above featuring Clinton. If a poll were to have a margin of error of +/- four points, the contest would be a statistical dead heat.
American innumeracy
Every three years, the OECD Programme for International Student Assessment (PTSA) surveys a randomly selected group of 15-year-old students from around the world on math, reading and science. (Covid-19 interrupted the 2021 cycle.)
In 2018, 600,000 students in 79 countries participated in the math survey. The U.S. placed 37th out of the 79 countries that participated in the math test, behind China (1st), Korea (7th), Slovenia (14th), Czech Republic (22nd) and Russia (30th).
4. Variable methodology
Comparing polling data across news organizations is less than straightforward.
How many consumers who rely on multiple news sources understand that there is no generally accepted standard for political polling methodological? From Pew Research:
Different polling organizations conduct their surveys in quite different ways. Survey methodology is undergoing a period of creative ferment. Currently, CNN and Fox News conduct polls by telephone using live interviewers, CBS News and Politico field their polls online using opt-in panels, and The Associated Press and Pew Research Center conduct polls online using a panel of respondents recruited offline. There is even a fourth group of pollsters that combine methods like robocalls and online surveying with opt-in samples. These different approaches have consequences for data quality, as well as accuracy in elections.
5-9: Jennifer Rubin chimes in
The headline of her 11 Sept 2023 column:
“I don’t write about polls. You shouldn’t bother with them, either.”
- The polling field is broken.
- Voters tell us utterly contradictory things.
- There’s no evidence that polling more than a year (or even eight months) before a presidential election is accurate.
- Polling seems designed to make a point. (See the problem with leading questions.)
- What voters know might be wrong — objectively wrong.
Gift link to the Washington Post.
There you have it: nine (9) reasons you should be ignoring political polls.
The list is incomplete.
To serve democracy and voters, news organizations should — but usually do not — provide context for survey results. (They should also report time-related monetary data in real dollars, but that’s another column.) A bit of agenda-setting theory is also to come!
Follow me: Facebook, Mastodon, Twitter
Featured image: photo 131140713 | Political Poll Concept © Tero Vesalainen | Dreamstime.com
Known for gnawing at complex questions like a terrier with a bone. Digital evangelist, writer, teacher. Transplanted Southerner; teach newbies to ride motorcycles. @kegill (Twitter and Mastodon.social); wiredpen.com