Angus Reid Institute
POLLSTERS AND POLLING organizations are frequently the target of criticism, derision and disdain. People don’t like the data, so folks get mad. We sometimes get it wrong, so folks get mad. And that’s fair play.
Most major pollsters really do everything in their power to conduct public opinion research in a way that is accurate, and fair. And while a lot of time is spent by observers dissecting the methodology pollsters use to collect data, an equally important but overlooked part of the survey process is the way the questions are written.
We can tell you, at the Angus Reid Institute, we take this part very seriously. We sweat over vocabulary. We fact check our questions. How much explanation is too much? Have we now created a leading question? Then it’s back to the drawing board. How much explanation is not enough? Will our respondents fully understand the context around the issues we’re canvassing?
We’ve been on conference calls with colleagues that have lasted hours, trying to get the phrasing just right. And we publish our survey questionnaires along with the data, so people can see for themselves what we asked and why we asked it.
And then…. there’s the Trump poll. Or rather, the “Mainstream Media Accountability Survey”, published on the Donald Trump campaign website Thursday afternoon.
When it started making the rounds on social media, and then around our offices, reactions ranged from horror to humor. This is, in our humble but professional opinions, a really, supremely, incredibly badly written poll.
But hey, we’re all about education and furthering polling discourse at the Angus Reid Institute, so let’s just turn this into a teachable moment, and walk through all the ways this survey could have been written better.
First of all, there’s nothing wrong with canvassing American trust in mainstream media organizations. It’s a legitimate subject. One that Pew Research tackles ably every day.
But these questions in the Trump survey? Let’s start by considering question 5, which asks: “On which issues does the mainstream media do the worst job of representing Republicans?” and then instructs respondents to “select as many that apply.” How can they be the things the media is “worst” at if we can pick them all?
Question 6 suffers from almost the opposite problem. It asks respondents from which television source they primarily get their news, then lists a grand total of four options: Fox News, CNN, MSNBC, and “local news.” There are other channels, Mr. President.
The rest of the 25-question survey, aside from a pair of open-ended prompts, follows the same pattern: A yes or no question is asked, and respondents are given four possible response choices: “yes,” “no,” “no opinion,” and “other, please specify.”
Here’s why that’s a terrible response metric:
First, it leads to nonsensical situations like this one:
Sometimes it’s good to have a “no opinion” option. Questions about personal experiences are not those times.
Shachi Kurl is executive director of the Angus Reid Institute; Ian Holliday is a research assistant. Read a longer version of this column at angusreid.org.