Advertisements
LATEST

KURL – Make public-opinion polling questionnaires great again

trump-poll-19feb2017By SHACHI KURL and IAN HOLLIDAY

Angus Reid Institute

POLLSTERS AND POLLING organizations are frequently the target of criticism, derision and disdain. People don’t like the data, so folks get mad. We sometimes get it wrong, so folks get mad. And that’s fair play.

kurl-colhedMost major pollsters really do everything in their power to conduct public opinion research in a way that is accurate, and fair. And while a lot of time is spent by observers dissecting the methodology pollsters use to collect data, an equally important but overlooked part of the survey process is the way the questions are written.

We can tell you, at the Angus Reid Institute, we take this part very seriously. We sweat over vocabulary. We fact check our questions. How much explanation is too much? Have we now created a leading question? Then it’s back to the drawing board. How much explanation is not enough? Will our respondents fully understand the context around the issues we’re canvassing?

We’ve been on conference calls with colleagues that have lasted hours, trying to get the phrasing just right. And we publish our survey questionnaires along with the data, so people can see for themselves what we asked and why we asked it.

And then…. there’s the Trump poll. Or rather, the “Mainstream Media Accountability Survey”, published on the Donald Trump campaign website Thursday afternoon.

When it started making the rounds on social media, and then around our offices, reactions ranged from horror to humor. This is, in our humble but professional opinions, a really, supremely, incredibly badly written poll.

But hey, we’re all about education and furthering polling discourse at the Angus Reid Institute, so let’s just turn this into a teachable moment, and walk through all the ways this survey could have been written better.

First of all, there’s nothing wrong with canvassing American trust in mainstream media organizations. It’s a legitimate subject. One that Pew Research tackles ably every day.

But these questions in the Trump survey? Let’s start by considering question 5, which asks: “On which issues does the mainstream media do the worst job of representing Republicans?” and then instructs respondents to “select as many that apply.” How can they be the things the media is “worst” at if we can pick them all?

Question 6 suffers from almost the opposite problem. It asks respondents from which television source they primarily get their news, then lists a grand total of four options: Fox News, CNN, MSNBC, and “local news.” There are other channels, Mr. President.

The rest of the 25-question survey, aside from a pair of open-ended prompts, follows the same pattern: A yes or no question is asked, and respondents are given four possible response choices: “yes,” “no,” “no opinion,” and “other, please specify.”

Here’s why that’s a terrible response metric:

First, it leads to nonsensical situations like this one:

trumo-poll-tweet-19feb2017

Sometimes it’s good to have a “no opinion” option. Questions about personal experiences are not those times.

Shachi Kurl is executive director of the Angus Reid Institute; Ian Holliday is a research assistant. Read a longer version of this column at angusreid.org.

Advertisements
About Mel Rothenburger (4455 Articles)
ArmchairMayor.ca is a forum about Kamloops and the world. It has more than one million views. Mel Rothenburger is the former Editor of The Daily News in Kamloops, B.C. (retiring in 2012), and past mayor of Kamloops (1999-2005). At ArmchairMayor.ca he is the publisher, editor, news editor, city editor, reporter, webmaster, and just about anything else you can think of. He is grateful for the contributions of several local columnists. This blog doesn't require a subscription but gratefully accepts donations to help defray costs.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: