The Art Of Asking Questions

As we enter the ramp-up to the next presidential election, what we read and how we think will be increasingly influenced by polls. That matters, especially in primary season, as voters cast their ballots based not just on who they want to win, but on who they think can and will win in the general election. In this way, polls not only reflect beliefs, but influence the actions of primary voters.

Of course, after Donald Trump’s surprise victory in the 2016 election, many people challenged how accurately surveys measured sentiments and intentions. That’s why the Pew Research Center’s ongoing efforts to experiment with how they ask questions — to assess whether the phrasing and position of a question has a bearing on how people respond to it — is so praiseworthy.

In a recent study, Pew researchers evaluated the impact of alternate ways to ask about people’s belief in human evolution and the role that God plays in it. Their findings illustrate how important it is for researchers to be mindful of all the ways that context (where and in what sequence questions are asked) and selection bias (who is available and chooses to respond) determine outcomes. But they illustrate a more fundamental truth as well, which is that human beings are far more conflicted, nuanced and sometimes simply muddled in our beliefs than blunt instruments like surveys easily allow for.

...human beings are far more conflicted, nuanced and sometimes simply muddled in our beliefs than blunt instruments like surveys easily allow for.

Their experiment was simple.

Historically, Pew researchers first asked respondents if they believed that humans had evolved over time, or if they have existed in their present form since the beginning of time. Those choosing the former were then asked whether human evolution was a product of a process like natural selection — one in which God played no part — or whether humans had evolved thanks to a process allowed or guided by God or some other higher power.

But in a recent survey, half the respondents were asked that two-part question, while the other half  were asked a single question with three answer options and asked to choose the statement closest to their view: “Humans have evolved over time due to processes such as natural selection; God or a higher power had no role in this process”; “Humans have evolved over time due to processes that were guided or allowed by God or a higher power”; or “Humans have existed in their present form since the beginning of time.”

The experiment yielded dramatic results. Among those who were asked the two-part question above, 68 percent said that humans have evolved over time, while 31 percent chose the creationist stance that humans have always existed in their present form. In contrast, among those who were asked the single question with three answer options, 81 percent said that humans have evolved over time and only 18 percent chose the creationist position.

Why would simple differences in wording and survey structure lead to such a drop in the percentage of respondents who flat-out reject evolution?

Pew theorizes that for many Americans, religious convictions and scientific beliefs are inextricably bound. When these people are simply asked whether humans have evolved over time — the first question in the two-step process — they experience this as being implicitly forced to choose between their belief in evolution and their belief in God. Because the answers to the first question didn’t allow a role for God right out of the gate, they chose to reject the notion that humans had evolved over time. But when asked just a single question with answer options that allow for the two beliefs to co-exist, these respondents were more ready to acknowledge that evolution had occurred.

If we extrapolate from this example, we might imagine that the number of believers in climate change would also rise if people were asked whether it was real and caused by humans, real but not caused by humans, or not real at all.

But does that mean the question should be asked in this way? Should researchers routinely provide answer options that might defy facts but more closely hew to people’s actual beliefs?

If we are genuinely trying to understand what and how people think — whether about the environment or about political candidates — then we have to make space for the biases, experiences and implicit fears that inform people’s “rational” responses. Perhaps in doing so, we’ll even find some common ground or at least new ways to frame the questions that we as a nation need to decide.

Should researchers routinely provide answer options that might defy facts but more closely hew to people’s actual beliefs?

But as Pew has demonstrated, even the seemingly innocuous act of inquiring can reflect political intention and influence results. After all, a question like the infamous “Have you stopped beating your wife yet?” has no exonerating answer. And a question like “Will a border wall slow the massive influx of criminals from Latin America?” may reflect a respondent’s convictions, but also perpetuates a falsehood.

That’s why those who may be tempted to vote for primary candidates based on their electability should closely examine what that belief is based on. Even the most honest and meticulous of pollsters can get things wrong for a variety of reasons, as this excellent analysis of the 2016 polling data reveals. And the less scrupulous, or at least more partisan firms like Rasmussen (which was the closest to being right about the 2016 election, but has otherwise demonstrated a Republican bias that’s led to pretty consistently wrong results) can further skew our personal predictions about what other voters are likely to do.

In the 2016 primaries, many who voted for Clinton did so on the basis of pragmatism, our hands guided by heads stuffed full of polling data about electability. In 2020, we can choose a different path, not by guessing at the hearts and minds of others, but by learning how to change them.