Post A Survey On Mechanical Turk And Watch The Results Roll In05:32

You can buy just about anything on Amazon.com — clothes, books, electronics. You can buy answers, too. College students and professors are doing all sorts of research on an Amazon site called Mechanical Turk.

Need 200 smokers for your survey on lung cancer? Have a moral dilemma to pose for your paper on Kierkegaard? Now researchers can log in, offer a few pennies in payment and watch the data roll in.

When you type in the Web address MechanicalTurk.com, you'll see a page full of gray rectangles, each one a different task. When I visited the site recently, I see one out of UCLA. They're paying 25 cents for a 5-minute survey on consumer decision-making. The next one is 10 cents and it's about dating preferences. When I scroll down, there's one from Ohio State paying 50 cents for workers to fill out a survey about their political beliefs.

'It's Very Fast. It's Very Cheap'

"It's very efficient. It's very fast. It's very cheap," says Adam Berinsky, a professor at MIT who studies public opinion. Some of those gray rectangles are his studies and surveys.

"There's been a huge explosion. So about four years ago, not a lot of folks were doing it. But now everyone I talk to at conferences is doing that," he says.

It all started about a decade ago. Amazon had a problem. It has millions of Web pages, each with a product for sale, but some were identical. The company wanted to weed out those duplicates, which is hard for computers but easy for people to do. So Amazon offered to pay people a few cents to do it for the company. Soon, other companies were posting tasks too and Mechanical Turk was born.

Why the name? Well, it references a famous hoax from the 1700s. There was a chess playing machine that could beat everyone. But actually, it was a chess master hiding inside. So, too, with Mechanical Turk. It seems like a computer is doing the work but it's actually a person.

"I do between six to eight hours and earn about $20 a day," says Ana Vasquez, who is unemployed. Between job interviews she Turks from her Brooklyn apartment. She does the math. She makes roughly $2.50 an hour.

"Which is basically, I guess, waitressing without the tips," she says.

'I Need The Money'

Mechanical Turk, or MTurk as it's called, has been criticized as a digital sweatshop. Vasquez knows the pay is peanuts but she really wants a job in social services.

"I'm enjoying the flexibility to keep pursuing that and not have to, say, choose something right now because I need the money," Vasquez says.

She's strategic. There are lots of different tasks on MTurk, such as Googling websites to boost their ratings, transcribing and copy editing. Research studies pay the most, Vasquez says.

"Recently, I did one on memory and it just involved clicking different numbers on the screen. It's probably for someone's Ph.D. or something like that," Vasquez says.

Quite likely. Jonathan Phillips is a Ph.D. student at Yale and he uses MTurk all the time. Compared to what he did before, it's a dream, he says. He used to print out questionnaires and hit the street.

"You'd spend two to three hours before you got too exhausted to continue and you'd get 15, 20 responses," Phillips says. "It's just so painstaking."

Not anymore.

How Reliable Are The Data?

"You come up with a study and then, an hour later you have, like, 200 responses, which is pretty fundamentally different," he says.

Berinsky, over at MIT, says researchers save not just weeks of effort, but great amounts of money as well. He's pays a couple cents per participant, compared to the $10 or $15 he used to pay.

OK, so MTurk is fast and cheap. Is it good? How reliable are the data?

The research shows that the population of Turkers is pretty representative, more so than signing up college students. There are other benefits too.

"I think it's actually improved the extent to which we can really be sure about our results," Berinsky says.

From his tucked away office, Phillips gives me an example. Say you put up a survey but the wording isn't quite right. That could skew the findings.

"People are much more willing to say, 'Well, I didn't really like that you used that one word there. So can you just change it and rerun it on Mechanical Turk,' knowing that it's not going to take you that long?"

Taking More Chances

Berinsky has another upside. "It's enabled people to take more chances. To do things that might seem crazy or sound crazy but are really clever and interesting and turn out to be right," Berinsky says.

What about the downsides? Maybe it's made research too easy and too cheap. Berinsky says some scholars might be putting up bad surveys.

"Maybe it would be better if people had to invest a little more of their own money before they ran with an idea," he says.

Not all research is suited to Mechanical Turk. If your study involves children, that's a no go. Need to observe your subject's facial expressions? Can't do that on MTurk. Those researchers can, of course, just do their research the old-fashioned way. But Phillips worries some researchers might be shying away from tough topics.

"I don't think we have any good sense of exactly which questions are now being neglected because they are harder to study," he says. "But I think it would be crazy to deny that it's having an influence on which questions we ask and how we ask them."

There's another change underway. We think about academics toiling away in a cubicle or sterile laboratories. One of Berinsky's colleagues surveys American voters on MTurk — but he does it from Indonesia. He might even be on the beach.

Copyright NPR 2022.