Completing surveys and doing data entry for strangers online isn’t just a sideline. It appears to be key for many people to pay the bills.
Most respondents to a new survey of users of Amazon’s Mechanical Turk said that at least sometimes they count on income from the marketplace “to make ends meet,” in the survey’s language. The results came in a new report from the creators of Turkopticon released yesterday. Turkopticon is a browser extension available from the University of California at San Diego that workers on Mechanical Turk use to rate the trustworthiness of those who request work through the system.
The primary purpose of the survey was to assess Turkopticon, not Mechanical Turk. Turkopticon users have expressed confusion through the survey and forums about the best way to use the extension. “Some problems affect lots of people a bit. Other problems affect a few people a lot,” wrote the survey’s authors, UCSD professors Lilly Irani and Six Silberman, who also created the extension.
Mechanical Turk may be the most curious part of Jeff Bezos’ empire. Perhaps the earliest iteration of the on demand economy, the site was launched in 2005. It allows people to request work for small amounts of money—for example, getting paid $0.01 for each item the worker transcribes from shopping receipts. These are called “Human Intelligence Tasks” or “HITs.”
Workers can be paid to do almost anything, so long as they can do it through a computer. One artist paid 91 workers $0.30 apiece to assess dates she went on, in progress. Tech Republic announced this week that it would use the service to make predictions about the U.S. Election.
However, many of these workers have gone unpaid after completing tasks, as The American Prospect reported. To organize against wage theft, turkers (as they call themselves) organized via this rating system to warn each other away from requesters who don’t pay. As the Turkopticon website says, “Bad reviews let you avoid shady employers and good reviews help you find fair ones.”
To improve the extension, the 2016 Turkopticon Community Survey came out last night. Of the 1,068 respondents, over 90 percent were workers on the service. Most were located in North America, with smatterings in India and Europe. The survey primarily concerned the Turkopticon system, rather than workers experience with Mechanical Turk.
Some of the key findings:
- To make ends meet, 26.6 percent reported that they sometimes rely on it and 28.5 percent said they always rely on the income.
- Something like 15 percent of respondents worked more than 35 hours in the seven days prior to completing the survey (this result was only shown in a table, so exact numbers can only be guessed).
- Overall, workers are satisfied with Turkopticon’s performance, at a rate of 78.8 percent, but they had lots of suggestions. More than 10 percent were willing to volunteer some of their time to the service.
- A few requesters also completed the survey, but only 7 percent of the responses came from people who were primarily or always a requester.
According to the Mechanical Turk FAQ, “A Requester may reject your work if they believe the answer is wrong, the HIT was not completed correctly or that the instructions were not followed.” Turkopticon originated in the conviction among workers that requesters were abusing that authority by denying payment while using its results.
This has led to a debate among the workers about what it means to assess a requester for “fairness,” one of the four qualities on which each requester receives a numeric score from turkers using the extension (along with promptness, generosity and communicativity). There’s a balance that has to be struck between the degree to which a requester builds in quality checks and looks for reasons not to pay workers, which is illuminated in a discussion about “attention checks” on a Google Forum devoted to workers.
Comments published in the community survey show that Turkopticon relies on moderators to make sure that ratings and comments square with the stated definitions of each category, so that workers are able to understand the ratings they are viewing.
The fuzziness of defining these four qualities has resulted in a system where Turkopticon users appear to favor the text reviews over the numerical ratings. A quarter of respondents preferred the descriptions to the numerical ratings and 63 percent reported that descriptions and numerical ratings were equally important.
Irani and Silberman created Turkopticon in 2008, maintaining it ever since. In their intro to the survey results, they wrote that the survey “is one step in making Turkopticon something more decentralized, more worker governed and something more people can actively have a hand in shaping.”
Amazon was not immediately available for comment.