Comments for Still No Killer Apps For Amazon Mechanical Turk

Still No Killer Apps For Amazon Mechanical Turk

2007-01-11T22:54:23Z

HIT stands for Human Intelligence Task and it's the term for a small job at Mechanical Turk, Amazon's AI task management service. According to Wikipedia's definition, "Mturk enables computer programs to co-ordinate the use of human intelligence to perform tasks which computers are unable to do." The payments are micro, but the idea is that they are tasks which groups of people would do for fun or to ease boredom.

Since its launch on 2 November 2005, Mechanical Turk has gradually built up a following - there is a forum for "Turkers" called Turker Nation, which appears to have light-to-medium level patronage.

However it's fair to say that Mturk isn't being used as much as the initial hype period in Nov-Dec 05. There are 259 current HITs, but even Turk fans seem to lament the lack of quality HITs. Says Turk Lurker:

"Personally, I've been scanning the available hits every few weeks myself and there's just doesn't seem like there's much worth doing out there right now. Still no "killer apps," which has plagued the project since the beginning."

]]> OpenGeoData writes about an interesting HIT currently available, from a company called Geospatial Vision - which is micro-paying people to do image recognition on sequential video stills. Notes OpenGeoData:

"You are paid 5 cents to tag 50 images with yellow lines, manholes, drains, bollards and pedestrian crossings. They are also, from looking at the videos, using these locations to then magically classify the sign type (one way, no entry, speed limit etc). Most images have only one feature if at all, there were about 2,000 HITs last night and at 25 frames a second that puts it at about an hour of footage for $100. That is insane."

The assignment itself states that "you will be given 50 images and asked to go through each image clicking on specific objects if you see them."

I'm not sure what would motivate a person to do a task like this, to be honest. It would probably take you half an hour or so to scroll through 50 images and click on specific objects (including mucking around getting set up). And for that you get 5c.

So it looks like Mechanical Turk is struggling to get any real traction, which makes me wonder if all the initial hype about it was overblown?

Comment from Emre Sokullu on 2007-01-11

2007-01-12T04:34:57Z

Pay per AI help idea of Jeff Bezos haven't worked so far. ChaCha follows the same pattern as well. The reason may be false marketing, they should market these products not in Silicon Valley but in 3rd world countries - people are not tended to do so boring jobs for so low prices - but this might not be the case in other places.

Comment from Indimingle on 2007-01-11

2007-01-12T06:04:54Z

It's just not worth it - probably not even for those who live in India! I reviewed it when it launched and the tasks are just not paying well enough. Granted there are some things that humans can do better than machines, there's a reason why it costs so much more. :-)

Comment from Peter Cooper on 2007-01-12

2007-01-12T08:52:31Z

I think there needs to be a system like Mechanical Turk that's a little higher up the food chain. That is, more into the area of 50 cents and higher per action. An eBay style marketplace for people's time. Perhaps I want some random person to test my webapp, for example, so I could offer that as a task for, say, $10, and then people can pick and choose tasks they want to do.

I have TONS of jobs I'd give out to the marketplace if I could deliver them to a market of workers piecemeal on a non-regular basis.. but instead I do them all myself as I don't want the headache of hiring and maintaining people when they're not needed.

Comment from Christian Flury on 2007-01-12

2007-01-12T13:55:51Z

Spot on! I, too, had a look at it quite a while ago shortly after it had been launched - and to my mind the main problem was a flawed concept of human intelligence: there are hardly any tasks that ANY human being could perform better than a computer, there's always a certain skill set involved, and in order to make sure the person performing a tasks does possess the necessary skills, humans constantly try to assess each other and build trust. In order for the Mechanical Turk Idea to work, it would have to replicate these mechanisms of human interaction in a way or another. Only then will clients be willing to pay better, and only then will it attract properly skilled persons.

Comment from Robert on 2007-01-12

2007-01-12T17:52:36Z

"..not in Silicon Valley but in 3rd world countries but in 3rd world countries"

Could you earn enough on Mechanical Turk in an hour to pay for an hour of internet use in a 3rd world country? It seems like people are making maybe $3 or $4 dollars an hour on Turk.

Comment from Billy on 2007-01-13

2007-01-13T18:05:47Z

Mech Turk doesn't require tasks that pay such low rates, those just happen to be the tasks put into the marketplace. So if there are tasks that you'd be willing to pay $1 for, you could use Mech Turk for that. Then issues of turker capability and fraud likely become a bigger issues.

I agree the 3rd world would be a great place to focus Mech Turk, but before they do that they need to find more tasks. Someone should set up 'Turk iCafes' in the 3rd world where people can come in, earn money using Turk, and the cafe takes a cut for providing the Internet access.

Comment from Jane on 2007-01-13

2007-01-13T18:08:15Z

Great idea and they seemed to have executed well on the very first version, but since its launched there seems to have been no progress.

So to me, this means one of the following:

1. It is no longer receiving attention (they have given up on it)
2. There was some sort of change in the team (people left after the launch and the new people can't execute)

Yeah, it wasn't a slam dunk success right out of the gates but it does have potential. Doesn't seem like the folks responsible for it know what to do with the potential.

Comment from Thomas on 2007-01-15

2007-01-15T17:24:51Z

Speaking as someone who's used the Mechanical Turk APIs a little bit: keep in mind that what you see at the Mechanical Turk site as a worker is really less than 1/2 of the product. The other part is the web service that lets you put work into the system and to pay for it using an API. They've been making decent progress on this part of the product, though not great, and they don't really seem all interested in what customers actually want. It's kind of a pain to use. It does have a sort of side-project feel to it all.

But it's cheap, and kind of fun if you have certain jobs that work well with it.