Humm... I always wondered about the biases introduced by using Mechanical Turk for feature testing. I've seen it used a lot in several different contexts (like almost every other talk at WSCD'09 and WSDM'09). Isn't there a risk that we are training "professional beta testers" that can considerably skew the results?
It seems like a clear case of where the different parts have very different incentives. The hackers want to have the best possible sample of users, while the Turks want to get their respective tasks done as quickly as possible so they can move on to the next paying one. I would think this would lead to Turks specializing in very specific (and relatively common and repetitive) tasks do they could maximize their through put.
In your case, I would prefer to "watch" users out in the wild, instead of "professional" turks to truly assess the quality of a feature implementation.
Thats a good point, almost like testing with users that are power users for your software. This would be fine though if your target users have the same experience as the turk users.
It seems like a clear case of where the different parts have very different incentives. The hackers want to have the best possible sample of users, while the Turks want to get their respective tasks done as quickly as possible so they can move on to the next paying one. I would think this would lead to Turks specializing in very specific (and relatively common and repetitive) tasks do they could maximize their through put.
In your case, I would prefer to "watch" users out in the wild, instead of "professional" turks to truly assess the quality of a feature implementation.