Worker Perception of Quality Assurance Mechanisms in Crowdsourcing and Human Computation Markets

Schulze, Thimo ; Nordheimer, Dennis ; Schader, Martin

schulze,_nordheimer,_schader_(2013)_-_Worker_Perception_of_Quality_Assurance_Mechanisms_in_Crowdsourcing_and_Human_Computation_Markets_(amcis).pdf - Published

Download (170kB)

Additional URL:
URN: urn:nbn:de:bsz:180-madoc-342315
Document Type: Conference or workshop publication
Year of publication: 2013
Book title: 19th Americas Conference on Information Systems 2013 : AMCIS 2013 Proceedings
Page range: Paper 6
Conference title: 19th Americas Conference on Information Systems (AMCIS 2013)
Location of the conference venue: Chicago, Ill.
Date of the conference: August 15-17, 2013
Publisher: Shim, J. P.
Place of publication: Atlanta, Ga.
Publishing house: AISeL
ISBN: 978-0-615-55907-0
Publication language: English
Institution: Business School > Wirtschaftsinformatik III (Schader 1991-2015, Em)
Subject: 004 Computer science, internet
650 Management
Keywords (English): Quality Assurance , Survey , Crowdsourcing , Human Computation , Mechanical Turk , Majority Vote , Labor Psychology
Abstract: Many human computation systems utilize crowdsourcing marketplaces to recruit workers. Because of the open nature of these marketplaces, requesters need to use appropriate quality assurance mechanisms to guarantee high quality results. Previous research has mostly focused on the statistical aspects of quality assurance. Instead, we analyze the worker perception of five quality assurance mechanisms (Qualification Test, Qualification Restriction, Gold Standard, Majority Vote, Validating Review) according to subjective (fairness, offense, benefit) and objective (necessity, accuracy, cost) criteria. Based on theory from related areas like labor psychology, we develop a conceptual model and test it with a survey on Mechanical Turk. Our results show big differences in perception, especially with respect to Majority Vote which is rated low by workers. On the basis of these results, we show implications for theory and give requesters on crowdsourcing markets the advice to integrate the worker view when selecting an appropriate quality assurance mechanism.

Dieser Eintrag ist Teil der Universitätsbibliographie.

Das Dokument wird vom Publikationsserver der Universitätsbibliothek Mannheim bereitgestellt.

Metadata export


+ Search Authors in

+ Download Statistics

Downloads per month over past year

View more statistics

You have found an error? Please let us know about your desired correction here: E-Mail

Actions (login required)

Show item Show item