Worker Perception of Quality Assurance Mechanisms in Crowdsourcing and Human Computation Markets


Schulze, Thimo ; Nordheimer, Dennis ; Schader, Martin


[img]
Preview
PDF
schulze,_nordheimer,_schader_(2013)_-_Worker_Perception_of_Quality_Assurance_Mechanisms_in_Crowdsourcing_and_Human_Computation_Markets_(amcis).pdf - Published

Download (170kB)

URL: https://ub-madoc.bib.uni-mannheim.de/34231
Additional URL: http://aisel.aisnet.org/amcis2013/VirtualCommuniti...
URN: urn:nbn:de:bsz:180-madoc-342315
Document Type: Conference or workshop publication
Year of publication: 2013
Book title: 19th Americas Conference on Information Systems 2013 : AMCIS 2013 Proceedings
Page range: Paper 6
Conference title: 19th Americas Conference on Information Systems (AMCIS 2013)
Location of the conference venue: Chicago, Ill.
Date of the conference: August 15-17, 2013
Author/Publisher of the book
(only the first ones mentioned)
:
Shim, J. P.
Place of publication: Atlanta, Ga.
Publishing house: AISeL
ISBN: 978-0-615-55907-0
Publication language: English
Institution: Business School > Wirtschaftsinformatik III (Schader -2015)
Subject: 004 Computer science, internet
650 Management
Keywords (English): Quality Assurance , Survey , Crowdsourcing , Human Computation , Mechanical Turk , Majority Vote , Labor Psychology
Abstract: Many human computation systems utilize crowdsourcing marketplaces to recruit workers. Because of the open nature of these marketplaces, requesters need to use appropriate quality assurance mechanisms to guarantee high quality results. Previous research has mostly focused on the statistical aspects of quality assurance. Instead, we analyze the worker perception of five quality assurance mechanisms (Qualification Test, Qualification Restriction, Gold Standard, Majority Vote, Validating Review) according to subjective (fairness, offense, benefit) and objective (necessity, accuracy, cost) criteria. Based on theory from related areas like labor psychology, we develop a conceptual model and test it with a survey on Mechanical Turk. Our results show big differences in perception, especially with respect to Majority Vote which is rated low by workers. On the basis of these results, we show implications for theory and give requesters on crowdsourcing markets the advice to integrate the worker view when selecting an appropriate quality assurance mechanism.

Dieser Eintrag ist Teil der Universitätsbibliographie.

Das Dokument wird vom Publikationsserver der Universitätsbibliothek Mannheim bereitgestellt.




+ Citation Example and Export

Schulze, Thimo ; Nordheimer, Dennis ; Schader, Martin Worker Perception of Quality Assurance Mechanisms in Crowdsourcing and Human Computation Markets. Open Access Shim, J. P. Paper 6 In: 19th Americas Conference on Information Systems 2013 : AMCIS 2013 Proceedings (2013) Atlanta, Ga. 19th Americas Conference on Information Systems (AMCIS 2013) (Chicago, Ill.) [Conference or workshop publication]
[img]
Preview


+ Search Authors in

+ Download Statistics

Downloads per month over past year

View more statistics



You have found an error? Please let us know about your desired correction here: E-Mail


Actions (login required)

Show item Show item