Evaluating representation bias of different survey recruitment designs

Rohr, Björn

Preview

PDF
Dissertation_Rohr_2025_public.pdf - Published
Download (13MB)

URN:	urn:nbn:de:bsz:180-madoc-711931
Document Type:	Doctoral dissertation
Year of publication:	2025
Place of publication:	Mannheim
University:	Universität Mannheim
Evaluator:	Keusch, Florian
Date of oral examination:	2025
Publication language:	English
Institution:	School of Social Sciences > Social Data Science and Methodology (Keusch 2022-)
Subject:	300 Social sciences, sociology, anthropology
Individual keywords (German):	Auswahlverzerrung , Zufallsstichprobe , Nicht-Zufallsstichprobe , Facebook-Umfragen , Piggybacking-Rekrutierung , Verzerrung bei der Schätzung von Korrelationen
Keywords (English):	selection bias , probability surveys , nonprobability surveys , facebook surveys , piggybacking recruitment , bias in estimates of correlations
Abstract:	Most studies in the social sciences strongly rely on accurate and unbiased survey estimates. Whether you use probability or nonprobability sampling, each survey method comes with its own advantages and disadvantages, but will they yield accurate univariate, bivariate, or multivariate estimates? In this dissertation, I address this question by comparing surveys collected through various methods with external benchmarks. The first study compares probability and nonprobability surveys against population benchmarks to evaluate whether univariate, bivariate, and multivariate estimates yield similarly accurate results. In the second study, I compare non-probability surveys, recruited with advertisements on Facebook, with two different methods to target members of the target population. While the first method – Simple Demographic Targeting – is less complicated to conduct, the alternative method – Complex Demographic Targeting – was often used in the past, as it is suggested that this method might mitigate the bias introduced by the Facebook Ads Distribution Algorithm. Nonetheless, this suggestion was rarely evaluated in the past. In a third study, I analyze whether piggybacking, a cost-efficient method of conducting a probability survey, where the survey respondents are recruited at the end of another survey, leads to higher amounts of nonresponse bias in univariate and bivariate estimates. Finally, I provide practical guidance on the circumstances under which each of the compared methods is more or less suitable, and I outline avenues for further methodological development in the field.