Item response theory for the analysis and construction of multidimensional forced-choice tests

Frick, Susanne

PDF
FrickSusanne_Dissertation_Druckversion.pdf - Published
Download (2MB)

URL:	https://madoc.bib.uni-mannheim.de/61151
URN:	urn:nbn:de:bsz:180-madoc-611519
Document Type:	Doctoral dissertation
Year of publication:	2021
Place of publication:	Mannheim
University:	Universität Mannheim
Evaluator:	Erdfelder, Edgar
Date of oral examination:	9 December 2021
Publication language:	English
Institution:	Außerfakultäre Einrichtungen > Graduiertenkolleg "Statistical Modeling in Psychology" (SMiP) School of Social Sciences > Psychologische Methodenlehre u. Diagnostik (Meiser 2009-)
License:	Creative Commons Attribution 4.0 International (CC BY 4.0)
Subject:	150 Psychology
Keywords (English):	multidimensional forced-choice , item response theory , Thurstonian IRT model , psychometrics
Abstract:	The multidimensional forced-choice (MFC) format has been proposed as an alternative to rating scales. In the MFC format, respondents indicate their relative preference for items measuring different attributes within blocks. Test construction for the MFC format is complex because how the items are combined affects the properties of the test. The aim of this thesis was to investigate and further develop IRT methods for the MFC format that can help to improve MFC test construction, focusing on the Thurstonian IRT model and a ranking instruction. In the first manuscript (Frick et al., 2021), we conducted an extensive simulation study on the normativity of Thurstonian IRT trait estimates. We investigated realistic test designs, removed a potential confounding with item parameter bias and compared recovery to that from classical test theory scoring and from rating scale and true-false formats. We found that with all positively keyed items, trait estimates showed ipsative properties. However, with mixed item keys, they were insensitive to otherwise suboptimal test designs. In an empirical study, we found that construct validity in the MFC format with three-item blocks was lower and criterion validity equal to the true-false format. In the second manuscript (Frick, 2021b), I developed the Faking Mixture model, a model for faking in the MFC format that allows to estimate the fakability of individual MFC blocks. A simulation study showed good parameter recovery. An empirical validation showed that the model can capture expected differences in item desirability, but also that matched blocks were not fully fake-proof. Therefore, it is worth to apply the Faking Mixture model in order to reduce fakability by removing or modifying blocks during test construction. In the third manuscript (Frick, 2021a), I proposed methods to estimate and summarize Fisher information for Thurstonian IRT models on the block level. Three simulation studies showed that the methods can accurately recover true information and are useful for test construction. It was examined how the proposed information summaries can be combined with algorithms for automated test assembly. Thus, block information can be used to assemble MFC tests that maximize reliability and have an ideal test design. In summary, this thesis provided both new methods and guidelines for MFC test construction. Modeling the block level did and will help to adequately capture the relative response process and item interactions and it can provide avenues for further psychometric developments.