|
Reliability and validity of a computer-based assessment of cognitive and non-cognitive facets of problem-solving competence in the business domain
Rausch, Andreas
;
Seifried, Jürgen
;
Wuttke, Eveline
;
Kögler, Kristina
;
Brandt, Steffen
|
DOI:
|
https://doi.org/10.1186/s40461-016-0035-y
|
|
URL:
|
https://ervet-journal.springeropen.com/articles/10...
|
|
Weitere URL:
|
https://link.springer.com/article/10.1186/s40461-0...
|
|
Dokumenttyp:
|
Zeitschriftenartikel
|
|
Erscheinungsjahr:
|
2016
|
|
Titel einer Zeitschrift oder einer Reihe:
|
Empirical Research in Vocational Education and Training
|
|
Band/Volume:
|
8
|
|
Heft/Issue:
|
9
|
|
Seitenbereich:
|
1-23
|
|
Ort der Veröffentlichung:
|
London [u.a.]
|
|
Verlag:
|
Springer
|
|
ISSN:
|
1877-6337 , 1877-6345
|
|
Sprache der Veröffentlichung:
|
Englisch
|
|
Einrichtung:
|
Fakultät für Betriebswirtschaftslehre > Wirtschaftspädagogik, Berufliches Lehren und Lernen (Seifried 2012-)
Fakultät für Betriebswirtschaftslehre > Wirtschaftspädagogik, Lernen im Arbeitsprozess (Rausch 2016-)
|
|
Fachgebiet:
|
370 Erziehung, Schul- und Bildungswesen
|
|
Freie Schlagwörter (Englisch):
|
Computer-based assessment , Problem-solving competence , Vocational education and training , Office work , Office simulation , Non-cognitive facets , Embedded experience sampling , Item response theory , Partial credits , Automated scoring
|
|
Abstract:
|
Background
To measure higher-order outcomes of vocational education and training (VET) we developed a computer-based assessment of domain-specific problem-solving competence. In modeling problem-solving competence, we distinguish four components of competence: (1) knowledge application, (2) metacognition, (3) self-concept, and (4) interest as well as thirteen facets of competence, each of which is assigned to one of the four components.
Methods
With regard to ecological and content validity, rather than apply highly structured items (e.g. multiple choice items), we developed three authentic problem scenarios and provided an open-ended problem space in terms of an authentic office simulation. The assessment was aimed at apprentice industrial clerks at the end of a 3-year apprenticeship program and focused on the domain of controlling (i.e., support of managerial decisions, cost planning, cost control, cost accounting, etc.). The computer-based office simulation provided typical tools (e.g., email client, spreadsheet software, file system, notebook, calculator, etc.). In order to assess the non-cognitive components in our competence model, we implemented an integrated measurement of self-concept and interest that we refer to as ‘Embedded Experience Sampling’ (EES). Test-takers are requested to spontaneously answer short prompts (EES items) during the test that are embedded in typical social interactions in the workplace. The empirical section is based on a study with 780 VET students from three commercial training occupations in Germany (industrial clerks and apprentices from two similar VET programs). The focus of the contribution is on testing a theoretically derived competence model based on item response theory, the implemented scoring methods and reliability of the instrument. Fine-grained response patterns from automated codings and human ratings were condensed into one partial credit item for each scenario and each of the facets in the cognitive component ‘knowledge application’.
Results
The multidimensional Rasch analysis revealed satisfactory EAP/PV reliabilities, which are between .78 and .84 for the ‘knowledge application’ facets and between .77 and .85 for the non-cognitive facets. Furthermore, the achievement differences between the industrial clerks and their comparison groups are as assumed.
Conclusions
In our study, we introduced an innovative method to measure non-cognitive facets of problem-solving competence in the course of complex problem scenarios. Furthermore, by using authentic problem scenarios and providing an open-ended and authentic problem space, our assessment of domain-specific problem-solving competence focuses on ecological validity but also ensures reliability.
|
 | Dieser Eintrag ist Teil der Universitätsbibliographie. |
 Suche Autoren in
BASE:
Rausch, Andreas
;
Seifried, Jürgen
;
Wuttke, Eveline
;
Kögler, Kristina
;
Brandt, Steffen
Google Scholar:
Rausch, Andreas
;
Seifried, Jürgen
;
Wuttke, Eveline
;
Kögler, Kristina
;
Brandt, Steffen
ORCID:
Rausch, Andreas  ORCID: 0000-0002-0749-2496 ; Seifried, Jürgen ORCID: 0000-0002-9460-7721 ; Wuttke, Eveline ; Kögler, Kristina ; Brandt, Steffen
Sie haben einen Fehler gefunden? Teilen Sie uns Ihren Korrekturwunsch bitte hier mit: E-Mail
Actions (login required)
 |
Eintrag anzeigen |
|
