Header

UZH-Logo

Maintenance Infos

Does understanding what a test measures make a difference? On the relevance of the ability to identify criteria for situational judgment test performance


Reznik, Nomi; Krumm, Stefan; Freudenstein, Jan‐Philipp; Heimann, Anna Luca; Ingold, Pia; Schäpers, Philipp; Kleinmann, Martin (2024). Does understanding what a test measures make a difference? On the relevance of the ability to identify criteria for situational judgment test performance. International Journal of Selection and Assessment, 32(2):210-224.

Abstract

Situational judgment tests (SJTs) are low‐fidelity simulations that are often used in personnel selection. Previous research has provided evidence that the ability to identify criteria (ATIC)—individuals' capability to detect underlying constructs in nontransparent personnel selection procedures—is relevant in simulations in personnel selection, such as assessment centers and situational interviews. Building on recent theorizing about response processes in SJTs as well as on previous empirical results, we posit that ATIC predicts SJT performance. We tested this hypothesis across two preregistered studies. In Study 1, a between‐subjects planned‐missingness design (N = 391 panelists) was employed and 55 selected items from five different SJTs were administered. Mixed‐effects‐modeling revealed a small effect for ATIC in predicting SJT responses. Results were replicated in Study 2 (N = 491 panelists), in which a complete teamwork SJT was administered with a high‐ or a low‐stakes instruction and showed either no or a small correlation with ATIC, respectively. We compare these findings with other studies, discuss implications for our understanding of response processes in SJTs, and derive avenues for future research.

Abstract

Situational judgment tests (SJTs) are low‐fidelity simulations that are often used in personnel selection. Previous research has provided evidence that the ability to identify criteria (ATIC)—individuals' capability to detect underlying constructs in nontransparent personnel selection procedures—is relevant in simulations in personnel selection, such as assessment centers and situational interviews. Building on recent theorizing about response processes in SJTs as well as on previous empirical results, we posit that ATIC predicts SJT performance. We tested this hypothesis across two preregistered studies. In Study 1, a between‐subjects planned‐missingness design (N = 391 panelists) was employed and 55 selected items from five different SJTs were administered. Mixed‐effects‐modeling revealed a small effect for ATIC in predicting SJT responses. Results were replicated in Study 2 (N = 491 panelists), in which a complete teamwork SJT was administered with a high‐ or a low‐stakes instruction and showed either no or a small correlation with ATIC, respectively. We compare these findings with other studies, discuss implications for our understanding of response processes in SJTs, and derive avenues for future research.

Statistics

Citations

Altmetrics

Downloads

2 downloads since deposited on 04 Jan 2024
2 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:06 Faculty of Arts > Institute of Psychology
Dewey Decimal Classification:150 Psychology
Scopus Subject Areas:Social Sciences & Humanities > General Business, Management and Accounting
Social Sciences & Humanities > Applied Psychology
Social Sciences & Humanities > General Psychology
Social Sciences & Humanities > Strategy and Management
Social Sciences & Humanities > Management of Technology and Innovation
Uncontrolled Keywords:ability to identify criteria, planned missingness, situational judgment test
Language:English
Date:1 June 2024
Deposited On:04 Jan 2024 15:11
Last Modified:08 May 2024 01:03
Publisher:Wiley-Blackwell Publishing, Inc.
ISSN:0965-075X
OA Status:Hybrid
Free access at:Publisher DOI. An embargo period may apply.
Publisher DOI:https://doi.org/10.1111/ijsa.12458
Project Information:
  • : FunderDeutsche Forschungsgemeinschaft
  • : Grant ID
  • : Project Title
  • Content: Published Version
  • Language: English
  • Licence: Creative Commons: Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)