Header

UZH-Logo

Maintenance Infos

A Comparison of Aggregation Rules for Selecting Anchor Items in Multigroup DIF Analysis


Huelmann, Thorben; Debelak, Rudolf; Strobl, Carolin (2019). A Comparison of Aggregation Rules for Selecting Anchor Items in Multigroup DIF Analysis. Journal of Educational Measurement:Epub ahead of print.

Abstract

This study addresses the topic of how anchoring methods for differential item functioning (DIF) analysis can be used in multigroup scenarios. The direct approach would be to combine anchoring methods developed for two‐group scenarios with multigroup DIF‐detection methods. Alternatively, multiple tests could be carried out. The results of these tests need to be aggregated to determine the anchor for the final DIF analysis. In this study, the direct approach and three aggregation rules are investigated. All approaches are combined with a variety of anchoring methods, such as the “all‐other purified” and “mean p‐value threshold” methods, in two simulation studies based on the Rasch model. Our results indicate that the direct approach generally does not lead to more accurate or even to inferior results than the aggregation rules. The min rule overall shows the best trade‐off between low false alarm rate and medium to high hit rate. However, it might be too sensitive when the number of groups is large. In this case, the all rule may be a good compromise. We also take a closer look at the anchor selection method “next candidate,” which performed rather poorly, and suggest possible improvements.

Abstract

This study addresses the topic of how anchoring methods for differential item functioning (DIF) analysis can be used in multigroup scenarios. The direct approach would be to combine anchoring methods developed for two‐group scenarios with multigroup DIF‐detection methods. Alternatively, multiple tests could be carried out. The results of these tests need to be aggregated to determine the anchor for the final DIF analysis. In this study, the direct approach and three aggregation rules are investigated. All approaches are combined with a variety of anchoring methods, such as the “all‐other purified” and “mean p‐value threshold” methods, in two simulation studies based on the Rasch model. Our results indicate that the direct approach generally does not lead to more accurate or even to inferior results than the aggregation rules. The min rule overall shows the best trade‐off between low false alarm rate and medium to high hit rate. However, it might be too sensitive when the number of groups is large. In this case, the all rule may be a good compromise. We also take a closer look at the anchor selection method “next candidate,” which performed rather poorly, and suggest possible improvements.

Statistics

Citations

Altmetrics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:06 Faculty of Arts > Institute of Psychology
Dewey Decimal Classification:150 Psychology
Uncontrolled Keywords:Applied Psychology, Psychology (miscellaneous), Education, Developmental and Educational Psychology
Language:English
Date:5 September 2019
Deposited On:12 Nov 2019 14:11
Last Modified:12 Nov 2019 14:16
Publisher:Wiley-Blackwell Publishing, Inc.
ISSN:0022-0655
OA Status:Closed
Publisher DOI:https://doi.org/10.1111/jedm.12246

Download

Full text not available from this repository.
View at publisher

Get full-text in a library