Header

UZH-Logo

Maintenance Infos

On-the-fly calibration in computerized adaptive testing


Verschoor, Angela; Berger, Stéphanie; Moser, Urs; Kleintjes, Frans (2019). On-the-fly calibration in computerized adaptive testing. In: Veldkamp, B P; Sluijter, C. Theoretical and Practical Advances in Computer-based Educational Measurement. Cham: Springer, 307-323.

Abstract

Research on Computerized Adaptive Testing (CAT) has been rooted in a long tradition. Yet, current operational requirements for CATs make the production a relatively expensive and time consuming process. Item pools need a large number of items, each calibrated with a large degree of accuracy. Using on-the-fly calibration might be the answer to reduce the operational demands for the production of a CAT. As calibration is to take place in real time, a fast and simple calibration method is needed. Three methods will be considered: Elo chess ratings, Joint Maximum Likelihood (JML), and Marginal Maximum Likelihood (MML). MML is the most time consuming method, but the only known method to give unbiased parameter estimates when calibrating CAT data. JML gives biased estimates although it is faster than MML, while the updating of Elo ratings is known to be even less time consuming. Although JML would meet operational requirements for running a CAT regarding computational performance, its bias and its inability to estimate parameters for perfect or zero scores makes it unsuitable as a strategy for on-the-fly calibration. In this chapter, we propose a combination of Elo rating and JML as a strategy that meets the operational requirements for running a CAT. The Elo rating is used in the very beginning of the administration to ensure that the JML estimation procedure is converging for all answer patterns. Modelling the bias in JML with the help of a relatively small, but representative set of calibrated items is proposed to eliminate the bias.

Abstract

Research on Computerized Adaptive Testing (CAT) has been rooted in a long tradition. Yet, current operational requirements for CATs make the production a relatively expensive and time consuming process. Item pools need a large number of items, each calibrated with a large degree of accuracy. Using on-the-fly calibration might be the answer to reduce the operational demands for the production of a CAT. As calibration is to take place in real time, a fast and simple calibration method is needed. Three methods will be considered: Elo chess ratings, Joint Maximum Likelihood (JML), and Marginal Maximum Likelihood (MML). MML is the most time consuming method, but the only known method to give unbiased parameter estimates when calibrating CAT data. JML gives biased estimates although it is faster than MML, while the updating of Elo ratings is known to be even less time consuming. Although JML would meet operational requirements for running a CAT regarding computational performance, its bias and its inability to estimate parameters for perfect or zero scores makes it unsuitable as a strategy for on-the-fly calibration. In this chapter, we propose a combination of Elo rating and JML as a strategy that meets the operational requirements for running a CAT. The Elo rating is used in the very beginning of the administration to ensure that the JML estimation procedure is converging for all answer patterns. Modelling the bias in JML with the help of a relatively small, but representative set of calibrated items is proposed to eliminate the bias.

Statistics

Citations

Dimensions.ai Metrics

Altmetrics

Downloads

9 downloads since deposited on 08 Jan 2020
9 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Book Section, refereed, original work
Communities & Collections:06 Faculty of Arts > Institute of Educational Evaluation
Dewey Decimal Classification:370 Education
Language:English
Date:1 January 2019
Deposited On:08 Jan 2020 08:50
Last Modified:08 Jan 2020 08:50
Publisher:Springer
ISBN:9783030184797
OA Status:Hybrid
Publisher DOI:https://doi.org/10.1007/978-3-030-18480-3_16

Download

Hybrid Open Access

Download PDF  'On-the-fly calibration in computerized adaptive testing'.
Preview
Content: Published Version
Language: English
Filetype: PDF
Size: 429kB
View at publisher
Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)
Get full-text in a library