Seminar in Psychometrics

Automatic Item Generation in Education: Current Techniques and Challenges

Date and time: Monday, April 7, 2025 (2:30 PM CET)
Place ICS CAS room 318, Pod Vodárenskou věží 2, Prague 8, also on Zoom.

Abstract:

The recent publication of large language models (LLMs) such as GPT-4 provides new opportunities for processing and generating texts, pictures, and other documents. An important application of this technology in education, psychology, and related fields is the generation of educational materials, such as new items for psychological and educational assessments. While generating new items using LLMs is relatively straightforward, several important challenges remain, including: a) predicting the psychometric characteristics of items, such as their difficulty and threshold; b) filtering items with more than one solution or other characteristics unsuitable for practical assessments; and c) integrating non-verbal materials, such as pictures. This talk discusses and illustrates current approaches to these challenges, drawing on examples from recent studies.

References:

Attali, Y., Runge, A., LaFlair, G. T., Yancey, K., Goodwin, S., Park, Y. and von Davier, A. A. (2022). The interactive reading task: Transformer-based automatic item generation. Front. Artif. Intell. 5:903077. doi: 10.3389/frai.2022.903077

von Davier, A. A., Runge, A., Park, Y., Attali, Y., Church, J., & LaFlair, G. (2024). The item factory: Intelligent automation in support of test development at scale. In: Hong Jiao & Robert W. Lissitz (Eds.), Machine learning, natural language processing, and psychometrics (pp. 1-25). Information Age Publishing. https://www.infoagepub.com/products/Machine-Learning-Natural-Language-Processing-and-Psychometrics

Rudolf Debelak

I am am Advanced Senior Scientist and Team Leader at the University of Zurich and EPFL, with a strong background in psychology, mathematics, statistics, and machine learning, particularly in R and Python. In my current role, I lead a research team focused on developing and applying machine learning methods and designing products in education and psychology. Prior to my academic career, I spent several years in the psychological testing industry, where I contributed to the development of software for psychological assessments.