Seminar in Psychometrics

COMPS seminar main page

Bayesian Nonparametric Models for Multiple Raters Data

Date and time: December 5, 2024 (4:00 PM CET)
Place ICS CAS room 318, Pod Vodárenskou věží 2, Prague 8, also on Zoom.

Abstract:

Rating procedure is crucial in many applied fields (e.g., educational, clinical, emergency). It implies that a rater (e.g., teacher, doctor) rates a subject (e.g., student, doctor) on a rating scale. Given raters’ variability, several statistical methods have been proposed for assessing and improving the quality of ratings. The analysis and the estimate of the Intraclass Correlation Coefficient (ICC) are major concerns in such cases. As evidenced by the literature, ICC might differ across different subgroups of raters and might be affected by contextual factors and subject heterogeneity. Model estimation in the presence of heterogeneity has been one of the recent challenges in this research line. Consequently, several methods have been proposed to address this issue under a parametric multilevel modelling framework, in which strong distributional assumptions are made. We propose a more flexible model under the Bayesian nonparametric (BNP) framework, in which most of those assumptions are relaxed. By eliciting hierarchical discrete nonparametric priors, the model accommodates clusters among raters and subjects, naturally accounts for heterogeneity, and improves estimates' accuracy. We propose a general BNP heteroscedastic framework to analyse rating data and possible latent differences among subjects and raters. Theoretical results about the ICC are provided together with computational strategies. Simulations results and a real-world application are presented, and possible further investigations are discussed.

References:

Mignemi, G., & Manolopoulou, I. (2024). Bayesian Nonparametric Models for Multiple Raters: a General Statistical Framework. arXiv preprint arXiv:2410.21498.

Yang, M., Dunson, D. B., & Baird, D. (2010). Semiparametric bayes hierarchical models with mean and variance constraints. Computational Statistics & Data Analysis, 54(9), 2172-2186. https://doi.org/10.1016/j.csda.2010.03.025.

ten Hove, D., Jorgensen, T. D., & van der Ark, L. A. (2024). Updated guidelines on selecting an intraclass correlation coefficient for interrater reliability, with applications to incomplete observational designs. Psychological methods, 29(5), 967–979. https://doi.org/10.1037/met0000516.