Seminar in Psychometrics
Methodological innovations in decision-making for research funding with a focus on transparency and reproducibility
Date and time: Monday, March 31, 2025 (4:00 PM CET)
Place ICS CAS room 318, Pod Vodárenskou věží 2, Prague 8, also on Zoom.
Decisions for competitive funding by national and international scientific funding organisations shape the research landscape, as they determine which research is done, and by whom. It is crucial that the funding decision-making process is conducted with care and accuracy, and funding systems are carefully designed to ensure limited resources are allocated according to the funding organisation’s priorities. Due to a lack of in-house expertise and availability of ready-to-use tools, many funding organisations rely on rather simplistic methodologies to aggregate expert evaluations into funding rankings and decisions. The aim of this presentation is to introduce the methodological gaps in research funding decision-making, and to discuss the limitations and opportunities of recently proposed methodological innovations. We will specifically look at the potential of Bayesian ranking procedures and how they can be used to make funding recommendations that take uncertainty and measurement error into account [1,2]. The presentation will include a demo of the methodology's R-implementation [3].
References:[1] Heyard, R., Ott, M., Salanti, G., & Egger, M. (2022). Rethinking the funding line at the Swiss national science foundation: Bayesian ranking and lottery. Statistics and Public Policy, 9(1), 110-121. https://doi.org/10.1080/2330443X.2022.2086190
[2] Heyard, R., Pina, D., Buljan, I., & Marusic, A. (2024). Assessing the potential of a Bayesian ranking as an alternative to consensus meetings for decision making in research funding: A Case Study of Marie Skłodowska-Curie Actions (To appear in PLOS ONE) https://doi.org/10.31222/osf.io/c4ujg
[3] ERforResearch https://snsf-data.github.io/ERforResearch/
Rachel Heyard
Rachel Heyard holds a PhD in Biostatistics and is currently a postdoctoral fellow in meta-research at the https://www.crs.uzh.ch/en.html, University of Zurich (UZH). After finalising her PhD at UZH in 2019, she left academia to join the data team of the Swiss National Science Foundation (SNSF) as a statistician. While working on diverse data and statistics projects informing science policy, she discovered her passion for meta-research - the interdisciplinary study of research itself. After a bit more than three years at the SNSF, she decided to go back to academia and started a postdoc in meta-research at the Center for Reproducible Science, where she teaches Good Research Practices and continues doing meta-research. Her research interests include developing and testing new approaches for allocating research funding, statistical methods in research, and improving reproducibility and the uptake of Open Science. She is further the vice-president of the https://stat.ch/, part of the steering committee of the CoARA (Coalition for Advancing Research Assessment) working group on https://github.com/rachelHey/metricsWG, the organiser of the Zurich ReproducibiliTea Seminar Series, and the co-organiser of the https://open.science-retreat.org/ (Swiss-edition).