Here you can openly access the main publication behind the Claims Reloaded toolkit.
Performance comparisons are fundamental in medical imaging Artificial Intelligence (AI) research, often driving claims of superiority based on relative improvements in common performance metrics. However, such claims frequently rely solely on empirical mean performance. In this paper, we investigate whether newly proposed methods genuinely outperform the state of the art by analyzing a representative cohort of medical imaging papers. We quantify the probability of false claims based on a Bayesian approach that leverages reported results alongside empirically estimated model congruence to estimate whether the relative ranking of methods is likely to have occurred by chance. According to our results, the majority (>80%) of papers claims outperformance when introducing a new method. Our analysis further revealed a high probability (>5%) of false outperformance claims in 86% of classification papers and 53% of segmentation papers. These findings highlight a critical flaw in current benchmarking practices: claims of outperformance in medical imaging AI are frequently unsubstantiated, posing a risk of misdirecting future research efforts.
Please cite the following paper if you use our online tool for a publication:
Christodoulou, E., Reinke, A., Andrè, P., Godau, P., Kalinowski, P., Houhou, R,. Erkan, S., Sudre, C.H., Burgos, N., Boutaj, S., Loizillon, S., Solal, M., Cheplygina, V., Heitz, C., Kozubek, M., Antonelli, M., Rieke, N., Gilson, A., Mayer, L.D., Tizabi, M.D., Cardoso, M. J., Simpson, A., Kopp-Schneider, A., Varoquaux, G., Colliot, O., Maier-Hein, L. False Promises in Medical Imaging AI? Assessing Validity of Outperformance Claims.(2025). https://arxiv.org/pdf/2505.04720
Bibtex entry:
@article{christodoulou2025false,
title={False Promises in Medical Imaging AI? Assessing Validity of Outperformance Claims},
author={Christodoulou, Evangelia and Reinke, Annika and Andr{\`e}, Pascaline and Godau, Patrick and Kalinowski, Piotr and Houhou, Rola and Erkan, Selen and Sudre, Carole H and Burgos, Ninon and Boutaj, Sofi{\`e}ne and others},
journal={arXiv preprint arXiv:2505.04720},
year={2025}
}