77 commitsLast commit ≈ 72 months ago39 stars8 forks
challengeR
challengeR: Methods and open-source toolkit for analyzing and visualizing challenge results
11
mentions
Description
challengeR is a tool for analyzing and visualizing challenge results in the field of image analysis and beyond.
Challenges have become the de facto standard for benchmarking image analysis algorithms. While the number of challenges is steadily increasing, surprisingly little effort has been invested in ensuring high quality design, execution and reporting for these international competitions. Specifically, results analysis and visualization in the event of uncertainties have been given almost no attention in the literature.
Given these shortcomings, challengeR aims to catalyze fast and widespread adoption of comprehensive analysis & visualization of competition results. challengeR offers an intuitive way to gain important insights into the relative and absolute performance of algorithms, including ranking uncertainties, which cannot be revealed by commonly applied visualization techniques.
Podium plot generated by challengeR © Division of Intelligent Medical Systems (IMSY) at DKFZ Heidelberg
Find challengeR on Helmholtz Imaging Connect
Authors: Wiesenfarth, M., Reinke, A., Landman, B.A., Eisenmann, M., Aguilera Saiz, L., Cardoso, M.J., Maier-Hein, L. and Kopp-Schneider, A., Kavur A.E
License
- GPLv2 or any later version
Participating organisations
Reference papers
Mentions
- 1.Author(s): Lena Maier-Hein, Annika Reinke, Patrick Godau, Minu D. Tizabi, Florian Buettner, Evangelia Christodoulou, Ben Glocker, Fabian Isensee, Jens Kleesiek, Michal Kozubek, Mauricio Reyes, Michael A. Riegler, Manuel Wiesenfarth, A. Emre Kavur, Carole H. Sudre, Michael Baumgartner, Matthias Eisenmann, Doreen Heckmann-Nötzel, Tim Rädsch, Laura Acion, Michela Antonelli, Tal Arbel, Spyridon Bakas, Arriel Benis, Matthew B. Blaschko, M. Jorge Cardoso, Veronika Cheplygina, Beth A. Cimini, Gary S. Collins, Keyvan Farahani, Luciana Ferrer, Adrian Galdran, Bram van Ginneken, Robert Haase, Daniel A. Hashimoto, Michael M. Hoffman, Merel Huisman, Pierre Jannin, Charles E. Kahn, Dagmar Kainmueller, Bernhard Kainz, Alexandros Karargyris, Alan Karthikesalingam, Florian Kofler, Annette Kopp-Schneider, Anna Kreshuk, Tahsin Kurc, Bennett A. Landman, Geert Litjens, Amin Madani, Klaus Maier-Hein, Anne L. Martel, Peter Mattson, Erik Meijering, Bjoern Menze, Karel G. M. Moons, Henning Müller, Brennan Nichyporuk, Felix Nickel, Jens Petersen, Nasir Rajpoot, Nicola Rieke, Julio Saez-Rodriguez, Clara I. Sánchez, Shravya Shetty, Maarten van Smeden, Ronald M. Summers, Abdel A. Taha, Aleksei Tiulpin, Sotirios A. Tsaftaris, Ben Van Calster, Gaël Varoquaux, Paul F. JägerPublished in 202410.1038/s41592-023-02151-z
- 2.Author(s): Annika Reinke, Minu D. Tizabi, Michael Baumgartner, Matthias Eisenmann, Doreen Heckmann-Nötzel, A. Emre Kavur, Tim Rädsch, Carole H. Sudre, Laura Acion, Michela Antonelli, Tal Arbel, Spyridon Bakas, Arriel Benis, Florian Buettner, M. Jorge Cardoso, Veronika Cheplygina, Jianxu Chen, Evangelia Christodoulou, Beth A. Cimini, Keyvan Farahani, Luciana Ferrer, Adrian Galdran, Bram van Ginneken, Ben Glocker, Patrick Godau, Daniel A. Hashimoto, Michael M. Hoffman, Merel Huisman, Fabian Isensee, Pierre Jannin, Charles E. Kahn, Dagmar Kainmueller, Bernhard Kainz, Alexandros Karargyris, Jens Kleesiek, Florian Kofler, Thijs Kooi, Annette Kopp-Schneider, Michal Kozubek, Anna Kreshuk, Tahsin Kurc, Bennett A. Landman, Geert Litjens, Amin Madani, Klaus Maier-Hein, Anne L. Martel, Erik Meijering, Bjoern Menze, Karel G. M. Moons, Henning Müller, Brennan Nichyporuk, Felix Nickel, Jens Petersen, Susanne M. Rafelski, Nasir Rajpoot, Mauricio Reyes, Michael A. Riegler, Nicola Rieke, Julio Saez-Rodriguez, Clara I. Sánchez, Shravya Shetty, Ronald M. Summers, Abdel A. Taha, Aleksei Tiulpin, Sotirios A. Tsaftaris, Ben Van Calster, Gaël Varoquaux, Ziv R. Yaniv, Paul F. Jäger, Lena Maier-HeinPublished in 202410.1038/s41592-023-02150-0
- 3.Author(s): Bo Qian, Hao Chen, Xiangning Wang, Zhouyu Guan, Tingyao Li, Yang Wen, Haoxuan Che, Gitaek Kwon, Jaeyoung Kim, Sungjin Choi, Seoyoung Shin, Felix Krause, Markus Unterdechler, Junlin Hou, Rui Feng, Yihao Li, Mostafa El Habib Daho, Dawei Yang, Qiang Wu, Ping Zhang, Xiaokang Yang, Yiyu Cai, Gavin Siew Wei Tan, Carol Y. Cheung, Wei-Ping Jia, Huating Li, Yih Chung Tham, Tien Yin Wong, Bin ShengPublished in 202310.2139/ssrn.4593632
- 4.Author(s): Arnaud Huaulmé, Kanako Harada, Quang-Minh Nguyen, Bogyu Park, Seungbum Hong, Min-Kook Choi, Michael Peven, Yunshuang Li, Yonghao Long, Qi Dou, Satyadwyoom Kumar, Seenivasan Lalithkumar, Ren Hongliang, Hiroki Matsuzaki, Yuto Ishikawa, Yuriko Harai, Satoshi Kondo, Mamoru Mitsuishi, Pierre JanninPublished in 202210.2139/ssrn.4088403
- 5.Author(s): Holger Roth, Ziyue Xu, Carlos Tor Diez, Ramon Sanchez Jacob, Jonathan Zember, Jose Molto, Wenqi Li, Sheng Xu, Baris Turkbey, Evrim Turkbey, Dong Yang, Ahmed Harouni, Nicola Rieke, Shishuai Hu, Fabian Isensee, Claire Tang, Qinji Yu, Jan Sölter, Tong Zheng, Vitali Liauchuk, Ziqi Zhou, Jan Moltz, Bruno Oliveira, Yong Xia, Klaus Maier-Hein, Qikai Li, Andreas Husch, Luyang Zhang, Vassili Kovalev, Li Kang, Alessa Hering, João Vilaça, Mona Flores, Daguang Xu, Bradford Wood, Marius LinguraruPublished in 202110.21203/rs.3.rs-571332/v1
- 6.Author(s): Lena Maier-Hein, Martin Wagner, Tobias Ross, Annika Reinke, Sebastian Bodenstedt, Peter M. Full, Hellena Hempe, Diana Mindroc-Filimon, Patrick Scholz, Thuy Nuong Tran, Pierangela Bruno, Anna Kisilenko, Benjamin Müller, Tornike Davitashvili, Manuela Capek, Minu D. Tizabi, Matthias Eisenmann, Tim J. Adler, Janek Gröhl, Melanie Schellenberg, Silvia Seidlitz, T. Y. Emmy Lai, Bünyamin Pekdemir, Veith Roethlingshoefer, Fabian Both, Sebastian Bittel, Marc Mengler, Lars Mündermann, Martin Apitz, Annette Kopp-Schneider, Stefanie Speidel, Felix Nickel, Pascal Probst, Hannes G. Kenngott, Beat P. Müller-StichPublished in 202110.1038/s41597-021-00882-2
- 7.Author(s): Alain Jungo, Olivier Scheidegger, Mauricio Reyes, Fabian BalsigerPublished in 202010.1016/j.cmpb.2020.105796
- 8.Author(s): Lena Maier-Hein, Annika Reinke, Michal Kozubek, Anne L. Martel, Tal Arbel, Matthias Eisenmann, Allan Hanbury, Pierre Jannin, Henning Müller, Sinan Onogur, Julio Saez-Rodriguez, Bram van Ginneken, Annette Kopp-Schneider, Bennett A. LandmanPublished in 202010.1016/j.media.2020.101796
- 9.Author(s): Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, Klaus H. Maier-HeinPublished in 202010.1038/s41592-020-01008-z
- 10.Author(s): Ross, Tobias, Reinke, Annika, Full, Peter M., Wagner, Martin, Kenngott, Hannes, Apitz, Martin, Hempe, Hellena, Filimon, Diana Mindroc, Scholz, Patrick, Tran, Thuy Nuong, Bruno, Pierangela, Arbeláez, Pablo, Bian, Gui-Bin, Bodenstedt, Sebastian, Bolmgren, Jon Lindström, Bravo-Sánchez, Laura, Chen, Hua-Bin, González, Cristina, Guo, Dong, Halvorsen, Pål, Heng, Pheng-Ann, Hosgor, Enes, Hou, Zeng-Guang, Isensee, Fabian, Jha, Debesh, Jiang, Tingting, Jin, Yueming, Kirtac, Kadir, Kletz, Sabrina, Leger, Stefan, Li, Zhixuan, Maier-Hein, Klaus H., Ni, Zhen-Liang, Riegler, Michael A., Schoeffmann, Klaus, Shi, Ruohua, Speidel, Stefanie, Stenzel, Michael, Twick, Isabell, Wang, Gutai, Wang, Jiacheng, Wang, Liansheng, Wang, Lu, Zhang, Yujie, Zhou, Yan-Jie, Zhu, Lei, Wiesenfarth, Manuel, Kopp-Schneider, Annette, Müller-Stich, Beat P., Maier-Hein, LenaPublished in 202010.48550/arxiv.2003.10299
- 11.Author(s): Nicholas Heller, Fabian Isensee, Klaus H. Maier-Hein, Xiaoshuai Hou, Chunmei Xie, Fengyi Li, Yang Nan, Guangrui Mu, Zhiyong Lin, Miofei Han, Guang Yao, Yaozong Gao, Yao Zhang, Yixin Wang, Feng Hou, Jiawei Yang, Guangwei Xiong, Jiang Tian, Cheng Zhong, Jun Ma, Jack Rickman, Joshua Dean, Bethany Stai, Resha Tejpaul, Makinna Oestreich, Paul Blake, Heather Kaluzniak, Shaneabbas Raza, Joel Rosenberg, Keenan Moore, Edward Walczak, Zachary Rengel, Zach Edgerton, Ranveer Vasdev, Matthew Peterson, Sean McSweeney, Sarah Peterson, Arveen Kalapara, Niranjan Sathianathen, Nikolaos Papanikolopoulos, Christopher WeightPublished in 202010.1016/j.media.2020.101821
Member of community
