We have a new landing page! It combines the structured challenge submissions across all conferences.
According to a recent study on common practice in biomedical image analysis, reproduction and adequate interpretation of challenge results are often not possible because only a fraction of relevant information is typically reported. To address this issue, we have implemented this online platform for challenge proposal submissions with structured descriptions of challenge protocols. The parameters for challenge design have been developed in collaboration with multiple institutions worldwide.
"The one practice that can universally commended is the transparent and complete reporting of all facets of a study, allowing a critical reader to evaluate the work and fully understand its strengths and limitations." Nature Neuroscience 2017, https://doi.org/10.1038/nn.4500
Challenges results will be interpretable.
The structured challenge submission systems will make challenges reproducible.
The standardized form makes it easy to include all important information without overseeing important aspects.
Altogether, your challenge will be of high quality, putting all aspects together!
Currently, the submission system is used for the following conferences:
- Website opens for submissions: 2 November 2020
- Proposal submission deadline: 10 December 2020 (23:59 CEST)
- Review deadline: 15 January 2021 (23:59 CET)
- First round of feedback: 27 January 2021
- Rebuttal deadline: 11 February 2021 (11:59 a.m. CET)
- Final feedback: 1 March 2021
- Website opens for submissions: 15 July 2020
- Proposal submission deadline: 2 October 2020 (23:59 CEST)
- Review deadline: 16 October 2020 (23:59 CEST)
- Rebuttal deadline: 21 October 2020 (23:59 CEST)
- Final feedback: 23 October 2020
- Website opens for submissions: 1 December 2019
- Proposal submission deadline: 13 January 2020 (23:59 CEST)
- First round of feedback: 3 February 2021
- Rebuttal deadline: 14 February 2021 (23:59 CEST)
- Final feedback: 21 February 2021
The number of biomedical image analysis challenges organized per year is steadily increasing. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results. To address the discrepancy between the impact of challenges and the quality (control), the Biomedical Image Analysis ChallengeS (BIAS) initiative developed a set of recommendations for the reporting of challenges. The BIAS statement aims to improve the transparency of the reporting of a biomedical image analysis challenge regardless of field of application, image modality or task category assessed.
The parameter list which is used to create challenge proposals was developed by the BIAS initiative that created a guideline on how to report and design challenges in a structured manner.
The Enhancing the QUAlity and Transparency Of health Research (EQUATOR) network is a global initiative with the aim of improving the quality of research publications and research itself. A key mission in this context is to achieve accurate, complete and transparent reporting of health research studies to support reproducibility and usefulness. Between 2006 and 2019, more than 400 reporting guidelines have been published under the umbrella of the equator network. A well-known guideline is the CONSORT statement developed for reporting of randomized controlled trials. Prominent journals, such as Lancet, JAMA or the British Medical Journal require the CONSORT checklist to be submitted along with the actual paper when reporting results of a randomized controlled trial.
The BIAS guideline was registered with the equator network: BIAS: Transparent repoting of biomedical image analysis challenges
Several challenges have been proposed in the structured challenge submission system for multiple conferences. In this section, we will show statistics based on all (accepted) challenges in the system for MICCAI, ISBI and MIDL conferences.
Organizing a challenge based on an accepted challenge proposal may be time-consuming, especially when large-scale data annotation is necessary. To address the traditionally tight schedule between MICCAI challenge acceptance and organization, we offered an early review of MICCAI 2022 challenge proposals in the call for MICCAI 2021 challenges. Organizers interested to organize a challenge in 2022 were able to receive an early acceptance that leaves time for careful preparation of the challenge. Only convincing challenge proposals were accepted in this round. Other challenges were asked to resubmit in the next call for challenges.
The team is associated with the Division of Computer Assisted Medical Interventions, German Cancer Research Center (DKFZ) Heidelberg.