Equator network

The Enhancing the QUAlity and Transparency Of health Research (EQUATOR) network is a global initiative with the aim of improving the quality of research publications and research itself. A key mission in this context is to achieve accurate, complete and transparent reporting of health research studies to support reproducibility and usefulness. Between 2006 and 2019, more than 400 reporting guidelines have been published under the umbrella of the equator network. A well-known guideline is the CONSORT statement developed for reporting of randomized controlled trials. Prominent journals, such as Lancet, JAMA or the British Medical Journal require the CONSORT checklist to be submitted along with the actual paper when reporting results of a randomized controlled trial.

The BIAS guideline was registered with the equator network: BIAS: Transparent repoting of biomedical image analysis challenges

1. Complete and transparent reporting

Validation studies such as challenges need to be reproducible and interpretable. To achieve this, we recommend you to report all important challenge design parameters. Below, you can download the checklist for challenge design parameters which can directly be filled out.
You wish to organize a MICCAI endorsed challenge? Please fill the form and send it to MICCAI-board-challenge-WG(at)dkfz-heidelberg.de.

2. Large dataset sizes

The dataset is one of the main components of your challenge. The rankings rely on the dataset as well as the general performance of the participating algorithms. In machine learning, you should try to acquire as much data as possible to avoid poor approximation of the algorithms. Of course, this is tough in the medical domain, but challenges with only very few cases may not attract a lot of researchers which therefore may lead to rejection.

3. Training-test dataset balance

The dataset should reflect the natural variability of the imaged objects. In the best case, it was acquired from multiple sources. The split of the dataset into training and test is very important for the algorithm performances. Be aware that imbalanced training data may lead to worse results. Furthermore, make sure that training and test data share similar distributions, but the test data should also be sufficient to test the generalization capabilities of algorithms.

4. Metrics/assessment aims reflecting the challenge goal

The metrics should be well chosen. Don't use a metric because many other researcher are using it without thinking it through. For example, the Dice Similarity metric is very popular in segmentation tasks, but it is not well-suited for small objects to be segmented. Define the goal of your challenge, define assessment aims and find the metrics that fit best. If you are unsure, here are some reads to consider:

• Common Limitations of Image Processing Metrics: A Picture Story
• A discovery dive into the world of evaluation — Do’s, don’ts and other considerations (blogpost)
• Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool

5. Statistical and/or uncertainty analysis

Organizing a challenge also includes the analysis of results. The algorithm performance should be validated in a thoughtful manner. Is the ranking robust against small perturbations? What would happen to the ranking if you slightly change the test dataset? What would happen if you combine some of the submitted methods? Statistics is important. Don't leave it out. Check out the following links for details:

• Methods and open-source toolkit for analyzing and visualizing challenge results (publication)
• Open-source toolkit for analyzing and visualizing challenge results (toolkit)

6. Challenge motivation

Why do you want to run your challenge? What is your motivation? Are you trying to find algorithms that exceed the current state of the art? Do you try to find solutions for an ill-posed problem? These questions should be pondered thoroughly from beginning.

7. Details and justification

Challenges are validation studies. As such, the design should be completely transparent and design choices should be justified and comprehensible. In the past, challenges were rejected because some design choices were not justified at all or not reproducible. We will present some examples of the most unclear parameters of rejected challenges. As described above, the dataset is highly important. It should be absolutely clear why you chose a specific training/test split. Next, the annotation process is as important as the data. Questions such as: 'Who annotated the data?' 'What was the experience level of annotaters?' 'Did multiple annotators annotate the data?' should be clearly answered as well as a link to the labeling protocol should be provided. Make sure to justify why you chose specific performance metrics. The metric should reflect your challenge goal (see below) and should assess valid properties. Be aware of mutually dependent metrics. The ranking computation should be clear, in the best case, write it in a pseudoalgorithmic form. Make clear why this computation was chosen. How does the submission of results work for the participants? Are they required to generate docker containers? Upload results? Upload code? All these questions are important to know.

8. Avoiding typos, grammatical errors, incomplete sentences

Finally, take your best shot when filling out the submission form. We know, it's tough, there are many parameters. But all of them matter. The reviewers will know immediately when you filled the parameters out in a rush. You don't submit a paper full of typos, grammatical errors and incomplete sentences, right? So you also don't want to submit a challenge proposal in such a way.