Skip NavigationSkip to Content
Return Return to Search Results

QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results

  1. Author:
    Mehta, Raghav
    Filos, Angelos
    Baid, Ujjwal
    Sako, Chiharu
    McKinley, Richard
    Rebsamen, Michael
    Dätwyler, Katrin
    Meier, Raphael
    Radojewski, Piotr
    Murugesan, Gowtham Krishnan
    Nalawade, Sahil
    Ganesh, Chandan
    Wagner, Ben
    Yu, Fang F
    Fei, Baowei
    Madhuranthakam, Ananth J
    Maldjian, Joseph A
    Daza, Laura
    Gómez, Catalina
    Arbeláez, Pablo
    Dai, Chengliang
    Wang, Shuo
    Reynaud, Hadrien
    Mo, Yuanhan
    Angelini, Elsa
    Guo, Yike
    Bai, Wenjia
    Banerjee, Subhashis
    Pei, Linmin
    Ak, Murat
    Rosas-González, Sarahi
    Zemmoura, Ilyess
    Tauber, Clovis
    Vu, Minh H
    Nyholm, Tufve
    Löfstedt, Tommy
    Ballestar, Laura Mora
    Vilaplana, Veronica
    McHugh, Hugh
    Maso Talou, Gonzalo
    Wang, Alan
    Patel, Jay
    Chang, Ken
    Hoebel, Katharina
    Gidwani, Mishka
    Arun, Nishanth
    Gupta, Sharut
    Aggarwal, Mehak
    Singh, Praveer
    Gerstner, Elizabeth R
    Kalpathy-Cramer, Jayashree
    Boutry, Nicolas
    Huard, Alexis
    Vidyaratne, Lasitha
    Rahman, Md Monibor
    Iftekharuddin, Khan M
    Chazalon, Joseph
    Puybareau, Elodie
    Tochon, Guillaume
    Ma, Jun
    Cabezas, Mariano
    Llado, Xavier
    Oliver, Arnau
    Valencia, Liliana
    Valverde, Sergi
    Amian, Mehdi
    Soltaninejad, Mohammadreza
    Myronenko, Andriy
    Hatamizadeh, Ali
    Feng, Xue
    Dou, Quan
    Tustison, Nicholas
    Meyer, Craig
    Shah, Nisarg A
    Talbar, Sanjay
    Weber, Marc-André
    Mahajan, Abhishek
    Jakab, Andras
    Wiest, Roland
    Fathallah-Shaykh, Hassan M
    Nazeri, Arash
    Milchenko, Mikhail
    Marcus, Daniel
    Kotrotsou, Aikaterini
    Colen, Rivka
    Freymann,John
    Kirby,Justin
    Davatzikos, Christos
    Menze, Bjoern
    Bakas, Spyridon
    Gal, Yarin
    Arbel, Tal

  2. Author Address

    1Centre for Intelligent Machines (CIM), McGill University, Montreal, QC, Canada. 2Oxford Applied and Theoretical Machine Learning (OATML) Group, University of Oxford, Oxford, England. 3Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA. 4Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA. 5Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA. 6Support Center for Advanced Neuroimaging (SCAN), University Institute of Diagnostic and Interventional Neuroradiology, University of Bern, Inselspital, Bern University Hospital, Bern, Switzerland. 7Human Performance Lab, Schulthess Clinic, Zurich, Switzerland. 8armasuisse S+T, Thun, Switzerland. 9Department of Radiology, University of Texas Southwestern Medical Center, Dallas, TX, USA. 10Department of Bioengineering, University of Texas at Dallas, Texas, USA. 11Advanced Imaging Research Center, University of Texas Southwestern Medical Center, Dallas, TX, USA. 12Universidad de los Andes, Bogotá, Colombia. 13Data Science Institute, Imperial College London, London, UK. 14NIHR Imperial BRC, ITMAT Data Science Group, Imperial College London, London, UK. 15Department of Brain Sciences, Imperial College London, London, UK. 16Machine Intelligence Unit, Indian Statistical Institute, Kolkata, India. 17Department of CSE, University of Calcutta, Kolkata, India. 18Division of Visual Information and Interaction (Vi2), Department of Information Technology, Uppsala University, Uppsala, Sweden. 19Department of Diagnostic Radiology, The University of Pittsburgh Medical Center, Pittsburgh, PA, USA. 20UMR U1253 iBrain, Université de Tours, Inserm, Tours, France. 21Neurosurgery department, CHRU de Tours, Tours, France. 22Department of Radiation Sciences, Umeå University, Umeå, Sweden. 23Department of Computing Science, Umeå University, Umeå, Sweden. 24Signal Theory and Communications Department, Universitat Politècnica de Catalunya, BarcelonaTech, Barcelona, Spain. 25Faculty of Medical and Health Sciences, University of Auckland, Auckland, New Zealand. 26Radiology Department, Auckland City Hospital, Auckland, New Zealand. 27Auckland Bioengineering Institute, University of Auckland, New Zealand. 28Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Boston, MA, USA. 29Massachusetts Institute of Technology, Cambridge, MA, USA. 30EPITA Research and Development Laboratory (LRDE), France. 31Vision Lab, Electrical and Computer Engineering, Old Dominion University, Norfolk, VA 23529, USA. 32EPITA Research and Development Laboratory (LRDE), Le Kremlin-Bicetre, France. 33School of Science, Nanjing University of Science and Technology. 34Research Institute of Computer Vision and Robotics, University of Girona, Spain. 35Department of Electrical and Computer Engineering, University of Tehran, Iran. 36School of Computer Science, University of Nottingham, UK. 37NVIDIA, Santa Clara, CA, US. 38Biomedical Engineering, University of Virginia, Charlottesville, USA. 39Radiology and Medical Imaging, University of Virginia, Charlottesville, USA. 40Department of Electrical Engineering, Indian Institute of Technology - Jodhpur, Jodhpur, India. 41SGGS Institute of Engineering and Technology, Nanded, India. 42Institute of Diagnostic and Interventional Radiology, Pediatric Radiology and Neuroradiology, University Medical Center Rostock, Rostock, Germany. 43Tata Memorial Centre, Homi Bhabha National Institute, Mumbai, India. 44Center for MR-Research, University Children's Hospital Zurich, Zurich, Switzerland. 45Institute for Surgical Technology and Biomechanics, University of Bern, Bern, Switzerland. 46Department of Neurology, The University of Alabama at Birmingham, Birmingham, AL, USA. 47Department of Radiology, Washington University, St. Louis, MO, USA. 48Neuroimaging Informatics and Analysis Center, Washington University, St. Louis, MO, USA. 49Department of Diagnostic Radiology, University of Texas MD Anderson Cancer Center, Houston, TX, USA. 50Leidos Biomedical Research, Inc, Frederick National Laboratory for Cancer Research, Frederick, MD, USA. 51Cancer Imaging Program, National Cancer Institute, National Institutes of Health, Bethesda, MD, USA. 52Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland. 53Department of Informatics, Technical University of Munich, Munich, Germany. 54MILA - Quebec Artificial Intelligence Institute, Montreal, QC, Canada.
    1. Year: 2022
    2. Date: Aug
  2. Journal: The journal of Machine Learning for Biomedical Imaging
    1. 2022
  4. Type of Article: Article
  1. Abstract:

    Deep learning (DL) models have provided state-of-the-art performance in various medical imaging benchmarking challenges, including the Brain Tumor Segmentation (BraTS) challenges. However, the task of focal pathology multi-compartment segmentation (e.g., tumor and lesion sub-regions) is particularly challenging, and potential errors hinder translating DL models into clinical workflows. Quantifying the reliability of DL model predictions in the form of uncertainties could enable clinical review of the most uncertain regions, thereby building trust and paving the way toward clinical translation. Several uncertainty estimation methods have recently been introduced for DL medical image segmentation tasks. Developing scores to evaluate and compare the performance of uncertainty measures will assist the end-user in making more informed decisions. In this study, we explore and evaluate a score developed during the BraTS 2019 and BraTS 2020 task on uncertainty quantification (QU-BraTS) and designed to assess and rank uncertainty estimates for brain tumor multi-compartment segmentation. This score (1) rewards uncertainty estimates that produce high confidence in correct assertions and those that assign low confidence levels at incorrect assertions, and (2) penalizes uncertainty measures that lead to a higher percentage of under-confident correct assertions. We further benchmark the segmentation uncertainties generated by 14 independent participating teams of QU-BraTS 2020, all of which also participated in the main BraTS segmentation task. Overall, our findings confirm the importance and complementary value that uncertainty estimates provide to segmentation algorithms, highlighting the need for uncertainty quantification in medical image analyses. Finally, in favor of transparency and reproducibility, our evaluation code is made publicly available at https://github.com/RagMeh11/QU-BraTS.

    See More

  1. Keywords:

External Sources

  1. PMID: 36998700
  2. PMCID: PMC10060060
  3. PII : https://www.melba-journal.org/papers/2022:026.html

Library Notes

  1. Fiscal Year: FY2022-2023
NCI at Frederick

You are leaving a government website.

This external link provides additional information that is consistent with the intended purpose of this site. The government cannot attest to the accuracy of a non-federal site.

Linking to a non-federal site does not constitute an endorsement by this institution or any of its employees of the sponsors or the information and products presented on the site. You will be subject to the destination site's privacy policy when you follow the link.

ContinueCancel