E pluribus unum: prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology crowdsourced initiative for multiobserver segmentation

Diana Lin; Kareem A. Wahid; Benjamin E. Nelms; Renjie He; Mohammed A. Naser; Simon Duke; Michael V. Sherer; John P. Christodouleas; Abdallah S.R. Mohamed; Michael Cislo; James D. Murphy; Clifton D. Fuller; Erin F. Gillespie

doi:10.1117/1.JMI.10.S1.S11903

E pluribus unum: prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology crowdsourced initiative for multiobserver segmentation

Diana Lin, Kareem A. Wahid, Benjamin E. Nelms, Renjie He, Mohammed A. Naser, Simon Duke, Michael V. Sherer, John P. Christodouleas, Abdallah S.R. Mohamed, Michael Cislo, James D. Murphy, Clifton D. Fuller, Erin F. Gillespie

Radiation Oncology

Research output: Contribution to journal › Article › peer-review

2 Scopus citations

Abstract

Purpose: Contouring Collaborative for Consensus in Radiation Oncology (C3RO) is a crowdsourced challenge engaging radiation oncologists across various expertise levels in segmentation. An obstacle to artificial intelligence (AI) development is the paucity of multiexpert datasets; consequently, we sought to characterize whether aggregate segmentations generated from multiple nonexperts could meet or exceed recognized expert agreement. Approach: Participants who contoured =1 region of interest (ROI) for the breast, sarcoma, head and neck (H N), gynecologic (GYN), or gastrointestinal (GI) cases were identified as a nonexpert or recognized expert. Cohort-specific ROIs were combined into single simultaneous truth and performance level estimation (STAPLE) consensus segmentations. STAPLEnonexpert ROIs were evaluated against STAPLEexpert contours using Dice similarity coefficient (DSC). The expert interobserver DSC (IODSCexpert) was calculated as an acceptability threshold between STAPLE nonexpert and STAPLEexpert. To determine the number of nonexperts required to match the IODSC expert for each ROI, a single consensus contour was generated using variable numbers of nonexperts and then compared to the IODSCexpert. Results: For all cases, the DSC values for STAPLEnonexpert versus STAPLEexpert were higher than comparator expert IODSCexpert for most ROIs. The minimum number of nonexpert segmentations needed for a consensus ROI to achieve IODSC expert acceptability criteria ranged between 2 and 4 for breast, 3 and 5 for sarcoma, 3 and 5 for H N, 3 and 5 for GYN, and 3 for GI. Conclusions: Multiple nonexpert-generated consensus ROIs met or exceeded expert-derived acceptability thresholds. Five nonexperts could potentially generate consensus segmentations for most ROIs with performance approximating experts, suggesting nonexpert segmentations as feasible cost-effective AI inputs.

Original language	English (US)
Article number	S11903
Journal	Journal of Medical Imaging
Volume	10
DOIs	https://doi.org/10.1117/1.JMI.10.S1.S11903
State	Published - Feb 1 2023

Keywords

artificial intelligence
autosegmentation
contouring
crowdsourcing
radiation oncology
segmentation

ASJC Scopus subject areas

Radiology Nuclear Medicine and imaging

Access to Document

10.1117/1.JMI.10.S1.S11903

Cite this

Lin, D., Wahid, K. A., Nelms, B. E., He, R., Naser, M. A., Duke, S., Sherer, M. V., Christodouleas, J. P., Mohamed, A. S. R., Cislo, M., Murphy, J. D., Fuller, C. D., & Gillespie, E. F. (2023). E pluribus unum: prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology crowdsourced initiative for multiobserver segmentation. Journal of Medical Imaging, 10, Article S11903. https://doi.org/10.1117/1.JMI.10.S1.S11903

Lin, D, Wahid, KA, Nelms, BE, He, R , Naser, MA, Duke, S, Sherer, MV, Christodouleas, JP, Mohamed, ASR, Cislo, M, Murphy, JD, Fuller, CD & Gillespie, EF 2023, 'E pluribus unum: prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology crowdsourced initiative for multiobserver segmentation', Journal of Medical Imaging, vol. 10, S11903. https://doi.org/10.1117/1.JMI.10.S1.S11903

@article{0d82b95dea3648f690b4d9ec19f63509,

title = "E pluribus unum: prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology crowdsourced initiative for multiobserver segmentation",

abstract = "Purpose: Contouring Collaborative for Consensus in Radiation Oncology (C3RO) is a crowdsourced challenge engaging radiation oncologists across various expertise levels in segmentation. An obstacle to artificial intelligence (AI) development is the paucity of multiexpert datasets; consequently, we sought to characterize whether aggregate segmentations generated from multiple nonexperts could meet or exceed recognized expert agreement. Approach: Participants who contoured =1 region of interest (ROI) for the breast, sarcoma, head and neck (H N), gynecologic (GYN), or gastrointestinal (GI) cases were identified as a nonexpert or recognized expert. Cohort-specific ROIs were combined into single simultaneous truth and performance level estimation (STAPLE) consensus segmentations. STAPLEnonexpert ROIs were evaluated against STAPLEexpert contours using Dice similarity coefficient (DSC). The expert interobserver DSC (IODSCexpert) was calculated as an acceptability threshold between STAPLE nonexpert and STAPLEexpert. To determine the number of nonexperts required to match the IODSC expert for each ROI, a single consensus contour was generated using variable numbers of nonexperts and then compared to the IODSCexpert. Results: For all cases, the DSC values for STAPLEnonexpert versus STAPLEexpert were higher than comparator expert IODSCexpert for most ROIs. The minimum number of nonexpert segmentations needed for a consensus ROI to achieve IODSC expert acceptability criteria ranged between 2 and 4 for breast, 3 and 5 for sarcoma, 3 and 5 for H N, 3 and 5 for GYN, and 3 for GI. Conclusions: Multiple nonexpert-generated consensus ROIs met or exceeded expert-derived acceptability thresholds. Five nonexperts could potentially generate consensus segmentations for most ROIs with performance approximating experts, suggesting nonexpert segmentations as feasible cost-effective AI inputs.",

keywords = "artificial intelligence, autosegmentation, contouring, crowdsourcing, radiation oncology, segmentation",

author = "Diana Lin and Wahid, {Kareem A.} and Nelms, {Benjamin E.} and Renjie He and Naser, {Mohammed A.} and Simon Duke and Sherer, {Michael V.} and Christodouleas, {John P.} and Mohamed, {Abdallah S.R.} and Michael Cislo and Murphy, {James D.} and Fuller, {Clifton D.} and Gillespie, {Erin F.}",

note = "Publisher Copyright: {\textcopyright} The Authors. Published by SPIE under a Creative Commons Attribution 4.0 International License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.",

year = "2023",

month = feb,

day = "1",

doi = "10.1117/1.JMI.10.S1.S11903",

language = "English (US)",

volume = "10",

journal = "Journal of Medical Imaging",

issn = "2329-4302",

publisher = "Elsevier Ireland Ltd",

}

TY - JOUR

T1 - E pluribus unum

T2 - prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology crowdsourced initiative for multiobserver segmentation

AU - Lin, Diana

AU - Wahid, Kareem A.

AU - Nelms, Benjamin E.

AU - He, Renjie

AU - Naser, Mohammed A.

AU - Duke, Simon

AU - Sherer, Michael V.

AU - Christodouleas, John P.

AU - Mohamed, Abdallah S.R.

AU - Cislo, Michael

AU - Murphy, James D.

AU - Fuller, Clifton D.

AU - Gillespie, Erin F.

N1 - Publisher Copyright: © The Authors. Published by SPIE under a Creative Commons Attribution 4.0 International License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

PY - 2023/2/1

Y1 - 2023/2/1

N2 - Purpose: Contouring Collaborative for Consensus in Radiation Oncology (C3RO) is a crowdsourced challenge engaging radiation oncologists across various expertise levels in segmentation. An obstacle to artificial intelligence (AI) development is the paucity of multiexpert datasets; consequently, we sought to characterize whether aggregate segmentations generated from multiple nonexperts could meet or exceed recognized expert agreement. Approach: Participants who contoured =1 region of interest (ROI) for the breast, sarcoma, head and neck (H N), gynecologic (GYN), or gastrointestinal (GI) cases were identified as a nonexpert or recognized expert. Cohort-specific ROIs were combined into single simultaneous truth and performance level estimation (STAPLE) consensus segmentations. STAPLEnonexpert ROIs were evaluated against STAPLEexpert contours using Dice similarity coefficient (DSC). The expert interobserver DSC (IODSCexpert) was calculated as an acceptability threshold between STAPLE nonexpert and STAPLEexpert. To determine the number of nonexperts required to match the IODSC expert for each ROI, a single consensus contour was generated using variable numbers of nonexperts and then compared to the IODSCexpert. Results: For all cases, the DSC values for STAPLEnonexpert versus STAPLEexpert were higher than comparator expert IODSCexpert for most ROIs. The minimum number of nonexpert segmentations needed for a consensus ROI to achieve IODSC expert acceptability criteria ranged between 2 and 4 for breast, 3 and 5 for sarcoma, 3 and 5 for H N, 3 and 5 for GYN, and 3 for GI. Conclusions: Multiple nonexpert-generated consensus ROIs met or exceeded expert-derived acceptability thresholds. Five nonexperts could potentially generate consensus segmentations for most ROIs with performance approximating experts, suggesting nonexpert segmentations as feasible cost-effective AI inputs.

AB - Purpose: Contouring Collaborative for Consensus in Radiation Oncology (C3RO) is a crowdsourced challenge engaging radiation oncologists across various expertise levels in segmentation. An obstacle to artificial intelligence (AI) development is the paucity of multiexpert datasets; consequently, we sought to characterize whether aggregate segmentations generated from multiple nonexperts could meet or exceed recognized expert agreement. Approach: Participants who contoured =1 region of interest (ROI) for the breast, sarcoma, head and neck (H N), gynecologic (GYN), or gastrointestinal (GI) cases were identified as a nonexpert or recognized expert. Cohort-specific ROIs were combined into single simultaneous truth and performance level estimation (STAPLE) consensus segmentations. STAPLEnonexpert ROIs were evaluated against STAPLEexpert contours using Dice similarity coefficient (DSC). The expert interobserver DSC (IODSCexpert) was calculated as an acceptability threshold between STAPLE nonexpert and STAPLEexpert. To determine the number of nonexperts required to match the IODSC expert for each ROI, a single consensus contour was generated using variable numbers of nonexperts and then compared to the IODSCexpert. Results: For all cases, the DSC values for STAPLEnonexpert versus STAPLEexpert were higher than comparator expert IODSCexpert for most ROIs. The minimum number of nonexpert segmentations needed for a consensus ROI to achieve IODSC expert acceptability criteria ranged between 2 and 4 for breast, 3 and 5 for sarcoma, 3 and 5 for H N, 3 and 5 for GYN, and 3 for GI. Conclusions: Multiple nonexpert-generated consensus ROIs met or exceeded expert-derived acceptability thresholds. Five nonexperts could potentially generate consensus segmentations for most ROIs with performance approximating experts, suggesting nonexpert segmentations as feasible cost-effective AI inputs.

KW - artificial intelligence

KW - autosegmentation

KW - contouring

KW - crowdsourcing

KW - radiation oncology

KW - segmentation

UR - http://www.scopus.com/inward/record.url?scp=85150744860&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85150744860&partnerID=8YFLogxK

U2 - 10.1117/1.JMI.10.S1.S11903

DO - 10.1117/1.JMI.10.S1.S11903

M3 - Article

C2 - 36761036

AN - SCOPUS:85150744860

SN - 2329-4302

VL - 10

JO - Journal of Medical Imaging

JF - Journal of Medical Imaging

M1 - S11903

ER -

E pluribus unum: prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology crowdsourced initiative for multiobserver segmentation

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this