Contour subregion error detection methodology using deep learning auto-segmentation

Background: Inaccurate manual organ delineation is one of the high-risk failure modes in radiation treatment. Numerous automated contour quality assurance (QA) systems have been developed to assess contour acceptability; however, manual inspection of flagged cases is a time-consuming and challenging process, and can lead to users overlooking the exact error location. Purpose: Our aim is to develop and validate a contour QA system that can effectively detect and visualize subregional contour errors, both qualitatively and quantitatively. Methods/Materials: A novel contour subregion error detection (CSED) system was developed using subregional surface distance discrepancies between manual and deep learning auto-segmentation (DLAS) contours. A validation study was conducted using a head and neck public dataset containing 339 cases and evaluated according to knowledge-based pass criteria derived from a clinical training dataset of 60 cases. A blind qualitative evaluation was conducted, comparing the results from the CSED system with manual labels. Subsequently, the CSED-flagged cases were re-examined by a radiation oncologist. Results: The CSED system could visualize the diverse types of subregional contour errors qualitatively and quantitatively. In the validation dataset, the CSED system resulted in true positive rates (TPR) of 0.814, 0.800, and 0.771; false positive rates (FPR) of 0.310, 0.267, and 0.298; and accuracies of 0.735, 0.759, and 0.730, for brainstem and left and right parotid contours, respectively. The CSED-assisted manual review caught 13 brainstem, 19 left parotid, and 21 right parotid contour errors missed by conventional human review. The TPR/FPR/accuracy of the CSED-assisted manual review improved to 0.836/0.253/0.784, 0.831/0.171/0.830, and 0.808/0.193/0.807 for each structure, respectively. Further, the time savings achieved through CSED-assisted review improved by 75%, with the time for review taking 24.81 ± 12.84, 26.75 ± 10.41, and 28.71 ± 13.72 s for each structure, respectively. Conclusions: The CSED system enables qualitative and quantitative detection, localization, and visualization of manual segmentation subregional errors utilizing DLAS contours as references. The use of this system has been shown to help reduce the risk of high-risk failure modes resulting from inaccurate organ segmentation.