The effect of prevalence of disease on performance of residents and fellows during training for interpreting DBT in a test-train-test observer study

Christiane M. Hakim, Lauren Q Chang Sen, Andrew Degnan, Joseph Delic, Sarah Pai, Hersh Sagreiya, Mark Sparrow, Ernestine Thomas, Michael Yannes, David Gur

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

There are no data on the effect of disease prevalence during training for interpreting digital breast tomosynthesis (DBT) based screening examinations on the performance of residents and fellows. We assessed the performance of six residents (four after one breast imaging rotation and two after two rotations) and two fellows in breast imaging when interpreting DBT screening examinations in a multi-case, mode balanced, test-train-test retrospective reader study (127 training and 160 testing cases). Half were trained with feedback of verified truth after reviewing each case with low prevalence of disease (13/127) and half with high prevalence (52/128). The pre- and post-training dataset was the same. Performance measures were compared (sensitivity, specificity and AUC). Readers trained with the low prevalence set decreased the overall recall rate of non-cancer cases (FPF from 0.21 to 0.13, p<0.001), and of cases with known malignancies (TPF from 0.70 to 0.61, p=0.004, due primarily to one clearly outlier reader). Readers trained with the high prevalence increased the overall recall rate (albeit, not statistically significant) of non-cancer cases (FPF from 0.16 to 0.18, p=0.07), and a borderline significant increase of cancer cases (TPF from 0.61 to 0.66, p=0.04). Fellows post six months of specialty training in each group had no significant changes in sensitivity, specificity, or AUC after training (smallest p>0.07). Both residents with two rotations experience had significant changes in sensitivity and specificity (highest p<0.028), but not in AUC. Early training with low disease prevalence of "what not to recall" should be included during training.

Original languageEnglish (US)
Title of host publicationMedical Imaging 2017
Subtitle of host publicationImage Perception, Observer Performance, and Technology Assessment
PublisherSPIE
Volume10136
ISBN (Electronic)9781510607170
DOIs
StatePublished - Jan 1 2017
EventMedical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment - Orlando, United States
Duration: Feb 12 2017Feb 13 2017

Other

OtherMedical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment
CountryUnited States
CityOrlando
Period2/12/172/13/17

Fingerprint

Mammography
breast
education
Screening
Imaging techniques
Area Under Curve
readers
Breast
screening
Sensitivity and Specificity
examination
Feedback
reviewing
sensitivity
Testing
Retrospective Studies

Keywords

  • Breast imaging
  • Observer performance
  • Prevalence of disease
  • Training

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Atomic and Molecular Physics, and Optics
  • Biomaterials
  • Radiology Nuclear Medicine and imaging

Cite this

Hakim, C. M., Chang Sen, L. Q., Degnan, A., Delic, J., Pai, S., Sagreiya, H., ... Gur, D. (2017). The effect of prevalence of disease on performance of residents and fellows during training for interpreting DBT in a test-train-test observer study. In Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment (Vol. 10136). [1013603] SPIE. https://doi.org/10.1117/12.2249341

The effect of prevalence of disease on performance of residents and fellows during training for interpreting DBT in a test-train-test observer study. / Hakim, Christiane M.; Chang Sen, Lauren Q; Degnan, Andrew; Delic, Joseph; Pai, Sarah; Sagreiya, Hersh; Sparrow, Mark; Thomas, Ernestine; Yannes, Michael; Gur, David.

Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment. Vol. 10136 SPIE, 2017. 1013603.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Hakim, CM, Chang Sen, LQ, Degnan, A, Delic, J, Pai, S, Sagreiya, H, Sparrow, M, Thomas, E, Yannes, M & Gur, D 2017, The effect of prevalence of disease on performance of residents and fellows during training for interpreting DBT in a test-train-test observer study. in Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment. vol. 10136, 1013603, SPIE, Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment, Orlando, United States, 2/12/17. https://doi.org/10.1117/12.2249341
Hakim CM, Chang Sen LQ, Degnan A, Delic J, Pai S, Sagreiya H et al. The effect of prevalence of disease on performance of residents and fellows during training for interpreting DBT in a test-train-test observer study. In Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment. Vol. 10136. SPIE. 2017. 1013603 https://doi.org/10.1117/12.2249341
Hakim, Christiane M. ; Chang Sen, Lauren Q ; Degnan, Andrew ; Delic, Joseph ; Pai, Sarah ; Sagreiya, Hersh ; Sparrow, Mark ; Thomas, Ernestine ; Yannes, Michael ; Gur, David. / The effect of prevalence of disease on performance of residents and fellows during training for interpreting DBT in a test-train-test observer study. Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment. Vol. 10136 SPIE, 2017.
@inproceedings{66bd19c4b7994b93a833b4809c3cc119,
title = "The effect of prevalence of disease on performance of residents and fellows during training for interpreting DBT in a test-train-test observer study",
abstract = "There are no data on the effect of disease prevalence during training for interpreting digital breast tomosynthesis (DBT) based screening examinations on the performance of residents and fellows. We assessed the performance of six residents (four after one breast imaging rotation and two after two rotations) and two fellows in breast imaging when interpreting DBT screening examinations in a multi-case, mode balanced, test-train-test retrospective reader study (127 training and 160 testing cases). Half were trained with feedback of verified truth after reviewing each case with low prevalence of disease (13/127) and half with high prevalence (52/128). The pre- and post-training dataset was the same. Performance measures were compared (sensitivity, specificity and AUC). Readers trained with the low prevalence set decreased the overall recall rate of non-cancer cases (FPF from 0.21 to 0.13, p<0.001), and of cases with known malignancies (TPF from 0.70 to 0.61, p=0.004, due primarily to one clearly outlier reader). Readers trained with the high prevalence increased the overall recall rate (albeit, not statistically significant) of non-cancer cases (FPF from 0.16 to 0.18, p=0.07), and a borderline significant increase of cancer cases (TPF from 0.61 to 0.66, p=0.04). Fellows post six months of specialty training in each group had no significant changes in sensitivity, specificity, or AUC after training (smallest p>0.07). Both residents with two rotations experience had significant changes in sensitivity and specificity (highest p<0.028), but not in AUC. Early training with low disease prevalence of {"}what not to recall{"} should be included during training.",
keywords = "Breast imaging, Observer performance, Prevalence of disease, Training",
author = "Hakim, {Christiane M.} and {Chang Sen}, {Lauren Q} and Andrew Degnan and Joseph Delic and Sarah Pai and Hersh Sagreiya and Mark Sparrow and Ernestine Thomas and Michael Yannes and David Gur",
year = "2017",
month = "1",
day = "1",
doi = "10.1117/12.2249341",
language = "English (US)",
volume = "10136",
booktitle = "Medical Imaging 2017",
publisher = "SPIE",

}

TY - GEN

T1 - The effect of prevalence of disease on performance of residents and fellows during training for interpreting DBT in a test-train-test observer study

AU - Hakim, Christiane M.

AU - Chang Sen, Lauren Q

AU - Degnan, Andrew

AU - Delic, Joseph

AU - Pai, Sarah

AU - Sagreiya, Hersh

AU - Sparrow, Mark

AU - Thomas, Ernestine

AU - Yannes, Michael

AU - Gur, David

PY - 2017/1/1

Y1 - 2017/1/1

N2 - There are no data on the effect of disease prevalence during training for interpreting digital breast tomosynthesis (DBT) based screening examinations on the performance of residents and fellows. We assessed the performance of six residents (four after one breast imaging rotation and two after two rotations) and two fellows in breast imaging when interpreting DBT screening examinations in a multi-case, mode balanced, test-train-test retrospective reader study (127 training and 160 testing cases). Half were trained with feedback of verified truth after reviewing each case with low prevalence of disease (13/127) and half with high prevalence (52/128). The pre- and post-training dataset was the same. Performance measures were compared (sensitivity, specificity and AUC). Readers trained with the low prevalence set decreased the overall recall rate of non-cancer cases (FPF from 0.21 to 0.13, p<0.001), and of cases with known malignancies (TPF from 0.70 to 0.61, p=0.004, due primarily to one clearly outlier reader). Readers trained with the high prevalence increased the overall recall rate (albeit, not statistically significant) of non-cancer cases (FPF from 0.16 to 0.18, p=0.07), and a borderline significant increase of cancer cases (TPF from 0.61 to 0.66, p=0.04). Fellows post six months of specialty training in each group had no significant changes in sensitivity, specificity, or AUC after training (smallest p>0.07). Both residents with two rotations experience had significant changes in sensitivity and specificity (highest p<0.028), but not in AUC. Early training with low disease prevalence of "what not to recall" should be included during training.

AB - There are no data on the effect of disease prevalence during training for interpreting digital breast tomosynthesis (DBT) based screening examinations on the performance of residents and fellows. We assessed the performance of six residents (four after one breast imaging rotation and two after two rotations) and two fellows in breast imaging when interpreting DBT screening examinations in a multi-case, mode balanced, test-train-test retrospective reader study (127 training and 160 testing cases). Half were trained with feedback of verified truth after reviewing each case with low prevalence of disease (13/127) and half with high prevalence (52/128). The pre- and post-training dataset was the same. Performance measures were compared (sensitivity, specificity and AUC). Readers trained with the low prevalence set decreased the overall recall rate of non-cancer cases (FPF from 0.21 to 0.13, p<0.001), and of cases with known malignancies (TPF from 0.70 to 0.61, p=0.004, due primarily to one clearly outlier reader). Readers trained with the high prevalence increased the overall recall rate (albeit, not statistically significant) of non-cancer cases (FPF from 0.16 to 0.18, p=0.07), and a borderline significant increase of cancer cases (TPF from 0.61 to 0.66, p=0.04). Fellows post six months of specialty training in each group had no significant changes in sensitivity, specificity, or AUC after training (smallest p>0.07). Both residents with two rotations experience had significant changes in sensitivity and specificity (highest p<0.028), but not in AUC. Early training with low disease prevalence of "what not to recall" should be included during training.

KW - Breast imaging

KW - Observer performance

KW - Prevalence of disease

KW - Training

UR - http://www.scopus.com/inward/record.url?scp=85020300313&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85020300313&partnerID=8YFLogxK

U2 - 10.1117/12.2249341

DO - 10.1117/12.2249341

M3 - Conference contribution

AN - SCOPUS:85020300313

VL - 10136

BT - Medical Imaging 2017

PB - SPIE

ER -