Measuring and Teaching Confidence Calibration Among Radiologists: A Multi-Institution Study

Objective: Our purpose was to assess the calibration of resident, fellow, and attending radiologists on a simple image classification task (presence or absence of an anterior cruciate ligament [ACL] tear based on interpretation of sagittal proton density, fat-saturated MR images) and to assess whether teaching residents could improve their calibration. Methods: We created a test containing 30 randomized, sagittal, proton density, fat-saturated MR images of the ACL (15 normal, 15 torn). This test was administered in person to 20 trainees and 3 attendings at one medical center in one state. An online version of the test was given to 23 trainees and 14 attendings from 11 other medical centers in nine other states. Subjects were asked to give their confidence level (0%-100%) that each ACL was torn. Results: Cross-sectional data were collected from 60 radiologists (mean time after medical school = 9.3 years, minimum = 1 year, maximum = 36 years). This demonstrated a statistically significant improvement in calibration as a function of increasing experience (P = .020). Longitudinal data were collected from 12 trainees at the start and end of their musculoskeletal radiology rotation, with an intervening review of the primary and secondary signs of ACL tear on MR. A statistically significant improvement in calibration was noted during the rotation (P = .028). Conclusions: Confidence calibration is a promising tool for quality improvement and radiologist self-assessment. Our study showed that calibration loss improves with experience in radiologists tested on a common and clinically important image classification task. We also demonstrated that calibration can be successfully taught to residents over a relatively short period (2-4 weeks).