TY - JOUR
T1 - On the choice of acceptance radius in free-response observer performance studies
AU - Haygood, Tamara Miner
AU - Ryan, J.
AU - Brennan, P. C.
AU - Li, S.
AU - Marom, E. M.
AU - McEntee, M. F.
AU - Itani, M.
AU - Evanoff, M.
AU - Chakraborty, D.
PY - 2013/1/1
Y1 - 2013/1/1
N2 - Objectives: Choosing an acceptance radius or proximity criterion is necessary to analyse free-response receiver operating characteristic (FROC) observer performance data. This is currently subjective, with little guidance in the literature about what is an appropriate acceptance radius. We evaluated varying acceptance radii in a nodule detection task in chest radiography and suggest guidelines for determining an acceptance radius. Methods: 80 chest radiographs were chosen, half of which contained nodules. We determined each nodule's centre. 21 radiologists read the images. We created acceptance radii bins of <5 pixels, <10 pixels, <20 pixels and onwards up to <200 and 200+ pixels. We counted lesion localisations in each bin and visually compared marks with the borders of nodules. Results: Most reader marks were tightly clustered around nodule centres, with tighter clustering for smaller than for larger nodules. At least 70% of readers' marks were placed within <10 pixels for small nodules, <20 pixels for medium nodules and <30 pixels for large nodules. Of 72 inspected marks that were less than 50 pixels from the centre of a nodule, only 1 fell outside the border of a nodule. Conclusion: The acceptance radius should be based on the larger nodule sizes. For our data, an acceptance radius of 50 pixels would have captured all but 2 reader marks within the borders of a nodule, while excluding only 1 true-positive mark. The choice of an acceptance radius for FROC analysis of observer performance studies should be based on the size of larger abnormalities.
AB - Objectives: Choosing an acceptance radius or proximity criterion is necessary to analyse free-response receiver operating characteristic (FROC) observer performance data. This is currently subjective, with little guidance in the literature about what is an appropriate acceptance radius. We evaluated varying acceptance radii in a nodule detection task in chest radiography and suggest guidelines for determining an acceptance radius. Methods: 80 chest radiographs were chosen, half of which contained nodules. We determined each nodule's centre. 21 radiologists read the images. We created acceptance radii bins of <5 pixels, <10 pixels, <20 pixels and onwards up to <200 and 200+ pixels. We counted lesion localisations in each bin and visually compared marks with the borders of nodules. Results: Most reader marks were tightly clustered around nodule centres, with tighter clustering for smaller than for larger nodules. At least 70% of readers' marks were placed within <10 pixels for small nodules, <20 pixels for medium nodules and <30 pixels for large nodules. Of 72 inspected marks that were less than 50 pixels from the centre of a nodule, only 1 fell outside the border of a nodule. Conclusion: The acceptance radius should be based on the larger nodule sizes. For our data, an acceptance radius of 50 pixels would have captured all but 2 reader marks within the borders of a nodule, while excluding only 1 true-positive mark. The choice of an acceptance radius for FROC analysis of observer performance studies should be based on the size of larger abnormalities.
UR - http://www.scopus.com/inward/record.url?scp=84871875724&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84871875724&partnerID=8YFLogxK
U2 - 10.1259/bjr/42313554
DO - 10.1259/bjr/42313554
M3 - Article
C2 - 22573302
AN - SCOPUS:84871875724
SN - 0007-1285
VL - 86
JO - British Journal of Radiology
JF - British Journal of Radiology
IS - 1021
M1 - 42313554
ER -