Using Localization Data from Image Interpretations to Improve Estimates of Performance Accuracy

A recently developed model uses the localization of abnormalities on images to improve statistical precision in measuring detection accuracy A z , the area below an observer's receiver operating characteristic (ROC) curve for ratings of sampled normal and abnormal cases. This study evaluated that improvement by investigating how much the standard error of estimated A z decreased when the statistical analysis included localization data. Comparisons of analyses with vs without localizations were made for: 1) the estimates of A z from observers' rating ROC curves for nodular lesions on clinical chest films and liver CT scans; 2) the probability of correct choices between paired samples of normal and abnormal cases (equivalent to A z ); and 3) the sampling distributions of A z measured in Monte Carlo simulations of 2,000 independent rating experiments. Localization information considerably improved the precision of A z estimates, particularly when detection accuracy was low ( A z ~ 0.60). These data provided roughly the same benefits in estimation precision as would two-to-fourfold increases in the sizes of both 1) the samples of positive and negative cases and 2) the observer samples used to estimate A z means. Key words: ROC analysis; LROC; observer performance accuracy; observer vanability; detection; localization; image interpretation. (Med Decis Making 2000;20:170-185)