Evaluation Challenges for Bridging Semantic Gap: Shape Disagreements on Pulmonary Nodules in the Lung Image Database Consortium

Evaluating the success of prediction and retrieval systems depends upon a reliable reference standard, a ground truth. The ideal gold standard is expected to result from the marking, labeling, and rating of domain experts. However experts disagree and lack of agreement challenges the development and evaluation of image-based feature prediction. This paper addresses the success and limitations in bridging the semantic gap between CT-based pulmonary nodule image features and radiologists’ ratings of diagnostic characteristics. The prediction of diagnostic characteristics promises to automatically annotate images with medically meaningful descriptions usable for indexing and retrieving in content-based image retrieval (CBIR) and computer aided diagnosis (CADx). Successful results in predicting texture characteristics will be contrasted with less successful results for boundary shapes. The two primary differences in agreement between radiologists will be discussed; the first concerns agreement about the existence of a nodule, while the second considers the variability in radiologists’ ratings.