As deep networks require large amounts of accurately labeled training data, a strategy to collect sufficiently large and accurate annotations is as important as innovations in recognition methods. This is especially true for building Computer Aided Detection (CAD) systems for chest X-rays where domain expertise of radiologists is required to annotate the presence and location of abnormalities on X-ray images. However, there lacks concrete evidence that provides guidance on how much resource to allocate for data annotation such that the resulting CAD system reaches desired performance. Without this knowledge, practitioners often fall back to the strategy of collecting as much detail as possible on as much data as possible which is cost inefficient. In this work, we investigate how the cost of data annotation ultimately impacts the CAD model performance on classification and segmentation of chest abnormalities in frontal-view X-ray images. We define the cost of annotation with respect to the following three dimensions: quantity, quality and granularity of labels. Throughout this study, we isolate the impact of each dimension on the resulting CAD model performance on detecting 10 chest abnormalities in X-rays. On a large scale training data with over 120K X-ray images with gold-standard annotations, we find that cost-efficient annotations provide great value when collected in large amounts and lead to competitive performance when compared to models trained with only gold-standard annotations. We also find that combining large amounts of cost efficient annotations with only small amounts of expensive labels leads to competitive CAD models at a much lower cost.
翻译:由于深网络需要大量贴有准确标签的培训数据,收集足够大和准确的说明的战略与表彰方法的创新同样重要,对于建立胸X光计算机辅助检测(CAD)系统来说尤其如此,因为需要放射学家的域域专长来说明X光图像异常的存在和位置;然而,缺乏具体证据来指导为数据注释分配多少资源,如由此产生的CAD系统达到预期业绩,从而得出大量准确的培训数据说明;没有这种知识,从业人员往往要回到尽可能详细收集成本低、尽可能多的数据的战略;在这项工作中,我们调查数据说明的成本最终如何影响CAD模型在前视X光图像中胸异常的分类和分解方面的功能;然而,我们确定以下三个层面的注释成本:标签的数量、质量和颗粒性。我们从每个层面分离出CAD模型在发现X光中发现10个胸异常的尽可能低的成本效率数据;在大规模成本标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准培训数据、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准培训数据、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准培训数据、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准、高标准