Recent studies on dense captioning and visual grounding in 3D have achieved impressive results. Despite developments in both areas, the limited amount of available 3D vision-language data causes overfitting issues for 3D visual grounding and 3D dense captioning methods. Also, how to discriminatively describe objects in complex 3D environments is not fully studied yet. To address these challenges, we present D3Net, an end-to-end neural speaker-listener architecture that can detect, describe and discriminate. Our D3Net unifies dense captioning and visual grounding in 3D in a self-critical manner. This self-critical property of D3Net also introduces discriminability during object caption generation and enables semi-supervised training on ScanNet data with partially annotated descriptions. Our method outperforms SOTA methods in both tasks on the ScanRefer dataset, surpassing the SOTA 3D dense captioning method by a significant margin.
翻译:尽管在这两个领域都取得了一些发展,但现有的三维视觉语言数据数量有限,导致三维视觉定位和三维密集字幕方法的问题过于合适。此外,对于如何对复杂三维环境中的物体进行区分描述,还没有进行充分研究。为了应对这些挑战,我们介绍了D3Net,这是一个端到端神经扬声器-听筒结构,可以探测、描述和区分。我们的D3Net以自我批评的方式统一了三维的密集字幕和视觉地面。D3Net的这种自我批评性属性在对象字幕生成过程中也引入了差异性,并能够进行带有部分附加说明说明的扫描网络数据半监督培训。我们的方法在扫描Refer数据集的两个任务中都超越了SOTA 3D稠密字幕方法,大大超过了SOTA 3D 密字幕方法。