Open-vocabulary object detection (OVD) aims to scale up vocabulary size to detect objects of novel categories beyond the training vocabulary. Recent work resorts to the rich knowledge in pre-trained vision-language models. However, existing methods are ineffective in proposal-level vision-language alignment. Meanwhile, the models usually suffer from confidence bias toward base categories and perform worse on novel ones. To overcome the challenges, we present MEDet, a novel and effective OVD framework with proposal mining and prediction equalization. First, we design an online proposal mining to refine the inherited vision-semantic knowledge from coarse to fine, allowing for proposal-level detection-oriented feature alignment. Second, based on causal inference theory, we introduce a class-wise backdoor adjustment to reinforce the predictions on novel categories to improve the overall OVD performance. Extensive experiments on COCO and LVIS benchmarks verify the superiority of MEDet over the competing approaches in detecting objects of novel categories, e.g., 32.6% AP50 on COCO and 22.4% mask mAP on LVIS.
翻译:开放词汇对象探测(OVD)旨在扩大词汇规模,以探测培训词汇以外的新类别物体。最近的工作在经过培训的视觉语言模型中依靠丰富的知识。但是,现有的方法在建议层面的视觉语言调整方面是无效的。与此同时,模型通常对基类产生信心偏向,对新颖类别则表现更差。为了克服挑战,我们提出MEDet,这是一个创新和有效的OVD框架,提出了采矿和预测均衡的建议。首先,我们设计了一个在线采矿提案,以完善从粗皮到精细的传承的视觉-语学知识,允许建议层面的探测导向特征调整。第二,根据因果推断理论,我们引入了从阶级角度出发的后门调整,以加强对新类别作出的预测,以提高OVD的总体绩效。关于CO和LVIS基准的广泛实验核实MET优于新类别物体探测的竞争性方法,例如,CO50的AP50占32.6%,LVIS的MAP22.4%。