Human-designed visual manuals are crucial components in shape assembly activities. They provide step-by-step guidance on how we should move and connect different parts in a convenient and physically-realizable way. While there has been an ongoing effort in building agents that perform assembly tasks, the information in human-design manuals has been largely overlooked. We identify that this is due to 1) a lack of realistic 3D assembly objects that have paired manuals and 2) the difficulty of extracting structured information from purely image-based manuals. Motivated by this observation, we present IKEA-Manual, a dataset consisting of 102 IKEA objects paired with assembly manuals. We provide fine-grained annotations on the IKEA objects and assembly manuals, including decomposed assembly parts, assembly plans, manual segmentation, and 2D-3D correspondence between 3D parts and visual manuals. We illustrate the broad application of our dataset on four tasks related to shape assembly: assembly plan generation, part segmentation, pose estimation, and 3D part assembly.
翻译:人类设计的视觉手册是形状组装活动的关键组成部分,它们为我们如何以方便和实际可行的方式移动和连接不同部分提供了逐步的指导。虽然一直在努力建设执行组装任务的代理物,但人类设计手册中的信息基本上被忽视。我们确认,这是因为:(1) 缺乏现实的三维组装对象,这些物体已经对齐了手册,(2) 从纯图像的手册中提取结构化信息有困难。根据这一观察,我们介绍了由102个IKEA物体和组装手册组成的数据集。我们提供了关于IKEA物体和组装手册的精细标记说明,包括组装部件、组装计划、人工分解、3D部件和视觉手册之间的2D-3D通信。我们举例说明了我们的数据集在与组装有关的四项任务上的广泛应用:组装计划的生成、部分分解、估计和3D部件组装。我们介绍了我们的数据集在与组装有关的四项任务上的广泛应用情况:组装设计:组装计划生成、部分分解、形状估计和3D部件组装。