We address the problem of tracking 3D object poses from touch during in-hand manipulations. Specifically, we look at tracking small objects using vision-based tactile sensors that provide high-dimensional tactile image measurements at the point of contact. While prior work has relied on a-priori information about the object being localized, we remove this requirement. Our key insight is that an object is composed of several local surface patches, each informative enough to achieve reliable object tracking. Moreover, we can recover the geometry of this local patch online by extracting local surface normal information embedded in each tactile image. We propose a novel two-stage approach. First, we learn a mapping from tactile images to surface normals using an image translation network. Second, we use these surface normals within a factor graph to both reconstruct a local patch map and use it to infer 3D object poses. We demonstrate reliable object tracking for over 100 contact sequences across unique shapes with four objects in simulation and two objects in the real-world. Supplementary video: https://youtu.be/JwNTC9_nh8M
翻译:我们解决了在手动操作过程中从触摸到3D天体构成的跟踪问题。 具体地说, 我们用基于视觉的触觉传感器来跟踪小物体, 在接触点提供高维触动图像测量。 虽然先前的工作依赖于关于天体局部化的优先信息, 我们删除了这一要求。 我们的关键洞察力是, 对象由几个本地表面补丁组成, 每个都有足够的信息, 足以实现可靠的天体跟踪。 此外, 我们可以通过提取嵌入每张触摸图像的本地表面正常信息, 来恢复本地补丁的几何性。 我们提出一个新的两阶段方法。 首先, 我们用图像翻译网络从触摸图图像到正常地表。 其次, 我们使用一个要素图中的这些表面常态, 来重建本地补丁图, 并用它来推断 3D 对象的配置。 我们展示了可靠的天体跟踪对象, 超过100个独特的形状的接触序列, 包括模拟中的4个物体和现实世界中的2个物体。 补充视频: http://youtu.be/ JwNC9_ nh8MMM。