We introduce NIFT, Neural Interaction Field and Template, a descriptive and robust interaction representation of object manipulations to facilitate imitation learning. Given a few object manipulation demos, NIFT guides the generation of the interaction imitation for a new object instance by matching the Neural Interaction Template (NIT) extracted from the demos to the Neural Interaction Field (NIF) defined for the new object. Specifically, the NIF is a neural field which encodes the relationship between each spatial point and a given object, where the relative position is defined by a spherical distance function rather than occupancies or signed distances, which are commonly adopted by conventional neural fields but less informative. For a given demo interaction, the corresponding NIT is defined by a set of spatial points sampled in the NIF of the demo object with associated neural features. To better capture the interaction, the points are sampled on the interaction bisector surface, which consists of points that are equidistant to two interacting objects and has been used extensively for interaction representation. With both point selection and pointwise features defined for better interaction encoding, NIT effectively guides the feature matching in the NIFs of the new object instances to optimize the object poses to realize the manipulation while imitating the demo interactions. Experiments show that our NIFT solution outperforms state-of-the-art imitation learning methods for object manipulation and generalizes better to objects from new categories.
翻译:我们引入了 NIF, 神经互动字段和模板, 一个描述性和强健的互动演示, 显示物体操纵的物体操作, 以便利模仿学习。 在少数物体操纵演示中, NIF 通过匹配从演示中提取的神经互动模版( NIT) 和为新对象定义的神经互动场( NIF), 指导为新对象生成的互动仿真。 具体地说, NIF 是一个神经场, 它将每个空间点和给定对象之间的关系编码起来, 相对位置由球形距离功能而不是占用或签名距离来定义, 常规神经场通常采用这种功能, 但信息量较少。 对于给定的演示互动, 相应的 NIT 由一组空间点来定义, 在演示对象的NIFS 样本中抽取一组空间点, 与相关的神经特征相匹配。 为了更好地捕捉互动, 这些点是在互动中标, 由两个相近于两个互动对象的点组成, 并被广泛用于互动演示。, 由点选择和点定位功能来更好地相互校正校正校正的物体, NIT,,, 将功能的特性定位到演示模型化到演示国 。