The task of learning to map an input set onto a permuted sequence of its elements is challenging for neural networks. Set-to-sequence problems occur in natural language processing, computer vision and structure prediction, where interactions between elements of large sets define the optimal output. Models must exhibit relational reasoning, handle varying cardinalities and manage combinatorial complexity. Previous attention-based methods require $n$ layers of their set transformations to explicitly represent $n$-th order relations. Our aim is to enhance their ability to efficiently model higher-order interactions through an additional interdependence component. We propose a novel neural set encoding method called the Set Interdependence Transformer, capable of relating the set's permutation invariant representation to its elements within sets of any cardinality. We combine it with a permutation learning module into a complete, 3-part set-to-sequence model and demonstrate its state-of-the-art performance on a number of tasks. These range from combinatorial optimization problems, through permutation learning challenges on both synthetic and established NLP datasets for sentence ordering, to a novel domain of product catalog structure prediction. Additionally, the network's ability to generalize to unseen sequence lengths is investigated and a comparative empirical analysis of the existing methods' ability to learn higher-order interactions is provided.
翻译:对神经网络来说,将输入内容映射成一个不固定的序列的任务对神经网络来说具有挑战性。在自然语言处理、计算机视觉和结构预测中,将设置到顺序的问题出现,其中大组合各元素之间的相互作用决定了最佳输出。模型必须展示关联推理,处理不同的基本特征,并管理组合复杂性。以前基于关注的方法要求其设定变换的层数,以明确代表美元-顺序关系。我们的目标是通过额外的相互依存部分,提高它们高效模拟更高顺序互动的能力。我们提出了一套新的神经元集编码方法,称为“设置自定义变换器”,能够将集的变异表达方式与其在任何基本性范围内的元素联系起来。我们将其与一个全套、三部分的组合到顺序的学习模块结合起来,并展示其在若干任务上的状态性能。从组合优化问题,通过对组合和已建立的用于排序的NLP数据集进行调换学习的挑战,我们提出了一套新式的神经元数据集,能够将数据集的变异式表达方式与任何基本基本基本的基本基本能力分析方法结合起来。