In this paper we explore the task of modeling (semi) structured object sequences; in particular we focus our attention on the problem of developing a structure-aware input representation for such sequences. In such sequences, we assume that each structured object is represented by a set of key-value pairs which encode the attributes of the structured object. Given a universe of keys, a sequence of structured objects can then be viewed as an evolution of the values for each key, over time. We encode and construct a sequential representation using the values for a particular key (Temporal Value Modeling - TVM) and then self-attend over the set of key-conditioned value sequences to a create a representation of the structured object sequence (Key Aggregation - KA). We pre-train and fine-tune the two components independently and present an innovative training schedule that interleaves the training of both modules with shared attention heads. We find that this iterative two part-training results in better performance than a unified network with hierarchical encoding as well as over, other methods that use a {\em record-view} representation of the sequence \cite{de2021transformers4rec} or a simple {\em flattened} representation of the sequence. We conduct experiments using real-world data to demonstrate the advantage of interleaving TVM-KA on multiple tasks and detailed ablation studies motivating our modeling choices. We find that our approach performs better than flattening sequence objects and also allows us to operate on significantly larger sequences than existing methods.
翻译:在本文中,我们探索了建模( 半) 结构化对象序列的任务; 特别是我们把注意力集中在为这些序列开发结构认知的输入代表器的问题上。 在这样的序列中, 我们假设每个结构化对象都由一组关键值对配对来代表结构化对象的属性。 在一个由键组成的宇宙中, 一个结构化对象序列可以被视为每个关键值的演进, 随时间推移。 我们用特定关键值( 临时值建模 - TVM) 的值来编码和构建一个顺序代表器。 然后在关键设定值序列的组合中, 将我们自我调整的每个结构化对象由一组关键值对配对来代表一个结构化对象序列( Key Agregation - KAKA) 。 我们预先调整和微调这两个组成部分, 并展示一个创新的培训时间表, 将两个模块与共同关注对象的训练相隔开来。 我们发现, 迭交式的两个模型比一个统一的网络的性更好, 分级编码, 并超越了其他方法, 使用一个固定的系统- 记录- 运行一个更精确的顺序的顺序的顺序的顺序的顺序, 显示一个更精确的顺序的顺序的顺序的顺序, 并显示一个更精确的顺序的顺序的顺序。