Neural networks that compute over graph structures are a natural fit for problems in a variety of domains, including natural language (parse trees) and cheminformatics (molecular graphs). However, since the computation graph has a different shape and size for every input, such networks do not directly support batched training or inference. They are also difficult to implement in popular deep learning libraries, which are based on static data-flow graphs. We introduce a technique called dynamic batching, which not only batches together operations between different input graphs of dissimilar shape, but also between different nodes within a single input graph. The technique allows us to create static graphs, using popular libraries, that emulate dynamic computation graphs of arbitrary shape and size. We further present a high-level library of compositional blocks that simplifies the creation of dynamic graph models. Using the library, we demonstrate concise and batch-wise parallel implementations for a variety of models from the literature.
翻译:计算图形结构的神经网络自然适合不同领域的问题,包括自然语言(分层树)和化学信息学(分子图)。然而,由于计算图表的形状和大小各异,这些网络并不直接支持分批培训或推断。它们也难以在以静态数据流图表为基础的广受欢迎的深层学习图书馆中实施。我们引入了一种称为动态批量的技术,它不仅将不同形状的不同输入图之间的操作组合在一起,而且将不同输入图的不同节点组合在一起。该技术使我们能够利用流行的图书馆创建静态图表,以模拟任意形状和大小的动态计算图。我们进一步展示了一个高层次的构造块库,简化了动态图形模型的创建。我们利用图书馆展示了各种文献模型的简洁和分批平行实施。