Exploring the generalization of a text-to-SQL parser is essential for a system to automatically adapt the real-world databases. Previous works provided investigations focusing on lexical diversity, including the influence of the synonym and perturbations in both natural language questions and databases. However, research on the structure variety of database schema~(DS) is deficient. Specifically, confronted with the same input question, the target SQL is probably represented in different ways when the DS comes to a different structure. In this work, we provide in-deep discussions about the structural generalization of text-to-SQL tasks. We observe that current datasets are too templated to study structural generalization. To collect eligible test data, we propose a framework to generate novel text-to-SQL data via automatic and synchronous (DS, SQL) pair altering. In the experiments, significant performance reduction when evaluating well-trained text-to-SQL models on the synthetic samples demonstrates the limitation of current research regarding structural generalization. According to comprehensive analysis, we suggest the practical reason is the overfitting of (NL, SQL) patterns.
翻译:探索文本到 SQL 剖析器的通用性对于自动调整真实世界数据库的系统来说至关重要。 以前的作品提供了侧重于词汇多样性的调查, 包括自然语言问题和数据库中的同义词和扰动的影响。 但是,对数据库 schema- (DS) 结构多样性的研究存在缺陷。 具体地说, 面对相同的输入问题, 目标 SQL 可能以不同的方式表现为当DS 进入不同的结构时。 在这项工作中, 我们提供有关文本到 SQL 任务结构化的深入讨论。 我们观察到当前数据集的模板太过广,无法研究结构化。 为了收集符合资格的测试数据, 我们提出了一个框架,通过自动同步( DS, SQL) 配对生成新的文本到 SQL 数据。 在实验中, 评估合成样品经过良好培训的文本到 SQL 模型时, 显著的性能下降。 在全面分析中, 我们建议的实际原因是( NL, SQL) 模式过于完善。