Text generative models trained via Maximum Likelihood Estimation (MLE) suffer from the notorious exposure bias problem, and Generative Adversarial Networks (GANs) are shown to have potential to tackle it. Existing language GANs adopt estimators like REINFORCE or continuous relaxations to model word distributions. The inherent limitations of such estimators lead current models to rely on pre-training techniques (MLE pre-training or pre-trained embeddings). Representation modeling methods which are free from those limitations, however, are seldom explored because of its poor performance in previous attempts. Our analyses reveal that invalid sampling method and unhealthy gradients are the main contributors to its unsatisfactory performance. In this work, we present two techniques to tackle these problems: dropout sampling and fully normalized LSTM. Based on these two techniques, we propose InitialGAN whose parameters are randomly initialized completely. Besides, we introduce a new evaluation metric, Least Coverage Rate, to better evaluate the quality of generated samples. The experimental results demonstrate that InitialGAN outperforms both MLE and other compared models. To the best of our knowledge, it is the first time a language GAN can outperform MLE without any pre-training techniques.
翻译:通过最大隐隐性估计(MLE)所培训的文本变色模型存在臭名昭著的暴露偏差问题,而General Adversarial Networks(GANs)已证明有可能解决这一问题。现有语言GANs采用REINFORCE等估算器,或不断放松文字分布模式。这种估算器的内在局限性导致当前模型完全依赖培训前技术(MLE培训前或预先培训前的嵌入)。但是,没有这些局限性的代表模型很少被探索,因为其以往尝试的表现不佳。我们的分析表明,无效的取样方法和不健康的梯度是其不令人满意的表现的主要促成者。在这项工作中,我们提出了解决这些问题的两种技术:辍学取样和完全正常化LSTM。根据这两种技术,我们建议MinisterGAN模型的参数完全随机初始化。此外,我们引入了新的评价指标,最小覆盖率,以更好地评估生成样品的质量。实验结果表明,初始GAN(GAN)超越了MLE和其他比较模型的完美性。对于我们的知识来说,任何最佳的学习前GAN语言都比起来。