Topic models have been widely used to learn representations from text and gain insight into document corpora. To perform topic discovery, existing neural models use document bag-of-words (BoW) representation as input followed by variational inference and learn topic-word distribution through reconstructing BoW. Such methods have mainly focused on analysing the effect of enforcing suitable priors on document distribution. However, little importance has been given to encoding improved document features for capturing document semantics better. In this work, we propose a novel framework: TAN-NTM which models document as a sequence of tokens instead of BoW at the input layer and processes it through an LSTM whose output is used to perform variational inference followed by BoW decoding. We apply attention on LSTM outputs to empower the model to attend on relevant words which convey topic related cues. We hypothesise that attention can be performed effectively if done in a topic guided manner and establish this empirically through ablations. We factor in topic-word distribution to perform topic aware attention achieving state-of-the-art results with ~9-15 percentage improvement over score of existing SOTA topic models in NPMI coherence metric on four benchmark datasets - 20NewsGroup, Yelp, AGNews, DBpedia. TAN-NTM also obtains better document classification accuracy owing to learning improved document-topic features. We qualitatively discuss that attention mechanism enables unsupervised discovery of keywords. Motivated by this, we further show that our proposed framework achieves state-of-the-art performance on topic aware supervised generation of keyphrases on StackExchange and Weibo datasets.
翻译:为了进行专题发现,现有神经模型使用文档组合词包(BoW)表示作为投入,然后进行变异推断,并通过重建 BoW 学习主题字的分布。这些方法主要侧重于分析在文件分发方面执行适当前科的影响。然而,很少重视将改进的文件特性编码,以便更好地获取文件语义。在这项工作中,我们提议了一个新颖的框架:TAN-NTM,该模型以输入层的标志顺序而不是BoW来记录,并通过LSTM进行处理,其产出被用来进行变异推断,然后是BOW去解码。我们注重LSTM产出,以便让该模型能够参加传递主题相关提示的相关词。我们假设,如果以主题指导的方式进行,并通过推理来确立这一经验性。我们把注意力纳入主题语言分配,以关注在输入层中实现状态结果,在输入输入输入输入输入的值为~15百分率的LSTMTM 进行流程过程过程。我们通过在现有的SOTA数据库中改进了质量数据模型的评分数,从而获得更好的数据。