This paper presents ViDeBERTa, a new pre-trained monolingual language model for Vietnamese, with three versions - ViDeBERTa_xsmall, ViDeBERTa_base, and ViDeBERTa_large, which are pre-trained on a large-scale corpus of high-quality and diverse Vietnamese texts using DeBERTa architecture. Although many successful pre-trained language models based on Transformer have been widely proposed for the English language, there are still few pre-trained models for Vietnamese, a low-resource language, that perform good results on downstream tasks, especially Question answering. We fine-tune and evaluate our model on three important natural language downstream tasks, Part-of-speech tagging, Named-entity recognition, and Question answering. The empirical results demonstrate that ViDeBERTa with far fewer parameters surpasses the previous state-of-the-art models on multiple Vietnamese-specific natural language understanding tasks. Notably, ViDeBERTa_base with 86M parameters, which is only about 23% of PhoBERT_large with 370M parameters, still performs the same or better results than the previous state-of-the-art model. Our ViDeBERTa models are available at: https://github.com/HySonLab/ViDeBERTa.
翻译:本文展示了ViDeBERTA,这是越南语的一个新的经过预先训练的单一语言模式,有三种版本,即ViDeBERTA_xsmill、ViDeBERTA_base和ViDeBERTA_pration,这些版本在使用DeBERTA结构的大规模高质量和多样化越南文本中经过预先培训。虽然许多以变异器为基础的成功经过训练的语言模式已被广泛推荐用于英语,但对于越南语来说,仍然很少有经过预先培训的越南语模式,这种语言是一种低资源语言,在下游任务上取得了良好的效果,特别是回答问题。我们微调了我们关于三种重要自然语言下游任务的模式,即部分语音标记、命名实体识别和问题回答。经验结果显示,ViDeBERTA,其参数远远少于以前越南特定自然语言理解任务方面的先进模式。值得注意的是,ViDeBERTA_base,它只有约23%的PhoBERT_ma参数,比370M参数还要高,仍然在前一个州-VABAR_VAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR_BAR