As a fundamental task in natural language processing, Chinese Grammatical Error Correction (CGEC) has gradually received widespread attention and become a research hotspot. However, one obvious deficiency for the existing CGEC evaluation system is that the evaluation values are significantly influenced by the Chinese word segmentation results or different language models. The evaluation values of the same error correction model can vary considerably under different word segmentation systems or different language models. However, it is expected that these metrics should be independent of the word segmentation results and language models, as they may lead to a lack of uniqueness and comparability in the evaluation of different methods. To this end, we propose three novel evaluation metrics for CGEC in two dimensions: reference-based and reference-less. In terms of the reference-based metric, we introduce sentence-level accuracy and char-level BLEU to evaluate the corrected sentences. Besides, in terms of the reference-less metric, we adopt char-level meaning preservation to measure the semantic preservation degree of the corrected sentences. We deeply evaluate and analyze the reasonableness and validity of the three proposed metrics, and we expect them to become a new standard for CGEC.
翻译:作为自然语言处理的基本任务,中国国名错误校正(CGEC)逐渐受到广泛关注,成为研究热点,但是,现有国名错误校正(CGEC)评价系统的一个明显缺陷是,评价值受到中文字分解结果或不同语言模式的重大影响。同一错误校正模式的评价值在不同字分解系统或不同语言模式下可能有很大差异。然而,预计这些指标应当独立于字分解结果和语言模式,因为它们可能导致不同方法评价缺乏独特性和可比性。为此,我们提议了三套新的评价指标,即基于参考和无参考。在基于参考的衡量标准方面,我们引入了判决级准确度和BLEU等级来评价经更正的句子。此外,在无参考标准方面,我们采用字级保护的含义来衡量经校正判决的语义保存程度。我们深入评估和分析了三项拟议衡量标准的合理性和有效性,我们期望它们成为该CC的新标准。