In recent years, automated approaches to assessing linguistic complexity in second language (L2) writing have made significant progress in gauging learner performance, predicting human ratings of the quality of learner productions, and benchmarking L2 development. In contrast, there is comparatively little work in the area of speaking, particularly with respect to fully automated approaches to assessing L2 spontaneous speech. While the importance of a well-performing ASR system is widely recognized, little research has been conducted to investigate the impact of its performance on subsequent automatic text analysis. In this paper, we focus on this issue and examine the impact of using a state-of-the-art ASR system for subsequent automatic analysis of linguistic complexity in spontaneously produced L2 speech. A set of 34 selected measures were considered, falling into four categories: syntactic, lexical, n-gram frequency, and information-theoretic measures. The agreement between the scores for these measures obtained on the basis of ASR-generated vs. manual transcriptions was determined through correlation analysis. A more differential effect of ASR performance on specific types of complexity measures when controlling for task type effects is also presented.
翻译:近年来,评估第二语言语言复杂性(L2)写作的自动化方法在衡量学习者业绩、预测学习者生产质量的人类评级和衡量L2发展基准方面取得了显著进展,相反,在演讲领域,特别是在完全自动化评估L2自发演讲方法方面,相对而言,工作较少;虽然普遍认识到良好表现的ASR系统的重要性,但很少进行研究,以调查其表现对随后自动文本分析的影响;在本文件中,我们着重研究这一问题,并研究使用最先进的ASR系统随后自动分析自发制作的L2语言复杂性的影响;考虑了一系列34项选定措施,可分为四类:合成、词汇、n-gram频率和信息-理论措施;根据ASR产生的对措施的分数通过相关分析确定。还介绍了ASR在控制任务类型影响时对特定类型复杂措施的更不同影响。