We evaluated the capability of generative pre-trained transformers (GPT), to pass assessments in introductory and intermediate Python programming courses at the postsecondary level. Discussions of potential uses (e.g., exercise generation, code explanation) and misuses (e.g., cheating) of this emerging technology in programming education have intensified, but to date there has not been a rigorous analysis of the models' capabilities in the realistic context of a full-fledged programming course with diverse set of assessment instruments. We evaluated GPT on three Python courses that employ assessments ranging from simple multiple-choice questions (no code involved) to complex programming projects with code bases distributed into multiple files (599 exercises overall). Further, we studied if and how successfully GPT models leverage feedback provided by an auto-grader. We found that the current models are not capable of passing the full spectrum of assessments typically involved in a Python programming course (<70% on even entry-level modules). Yet, it is clear that a straightforward application of these easily accessible models could enable a learner to obtain a non-trivial portion of the overall available score (>55%) in introductory and intermediate courses alike. While the models exhibit remarkable capabilities, including correcting solutions based on auto-grader's feedback, some limitations exist (e.g., poor handling of exercises requiring complex chains of reasoning steps). These findings can be leveraged by instructors wishing to adapt their assessments so that GPT becomes a valuable assistant for a learner as opposed to an end-to-end solution.
翻译:我们评估了基因改造前受过训练的变压器(GPT)的能力,以便在中学后一级的入门和中间俾丹方案课程中通过评估,在入门和中间俾丹方案课程中通过评估。关于这种新兴技术在方案规划教育中的潜在用途(例如,练习生成、代码解释)和滥用(例如,欺骗)的讨论已经得到加强,但是,迄今为止,还没有在一个全面的方案拟订课程的现实背景下,对模型的能力进行严格的分析,其中包括一套全方位的方案拟定课程,并配有一套不同的评估工具。我们评估了三次Python课程中的Gython方案能力,这些课程利用了从简单的多选问题(不涉及代码)到复杂的方案拟定项目的评估,其代码基础分布在多个文档中(共599项演习)。 此外,我们研究了全球贸易点模式是否和如何成功地利用了由自动毕业者提供的这种技术的反馈。我们发现,目前的模型无法在Python方案拟定课程中通常涉及的全部评估(在入门阶段的模块上 < 70% ) 。然而,显然,这些易于获取的适应的模式可以直接应用使学习者获得非三分部分的全面评价,在总体的学习阶段的学习阶段,在总的推算方法中(例如,在示范课程中的推介阶段的推介中,有某种推介式的推介。 ),这些推介式的推介式的推介方法,在了某种推介能力。</s>