Neural Language Models of Code, or Neural Code Models (NCMs), are rapidly progressing from research prototypes to commercial developer tools. As such, understanding the capabilities and limitations of such models is becoming critical. However, the abilities of these models are typically measured using automated metrics that often only reveal a portion of their real-world performance. While, in general, the performance of NCMs appears promising, currently much is unknown about how such models arrive at decisions. To this end, this paper introduces $do_{code}$, a post-hoc interpretability methodology specific to NCMs that is capable of explaining model predictions. $do_{code}$ is based upon causal inference to enable programming language-oriented explanations. While the theoretical underpinnings of $do_{code}$ are extensible to exploring different model properties, we provide a concrete instantiation that aims to mitigate the impact of spurious correlations by grounding explanations of model behavior in properties of programming languages. To demonstrate the practical benefit of $do_{code}$, we illustrate the insights that our framework can provide by performing a case study on two popular deep learning architectures and nine NCMs. The results of this case study illustrate that our studied NCMs are sensitive to changes in code syntax and statistically learn to predict tokens related to blocks of code (e.g., brackets, parenthesis, semicolon) with less confounding bias as compared to other programming language constructs. These insights demonstrate the potential of $do_{code}$ as a useful model debugging mechanism that may aid in discovering biases and limitations in NCMs.
翻译:代码的神经语言模型或神经代码模型(NCM)正在从研究原型迅速进步到商业开发工具。 因此,理解这些模型的能力和局限性正在变得至关重要。 但是,这些模型的能力通常使用自动度量来衡量,往往只显示其真实世界性能的一部分。 虽然一般来说,NCMS的表现看似有希望,但对于这些模型是如何形成决定的,目前还不清楚。为此,本文件引入了$do ⁇ code}美元,这是一个适合NCMS的、能够解释模型预测的后可理解性方法。 $do ⁇ code}美元基于因果推论,以便能够编程以语言为导向的解释解释。尽管美元代码的理论基础通常能够用来探索不同的模型属性。我们提供了具体的即时速,目的是通过用模型的特性解释模型行为来减轻这些模型行为的影响。为了证明$do ⁇ codecodecol$的实用效益,我们的框架可以通过进行案例研究来解释。 $dodo{codecode$(codeal)$) 美元, 美元基于因果关系推算性推算性推算, 也就是我们国家数据库的模型模型的模型的模型的模型和模型的模型的模型的模型的模型的模型的模型的模型的模型的模型的模型的模型的模型的变数, 和模型的变数, 解算法学变法学变的模型的解算法学的模型的变。