Improving Vulnerability Inspection Efficiency Using Active Learning

Society needs more secure software. But the subject matter experts in software security are in short supply. Hence, development teams are motivated to make the most of their limited time. The goal of this paper is to improve software vulnerability inspection efficiency via an active learning based inspection support tool HARMLESS. HARMLESS incrementally updates its vulnerability prediction model (VPM) based on latest human inspection results and then applies the model to prioritize human inspection efforts to source code files that are more likely to contain vulnerabilities. HARMLESS is designed to have three advantages over conventional software vulnerability prediction methods. Firstly, by integrating human and vulnerability prediction model in an active learning environment, HARMLESS keeps refining its VPM and can find vulnerabilities with reduced human inspection effort before a software's first release. Secondly, by estimating the total number of vulnerabilities in a software project, HARMLESS guides human to stop the inspection at a target recall. Thirdly, HARMLESS applies redundant inspection (source code files inspected multiple times by different humans) on source code files that are more likely to contain missing vulnerabilities, so that vulnerabilities missed by human inspectors can be retrieved efficiently. We evaluate HARMLESS via a simulation with Mozilla Firefox vulnerability data. Our results show that (1) HARMLESS finds 60, 70, 80, 90, 95, 99% vulnerabilities by inspecting 6, 8, 10, 16, 20, 34% source code files, respectively. (2) During the simulation, when targeting at 90, 95, 99% recall, HARMLESS could stop early at 23, 30, 47% source code files inspected, respectively. (3) Even when human reviewers fail to identify half of the vulnerabilities, HARMLESS is able to cover 96% of the missing vulnerabilities by redundantly inspecting half of the classified files.