An Empirical Investigation on the Challenges of Creating Custom Static Analysis Rules for Defect Localization

Background: Custom static analysis rules, i.e., rules specific for one or more applications, have been successfully applied to perform corrective and preventive software maintenance. Their usage can reduce the costs of verification and improve the reliability and security of applications. Pattern-Driven Maintenance (PDM) is a method designed to support the creation of such rules during software maintenance. However, as PDM was recently created, few maintainers have reported on its usage. Hence, the challenges and skills needed to apply PDM properly are unknown. Aims: In this paper, we investigate the challenges faced by maintainers on applying PDM for creating custom static analysis rules for defect localization. Method: We conducted an observational study on novice maintainers creating custom static analysis rules by applying PDM. The study was divided into three tasks: (i) identifying a defect pattern, (ii) programming a static analysis rule to locate instances of the pattern, and (iii) verifying the located instances. We analyzed the efficiency of maintainers on applying each task and their comments on task challenges. We also analyzed the acceptance of PDM by the maintainers. Results: We observed that previous knowledge on debugging, the subject software, and related technologies influenced the performance of maintainers. However, the method's bottleneck was static analysis rules programming, being the task that maintainers had more difficulties in completing. Besides those difficulties, maintainers found PDM useful and demonstrated the intention of using it in practice. Conclusions: The results strengthen our confidence that PDM can help maintainers in producing custom static analysis rules for locating defects. However, a better approach for programming those rules and the proper selection and training of maintainers is needed to apply PDM effectively.