The use of Large Language Models (LLMs) to support tasks in software development has steadily increased over recent years. From assisting developers in coding activities to providing conversational agents that answer newcomers' questions. In collaboration with the Mozilla Foundation, this study evaluates the effectiveness of Retrieval-Augmented Generation (RAG) in assisting developers within the Mozilla Firefox project. We conducted an empirical analysis comparing responses from human developers, a standard GPT model, and a GPT model enhanced with RAG, using real queries from Mozilla's developer chat rooms. To ensure a rigorous evaluation, Mozilla experts assessed the responses based on helpfulness, comprehensiveness, and conciseness. The results show that RAG-assisted responses were more comprehensive than human developers (62.50% to 54.17%) and almost as helpful (75.00% to 79.17%), suggesting RAG's potential to enhance developer assistance. However, the RAG responses were not as concise and often verbose. The results show the potential to apply RAG-based tools to Open Source Software (OSS) to minimize the load to core maintainers without losing answer quality. Toning down retrieval mechanisms and making responses even shorter in the future would enhance developer assistance in massive projects like Mozilla Firefox.
翻译：近年来，大型语言模型（LLMs）在软件开发任务支持中的应用稳步增长，从协助开发人员进行编码活动到提供回答新手问题的对话代理。本研究与Mozilla基金会合作，评估了检索增强生成（RAG）在协助Mozilla Firefox项目内开发人员方面的有效性。我们进行了一项实证分析，通过使用Mozilla开发者聊天室中的真实查询，比较了人类开发人员、标准GPT模型以及经RAG增强的GPT模型的回答。为确保评估的严谨性，Mozilla专家根据回答的有用性、全面性和简洁性对回答进行了评估。结果显示，RAG辅助的回答比人类开发人员的回答更全面（62.50%对54.17%），并且几乎同样有用（75.00%对79.17%），这表明RAG具有增强开发人员协助的潜力。然而，RAG的回答不够简洁，常常显得冗长。结果表明，将基于RAG的工具应用于开源软件（OSS）有可能在保持回答质量的同时，减轻核心维护者的负担。未来若能优化检索机制并进一步缩短回答长度，将能更好地增强如Mozilla Firefox这类大型项目中的开发人员协助。
相关内容
微信扫码咨询专知VIP会员