Error attribution in Large Language Model (LLM) multi-agent systems presents a significant challenge in debugging and improving collaborative AI systems. Current approaches to pinpointing agent and step level failures in interaction traces - whether using all-at-once evaluation, step-by-step analysis, or binary search - fall short when analyzing complex patterns, struggling with both accuracy and consistency. We present ECHO (Error attribution through Contextual Hierarchy and Objective consensus analysis), a novel algorithm that combines hierarchical context representation, objective analysis-based evaluation, and consensus voting to improve error attribution accuracy. Our approach leverages a positional-based leveling of contextual understanding while maintaining objective evaluation criteria, ultimately reaching conclusions through a consensus mechanism. Experimental results demonstrate that ECHO outperforms existing methods across various multi-agent interaction scenarios, showing particular strength in cases involving subtle reasoning errors and complex interdependencies. Our findings suggest that leveraging these concepts of structured, hierarchical context representation combined with consensus-based objective decision-making, provides a more robust framework for error attribution in multi-agent systems.
翻译：大型语言模型（LLM）多智能体系统中的错误归因，对于调试和改进协作式人工智能系统构成了一项重大挑战。当前用于定位交互轨迹中智能体与步骤层面故障的方法——无论是采用一次性评估、逐步分析还是二分搜索——在分析复杂模式时均存在不足，难以兼顾准确性与一致性。本文提出ECHO（基于上下文层次结构与客观共识分析的错误归因算法），这是一种结合层次化上下文表征、基于客观分析的评估以及共识投票机制的新型算法，旨在提升错误归因的准确性。我们的方法通过基于位置的层级化上下文理解，同时保持客观评估标准，最终通过共识机制得出结论。实验结果表明，ECHO在多种多智能体交互场景中均优于现有方法，尤其在涉及细微推理错误与复杂相互依赖性的案例中表现出显著优势。我们的研究结果表明，利用这种结构化、层次化的上下文表征概念，结合基于共识的客观决策机制，能为多智能体系统的错误归因提供一个更稳健的框架。
相关内容
微信扫码咨询专知VIP会员