Message Flow Analysis with Complex Causal Links for Distributed ROS 2 Systems

Distributed robotic systems rely heavily on publish-subscribe frameworks, such as ROS, to efficiently implement modular computation graphs. The ROS 2 executor, a high-level task scheduler which handles messages internally, is a performance bottleneck. In previous work, we presented ros2_tracing, a framework with instrumentation and tools for real-time tracing of ROS 2. We now extend on that instrumentation and leverage the tracing tools to propose an analysis and visualization of the flow of messages across distributed ROS 2 systems. Our proposed method detects one-to-many and many-to-many causal links between input and output messages, including indirect causal links through simple user-level annotations. We validate our method on both synthetic and real robotic systems, and demonstrate its low runtime overhead. Moreover, the underlying intermediate execution representation database can be further leveraged to extract additional metrics and high-level results. This can provide valuable timing and scheduling information to further study and improve the ROS 2 executor as well as optimize any ROS 2 system. The source code is available at: https://github.com/christophebedard/ros2-message-flow-analysis.