如何评估分布式协调系统？——一项调研与分析 (How to Evaluate Distributed Coordination Systems? -- A Survey and Analysis)

from arxiv, The paper has been revised and extended to include (1) additional distributed coordination algorithms and (2) additional benchmarking tools in the survey. Also, the selection criteria of the algorithms, services, and tools was specified

Coordination services and protocols are critical components of distributed systems and are essential for providing consistency, fault tolerance, and scalability. However, due to the lack of standard benchmarking and evaluation tools for distributed coordination services, coordination service developers/researchers either use a NoSQL standard benchmark and omit evaluating consistency, distribution, and fault tolerance; or create their own ad-hoc microbenchmarks and skip comparability with other services. In this study, we analyze and compare the evaluation mechanisms for known and widely used consensus algorithms, distributed coordination services, and distributed applications built on top of these services. We identify the most important requirements of distributed coordination service benchmarking, such as the metrics and parameters for the evaluation of the performance, scalability, availability, and consistency of these systems. Finally, we discuss why the existing benchmarks fail to address the complex requirements of distributed coordination system evaluation.

翻译：协调服务与协议是分布式系统的关键组成部分，对于提供一致性、容错性与可扩展性至关重要。然而，由于缺乏针对分布式协调服务的标准化基准测试与评估工具，协调服务的开发者/研究者要么使用NoSQL标准基准测试而忽略对一致性、分布性与容错性的评估，要么自行创建临时性微基准测试并放弃与其他服务的可比性。本研究分析并比较了已知且广泛使用的共识算法、分布式协调服务以及基于这些服务构建的分布式应用的评估机制。我们识别了分布式协调服务基准测试中最重要的需求，例如评估这些系统性能、可扩展性、可用性与一致性的指标与参数。最后，我们讨论了现有基准测试为何未能满足分布式协调系统评估的复杂需求。