Accessibility-Based Clustering for Efficient Learning of Robot Fall Recovery

For the model-free deep reinforcement learning of quadruped fall recovery, the initialization of robot configurations is crucial to the data efficiency and robustness. This work focuses on algorithmic improvements of data efficiency and robustness simultaneously through automatic discovery of initial states, which is achieved by our proposed K-Access algorithm based on accessibility metrics. Specifically, we formulated accessibility metrics to measure the difficulty of transitions between two arbitrary states, and proposed a novel K-Access algorithm for state-space clustering that automatically discovers the centroids of the static-pose clusters based on the accessibility metrics. By using the discovered centroidal static poses as initial states, we improve the data efficiency by reducing the redundant exploration, and enhance the robustness by easy explorations from the centroids to sampled static poses. We studied extensive validation using an 8-DOF quadrupedal robot Bittle. Compared to random initialization, the learning curve of our proposed method converges much faster, requiring only around 60% of training episodes. With our method, the robot can successfully recover standing poses in 99.4% of tests within 3 seconds.