Reinforcement Learning and Action Space Shaping for a Humanoid Agent in a Highly Dynamic Environment

Reinforcement Learning (RL) is a powerful tool and has been increasingly used in continuous control tasks such as locomotion and balancing in robotics. In this paper, we tackle a balancing task in a highly dynamic environment, using a humanoid robot agent and a balancing board. This task requires complex continuous actuation in order for the agent to stay in a balanced state. In this work, we propose an RL algorithm structure based on the state-of-the-art Proximal Policy Optimization (PPO) using GPU-based implementation; the agent achieves successful balancing in under 40 min of real-time. We sought to examine the impact of action space shaping on sample efficiency and designed 6 distinct control modes. Our constrained parallel control modes outperform the naive baseline in both sample efficiency and variance to the starting seed. The best-performing control mode, using parallel configuration, including lower body and shoulder roll joints named (PLS-R), is 33% more sample efficient than all the other defined modes, indicating the impact of action space shaping on the sample efficiency of our approach.Our implementation is open-source and freely available at: https://github.com/NTNU-ERC/Robinion-Balance-Board-PPO.