一策应对一切:多形态运动的端到端学习方法
One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion
摘要 Abstract
深度强化学习技术在稳健的腿式运动方面取得了最先进的成果。尽管存在四足机器人、人形机器人和六足机器人等多种腿式平台,但该领域仍缺乏一个单一的学习框架,能够轻松且有效地控制这些不同的形态,并可能实现对未见过的机器人形态的零样本或少量样本迁移。我们引入了统一机器人形态架构(URMA),以填补这一空白。我们的框架将端到端的多任务强化学习方法引入腿式机器人领域,使学到的策略能够控制任何类型的机器人形态。我们方法的关键思想是允许网络学习一个抽象的运动控制器,由于我们的形态无关编码器和解码器,该控制器可以在形态之间无缝共享。这种灵活的架构可以被视为构建腿式机器人运动基础模型的第一步。我们的实验表明,URMA能够在多种形态上学习运动策略,并能轻松迁移到模拟和现实世界中的未见过的机器人平台。
Deep Reinforcement Learning techniques are achieving state-of-the-art results in robust legged locomotion. While there exists a wide variety of legged platforms such as quadruped, humanoids, and hexapods, the field is still missing a single learning framework that can control all these different embodiments easily and effectively and possibly transfer, zero or few-shot, to unseen robot embodiments. We introduce URMA, the Unified Robot Morphology Architecture, to close this gap. Our framework brings the end-to-end Multi-Task Reinforcement Learning approach to the realm of legged robots, enabling the learned policy to control any type of robot morphology. The key idea of our method is to allow the network to learn an abstract locomotion controller that can be seamlessly shared between embodiments thanks to our morphology-agnostic encoders and decoders. This flexible architecture can be seen as a potential first step in building a foundation model for legged robot locomotion. Our experiments show that URMA can learn a locomotion policy on multiple embodiments that can be easily transferred to unseen robot platforms in simulation and the real world.