面向稀疏奖励的机器人操作技能学习

Robot manipulation skills learning for sparse rewards

中文关键词  机器人操作技能学习  强化学习  稀疏奖励  最大熵方法  自适应温度参数  元学习

英文关键词  robot manipulation skills learning  reinforcement learning  sparse reward  maximum entropy methods  adaptive temperature parameters  meta-learning

中文摘要

      基于深度强化学习的机器人操作技能学习成为研究热点, 但由于任务的稀疏奖励性质, 学习效率较低. 本 文提出了基于元学习的双经验池自适应软更新事后经验回放方法, 并将其应用于稀疏奖励的机器人操作技能学习 问题求解. 首先, 在软更新事后经验回放算法的基础上推导出可以提高算法效率的精简值函数, 并加入温度自适应 调整策略, 动态调整温度参数以适应不同的任务环境; 其次, 结合元学习思想对经验回放进行分割, 训练时动态调整 选取真实采样数据和构建虚拟数的比例, 提出了DAS-HER方法; 然后, 将DAS-HER算法应用到机器人操作技能学 习中, 构建了一个稀疏奖励环境下具有通用性的机器人操作技能学习框架; 最后, 在Mujoco下的Fetch和Hand环境 中, 进行了8项任务的对比实验, 实验结果表明, 无论是在训练效率还是在成功率方面, 本文算法表现均优于其他算 法.

英文摘要

      Robot manipulation skills learning based on deep reinforcement learning has become a research hotspot. However, due to the sparse reward nature of robot manipulation skills learning, the learning efficiency is low. In this paper, a double experience replay buffer adaptive soft hindsight experience replay (DAS-HER) algorithm based on metalearning is proposed, and applied to solve the manipulation skills learning problem with sparse reward. Firstly, based on the soft hindsight experience replay (SHER) algorithm, a simplified value function which can improve the efficiency of the algorithm is derived, and a temperature adaptive adjustment strategy is introduced which can dynamically adjust the temperature parameters to adapt to different task environments. Secondly, combined with meta-learning, the experience replay is segmented, dynamically adjust the ratio of real sampling data and construct virtual data during training, and the DAS-HER algorithm is proposed. Thirdly, a generalized framework for robot manipulation skills learning under a sparse reward environment is constructed, and DAS-HER algorithm is applied to robot manipulation skills learning. Finally, comparative experiments for eight tasks are conducted both in Fetch and Hand environments under Mujoco environment, and the results show that the proposed algorithms outperform other algorithms in terms of training efficiency and success rate.