融合强化学习和进化算法的高超声速飞行器航迹规划

Trajectory planning for hypersonic vehicle combined with reinforcement learning and evolutionary algorithms

中文关键词  强化学习  深度强化学习  高超声速飞行器  航迹规划

英文关键词  reinforcement learning  deep reinforcement learning  hypersonic vehicles  trajectory planning

中文摘要

      由于高超声速飞行器的复杂特性, 对其进行航迹规划是一项非常困难的任务. 本文针对高超声速飞行器巡 航段, 提出了一种将无模型的强化学习和交叉熵方法相结合的在线航迹规划算法. 本文将航迹规划问题建模为环境 信息缺失程度不同的马尔可夫决策过程, 利用(PPO)算法在建立的飞行环境模拟器中离线训练智能体, 并通过提高 智能体的动作在时间上的相关性来保证航迹的曲率平滑. 交叉熵方法则以已训练的智能体由观测到的状态给出的 动作作为一种先验知识, 进一步在线优化规划策略. 实验结果表明了本文的方法可以生成曲率平滑的航迹, 在复杂 的飞行环境中具有较高的成功率, 并且可以泛化到不同的飞行环境中.

英文摘要

      It is difficult to plan the flight trajectory for hypersonic vehicle because of its sophisticated characteristics. In this paper, an online trajectory planning algorithm combining model-free reinforcement learning and cross-entropy method for hypersonic vehicle in the cruise phase is proposed. The trajectory planning problem is modeled as Markov decision processes with different degrees of missing environmental information. The agent is trained off-line in the flight environment simulator by using proximal policy optimization (PPO) algorithm, and the curvature smoothness of the trajectory is ensured by improving the temporal correlation of the agent’s action. The cross-entropy method uses the actions of the trained agent from the observed state as a kind of prior knowledge to further optimize the planning policy online. Simulation results provide the evidence that the proposed method can generate curvature smooth trajectories with high success rate in complex flight environment, and can be generalized to different flight environments.