多层感知器模型互反奇异性区域学习动态的理论分析

Theoretical analysis of learning dynamics near the opposite singularities in multilayer perceptrons

中文关键词  多层感知器  神经网络  学习动态  奇异性  互反

英文关键词  multilayer perceptrons  neural networks  learning dynamics  singularity  opposite

中文摘要

      多层感知器神经网络(MLPs)的学习过程经常发生一些奇异性行为, 容易陷入平坦区, 这都和MLPs的参数 空间中存在的奇异性区域有直接关系. 当MLPs的两个隐节点的权值接近互反时, 置换对称性会导致学习困难. 对 MLPs的互反奇异性区域附近的学习动态进行分析. 本文首先得到了平均学习方程的解析表达式, 然后给出了互反 奇异性区域附近的理论学习轨迹, 并通过数值方法得到了其附近的实际学习轨迹. 通过仿真实验, 分别观察了MLPs 的平均学习动态, 批处理学习动态和在线学习动态, 并进行了比较分析.

英文摘要

      Owing to the existence of singularities in the parameter space, multilayer perceptrons (MLPs) may behave extremely slowly in learning or even be trapped in plateaus. When weights of two hidden units are nearly mutually opposite, the learning process will encounter difficulties because of the permutation symmetry. We investigate the learning dynamics of MLPs near opposite singularities, and derive the analytical expressions for averaged learning equations. Then, we obtain the theoretical learning trajectories near the opposite singularities. Furthermore, real learning trajectories near the opposite singularities are also calculated by using numerical methods. In simulations, we study the averaged learning dynamics, the batch mode learning dynamics and the online learning dynamics, respectively.