Centre manifold analysis of plateau phenomena in learning of three-layer perceptron (Research on the Theory of Random Dynamical Systems and Fractal Geometry)

Abstract

A three-layer perceptron is the most basic model of hierarchical neural networks. We treat a gradient system representing the learning process of the three-layer perceptron. In its parameter space, a three-layer perceptron has one-dimensional singular regions comprising both attractive and repulsive parts, which is often called a Milnor-like attractor. In this paper, we introduce an analysis of the learning process in the vicinity of a Milnor-like attractor based on the centre manifold theory.