Hierarchical architecture with modular network SOM and modular reinforcement learning

Abstract

We propose a hierarchical architecture composed of a modular network SOM (mnSOM) layer and a modular reinforcement learning (mRL) layer. The mnSOM layer models characteristics of a target system, and the mRL layer provides control signals to the target system. Given a set of inputs and outputs from the target system, a winner module which minimizes the mean square output error is determined in the mnSOM layer. The corresponding module in the mRL layer is trained by reinforcement learning to maximize accumulated future rewards. An essential point, here, is that neighborhood learning is adopted at both layers, which guarantees a topology preserving map based on similarity between modules. Its application to a pursuit-evasion game demonstrates usefulness of interpolated modules in providing appropriate control signals. A modular approach to both modeling and control proposed in the paper provides a promising framework for wide-ranging tasks. © 2009 Springer Berlin Heidelberg.