Strategy Acquisition for Games Based on Simplified Reinforcement Learning Using a Strategy Network

Abstract

We propose a simplified form of reinforcement learning (RL) for game strategy acquisition using a strategy network. RL has been applied to a number of games, such as backgammon, checkers, etc. However, the application of RL to Othello or Shogi, which have very large state spaces, is more difficult because these games take a very long time to learning. The proposed strategy network is composed of N lines from N nodes on the game board with a single evaluation node as a 2-layer perceptron. These nodes denote all possible states of every square on the game board and can easily represent the evaluation function. Moreover, these nodes can also denote imaginary states, such as pieces that may exist in the next step, or denote every positional relation of two arbitrary pieces or other various board phases. After several thousands of games had been played, the strategy network quickly acquired a better evaluation function than that using a normalized Gaussian network. The computer player employing the strategy network beat a heuristic-based player that evaluates the values of pieces or places on the game board. The proposed strategy network was able to acquire good weightings of various features of game states. In addition, the player employing the strategy network for a 4×4 Othello task after co-evolutionary training acquired a winning strategy.