Levels of Metacognition and Their Applicability to Reinforcement Learning

Abstract

Recognizing patterns is an integral aspect of human intelligence; we know that every winter brings cold weather and snowfall. We therefore go to the stores beforehand to purchase coats and tools that will ensure our comfort and survival. We are not shocked when the season changes, instead we learn to manage in each new season. We instilled this ability to detect and cope with seasonality into an autonomous agent- Chippy. Chippy uses a reinforcement learner to gather rewards as it explores its environment. Seasonal changes are constructed into Chippy’s environment by changing rewards at regular intervals. We allowed Chippy to operate at different levels of metacognition and compared the amount of rewards gathered when Chippy operates at each level. Results show that Chippy’s reinforcement learner performs best when Chippy metacognitively monitors not only patterns in expectation violations but also patterns in the suggestions made.