Influence of surprise on reinforcement learning in younger and older adults

Abstract

Surprise is a key component of many learning experiences, and yet its precise computational role, and how it changes with age, remain debated. One major challenge is that surprise often occurs jointly with other variables, such as uncertainty and outcome probability. To assess how humans learn from surprising events, and whether aging affects this process, we studied choices while participants learned from bandits with either Gaussian or bimodal outcome distributions, which decoupled outcome probability, uncertainty, and surprise. A total of 102 participants (51 older, aged 50—73; 51 younger, 19–30 years) chose between three bandits, one of which had a bimodal outcome distribution. Behavioral analyses showed that both age-groups learned the average of the bimodal bandit less well. A trial-by-trial analysis indicated that participants performed choice reversals immediately following large absolute prediction errors, consistent with heightened sensitivity to surprise. This effect was stronger in older adults. Computational models indicated that learning rates in younger as well as older adults were influenced by surprise, rather than uncertainty, but also suggested large interindividual variability in the process underlying learning in our task. Our work bridges between behavioral economics research that has focused on how outcomes with low probability affect choice in older adults, and reinforcement learning work that has investigated age differences in the effects of uncertainty and suggests that older adults overly adapt to surprising events, even when accounting for probability and uncertainty effects. 51 younger adults to play a reinforcement learning task that confronted them with surprising outcomes and analyzed their choices. We found that both age groups showed heightened sensitivity to surprising outcomes that resulted in distinctive behavioral adjustments. Notably, older adults weighted these surprising events more than younger adults. Comparing the choices of participants with computational models that incorporated surprises in different ways, we found a model that modulated its learning with the amount of surprise to mimic participants’ choices best. The results help to better understand the role of past surprising events during learning in older and younger adults.