A robot model of the basal ganglia: Behavior and intrinsic processing* Tony J. Prescotta,*, Fernando M. Montes Gonzalezb, �� Kevin Gurneya, Mark D. Humphriesa, Peter Redgravea a Adaptive Behavior Research Group, Department of Psychology, University of Sheffield, Sheffield, Western Bank, South Yorkshire, Sheffield S10 2TN, UK b Department de Fisica e Inteligencia Artificial, Universidad Veracruzana, Xalapa, Veracruz, Mexico Received 13 August 2004 accepted 9 June 2005 Abstract The existence of multiple parallel loops connecting sensorimotor systems to the basal ganglia has given rise to proposals that these nuclei serve as a selection mechanism resolving competitions between the alternative actions available in a given context. A strong test of this hypothesis is to require a computational model of the basal ganglia to generate integrated selection sequences in an autonomous agent, we therefore describe a robot architecture into which such a model is embedded, and require it to control action selection in a robotic task inspired by animal observations. Our results demonstrate effective action selection by the embedded model under a wide range of sensory and motivational conditions. When confronted with multiple, high salience alternatives, the robot also exhibits forms of behavioral disintegration that show similarities to animal behavior in conflict situations. The model is shown to cast light on recent neurobiological findings concerning behavioral switching and sequencing. q 2005 Elsevier Ltd. All rights reserved. Keywords: Basal ganglia Action selection Behavior switching Embodied computational neuroscience Robot Rat 1. Introduction The basal ganglia are a group of highly interconnected central brain structures with a critical influence over movement and cognition. Interest in these structures derives in part from their importance for a cluster of brain disorders that includes Parkinson���s disease, Huntington���s disease, Tourette���s syndrome, schizophrenia, and attention deficit hyperactivity disorder, and has driven more than a century of neurobiological study. This extensive research effort has given rise to a wealth of relevant data, and consequently a pressing need for a better functional understanding of these structures. The basal ganglia, therefore, present one of the most exciting prospects for computational modeling of brain function and have been the focus of extensive modeling research efforts (for reviews see Gillies & Arbuthnott, 2000 Gurney, Prescott, Wickens, & Redgrave, 2004 Houk, Davis, & Beiser, 1995 Prescott, Gurney, & Redgrave, 2002 Wickens, 1997). A recurring theme in the basal ganglia literature is that these structures operate to release inhibition from desired actions while maintaining or increasing inhibition on undesired actions (Cools, 1980 Denny-Brown & Yanagi- sawa, 1976 Hikosaka, 1994 Mink, 1996 Robbins & Brown, 1990 Wickens, 1997). In our own theoretical work (Prescott, Redgrave, & Gurney, 1999 Redgrave, Prescott, & Gurney, 1999a) we have developed the idea that the basal ganglia acts as an action selection mechanism���resolving conflicts between functional units that are physically separated within the brain but are in competition for behavioral expression. We have shown how this proposal relates to known anatomy and physiology and meets several high-level computational requirements for an effective action selection device. In line with this hypothesis we also embarked on a program of modeling the circuitry of the basal ganglia and related structures at several levels of abstraction. A key focus has been to investigate ���system��� level models of the basal ganglia constrained by the known functional anatomy in which neural populations are represented by simple leaky integrator units (Gurney, Prescott, & Redgrave, 2001a,b Gurney, Humphries, Neural Networks 19 (2006) 31���61 www.elsevier.com/locate/neunet 0893-6080/$ - see front matter q 2005 Elsevier Ltd. All rights reserved. doi:10.1016/j.neunet.2005.06.049 *UK Engineering and Physical Sciences Research Council (EPSRC) GR/R95722/01. * Corresponding author. Tel.: C44 114 2226547, fax: C44 114 2766515. E-mail address: t.j.prescott@sheffield.ac.uk (T.J. Prescott).

Wood, Prescott, & Redgrave, 2004 Humphries & Gurney, 2002). At lower levels of neurobiological detail we have studied the patterning of signals encoded by trains of action potentials (���spikes���) (Humphries, 2002 Humphries & Gurney, 2001), and have investigated biophysical models of the membrane dynamics of striatal neurons (Wood, Gurney, & Wilson, 2004). Studies at all of these levels have generated complementary results confirming that the biological architecture of the basal ganglia can operate as an effective selection mechanism. In our view, however, this computational neuroscience approach, in which specific brain systems are modeled in isolation of the wider context, still leaves many important questions unanswered. First, we are left wondering how best to interpret the inputs and outputs of the model���we might choose to think of inputs as, say, ���sensory��� signals, or of outputs as ���motor��� signals, but such assignments are essentially ungrounded. Second, without locating a model within any wider context, we are unable to judge whether it can fulfill its hypothesized functional role within a more fully specified control architecture. Third, without any linkage to sensory and motor systems, we may question whether a model could cope with noisy or ambiguous sense data, or as part of a system challenged with coordinating the movements of real effector systems. Finally, without the context of multiple demands, such as the need to maintain physical integrity, avoid threats, and discover and exploit resources, we will be unsure whether or not a model can meet some of the basic requirements for adaptive behavior. In this article we, therefore, describe an embedding of the system-level model of the basal ganglia and associated thalamocortical connections within the control architecture of a small mobile robot engaged in a simulated foraging task that requires the robot to select appropriate actions under changing sensory and motivational conditions and thereby generate sequences of integrated behavior. We describe the methodology we are applying as embodied computational neuroscience. Preliminary results for the robot model have been described in (Montes Gonzalez, Prescott, Gurney, Humphries, & Redgrave, 2000), and a version of the model has been shown to have better selection properties than a standard winner-takes-all selection mechanism in a robotic survival task (Girard, Cuzin, Guillot, Gurney, & Prescott, 2003). The current article, however, provides the first full account of the robot embedding of the basal ganglia model together with an extensive evaluation of the model���s behavior in comparison to relevant neurobehavioral studies. We also present results showing the behavior of the robot model when faced with multiple high-salience alternatives, and draw comparisons with studies of animal behavior in conflict situations. The remainder of the article is organized as follows. The action selection hypothesis of the basal ganglia and related modeling work is reviewed in Section 2. The motivation for the robot basal ganglia model, full details of the robot implementation, and a summary of action selection metrics, are described in Section 3 (and the accompanying appendices). Section 4 then describes the results of three experiments: experiment 1, a systematic search of a salience space using a disembodied version of the extended basal ganglia model (extending earlier analyses of this model by Humphries and Gurney, 2002) experiment 2, our main investigation of the action selection by the robot basal ganglia model and experiment 3, an investigation of robot behavior in the context of high salience alternatives. Section 5 provides our discussion of the experimental results focusing on comparisons with biological data. 2. Background: the basal ganglia viewed as an action selection device There have been many excellent summaries of the functional anatomy of the basal ganglia (e.g. Gerfen & Wilson, 1996 Mink, 1996 Smith, Bevan, Shink, & Bolam, 1998), the following, therefore, focuses on those aspects most relevant to understanding the models discussed below. The principle structures of the rodent basal ganglia (Fig. 1a) are the striatum (consisting of the caudate, the putamen, and the ventral striatum), the subthalamic nucleus (STN), the globus pallidus (GP), the substantia nigra (SN, consisting of the pars reticulata SNr and pars compacta SNc), and the entopeduncular nucleus (EP) (homologous to the globus pallidus internal segment, or GPi, in primates). These structures are massively interconnected and form a functional sub-system within the wider brain architecture (Fig. 1b). The input nuclei of the basal ganglia are the striatum and the STN. Afferent connections to both of these structures originate from virtually the entire brain including cerebral cortex, many parts of the brainstem (via the thalamus), and the limbic system. These connections provide phasic excitatory input. The main output nuclei are the substantia nigra pars reticulata (SNr), and the entopeduncular nucleus (EP). These structures provide extensively branched efferents to the thalamus (which in turn project back to the cerebral cortex), and to pre-motor areas of the midbrain and brainstem. Most output projections are tonically active and inhibitory. To understand the intrinsic connectivity of the basal ganglia it is important to recognize that the main projection neurons from the striatum (medium spiny cells) form two widely distributed populations differentiated by their efferent connectivity and neurochemistry. One population contains the neuropeptides substance P and dynorphin, preferentially expresses the D1 subtype of dopamine receptors, and projects primarily to the output nuclei (SNr and EP). In the prevailing informal model of the basal ganglia (Albin, Young, & Penney, 1989) this ���D1 striatal��� projection constitutes the so-called direct pathway to the output nuclei. Efferent activity from these neurons suppresses the tonic inhibitory firing in the output structures T.J. Prescott et al. / Neural Networks 19 (2006) 31���61 32

which in turn disinhibits targets in the thalamus and brainstem. A second population of striatal projection neurons contains enkephalin and preferentially expresses D2 subtype dopamine receptors. This group projects primarily to the globus pallidus (GP) whose tonic inhibitory outputs are directed both to the output nuclei (SNr and EP) and to the STN. The inhibitory projection from these ���D2 striatal��� neurons constitutes the first leg of an indirect pathway to the output nuclei. Since this pathway has two inhibitory links (Striatum-GP, GP-STN), followed by an excitatory one (STN-EP/SNr), its net effect is to activate output nuclei thereby increasing inhibitory control of the thalamus and brainstem. The main source of dopamine innervation to the striatum is the substantia nigra pars compacta (SNc). Dopaminergic modulation of basal ganglia is generally considered to act at two time-scales (Grace, 1991 Walters, Ruskin, Allers, & Bergstrom, 2000). One is a short-latency phasic response (100 ms burst) that correlates with the onset of biologically significant stimuli and appears to be critical for some forms of incentive learning (Redgrave, Prescott, & Gurney, 1999b Schultz, Dayan, & Montague, 1997), the other is a tonic level of activity (1���8 Hz) that is altered by various brain pathologies, such as Parkinson���s disease, and in the normal brain may be subject to modulation by structures such as the frontal cortex. Interestingly, the D1 and D2 striatal populations respond differently to variations in dopamin- ergic transmission. Whilst a range of effects have been reported, one simplifying hypothesis, that accounts for a significant proportion of available findings, is that dopamine enhances the effectiveness of other synaptic inputs when acting via D1 receptors (Akkal, Burbaud, Audin, & Bioulac, 1996) whilst reducing such efficacy when acting at D2 receptors (Gerfen, Engber, Mahan, Susel, Chase and Monsma, 1990 Harsing & Zigmond, 1997). This arrange- ment seems to provide dopaminergic control of a ���push/pull��� mechanism subserved by the direct (inhibitory) and indirect (net excitatory) basal ganglia pathways. The effects of variations in this tonic dopamine level on our robot model are the subject of a separate article in the current work we report results in which the simulated dopamine level is fixed at an intermediate level. Likewise, the current article does not address the problem of learning (and the role of dopamine therein), but the logically distinct question of whether the basal ganglia are suitably configured to support action selection in an embodied agent. A key assumption of our basal ganglia model is that the brain is processing, in parallel, a large number of sensory and cognitive streams or channels, each one potentially carrying a request for action to be taken. For effective behavior, the majority of these requests must be suppressed to allow the expression of only a limited number (perhaps just one). This channel-based scheme is consistent with evidence that basal ganglia input occurs via a series of topographically organized, parallel processing streams (Alexander & Crutcher, 1990). The action selection hypothesis of the basal ganglia further suggests that the activity of cell populations in the striatum and STN encodes the salience, or propensity for selection, of candidate actions. At the same time, the basal ganglia output structures, SNr and EP, are viewed as gating candidate actions via a reduction in their inhibitory output for winning channels. When considered in isolation of the wider brain architecture, this action selection thesis is best restated in terms of the context-neutral problem of ���signal selection��� in other words, the proposal is that large signal inputs at striatum and STN select for low signal outputs at EP/SNr. From a signal selection perspective multiple mechanisms within the basal ganglia and related circuitry appear to be suitably configured to resolve conflicts between competing channels and provide the required clean and rapid switching between winners. Our initial system-level model of Fig. 1. Basal ganglia anatomy of the rat: (a) internal pathways, (b) external pathways. Not all connections are shown. Abbreviations: STN, subthalamic nucleus EP, entopeduncular nucleus GP, globus pallidus SNc, substantia nigra pars compacta SNr, substantia nigra pars reticulata D1 D2, striatal neurons preferentially expressing dopamine receptors subtypes D1 and D2. T.J. Prescott et al. / Neural Networks 19 (2006) 31���61 33

the basal ganglia (Gurney et al., 2001a,b) focused on the following candidate selection mechanisms. First, at the cellular level considerable interest has focused on an intrinsic property of striatal projection neurons such that, at any given moment, a majority of cells are in an inactive ���down-state���, and can only be triggered into an active ���up-state��� (where they can fire action potentials) by a significant amount of coincident input (Wilson & Kawaguchi, 1996). This bistable behavior could act as a high-pass filter to exclude weakly supported ���requests���. Second, computational theory suggests that a feed- forward, off-centre, on-surround network is an appropriate mechanism for enhancing signal selection. In the basal ganglia, this type of selection circuit appears to be implemented by a combination of focused striatal inhibition of the output nuclei (the off-centre) and diffuse STN excitation of the same (the on-surround) (Parent & Hazrati, 1995). On closer examination, however, it appears that there are actually two such feed-forward networks in the basal ganglia intrinsic circuitry (see Fig. 2a and b), differentiated by the projection targets of the D1-type and D2-type sub- populations of striatal neurons. One instantiation (Fig. 2a) makes use of EP/SNr as its ���output layer��� since this is clearly consistent with our signal selection hypothesis for the basal ganglia we have designated this circuit the selection pathway. However, there is also a second implementation of the feed-forward architecture whose target is the GP (Fig. 2b). Since the efferent connections of the GP are confined to other basal ganglia nuclei it is not immediately clear in what sense this second implementation can contribute to the overall selection task. This question can be resolved by supposing that this second sub-system forms a control pathway that functions to regulate the properties of the main selection mechanism. The control signals emanating from GP are evident when the two sub- systems are combined to give the overall functional architecture shown in Fig. 2c. In our original system-level model, we operationalized the above circuit (Fig. 2c) as a multi-channel system where, for every basal ganglia nucleus, the neural population encoding each channel is simulated by a suitably configured leaky integrator unit. Analytical and simulation studies (Gurney et al., 2001a,b) conducted with this model demonstrated that it has the capacity to support effective switching between multiple competitors. In simulation, two or more channels of the model were provided with afferent input in the form of hand-crafted signals of different amplitude. Results showed that the largest signal input always generates the smallest signal output (thus showing signal selection), and that the system rapidly switches from a currently selected channel to a competing channel that suddenly has a larger input. We were also able to generate signal characteristics in the component circuits of our basal ganglia model that follow similar temporal patterns to single-unit recordings of neural firing in GP (Ryan & Clark, 1991) and SNr (Schultz, 1986). Humphries and Gurney extended the original model of intrinsic basal ganglia processing to include basal ganglia- thalamocortical loops (Humphries & Gurney, 2002). This work led to the proposal that the thalamic complex���the ventro-lateral (VL) thalamus and thalamic-reticular nucleus (TRN)���acts to provide additional selection-related functionality. Specifically, as shown in Fig. 3, these circuits can be understood as sub-serving two important roles. First, disinhibition of VL thalamic targets by EP/SNr enables a positive feedback loop whereby winning basal ganglia channels can increase the activation of their own cortical inputs. Second, the within- and between- channel connec- tions between the TRN and the VL thalamus appear to implement a distal lateral-inhibition network that serves to increase the activity of the most strongly innervated channel at the expense of its neighbors. In simulation, again with hand-crafted signals, the additional selective functions of these extra-basal ganglia mechanisms were found to promote several desirable selection features including cleaner switching between channels of closely matched salience, and the ability to ignore transient salience interrupts. Recently, we have also shown that the model can accommodate new data on striato pallidal projections, Fig. 2. The basal ganglia viewed as an action selection mechanism. Abbreviations as per Fig. 1. Our analysis of the basal ganglia intrinsic connectivity (Gurney et al., 2001a,b) indicated the presence of two off-centre, on-surround, feed-forward networks. One instantiation: (a) makes use of EP/SNr as its ���output layer��� and is designated the selection pathway, the second (b) targets GP and is designated the control pathway. The control signals emanating from GP are evident when the two sub-systems are combined to give the overall functional architecture shown in Figure c. T.J. Prescott et al. / Neural Networks 19 (2006) 31���61 34

and on local inhibitory connections within the globus pallidus and substantia nigra (Gurney, Humphries et al., 2004 Humphries, Prescott, & Gurney, 2003). Both extensions also appear to enhance the selectivity of the system and, in adding further biological realism, lend further support to the selection hypothesis of basal ganglia function. An effective action selection mechanism should be sensitive to changes in salience weightings that alter the relative urgency of competing behaviors in a given context. It is less evident, however, how a selection mechanism should respond to changes in salience weightings that leave relative salience unchanged whilst scaling the overall level of the selection competition. The assumption encapsulated by the widely used winner-takes-all selection mechanism, for example, is that the overall level of salience is irrelevant (the competitor with highest salience is always preferred). We have previously demonstrated that the selection proper- ties of both the intrinsic (Gurney et al., 2001b) and extended (Humphries and Gurney, 2002) basal ganglia models do not conform to this assumption, but instead, vary according to the overall ���intensity��� of the selection competition. We will extend this work below by showing that that the degree of hysteresis, or persistence, of the winning sub-system may change as a consequence of changes in the overall level of salience. Our previous studies noted interesting patterns of ���multiple channel��� selection when the model is presented with multiple, high salience alternatives. We, therefore, investigate the behavior of the robot model in these circumstances, and consider possible parallels with obser- vations derived from ethological studies of behavioral conflict. 3. Developing a robot model of action selection by the basal ganglia The modeling work considered above serves to demon- strate signal selection by the basal ganglia rather than action selection per se. To show convincingly that the basal ganglia model is able to operate as an effective action selection device we believe it needs to be embedded in a real-time sensorimotor interaction with the physical world. An important goal has, therefore, been to construct an embedded basal ganglia model in which selection occurs between multiple, physically realized behaviors in a mobile robot. Since the use of robotics in computational neuro- science is relatively new, we preface our description of this model with a brief explanation of how we approach this task of embedding a computational neuroscience model within a robot architecture that generates observable behavior. 3.1. A methodology for embodied computational neuroscience Any computational neuroscience model, robotic or otherwise, is composed of components that are ���biomi- metic������that is, they are intended to directly simulate neurobiological processes (at some appropriate level), and those that are merely ���engineered��� so as to provide an interface that will allow the model to be interrogated and evaluated. The need for engineered components is particu- larly obvious in the case of robotic models where simulations of neural circuits must, at some point, be interfaced with (usually) very-non-neural robot hardware. Furthermore, in models that seek to simulate complete behavioural competences it is also generally impractical, because of the scale of the task, or impossible, because of the lack of the necessary neurobiological data, to simulate all components of the neural substrate for the target competence at a given level of detail. In this situation, engineered components are also required to substitute for the function of some of the neural circuits, known or non- known, that are involved in the production of that competence in an animal. In the current model, since the biological substrate of interest is the basal ganglia, the system components that provide the interface between the robot hardware (and low-level controllers) and the models of the basal ganglia and related nuclei have been constructed as a set of engineered sub-systems that we collectively denote as the embedding architecture. While broadly ���biologically inspired���, we would stress that this embedding Fig. 3. The extended basal ganglia model of Humphries and Gurney (2002). Abbreviations: SSC, somatosensory cortex MC, motor cortex VL, ventro- lateral thalamus TRN, thalamic-reticular nucleus, others as per Fig. 1. Connectivity within the basal ganglia component of the model is as shown in Fig. 2c. Basal ganglia-thalamocortical loops can be understood as providing additional mechanisms that can contribute to effective action selection. First, the removal of basal ganglia inhibition from VL completes a positive feedback loop to the motor cortex. Second, the diffuse inhibitory connections from TRN to VL, which are stronger between channels than within channels (as indicated by the plain and dotted inhibitory connections in the figure), together with within-channel excitation from VL to TRN, produces a form of mutual inhibition between channels. See text and Humphries and Gurney (2002) for further explanation. T.J. Prescott et al. / Neural Networks 19 (2006) 31���61 35