letters to nature NATURE | VOL 408 | 16 NOVEMBER 2000 | www.nature.com 361 Data analysis A neuron was considered selective to a stimulus group if: (1) the firing rate during stimulus presentation was different from the preceding baseline (Wilcoxon test, , 0.05), (2) an analysis of variance and pairwise comparisons (Wilcoxon test) addressing whether there were differences among the stimulus groups yielded P , 0.05 and (3) an ANOVA (parametric and non-parametric) comparing the variability to distinct stimuli within the selective category to the variability to repeated presentations of the same stimulus showed P.0.05. We observed neurons selective to faces, objects, spatial layouts and other stimuli. If the across-groups comparisons were not significant but the activity was different from baseline, the neuron was defined as responsive but non-selective. To take into account any effects due to the different intervals we also compared the responses in a 600-ms window centred on the peak firing rate. The peak, latency and duration were estimated from the spike density function15. For the selective neurons we computed the probability of error, Pe, for classifying the stimulus as belonging to the preferred stimulus category or not15,16. We did not observe any difference between the right and left hemisphere neurons. Received 21 July accepted 22 August 2000. 1. Kosslyn, S. M. Image and Brain (MIT Press, Cambridge, 1994). 2. Farah, M. J. Is visual imagery really visual? Overlooked evidence from neuropsychology. Psychol. Rev. 95, 307���317 (1988). 3. Behrmann, M., Winocur, G. & Moscovitch, M. Dissociation between mental imagery and object recognition in a brain-damaged patient. Nature 359, 636���637 (1992). 4. Kosslyn, S. M., Thompson, W. L. & Alpert, N. M. Neural systems shared by visual imagery and visual perception: a PET study. Neuroimage 6, 320���334 (1997). 5. Roland, P. E. & Gulyas, B. Visual imagery and visual representation. Trends Neurosci. 17, 281���287 (1994). 6. D���Esposito, M. et al. A fMRI study of mental image generation. Neuropsychologia 35, 725���730 (1997). 7. O���Craven, K. & Kanwisher, N. Mental imagery of faces and places activates corresponding stimulus- specific brain regions. J. Cog. Neurosci. (in the press). 8. Frith, C. & Dolan, R. J. Brain mechanisms associated with top-down processes in perception. Phil. Trans. R. Soc. Lond. 352, 1221���1230 (1997). 9. Ishai, A. & Sagi, D. Common mechanisms of visual imagery and perception. Science 268, 1772���1774 (1995). 10. Rainer, G., Rao, S. & Miller, E. Prospective coding for objects in primate prefrontal cortex. J. Neurosci. 19, 5493���5505 (1999). 11. Miyashita, Y. & Chang, H. S. Neuronal correlate of pictorial short-term memory in the primate temporal cortex. Nature 331, 68���71 (1988). 12. Tomita, H., Ohbayashi, M., Nakahara, K., Hasegawa, I. & Miyashita, Y. Top-down signal from prefrontal cortex in executive control of memory retrieval. Nature 401, 699���703 (1999). 13. Bartolomeo, P. et al. Multiple-domain dissociation between impaired visual perception and preserved mental imagery in a patient with bilateral extrastriate lesions. Neuropsychologia 36, 239���249 (1998). 14. Fried, I., MacDonald, K. A. & Wilson, C. Single neuron activity in human hippocampus and amygdala during recognition of faces and objects. Neuron 18, 753���765 (1997). 15. Kreiman, G., Koch, C. & Fried, I. Category-specific visual responses of single neurons in the human medial temporal lobe. Nature Neurosci. 3, 946���953 (2000). 16. Green, D. & Swets, J. Signal detection theory and psychophysics (Wiley, New York, 1966). 17. Kanwisher, N. & Moscovitch, M. The cognitive neuroscience of face processing: An introduction. Cogn. Neuropsychol. 17, 1���11 (2000). 18. Epstein, R. & Kanwisher, N. A cortical representation of the local visual environment. Nature 392, 598���601 (1998). 19. Logothetis, N. K. & Sheinberg, D. L. Visual object recognition. Annu. Rev. Neurosci. 19, 577���621 (1996). 20. Tanaka, K. Inferotemporal cortex and object vision. Annu. Rev. Neurosci. 19, 109���139 (1996). 21. Gross, C. G. How inferior temporal cortex became a visual area. Cereb. Cortex 5, 455���469 (1994). 22. Rolls, E. Neural organization of higher visual functions. Curr. Opin. Neurobiol. 1, 274���278 (1991). 23. Miyashita, Y. Inferior temporal cortex: Where visual perception meets memory. Annu. Rev. Neurosci. 16, 245���263 (1993). 24. Chelazzi, L., Duncan, J., Miller, E. K. & Desimone, R. Responses of neurons in inferior temporal cortex during memory-guided visual search. J. Neurophys. 80, 2918���2940 (1998). 25. Suzuki, W. A. Neuroanatomy of the monkey entorhinal, perirhinal and parahippocampal cortices: Organization of cortical inputs and interconnections with amygdala and striatum. Semin. Neurosci. 8, 3���12 (1996). 26. Warrington, E. & McCarthy, R. Categories of knowledge���Further fractionations and an attempted integration. Brain 110, 1273���1296 (1987). 27. Meunier, M., Hadfield, W., Bachevalier, J. & Murray, E. Effects of rhinal cortex lesions combined with hippocampectomy on visual recognition memory in rhesus monkeys. J. Neurophysiol. 75, 1190���1205 (1996). 28. Fried, I., Mateer, C., Ojemann, G., Wohns, R. & Fedio, P. Organization of visuospatial functions in human cortex. Brain 105, 349���371 (1982). 29. Ojemann, G. & Mateer, C. Human language cortex: localization of memory, syntax, and sequential motor-phoneme identification systems. Science 205, 1401���1403 (1979). 30. Penfield, W. & Jasper, H. Epilepsy And The Functional Anatomy Of The Human Brain (Little, Brown & Co., Boston, 1954). Acknowledgements This work was supported by grants from NIH, the Centre for Consciousness Studies at the University of Arizona and the Keck Foundation. We thank M. Zirlinger for discussions, T. Fields, C. Wilson, E. Isham and E. Behnke for help with the recordings, F. Crick for comments and I. Wainwright for editorial assistance. We also thank all the patients who participated in these studies. Correspondence and requests for materials should be addressed to I.F. (e-mail: ifried@mednet.ucla.edu). ................................................................. Real-time prediction of hand trajectory by ensembles of cortical neurons in primates Johan Wessberg*, Christopher R. Stambaugh*, Jerald D. Kralik*, Pamela D. Beck*, Mark Laubach*, John K. Chapin��, Jung Kim���, S. James Biggs���, Mandayam A. Srinivasan��� & Miguel A. L. Nicolelis*��k * Department of Neurobiology �� Department of Biomedical Engineering k Department of Psychology-Experimental, Duke University, Durham, North Carolina 27710, USA �� Department of Physiology and Pharmacology, State University of New York Health Science Center, Brooklyn, New York 11203, USA ��� Laboratory for Human and Machine Haptics, Department of Mechanical Engineering and Research Laboratory of Electronics, MIT, Cambridge, Massachusetts 02139, USA .............................................................................................................................................. Signals derived from the rat motor cortex can be used for controlling one-dimensional movements of a robot arm1. It remains unknown, however, whether real-time processing of cortical signals can be employed to reproduce, in a robotic device, the kind of complex arm movements used by primates to reach objects in space. Here we recorded the simultaneous activity of large populations of neurons, distributed in the pre- motor, primary motor and posterior parietal cortical areas, as non-human primates performed two distinct motor tasks. Accu- rate real-time predictions of one- and three-dimensional arm movement trajectories were obtained by applying both linear and nonlinear algorithms to cortical neuronal ensemble activity recorded from each animal. In addition, cortically derived signals were successfully used for real-time control of robotic devices, both locally and through the Internet. These results suggest that long-term control of complex prosthetic robot arm movements can be achieved by simple real-time transformations of neuronal population signals derived from multiple cortical areas in primates. Several interconnected cortical areas in the frontal and parietal lobes are involved in the selection of motor commands for produc- ing reaching movements in primates2���8. The involvement of these areas in many aspects of motor control has been documented extensively by serial single-neuron recordings of primate behaviour2,3,8,9, and evidence for distributed representations of motor information has been found in most of these studies10���13, but little is known about how these cortical areas collectively influence the generation of arm movements in real time. The advent of multi-site neural ensemble recordings in primates14 has allowed simultaneous monitoring of the activity of large popula- tions of neurons, distributed across multiple cortical areas, as animals are trained in motor tasks15. Here we used this technique to investigate whether real-time transformations of signals gener- ated by populations of single cortical neurons can be used to mimic in a robotic device the complex arm movements used by primates to reach for objects in space. Microwire arrays were implanted in multiple cortical areas of two owl monkeys (Aotus trivirgatus)14���16. In the first monkey, 96 micro- wires were implanted in the left dorsal premotor cortex (PMd, 16 wires), left primary motor cortex (MI, 16 wires)17,18, left posterior parietal cortex (PP, 16 wires), right PMd and MI (32 wires), and right PP cortex (16 wires). In the second monkey, 32 microwires were implanted in the left PMd (16 wires) and in the left MI (16 wires). Recordings of cortical neural ensembles began 1���2 weeks after the implantation surgery and continued for 12 months in monkey 1, and 24 months in monkey 2. During this period, the monkeys were trained in two distinct motor tasks. In task 1, animals �� 2000 Macmillan Magazines Ltd

letters to nature NATURE | VOL 408 | 16 NOVEMBER 2000 | www.nature.com 363 made one-dimensional hand movements to displace a manipulan- dum in one of two directions (left or right) following a visual cue. In task 2, the monkeys made three-dimensional hand movements to reach for small pieces of food randomly placed at four different positions on a tray. Cortical recordings were obtained while the two subjects were trained and tested on both tasks (Fig. 1a). Figure 1b and c illustrate samples of the raw neuronal data obtained while the animals performed task 1. In both monkeys, coherence analysis19���21revealed that the activity of most single neurons from each of the simultaneously recorded cortical areas was significantly correlated with both one-dimensional (Fig. 2a and b) and three-dimensional hand trajectories, although the degree and frequency range of these correlations varied considerably within and between cortical areas. We then investigated whether both linear19���21 and artificial neural network (ANN)22,23 algorithms could be used to predict hand position in real time. For one- dimensional movements, we observed that both algorithms yielded highly significant real-time predictions in both monkeys (Fig. 2c and d). These results were obtained in spite of the fact that the trajectories were quite complex, involving different starting posi- tions, as well as movements at different velocities. For example, in the session represented in Fig. 2c, the activity of 27 PMd, 26 MI, 28 PP, and 19 ipsilateral MI/PMd neurons in monkey 1 allowed us to achieve an average correlation coefficient of 0.61 between the observed and predicted hand position (60-minute session, range 0.50���0.71, linear model 0.45���0.73, ANN P ,0.00119���21). Figure 2d illustrates similar real-time results obtained by using a smaller sample of neurons (8 PMd and 27 MI) in monkey 2 (average r = 0.72, range 0.47���0.79, linear model average r = 0.66, range 0.42���0.71, ANN, P , 0.001). No large differences in fitting accuracy were observed between linear and ANN algorithms in either animal (Fig. 2c and d, linear prediction shown as green line, ANN as red line). As shown in Fig. 2e (monkey 1) and Fig. 2f (monkey 2), the performance of both algorithms improved in the first few minutes of recordings and then reached an asymptotic level that was maintained throughout the experiment. In both monkeys, highly significant predictions of hand movement trajectories were obtained for several months. To reduce the influence of dynamic changes in the coupling between neuronal activity and movements and other non-station- ary influences in our real-time predictions, both linear and ANN models were continuously updated throughout the recording ses- sions. This approach significantly improved the prediction of hand trajectories. For example, when predicting the last 10 minutes of 50���100-minute sessions, the adaptive algorithm performed 55% (20 sessions, median) better than a fixed model based on the initial 10 minutes, or 20% better than a model based on the 30���40-minute segment of the session. Because accurate hand trajectory predictions were achieved early on in each recording session and remained stable for long periods of time, we were able to use brain-derived signals to control the movements of robotic devices (Phantom, SensAble Technologies)24 in real time (Fig. 2g). In addition, we were able to broadcast these motor control signals to multiple computer clients by using a regular Internet communication protocol (TCP/IP, Fig 1a) and control two distinct robots simultaneously: one at Duke University (Fig. 2g, blue line) and one at MIT (Fig. 2g, red line). Next, we investigated whether the same cortical ensemble activity and models could be used to predict the complex sequences of three-dimensional hand movements used by primates in a food- reaching task (task 2). These movements involved four phases: 0 20 40 60 80 100 mm 80 60 40 20 0 0 20 40 60 80 100 Up Down Tray 30 mm Monkey sits here Left Right100 Tray Mouth Start Tray Mouth Start Proximal Distal 0 20 40 60 80 100 mm 100 80 60 40 20 0 0 20 40 60 80 100 0 20 40 60 80 100 mm 100 80 60 40 20 0 0 20 40 60 80 100 a d b X Y Z c e 0 20 40 60 mm 60 40 20 0 0 20 40 60 0 20 40 60 mm 60 40 20 0 0 20 40 60 0 20 40 60 mm 60 40 20 0 0 20 40 60 Start Start Tray Mouth Tray Start Tray Mouth Observed Predicted Targets 2 1 4 3 2 1 4 3 0 5 10 15 20 25 30 0 0.2 0.4 0.6 0.8 1 Time (minutes) Time (minutes) 0 5 10 15 20 25 0 0.2 0.4 0.6 0.8 1 Correlation coeff. (R) Monkey 1 Monkey 1 Monkey 2 Monkey 2 Proximal/Distal (X) Right/Left (Y) Up/Down (Z) f g 60 mm Mouth Mouth Start Tray Figure 3 Real-time prediction of 3D hand movements. a, b, 3D hand movement trajectories produced by monkey 1 (a) and 2 (b) during single experimental sessions. c, schematic diagram of the four possible target locations in the food reaching task. d, e, samples of observed (black line) and real-time predicted (red line) 3D hand movement for monkey 1 (d) and 2 (e). f, g, Correlation coefficient variation for x (black line), y (blue line) and z (red line) dimensions of predicted 3D hand movements using the linear model. �� 2000 Macmillan Magazines Ltd