The Future of Time

Abstract

Preface This book is the result of an unsuccessful joke. During the summer of 1990, we were both participating in the Complex Systems Summer School of the Santa Fe Institute. Like many such programs dealing with "complexity," this one was full of exciting examples of how it can be possible to recognize when apparently complex behavior has a simple understandable origin. However, as is often the case in young disciplines, little effort was spent trying to understand how such techniques are interrelated, how they relate to traditional practices, and what the bounds on their reliability are. These issues must be addressed if suggestive results are to grow into a mature discipline. Problems were particularly apparent in time series analysis, an area that we both arrived at in our respective physics theses. Out of frustration with the fragmented and anecdotal literature, we made what we thought was a humorous suggestion: run a competition. Much to our surprise, no one laughed and, to our further surprise, the Santa Fe Institute promptly agreed to support it. The rest is history (630 pages worth). Reasons why a competition might be a bad idea abound: science is a thoughtful activity, not a simple race; the relevant disciplines are too dissimilar and the questions too difficult to permit meaningful comparisons; and the required effort might be prohibitively large in return for potentially misleading results. On the other hand, regardless of the very different techniques and language games of the different disciplines that study time series (physics, biology, economics,.. .), very Times Series Prediction: Forecasting the Future and Understanding the Past, A. S. Weigend and N. A. Gershenfeld, eds. Reading, MA: Addison-Wesley, 1993. i ii similar questions are asked: What will happen next? What kind of system produced the time series? How can it be described? How much can we know about the system? These questions can have quantitative answers that permit direct comparisons. And with the growing penetration of computer networks, it has become feasible to announce a competition, to distribute the data (withholding the continuations), and subsequently to collect and analyze the results. We began to realize that a competition might not be such a crazy idea. The Santa Fe Institute seemed ideally placed to support such an undertaking. It spans many disciplines and addresses broad questions that do not easily fall within the purview of a single academic department. Following its initial commitment, we assembled a group of advisors [1] to represent many of the relevant disciplines in order to help us decide if and how to proceed. These initial discussions progressed to the collection of a large library of candidate data sets, the selection of a representative small subset, the specification of the competition tasks, and finally the publicizing and then running of the competition (which was remotely managed by Andreas in Bangkok and Neil in Cambridge, Massachusetts). After its close, we ran a NATO Advanced Research Workshop to bring together the advisory board, representatives of the groups that had provided the data, successful participants, and interested observers. This heterogeneous group was able to communicate using the common reference of the competition data sets; the result is this book. It aims to provide a snapshot of the range of new techniques that are currently used to study time series, both as a reference for experts and as a guide for novices. Scanning the contents, we are struck by the variety of routes that lead people to study time series. This subject, which has a rather dry reputation from a distance (we certainly thought that), lies at the heart of the scientific enterprise of building models from observations. One of our goals was to help clarify how new time series techniques can be broadly applicable beyond the restricted domains within which they evolved (such as simple chaos experiments), and, at the same time, how theories of everything can be applicable to nothing given the limitations of real data. We had another hidden agenda in running this competition. Any one such study can never be definitive, but our hope was that the real result would be planting a seed for an ongoing process of using new technology to share results in what is, in effect, a very large collective research project. The many papers in this volume that use the competition tasks as starting points for the broader and deeper study of these common data sets suggests that our hope might be fulfilled. This survey of what is possible is in no way meant to suggest that better results are impossible. We will be pleased if the Santa Fe data sets and results become common reference