Journal of Artificial Intelligence Research 31 (2008) 591-656 Submitted 11/07 published 3/08 A Multiagent Approach to Autonomous Intersection Management Kurt Dresner kdresner@cs.utexas.edu Peter Stone pstone@cs.utexas.edu Department of Computer Sciences, University of Texas at Austin 1 University Station [C0500], Austin, TX 78712 USA Abstract Artificial intelligence research is ushering in a new era of sophisticated, mass-market transportation technology. While computers can already fly a passenger jet better than a trained human pilot, people are still faced with the dangerous yet tedious task of driving au- tomobiles. Intelligent Transportation Systems (ITS) is the field that focuses on integrating information technology with vehicles and transportation infrastructure to make transporta- tion safer, cheaper, and more e���cient. Recent advances in ITS point to a future in which vehicles themselves handle the vast majority of the driving task. Once autonomous vehicles become popular, autonomous interactions amongst multiple vehicles will be possible. Cur- rent methods of vehicle coordination, which are all designed to work with human drivers, will be outdated. The bottleneck for roadway e���ciency will no longer be the drivers, but rather the mechanism by which those drivers��� actions are coordinated. While open-road driving is a well-studied and more-or-less-solved problem, urban tra���c scenarios, especially intersections, are much more challenging. We believe current methods for controlling tra���c, specifically at intersections, will not be able to take advantage of the increased sensitivity and precision of autonomous vehicles as compared to human drivers. In this article, we suggest an alternative mechanism for coordinating the movement of autonomous vehicles through intersections. Drivers and intersections in this mechanism are treated as autonomous agents in a multiagent system. In this multiagent system, intersections use a new reservation-based approach built around a detailed communication protocol, which we also present. We demonstrate in simulation that our new mechanism has the potential to significantly outperform current intersection control technology���tra���c lights and stop signs. Because our mechanism can emulate a tra���c light or stop sign, it subsumes the most popular current methods of intersection control. This article also presents two extensions to the mechanism. The first extension allows the system to control human-driven vehicles in addition to autonomous vehicles. The second gives priority to emergency vehicles without significant cost to civilian vehicles. The mechanism, including both extensions, is implemented and tested in simulation, and we present experimental results that strongly attest to the e���cacy of this approach. 1. Introduction Few concepts, if any, embody the goals and aspirations of artificial intelligence as well as fully autonomous robots. Countless films and stories have been made that focus on a future filled with such humanoid agents which, when not violently overthrowing their human masters, run errands, complete menial tasks, or perform duties that would be too di���cult or dangerous for humans. However, machines that sense, think about, and take actions in the real world around us are no longer just the stuff of science fiction and fantasy. c 2008 AI Access Foundation. All rights reserved.

Dresner & Stone Research initiatives like Robocup (?) and the DARPA Grand Challenge (?) have shown that current AI can produce autonomous, embodied, competent agents for complex tasks like playing soccer or navigating the Mojave Desert, respectively. While certainly no small feat, traversing a barren desert devoid of pedestrians, narrow lanes, and multitudes of other fast-moving vehicles is not a typical daily task for humans. As Gary Bradski, a researcher at Intel Corp. said following the successful completion of the 2005 Grand Challenge by ���Stanley,��� a modified Volkswagen Touareg, ���Now we need to teach them how to drive in tra���c��� (?). Since then, competitors in the 2007 DARPA Urban Challenge took significant strides towards this next milestone, though in the competition cars did not need to sense tra���c signs or signals and tra���c was relatively sparse���more characteristic of suburban than dense urban settings. In modern urban settings, automobile tra���c and collisions lead to endless frustration as well as significant loss of life, property, and productivity. A 2004 study of 85 U.S. cities by researchers at Texas A&M University estimated the annual time spent waiting in tra���c to be 46 hours per capita, up from 16 hours in 1982 (?). Americans burn approximately 5.6 billion gallons of fuel each year simply idling their engines. All told, the annual financial cost of tra���c congestion has swollen from $14 billion to more than $63 billion (in 2002 US dollars) in this period. The cost of all the wasted time and fuel due to congestion pales in comparison to the costs associated with automobile collisions. In a 2002 report, the National Highway Tra���c Safety Administration (NHTSA) put the annual societal cost of automobile collisions in the U.S. at $230 billion (?). Fully autonomous vehicles may be able to spare us much, if not nearly all of these costs. An autonomous driver agent can much more accurately judge distances and velocities, attentively monitor its surroundings, and react instantly to situations that would leave a (relatively) sluggish human driver helpless. Furthermore, an autonomous driver agent will not get sleepy, impatient, angry, or drunk. Alcohol, speeding, and running red lights are the top three causes of automobile collision fatalities. Autonomous driver agents���properly programmed���would eliminate all three. A fully autonomous vehicle that will drive in tra���c will have to do everything from obeying the speed limit and staying in its lane to detecting and tracking pedestrians or choosing the best route to the mall. While this is certainly a complex task, advances in artificial intelligence, and more specifically, Intelligent Transportation Systems (ITS), suggest that it may soon be a reality (?). Cars can already be equipped with features of autonomy such as adaptive cruise control, GPS-based route planning (?, ?), and autonomous steering (?, ?). Some current production vehicles even sport these features. DaimlerBenz���s Mercedes-Benz S-Class has an adaptive cruise control system that can maintain a safe following distance from the car in front of it, and will apply extra braking power if it determines that the driver is not braking hard enough. Both Toyota and BMW are currently selling vehicles that can parallel park completely autonomously, even finding a space in which to park without any driver input. In 2008, General Motors (GM) plans to release a nearly autonomous vehicle under its European ���Opel��� brand. The 2008 Opel Vectra will be able to drive itself at speeds up to 60 miles per hour, even in heavy tra���c. Using a video camera, lasers, and a lot of processing power, the car will be able to identify tra���c signs, curves in the street, lane markings, as well as other vehicles. By the end of the decade, GM hopes to incorporate the system into many other models. 592

A Multiagent Approach to Autonomous Intersection Management Autonomous vehicles are coming. In this article, we present a well-defined multiagent framework to manage large numbers of autonomous vehicles at intersections. While there still exist many technical hurdles and rigorous safety tests, we show in simulation that this framework may someday dramatically improve the safety and e���ciency of our roadways. 1.1 Multiagent Systems As autonomous vehicles become more and more prevalent, the possibility of autonomous interactions among multiple vehicles becomes more interesting. Multiagent Systems (MAS) is the subfield of AI that aims to provide both principles for construction of complex sys- tems involving multiple agents and mechanisms for coordination of independent agents��� behaviors (?). Automobile tra���c is a vast multiagent system involving millions of hetero- geneous agents: commuters, truck drivers, pedestrians, cyclists, and even tra���c-directing police o���cers. The mechanism that coordinates the behavior of these agents is a complex conglomeration of laws, signs, and signaling systems that vary slightly from state to state and widely from country to country. The mechanism is designed to work closely with the agents���the humans���that populate the multiagent system. Tra���c lights leave time in be- tween green lights to allow slower or perhaps impatient drivers to clear intersections. Street signs are colored brightly to make them easier to see and use simple designs to make them easy to understand. Drivers must maintain a su���cient following distance to make up for slow reaction times. Speed limits ensure that humans have enough time to process all the necessary information about the position and velocities of other vehicles. Safety buffers of myriad sorts are built into almost every part of the system to compensate for the limitations of humans. The first generation of autonomous vehicles will undoubtedly need to work within this system. Processing-intensive vision algorithms will identify and extract semantic informa- tion from signs and signals, special subroutines will ensure that the vehicles do not exceed the speed limit, and in the middle of the night, with not another moving vehicle for blocks, an autonomous vehicle will come to a stop at a red light. However, once most vehicles are autonomous and the limitations are eliminated, it will not make sense to use a mechanism designed to control fundamentally different agents���it will be ine���cient, both in terms of processing power and getting vehicles to their destinations quickly. Replacing this soon-to-be-outdated mechanism is inherently a multiagent challenge for several reasons. First, there are no viable single-agent solutions one computer cannot handle all the vehicles in the world. Second, with vehicles constantly entering and leaving countries, states, cities, and towns, any solution will have to be flexible and distributed. Third, the different agents have separate, and sometimes conflicting objectives. As with human-driven vehicles, autonomous vehicles will act in their own self-interest, attempting to minimize travel time, distance, and fuel use. Other types of agents may aim to maximize social welfare, minimizing these quantities for the average vehicle. Finally, even if a single computer could control a city���s worth of tra���c, it would be a very sensitive point of failure. 1.2 Intersections On the open road, automobiles can be more or less completely autonomous. Furthermore, there is little need for more than a simple reactive behavior that keeps the vehicle in the 593

Dresner & Stone lane, maintains a reasonable distance from other vehicles, and avoids obstacles. Even lane changing can be safely and e���ciently accomplished by an autonomous vehicle (?). The algorithmic and AI aspects of open-road driving are essentially solved. The problem itself is not too di���cult: there are no pedestrians or cyclists and vehicles travel in the same direction at similar velocities relative movement is smooth and rare. Intersections are a completely different story: vehicles constantly cross paths, in many different directions. A vehicle approaching an intersection can quickly find itself in a situa- tion in which a collision is unavoidable, even when it has acted optimally. Tra���c statistics support the sensitive nature of intersections. Vehicle collisions at intersections account for anywhere between 25% and 45% of all collisions. As intersections make up a very small portion of the roadway, this is a wildly disproportionate amount. Collisions at intersec- tions tend to involve cars traveling in different directions, and thus they frequently result in greater injury and damage. Most modern-day intersections are controlled with tra���c lights or stop signs, the former usually reserved for larger, busier intersections. At the busiest of intersections���freeway interchanges���large, extremely expensive cloverleaf junctions are built. With the vastly improved precision control and sensing that autonomous vehicles will offer, there must be a more e���cient and safe way to manage intersections. Imagine the scenario in which an autonomous vehicle stops at a red light in the middle of the night with no other vehicles nearby. At the very least, the vehicle should be able to communicate its presence to the intersection, which can verify that no other vehicles are nearby, and turn the light green for the stopped vehicle. In a more ambitious implementation, the intersection could turn the light green preemptively, obviating the stop altogether. In this article, we go a step further, allowing vehicles to ���call ahead��� and reserve space-time in the intersection. The remainder of this article is organized as follows. In Section 2, we describe the prob- lem of autonomous intersection management and a framework with which we will attempt to solve this problem. In Section 3, we describe the implementation of the solution frame- work. Section 4 presents our experiments and empirical results. In Section 5, we conduct a failure mode analysis of the proposed mechanism. Related work is discussed in Section 6. Section 7 briefly explores some avenues for future research and concludes. 2. Problem Statement and Solution Framework Automobile tra���c is already a huge multiagent system with millions of human driver agents, various signaling and control mechanisms, and a complicated protocol governing the actions of the driver agents, in the form of tra���c laws. However, if human drivers are to be replaced by autonomous driving agents, the other elements of the multiagent system should be rethought. Tra���c lights, stop signs, and our current tra���c laws are all designed with human drivers in mind and fail to take advantage of the increased sensitivity and precision of computerized driver agents. If we want autonomous vehicles to operate with high e���ciency and safety, we must design a new way to coordinate them. In this section, we formulate the problem we are trying to solve and present a framework within which we believe the problem can best be solved. 594

A Multiagent Approach to Autonomous Intersection Management 2.1 Desiderata In designing a mechanism by which tra���c is controlled at intersections, we aim to satisfy the following list of properties. Autonomy Each vehicle should be an autonomous agent. If the entire mechanism were centrally controlled, it would be more susceptible to single-point failure, require massive amounts of computational power, and exert unnecessary control over vehicles in situations where they are perfectly capable of controlling themselves. Low Communication Complexity By keeping the number of messages and amount of information transmitted to a minimum, the system can afford to put more communication reliability measures in place. Furthermore, each vehicle, as an autonomous agent, may have privacy concerns which should be respected. Keeping the communication complexity low will also make the system more scalable. Sensor Model Realism Each agent should have access only to sensors that are available with current-day technology. The mechanism should not rely on fictional sensor technology that may never materialize. Protocol Standardization The mechanism should employ a simple, standardized pro- tocol for communication between agents. Without a standardized protocol, each agent would need to understand the internal workings of every agent with which it interacts. This requirement would forbid the introduction of new agents into the system. An open, stan- dardized protocol would make adoption of the system easier and simpler for private vehicle manufacturers. Deadlock/Starvation Avoidance Deadlocks and starvation should not occur in the system. Every vehicle approaching an intersection should eventually cross, even if it is better for the rest of the agents to leave that vehicle stranded. Incremental Deployability The system should be incrementally deployable, in two senses. First, it should be possible to set up selected intersections to use the system, and then slowly expand to other intersections as needed. Second, the system should function even with few or no autonomous vehicles. At each stage of deployment, whether it is an increase in the proportion of autonomous vehicles or the number of equipped intersections, overall performance of the system should improve. At no point should a net disincentive to continue deploying the system exist. Safety Excepting for gross vehicle malfunction or extraordinary circumstances (e.g. nat- ural disasters), as long as they follow the protocol, vehicles should never collide in the intersection. Note that no stronger guarantee is possible���as with modern mechanisms, a suicidal human driver can always steer a vehicle into oncoming tra���c. Furthermore, the system should be safe in the event of total communication failure. If messages are dropped or corrupted, the safety of the system should not be compromised. It is impossible to pre- vent all negative effects due to communication failures, but those negative effects should be isolated to e���ciency. If a message gets dropped, it can make someone arrive 10 seconds later at their destination, but it should not cause a collision. In the rare but unpreventable 595

Dresner & Stone case of gross vehicle malfunction, the system should react and attempt to minimize damage and casualties. E���ciency Vehicles should get across the intersection and on their way in as little time as possible. To quantify e���ciency, we introduce delay, defined as the amount of additional travel time incurred by the vehicle as the result of passing through the intersection. 2.2 The Reservation Idea Of the desiderata, modern-day tra���c lights and stop signs completely satisfy all but the last one. While many accidents take place at intersections governed by tra���c lights, these accidents are rarely, if ever, the fault of the tra���c light system itself, but rather that of the human drivers. However, as we will show, tra���c lights and stop signs are terribly ine���cient. Not only do vehicles traversing intersections equipped with these mechanisms experience large delays, but the intersections themselves can only manage a somewhat limited amount of tra���c. Any stretch of open road can accommodate a certain level of tra���c at a given velocity. The capacity of an intersection involving a road is trivially bounded above by the capacity of the road. As we will also show, the capacity of tra���c lights and stop signs is much less than that of the roads that feed into them. The aim of this research is to create an intersection control mechanism that exceeds the e���ciency of tra���c lights and stop signs, while maintaining each of the other desiderata. With the desiderata in mind, we developed a multiagent approach to direct vehicles through intersections more e���ciently. In this approach, computer programs called driver agents control the vehicles, while an arbiter agent called an intersection manager is placed at each intersection. The driver agents ���call ahead��� and attempt to reserve a block of space-time in the intersection. The intersection manager decides whether to grant or re- ject requested reservations according to an intersection control policy. Figure 1 shows one interaction between a driver agent and an intersection manager. The system functions analogously to a human attempting to make a reservation at a hotel���the potential guest specifies when he or she will be arriving, how much space is required, and how long the stay will be the human reservation agent determines whether or not to grant the reservation, according to the hotel���s reservation policy. Just as the guest does not need to understand the hotel���s decision process, the driver agents should not require any knowledge of the intersection control policy used by the intersection manager. When a vehicle approaches the intersection, the vehicle���s driver agent transmits a reser- vation request, which includes parameters such as time of arrival, velocity of arrival, as well as vehicle characteristics like size and acceleration/deceleration capabilities, to the intersec- tion manager. The intersection manager then passes this information to the policy, which determines whether or not it is safe for the vehicle to cross the intersection. If the policy deems it to be safe, the intersection manager responds to the driver agent with a message indicating the reservation has been accepted and including any supplemental restrictions the driver must observe in order to guarantee the safety of the traversal. Otherwise, the intersection manager sends a message indicating that the reservation request has been re- jected, possibly including the grounds for rejection. In addition to confirming or rejecting the request, the intersection manager may respond with a counter-offer. The driver agent may not pilot the vehicle into the intersection without a reservation. Even with a reserva- 596

A Multiagent Approach to Autonomous Intersection Management REQUEST Intersection Control Policy REJECT CONFIRM Preprocess Postprocess Yes, Restrictions No, Reason Driver Agent Intersection Manager Figure 1: One of the driver agents attempts to make a reservation. The intersection man- ager responds based on the decision of an intersection control policy. tion, a driver agent may only proceed through the intersection according to the parameters and restrictions associated with the reservation. For the sake of brevity, we may refer to a vehicle having or obtaining a reservation, rather than specifically stating that the driver agent of that vehicle has or obtains a reservation. 3. Building The System This section describes the realization of the reservation idea as an implemented algorithm. This process involved developing a simulator in which to run the algorithm, as well as creating behaviors for each of the agents and a protocol by which they can communicate. 3.1 Custom Simulator In order to empirically evaluate the reservation idea, we built a custom time-based simulator. The simulator models an area that is 250 m �� 250 m. The intersection is located at the center of that area, and its size is determined by the number of lanes traveling in each direction, which is variable. We assume throughout that vehicles drive on the right side of the road, however this assumption is not required for the system to work properly. Figure 2 shows a screenshot of the simulator���s graphical display. During each time step, the simulator: 1. Probabilistically spawns new vehicles 2. Provides sensor input to all vehicles 3. Allows all driver agents to act 4. Updates the position of all vehicles according to the physical model 5. Removes any vehicles outside the simulated area that have completed their journey 3.1.1 Vehicles Vehicles in the simulator have the following properties: ��� Vehicle Identification Number (VIN) ��� Length 597

Dresner & Stone Figure 2: A screenshot of the simulator in action. ��� Width ��� Distance from front of vehicle to front axle ��� Distance from front of vehicle to rear axle ��� Maximum velocity ��� Maximum acceleration ��� Minimum acceleration ��� Maximum steering angle ��� Sensor range and the following state variables: ��� Position ��� Velocity ��� Heading ��� Acceleration ��� Steering angle The driver agent assigned to pilot the vehicle may access each of these quantities, with or without noise, depending on the configuration of the simulator. The driver agent may also access several simulated external sensors: a list of all vehicles within the sensor range, and a simplified laser range finder. A detailed description of the simplified laser range finder can be found in Appendix A. The steering angle is the angle of the front wheels with respect to the vehicle. This angle can be changed by the driver agent, but the simulator limits the rate at which it can be changed. This limitation simulates the fact that even a computerized driver cannot move the steering wheel infinitely fast. By introducing this limitation, we more accurately approximate vehicle turning, including some of the more dangerous aspects. If the driver cannot turn the wheels instantaneously, it must ensure that it does not drive around corners 598

A Multiagent Approach to Autonomous Intersection Management at too high a velocity���it may not be able to straighten out quickly enough and wind up veering off the road instead. The constants representing the distance from the front of the vehicle to the front and rear axles allow more accurate simulation of vehicle turning. Specifically, they allow the simulator to treat different styles of vehicle differently. The distance between the front and rear axles is known as the wheelbase. Vehicles with shorter wheelbases can turn more sharply than those with longer wheelbases���if the simulator is to accurately model turning, it needs access to these important parameters. Furthermore, a vehicle with a long hood will turn differently than a vehicle whose front wheels are located nearer to the front of the vehicle. 3.1.2 Lanes Lanes in our system consist of a directed line segment, a width, left and right borders that vehicles may or may not be permitted to cross, and references to which lanes, if any, border on the right and left side. In a real-life implementation, this would be a software construct the vehicles and driver agents would use to perform lane following and changing. If a vehicle wants to change lanes to the left or right, it must first establish that the vehicle is allowed to cross the border between the lanes, after which it can feed its lane-following algorithm the reference to the desired lane. 3.1.3 Physical Model At each time step, the simulator must update the position of every vehicle. Because we model only planar vehicle kinematics and not dynamics, we must make a few assumptions. First, we assume that vehicles do not skid on the road. Second, we assume that vehicles move according to the following differential equations for non-holonomic motion: ���x ���t = v �� cos(��) ���y ���t = v �� sin(��) ����� ���t = v �� tan �� L In these equations, x, y, and �� describe the vehicle���s position and orientation, v repre- sents the vehicle���s velocity, �� describes the vehicle���s steering angle, and L is the vehicle���s wheelbase. We solve these equations holding v and �� constant for each time step. 3.1.4 Measuring Delay In Section 2.1, we introduced delay���the increase in travel time for a vehicle due to the presence of the intersection. In the simulation, this is measured by first assuming that on the open road, a vehicle can maintain its velocity at the speed limit. Each vehicle is timestamped when it enters the simulation and keeps track of how far it has traveled. When the vehicle is removed from simulation, its total delay is calculated as the difference between how long it actually took to travel as far as it did and how long it would take were the vehicle to travel at the speed limit for the entire journey. By this measure, a zero delay is 599

Dresner & Stone not possible when the vehicle is turning, as it needs to slow down in order to safely make the turn. In practice, we compare the delays of all vehicles to delays using a policy that allows vehicles through the intersection unhindered, which will also be non-zero if any of the vehicles turn or if the road is congested. In this way, we can quantify the effect of the intersection on the vehicle, both directly (not being able to go through the intersection because requests were rejected) and indirectly (having to decelerate because another vehicle cannot get through). 3.2 Communication Protocol This section presents a detailed communication protocol by which vehicles and intersections can coordinate their behavior. The protocol as presented here offers three major benefits: ��� All information between the agents goes through one monitorable channel, which makes reasoning about the communication straightforward. ��� By limiting the interactions of the agents to a few message types, we can ensure that no agent has an unrealistic amount of control over another. ��� The agents have a way to communicate that is identical for any intersection manage- ment policy or driver agent policy. Thus, a vehicle can cross an intersection without having any idea what policy the intersection manager is using���it simply sends and receives messages and obeys the rules. The protocol consists of several message types for each kind of agent, as well as some rules governing when the messages should be sent and what sorts of guarantees accompany them. Driver agents can send Request, Change-Request, Cancel, and Done mes- sages. Request and Change-Request are used when the driver agent wants to make a reservation or change an existing reservation, respectively. Both types of request message include all the relevant properties of the vehicle. Driver agents send a Cancel message when they want to cancel an existing reservation. When a vehicle has successfully crossed the intersection, its driver agent sends a Done message to the intersection manager. Both the Cancel and Done messages include the VIN of the vehicle, as well as an identifier for the reservation to be cancelled or reported as complete. Intersection managers can send Confirm, Reject, and Acknowledge messages, as well as a special Emergency-Stop message, which is only used when the intersection manager detects a major problem in the intersection (see Section 5). Confirm is sent when the intersection manager approves a Request or Change-Request message. It includes information describing the reservation���a unique identifier for the reservation, a start time, a start lane, a departure lane (which will be identical to the start lane unless the vehicle is turning), and a list of constraints for the vehicle���s acceleration while it is in the intersection. The Reject message is used to reject either a Request or Change-Request message. The intersection sends an Acknowledge message in response to Cancel and Done messages sent by the vehicles. A more detailed specification of the protocol including full syntax and semantics can be found in Appendix B. 600

A Multiagent Approach to Autonomous Intersection Management 3.2.1 Message Corruption and Loss We assume that messages can be digitally signed, such that the possibility of an undetected message corruption is acceptably small. The protocol is designed specifically to be robust to message loss. If a message is sent but not received���or deemed corrupted���the worst thing that can happen is additional delay. No collisions can occur due to lost messages. When a vehicle makes a reservation request, it does not assume the space is reserved until it receives a confirmation from the intersection manager. If a Request message is dropped, no Confirm message will follow. If a Confirm or Reject message is dropped, the vehicle will simply try again���it won���t assume that it has a valid reservation. 3.2.2 Enabling Policy Switching The protocol hides the implementation of the policy from the driver agents ��� they have no idea how the intersection manager is making its decisions, they are just guaranteed that if they follow them, they will be safe. Thus, there are no stipulations that the policy must remain fixed. An intersection manager could use one policy one moment and then switch to a more appropriate policy later, provided it can still guarantee that vehicles following the protocol make it safely across the intersection. 3.2.3 Intersection Manager The intersection manager acts as a stable communication interface between the driver agents and the intersection control policy and therefore does not contain a lot of functionality. However, regardless of how the policy makes its decision, the intersection manager must present the same interface to the driver agents. The general intersection manager algorithm is shown in Algorithm 1. In it, Cancel messages and Done messages are treated almost identically���when a Done message is received, the intersection manager knows that the policy can erase any information about the related reservation. However, the Done message also may contain information that is useful to the intersection manager and policy. For example, when a vehicle sends a Done message, it could include the delay it experienced crossing the intersection, providing the intersection manager with a sort of reward signal, by which it can judge its performance. 3.3 Driver Agent The vast majority of this research focuses on how to make a better intersection manager and control policy. These parts are designed to work with any driver agent that follows the protocol. However, for testing purposes, a driver agent implementation is required. Despite the fact that a lot of work went into the driver agent (it is probably the most intricate part of the system), it is not the focus of this article. We refer the interested reader to Appendix C, which explains the driver agent in detail. In brief, the driver agent estimates the time and velocity at which it will reach the intersection, and requests an appropriate reservation. If granted a reservation, it attempts to arrive on schedule. If it determines that it is unable to keep the reservation, it cancels the reservation. If it believes it will be substantially early, it attempts to change to an earlier reservation. If it is unable to get a reservation, it decelerates (down to a minimum velocity) and requests again. It does not 601

Dresner & Stone Algorithm 1 The intersection manager algorithm. Vehicle V sends a message to the intersection manager, which responds according to policy P . 1: loop 2: receive message from V 3: if message type is Request then 4: process request for new reservation with P 5: if P accepts the request then 6: send Confirm message to V containing the reservation returned by P 7: else 8: send Reject message to V 9: else if message type is Change-Request then 10: process request for change of reservation with P 11: if P accepts the request then 12: send Confirm message to V containing the reservation returned by P 13: else 14: send Reject message to V 15: else if message type is Cancel then 16: process cancel with P 17: send Acknowledge message to V 18: else if message type is Done then 19: record any statistics supplied in message 20: process cancel with P 21: send Acknowledge message to V enter the intersection without a reservation. On the open road, the driver agent employs a simple lane-following algorithm, and maintains a following distance of one second between its vehicle and the vehicle in front of it. 3.4 The FCFS Policy To this point, we���ve described the substrate infrastructure that enables our research. The remainder of Section 3 introduces the core contribution of this article and the main payoff for creating this infrastructure, namely an intersection control policy that enables fine-grained coordination of vehicles at intersections, and a subsequent dramatic decrease in delays. While the intersection manager communicates directly with the driver agents, the inter- section control policy is the ���brains��� behind the operation. Here we describe an intersection control policy created from the reservation idea as discussed in Section 2.2. Because of the ���First Come, First Served��� nature of the policy, we name this policy FCFS. The main part of the policy���the request processing���is shown in Algorithm 2. Recall that FCFS enables a car to reserve in advance the space-time it needs to cross the intersection. Planning ahead allows vehicles coming from all directions to traverse the intersection simultaneously with minimal delay. The policy works as follows: ��� The intersection is divided into an n �� n grid of reservation tiles, where n is the granularity of the policy. 602

A Multiagent Approach to Autonomous Intersection Management ��� Upon receiving the reservation parameters from an approaching driver agent, the pol- icy runs an internal simulation of the trajectory of the vehicle across the intersection using these parameters. ��� At each time step of the internal simulation, the policy determines which reservation tiles will be occupied by the vehicle ��� If at any time during the simulation the requesting vehicle occupies a reservation tile that is already reserved by another vehicle, the policy rejects the request. Otherwise, the policy accepts the reservation and reserves the appropriate tiles for the times they will be required. Figure 3 shows a graphical depiction of the concept behind the FCFS policy. (a) Successful (b) Rejected Figure 3: The internal simulation of a granularity-8 FCFS policy. The black rectangles rep- resent vehicles, and the shaded tiles are tiles that are currently reserved. In 3(a), a vehicle���s request is accepted, and the intersection reserves a set of tiles at time t. In 3(b), a second vehicle���s request is rejected because during the simulation of its trajectory, the policy determines that it requires a tile (darkly shaded) already reserved by the first vehicle at time t. While the concept behind FCFS is sound, it requires some modifications before it will work reliably, safely, and e���ciently���even in simulation. In the remainder of this section, we present these modifications, most of which were created in response to early experimental results documented in Section 4. 3.4.1 Determining the Outbound Lane In our first implementation of the reservation system, vehicles were capable of traveling only in straight lines. Once we allowed vehicles to turn, it became apparent that the driver agents should not determine which lane they use to exit the intersection. Instead, the intersection manager, which has more information about the intersection, makes this decision. Driver agents indicate in their request message which way they intend to turn, or for more complicated intersections, which direction they intend to go. The intersection control policy then decides in which outbound lane to place the vehicle. For all experiments documented in this article, the FCFS policy chooses the most natural lane: for left and 603

Dresner & Stone right turns, it chooses the nearest lane, whereas for vehicles that are not going to turn, it chooses the lane in which they are planning to arrive at the intersection. However, a policy could behave differently if configured to do so. For example, the policy can create a priority list of outbound lanes based on the inbound lane, and then run internal simulations using each of these lanes until it found an acceptable configuration. For turning vehicles, this list would be the set of outbound lanes in the correct direction, sorted from nearest to farthest. For vehicles not turning, it would ���spiral��� out from the lane in which they arrive���first the arrival lane, then the lane to the left, then the lane to the right, then two lanes to the left, and so forth. In this manner, a vehicle that might otherwise have had its request rejected can obtain a reservation for a different path through the intersection. 3.4.2 Acceleration In The Intersection Given a set of reservation parameters, there are an infinite number of possible trajectories a vehicle can take, if it is allowed to accelerate in the intersection. This is because at each time step, the driver agent could set its vehicle���s acceleration to any value within the limits of the vehicle���s capabilities. Depending on the trajectory, the intersection manager may or may not be able to grant the reservation���one set of accelerations may cause it to collide with another vehicle, while a second set might let the vehicle through safely. For this reason, acceleration in the intersection must be constrained by the intersection control policy. Allowing driver agents to decide their own acceleration within the intersection would require the policy to be much more conservative in estimating vehicle trajectories, thereby reducing e���ciency substantially. Instead, it is the responsibility of the intersection control policy to choose a safe and e���cient acceleration schedule and include it in the Confirm message, if the driver agent���s request is accepted. Choosing the best acceleration schedule for the requesting vehicle, or on an even more basic level, finding a schedule for which the intersection manager can grant the reserva- tion, is a di���cult challenge for the intersection control policy. Our initial solution was to allow no acceleration within the intersection driver agents were required to maintain the same velocity throughout the entire trajectory. This approach had several major flaws, the most severe of which was causing a deadlock scenario as vehicles traversed the intersection more and more slowly, unable to recover from the slightest decelerations. This scenario is described in much more detail in Section 4.2. The FCFS policy, as we have implemented it still takes a fairly straightforward approach to the problem of determining acceleration schedules for reservation requests. It first at- tempts a trajectory in which the requesting vehicle accelerates as quickly as possible to maximum velocity as soon as it enters the intersection. If it cannot grant a reservation based on that trajectory, it tries one in which the requesting vehicle maintains a constant velocity throughout the intersection. If neither work, it rejects the request. Furthermore, if the request indicates that the vehicle will arrive at a su���ciently slow velocity���in our case 10 m/s���it does not grant a fixed-velocity reservation. Were it to grant arbitrarily slow reservations, a vehicle could use an excessively large amount of space-time in the intersec- tion, causing other vehicles undue delay. By enforcing a minimum velocity for fixed-velocity reservations, the policy ensures that no vehicle will spend too long in the intersection. While more complex solutions exist, this solution is good for several reasons. First, it is compu- 604

A Multiagent Approach to Autonomous Intersection Management d��� d Figure 4: Several vehicles are waiting at the intersection. With a reservation distance of d, the front (white) vehicle is incapable of obtaining a reservation because the vehicles behind it (shaded) hold conflicting reservations. Once the white vehicle���s request is rejected, the reservation distance is decreased to d . Once the shaded vehicles cancel their reservations, the white vehicle can obtain a reservation un- contested. tationally tractable: the policy runs at most two internal simulations per request. Second, it allows vehicles that are stopped or moving very slowly at the intersection to clear the in- tersection in a timely manner once they get a reservation. Third, it eliminates the deadlock scenario presented in Section 4.2 by allowing vehicles to recover from decelerating when they cannot obtain a reservation even a vehicle that comes to a full stop at the intersection can accelerate back up to a reasonable velocity as it crosses the intersection. 3.4.3 Reservation Distance Allowing accelerations in the intersection helps eliminate deadlocks, but other problems arose in our prototype implementation that significantly impaired the performance of the system. Frequently, a lane of tra���c would become congested when many vehicles were spawned in that lane. Even when the simulator stopped spawning vehicles in that lane, the lane would remain congested. The problem is that FCFS, as first described, does nothing to control how vehicles in the same lane are alloted reservations. At best, the frontmost vehicle will get a reservation and make it through the intersection unhindered. However, this is often not the case. Sometimes the vehicle in front cannot obtain a reservation (due to congestion), and must decelerate. As shown in Figure 4, driver agents in vehicles further back may expect to accelerate soon and successfully reserve space-time in the intersection that the frontmost vehicle needs. While all vehicles will eventually make it through (a vehicle might get a reservation immediately after vehicles behind it cancel), this process can repeat many times before the frontmost vehicle gets a reservation. In the worst scenarios, a single vehicle can continue for quite some time to obtain reservations that prevent the front car from crossing the intersection. If we could maintain the invariant that vehicles do not get reservations unless all cars in front of them (in their lane) have reservations, this scenario could be avoided entirely. A 605