VR‐based dataset for autonomous‐driving system

Abstract

At present, visual recognition systems have acquired wide employment in the autonomous-driving area. The lack of fully featured benchmarks that mimic scenarios faced by autonomous-driving system is the core factor limiting the visual understanding of complex urban traffic scenes. However, to establish a dataset adequately captures the complexity of real-world urban traffics consuming time and effort. In order to solve these difficulties, authors involve virtual reality to develop a large-scale dataset, which trains and tests approaches for autonomous-driving vehicles. Using the label of the object in virtual scenes, the coordinate transformation of a 3D object to a 2D plane is calculated, which makes the label of the pixel block corresponding to the object in the 2D plane accessible. Their recording platform is equipped with video camera models, LiDAR model and positioning system. By using the pilot-in-the-loop method with driving simulator hardware and VR devices, the authors acquire and establish a large, diverse dataset comprising stereo video sequences recorded in streets and mountain roads from several different environments. Their pioneering method of using VR technology significantly mitigates the costs of acquisition of training data. Crucially, their effort exceeds previous attempts in terms of dataset size, scene variability and complexity. 11Introduction Currently, the rise of deep learning [1] has brought a major impact on the state of the art in computer vision and machine learning. The availability of full-featured, publicly available datasets such as Microsoft COCO [2], PASCAL-Context [3], PASCAL VOC [4] and ImageNet [5], which allow deep neural networks to develop their full potentiality, could be a crucial factor to their success. Despite the existing gap to human performance, scene understanding approaches have started to play an essential part in advanced artificial intelligence systems. Based on this, developing autonomous systems that are able to assist humans in everyday tasks becomes one of the grand challenges in modern computer science. One typical example is the autonomous-driving system, which is developed to replace human driving in order to decrease fatalities caused by traffic accidents. In this area, research progress is closely related to the datasets. Autonomous-driving research is also critically dependent on vast quantities of real-world data for development, testing and validation of algorithms before being applied in the real world. Fig. 1 is the panorama of three types of traffic environment used to acquire data in this paper. Following the benchmark-driven approach of the computer vision community, a number of vision-based autonomous-driving datasets have been released including [6-9] notably the KITTI dataset in [10], the recent Cityscapes dataset in [11] and Oxford dataset in [12]. These datasets focus primarily on the development of algorithmic competencies for autonomous-driving: motion estimation as in [13, 14], stereo reconstruction as in [15], pedestrian and vehicle detection as in [16], semantic classification as in [17]. On top of the above, for long-term autonomy, the other two aspects were taken into account: chiefly, localisation in the same environment under significantly different conditions as in [18] and mapping in the presence of structural change over time as in [19]. However, these urban scene datasets are often much smaller than datasets addressing full-featured and more general settings. We argue that they do not fully capture the complexity and variability of real-world urban traffic scenes. Additionally, to establish a dataset as described above requires vast quantities of time and effort, notably work for increasing annotation richness. In view of the above-mentioned defects in the establishment of the dataset, our thought is to introduce virtual reality (VR) technology into data acquisition in order to mitigate these difficulties by a wholly new approach. Based on our previous research [20], the graphic transformation of a 3D object to the 2D plane is completed by implementing the functional instance of build-in labels in our virtual environment, which makes the auto-labelling of the pixel block corresponding to the object in the 2D plane accessible. By using the pilot-in-the-loop method with driving simulator hardware and VR devices, we acquire and establish a large, diverse dataset comprising stereo video sequences recorded in streets and mountain roads from several different environments. Specifically, our dataset is tailored for autonomous driving, not only in an urban environment but also in landscapes. Beyond this, a much wider range of highly complex scenes that were recorded in different cities, neighbourhoods, countries and mountain landscape was involved. These make our dataset significantly surpass previous efforts not only in terms of size, annotation richness, but also, more importantly, with variability and complexity. We go beyond the pixel-level semantic labelling by considering the instance-level semantic labelling. More than that, we also provide depth information through the stereo vision to facilitate research on the 3D scene understanding. 22Data acquisition The data acquisition work had continued since May 2018 when we finished the development of our data acquisition platform [20], which involved >200 h of recorded driving in a virtual environment. Our crews using driving simulator hardware instead of autonomous capabilities drove the vehicle manually throughout the data acquisition platform. Fig. 2 presents a montage of images taken at the same location on different acquisition traversals, which illustrate the range of appearance changes with weather and time. Fig. 3 lists the summary statistics for our dataset. 2.1 Acquisition method The data acquisition in a virtual environment relies on the virtual driving simulation environment. We choose the human-in-the-loop driving simulation method (illustrated in Fig. 4) using VR head