Frontiers of transportation video structural analysis in the smart city

Abstract

As the construction of smart cities continues to deepen, our country gradually builds multidimensional and omnidirectional sensor systems in roads, railways, and urban rails and other ground transportation fields to build strong data support for smart transportation. Faced with all-weather traffic data collected by sensors, analyzing the data by relying solely on human resources is no longer possible. Therefore, studying the structural analysis technology of traffic video and establishing a safe, flexible, and efficient intelligent transportation system has significant social benefits and application value. Traffic video structural analysis is the core technology in smart transportation. It aims to use artificial intelligence algorithms to parse unstructured traffic video data into structured semantic information that is easy for workers and computers to understand and provide basic technical support for subsequent related tasks. The structural analysis of traffic video is a key technology for smart city construction. It can help the police in quickly locating criminal vehicles and travel routes, greatly improve the police's efficiency in solving crimes, and maintain city safety; it can also automatically identify illegal vehicles and types of violation, constrains people to abide by the traffic order, and realize a smooth urban traffic environment. With the advent of the 5G internet of things era, ultrahigh network bandwidth and transmission speed further improves the quality and efficiency of vehicle video transmission. Efficiently and accurately conducting traffic video structure analysis will be the focus of research in the next few years. Traffic video structural analysis includes vehicle video structural analysis, personnel structural analysis, and behavior analysis. Among them, as a complex, multistep task, vehicle structuring is mainly composed of three subtasks, namely, vehicle detection, vehicle attributes (license plate, type, and color) recognition, and vehicle retrieval and reidentification. Human face structuring and pedestrian structuring are two important research directions in the intelligent analysis of traffic videos. They mainly analyze some apparent attributes of human faces or pedestrians, such as age, gender, mask, backpack, clothing color, and length. Pedestrian behavior analysis refers to the identification and prediction of pedestrian actions. For example, the speed at which pedestrians currently head and in which direction, whether they are answering calls, and whether they have to cross the road. For the task of vehicle structure analysis, first, the object detection technology must be used to quickly and accurately locate the vehicle. Second, on the basis of positioning the vehicle, it fully excavates the visual characteristics of the vehicle, realizes the identification of the inherent attributes of vehicle, and generates structured tags about the vehicle. Finally, on the basis of structured tags, the retrieval technology and reidentification technology are further combined to realize the retrieval and reidentification of a specific vehicle in the massive video data. Personnel structural analysis and behavior analysis can detect and identify pedestrians in traffic videos and conduct structured data extraction and behavior analysis of detected personnel. In the analysis of personnel structure, a person is extracted as a descriptive individual. In terms of face structure, it includes accurate facial positioning, facial feature extraction, and facial feature comparison. In terms of pedestrian structure, it includes gender, age, and age of the person. Various descriptive information includes height, hair accessories, clothing, carrying items, and walking patterns. Pedestrian behavior analysis is carried out on the basis of personnel structure analysis. Behavior analysis refers to the recognition, comprehension, and prediction of pedestrian actions. In the area of big data processing and analysis of traffic video, research on vehicle structuring started earlier and related technologies have also developed rapidly, but it can still be remarkably developed. The premise of vehicle structuring is vehicle detection, which is affected by the shooting scene and the moving speed of the vehicle. Accurately locating the vehicle in the case of low light and the fast vehicle speed is still a problem to be solved. Many types of vehicles are found in the market, and the differences between models of similar brands are small. License plate recognition has become more important. In complex and changeable scenes, the generalization and accuracy of the positioning and recognition algorithm should be further improved. The extensive deployment of traffic monitoring equipment realizes all-weather monitoring of relevant road systems and further increases the difficulty of vehicle retrieval and reidentification tasks. Rapid retrieval or reidentification of target vehicles in complex and changeable scenes is crucial. It requires continuous investment and a much innovative research by scientific researchers. The need for structured pedestrian analysis has gradually emerged with further improvement of urban management. Pedestrian structuring mainly analyzes some apparent attributes of faces or pedestrians, such as age, gender, and clothing style, and provides more detailed data support for subsequent related tasks. Pedestrian structured analysis technology has also ushered in a period of rapid development with the development of deep learning. However, the structured analysis of pedestrians for specific scenarios, such as accurately identifying the age and gender of a person in an unconstrained environment, implementing the deployment of high-precision models in terminal systems with limited resources, and integrating multimodal information to further improve the accuracy of pedestrian attribute recognition, needs further research. Pedestrian behavior analysis is a more advanced task in traffic video big data processing and analysis. It is more challenging due to factors, such as shooting scenes, moving cameras, viewing angles, and lighting changes. Judging from the behavior recognition effect of the mainstream neural network architecture, the current model does not achieve the desired effect on the large-scale behavior data set Kinetic because the existing model still fails to fully learn and model the behavioral timing relationship. In the field of behavior recognition, future research can still focus on recognition models for designing long-time-dependent network architectures, adapting large-scale data sets, and achieving lightweight behavior. With the development of Internet of Things and 5G technologies, the promotion of new technologies has also played an important role in the structural analysis of traffic video. To be equipped with IoT devices has become an inevitable trend for modern cars. Vehicles can be connected to basic transportation facilities (vehicle to infrastructure, V2I) or to surrounding vehicles (vehicle-to-vehicle, V2V). The development of these technologies depends on the common progress of vehicle video structuring and internet of things technology. With the global popularity of 5G technology, rapid transmission of high-quality video data has become a reality. Extracting structured information more efficiently from traffic videos, such as vehicle information, pedestrian information, and behavior prediction, has become more urgent. Researchers should study on improving the performance of related algorithms, should design more efficient hardware systems, and build more efficient traffic video structured analysis systems through software and hardware collaborations. We discuss the related work on traffic video structural analysis in detail from three aspects, as follows: vehicle, personnel, and behavior analysis. Moreover, we summarize these research works and provide some reasonable directions for future work.