Massive AIS data storage and query based on Hadoop platform

Abstract

There are nearly 50 million AIS data generated every day in the Yangtze River Basin. Efficient storage and query of AIS data become the key problem of smart maritime administration. To overcome this problem, a distributed storage and query scheme based on Hadoop platform is proposed. Firstly, the original AIS message is parsed, the abnormal data is removed and the parsed data is pushed to the Kafka message queue. Secondly, the messages in the queue are pushed to different servers to complete the storage of real-time cache, historical trajectory and regional information. Finally, the query interface is provided to realize the real-time query of ship status, trajectory and regional ship in a certain period of time. The real applications show that compared with the traditional relational database, the solution can meet the requirements of real-time query and has high scalability.