Design and implementation of craweper based on scrapy

Abstract

With the development of the Internet, network data covers various fields, but with the increasing amount of network data and diversified data formats, it becomes more and more difficult for users to obtain valuable data from massive data. At present, data collection technology has been studied at home and abroad, and has found that network resources can be automatically obtained through network crawler technology. This article takes the information of second-hand housing in Chongqing as an example, designs a crawler program based on Scrapy framework, captures and stores the housing information in some central and western regions, and finally uses Excel data analysis to analyze the second-hand housing resources in Chongqing according to the region and house type. The results show that this program can automatically collect housing information from Anju customers, improve the speed and quality of user obtaining information, and provide a data source for user data analysis.