Web robots periodically crawl Web sites to download their content, thus producing potential bandwidth overload and performance degradation. To cope with their presence, it is then important to understand and predict their behavior. The analysis of the properties of the traffic generated by some commercial robots has shown that their access patterns vary: some tend to revisit the pages rather often and employ many cooperating clients, whereas others crawl the site very thoroughly and extensively following regular temporal patterns. Crawling activities are usually intermixed with inactivity periods whose duration is easily predicted. © 2013 Springer-Verlag London.
CITATION STYLE
Calzarossa, M. C., & Massari, L. (2013). Temporal analysis of crawling activities of commercial web robots. In Computer and Information Sciences III - 27th International Symposium on Computer and Information Sciences, ISCIS 2012 (pp. 429–436). Kluwer Academic Publishers. https://doi.org/10.1007/978-1-4471-4594-3_44
Mendeley helps you to discover research relevant for your work.