XIEv: Dynamic analysis for crawling and modeling of web applications

Abstract

Researchers and practitioners in the fields of testing, security assessment and web development seeking to evaluate a given web application often have to rely on the existence of a model of the respective system, which is then used as input to task-specific tools. Such models may include information on HTTP endpoints and their parameters, available user actions/event listeners and required assets. Unfortunately, this data is often unavailable in practice, as only rigorous development practices or manual analysis guarantee their existence and correctness. Crawlers based on static analysis have traditionally been used to extract required information from existing sites. Regrettably, these tools can not accurately account for the dynamic behavior introduced by JavaScript and other technologies that are prevalent on modern sites. While methods based on dynamic analysis exist, they are not fully capable of identifying event listeners and their effects. This work presents XIEv, an approach for dynamic analysis of web applications that produces an execution trace usable for the extraction of navigation graphs, identification of bugs at runtime and enumeration of resources requested by each page. It offers improved recognition and selection of event listeners as well as a greater range of observed effects compared to existing approaches.