HTTP Requests

Abstract

In this chapter, we focus on general, all-purpose infrastructure we can use in R for accessing networks and the Web. The RCurl package provides both high-level and intermediate-level functionality within R that allow us to make rich and flexible requests to a large variety of different servers, including Web servers and applications that speak different protocols. RCurl generalizes the functionality built into R for downloading documents, etc., by a) supporting more protocols, b) allowing us to control many more aspects of the requests. We use RCurl as the foundation in R for scraping data from static Web pages, submitting HTML forms, interacting with RESTful APIs (Application Programming Interfaces) on both Web servers and local applications, invoking methods provided by SOAP servers, and using XML-RPC for remote procedure calls. RCurl provides support for many protocols used on the Internet (e.g., HTTP, HTTPS, FTP) and a large collection of options for controlling the requests (e.g., cookies, login and passwords, content type, header fields). In this chapter, we will focus primarily on HTTP and illustrate how to use RCurl generally by developing interfaces to different Web APIs.