Web scraping

A text mining function for websites

For one of my projects I needed to download text from multiple websites. I did this with rvest and dplyr. While this can be relatively easy if the sources come from the same websites, it can be pretty tedious when the website hosts are various. The reason is how the content is kept in the HTML of the website. Assume that you want to extract the title, author information, publish date, and of course the main article text. You can access that information via CSS or XPath. The following text will walk you through an example and provide the relevant code.