LoginSign Up
← Back to all sparks
R

rvest

ANALYTICS
Velocity0.0

Scrape and parse data from HTML web pages in R

github.com/tidyverse/rvest ↗

rvest grew a real browser, turning a static scraper into a dynamic one.

rweb-scrapinghtml-parsingheadless-browsertidyverse
◆Current state
rvest is the tidyverse's HTML scraping package. The 1.0.0 line rebuilt table parsing to mimic how browsers actually read tables, and 1.0.4 added read_html_live(), which drives a headless browser so JavaScript-rendered pages can be scraped at all. 1.0.5 is consolidation on that newer path.
◆Where it's heading
The package has split into two modes: fast static parsing for ordinary documents, and a live browser session for client-rendered ones. Recent effort concentrates on hardening the live path, where page navigation and interaction bugs surface that never existed in static parsing.
◆Prediction
Expect further fixes and documentation around read_html_live() as more scraping targets move rendering to the client.

◆Recent moves

  1. 11mo ago

    rvest 1.0.5 fixes stale nodes after click navigation in LiveHTML

    Fixes LiveHTML objects erroring with 'could not find node with given id' when a click triggered page navigation — a failure mode specific to the live-browser path. The new vignette scraping JS-rendered starwars data shows where the team expects users to need it.

    View source ↗
  2. 2y ago

    rvest 1.0.4 scrapes JavaScript pages via a live browser session

    ⚡ SPARK

    The release that redefined what rvest is for: read_html_live() loads pages into a real browser, making JavaScript-rendered content scrapable and allowing simulated clicks, typing and scrolling. Everything since has been hardening this path.

    View source ↗
  3. 3y ago

    rvest 1.0.3 re-documents to fix .Rd HTML issues

    A re-documentation pass to fix HTML issues in the .Rd files. CRAN hygiene with no functional change.

    View source ↗
  4. 4y ago

    rvest 1.0.2 returns empty tibbles for empty tables

    CRAN-driven fixes plus html_table() returning empty tibbles for empty tables. A small correctness edge in the table parser rewritten at 1.0.0.

    View source ↗
  5. 5y ago

    rvest 1.0.1 parses blank rowspan and colspan as 1

    html_table() now reads blank rowspan and colspan attributes as 1, matching browser behaviour. Another rough edge in the new table algorithm rather than a change in capability.

    View source ↗
  6. 5y ago

    rvest 1.0.0 rewrites table parsing to match browser behaviour

    The 1.0.0 milestone: html_table() rewritten from scratch to follow the algorithm browsers use, and a new html_text2() that renders nodes closer to what a reader sees, turning <br> into newlines and dropping insignificant whitespace. Fidelity work that set up the later move to a live browser.

    View source ↗