LoginSign Up
← Back to all sparks
W

wasserportal

ANALYTICS
Velocity0.0

R Package with Functions for Scraping Data of Wasserportal Berlin

github.com/KWB-R/wasserportal ↗

A Berlin groundwater scraper grew a full IoT publishing pipeline in two days.

groundwater-monitoringweb-scrapingthingsboardiot-telemetryopen-datar-package
◆Current state
wasserportal scrapes groundwater and surface water monitoring data from Wasserportal Berlin — station metadata, water levels, quality series. For its first four years that was the whole product, with releases tracking the portal's own churn: a new API version in 2023, new three-letter variable codes, and a 2026 fix decoding pages as windows-1252 because the server declares UTF-8 but emits Latin-1. In June 2026 it added a ThingsBoard integration, then extended it to self-hosted instances two days later.
◆Where it's heading
The package is repositioning from a data-access library into the first half of a monitoring pipeline. 0.6.0 shipped push helpers, device discovery and cleanup, an importable dashboard and a GitHub Actions workflow; 0.7.0 immediately generalised the authentication so it works against self-hosted Community Edition rather than only the Cloud tier, and lifted the push from a five-station demo to the entire groundwater archive. The hardening detail in 0.7.0 — retries on 5xx, warnings when only one of two credentials is set — reads like a maintainer who has run this against a real deployment.
◆Prediction
The obvious next step is extending the ThingsBoard path to the surface water stations, which the package has flagged as under-supported since 0.1.0. The entries show no other direction under construction.

◆Recent moves

  1. 1mo ago

    Self-hosted ThingsBoard support via JWT login

    The two-day-old ThingsBoard integration gets generalised: tb_login() adds username/password JWT authentication across the tenant-API helpers so self-hosted Community Edition works, and the push scales from a five-station demo to the whole groundwater archive. Retries on 5xx and a warning for half-configured credentials are the marks of a path that has been run in anger.

    View source ↗
  2. 1mo ago

    ThingsBoard integration turns the scraper into a pipeline

    ⚡ SPARK

    The release that changes what the package is for. Four years of read-only scraping gains a write side — telemetry push, device management, a shipped dashboard and scheduled automation — and 0.7.0 two days later exists only to generalise it.

    View source ↗
  3. 3mo ago

    windows-1252 decoding fixes umlaut handling on Windows

    A real correctness fix underneath a CI refresh: the portal declares UTF-8 but serves Latin-1 bytes, which left umlauts mis-marked and broke character substitution on Windows R. Table extraction moves to xml2 directly to sidestep locale-sensitive trimming. The rest is Actions modernisation ahead of a Node 20 deprecation.

    View source ↗
  4. 3y ago

    Export and import functions for scraped datasets

    Round-trip persistence for scraped groundwater and surface water datasets, so results survive between sessions. An early step toward the pipeline framing that the ThingsBoard work completes three years later.

    View source ↗
  5. 3y ago

    Adapted to Wasserportal's new API version

    The portal changed its API and its variable codes to three letters; the package followed, adding an api_version argument and an n_cores option, plus a readability pass over duplicated code. The recurring cost of building on someone else's public portal.

    View source ↗
  6. 4y ago

    Temperature parameter naming corrected

    A one-line bugfix over 0.1.0 correcting a mislabeled temperature parameter in the stations table; the rest of the note restates the package's existing function set. Surface water support is already flagged as limited here, and stays that way.

    View source ↗