LoginSign Up
← Back to all sparks
C

climate

INFRA · APIS
Velocity0.0

Interface to Download Meteorological (and Hydrological) Datasets

github.com/bczernecki/climate ↗

climate stops scraping and starts parsing: an in-house SYNOP engine replaces the HTML path.

weather-datadata-acquisitionsynop-parsingdependency-reductionr-package
◆Current state
climate pulls meteorological, hydrological and sounding data in R from public providers, chiefly the Polish IMGW-PIB repository, OGIMET and NOAA. Version 1.4.0 changes how it gets that data rather than what it offers: a new synop_parser() reads raw SYNOP messages directly, the OGIMET functions take a parser argument and default to SYNOP for hourly data with HTML kept for daily summaries, and the XML parsing engine has been replaced by in-house code to cut dependencies. The sounding functions also lost their server-side thermodynamic indices when the legacy Wyoming interface shut down.
◆Where it's heading
This package's history is largely a record of upstream providers changing underneath it, and 1.4.0 is the release where it stops absorbing those changes and reduces its exposure to them. The 1.2.9 notes described a 20-second pause between OGIMET requests as a temporary measure and named a SYNOP parsing engine as the intended fix; that engine is now here, and SYNOP messages can be retrieved in bulk rather than one scrape at a time. Owning the parser and dropping XML means the next provider-side HTML change is a smaller problem than the last one was.
◆Prediction
Expect the SYNOP path to become the default for more of the OGIMET surface as it proves out, with the HTML parser retained only where the provider offers nothing else. Whether the thermodynamic indices return in some form, or stay permanently delegated to thundeR, is the open question these notes leave.

◆Recent moves

  1. 1mo ago

    In-house SYNOP parser replaces the HTML and XML path

    ⚡ SPARK

    The release that changes how climate acquires data rather than what it exposes: synop_parser() reads raw SYNOP messages, OGIMET functions choose between SYNOP and HTML via a parser argument, and the XML engine is replaced by in-house code. It delivers the parsing engine the 1.2.9 notes had flagged as the intended replacement for a rate-limit workaround.

    View source ↗
  2. 6mo ago

    IMGW header fixes and faster selective station downloads

    Repairs corrupted header handling after changes in the IMGW-PIB repository, speeds up selective downloads by station name while using less memory, and moves NOAA retrieval to the current NCEI host. The 20-second pause added between OGIMET requests is described in the notes as temporary, pending the SYNOP engine that lands in 1.4.0.

    View source ↗
  3. 10mo ago

    Hydrological telemetry datastore access added

    Adds retrieval of hydrological station data from the IMGW-PIB telemetry datastore along with its coordinate metadata, extending coverage into a data source the package did not previously reach. The accompanying OGIMET repairs are another instance of the upstream churn that 1.4.0 later addresses structurally.

    View source ↗
  4. 1y ago

    Corrupted IMGW archives skipped instead of breaking downloads

    Adapts to IMGW-PIB dataset changes since 2024 by detecting corrupted zip files, skipping them rather than aborting the run, and collecting errors into an end-of-download summary instead of messaging as it goes. Downloading daily data by station ID was removed in the process.

    View source ↗
  5. 3y ago

    allow_failure keeps CRAN checks green when services are down

    Introduces an allow_failure argument, on by default, so that network problems and temporarily unavailable providers do not surface as errors or warnings, which is the standing tension for any package whose tests depend on live external services. Documentation gains a note on using climate from Python.

    View source ↗
  6. 4y ago

    Multiple station names fixed; units moved to attributes

    A maintenance release fixing multiple-name arguments in the meteo_imgw family and moving units out of column names into attributes for the nearest-stations functions. Mostly cosmetic against the provider-adaptation work that dominates the releases around it.

    View source ↗