LoginSign Up
← Back to all sparks
S

spanishoddata

ANALYTICS
Velocity0.0

Get Spanish Origin-Destination Data

github.com/rOpenSpain/spanishoddata ↗

spanishoddata spent a year finding out its 2020-2021 data was quietly incomplete.

mobility-dataorigin-destinationduckdbopen-datadata-fidelityr-package
◆Current state
spanishoddata provides access to Spain's open mobility origin-destination datasets from the Ministry of Transport, converting them into DuckDB and parquet for analysis at scale. Nearly every release in this window is a data-fidelity fix rather than a feature: district-to-municipal reaggregation was wrong for the 2020-2021 vintage, literal 'NA' strings in the source CSVs broke DuckDB enum casting, and the Amazon S3 metadata bucket turned out to be truncated at March 2021, silently hiding data.
◆Where it's heading
The package is in a trust-building phase. The pattern across 0.2.1 through 0.2.6 is the maintainers repeatedly discovering that upstream metadata and the package's own aggregation were misrepresenting what data existed, then fixing it and adding a check so it surfaces next time. That is now backed by infrastructure: comprehensive unit tests plus weekly live-data runs on GitHub workers that alert maintainers when the upstream ministry changes something. The last feature release sits outside the six-entry window, which is itself the story.
◆Prediction
Expect continued upstream-tracking fixes as the ministry's API and S3 layout shift, with the experimental quick-access and checksum functions the most likely candidates for promotion to stable.

◆Recent moves

  1. 2mo ago

    Redirected district files identified in v1 metadata

    A narrow follow-on to the 0.2.5 metadata switch, catching redirected district files and alerting users when a query matches nothing. The alert is the more useful half — silent empty results were part of what made the earlier truncation hard to spot.

    View source ↗
  2. 2mo ago

    S3 metadata truncation at March 2021 bypassed via XML feed

    The most consequential fix in the window: the Amazon S3 bucket the package read metadata from was artificially cut off at 18 March 2021, so users requesting 2020-2021 data received an incomplete picture without being told. Metadata fetching reverts to the XML feed.

    View source ↗
  3. 4mo ago

    Literal NA strings no longer break DuckDB casting

    Source CSVs carry literal 'NA' text in columns like renta, distancia, origen and destino, which failed when cast to DuckDB enums and other non-string types. Handled across both data vintages, along with better resilience to network failures.

    View source ↗
  4. 4mo ago

    Large urban area zones can be reloaded again

    A reload bug affecting gau (large urban area) zones in spod_get_zones. Single-zone-type defect with no wider effect.

    View source ↗
  5. 5mo ago

    time_slot column removed; test coverage goes live-weekly

    The deprecated time_slot column is finally dropped in favor of hour, completing a rename announced a year earlier. More durable is the testing shift: comprehensive unit coverage plus weekly live-data runs that alert maintainers when upstream changes — infrastructure built in direct response to this window's fidelity failures.

    View source ↗
  6. 1y ago

    District-to-municipal reaggregation corrected for 2020-2021

    Municipal-level figures derived from district data for the v1 vintage were being aggregated incorrectly — a defect that produced plausible but wrong numbers rather than an error. The first of several discoveries that the 2020-2021 data path needed auditing.

    View source ↗