LoginSign Up
← Back to all sparks
H

hubData

ANALYTICS
Velocity0.0

Tools for accessing and working with hubverse data

github.com/hubverse-org/hubData ↗

The Arrow data layer for forecast hubs, spending its releases on cloud and materialisation bugs.

data-accessarrowcloud-storagehubverseepidemiologyr-package
◆Current state
hubData is the access layer for hubverse forecasting hubs, connecting to local and cloud-stored model output through Arrow and handing back lazy connections or materialised tibbles. Its releases divide sharply between schema and utility additions in the 1.x line and, more recently, a run of defect fixes in the cloud and Arrow integration. Two of those fixes involved data being silently wrong rather than an error being raised.
◆Where it's heading
The package has largely finished adding surface and is now paying down the cost of sitting on top of Arrow and S3: ALTREP-backed columns escaping into user sessions, cloud hubs whose declared format differs from what is actually written, and metadata arrays parsing inconsistently. Each fix narrows the gap between what the storage layer does and what an R user expects. The performance-motivated default flip in 2.0.0 points the same way, prioritising large cloud hubs over conservative local behaviour.
◆Prediction
Expect continued fixes at the Arrow and cloud boundary, particularly where declared hub configuration and actual stored format disagree, which has now produced defects twice.

◆Recent moves

  1. 1mo ago

    Cloud hubs declaring CSV no longer return an empty connection

    connect_hub() returned nothing but a no-files warning for S3 hubs whose admin.json declares a non-parquet submission format, because the hubverse cloud sync always writes model output to parquet on S3 regardless of what was declared. It now also accepts parquet on S3 when no file format is given, leaving local hubs and genuinely non-parquet cloud hubs unaffected. A clean example of configuration and storage reality diverging.

    View source ↗
  2. 3mo ago

    collect_hub() returns plain vectors instead of ALTREP views

    Returned tibbles carried arrow-ALTREP-backed columns that could not be safely saved, serialized, or sent to parallel workers in sessions without arrow installed — they came back length-zero, silently corrupting data rather than failing. Setting arrow.use_altrep to FALSE for the duration of the call restores the expected collect() contract of materialising into ordinary R vectors. The severity here is the silence, not the mechanism.

    View source ↗
  3. 3mo ago

    Array-valued metadata fields now parse as list columns

    Fixes load_model_metadata() mishandling array-valued top-level fields in model metadata YAML, which now become list columns per the specification. The side effect is worth noting: a length-one array and a scalar previously parsed identically and now do not, so metadata that round-tripped before may change type. Small in scope, but it alters how existing hub metadata reads.

    View source ↗
  4. 7mo ago

    date_col parameter for oracle output schemas

    Adds date_col to create_oracle_output_schema() and connect_target_oracle_output(), letting users name the date column in inference mode so it is typed correctly, and ignored under v6+ config where the schema already defines it. Schema creation also warns rather than errors when no date column is found. Aligns oracle output handling with the time-series equivalent.

    View source ↗
  5. 8mo ago

    connect_hub() skips file validation by default (breaking)

    Flips skip_checks from FALSE to TRUE, dropping the high-I/O file validation that made connecting to large cloud hubs slow, on the reasoning that hubs validated through hubValidations should not need it again at read time. The old behaviour remains available explicitly, and the new default matches the Python hubdata package. A breaking default change that trades a safety net for cloud performance, and a visible instance of the R and Python implementations converging.

    View source ↗
  6. 10mo ago

    Arrow schema conversion and validation utilities

    Adds as_r_schema(), arrow_schema_to_string(), is_supported_arrow_type() and validate_arrow_schema() for converting Arrow schemas to R types and checking which fields are supported. Infrastructure rather than user-facing capability, but it makes the type boundary between Arrow and R explicit and checkable — the same boundary that later produced the ALTREP defect.

    View source ↗