LoginSign Up
← Back to all sparks
H

hubUtils

ANALYTICS
Velocity0.0

Core 'hubverse' Utilities

github.com/hubverse-org/hubUtils ↗

The hubverse's shared plumbing, tracking schema versions and converting output types.

hubverseschemaepidemiologyr-packageperformance
◆Current state
hubUtils is the low-level dependency the rest of the hubverse builds on: schema version tracking, config file reading, example test hubs, and conversion between forecast output types. Its releases are small and cadenced to the hubverse schema itself, with a version bump arriving whenever the config schema advances. The recent work is performance rather than surface.
◆Where it's heading
The through-line is that this package absorbs whatever the schema is doing — v5, then v6 with target-data configuration, each arriving with matching accessors and example hubs so the sibling packages can be tested against something real. convert_output_type() is the one piece of genuine computation here, and it has now been optimised by roughly an order of magnitude, suggesting it is being used at scales the original implementation did not anticipate. Everything else is accessors and fixtures.
◆Prediction
Expect the next substantive release to track the next hubverse schema version, with any independent work concentrated on convert_output_type(), the only performance-sensitive function in the package.

◆Recent moves

  1. 1mo ago

    convert_output_type() drops an order of magnitude in time and memory

    Conversions are now computed per group of model and task ID combinations rather than materialising an n_samples by n_output_type_ids intermediate table, cutting both runtime and allocation by roughly ten times on large sample tables. Reported by a user, which indicates the function is being run at sizes the original design did not anticipate. It is the only computationally significant function in an otherwise accessor-shaped package.

    View source ↗
  2. 7mo ago

    Graceful failure when remote resources are unreachable

    Functions that fetch remote URLs now fail with informative errors instead of whatever the underlying request produced. Relevant because schema URLs are resolved over the network, so an offline session previously failed obscurely. No change in capability.

    View source ↗
  3. 9mo ago

    Accessors for target-data.json configuration

    Adds a family of getters for the v6 target-data config — date column, observable unit with dataset-specific overrides, whether target data is versioned with inheritance from global settings, and the non-task-id schema for time series. These are the primitives the sibling packages need to read v6 hubs, and they arrive shortly after the schema itself. Characteristic of this package's role as the place schema knowledge is centralised.

    View source ↗
  4. 9mo ago

    Schema v6.0.0 support and two example v6 hubs

    Adds the v6.0.0 schema and threads target-data config files through get_schema_url(), read_config(), get_version_hub() and is_v3_hub(), alongside two lightweight example hubs covering single-file and hive-partitioned target data. The version number tracks the schema more than the package's own maturity. Providing fixtures for both target-data layouts is what lets the rest of the stack develop against v6 at all.

    View source ↗
  5. 10mo ago

    Lightweight example v5 hubs added for tests

    Bundles two complete, valid v5 example hubs so examples and checks no longer need a clone of the public example hub repository. Test infrastructure with no user-facing effect, though it speeds up development across the sibling packages. The same pattern repeats a month later for v6.

    View source ↗
  6. 1y ago

    convert_output_type() introduced for sample-to-summary conversion

    Introduces convert_output_type(), converting model outputs between types — initially samples to mean, median and quantile — and extends it to handle output type IDs that depend on task-id variable values. This is the function later optimised by an order of magnitude, and the only real computation the package performs. Its arrival alongside a schema version bump is the standard shape of a hubUtils release.

    View source ↗