LoginSign Up
← Back to all sparks
H

hubPredEvalsData

ANALYTICS
Velocity0.0

Data Creation for Hub Prediction Evaluations via predevals

github.com/hubverse-org/hubPredEvalsData ↗

The pipeline turning hub forecasts into dashboard-ready evaluation data.

forecast-evaluationhubverseconfigurationdashboardsepidemiology
◆Current state
hubPredEvalsData generates the scored evaluation data that hubverse prediction dashboards read, driven by a predevals-config.yml and scoring through hubEvals underneath. It is the youngest package in this part of the stack and the fastest-moving in configuration terms, having already passed a breaking 1.0.0 and a schema-versioned feature addition. Its output contract is a scores.csv file consumed downstream, which shapes what its releases care about.
◆Where it's heading
Each release widens what the config file can express — round selection, then scale transformations with per-target overrides, then target labelling pulled from the hub's own task metadata. The pattern is consistent: capability that already exists in hubEvals gets a declarative surface here so hub maintainers configure it rather than write code. Recent attention to byte-stable output ordering shows the file is being treated as a reproducible artifact, not just a report.
◆Prediction
Expect the config schema to keep absorbing hubEvals capabilities as declarative options, with continued attention to making scores.csv reproducible and diffable between runs.

◆Recent moves

  1. 1mo ago

    Target names pulled from hub metadata; scores.csv made byte-stable

    Options generation now pulls target_name and target_units from the hub's own tasks.json target metadata, so dashboard menus label targets from a single source rather than a duplicated one. Separately, scores.csv rows are sorted on a fixed key immediately before writing, making the file byte-identical across runs regardless of arrow's scan order. The second item matters more than it sounds: it turns the output into something you can diff or checksum to confirm a rerun changed nothing.

    View source ↗
  2. 2mo ago

    Versioned target data with an as_of column no longer fails scoring

    Oracle output carrying an as_of provenance column caused generate_eval_data() to fail, because hubEvals rejects columns outside the task-id keys plus oracle_value. Since oracle output holds one value per observational unit, the column is now dropped before scoring. A narrow compatibility fix at the boundary between versioned target data and the scoring layer.

    View source ↗
  3. 2mo ago

    Scale transformations become configurable per target

    Adds transform support to the config schema — a global default plus per-target overrides that can also opt a target out entirely — covering log_shift, sqrt, log1p, log, log10 and log2, with transformed metrics emitted in a wide-format scores.csv. This is the declarative surface for the transform capability hubEvals gained earlier, which is the recurring division of labour between the two packages. Hub maintainers now choose scales in configuration rather than in code.

    View source ↗
  4. 4mo ago

    Multi-round hub support via a required rounds_idx property (breaking)

    Extends the package to hubs running multiple rounds, requiring config files to move to schema version 1.0.1 and declare which round to use by index. Breaking for every existing config, with migration steps spelled out in the notes. The version number marks a real contract change rather than a maturity claim, and it establishes the schema-versioning discipline the later transform release reuses.

    View source ↗