LoginSign Up
← Back to all sparks
H

hubEvals

ANALYTICS
Velocity2.5

Tools for Scoring and Evaluating Hubverse Model Outputs

github.com/hubverse-org/hubEvals ↗

Forecast-hub scoring that learned to handle joint, sample-based predictions.

forecast-evaluationscoringepidemiologyr-packagehubverse
◆Current state
hubEvals scores model output from collaborative forecasting hubs, wrapping scoringutils and translating hubverse formats into forecast objects it can evaluate. The package has moved quickly from a thin translation layer to something that handles every output type the hubverse defines — quantile, mean, median, nominal and ordinal pmf, and samples. The most recent releases are almost entirely about the failure modes of relative skill scoring rather than about new metrics.
◆Where it's heading
Two threads dominate. The first is coverage of output types, which reached its widest point with sample-based and compound scoring. The second, and the one occupying every recent release, is making relative skill degrade gracefully: single-model input, comparison groups with one model, and groups missing the requested baseline have each been converted from a cryptic upstream abort into a defined result. That pattern — inherited scoringutils errors being caught and given hub-specific meaning — is the clearest signal of where this package adds value.
◆Prediction
Expect continued work smoothing scoringutils error surfaces into hub-aware behaviour, and performance attention on relative skill, which was explicitly optimised in the latest release.

◆Recent moves

  1. 24d ago

    Scored-forecast counts and faster relative skill

    Adds an include_count argument that reports how many forecasts went into each summary group, which matters when comparing models evaluated over different amounts of data. Relative skill computation is faster and no longer emits scoringutils warnings about exact p-values with ties. Both items continue the recent focus on the relative-skill path rather than on new metrics.

    View source ↗
  2. 1mo ago

    Disaggregated relative skill no longer aborts the whole call

    Where some comparison groups cannot be compared, scoring now resolves per group instead of failing everything with a cryptic baseline-missing error: single-model groups get relative skill 1, and groups missing the requested baseline report NA with a warning naming them, while absolute scores survive. This is the third consecutive release addressing the same class of problem. It reflects a package whose main job is turning upstream statistical errors into results a hub maintainer can act on.

    View source ↗
  3. 1mo ago

    Single-model scoring returns relative skill of 1 instead of erroring

    Requesting relative metrics with only one model previously failed through scoringutils with a not-enough-comparators error; the relative-skill columns are now filled with 1, and supplying a mismatched baseline gives a clear message instead. A getting-started vignette covering every supported output type ships alongside. The vignette is the more telling item — it marks the point where the output-type coverage was complete enough to document as a whole.

    View source ↗
  4. 5mo ago

    Sample output types and multivariate compound scoring

    ⚡ SPARK

    The release that widened what hubEvals can evaluate at all: sample-based forecasts become scorable, and compound scoring introduces genuinely multivariate metrics over joint predictions rather than per-target ones. Everything after this release refines the relative-skill machinery rather than adding output types, which makes this the point where the package's coverage was finished. It also sets up the transform work that hubPredEvalsData would later expose through configuration.

    View source ↗
  5. 6mo ago

    Scoring on transformed scales via transform arguments

    Adds transform, transform_append and transform_label to score_model_out(), so scores can be computed on log or square-root scales for quantile, mean and median output types. Relevant wherever forecast errors are multiplicative rather than additive, as in epidemic growth. This is the origin of a capability that later surfaced as declarative configuration in hubPredEvalsData.

    View source ↗
  6. 11mo ago

    First release: score_model_out() and the scoringutils bridge

    The initial versioned release, exporting the transform_*_model_out() functions that convert hubverse model output into scoringutils forecast objects, alongside score_model_out() itself. Dependencies move to CRAN releases where available. Everything the package became is visible in outline here: a translation layer between hub formats and an established scoring engine.

    View source ↗