LoginSign Up
← Back to all sparks
S

scoringutils

ANALYTICS
Velocity0.0

Utilities for Scoring and Assessing Predictions

github.com/epiforecasts/scoringutils ↗

scoringutils pushes forecast scoring past univariate outcomes into multivariate and ordinal ones.

forecast evaluationprobabilistic scoringmultivariate forecastss3 classesepidemiology
◆Current state
scoringutils evaluates probabilistic forecasts in R. Since the 2.0.0 rewrite it is organised around typed forecast objects — quantile, sample, binary, point, nominal — built by as_forecast_<type>() constructors and scored through S3 methods. Version 2.2.0 adds multivariate sample and point types with the variogram score, and 2.1.0 added ordinal forecasts.
◆Where it's heading
The forecast-type system introduced in 2.0.0 is the engine of everything since: each release fits another outcome shape into it rather than reworking the scoring interface. Multivariate support is the largest of those additions because it scores the dependence structure between variables, not just marginal accuracy. Type and constructor names are still being reconciled — forecast_sample_multivariate was renamed to forecast_multivariate_sample with a deprecation window.
◆Prediction
Expect further forecast types and metrics slotted into the same constructor pattern, and the deprecated forecast_sample_multivariate alias and is_forecast_sample_multivariate() to be removed once that window closes.

◆Recent moves

  1. 4mo ago

    Multivariate forecast scoring and the variogram score

    ⚡ SPARK

    The typed-forecast system takes on its first genuinely multi-dimensional outcomes, with the variogram score as the default metric for multivariate samples. Naming inconsistencies from the feature's introduction are cleaned up in the same release.

    View source ↗
  2. 11mo ago

    Quantile levels rounded to avoid float duplicates

    A patch handling quantile levels that differ only by floating-point noise, plus ggplot2 label compliance in example plots.

    View source ↗
  3. 1y ago

    Optional p-values in pairwise comparisons; PIT fix

    Pairwise comparison gains a test_type = NULL escape from p-value computation, and a bug that silently forced discrete PIT histograms on continuous data is fixed. The R floor moves to 4.1.0 under dependency pressure.

    View source ↗
  4. 1y ago

    Ordinal forecasts get their own class and metrics

    Ordinal outcomes become a first-class forecast type with matching metrics, and conversion methods now exist between forecast types. This is the 2.0.0 class system doing what it was built for rather than a change of direction.

    View source ↗
  5. 1y ago

    Rewrite: typed forecast objects and pluggable metrics

    ⚡ SPARK

    The rewrite every later release builds on — typed forecast objects, as_forecast_<type>() validation constructors, S3 scoring methods and user-supplied scoring functions — at the cost of breaking the 1.x interface outright.

    View source ↗
  6. 2y ago

    Two bug fixes and package-site infrastructure

    A maintenance release before the rewrite, notable mainly for the startup message warning users that breaking changes were coming.

    View source ↗