LoginSign Up
← Back to all sparks
P

probably

ANALYTICS
Velocity0.0

Tools for post-processing class probabilities and calibrating R model predictions

github.com/tidymodels/probably ↗

The package that made calibration a step instead of an afterthought.

calibrationconformal-inferencetidymodelsuncertaintydeploymentprobability
◆Current state
probably started as a small utility for class predictions and equivocal zones, and version 1.0.0 turned it into tidymodels' calibration and uncertainty package: cal_plot_*, cal_estimate_*, cal_validate_* and cal_apply across binary, multiclass and regression problems, plus conformal prediction intervals. Since then the work has been consolidation — a large internal refactor with no API change, split conformal and conformal quantile regression, bound_prediction(), and required_pkgs() and butcher methods so conformal objects can be deployed and stripped.
◆Where it's heading
The recent releases are about making these objects survive leaving the session. butcher and required_pkgs() methods are what a model needs to be pinned, containerised and served, and their arrival alongside workflows adding a tailor postprocessing stage and vetiver adding probably support points the same way: calibration is being moved out of analysis scripts and into the deployed pipeline. The cal_*_none() reference implementations are the tell that calibration is now something people tune rather than apply once.
◆Prediction
Expect the calibration functions to be reachable directly from a tuned workflow's postprocessing stage rather than applied to predictions afterwards, following the tailor integration that workflows just shipped.

◆Recent moves

  1. 10mo ago

    Conformal objects gain required_pkgs() and butcher methods

    Adds required_pkgs() and butcher methods to all four conformal interval functions. Unglamorous plumbing with a clear purpose: an object that cannot declare its dependencies or shed its stored environment is an object you cannot deploy.

    View source ↗
  2. 1y ago

    ggplot2 test updates and a clearer validation-set error

    Unit tests updated for a new ggplot2 release and a clearer error when cal_validate_*() is given a validation set. Maintenance against an upstream release.

    View source ↗
  3. 1y ago

    Calibration internals refactored; isotonic bootstrap bug fixed

    A large internal refactor of the calibration code with no user-facing API change, plus cal_*_none() no-op calibrators to serve as a tuning reference, bound_prediction() for constraining numeric predictions, and a fix where cal_estimate_isotonic() always used a bootstrap sample. That last one silently changed results for anyone using isotonic calibration.

    View source ↗
  4. 2y ago

    Fix grouping sensitivity to variable type

    One bug fix where calibration grouping behaved differently depending on the type of the grouping variables. Narrow correctness fix.

    View source ↗
  5. 3y ago

    Split conformal and conformal quantile regression added

    Adds split conformal inference and conformal quantile regression, renames the int_conformal_infer_*() functions three weeks after introducing them, and centres cross-validated intervals on the averaged resampled prediction. Split conformal is the cheap option that makes conformal intervals practical on large data.

    View source ↗
  6. 3y ago

    Calibration and conformal inference arrive in tidymodels

    ⚡ SPARK

    The release that made probably a calibration package: a full estimate, validate, plot and apply workflow across binary, multiclass and regression problems, plus conformal prediction intervals. Everything since has been hardening what this release established.

    View source ↗