LoginSign Up
← Back to all sparks
M

MachineShop

INFRA · APIS
Velocity0.0

Machine Learning Models and Tools

github.com/brian-j-smith/machineshop ↗

A mature R modelling framework refining variable importance and resampling controls

machine-learningr-packagemodel-frameworkvariable-importanceresampling
◆Current state
MachineShop provides a unified interface over a wide set of R model packages, handling fitting, resampling, performance metrics and variable importance behind one API. Recent releases are narrow and mostly corrective: 3.9.2 removed dead documentation links and fixed a Java parameter in a BART example, 3.9.1 ensured global settings reach compute nodes when varimp() runs in parallel and patched XGBoost model compatibility. The last release with real surface change was 3.9.0, which added offset support to XGBModel and a pool argument to calibration() controlling whether calibration curves are computed on pooled predictions or averaged across resampling iterations.
◆Where it's heading
Development has concentrated on variable importance and resampling rather than on adding models. 3.8.0 restructured the VariableImportance class to record which method and metric produced it, with an update() method to migrate objects from earlier versions, and extended term-specific p-values to Cox, POLR and survival regression models. 3.7.0 added grouped and stratified resampling to the control objects. The pace has slowed markedly - four releases in the last two years against six in the two before - and the recent content is compatibility work against XGBoost, parsnip, ggplot2 and recipes.
◆Prediction
Expect the deprecated calibration pooling behaviour to be removed in a future release as the notes state, with the intervening versions continuing to track upstream model package changes.

◆Recent moves

  1. 5mo ago

    Documentation link cleanup and a BART example fix

    Removes dead Rd cross-references to ParBayesianOptimization and sets the Java parameters an example needed. CRAN housekeeping with no effect on modelling behaviour.

    View source ↗
  2. 8mo ago

    Global settings now reach compute nodes during parallel varimp

    Fixes parallel variable importance runs where compute nodes did not inherit the session's global settings, meaning results could differ from a serial run of the same code. Paired with XGBoost compatibility fixes across the DART, linear and tree variants.

    View source ↗
  3. 1y ago

    Offset support for XGBoost and per-iteration calibration curves

    Adds offsets to XGBModel and introduces a pool argument letting calibration() average per-iteration curves rather than pooling all resampled predictions, with the older behaviour deprecated. The note also warns that pooling can exhaust memory when fitting smooth curves without breaks, which is likely why the default changed.

    View source ↗
  4. 1y ago

    Variable importance objects record their own method and metric

    Adds method and metric slots to the VariableImportance class so a result carries whether it came from permutation or the model itself and under which metric, with an update() method for objects made by older versions. Term-specific p-values via drop1() extend to Cox, POLR and survival regression, aligning them with the linear and generalized linear cases.

    View source ↗
  5. 2y ago

    Grouped and stratified resampling in the control objects

    Enables resampling by a grouping variable across BootControl, OOBControl and SplitControl, plus stratified splitting - the controls needed to keep clustered observations from spanning training and test sets. Also raises the R requirement to 4.1.0.

    View source ↗
  6. 3y ago

    Backward compatibility for older model objects

    Handles MLModel objects saved before the na.rm slot existed, clears an S3 generic consistency warning, and relaxes the requirement that role variables appear in prediction data. Maintenance for users with serialized models from earlier versions.

    View source ↗