LoginSign Up
← Back to all sparks
C

cvms

INFRA · APIS
Velocity0.0

Cross-Validation for Model Selection

github.com/ludvigolsen/cvms ↗

A cross-validation package whose real development has moved to its plotting function

cross-validationr-packagemodel-evaluationvisualizationconfusion-matrix
◆Current state
cvms runs repeated cross-validation over model formulas and reports comparable metrics. The 2.0.0 release was a breaking correctness fix: every function accepting fold_cols mismatched training and testing data when fold indices were non-sequential, did not start at 1, or were strings, because the iteration index was compared against the raw fold value rather than its factor level index. 2.0.1 restored coefficient extraction for nnet::multinom and mixed models by supplying an environment containing the training data, and followed lme4's move of findbars() into the reformulas package.
◆Where it's heading
Two threads run in parallel and only one is about cross-validation. The plotting function plot_confusion_matrix() has absorbed most feature work since 1.5.0 - custom gradient palettes, intensity limits, per-tile settings, dynamic font colors keyed to value thresholds, and arguments that accept functions rather than constants - to the point where a companion web application exists for using it without code. The cross-validation core, by contrast, sees maintenance: upstream compatibility fixes for pROC, ggnewscale and ggplot2, and the fold-matching correction that finally forced a major version.
◆Prediction
Expect continued option growth in the confusion matrix plotting surface, since that is where nearly every release since 1.5.0 has spent its changes, with core cross-validation changes arriving only as upstream packages force them.

◆Recent moves

  1. 2mo ago

    Coefficient extraction restored for multinom and mixed models

    Extracts model coefficients with an environment carrying the training data, which restores nnet::multinom and mixed-model coefficient extraction where the parameters and insight backends need the original data. Also follows findbars() out of lme4 into the reformulas package, a dependency shuffle the wider mixed-model ecosystem is working through.

    View source ↗
  2. 9mo ago

    Breaking fix for mismatched folds with non-sequential fold IDs

    The first major version bump in the package's recent history, and it is a correctness fix rather than a feature: non-sequential or string fold IDs were matched against raw values instead of factor level indices, so training and testing splits could be drawn from the wrong folds. The maintainer scopes the blast radius narrowly to users who resampled already-folded data, which is why this reads as a serious but contained repair.

    View source ↗
  3. 11mo ago

    Test compatibility with pROC 1.19 and a deprecation warning fix

    Removes an over-specified test of pROC curve contents and clears a deprecation warning when add_sums is enabled without custom sum settings. Upstream housekeeping, contributed largely by the affected package maintainers.

    View source ↗
  4. 1y ago

    Confusion matrix fonts and colors can now be computed from the values

    The clearest example of where this package's effort goes: font and arrow colors can switch above and below a threshold via dynamic_font_colors, and most font() arguments now accept a function that decides the setting from the underlying counts. Presentation logic in the plotting layer, not model evaluation.

    View source ↗
  5. 1y ago

    Tile intensity by row or column percentages

    Adds row_percentages and col_percentages as intensity bases for the main tiles, useful when class imbalance makes raw counts uninformative. Also fixes an error combining filtered input with sum tiles or diagonal-only percentages.

    View source ↗
  6. 1y ago

    Multinom coefficient extraction fix after a parameters update

    A single upstream-tracking fix for nnet::multinom coefficient extraction. Notably the same area broke again in 2.0.1 for a different reason, which suggests this integration point stays fragile.

    View source ↗