LoginSign Up
← Back to all sparks
M

mlr3benchmark

AI-ASSISTANTS
Velocity0.0

Analysis and tools for benchmarking in mlr3 and beyond.

github.com/mlr-org/mlr3benchmark ↗

A small mlr3 add-on for comparing learners, spending most releases making its statistics honest.

benchmarkingmachine-learningstatistical-testingmlr3visualization
◆Current state
mlr3benchmark handles the statistical end of the mlr3 ecosystem: aggregating benchmark results into BenchmarkAggr objects, running Friedman and post-hoc tests across them, and drawing critical difference plots. The four visible releases span two years and are dominated by correctness work on those tests and plots rather than new comparison methods. The package changed maintainer at 0.1.4 and has not shipped since.
◆Where it's heading
The arc is a package tightening the gap between what its plots show and what its tests actually support. Overlapping bars in CD plots were producing misleading comparisons in 0.1.1; construction was loosened so column naming stopped being rigid; then 0.1.2 tightened the other way, requiring factors rather than silently coercing them. By 0.1.4 the friedman_global escape hatch lets users proceed past a non-significant global test deliberately rather than being blocked by it.
◆Prediction
The maintainer handover at 0.1.4 with no release since is the clearest signal in these entries, and it points to continuity work rather than expansion. Nothing here indicates which additional post-hoc tests, if any, are planned.

◆Recent moves

  1. 3y ago

    friedman_global lets post-hoc tests run past a failed global test

    Adds a friedman_global argument so post-hoc tests and autoplots can run even when the global Friedman test fails to reject the null. That is a deliberate loosening of a statistical guardrail, handed to the user as an explicit choice. The release also records a maintainer change to Sebastian Fischer.

    View source ↗
  2. 4y ago

    PMCMRplus compatibility fix

    A README correction and a compatibility fix for PMCMRplus, the package supplying the post-hoc tests. No user-facing change beyond continuing to work against its dependency.

    View source ↗
  3. 5y ago

    BenchmarkAggr now requires factor columns; critical construction fix

    A critical patch for a bug in BenchmarkAggr construction, paired with a breaking requirement that task and learner columns arrive as factors with no internal coercion. Trading implicit convenience for predictable typing, and continuing the CD plot fixes started in 0.1.1.

    View source ↗
  4. 5y ago

    Overlapping CD-plot bars fixed; flexible BenchmarkAggr construction

    Fixes CD plots whose overlapping bars had been giving misleading results, which is the correctness problem the later releases keep circling. Also relaxes BenchmarkAggr construction so columns need not be named task_id and learner_id, returns the full test object from friedman_test in the single-measure case, and adds a $subset method.

    View source ↗