LoginSign Up
← Back to all sparks
R

rush

INFRA · APIS
Velocity2.5

Package designed to solve large-scale problems asynchronously across a distributed network

rush.mlr-org.com ↗

Rush spent 2026 making distributed task ownership correct, starting with a breaking 1.0.

mlr3distributed-computingredisfault-toleranceworker-lifecycle
◆Current state
rush is the Redis-backed distributed computing backend behind mlr3's parallel tuning: workers pull tasks, push results and are tracked through heartbeats. Since 1.0.0 in March 2026 it has shipped five releases in four months, almost all correcting how tasks and workers are owned and identified under failure. The most recent lets each worker start in its own mirai call so it can spawn local daemons.
◆Where it's heading
The through-line is moving responsibility from the controller to the worker. 1.0.0 rewrote the task lifecycle and dropped the seed mechanism; 1.2.0 moved fail_tasks(), finish_tasks(), pop_task() and push_running_tasks() from Rush onto RushWorker so only the worker that ran a task can mark it done, and gave worker ids random suffixes so they cannot collide. 1.1.0 had already removed phantom failed tasks when a worker crashes between evaluations.
◆Prediction
With ownership and identity settled, the remaining pressure points are heartbeat and lost-worker detection; expect the next releases to tighten failure recovery rather than add API surface.

◆Recent moves

  1. 21d ago

    Workers start in their own mirai call so they can create local daemons

    Replacing the single mirai_map() dispatch with one mirai() call per worker allows nested parallelism — a worker can now spin up its own daemons. A small change with real consequences for anyone running parallel learners inside a distributed tuning run.

    View source ↗
  2. 1mo ago

    Task lifecycle moves onto the worker; worker ids can no longer collide

    The methods that mark a task failed, finished, running or popped move from Rush to RushWorker, so only the worker that processed a task changes its state. Worker ids gain random suffixes, deprecated local/remote worker types are removed, and Redis 7.0 becomes a declared system requirement.

    View source ↗
  3. 3mo ago

    Phantom failed tasks and missed results fixed

    detect_lost_workers() no longer invents failed tasks when a worker dies between evaluations, and fetch_new_tasks() tracks seen results by actual cache size rather than the Redis counter, closing a window where results could be missed or duplicated.

    View source ↗
  4. 4mo ago

    Smaller Redis payloads and vector-valued task parameters

    start_workers() strips the enclosing environment from worker_loop before serialisation, so closures stop bloating Redis payloads, and task fetching survives parameters holding vectors rather than scalars.

    View source ↗
  5. 4mo ago

    Rush 1.0 rewrites the task lifecycle and drops the seed mechanism

    ⚡ SPARK

    The release that set the shape of everything since: the result cache becomes a data.table, wait and fetch collapse into a single fetch_new_tasks() with a timeout, push_results() and push_failed() are renamed to finish_tasks() and fail_tasks(), and the RNG seed machinery is removed outright. Every subsequent release refines the ownership model this one established.

    View source ↗
  6. 9mo ago

    wait_for_workers() can target a count or specific ids

    Waiting becomes precise — a specific number of workers or a named set — and registration is checked against the network. An early step toward the worker-identity work 1.2.0 finished.

    View source ↗