LoginSign Up
← Back to all sparks
J

joblib

DEVOPS
Velocity0.0

Lightweight pipelining and caching for Python

github.com/joblib/joblib ↗

The library behind scikit-learn's n_jobs is adding streaming and async caching.

parallelismcachingasyncscikit-learnpython
◆Current state
joblib is at 1.4.0, the layer scikit-learn and much of scientific Python lean on for process-level parallelism and disk memoization. That release added an unordered generator return mode, vendored cloudpickle 3.0.0, dropped Python 3.7 and extended caching to coroutine functions. The two releases before it were pure bug fixes, one of them just a vendored loky bump.
◆Where it's heading
The direction is toward returning results as they finish rather than in submission order, and toward covering async code that the original synchronous design ignored. Both changes serve callers who want throughput from long, uneven workloads instead of a single blocking join.
◆Prediction
Given the generator work and the coroutine caching in 1.4.0, the next release is most likely to extend or stabilize those async and streaming paths rather than change the Parallel API itself.

◆Recent moves

  1. 2y ago

    Unordered generator results and coroutine caching

    Adds an unordered generator return mode so results surface as workers finish, and extends Memory caching to coroutine functions. Vendoring cloudpickle 3.0.0 and dropping Python 3.7 clear the way for that async work.

    View source ↗
  2. 3y ago

    Bug fixes: n_jobs default and Parallel logger

    A bug-fix release: n_jobs=None now behaves as the documented default and the Parallel logger initializes its parent class. Housekeeping ahead of the 1.4.0 feature work.

    View source ↗
  3. 3y ago

    Patch: vendors loky 3.4.1 for compatibility

    A single-purpose patch that vendors loky 3.4.1 for compatibility. No user-facing change beyond keeping the process backend working.

    View source ↗