LoginSign Up
← Back to all sparks
A

Apache Uniffle

ANALYTICS
Velocity0.0

Unified remote shuffle service for distributed compute engines like Spark and MapReduce

uniffle.apache.org ↗

Uniffle's remote shuffle service finally makes its fast path the default

remote-shufflesparknetty-transportpartition-skewbig-data
◆Current state
Apache Uniffle is a remote shuffle service for Spark, MapReduce and Tez. The 0.10.0 release in September 2025 flipped the Netty-based transport (GRPC_NETTY) from opt-in to the default, promoted partition reassignment to general availability, and added partition splitting for oversized shuffle partitions. Releases before that were largely stabilization work: a dashboard in 0.9.0, log and layout cleanups in 0.9.1, and pure license housekeeping in 0.9.2.
◆Where it's heading
The arc runs from 'Netty is available' in 0.8.0, to 'Netty is production ready' in 0.9.0, to 'Netty is on by default' in 0.10.0 — a three-release migration off the original gRPC transport, executed conservatively. The parallel theme is huge-partition survival: reassignment and splitting both exist to stop a single skewed partition from taking down a shuffle write. Uniffle is optimizing for the failure modes of very large Spark jobs rather than for breadth of features.
◆Prediction
The Rust shuffle server introduced experimentally in 0.9.0 is the obvious next promotion candidate, following the same available-then-default path Netty took. The release notes do not indicate a timeline.

◆Recent moves

  1. 1y ago

    Netty transport becomes the default in Uniffle 0.10

    ⚡ SPARK

    The release that completes the transport migration the last two versions were building toward: GRPC_NETTY is now on by default, partition reassignment for Spark reaches general availability, and partition splitting arrives for oversized shuffles. Remote merge extends to MapReduce and Tez, and a Spark-side UI lands for observability.

    View source ↗
  2. 1y ago

    License and NOTICE housekeeping in Uniffle 0.9.2

    Entirely licensing and metadata work — NOTICE corrections, added copyright lines, a security.md, and renaming DISCLAIMER-WIP. Compliance maintenance on the 0.9 line with no runtime change.

    View source ↗
  3. 1y ago

    Dashboard and block-ID layout cleanup in Uniffle 0.9.1

    Consolidation on the 0.9 line: dashboard refinements, human-readable timestamps, and reduced block-ID layout constraints for MapReduce and Tez. The block-ID work continues the configuration simplification 0.9.0 started for Spark, extending it to the other engines.

    View source ↗
  4. 1y ago

    v0.9.1: [MINOR] chore: Fix the issue of license loss (#2246)

    A maintenance tag on the 0.9.1 line correcting a lost license file. Packaging hygiene ahead of the broader compliance sweep that landed in 0.9.2.

    View source ↗
  5. 2y ago

    v0.9.1-rc1: Revert "Update create-package.sh (#2019)"

    A release candidate whose only content is reverting a packaging script change. Build-process churn with no user-visible effect.

    View source ↗
  6. 2y ago

    Uniffle 0.9 adds a dashboard and a Rust shuffle server

    The middle step of the transport migration: Netty is declared production-available here, one release before it becomes the default. A web dashboard and Spark 3.5 support arrive, and an experimental Rust implementation of the shuffle server appears — a second runtime that has not yet displaced the Java one.

    View source ↗