LoginSign Up
← Back to all sparks
A

Apache Arrow Rust

DEVOPS
Velocity5.0

Rust implementation of the Apache Arrow columnar in-memory format

arrow.apache.org/rust ↗

Rust Arrow ships monthly, with release-candidate commits cluttering the feed between them.

apache-arrowrustparquetcolumnarvariant-type
◆Current state
arrow-rs is the Rust implementation of Apache Arrow plus the Parquet crates that sit on top of it, and it ships a minor release roughly monthly with a maintenance branch running alongside. The feed interleaves real changelogs with release-candidate tags whose bodies are just the version-bump pull request, so half the entries carry no product information. The substantive work in this window splits between Parquet reader correctness — nested columns across page boundaries, mask filtering across skipped pages — and the ongoing Variant type implementation.
◆Where it's heading
Two things are being built out steadily. Variant support is accumulating conversion and field APIs release over release, moving toward a usable semi-structured type. Separately the crate is filling in ergonomic gaps that downstream engines hit: a sans-IO IPC stream encoder, unchecked builder methods, builder buffer capacity accessors, and stricter DataType parsing. The 58_maintenance branch takes security and encryption backports in parallel.
◆Prediction
Variant should keep gaining API surface each release until it reaches parity with the other array types; the Parquet nested-read fixes suggest that area is not yet settled either.

◆Recent moves

  1. 5d ago

    Sans-IO IPC stream encoder and Parquet mask filtering across skipped pages

    The sans-IO IPC stream encoder is the notable addition — it lets embedders drive encoding without the crate owning the I/O, which matters for async and non-standard transports. On the Parquet side, mask filtering now works across skipped pages, and arrow-avro's object_store integration is deprecated.

    View source ↗
  2. 8d ago

    Release candidate tag for 59.2.0

    A release-candidate tag whose body is the version-bump commit, noting the switch of changelog generation to git-cliff. No product content of its own.

    View source ↗
  3. 20d ago

    Maintenance branch release: more Parquet encryption algorithms, audit backports

    A 58_maintenance branch release carrying additional Parquet encryption algorithms and backported cargo audit fixes. It exists so users pinned to the 58 line get security work without taking the 59 API surface.

    View source ↗
  4. 24d ago

    Release candidate syncing the 58.4.0 changelog

    A release candidate whose entire content is a changelog sync for one previously merged pull request. Feed noise from the release process rather than a release.

    View source ↗
  5. 1mo ago

    Product aggregate kernel, CSV header validation, nested dictionary cast fast path

    A broad release: a product aggregate kernel, CSV headers validated against a schema, IPC writer compression levels, and a fast path for nested DictionaryArray casting. It also fixes reading multiple nested columns when a list continues across a DataPageV2 boundary, one of the Parquet correctness issues this window keeps returning to.

    View source ↗
  6. 1mo ago

    Release candidate tag for 59.1.0

    The release-preparation pull request tagged as a candidate — version bump and changelog update only. The actual changes ship under the 59.1.0 tag four days later.

    View source ↗