LoginSign Up
← Back to all sparks
P

Polars

ANALYTICS
Velocity5.0

High-performance DataFrame library for analytics

pola.rs ↗

The streaming engine is stable and the API is being narrowed — Polars is clearing ground for a breaking release

dataframesstreaming-enginedeprecationscloud-ioperformancerust
◆Current state
Polars publishes two trains into one feed: Python releases roughly weekly through 1.42.0 to 1.43.2, and Rust releases on their own numbering, with 0.54.4 carrying the milestone that the streaming engine is stabilized. The dominant thread across the Python releases is deprecation — casts from string to temporal types, numeric-to-categorical and categorical-to-integer casts, casts from non-nested dtypes into lists, bitwise ops between integers and booleans, cat.get_categories(), cat.to_local(), LazyFrame.profile(), and to_struct() calls without field names. Alongside it, cloud IO keeps getting attention: bytes-based concurrency control, callback sinks on cloud, and non-blocking path expansion.
◆Where it's heading
A deprecation batch this size is not routine tidying — it is the removal list for a future major, and the common theme is closing implicit conversions that silently change semantics. The performance and correctness work points the same way, toward the streaming engine as the default execution path rather than a mode: nested common subplan elimination, streaming grouped AsOf joins, hand-written Thrift for parquet metadata decode, and repeated fixes to sortedness and chunking on the streaming path. Cloud is the third leg, with the engine being taught to run against object storage without materializing.
◆Prediction
With the streaming engine marked stable and this many APIs deprecated in a single wave, the deprecations are the visible countdown to a release that removes them. The entries do not say when, so the safer read is that the next Python releases keep pairing streaming-path fixes with further deprecation notices rather than breaking anything yet.

◆Recent moves

  1. 7h ago

    Polars 1.43.2: scan_csv schema inference, more casts deprecated

    Adds infer_schema_files to scan_csv and pushes len() down to concat and union inputs, while deprecating categorical-to-integer casts and unset plan_stage in show_graph(). Another instalment of the same pattern: a small capability addition wrapped in continued API narrowing.

    View source ↗
  2. 5d ago

    Polars 1.43.1: callback sinks on cloud, streaming and lakehouse scan fixes

    Enables callback sinks on cloud and fixes several streaming-path defects — a self-join panic on scan_delta and scan_iceberg, undefined behaviour in first/last_non_null on empty chunks, and sortedness propagation through Gather. The cloud and streaming threads that run through this whole window, in one patch.

    View source ↗
  3. 11d ago

    Polars 1.43.0 deprecates categorical casts, profile() and implicit conversions

    The largest deprecation batch in the window: numeric-to-categorical casts, cat.get_categories(), cat.to_local(), LazyFrame.profile(), casts from non-nested dtypes into lists, integer-boolean bitwise ops, and to_struct() without field names. Nothing breaks yet, but this is the release that names what is going away.

    View source ↗
  4. 1mo ago

    Polars 1.42.1: parquet metadata sampling and IO tweaks

    A patch adding sampled resolve for multi-file parquet metadata, a faster small-dtype sum kernel, and non-blocking path expansion, plus replacement of pl.concat's strict parameter. Incremental performance work on paths the streaming engine leans on.

    View source ↗
  5. 1mo ago

    Polars 1.42.0: cloud IO concurrency control and streaming throughput

    Focused on cloud and streaming throughput: bytes-based concurrency control for cloud IO, no multiplexer flush on phase change, cheaper rechunking, faster struct unnest inside list eval on streaming, and elimination of contradictory-predicate filters. Also starts the cast deprecations that 1.43.0 expands.

    View source ↗
  6. 1mo ago

    Rust Polars 0.54.4 stabilizes the streaming engine

    ⚡ SPARK

    The Rust release that marks the streaming engine as stabilized, alongside nested common subplan elimination, streaming grouped AsOf joins, LazyFrame.gather and a hand-written Thrift parquet metadata decoder. Every streaming fix in the Python releases since reads as follow-through on this.

    View source ↗