Shynet vs Polars

Polars ships on two trains: the Python package, now at 1.43.2, and the Rust crate at 0.55.1 whose DSL is pinned to a matching Python version. Recent work concentrates in three places — the streaming engine, stabilized in the Rust 0.54.4 release and given out-of-core spilling in Python 1.42.0; the query optimizer, with predicate canonicalization, contradictory-filter elimination and nested common subplan elimination; and lakehouse table formats, where Iceberg, Delta and hive-partitioned layouts get dedicated join rewrites and scan parallelism. A steady deprecation wave runs alongside, mostly narrowing which casts the Categorical and Enum types permit.

The engine work is pushing Polars past the fits-in-memory, single-machine dataframe it became known for. Spilling and a stabilized streaming engine chip at the memory ceiling; the cloud IO changes — global DNS cache, bytes-based concurrency control, non-blocking path expansion — target remote object storage rather than local files; and the hive, Iceberg and Delta join rewrites only pay off when reading a partitioned lake. The deprecations run the opposite direction, tightening a type system that had been permissive about casts.

The accumulating deprecations around categorical casts, list casts and integer-boolean bitwise ops, several already emitting FutureWarnings, point toward a breaking major release that removes them. On the engine side, the explicitly naive out-of-core spilling is the obvious next thing to be reworked.