LoginSign Up
← Back to all sparks
P

PyTables

DEVOPS
Velocity0.0

Hierarchical dataset management in Python

github.com/PyTables/PyTables ↗

PyTables opened a path around HDF5's filter pipeline, then chased Python's runtime.

hdf5chunkingfree-threadingnumpypython
◆Current state
PyTables is at 3.11.1, a one-line blosc2 loading fix. The structural change in the window is 3.10.0's direct chunking API, which lets callers read and write raw chunk data without going through the HDF5 filter pipeline, funded by a NumFOCUS grant. Since then the work has been runtime currency: NumPy 2, Python 3.13 and 3.14, free-threading compatibility and abi3 wheels.
◆Where it's heading
Two threads, both about overhead. The direct chunking API removes the filter pipeline from the hot path for callers who already know their compression; free-threading compatibility and threadsafe HDF5 wheels remove locking from concurrent reads. PyTables is positioning as the low-overhead route to HDF5 rather than competing on features with the format itself.
◆Prediction
With the free-threading directive set and abi3 wheels shipping, the next release most likely consolidates that threading story — the notes already point readers to a separate threading cookbook — rather than extending the chunking API.

◆Recent moves

  1. 5mo ago

    Fixes blosc2 loading

    A single fix to blosc2 loading. It repairs an install-time problem and changes nothing about the API.

    View source ↗
  2. 5mo ago

    Python 3.14, free-threading compatibility and abi3 wheels

    Python 3.14 support arrives with the Cython free-threading directive set and wheels built against the stable ABI. It is the runtime half of the low-overhead thread the direct chunking API started.

    View source ↗
  3. 1y ago

    Python 3.13 wheels, multi-dimensional chunkshape, dtype descriptions

    Python 3.13 wheels, multi-dimensional chunkshape when creating arrays, and table descriptions accepted as IsDescription, dict or numpy dtype. A full PEP 8 reformat with linting in CI comes alongside.

    View source ↗
  4. 1y ago

    Fixes NumPy version constraint blocking NumPy 2

    Corrects a NumPy version constraint that wrongly blocked NumPy 2.x, plus a PyPy compatibility fix. A packaging repair on top of 3.10.0.

    View source ↗
  5. 1y ago

    Direct chunking API bypasses the HDF5 filter pipeline

    ⚡ SPARK

    The release that changed what PyTables is for: raw chunk access that skips the HDF5 filter pipeline entirely. NumPy 2 compatibility and type hints across the API land in the same version.

    View source ↗
  6. 2y ago

    Threadsafe HDF5 wheels; HDF5 1.8 API support dropped

    Wheels move to a threadsafe HDF5 build, so concurrent reads no longer need external locking or monkey-patching. Compatibility with the obsolete HDF5 1.8 API is dropped, raising the floor to 1.10.

    View source ↗