LoginSign Up
← Back to all sparks
A

Apache TsFile

ANALYTICS
Velocity2.5

Columnar storage file format designed for time-series data in IoT scenarios

tsfile.apache.org ↗

TsFile is quietly rebuilding itself as an Arrow-speaking interchange format

time-seriescolumnar-formatapache-arrowpython-bindingsiotdb
◆Current state
Apache TsFile is the columnar time-series file format underlying IoTDB, maintained as three parallel implementations in Java, C++ and Python. Recent releases have concentrated on the C++ and Python ends: SIMD paths and parallel reads in 2.4.0, an Arrow-compatible result path from C++ through to Python DataFrames in 2.3.0, and conversion scripts from CSV, Parquet and Arrow into TsFile in 2.3.1. The Java side gets steadier, smaller work — serialized-size calculation, schema modification during writes, encryption configuration.
◆Where it's heading
The centre of gravity has moved from format features to ecosystem reach. Arrow-backed DataFrames and format converters are not about storing time series better; they are about making TsFile readable by the Python analytics stack without a translation layer, which is the gap that keeps a specialized format confined to its own database. The C++ performance work in 2.4.0 serves the same end, since the Python bindings sit on top of it. Version numbering runs on two lines at once, with 1.1.x backports still shipping alongside the 2.x series.
◆Prediction
Given the direction of the Arrow work, the Python interface is the most likely target for further capability rather than the Java one. The notes do not indicate when the 1.1 maintenance line ends.

◆Recent moves

  1. 19d ago

    SIMD and parallel read paths in Apache TsFile 2.4.0

    The heaviest C++ performance release in the window: batch codec APIs, single-pass TsBlock decoding, AVX2 and NEON SIMD paths, chunk-level parallel reads and column-parallel writes, plus faster metadata and tag lookups. On the Python side TsFileDataFrame extends to tree-model files, continuing the DataFrame reach that 2.3.0 established.

    View source ↗
  2. 2mo ago

    CSV, Parquet and Arrow conversion scripts for TsFile 2.3.1

    Adds tooling to convert CSV, Parquet and Arrow data into TsFile, with follow-up fixes for date and timestamp types, uppercase column names and non-ASCII table names. Together with the Arrow read path in 2.3.0, this closes the loop on moving data both directions between TsFile and the mainstream columnar ecosystem.

    View source ↗
  3. 3mo ago

    Arrow-backed DataFrames and paginated reads in TsFile 2.3.0

    The release that makes the Python path fast rather than merely present: the C++ layer returns Arrow-compatible results so the Python binding can hand back Arrow-backed DataFrames directly. Alongside it come row-number queries with limit and offset, nested-directory traversal, and C++ interfaces for series-level timestamp ranges and for salvaging data from corrupted files.

    View source ↗
  4. 3mo ago

    Write-time schema changes and read/write encryption in TsFile 2.2.1

    A Java-weighted release: schema can be modified mid-write, memory usage can be estimated for Tablets, I/O sizes are recorded during reads, and encryption becomes configurable for both directions. Python gains DataFrame-to-TsFile conversion and the newer date and timestamp types, the write-side counterpart to the read work in 2.3.0.

    View source ↗
  5. 7mo ago

    Python text types and C++ tag filtering in TsFile 2.2.0

    Fills in parity gaps across the three implementations: full TEXT and STRING support plus a to_dataframe method in Python, RLE and CAMEL encodings and TEXT writes in C++, and tag filtering in both the C++ and Java table models. Most of the accompanying fixes address null and empty-column handling in the C++ writer.

    View source ↗
  6. 7mo ago

    Backport maintenance on the TsFile 1.1 line

    A backport-only patch to the dev/1.1 branch — memory-size calculation, bit-manipulation helpers, iterator return types and assorted fixes, each cherry-picked from mainline. Housekeeping for users who have not moved to the 2.x series.

    View source ↗