LoginSign Up
← Back to all sparks
N

nanoparquet

ANALYTICS
Velocity0.0

R package to read and write Parquet files. Contribute to r-lib/nanoparquet development by creating an account on GitHub.

github.com/r-lib/nanoparquet ↗

nanoparquet is chasing byte-level agreement with the Java and Rust Parquet readers, not feature count.

parquetr-languageinteroperabilitydata-formatstype-mapping
◆Current state
nanoparquet reads and writes Parquet from R with no Arrow dependency, which is its entire reason to exist. The 0.4.0 line renamed the reader API and added schema authoring plus `append_parquet()`, and the 0.5.x releases have gone after interoperability: definition and repetition level encodings the Apache Parquet Java library expects, flatbuffer alignment the Rust arrow-rs reader expects, 128-bit decimals, and Polars-written files that omit the dictionary page offset. The newest release adds `bit64::integer64` columns and writing to stdout.
◆Where it's heading
Almost every entry since 0.4.0 names another engine — Java, arrow-rs, Polars, Arrow schema metadata — which tells you the maintainers are treating cross-reader fidelity as the product rather than R-side ergonomics. The type system is filling in from the edges: DECIMAL beyond 8 bytes, UUID, FLOAT16 and INTERVAL as raw lists, and now 64-bit integers with an explicit read-type option instead of a silent cast to double. Writing to `:stdout:` points at a second audience, shell pipelines rather than interactive R.
◆Prediction
The remaining unmapped Parquet types the changelog has been parking in raw-vector lists — FLOAT16 and INTERVAL — are the obvious next targets, following the same pattern by which DECIMAL and UUID graduated to real R types.

◆Recent moves

  1. 3mo ago

    64-bit integer columns and writing Parquet to stdout

    Two additions that widen where nanoparquet fits. `bit64::integer64` columns now round-trip as INT64 with a `read_int64_type` option to control the read side, closing a precision gap that the default double conversion left open. Writing to `:stdout:` makes a one-line `Rscript` invocation into a shell-pipeline Parquet writer.

    View source ↗
  2. 4mo ago

    Files now readable by the Java and Rust Parquet libraries

    The clearest statement of the current priority: data page headers now set RLE level encodings for the Apache Parquet Java library, and ARROW:schema metadata gets correct flatbuffer alignment for the Rust arrow-rs reader. Files nanoparquet wrote were previously being rejected downstream — that is a worse failure than a missing feature, and this release closes both cases.

    View source ↗
  3. 1y ago

    Reads Polars files that omit the dictionary page offset

    Another writer-specific accommodation — Polars produces files without the dictionary page offset in their metadata, and nanoparquet now handles them. Small in diff, but it extends the set of real-world files the reader accepts, which is the axis this package competes on.

    View source ↗
  4. 1y ago

    Date, FLOAT, and mixed-encoding read fixes

    A pure bug-fix release covering double `Date` column conversion, FLOAT columns spanning multiple row groups, and column chunks that mix dictionary-encoded and plain pages. Narrow edge cases rather than a change in what the package can do.

    View source ↗
  5. 1y ago

    Schema authoring and append_parquet arrive with a renamed API

    ⚡ SPARK

    The release that turned a reader with a writer attached into a full Parquet toolkit. Schema construction from scratch means users control type mapping instead of accepting inference, and `append_parquet()` makes existing files extendable. The coordinated `read_*` rename set the naming convention every release since has followed.

    View source ↗
  6. 1y ago

    Fixes a write_parquet crash

    A single-issue patch release fixing a crash in `write_parquet()`. No user-visible change beyond the failure no longer happening.

    View source ↗