LoginSign Up
← Back to all sparks
G

git2rdata

ANALYTICS
Velocity0.0

An R package for storing and retrieving data.frames in git repositories.

github.com/ropensci/git2rdata ↗

git2rdata keeps sharpening one idea: a data frame that produces a readable git diff.

version-controlreproducibilityr-languagedata-storagemetadata
◆Current state
git2rdata stores data frames as plain text plus a metadata sidecar so that version control sees meaningful line-level diffs instead of binary churn. The recent releases have all pushed on the metadata half of that pair: 0.4.1 added `update_metadata()`, 0.5.1 made arbitrary data frame metadata round-trip through storage, and 0.5.2 adds a `convert` argument that records column conversions in the metadata and reverses them on read.
◆Where it's heading
The file format itself settled years ago — the last breaking change was the 0.2.0 hash rework — and development since has been about what travels alongside the data. Storage decisions that used to be implicit are becoming declarative and recorded: significant digits in 0.5.0, arbitrary attributes in 0.5.1, type conversions in 0.5.2. The other steady thread is determinism, from C-locale sorting through `icuSetCollate()`, because unstable ordering is what turns a one-row change into a whole-file diff.
◆Prediction
The metadata system has absorbed digits, attributes and conversions in three consecutive releases, so the next likely addition is another storage decision moved into metadata rather than any change to the on-disk format.

◆Recent moves

  1. 4mo ago

    Column conversions recorded in metadata and reversed on read

    `write_vc()` gains a `convert` argument that applies column conversions before storage and records them in the metadata, and `read_vc()` reverses them on the way back. The third consecutive release to move a storage decision out of user code and into the metadata sidecar, where it travels with the data through git.

    View source ↗
  2. 8mo ago

    Data frame metadata now round-trips through storage

    Metadata attached to a data frame survives `write_vc()` and comes back from `read_vc()`, alongside handling for empty factors and for multiple commits landing within the same second. The commit-timestamp fix matters more than it reads — `recent_commit()` is how the package locates a stored object's history.

    View source ↗
  3. 1y ago

    Significant digits become an explicit storage option

    A `digits` argument on `write_vc()` and `meta()` sets how many significant digits numeric columns are stored with. For a package whose value is diff quality, controlling float precision at write time is the difference between a stable file and one that churns on every regeneration. Empty datasets stored with `split_by` also read back correctly now.

    View source ↗
  4. 1y ago

    update_metadata() for editing a stored object's description

    Adds `update_metadata()` so an object's description can be revised without rewriting the data. The first of the metadata-focused additions that the 0.5.x line then built out.

    View source ↗
  5. 4y ago

    Non-optimised files switch to CSV; verify_vc() added

    Non-optimised output moves from tab-separated to comma-separated on the reasoning that people recognise `.csv` as a data file — a legibility argument, consistent with a package that exists so humans and code review can read stored data. `verify_vc()` adds a contract check that required variables are present before returning the data.

    View source ↗
  6. 4y ago

    Standardised sorting via icuSetCollate()

    A one-line refinement of the locale-independent ordering established back in 0.2.0, now using `icuSetCollate()`. Invisible in use, but it is the mechanism that keeps a row reordering from showing up as a whole-file diff.

    View source ↗