LoginSign Up
← Back to all sparks
T

tidyaudit

INFRA · APIS
Velocity0.0

Pipeline Audit Trails and Data Diagnostics for 'tidyverse' Workflows

github.com/fpcordeiro/tidyaudit ↗

Pipeline provenance for tidyverse workflows, recording what changed at each step without keeping the data.

data-qualityprovenancetidyversepipeline-auditingdata-validationr-package
◆Current state
tidyaudit records lightweight metadata snapshots as data flows through a pipeline — row and column counts, NA counts, and structured diffs between any two points — without storing the data itself. Taps are operation-aware, so join, filter, and anti-join steps each report what that operation specifically did, and validation helpers cover join integrity, primary keys, and variable relationships. The trail can now be exported as a self-contained interactive HTML diagram or serialized to JSON or RDS.
◆Where it's heading
The arc is from inspection to artifact. The first release made the trail something you print and read; 0.2.0 made it something you can hand to someone else or feed to another program, with the HTML export deliberately requiring no server and no Shiny. Reporting has been refined in the same direction, with a tabular changes block showing from-and-to values with row, column, and NA deltas. The remaining work in the window is defensive — a factor-handling path rebuilt because R-devel tightened what as.data.frame.table() accepts in row names.
◆Prediction
With serialization and a standalone export in place, the natural next step is making trails comparable across runs rather than only across steps within one, though nothing in the entries commits to it yet.

◆Recent moves

  1. 3mo ago

    Factor auditing fixed against a stricter R-devel

    R-devel began rejecting NA in row names, which broke the frequency-table builder for every factor input because the always-NA bucket carries one. The table is now constructed directly, with the notes stating no user-visible output change.

    View source ↗
  2. 4mo ago

    Trails export to standalone HTML and machine-readable formats

    The release that makes an audit trail portable: a self-contained interactive pipeline diagram with clickable nodes needing no server or Shiny, plus serialization to RDS and JSON. Frequency tables and crosstabs join the tap family, and snapshot scope can be limited on wide datasets. It extends the trail concept rather than replacing it, but it is what turns a console output into something shareable.

    View source ↗
  3. 5mo ago

    First release: pipeline audit trails for tidyverse

    ⚡ SPARK

    The founding release and the source of the design decision everything else rests on: capture metadata at each pipe step, never the data. Operation-aware taps and diff-by-label are what make the resulting trail readable rather than a pile of dimensions.

    View source ↗