LoginSign Up
← Back to all sparks
D

dqcheckr

INFRA · APIS
Velocity2.5

Automated Data Quality Checks for Recurring Dataset Deliveries

github.com/mickmioduszewski/dqcheckr ↗

dqcheckr adds drift analysis, then removes the YAML a user had to hand-write.

data-qualityduckdbdrift-analysisyaml-configquartor-package
◆Current state
dqcheckr runs configurable data-quality checks over files and DuckDB tables, driven by YAML dataset configs and recording results as snapshots. The 0.2.0 release added the ability to compare two historical snapshots and report per-column statistical drift, schema changes and trend charts, extending the tool from point-in-time checking into change over time. The most recent tag, 0.3.0, attacks the other friction point by generating the config itself from a sniff pass over the data.
◆Where it's heading
Both moves point the same way: reduce what the operator has to write and know. Config generation removes the hand-authored YAML that gated first use, list_runs() and validate_config() make an existing setup inspectable, and the snapshot comparison turns accumulated run history into a second product surface. Check coverage keeps widening underneath — outlier detection, composite keys, row-count and file-size ceilings — and the reporting layer moved from rmarkdown to Quarto, with existing 0.1.x databases auto-migrated on first run.
◆Prediction
Expect the generated configs and the drift reports to converge, so a sniffed config can seed thresholds from the snapshot history rather than from defaults, plus continued growth in the numbered QC check catalogue.

◆Recent moves

  1. 24d ago

    Config generation from data sniffing; run listing added

    The onboarding release: generate_dataset_config() and generate_global_config() derive the YAML from the data instead of asking the user to write it, with validate_config() and list_runs() making an existing setup inspectable. The body is terse, but the direction matches the 0.2.0 push to reduce operator effort.

    View source ↗
  2. 2mo ago

    DuckDB CSV ingestion fixed for undetectable delimiters

    A single bug fix: delimiter and encoding from the dataset config are now passed to DuckDB's read_csv_auto for both column detection and table creation, so files whose delimiter cannot be auto-detected ingest correctly. No new capability.

    View source ↗
  3. 2mo ago

    Snapshot drift analysis arrives; reports move to Quarto

    ⚡ SPARK

    The release that gave dqcheckr a second axis. Alongside the usual check-catalogue growth, compare_snapshots() turns the accumulated run history into drift analysis, which is the capability the 0.3.0 config work is now making easier to reach.

    View source ↗