LoginSign Up
← Back to all sparks
M

Marquez

ANALYTICS
Velocity0.0

Metadata service for collecting, aggregating and visualising data lineage

marquezproject.ai ↗

Marquez spent 2024 turning a lineage store into a UI, then stopped releasing

openlineagedata-lineageobservabilitymetadatacolumn-lineage
◆Current state
Marquez is the reference metadata server for OpenLineage, and this window captures a concentrated push on its web interface: a column-lineage page and redesigned lineage graph in 0.45.0, data quality and job status displays in 0.47.0, and a data observability dashboard in 0.50.0 showing event stats over 24-hour and 7-day windows with per-source, per-dataset and per-job views. The API side moved in step — job tagging, dataset schema versions, paging on jobs and dataset versions, job-to-job lineage, richer metrics labels. One contributor accounts for nearly all the web work in these entries. Note that the 0.45 through 0.49 releases were all stamped within three minutes of each other and in reverse version order, so publication timestamps here do not reflect release order.
◆Where it's heading
The arc is a backend project growing a front end: lineage data that existed in the API was progressively given pages, panels and paging, and by 0.50.0 the framing had shifted from lineage graph to observability dashboard. Tagging deepened in the same period, moving from dataset tags to field-level tags to job tags. That arc stops abruptly — 0.50.0 in October 2024 is the last entry, with nothing published in the roughly two years since.
◆Prediction
The entries give no roadmap signal past 0.50.0, and the job-to-job lineage change was explicitly framed as a prompt for a wider parent/child hierarchy discussion that no later release picks up, so that hierarchy work is best read as unfinished rather than planned.

◆Recent moves

  1. 1y ago

    Marquez 0.50.0 adds a data observability dashboard

    The largest and last release here, adding a dashboard of OpenLineage event statistics over 24-hour and 7-day windows with breakouts by source, dataset and job, plus a job list showing the latest runs and their durations. It completes the shift from lineage viewer to something closer to an operational monitoring surface, and then the feed goes quiet.

    View source ↗
  2. 2y ago

    Marquez 0.45.0 ships a column lineage page and redesigned graph

    Column-level lineage gets its own page and the lineage graph is redesigned around a shared set of navigation principles, with collapsible dataset nodes and zoom-to-selection. This is the release that started the year-long UI push the later versions build on.

    View source ↗
  3. 2y ago

    Marquez 0.46.0 cleans up the UI and fixes streaming terminal events

    A small release removing non-useful UI elements and correcting how COMPLETE and FAIL events are handled for streaming jobs, so a terminal event without dataset information no longer creates a spurious job version. Cleanup following the 0.45.0 redesign.

    View source ↗
  4. 2y ago

    Marquez 0.47.0 adds data quality and job status views, job tagging

    Data quality and job status displays arrive in the web UI while job tagging lands in the API, extending the tagging thread that ran from dataset tags through field-level tags. Helm chart values for service accounts and extra containers point at deployments being run by platform teams rather than by hand.

    View source ↗
  5. 2y ago

    Marquez 0.49.0 adds job-to-job lineage to the API

    A deliberately small backend change returning job UUIDs alongside parent names, which the release notes present as a prompt for a broader parent/child hierarchy design rather than a finished feature. It ships with critical npm audit fixes and URL encoding for job, dataset and tag names in API requests.

    View source ↗
  6. 2y ago

    Marquez 0.48.0 adds paging and dataset schema versions

    Paging on the jobs and dataset-versions panels plus dataset schema versions in the model — the practical consequence of instances accumulating enough history that unpaged panels stopped working. Metrics gain DAO and endpoint labels, making the server easier to monitor in production.

    View source ↗