LoginSign Up
← Back to all sparks
D

Dagster

ANALYTICS
Velocity5.0

Data orchestration platform for building and observing data pipelines.

dagster.io ↗

Dagster extends Declarative Automation from assets to jobs, unifying the orchestration model.

orchestrationdata-engineeringdeclarative-automationsnowflake-dbtmcpdagster-plus
◆Current state
Dagster is shipping steady incremental improvements across 1.13.x: asset health status for retrying partitions, SnowflakeDbtProjectComponent maturation, alert policy YAML enforcement in Dagster+, and an automation condition UI that shows why historical conditions fired. The most architecturally significant change in the window is Declarative Automation for jobs, now in preview — it applies the same condition-based trigger model that assets use to scheduled batch jobs.
◆Where it's heading
Declarative Automation expanding to jobs is the directional story. If it exits preview and gains adoption, it reduces the need for sensors and schedules as separate primitives — the same declarative condition layer covers everything. The Dagster+ MCP server gaining OAuth documentation signals growing interest in using Dagster as an orchestration target for AI agents, not just a data pipeline tool.
◆Prediction
Declarative Automation for jobs will exit preview within 1-2 minor releases, and the next visible investment is likely deeper first-class support for AI workload orchestration alongside the existing data pipeline primitives.

◆Recent moves

  1. 1mo ago

    1.13.21: backfill completion fix, YAML-enforced alert policies

    Backfills now correctly complete as failed when all runs finish but partitions remain unresolved, closing a stuck-in-progress edge case. Alert policies in Dagster+ can now be fully managed from YAML files, removing the ability to edit them in the UI — enforcing infrastructure-as-code patterns for teams that need it.

    View source ↗
  2. 2mo ago

    1.13.20 (core) / 0.29.20 (libraries)

    Dynamic partitions gain a combined wipe-and-delete action in Dagster+, clearing degraded health status in one step. Logo and favicon refresh, minor UI improvements. No changes to orchestration behavior.

    View source ↗
  3. 2mo ago

    1.13.19 (core) / 0.29.19 (libraries)

    An expand/collapse toggle for automation condition evaluation tables and a fix for a sensor dry-run permission edge case. Maintenance update with no new orchestration capabilities.

    View source ↗
  4. 2mo ago

    1.13.18 (core) / 0.29.18 (libraries)

    Failed partitions awaiting automatic retry now show a WARNING health status instead of DEGRADED — a UX clarification that prevents unnecessary alarm. SnowflakeDbtProjectComponent gains defs_state support for versioned state storage.

    View source ↗
  5. 2mo ago

    1.13.17: spark command injection fix, MCP OAuth docs, asset graph search improvements

    A command injection vulnerability in dagster-spark is patched — spark-submit is now invoked directly instead of through a shell, preventing shell metacharacter interpretation in application arguments. Also ships improved asset graph search and documentation for connecting to the Dagster+ MCP server via OAuth.

    View source ↗
  6. 3mo ago

    1.13.16: Declarative Automation for jobs (preview)

    ⚡ SPARK

    Declarative Automation gains job support in preview: pass an automation_condition to define_asset_job and Dagster triggers a run when the condition becomes true. The same model that assets use now extends to the job primitive — the first step toward a unified declarative orchestration layer.

    View source ↗