LoginSign Up
← Back to all sparks
D

Dagster

ANALYTICS
Velocity5.0

Data orchestration platform for building and observing data pipelines.

dagster.io ↗

Dagster is spending this cycle on scale problems in the UI and correctness in asset state

data-orchestrationdbtasset-graphui-performanceweekly-releases
◆Current state
Weekly releases on the 1.13 line, each small and specific. The recurring subjects are UI performance at scale (a virtualized asset catalog, a bounded preview for runs targeting large asset selections), correctness around wiped assets and stuck backfills, and integration surface — a preview Snowflake component for orchestrating dbt projects natively, an insights option for the dbt component, and EMR Serverless dashboard tuning.
◆Where it's heading
The pattern points at customers running large graphs: the fixes that matter here are the ones that stop the UI freezing on many asset groups, stop backfills stalling on mismatched partition definitions, and stop wiped assets reporting stale materializations. Alongside that, dbt keeps drawing integration work from both the Snowflake and core dbt sides.
◆Prediction
Expect the Snowflake dbt component to move out of preview and more of the asset catalog and runs feed to adopt bounded-then-load-on-demand fetching as the default for large workspaces.

◆Recent moves

  1. 6d ago

    Preview Snowflake component for orchestrating dbt projects

    Adds SnowflakeDbtProjectComponent in preview for running dbt projects natively on Snowflake, plus documentation for the Dagster MCP server. It continues the pattern of dbt drawing the most integration attention this cycle.

    View source ↗
  2. 13d ago

    Serverless I/O manager materialization fixes

    Fixes 401 and 400 errors from the serverless I/O manager for multi-deployment organizations and storage keys with special characters. Narrow correctness work on the hosted path.

    View source ↗
  3. 20d ago

    Install-time gRPC conflict and automation tick fixes

    Resolves a fresh-install dependency conflict that broke code server imports and fixes tick details for declarative automation runs. Maintenance rather than new surface.

    View source ↗
  4. 27d ago

    Bounded runs feed previews and an EMR Serverless refresh interval

    The runs feed now fetches a bounded preview for large asset selections, and EMR Serverless gained a dashboard refresh interval so Spark UI links survive a run. Both are scale-driven fixes for workspaces with large graphs.

    View source ↗
  5. 1mo ago

    Virtualized asset catalog and dbt insights from YAML

    The asset catalog renders as a single virtualized list so large workspaces no longer freeze, and the dbt component can enable Insights tracking from YAML config. GraphQL gained bounded selection counts to support the same pattern.

    View source ↗
  6. 1mo ago

    Stale materialization and stuck backfill fixes

    Fixes wiped assets reporting pre-wipe materializations as current, backfills that emit no runs but never complete across mismatched partition definitions, and slow partition counts for cron schedules with explicit minute lists. These are the failures large deployments actually hit.

    View source ↗