LoginSign Up
← Back to all sparks
A

Apache DolphinScheduler

INFRA · APIS
Velocity0.0

Distributed visual workflow scheduler for big data pipelines

dolphinscheduler.apache.org ↗

A slow, deliberate release train that keeps widening its cloud-task surface

workflow-orchestrationtask-pluginscloud-computeconnection-centeroperator-visibility
◆Current state
DolphinScheduler ships on a multi-month cadence, with 3.4.2 (June 2026) and 3.4.1 (March 2026) as the recent stable points. The work is split between DSIP design proposals, new task plugins, and operator-facing hardening. Nothing in the window suggests a rewrite; this is a mature orchestrator accreting integrations.
◆Where it's heading
The consistent direction is coverage of managed cloud compute — Amazon EMR Serverless joins the task-plugin catalog, following the earlier consolidation of Zeppelin, SageMaker and Kubernetes connections into a shared connection center. Alongside that, the project keeps tightening what operators can see and control at runtime: monitor-page visibility into running tasks per master/worker, configurable maximum runtime, and dispatch timeout handling for missing worker groups.
◆Prediction
Expect the next release to add another managed-compute task plugin and continue routing its credentials through the connection center, since that is the pattern every recent integration has followed.

◆Recent moves

  1. 1mo ago

    EMR Serverless task plugin and live task monitoring

    Adds an Amazon EMR Serverless task plugin and a monitor page that shows the running tasks and workflows on any active master or worker, extending the cloud-coverage and operator-visibility threads that define this phase. Supporting work includes parameter replacement for Flink tasks and disabling HTTP TRACE via configuration.

    View source ↗
  2. 4mo ago

    Configurable workflow runtime limits and dispatch timeouts

    Workflow and task instances gain a configurable maximum runtime, and the master now times out dispatch when a worker group is missing or has no available workers. Both close long-standing failure modes where a stuck job had no ceiling, fitting the run-time control theme rather than adding capability.

    View source ↗
  3. 1y ago

    Connection center centralizes credentials across task plugins

    ⚡ SPARK

    The alpha that set up the current architecture: Zeppelin, SageMaker and Kubernetes connections all move into a shared connection center, with the storage SPI reworked and remote logging configuration split out. Every integration shipped since has followed the pattern this release established.

    View source ↗