LoginSign Up
← Back to all sparks
O

ONNX Runtime

AI-ASSISTANTS
Velocity7.5

Cross-platform inference and training engine for ONNX-format machine-learning models.

onnxruntime.ai ↗

ONNX Runtime is dismantling itself into a core plus detachable accelerator plug-ins, CUDA included.

inference-runtimeexecution-providerswebgpucudamodel-optimizationplugin-architecture
◆Current state
The runtime's accelerators are leaving the main binary. WebGPU went first as a standalone plug-in execution provider, and CUDA — the backend most GPU deployments actually use — followed in August as a separately packaged plug-in that registers with an existing installation and is now the default CUDA implementation. Alongside that, onnxruntime-web has announced the end of WebGL and JSEP with native WebGPU as the only forward path, and the latest patch adds device-free WebGPU compilation so graphs can be transformed and serialized offline with no GPU present.
◆Where it's heading
The direction is decoupling on two axes. Vertically, accelerator support is being pulled out of the core release train so CUDA fixes and new vendor features no longer wait on a core version, with a plug-in ABI carrying version-gated callbacks as the compatibility surface. Horizontally, the core itself is getting lighter — cuDNN and cuFFT made optional, nvrtc unlinked, the CUDA redistributable footprint cut. Note the release numbering does not read chronologically: the 1.28.1 patch shipped after both 1.29.0 and the CUDA plug-in, because the 1.28 line is being serviced in parallel.
◆Prediction
Expect the plug-in EPs to take over release cadence from the core, with CUDA 12 removed in 1.27 as announced and further backends following WebGPU and CUDA out of the main binary.

◆Recent moves

  1. 1d ago

    Device-free WebGPU compilation for offline model optimization

    A 1.28-line patch whose one real addition matters for build pipelines: compile-only WebGPU sessions that transform graphs and serialize optimized models without GPU hardware present, which makes model optimization a CI step rather than a GPU-host step. The rest is hardening — a Win32k lockdown access violation, zero-input EPContext nodes, and fusion pattern validation.

    View source ↗
  2. 2d ago

    CUDA becomes a standalone plug-in execution provider

    ⚡ SPARK

    The plug-in architecture reaches the backend that matters most. CUDA ships as its own package with arena allocation, resource accounting, user compute streams, and external allocators, and becomes the default CUDA provider rather than an alternative to the in-core one.

    View source ↗
  3. 8d ago

    ONNX Runtime 1.29 deprecates WebGL and JSEP, adds POSIX telemetry

    ⚡ SPARK

    Two independent decisions in one release: browser inference consolidates on native WebGPU with WebGL and JSEP deprecated, and telemetry extends off Windows onto Linux, macOS, Android and iOS. The first closes a migration path, the second opens a question every downstream vendor now has to answer.

    View source ↗
  4. 8d ago

    ONNX Runtime 1.26 adds RISC-V vector support and .ort memory mapping

    RISC-V vector instructions reach the CPU provider and .ort models gain optional memory mapping, both aimed at constrained deployment targets. The release also gave notice of two things since delivered: CUDA 12 removal, and CUDA moving to a dedicated execution provider.

    View source ↗
  5. 21d ago

    WebGPU plug-in: FlashAttention fusions, Qwen3 and Gemma 4 paths

    Attention-heavy LLM work concentrated in the WebGPU plug-in — fused FlashAttention decode kernels for any sequence length, QKV and MLP fusions for Qwen3-style models, and Gemma 4 KV-shared decoder support. It demonstrates the point of the plug-in split: model-specific optimization shipping on its own schedule rather than the core's.

    View source ↗
  6. 26d ago

    ONNX 1.22 upgrade, slimmer CUDA footprint, experimental C API

    cuDNN and cuFFT become optional at runtime and nvrtc is unlinked, cutting the CUDA redistributable footprint substantially — the core shedding weight ahead of CUDA leaving it entirely. An experimental C/C++ API surface arrives with explicit notice that it may change.

    View source ↗