LoginSign Up
← Back to all sparks
K

KServe

AI-ASSISTANTS
Velocity5.0

Standardized model inference serving platform on Kubernetes

kserve.github.io/website ↗

KServe pivots to LLM-first serving: disaggregated inference and model-based routing in v0.21 RC

ml-servingkubernetesllm-inferencedisaggregated-inferencecncfmodel-serving
◆Current state
KServe is midway through a significant architectural shift, building LLMInferenceService (llmisvc) as a first-class CRD alongside the original InferenceService. The v0.21.0 release candidate adds disaggregated inference support — splitting prefill and decode stages across separate pods via KV-transfer config — and model-based routing gates that hold traffic until a model's health status confirms readiness. Both v0.21.0 RCs are light on changelog detail, consistent with a project in final pre-release hardening.
◆Where it's heading
KServe is repositioning from a generic ML model server to an LLM-optimized inference platform. The disaggregated inference work targets the high-throughput LLM serving use case where prefill and decode stages have different compute profiles and benefit from separate scaling. Model-based routing gates and live config caching (introduced in v0.20.0) are the operational primitives needed to run multi-model fleets reliably. The ZMQ-based multi-node coordination added in v0.18 completes the architectural picture for large-scale LLM deployment.
◆Prediction
The GA of v0.21.0 will be the marker to watch — these RC cycles are unusually slow, suggesting either significant integration testing or enterprise adoption pressure shaping the release criteria. A production-stable LLMInferenceService with disaggregated inference would make KServe a credible alternative to proprietary serving stacks like Triton for teams already running Kubernetes.

◆Recent moves

  1. 2d ago

    v0.21.0-rc1 release prep

    v0.21.0-rc1 is a release prep commit with no visible user-facing changes recorded in the changelog. The RC cycle indicates final pre-release hardening is ongoing.

    View source ↗
  2. 14d ago

    v0.21.0-rc0 release prep

    v0.21.0-rc0 is the opening release candidate commit for the 0.21 cycle. No feature changelog visible at this stage.

    View source ↗
  3. 1mo ago

    v0.20.0-rc1

    v0.20.0-rc1 updates TLS flag handling for the disaggregated inference sidecar and fixes quote escaping in the KV-transfer config generator — both are reliability fixes for the disaggregated inference path introduced in rc0. The InfPool reconcile skip when the CRD is absent prevents crashes in environments without the extension installed.

    View source ↗
  4. 2mo ago

    v0.20.0-rc0

    ⚡ SPARK

    v0.20.0-rc0 introduces model-based routing gates — traffic is held until the serving model reports healthy in the InferenceService status — and live caching of the inferenceservice-config with change watching. These primitives make multi-model fleet management operationally viable: routes update without pod restarts, and unhealthy model deployments don't receive traffic automatically.

    View source ↗
  5. 3mo ago

    v0.19.0-rc0

    v0.19.0-rc0 propagates spec.labels and annotations from LLMInferenceService to the generated services, enabling standard Kubernetes observability tooling (Prometheus relabeling, network policies) to work without workarounds. Deployment mode normalization in status fixes a compatibility issue for operators that had migrated from legacy configuration shapes.

    View source ↗
  6. 5mo ago

    v0.18.0-rc1

    v0.18.0-rc1 adds CSV and Parquet marshallers for data ingestion use cases and fixes ZMQ leader address resolution for multi-node LLM inference via LeaderWorkerSet. The ZMQ fix is operationally significant: multi-node LLM inference with ZMQ-based inter-node coordination had been broken for affected deployments.

    View source ↗