LoginSign Up
← Back to all sparks
K

KServe

AI-ASSISTANTS
Velocity5.0

Standardized model inference serving platform on Kubernetes

kserve.github.io/website ↗

KServe now releases almost entirely for its LLM inference service.

model-servingkubernetesllm-inferencegpu-schedulingrelease-candidates
◆Current state
KServe publishes release candidates rather than finals to this feed, running rc0 and rc1 pairs through the 0.18, 0.19 and 0.20 cycles. The commit lists are dominated by llmisvc, the LLMInferenceService controller: model-based routing gates with models surfaced in status, cached inference service configuration with change watching, heterogeneous GPU load balancing, TLS flags for the disaggregation sidecar, and graceful handling when the LeaderWorkerSet or InferencePool CRDs are absent.
◆Where it's heading
The centre of gravity has moved from generic model serving to serving large language models specifically, with the surrounding Kubernetes ecosystem — Gateway API Inference Extension CRDs, LeaderWorkerSet, InferencePool — treated as dependencies rather than options. Handling missing CRDs gracefully in release after release says the project expects to run in clusters that have only some of that stack. The CSV and Parquet marshallers and CloudEvents logging improvements are the remaining generic-serving work.
◆Prediction
The 0.20 candidates are converging on a small change set, so a 0.20.0 final is close; disaggregated serving is the newest area and the most likely focus after it.

◆Recent moves

  1. 5d ago

    Second 0.20 candidate: four llmisvc fixes

    A four-commit candidate covering a disaggregation sidecar TLS flag, quote escaping in generated kv-transfer config, and skipping v1alpha2 InfPool reconciliation when the CRD is absent. The small delta suggests 0.20 is stabilising.

    View source ↗
  2. 23d ago

    Model-based routing gates and cached inference config

    The substantive 0.20 candidate. Model-based routing gates with models reported in status move llmisvc toward routing decisions made on model identity, and caching the inference service config with change watching cuts a per-reconcile cost that shows at scale.

    View source ↗
  3. 2mo ago

    Heterogeneous GPU load balancing and label propagation

    Heterogeneous GPU load balancing is the notable addition — it acknowledges that inference clusters are assembled from mixed hardware rather than uniform pools. Label and annotation propagation to the service is small but removes a common workaround.

    View source ↗
  4. 3mo ago

    Second 0.18 candidate, restating rc0's change list

    Published two days after rc0 with an identical visible change list; the fetched body is truncated at the same point, so any delta is not observable from this entry. Treat it as a retag of the 0.18 candidate rather than new work.

    View source ↗
  5. 3mo ago

    Inference Extension CRDs bundled; CSV and Parquet marshallers

    Bundling the Gateway API Inference Extension CRDs into the build is the structural change — it makes the inference gateway stack part of KServe's own surface. CSV and Parquet marshallers plus LeaderWorkerSet address resolution round out a release aimed at real deployment shapes.

    View source ↗