KServe vs Together AI

KServe's last two major release cycles (v0.19.0, v0.20.0, now v0.21.0) delivered a comprehensive LLM serving rework: native support for OpenAI Completions, Responses API, and Anthropic Messages API; KV cache offloading for CPU tiering; traffic splitting for controlled LLM deployments; Managed DRA (Kubernetes Dynamic Resource Allocation) for GPU resource management; vLLM as a first-class runtime; LoRA adapter affinity scoring; confidential model serving; and autoscaling via KEDA and HPA. The LLMInferenceService (llmisvc) is now the platform's primary development surface, not the classic InferenceService.

KServe is repositioning as the Kubernetes-native LLM inference platform for enterprise, not just a generic ML serving abstraction. The prefill/decode disaggregation work (llm-d integration), KV cache tiering, distributed tracing, and multi-API protocol support (OpenAI, Anthropic) all target production LLM workloads at scale. Confidential model serving and Managed DRA integration signal intent to serve regulated environments where GPU resource isolation and data protection are requirements. The llmisvc trajectory points toward multi-model routing behind a single endpoint and increasingly sophisticated scheduling.

The v0.21.0 release cycle likely consolidates the llmisvc API into a stable surface. The next major release will probably ship autoscaling policies based on KV cache utilization rather than request count alone, and extend multi-model routing to cover model versioning and A/B deployments.