Issue 5: Multi-Tenant KV Safety and Non-Portable Speedups

Breaking release: Python 3.10 dropped, max_tokens semantics change.

Why it matters:

v0.32.0 shipped on Oct 18: minimum Python moves to 3.11, default CORS origins are restricted, cascade attention and mamba_cache_mode='all' are removed, and `max_tokens` is no longer reserved from the context window, which can change long-context behavior for existing clients. Model Runner V2 keeps gaining ground with a GPU n-gram speculator, drain-then-commit scaling for Elastic EP, PP with PCP for MLA models, and a STOCK_TORCH_COMPILE mode.

Relevance to edge / on-device deployment:

Not an edge item per se, but the max_tokens semantic change affects every self-hosted client and edge gateway needs to adapt too.

Source: GitHub vLLM Releases

Failed PD transfers wrote into other requests' KV; coordinated disclosure.

Why it matters:

Cloudflare found the flaw in production in September and reported it to SGLang: in disaggregated PD serving, failed Mooncake/NIXL KV transfers were still acked as drained, so the decode side freed slots early and leftover writes could land in another request's KV cache, a cross-tenant data exposure. The fix is deferred release of KV slots (#43719, #45973 and related), shipped in v0.5.22/v0.5.23, with the full timeline disclosed jointly on Oct 26.

Relevance to edge / on-device deployment:

Single-user on-device setups are unaffected, but shared KV lifecycle design is an audit surface for multi-tenant edge gateways too.

Source: LMSYS Blog

SGLang and vLLM run on TPUs via TorchTPU, code out with the talk.

Why it matters:

At PyTorch Conference (Oct 21), a Meta×Google session presented a new PyTorch-native TPU backend that runs SGLang and vLLM on Cloud TPUs while preserving each engine's scheduler, batching, and OpenAI-compatible API, covering Pallas attention, FP8 MoE (including Qwen3-Coder-480B), PD disaggregation, and speculative decoding; both codebases were open-sourced with the talk. Fujitsu also shared vLLM+OpenVINO numbers on Arm: roughly 2× throughput for GPT-OSS/Llama on Graviton3e.

Relevance to edge / on-device deployment:

TPUs are far from the edge, but the Arm CPU track (SVE-optimized SDPA, U8 KV cache) is worth borrowing for edge servers.

Source: PyTorch Blog

VTCM memory size now configurable for Snapdragon tiers.

Why it matters:

b11913 adds GGML_HEXAGON_VTCM_SIZE and GGML_HEXAGON_VTCM_MB env vars to the Hexagon backend, exposing the vector tightly-coupled memory size so quantized inference can be tuned per Snapdragon tier. The same week llama.cpp merged support for the Flint model architecture (b11803) and a Realtime API audio-commit timing fix (b11828).

Relevance to edge / on-device deployment:

Squarely on-device tuning: VTCM is the Hexagon NPU's on-chip memory, and sizing it correctly decides real throughput for quantized models.

Source: GitHub llama.cpp Releases

Training-free MLA cache quantization to 2.5 bits; 6.5× capacity.

Why it matters:

MLA caches a compressed KV latent plus a narrow positional key that carries position information, and the two tensors play different roles, so MHA-style quantization loses accuracy. RAQ-MLA fits an invertible transform per tensor against the error that tensor delivers (attention-weighted value error for the latent, score perturbation for the positional key). Training-free, it quantizes the full MLA cache to about 2.5 bits per element, a 6.5× increase in theoretical KV capacity, staying nearly lossless on reasoning and coding tasks across four MLA models.

Relevance to edge / on-device deployment:

MLA is now the mainstream attention in strong on-device models (DeepSeek, Kimi families); this is the first training-free quantization designed for the MLA cache structure.

Source: arXiv 2610.33625

Speculative decoding reduces throughput on Apple; speedups don't port.

Why it matters:

The authors evaluate four optimization families across four Apple machines: KV-cache pruning reaches up to 1.75× end-to-end throughput but can slow desktop NVIDIA GPUs; every speculative decoding method they test reduces throughput on Apple despite reported 2–6× gains on server GPUs; CUDA kernel designs transfer only partially (public Metal lacks asynchronous copy), with transferable parts yielding up to 1.20× decode throughput. Their conclusion: a speedup is a joint function of technique, runtime, hardware contract, and workload regime, not a property of the technique alone.

Relevance to edge / on-device deployment:

Required reading for on-device stack selection: on Apple Silicon, validate KV-pruning-style methods before importing server-side speculative-decoding conclusions.

Source: arXiv 2610.31479

Adaptive draft-KV refresh keeps 98% acceptance at an 8K budget.

Why it matters:

Speculative-decoding drafters also carry a KV cache for long outputs, and selection-based compression tuned on prefill information goes stale as generation proceeds (decode staleness). RIFT refreshes cache selection during generation, cuts refresh overhead with an adaptive strategy, and adds compressibility training via sparse attention masks. The result: Qwen3.5-4B drafting for Qwen3.5-122B retains over 98% of full-cache acceptance length at 32K input tokens, 4K output tokens, and an 8K KV budget.

Relevance to edge / on-device deployment:

Long-output agents on-device need both speculative decoding and memory discipline; drafter-side KV compression sits exactly at that intersection.

Source: arXiv 2610.31098