Issue 7: Inference Control Planes and the FP4 Kernel War

Default batch tokens doubled; ten legacy architectures removed.

Why it matters:

927 commits from 389 contributors: the default max_num_batched_tokens rises from 8192 to 16384, Mamba models get prefix caching by default, and new request admission-control flags land. Breaking changes include removal of ten deprecated model architectures, migrating FlexOlmo/Olmo3/Hunyuan V1/VL to the Transformers backend, dropping the PyAV video decoder, and ending CUDA 12.9 builds. New models include Reflection Beam-1.0 (501B MoE, 23B active) and Kimi K3 NVFP4 checkpoints.

Relevance to edge / on-device deployment:

Not an edge item per se, but the doubled default batch size and admission-control flags matter for capacity planning on small self-hosted and edge gateways too.

Source: GitHub vLLM Releases

DSpark works under PD with decode context parallelism; CPU simulator ships.

Why it matters:

DSpark speculative decoding now runs under disaggregated PD with decode context parallelism for hybrid models such as Kimi-Linear, verified on 8× B300 over NIXL and Mooncake up to 256K input; Responses API storage becomes opt-in; the CPU-only SGLang Simulator runs the real scheduler with a latency predictor, estimating TTFT within about 6% (10% on the longest 32K–128K traces) and prefix reuse within 0.05 percentage points; prefill CP v1 is removed.

Relevance to edge / on-device deployment:

The simulator enables capacity and cache-policy evaluation without GPUs, which helps edge cluster sizing directly; the DSpark DCP work is datacenter-class.

Source: GitHub SGLang Releases

Buoyant open-sources a control plane; 95% lower TTFT on shared prefixes.

Why it matters:

On Nov 4, Linkerd maker Buoyant open-sourced Kinference (Apache 2.0, Rust, v0.1.0-alpha.7): it watches vLLM KV-cache events and engine metrics and routes requests by declarative policy across five Kubernetes CRDs instead of kube-proxy round-robin. In its own benchmark on 3× L40S, median TTFT falls from 6.88s to 0.31s on shared-prefix workloads, with 7.4× goodput on shared-prefix and 3.5× on multiturn traffic; the no-shared-prefix control shows no gain. It builds on NVIDIA's dynamo-kv-router crate. Clear gaps: alpha-stage, with rate limiting, semantic classifiers, Anthropic-style /v1/messages, and multi-replica operation all unimplemented.

Relevance to edge / on-device deployment:

Useless for single-box setups; for multi-replica edge clusters that need a routing policy layer, it is one of the Apache 2.0 starting options.

Source: vLLM Blog

Hybrid-attention architecture lands; NVFP4 gated matmuls fused.

Why it matters:

b12180 adds Kimi-K3.1-Pro with its hybrid attention combining full-attention MLA and sparse DSA layers; b12121 fuses NVFP4 gate/up matmuls and the GLU in CUDA MMQ, directly speeding up NVFP4-quantized models; b12119 adds an indexer op to the Hexagon backend.

Relevance to edge / on-device deployment:

Kimi K3.1-Pro can now run locally; the NVFP4 fusion makes FP4-quantized decode faster on RTX-class cards, and the Hexagon patch continues the Qualcomm line.

Source: GitHub llama.cpp Releases

Algorithm-system co-design for FP4 KV; 2.4–3.0× throughput on B300.

Why it matters:

For NVFP4/MXFP4 KV caches, three algorithmic fixes hold accuracy: dual-residual query quantization cuts query logit error from 60% to 1%, underflow-aware probability scaling stops long-context probabilities rounding to zero, and outlier-aware key quantization combines smoothing with Hadamard rotation. On the systems side, transposed softmax with fused probability quantization removes the softmax bottleneck. Across dense and MoE models with GQA and MLA it stays within 0.5 points of BF16 KV with NVFP4 and remains stable in RL rollouts; on B300 with Qwen3-4B and Qwen3-30B-A3B it delivers 2.4–3.0× higher peak serving throughput than BF16 KV. Code is open-sourced.

Relevance to edge / on-device deployment:

B300 is a datacenter card, but the error-control ideas (query splitting, probability scaling) transfer to on-device FP4 kernel implementations.

Source: arXiv 2611.09825

0.5-bit KV quantization for looped transformers, integrated in SGLang.

Why it matters:

Looped models reuse each layer's parameters but still store per-loop-step KV, so the cache grows linearly with steps. LoopVQ finds the cache highly redundant along the loop axis and the steps differ sharply in quantization sensitivity, so it codes channels jointly across steps and fine-tunes codebooks against full-precision outputs: at 0.5 bits per element it beats existing vector-quantization methods built for standard transformers on general, long chain-of-thought, and long-context retrieval tasks, and combined with TaSQ it keeps most BF16 accuracy even at 0.25 bits. In SGLang it nearly halves decode latency at 32K context and raises peak throughput by 4.3× and 6.1× on the 1.4B and 2.6B Ouro models.

Relevance to edge / on-device deployment:

Looped small models (Ouro, Huginn class) are edge-friendly architectures, and 0.5-bit KV further reduces their long-context footprint; the SGLang integration means the same path runs server-side.

Source: arXiv 2611.05304

Streaming precision back from RAM/SSD rescues 2-bit quantization.

Why it matters:

StrataFlux turns host RAM and SSDs into sources of extra precision: selective precision streaming reduces transfer volume, overlaps delivery with compute, and reuses fetched precision across decode steps. For 2-bit Llama-3.1-8B-Instruct it cuts WikiText-2 perplexity from 9.46 to 8.40 while keeping 95.4% of the quantized baseline's throughput on an RTX 5090; with 2-bit weights and KV, Qwen3-8B recovers from 11.0% to 47.1% on AIME 2024/2025 at unchanged GPU memory; at 128K context, 3-bit weights plus 2-bit KV match vLLM's 4-bit AWQ weights with 8-bit KV in perplexity and speed at roughly half the GPU memory. Evaluations cover discrete GPUs and unified-memory platforms.

Relevance to edge / on-device deployment:

Unified-memory platforms get dedicated evaluation: Mac-class devices with large RAM can stream precision back to compensate for 2-bit quantization losses.

Source: arXiv 2611.08363