Issue 3: Rubin Day-0 and the KV-Compression Engineering Turn

Day-0 Vera Rubin NVL72 support with Rubin-tuned FlashInfer kernels and locality-aware MoE.

Why it matters:

The official figure is 7.8x the per-GPU throughput of GB200 NVL72 on the AgentX benchmark. Inference headroom on new hardware is now captured from day one, and an engine's kernel-engineering bandwidth is what converts it.

Relevance to edge / on-device deployment:

Datacenter-class with no direct edge angle; the kernel and scheduling ideas tend to migrate to workstation and RTX paths later.

Source: vLLM Blog

Three weeks after launch: 47% lower latency at low concurrency, 5x AgentX throughput.

Why it matters:

The gains come from SWA bounded replay, CUDA graph fusion, and kernels shipped with the model itself; at a fixed 155 tok/s/user operating point, per-chip throughput rose from 27k to 144k tok/s. The repeatable pattern: model teams ship kernels, engine teams own scheduling.

Relevance to edge / on-device deployment:

Bounded-replay style time-locality tricks carry over to long-context on-device inference.

Source: vLLM Blog

Prefill nodes reuse KV computed by decode nodes on later turns; TTFT improves up to 3.3x.

Why it matters:

In PD-disaggregated serving, later conversation turns no longer recompute the full prefix on the prefill side. For multi-turn agent workloads this removes most redundant prefill; at 20k input tokens, TTFT drops from roughly 2300 ms to 700 ms.

Relevance to edge / on-device deployment:

The compute-once-reuse-where-needed idea transfers to on-device setups where multiple sessions or agents share context.

Source: vLLM Blog

Pure-C local inference for DeepSeek 4 Flash/PRO on Metal, CUDA, and ROCm; 23K+ stars in days.

Why it matters:

The creator of Redis building an engine against a brand-new architecture is a strong code-quality signal. It is worth reading as a second reference implementation alongside llama.cpp.

Relevance to edge / on-device deployment:

Single-machine inference is the whole point; Apple Silicon and consumer GPUs are the primary targets.

Source: GitHub Trending / coddykit analysis

Six builds in seven days touch the Hexagon, SYCL, OpenCL, and Vulkan paths.

Why it matters:

Highlights include Hexagon DMA scalar broadcast for Qualcomm DSP builds, Q5_K reorder-layout MMVQ with fused GLU on SYCL, Gemma-4 decode optimizations on OpenCL, and a server fix for prompt-cache updates on busy slots. Phone DSPs, integrated GPUs, and mobile GPUs now get maintenance density comparable to the CUDA path.

Relevance to edge / on-device deployment:

Directly relevant: every fix this week lands on edge or highly fragmented hardware.

Source: GitHub llama.cpp Releases (via freedom.tech)

Paired non-orthogonal Q/K transforms let INT2 KV quantization and query pruning coexist.

Why it matters:

At 128K context the cache shrinks to roughly 1/6.8 of BF16 and KV read volume drops by about 88%; the SGLang implementation reaches 3.75x the decode throughput of unpruned BF16. It is training-free and already integrated into an engine.

Relevance to edge / on-device deployment:

2-bit KV with channel pruning is the combination long-context on-device deployment has been missing.

Source: arXiv 2610.09827

Spatial-temporal post-training quantization for delta-rule recurrent states, merged into SGLang.

Why it matters:

6-bit states match FP32 accuracy on Qwen3.8-27B and Kimi-Linear-48B-A3B while cutting recurrent-state memory by more than 80%; total serving memory drops by up to 68.7%. Code is open-source.

Relevance to edge / on-device deployment:

For on-device linear-attention models, the fixed-size recurrent state is the new KV cache, and this is its first practical quantization recipe.

Source: arXiv 2609.38169