← LLM Implementation Weekly
中文Home

Issue 8: llama.cpp 1.0 and the Vera Rubin Week

Nov 11 – Nov 17, 2026 · Published 2026-11-17

This week is infrastructure week. llama.cpp and ggml shipped 1.0 together: 35 deprecated functions removed, the NextN speculative-decoding API promoted to the public header, FP8 KV caches and a host-memory MoE expert cache added, plus a commitment that any breaking change to the public C API earns a major-version bump. On hardware, vLLM published DeepSeek-V4.1-Flash numbers on NVIDIA's new Vera Rubin NVL72: 58× the throughput of GB300 NVL72 with a 387 tok/s per-user P90. For multimodal serving, vLLM introduced encoder disaggregation (EPD); in the quantization war, Unsloth extended Dynamic V3.1 to the Qwen3.8 family (still no public response from the GSQ-RCO camp); and FlashOffloading delivered a systems design for offloading the entire KV cache to host memory on Ascend 910C. The first SGLang Summit ran Nov 12–13 in San Francisco.

This Week's Picks

Release

llama.cpp 1.0 Ships with a Stability Promise

Semver commitment lands alongside FP8 KV and an MoE expert cache.

Why it matters:
llama.cpp and ggml released 1.0 on Nov 13: llama_set_warmup and 35 deprecated functions plus defrag_thold are gone, the NextN speculative API moves from llama-ext.h into the public llama.h, OCP FP8 KV caches arrive (-ctk/-ctv f8_e4m3 with optional static per-tensor scales), and a host-memory MoE expert cache (moe_cache_size, moe_cache_cpu_share, LRFU replacement) lets small GPUs run large MoE models. New architectures include Kimi-K3.1-Pro, Ripple-65B-Preview, xing4_0, drama-3 TTS, and two vision models, bringing the total past 150. The team commits to bumping the major version for any incompatible public C API or ABI change. llama-server security defaults tighten: tools or MCP on a non-loopback address now require an API key (or explicit --allow-unauthenticated-tools), HF_TOKEN is restricted to huggingface.co, and DNS rebinding plus SSRF holes are fixed.
Relevance to edge / on-device deployment:
Directly relevant: FP8 KV saves memory for long context on-device, the expert cache helps small-VRAM devices run large MoE models, and the stricter server defaults affect every self-hoster.

Source: GitHub llama.cpp Releases

Release

DeepSeek at 58× Throughput on Vera Rubin NVL72

1.7M tok/s per GPU on NVL72; 387 tok/s per-user P90.

Why it matters:
The vLLM blog (Nov 12) reports DeepSeek-V4.1-Flash serving on NVIDIA's new Vera Rubin NVL72: 58× the throughput of GB300 NVL72 on the SemiAnalysis AgentX benchmark, reaching 1.7M tok/s per GPU; on the interactivity metric, per-user P90 hits 387 tok/s. It is official same-week adaptation data for new silicon, and another case of per-user speed replacing per-GPU throughput as the headline metric.
Relevance to edge / on-device deployment:
No edge angle per se, but the 387 tok/s per-user interactivity bar is becoming the consensus serving target and will shape speed expectations for on-device products too.

Source: vLLM Blog

Release

vLLM Lands Encoder Disaggregation (EPD)

Multimodal encoders become standalone services with cross-encoder caching.

Why it matters:
The vLLM blog (Nov 11) introduces encoder disaggregation: vision and audio encoders run as separate services decoupled from LLM decoding, with metadata-only multimodal requests, remote embedding transfer, cross-encoder caching, and lower-overhead EPD routing, aiming to take multimodal scaling concerns out of the decoding engine.
Relevance to edge / on-device deployment:
On-device multimodal devices can borrow the same split: keep the encoder resident and load the decoder on demand, easing deployment on small-memory hardware.

Source: vLLM Blog

Tool

UD V3.1 Extends to the Qwen3.8 Family

Qwen3.8-Flash-Next 1-bit hits 54.0% on DeepSWE.

Why it matters:
Unsloth's Dynamic V3.1 quantization (docs updated Nov 11) now covers Qwen3.8-27B, DeepSeek-V4.1-Flash-1017, and Qwen3.8-Flash-Next, claiming 10–15% higher accuracy and 50% fewer tokens than same-size alternatives. The Qwen3.8-Flash-Next 1-bit build scores 54.0% on DeepSWE v1.1 and 68.3% on Terminal Bench 2.1; DeepSeek-V4.1-Flash-1017 IQ1_S reaches 65.8% (DeepSWE) and 69.5% (TB). On last week's reproducibility challenge, the ISTA/GSQ-RCO camp has still not responded publicly as of this week.
Relevance to edge / on-device deployment:
Directly relevant: a 1-bit GGUF of Qwen3.8-Flash-Next makes a flagship-class agent model runnable on a single 12GB card, a concrete selection datapoint for local agent setups.

Source: Unsloth Docs

Paper

FlashOffloading: Host-Memory KV Offload

Full KV offload to host DRAM under PD; 1.38–1.91× throughput.

Why it matters:
Long contexts fill HBM and cap concurrency, while naive host-memory offloading adds management overhead. FlashOffloading is a phase-specialized architecture: prefill uses a four-buffer rolling HBM cache pipeline overlapping load, compute, and writeback; decode uses a hot cache with a reverse token-to-slot sparse index that adds zero extra miss-lookup cost, plus sparse-attention kernels overlapping H2D transfers. Deployed with GLM-5.2 and MTP3 speculative decoding on an Ascend 910C PD-disaggregated platform under a 30ms TPOT constraint, it improves prefill throughput by up to 1.38× and decode by up to 1.91×, turning 512K–1M-token contexts from 'does not fit' into servable.
Relevance to edge / on-device deployment:
Ascend is server silicon, but treating host memory as a KV extension layer overlapped with compute is isomorphic to unified-memory device management and worth borrowing for on-device long context.

Source: arXiv 2611.07542

Quick Takes

Signal vs Noise This Week

Signal The inference ecosystem enters stabilization

llama.cpp 1.0's semver promise, vLLM 0.33's tightened defaults and legacy cleanup, and the first SGLang Summit together mark a shift from feature races to enterprise-grade stability and governance. Downstream apps and distros can start making real long-term version plans.

Signal Same-week adaptation becomes table stakes

Vera Rubin had official vLLM numbers within its launch week, and Ascend 910C now has targeted systems papers like FlashOffloading. Engine adaptation within a week of silicon launch is no longer a differentiator but an entry ticket, and this will keep escalating.

Noise Summit and Day-0 announcements still lack third-party data

SGLang Summit sessions and Day-0 posts for Inkling 2 and H3-Image are publisher-side numbers without independent reproduction; wait for community measurements before assigning weight.

Worth a Star / a Read

Watchlist for Next Week

This issue is curated from public sources; every link points to the original publisher. If information has changed after publication, the source page takes precedence.