This Week's Picks
Release
SGLang's Main Stack Lands on Apple Silicon
v0.5.22 runs the full stack on Apple Silicon; 65.9 tps on an M4 Pro.
- Why it matters:
- v0.5.22 adds a standard PyTorch/MPS backend for Apple Silicon, so a Mac now runs the same scheduler, model runner, and memory pool as GPU servers. The team reports 65.9 tps median decode and 2,531 tps prefill for Qwen3-0.6B at offline batch 4 on an M4 Pro with 48 GB. The release also ships 710 PRs, and overlapped checkpoint staging cuts Qwen3-32B startup on H100 from 84.8s to 35.6s.
- Relevance to edge / on-device deployment:
- Directly relevant: Apple Silicon gains a second mainline path next to the MLX backend, with memory planning that respects the unified-memory limit.
Source: GitHub SGLang Releases
Release
Kimi K3 Matches B300 per GPU on MI355X
vLLM reports MI355X per-GPU throughput on par with HGX B300 for Kimi K3.
- Why it matters:
- The vLLM blog presents measurements of Kimi K3 on AMD Instinct MI355X reaching HGX B300 per-GPU throughput, a rare public head-to-head of AMD's flagship against NVIDIA on agentic serving.
- Relevance to edge / on-device deployment:
- No edge angle per se; if the per-card economics hold, small on-prem clusters gain a second hardware option.
Source: vLLM Blog
Release
llama.cpp 0.7.0 and a Week of Hexagon Quantization
v0.7.0 tagged Oct 15; Hexagon gains Q1_0/Q2_0 quantization formats.
- Why it matters:
- v0.7.0 followed v0.6.0 (extended batch API, GLM-5.3-Flash, MTP speculative decoding). This week the Hexagon backend picked up Q1_0/Q2_0 formats with Hadamard ops and reworked quantized weight repacking, and the Vulkan backend got L2-friendly workgroup scheduling for matmul.
- Relevance to edge / on-device deployment:
- Hexagon is Qualcomm's DSP: Q1_0/Q2_0 support brings sub-2-bit inference paths to Snapdragon-class devices.
Source: GitHub llama.cpp Releases
Tool
fastsafetensors Cuts Model Loading Time
IBM's open-source loader makes safetensors loading 4.8–7.5× faster.
- Why it matters:
- By removing per-tensor copies, coalescing fragmented I/O, and skipping host staging, checkpoint loading in PyTorch inference systems gets 4.8–7.5× faster with up to 28 GB/s of NVMe read throughput. Parallel loading, ROCm, and unified-memory support are included, with a lightning talk at PyTorch Conference on Oct 20.
- Relevance to edge / on-device deployment:
- Cold-start time is a real pain point when shipping large models to edge devices, and faster loading changes the first-run experience.
Source: PyTorch Blog
Paper
PN-Mix: Mixed-Precision Key-Cache Quantization
Allocates precision by channel-pair saliency for 2-bit key caches.
- Why it matters:
- Analyzing attention-logit perturbation at the RoPE channel-pair level, the authors show channel pairs with large query–key pair-norm products dominate quantization error. Saliency is identified once during prefill and reused through decoding. At 2.26-bit average precision PN-Mix matches or beats prior mixed-precision methods that use 2.32–2.39 bits, and it degrades gracefully down to 1.5 bits.
- Relevance to edge / on-device deployment:
- Key-cache quantization is the binding constraint for long context on-device; a calibrate-once design fits decode-dominated edge workloads.
Source: arXiv 2610.19605
Paper
BELT: Learned Low-Rank KV Compression
Learned top-k rank allocation for low-rank KV compression; 3.3× decode throughput.
- Why it matters:
- BELT trains a global rank allocator with soft selection at training time and hard selection at deployment to meet a user-specified budget, with a key-side objective aligned to attention distributions. Across three models it reaches up to 70% low-rank compression and beats the strongest baseline by up to 23.3 points on LongBench; combined with quantization it cuts the KV cache by about 94% (17.8×), and the Triton kernels deliver 3.3× end-to-end decode throughput.
- Relevance to edge / on-device deployment:
- Low-rank plus quantization is another viable compression path for memory-constrained devices, and the released Triton kernels are reusable.
Source: arXiv 2610.17966
Paper
MultiAlloc: Budget-Aware KV Cache Quantization
Allocates quantization precision across layers and tokens under a fixed memory budget.
- Why it matters:
- The paper reframes KV cache quantization as a resource-allocation problem. Lightweight offline profiling characterizes layer sensitivity and token importance, and the framework derives a precision-allocation policy between layer-wise enhancement and token-wise preservation without online search. Under matched memory budgets it consistently improves over single-granularity methods on Mistral, Llama, DeepSeek-Qwen, and QwQ.
- Relevance to edge / on-device deployment:
- Fixed memory budgets are the norm on edge devices, so budget-aware allocation tracks deployment reality better than chasing ever-lower bit-widths.
Source: arXiv 2610.18642
Signal vs Noise This Week
Signal Serving stacks enter a heterogeneous-hardware sprint
SGLang's main stack on Apple Silicon, vLLM's MI355X numbers, and llama.cpp's Hexagon push all landed in one week. The competition has moved from CUDA-only to demanding a mainline path for every serious chip, and this will run through year-end.
Signal KV cache compression becomes a budget-allocation problem
PN-Mix, MultiAlloc, and BELT appeared in the same week, all asking how to spend a fixed precision or rank budget rather than how low the bit-width can go. These profiling-driven methods are cheap to implement and will likely be absorbed by serving ecosystems quickly.
Noise Day-0 announcement density is outrunning evidence
vLLM published three Day-0 posts in three days (Nemotron 3.5 Super VL, Muse Spark Open, Reflection Beam), most reading like partner announcements without reproducible numbers. Wait for independent benchmarks before assigning weight.
This issue is curated from public sources; every link points to the original publisher. If information has changed after publication, the source page takes precedence.