Issue 2: The Fall Release Wave of Inference Engines

v0.31.0: DeepSeek upgrade, restart without reload, tighter security defaults.

Why it matters:

The new `vllm preload` daemon keeps quantized weights resident in GPU memory across engine restarts; DeepSeek V4.1-Flash now defaults to SM100 FlashMLA mega attention, MXFP8, and sparse MQA; untrusted per-request multimodal kwargs are rejected by default, raising the production security bar.

Relevance to edge / on-device deployment:

Adds a DeepSeek V4 CPU backend (AVX512/AMX sparse MLA) and cgroup-aware KV cache sizing, benefiting GPU-less x86 servers and small edge clusters.

Source: GitHub vLLM Releases

779 PRs, hot-swap prefill/decode instances, new Decisions API.

Why it matters:

PD-serving instances can switch between prefill and decode roles without a restart, cutting scheduling fragmentation during elastic scaling; the Decisions API turns an LLM/VLM into a low-latency classifier or scorer, useful for tool selection in agent workflows.

Relevance to edge / on-device deployment:

LFM2-VL DSpark speculative decoding is merged, giving 1.66–2.56× decode speedup for edge VLMs, though end-to-end gains are still capped by the vision encoder.

Source: GitHub SGLang Releases

PyTorch becomes the sole backend; TensorRT engine removed.

Why it matters:

NVIDIA removed the TensorRT backend, AutoDeploy, and the separate draft-engine path, standardizing on PyTorch. For operators, fewer config knobs and clearer errors; for the ecosystem, TensorRT-LLM increasingly resembles a PyTorch inference optimization suite rather than a standalone compiler.

Relevance to edge / on-device deployment:

Limited direct edge impact, but DGX Spark Beta adds local/edge small-node support for GPT-OSS-20B/120B and others.

Source: NVIDIA TensorRT-LLM Release Notes

Schedules multi-agent inference on CPU-GPU unified memory.

Why it matters:

The paper argues the bottleneck for on-device multi-agent workloads is not single-request latency but concurrent scheduling and context switching on unified memory; it proposes a task-graph scheduler and memory-aware KV sharing, cutting end-to-end task completion time by 27–41% on Apple Silicon and Snapdragon laptop prototypes.

Relevance to edge / on-device deployment:

Targets on-device multi-agent deployment directly and provides a scheduling prototype for CPU-GPU unified memory.

Source: arXiv 2610.03394

KL divergence misses decision drift in compressed models.

Why it matters:

Comparing dense models with QAT/PTQ-quantized and pruned variants, the authors find low-KL models can still flip decisions at critical tokens, dropping pass@1 on code generation; they propose evaluating compression by task-level flip rate jointly with confidence decay.

Relevance to edge / on-device deployment:

Edge deployments often must use low-bit quantization; this metric predicts real accuracy loss better than perplexity or KL alone.

Source: arXiv 2610.00694

macOS default backend switches from llama.cpp to MLX.

Why it matters:

Defaulting to MLX on Apple Silicon improves first-token latency and sustained throughput for many models, aligns with Apple's native memory model, and cuts unnecessary copies on unified memory; for end users it is an out-of-the-box local performance gain.

Relevance to edge / on-device deployment:

A direct win for Apple Silicon edge users and a sign that local-LLM toolchains are aligning with platform-native backends.

Source: GitHub Ollama Releases

Models that answer by scoring options instead of generating text.

Why it matters:

A Hugging Face blog covers new llama.cpp support for decision models: feed candidate options and the model returns a score for each, useful for classification, preference alignment, and tool selection. Lower latency and more stable output than generating and parsing an answer.

Relevance to edge / on-device deployment:

On-device classification and decision tasks can be done with smaller models and lower latency, fitting local agent tool-routing.

Source: Hugging Face Blog