Issue 6: The Reproducibility War in Quantization

New quants claim broad wins over GSQ-RCO; its lossless claims don't reproduce.

Why it matters:

Unsloth released the Dynamic V3.1 preview algorithm with new DeepSeek-V4.1-Flash-1017 GGUFs and a 35.1B-token evaluation, about 27× their previous benchmarking volume, covering agentic suites like Terminal Bench 2.1 and DeepSWE v1.1. Their conclusion: UD V3.1 beats ISTA's GSQ-RCO in nearly every setting, and GSQ-RCO's claimed 100% AIME25 recovery could not be reproduced. Unsloth also concedes that lower-bit quantization lengthens reasoning chains, so budget more reasoning tokens.

Relevance to edge / on-device deployment:

Squarely on-device: whether a 256GB Mac or a pair of DGX Sparks can run DeepSeek V4.1 (552B plus 196B Engram) depends on which quantization claim holds.

Source: Unsloth Docs

Matryoshka-style per-expert quantization; 1.3–3.7× on a Raspberry Pi 5.

Why it matters:

MatMoE lets the same expert run at multiple precisions (Matryoshka PTQ), removes redundant memory access in the nested representation with floor-and-compensation quantization, and allocates bit-widths per expert from routing scores to meet latency constraints. Implemented on llama.cpp for CPU and GPU, it achieves 4.0–18.3% higher accuracy than state-of-the-art static mixed-precision across three MoE models, with 1.3–3.7× speedup on a Raspberry Pi 5 and 1.2–1.3× on an RTX 4090 and a Jetson AGX Orin.

Relevance to edge / on-device deployment:

A rare MoE quantization paper with real measurements on a Raspberry Pi 5 and Jetson; dynamic per-expert bit-widths are a concrete path for edge MoE deployment.

Source: arXiv 2611.01704

A state-quantization pipeline for linear recurrent models, live in vLLM.

Why it matters:

The paper shows quantization for linear-recurrence models differs from softmax Transformers: extra-state weights and activations are easier to quantize because decayed recurrences develop weaker residual outlier channels, while the real problem is the recurrent state itself, updated repeatedly during decode. LoReS combines value-axis scale grouping, decay-aware mixed precision, and stochastic rounding, delivering robust 8-bit state quantization on Qwen-3.5 9B, Nemotron-H 8B, and Kimi-Linear 48B-A3B, with an implementation in vLLM that yields end-to-end decode speedups.

Relevance to edge / on-device deployment:

Linear models carry a fixed-size state that suits long decode on-device; state quantization closes the last gap for deploying them, and the vLLM integration makes the same path available server-side.

Source: arXiv 2611.01864

A two-parameter closed-form grid; NVFP4 KLD drops up to 67%.

Why it matters:

Block-scaled lookup-table formats, from NF4 and BOF4 to the 3-bit LUT instruction in NVIDIA Rubin, ship grids copied from numeric types or assumed distributions; the authors prove the best fixed grid varies with bit-width (BOFS at 2 bits, NF at 4–5, INT at 6). SHq parameterizes the whole grid family with two degrees of freedom: a companding strength α interpolating spacing from uniform to float-like, and an offset c that reuses the FP8 block-scale sign bit to mirror the grid, up to doubling distinct values per block. It only changes table contents, drops into any such format, achieves the lowest KL divergence in every setting at 3–4 bits, beats the best fixed grid by up to 34%, and at 4.5 bits cuts NVFP4's KLD by 41–67% while improving downstream accuracy.

Relevance to edge / on-device deployment:

LUT-style formats are the mainstream implementation for on-device quantization; a grid-only change can be absorbed by llama.cpp-class stacks at low cost.

Source: arXiv 2611.02966

4-bit quant keeps accuracy but turns one-shot solves into loop-based repair.

Why it matters:

On OJBench, comparing BF16 with 4-bit AWQ and GPTQ across Qwen2.5-Coder 7B/14B/32B and Devstral-Small-2507 over three seeds, final accuracy shows no statistically significant loss from quantization, yet behavior shifts substantially: AWQ cuts Qwen-7B one-shot solves from 17 to 0 while raising loop-based solves from 10 to 25, and failure-mode distributions move under quantization. The takeaway: quantization robustness for coding agents should be assessed via solution trajectories, tool use, and failure modes, not final scores alone.

Relevance to edge / on-device deployment:

On-device coding agents are mostly small models wrapped in execute-and-repair loops; this paper offers an evaluation lens closer to real usage than score deltas.

Source: arXiv 2611.02368

MLX backend gains DFlash2; faster prefill for gemma/flint/laguna.

Why it matters:

v0.40.12 (Oct 27) adds DFlash2 speculative-decoding support to the MLX and MLX-C engines, speeds up prompt processing for gemma/flint/laguna models, and bumps the llama.cpp dependency to b11901. v0.40.13 (Oct 29) then brings Liquid AI's LFM3 family, including lfm3-audio TTS and lfm3-omni, to Linux and Windows.

Relevance to edge / on-device deployment:

With Ollama defaulting to the MLX engine on Apple Silicon, DFlash2 fills the speculative-decoding gap on that path, and Mac local inference decode speed benefits directly.

Source: GitHub Ollama Releases