PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

PolyQ addresses the problem that existing low-bit quantization for CPU-based LLM inference offers either coarse operating points or fine-grained mixed precision that is inefficient to execute. The method is a compiler/quantization co-design that assigns per-channel bit-widths from {2,3,4,8,16} and uses a compile-time model compiler to permute and cluster channels into bit-homogeneous blocks with SIMD- and LUT-compatible kernels. On Falcon-H1-3B, Llama2-13B, and Qwen3-32B, PolyQ improves perplexity by 2.4–32.1% over prior methods at a 3-bit target, and compiler layout regularization reduces activation reorder traffic by up to 70.8% on three representative CPUs. This matters because it demonstrates that fractional-bit CPU deployment is practical, predictable, and energy-efficient across diverse edge targets, enabling scalable on-device LLM inference.