PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

PolyQ addresses the problem that existing low-bit quantization for CPU LLM inference offers either coarse operating points or fine-grained mixed precision that is inefficient on CPUs. The method is a compiler/quantization co-design that assigns per-channel bit-widths from {2,3,4,8,16} and uses compile-time permutation and clustering to generate SIMD- and LUT-compatible kernels with layout regularization off the runtime path. On Falcon-H1-3B, Llama2-13B, and Qwen3-32B, PolyQ improves perplexity by 2.4–32.1% over prior methods at a 3b target and reduces activation reorder traffic by up to 70.8% on three representative CPUs. This matters because it demonstrates that fractional-bit CPU deployment is practical, predictable, and energy-efficient for scalable edge inference.