Filtered by: Runtime × Clear all

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

Hyunwoo Oh, Suyeon Jang, Hanning Chen, KyungIn Nam 2026-07-19
Inference LLM Quantization Compiler Low Bit Runtime ×

PolyQ addresses the problem that existing low-bit quantization for CPU LLM inference offers either coarse operating points or fine-grained mixed precision that is inefficient on CPUs. The method is a compiler/quantization co-design that assigns per-channel bit-widths from {2,3,4,8,16} and uses compile-time permutation and clustering to generate SIMD- and LUT-compatible kernels with layout regularization off the runtime path. On Falcon-H1-3B, Llama2-13B, and Qwen3-32B, PolyQ improves perplexity by 2.4–32.1% over prior methods at a 3b target and reduces activation reorder traffic by up to 70.8% on three representative CPUs. This matters because it demonstrates that fractional-bit CPU deployment is practical, predictable, and energy-efficient for scalable edge inference.

PDF

Valinor: Architectural Support for Fast, Energy-Efficient and Programmable Physical Memory Allocation

Konstantinos Kanellopoulos, Spiros Galanopoulos, Konstantinos Sgouras, Vlad-Petru Nitu 2026-07-19
Workload Runtime × Hardware Simulation

The problem is that physical memory allocation in current systems incurs high overhead from minor page faults, which can account for up to 54% of runtime and 40% of system energy in short-lived workloads like serverless functions. The method, Valinor, is a hardware-OS cooperative substrate that introduces a programmable hardware allocation engine executing compact OS-supplied allocation libraries at near fixed-hardware speed. On a BOOM RISC-V soft core running Linux, Valinor accelerates allocation by 17x, improves end-to-end performance by 16%, and reduces energy consumption by up to 8%, with full-system simulation confirming hardware-class performance across six allocation libraries. This matters because Valinor provides the flexibility to support diverse allocation policies and adapt to new hardware conditions while achieving the performance and energy efficiency of dedicated hardware.

PDF