ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM

The problem is that low-bit GEMM is central to efficient ML inference, but very-low-bit execution is poorly suited for conventional CPUs, and fragmented precision regimes make lightweight CPU support selection a first-class design challenge. ExaGEMM proposes a workload-aware codesign framework for CPU-native low-bit GEMM using register-resident LUT execution, requiring only an in-register select/feed mechanism as new hardware. Experimental results show that ExaGEMM improves latency by 13.29x over software-only baselines across representative ML models and CPU targets, with workload-aware frontier selection being especially important for mixed-precision LLM workloads. This matters because it enables efficient CPU-driven ML inference for low-bit GEMM without major hardware redesign, addressing a critical gap in practical deployment.