ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM

The problem is that low-bit GEMM is central to efficient ML inference, but very-low-bit execution is poorly suited to conventional CPUs, and the fragmented design space of precision regimes makes lightweight CPU support selection a first-class challenge. ExaGEMM proposes a workload-aware codesign framework for CPU-native low-bit GEMM using register-resident LUT execution, requiring only an in-register select/feed mechanism as new hardware. Experimental evidence shows ExaGEMM improves latency by 13.29x over software-only baselines across representative ML models and CPU targets, while pruning the candidate space by 99.2% before simulation. This matters because it enables practical, efficient CPU-driven ML inference for mixed-precision LLM workloads, demonstrating that workload-aware frontier selection is critical for low-bit GEMM support.