ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM

The problem is that low-bit GEMM is central to efficient ML inference, but very-low-bit execution is poorly suited for conventional CPUs, and fragmented precision regimes make lightweight CPU support selection a first-class design challenge. ExaGEMM proposes a workload-aware codesign framework for CPU-native low-bit GEMM via register-resident LUT execution, requiring only an in-register select/feed mechanism with explicitly modeled cost. Experimental evidence shows ExaGEMM improves latency by 13.29x over software-only baselines across representative ML models and CPU targets, while pruning the candidate space by 99.2% before simulation. This matters because it demonstrates that workload-aware frontier selection is especially important for mixed-precision LLM workloads, enabling efficient CPU-driven ML inference without major hardware redesign.