HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

HeteroMosaic addresses the problem that existing LLM runtimes underutilize heterogeneous resources in edge SoCs by making coarse device-level decisions or optimizing operators in isolation. The method introduces a heterogeneity-first scheduling framework that uses a heterogeneous roofline model, dependency-preserving micro-batches, and trace-guided co-optimization of scheduling and device allocation. On a balanced AMD Ryzen AI platform, HeteroMosaic achieves up to 1.73X speedup over an iGPU baseline, 1.78X over an NPU baseline, and 2.05X over frameworks like llama.cpp, while reducing energy by up to 45.3%. This matters because it demonstrates significant performance and energy efficiency gains for edge LLM inference by effectively exposing and exploiting cross-accelerator execution opportunities on unified-memory platforms.