Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices

Problem: Running large language models on consumer devices is challenging because model weights exceed GPU memory, and existing offloading systems use coarse layer-level scheduling that ignores tensor heterogeneity and adapts poorly to changing hardware loads. Method: ATSInfer performs offloading at tensor granularity, combining static tensor placement with load-aware dynamic transfer and asynchronous CPU-GPU coordination to schedule storage, data movement, and computation. Finding: On representative consumer platforms with dense and MoE models, ATSInfer improves prefill throughput by up to 1.94× and decode throughput by up to 3.29×, while increasing GPU utilization and PCIe bandwidth usage. Why it matters: ATSInfer substantially improves the user experience of local LLM deployment on personal consumer devices by enabling efficient hybrid CPU-GPU inference.