Filtered by: Inference × Clear all

CODA: Algorithm-Hardware Co-design for Edge Video Diffusion via NMP-Enabled Compute-Cache Operator Disaggregation

Yuanpeng Zhang, YuXuan Wu, Yitong Xiao, Chenhao Xue 2026-07-18
Inference × Transformer Hardware Design Cache

The problem is that deploying Video Diffusion Models on edge devices is too slow for practical local inference due to iterative Transformer-based denoising. The method, CODA, uses algorithm-hardware co-design to disaggregate compute and cache operators via a DIMM-side near-memory engine, reorganizing cache activity into coalesced segments and overlapping compute with cache execution. Experimental evidence shows CODA achieves up to 1.80x end-to-end speedup and 1.74x higher energy efficiency while preserving generation quality. This matters because it enables practical, privacy-preserving video generation on memory-constrained edge GPUs by overcoming the communication and serialization bottlenecks of Cross-Timestep Caching.

PDF

Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent

Lingyun Yang, Yuxiao Wang, Shenghao Liang, Linfeng Yang 2026-07-18
Inference × LLM GPU Hardware Agent

Atrex-Bench addresses the problem that existing GPU kernel benchmarks use synthetic or curated workloads, not production traces. The method samples 30 operators and 440 shapes from full-cluster production inference traces, weighting each by observed GPU time and card-hours. Experimental evidence shows the best vanilla coding agent reaches only ~10% of the hardware roofline on production operators, with much of the pass rate from PyTorch fallbacks. The co-released Atrex-Kernel-Agent (AKA) converts zero-FlyDSL fallbacks into real kernels matching hand-tuned baselines, demonstrating that profile-driven optimization is critical for production readiness.

PDF

ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM

Hyunwoo Oh, Suyeon Jang, Hanning Chen, Sanggeon Yun 2026-07-18
Gem5 Microarchitecture Simulation Inference ×

The problem is that low-bit GEMM is central to efficient ML inference, but very-low-bit execution is poorly suited for conventional CPUs, and fragmented precision regimes make lightweight CPU support selection a first-class design challenge. ExaGEMM proposes a workload-aware codesign framework for CPU-native low-bit GEMM using register-resident LUT execution, requiring only an in-register select/feed mechanism as new hardware. Experimental results show that ExaGEMM improves latency by 13.29x over software-only baselines across representative ML models and CPU targets, with workload-aware frontier selection being especially important for mixed-precision LLM workloads. This matters because it enables efficient CPU-driven ML inference for low-bit GEMM without major hardware redesign, addressing a critical gap in practical deployment.

PDF

NIFA: Nonlinear IMC enhanced FPGA for efficient ML inference

Jiajun Hu, Ruthwik Reddy Sunketa, Lei Zhao, Archit Gajjar 2026-07-18
Design Space Exploration Agentic Architecture Inference ×

The problem is that conventional ReRAM-based analog in-memory computing (IMC) blocks in FPGAs only support static-weight vector-matrix multiplication, limiting efficiency gains for Transformer models that require frequent nonlinear and dynamic matrix-matrix multiplication (DIMM) operations, while the ADCs within each IMC block consume over 70% of its area and power. The method proposes a novel FPGA architecture integrating an ADC-free IMC block that replaces ADCs with analog content-addressable memories (ACAMs) to natively perform nonlinear operations, along with FPGA-aware design-space exploration and efficient mapping to enable DIMM operations for attention computation. Experimental evidence shows that on CNN and Transformer-based benchmarks, the architecture achieves up to 40x and 1.9x higher energy efficiency and 4.1x and 2.5x higher area efficiency, respectively. This matters because it significantly improves FPGA deep learning inference efficiency, particularly for Transformer-based workloads across long input sequences, advancing domain-specialized FPGA design.

PDF