ELiTeFormer: An Efficient Transformer for FPGAs

ELiTeFormer addresses the deployment challenge of Transformer blocks in LLMs by co-designing hybrid linear attention with ultra-low-precision ternary projections specifically for FPGAs. The method introduces a novel processing element micro-architecture that eliminates all multiplications in ternary linear projections using bitmasking, avoiding DSP blocks entirely. Experimental results show 10x model weight compression and 12.8x KV cache compression versus LLaMA 3, with block-level simulations achieving 9.6x speedup for FFN and 4.4x for attention, and end-to-end deployment delivering up to 3.9x lower latency and 3.2x better energy efficiency than an NVIDIA A100 GPU. This matters as it demonstrates the first FPGA realization combining linear attention with ternary quantization, proving algorithm-architecture co-design viability for next-generation LLM acceleration.