ArchEval: Measuring AI Agents as Computer Architects

ArchEval introduces a benchmark and platform with 20 challenges across CPU cores, memory systems, and accelerators, backed by eight simulators, to evaluate LLM agents on computer architecture design and optimization. The method tests agents under three settings: L1 with full simulator feedback, L2 with simulator source code but no automated workflow, and L3 with no runnable feedback before submission. Experimental evidence shows that with L1 support, all four agents reach or exceed baseline performance, but removing support exposes weaknesses—only GPT-5.5 + Codex remains above baseline in L3 (1.21x geomean, 65% win rate), while the other three fall below baseline. This matters because it frames current agents as useful optimization assistants rather than autonomous architects, identifying needed capabilities like simulator-tool use, calibrated prediction, and pre-feedback judgment.