Can LLMs Perform Deep Technical Comprehension of Computer Architecture Papers?

Nishant Aggarwal, Ayushi Dubal, Sreeraj Kannakarankodi, Ian McDougall 2026-07-16

The problem is that existing LLM evaluations focus on summarization rather than deep technical comprehension, which requires structured critique identifying core mechanisms, buried assumptions, and cross-paper contributions. The method introduces Gauntlet, an open-source pipeline using five independent expert-persona reviewers and an adversarial synthesis stage to analyze computer architecture papers. On 20 ISCA 2025 and HPCA 2026 papers, evaluators preferred Gauntlet over human analysis in 15 of 20 comparisons, with significant advantage on Critical Rigor and only vanishing on Calibration, while humans won on trust and usefulness rather than depth. This matters because Gauntlet demonstrates that multi-agent LLM pipelines can outperform humans in deep technical critique, and the released analyses, scores, and rubric provide a community resource for advancing automated paper comprehension.

PDF