LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget

LongStraw addresses the growing gap between inference context lengths and RL post-training, where inference systems handle million-token contexts but post-training typically stays at 256K tokens or below, especially critical for AI agents with long trajectories. The method is an architecture-aware execution stack for million-token RL post-training under a fixed GPU budget, using GRPO to evaluate shared prompts without autograd, retain only model-specific state, and replay short response branches one at a time to reduce the live training graph. On eight H20 GPUs, LongStraw completes grouped Qwen scoring and response backward at 2.1M positions for groups of 2 and 8, with only 0.21 GB added peak memory per group size increase, and a separate stress test reaches 4.46M positions; on 32 H20 GPUs, it validates the end-to-end execution path for a 2.1M-token prompt across all 78 layers of GLM-5.2. This matters because it demonstrates execution capacity for million-token RL post-training under fixed GPU budgets, though the abstract does not disclose complete training correctness due to detached prompt state and incomplete distributed paths.