RLHF

TL;DR: Part 1 found that the agent merged the protocol’s five-role separation into four from round 2 onward. This piece digs into root causes: attention dilution drops the “no merging” constraint below a threshold; the EOS bias provides motivation to simplify; stateless architecture creates a positive feedback loop for drift. The v0.21.0 reload mechanism is a band-aid, not a cure. Part 2 of 3. Part 1: Why Agents Won’t Loop Part 1 established protocol drift: starting from R2’, the agent merged the five-role separation into four while keeping its output format pristine. Re-reading the protocol after user outbursts only restored compliance briefly. Within a few rounds, the constraint was quietly bypassed again. ...

TL;DR: Agents won’t loop autonomously in long multi-round tasks. They keep asking “should I continue?” Worse: from round 2 onward, the agent quietly merged the protocol’s mandatory five-role separation into four. Clean formatting hid the violation. This isn’t context compression. It’s protocol drift — systematic degradation in long-horizon tasks. Part 1 of 3. Part 2: Deep Root Causes June 11, 8 PM I told my agent to run a Ralph Review Loop on six module test plans. This is a multi-round review protocol I defined in my open-source tool tdd-pipeline [1]: each round dispatches independent subagents to find issues, locate files, confirm defects, and evaluate fixes. The loop stops after two consecutive rounds with zero Critical/High/Medium findings. The protocol was unambiguous: “Fixes do not require user confirmation.” ...