Notes on Long-Horizon Agents
What actually breaks when an agent has to remember, plan, and recover across hundreds of steps — and the small architectural bets that keep it coherent.
Most agent demos live and die inside a single context window. The hard problems start one horizon later — when the task outlasts the model's working memory and the agent has to carry intent forward without a human holding its hand.
I've spent the last few months building Hermes, an agent that maintains a portfolio's experiment log and drafts release notes. It runs unattended for long stretches, so it has been a useful forcing function for thinking about memory, planning, and recovery as first-class concerns rather than afterthoughts.
Memory is a retrieval problem, not a storage one
The naive move is to stuff everything into the prompt. It works until it doesn't — costs climb, latency creeps, and the model starts to lose the thread in the noise. The better framing is that memory is a retrieval problem: store cheaply, retrieve precisely, and only pay for the tokens that earn their place.
In practice that means a hot working set the agent reads first, a structured index it can scan, and a cold store it drills into only when the cheaper layers miss. The discipline is to stop at the first layer that resolves the query.
Plans should be cheap to throw away
A long-horizon agent that commits hard to a plan is brittle. The world moves; a step fails; an assumption turns out wrong. The agents that hold up treat plans as disposable scaffolding — useful for the next few moves, rewritten freely when reality disagrees.
The goal isn't a perfect plan. It's a system that notices when the plan is wrong fast enough to do something about it.
What I'd tell my past self
- Instrument the agent before you optimize it — you can't fix what you can't see.
- Summarize aggressively; a good summary is worth a thousand stale tokens.
- Make recovery a designed path, not an exception you hope never fires.
None of this is solved. But framing memory, planning, and recovery as the real surface area — rather than the model itself — has been the most useful shift in how I build.