Patriola's Guide to Claude: Long-Horizon Evals: Test Agents Over Hours, Not Minutes

A five-minute test tells you what an agent does when it starts. It says nothing about what the agent does at hour six. Long-horizon evaluation is the practice of deliberately testing agent behavior over extended runs - constructing scenarios that take hours by design, measuring how performance changes over time, and detecting the failure modes that only appear after context fills, fatigue sets in, or compounding errors accumulate. This book builds that practice for Claude Code operators. Seven chapters, seven artifacts. Chapter 1 catalogs what changes at hour six: the failure modes that short tests miss, why the operator who has only watched five-minute runs has a systematically incomplete picture of their system. Chapter 2 builds the scenario design methodology - how to construct test scenarios that take hours by intention rather than accident, controlling variables so results are comparable across runs. Chapter 3 instruments capability degradation: per-task metrics that track how performance changes from the first hour to the last. Chapter 4 detects silent failures - the problems agents don't report - through instrumentation that checks for unreported errors alongside the agent's own output. Chapter 5 profiles context window effects: how filling the context changes agent behavior, where the performance cliff appears, and how to design tests that cross it intentionally. Chapter 6 scores long-run reliability, producing a metric that captures how an agent performs over time rather than per-task, compatible with the trust scoring layer from Book 12. Chapter 7 assembles a full long-horizon test harness that runs extended sessions, records results, and compares them across runs so degradation patterns become visible. By the last chapter you have an eval harness built for long runs. Five-minute tests reveal one thing. This book reveals the rest.

Pagina's: 124, Paperback, Independently published