Long-Horizon Agents: Why Multi-Turn Reasoning Breaks and the Practical Training Tricks That Fix It

Almost every modern language model looks impressive on a two-step demo. You ask it to check a database or summarize a document, it calls the right tool, formats the answer, and looks like an autonomous engineer.

The illusion falls apart the moment you ask that same model to complete a 30- or 50-step workflow: diagnosing a failing Kubernetes cluster, navigating a multi-file pull request, or running a 48-hour industrial simulation.

Around step 10, the agent makes a small typo in a terminal command. By step 15, it misinterprets a confusing error message. By step 25, it has forgotten its original plan and begins arguing with its own terminal history. By step 35, its context window is overflowing with thousands of noisy stderr lines, and the run crashes in an expensive loop.

This failure is not solved by simply adding a bigger system prompt. It is a systems problem in how agents explore, learn from mistakes, and receive reinforcement learning signals.

In this guide, we break down why multi-turn agents collapse, why standard RL algorithms burn thousands of GPU hours learning nothing, and the pragmatic tricks—proven in recent frontier reasoning models (such as MiniMax-M1 and DeepSeek-R1)—that turn fragile single-turn models into reliable long-horizon agents.

Start here

Learn why single-turn benchmarks lie, why standard GRPO policy updates produce zero learning on long tasks, how the 75% pass-rate rule cuts GPU training costs in half, why smooth gradient clipping (CISPO) saves words of doubt, and how to train through real agent harnesses without building fragile mocks.

• Horizon (T): The number of sequential interaction turns (actions and tool responses) an agent takes to finish a task.
• Markovian State Drift: When an early mistake leaves the environment in an unfamiliar, broken state, causing subsequent actions to fail at much higher rates.
• Zero-Gradient Trap: When all rollouts in a training group either fail completely (0/8) or succeed easily (8/8), producing zero mathematical advantage to update the model.
• The 75% Rule: A dynamic curriculum trick that removes tasks once the model solves them over 75% of the time, saving up to 50% of rollout compute.
• CISPO: An RL objective that clips importance-sampling weights instead of dropping tokens, so rare exploration tokens like 'Wait' and 'Let me verify' keep their gradient.
• Gated Rewards: Binary pass/fail checks (strict JSON format, no conversational chatter) that an answer must pass before receiving any quality score.
• Polar Proxy: A non-invasive proxy that records tool traffic between real agent wrappers (like Claude Code or mini-SWE-agent) and the model during training.

1. The Step-20 Cliff: Why Agents Break in Practice

Why do agents fail over long horizons? The basic math seems simple: if an agent has a 95% chance of making the right tool call on any single turn, its probability of getting 20 turns right in a row is:

0.95²⁰ ≈ 35.8%

At 50 steps, that drops to 7.7%. That means more than 9 out of 10 runs fail, even with a seemingly high 95% single-step accuracy.

If every step were independent, success would be p to the power of the number of steps: 95% per step leaves 35.8% at 20 turns and 7.7% at 50. Real environments are worse.

But real-world software engineering and industrial environments are far worse than this textbook formula. Steps are not independent coin flips; they are tied together by environmental state:

1. The Domino Effect: On step 6, the agent passes an invalid flag to a bash command or deletes a necessary lockfile.
2. Entering Unfamiliar Territory: The system returns an unexpected error traceback or an exit code 127. Because base models are mostly trained on clean, successful tutorials, the model has rarely seen this exact broken state.
3. The Hallucination Spiral: Instead of stepping back and running git status or inspecting the filesystem, the agent invents an excuse, tries to run a more complicated command to patch its mistake, and creates a circular error loop.

In practice, once an agent makes an unhandled error, its chance of making a second error on the very next step jumps from 5% to over 60%. This is why long-horizon tasks fail at an exponential cliff.

2. The Zero-Gradient Trap: Why 90% of Agent Rollouts Learn Nothing

When teams try to fix this by applying standard reinforcement learning (like GRPO or PPO), they hit an immediate wall: the cold-start problem.

Reinforcement learning works by comparing multiple attempts (rollouts) for the same problem. The attempts that score better than average are reinforced; the attempts that score worse are discouraged:

If a task is too hard and the model never succeeds by chance, all attempts get a score of 0. The difference between attempts is zero, the gradient is zero, and the model updates nothing. You can burn tens of thousands of dollars on H100 clusters, but the model will not take a single step forward.

GRPO compares 8 attempts at the same task. When all fail or all pass, every advantage is zero and the update learns nothing; only mixed groups carry signal.

3. The 75% Rule: Dynamic Difficulty Filtering

One of the most practical engineering insights in recent online RL literature is the 75% pass-rate cutoff.

As training progresses, tasks that were once difficult become easy. A coding problem that the model solved 1 out of 8 times at step 100 might be solved 8 out of 8 times by step 1,000. Once a model solves a task every time, that task ceases to teach it anything.

How the trick works:

• Before each training stage, run the current checkpoint across the task pool and measure the pass rate (e.g., across 8 samples).
• Filter out any task solved more than 75% of the time: The model has already mastered it. Keeping it in the training batch only dilutes the gradients and wastes rollout compute.
• Focus strictly on the 20%–75% struggle zone: This is where the variance between good and bad attempts is highest, providing the strongest reinforcement learning signal.

In practice, this simple automated filter cuts rollout inference compute by over 50%. Instead of wasting GPU cycles generating answers to tasks the model already knows, every dollar of compute goes toward pushing the capability frontier.

The chance that an 8-attempt group contains at least one pass and one fail. Tasks below 20% or above 75% waste most rollouts, which is why the 75% filter pays off.

4. CISPO: Preserving the Words of Doubt That Make Reasoning Work

Standard RL algorithms (like PPO and GRPO) enforce a mathematical safety clamp: if a token’s probability changes too drastically in a single step, the algorithm clips the gradient to zero to prevent the model from destabilizing.

For standard text generation, this clipping is fine. But for multi-turn reasoning agents, it causes a silent disaster:

In a reasoning model, the most valuable tokens are the words of doubt: “Wait, let me check the exit code...”, “However, the file was not created...”, or “Hold on, that contradicts step 2.”

An SFT model almost never produces these tokens. They start with tiny probabilities (e.g., 0.001%). When reinforcement learning starts discovering that hesitating and checking intermediate outputs leads to higher rewards, the probability of these tokens begins to rise rapidly.

The GRPO Trap: Because these tokens experience the biggest percentage change, standard GRPO marks them as outliers and clips their gradients to zero. The model is blocked from learning how to doubt itself.

The CISPO Fix: Instead of clipping out-of-bounds tokens to zero, the CISPO objective (introduced by MiniMax-M1) clips the importance-sampling weight instead of the update: the weight is capped and treated as a constant, but every token still contributes its gradient. Training stays stable, and the learning signal on rare exploration tokens is never switched off. The result: agents learn to self-correct and backtrack significantly faster than under vanilla GRPO.

PPO/GRPO clipping drops tokens whose probability ratio leaves the trust region, which hits rare words like 'Wait' first. CISPO caps the importance weight but keeps every token's gradient.

5. Gated Rewards: Binary Filters Before Scoring

A classic trap when building multi-turn agents is using a single blended reward score: giving the model 0.7 points for a partially working answer, 0.2 for nice explanation formatting, and 0.1 for being polite.

Models exploit blended scores ruthlessly. In coding and function-calling environments, a model trained on soft scores quickly learns a catastrophic habit: conversational chatter around structured output.

Instead of emitting raw, parseable JSON:

{"action": "read_file", "path": "src/main.py"}

The model starts outputting polite commentary:

Certainly! Here is the tool call you requested to inspect the file:
{"action": "read_file", "path": "src/main.py"}
I hope this helps!

To a human reviewer, this looks helpful. To a production execution pipeline or JSON parser, it is an immediate syntax crash.

The Gated Reward Architecture:

• Gate 1 (Strict Formatting): Did the model output strictly valid JSON/code with zero conversational chatter outside the payload? If NO → Reward = 0. Stop immediately.
• Gate 2 (Language & Environment Invariants): Did the model switch languages mid-sentence or run a prohibited command? If YES → Reward = 0. Stop immediately.
• Gate 3 (Execution Verification): Did the unit tests or environment transition succeed? Only outputs that pass Gates 1 and 2 are evaluated on task performance.

Gated rewards: format, invariants and execution are pass/fail checks in front of the task score, so a chatty but unparseable answer earns exactly 0.

This binary gate eliminates format drift within the first 50 steps of training. The model learns that no amount of eloquence can compensate for a broken interface contract.

6. Adaptive Length Penalties: Curing the Infinite Thinking Loop

Once you give a model the ability to reason and self-correct, a new pathology emerges: verbose procrastination.

The model realizes that writing more reasoning tokens slightly improves its chances of catching an error. Within a few hundred training steps, average reasoning length explodes: the agent begins spending 4 minutes and 3,000 tokens “thinking” about whether to run ls or pwd.

A naive penalty (e.g., subtracting 0.001 points per token) ruins the model: hard multi-step problems genuinely require long chains of reasoning, and a flat penalty causes the model to give up prematurely.

The solution is an adaptive length penalty scaled by task difficulty:

• For easy tasks (Pass Rate > 80%): Apply a strong length penalty. If a task is straightforward, the model should execute it cleanly in 1–2 turns without writing a dissertation.
• For difficult tasks (Pass Rate < 20%): Reduce the length penalty toward zero. Give the model the full token budget to explore hypotheses and recover from failures.

This keeps simple tool calls lightning-fast while preserving deep multi-turn persistence on genuine edge cases.

7. The Polar Proxy: Train Through Real Agent Harnesses

How should you wire an agent into an RL training loop?

Most research labs make the mistake of rewriting the agent harness inside Python: building a simplified mock of the terminal, the filesystem, and the tool definitions. This is a dead end. Within weeks, your real production agent framework (whether it’s Claude Code, Codex, OpenHands, or mini-SWE-agent) evolves, and your training environment becomes an obsolete toy that no longer matches reality.

The production pattern (used in NVIDIA’s Polar and adapted by industrial labs) is to train through a transparent proxy:

With a proxy in place, the real agent harness believes it is talking to a standard OpenAI-compatible API. The proxy intercepts every prompt and completion, records the exact multi-turn trajectory, and forwards it to the RL training cluster. You train the model in the exact same environment where it will operate in production.

Training through a transparent proxy: the real harness talks to what looks like a normal model API, while the proxy records each trajectory for the RL trainer.

8. Multi-Turn Memory: Writing Is Harder Than Reading

As interaction horizons stretch past 30 turns, an agent cannot rely solely on raw context stuffing. If an agent dumps every terminal output into the prompt, the context window quickly bloats past 60,000 tokens, causing inference latency to spike and attention heads to suffer from “lost in the middle” syndrome.

Long-horizon agents require explicit memory tools (e.g., scratchpads, key-value stores, or state summaries). However, empirical training reveals a crucial asymmetry:

Reading memory is easy; writing memory is where agents fail.

Retrieving a stored fact via search is a standard retrieval task that LLMs handle well. The real danger is memory corruption during writes:

• An agent records duplicate facts under slightly different keys, confusing future lookups.
• An agent mistakes a temporary intention for a permanent state (e.g., recording “database migration failed” as a permanent truth, even after a subsequent step fixed it).
• Conflicting records accumulate over turns, causing the agent to hallucinate about whether a task was completed.

In state-of-the-art post-training, memory skills are trained with separated reward channels: one benchmark evaluating retrieval accuracy (reading), and a dedicated multi-turn benchmark evaluating memory hygiene (writing only unambiguous, non-redundant, verified state diffs).

How much do these recipes help together? The original article includes an interactive toy model in which per-step reliability starts at 94%, 97% or 99% and decays slightly with every turn. In that model, the combined recipe keeps about 67% of 20-turn tasks and 17% of 50-turn tasks on track, against about 5% and under 1% for a prompted model. The numbers are assumptions, not measurements; the point is the shape: better training moves the cliff later, it does not remove it.

A toy model with assumed per-step reliability (94%, 97%, 99%) that decays slightly each turn. Better recipes move the cliff later; they do not remove it.

9. Practical Blueprint for 50-Step Autonomy

If your team is deploying or post-training an agent intended to run reliably past 20 turns, here is the pragmatic systems checklist:

1. Stop trusting single-turn leaderboards: A model with a 95% tool-calling score on Berkeley Function Calling can still have a sub-10% completion rate on real 30-step workflows. Benchmark at your actual production horizon.
2. Enforce the 75% difficulty filter: Discard training tasks once your checkpoint achieves >75% pass rate. Focus 100% of your GPU budget on the learning zone where success is between 20% and 75%.
3. Use CISPO instead of naive clipping: Never use rigid PPO/GRPO clipping that zeroes out gradients on rare tokens. Give self-doubt and backtracking tokens room to grow.
4. Put gated filters in front of rewards: Fail with zero reward if the model outputs conversational preamble, markdown fences, or malformed JSON. Clean interfaces must be a non-negotiable prerequisite.
5. Scale length penalties by task difficulty: Punish verbose reasoning on simple tasks; allow breathing room on genuinely hard tasks.
6. Train through real harnesses with proxies: Don’t waste months maintaining mock simulators. Use a transparent proxy to train directly against your production agent CLI in isolated containers.

Research executed September 2026. Primary systems references: MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention (MiniMax, 2025), RLVR & Reward Hacking, Multi-Reward RL in Agent Environments, and g factor Private Task Gyms.

Applying this to your own models? We build private RL training environments with deterministic verifiers for teams' own workflows. Private training environments.

Originally published at g-ftech.com.