Measure Your Agent's pass^k Before You Let It Run Overnight

Most teams shipping an agent know one number about it: how often it gets a task right. That number is measured with somebody watching. Your night queue has nobody watching, and it needs a different number.

A recent benchmark put figures on the difference. Across 507 workflows that mutate stored state, twenty attempts each, the top model came out right on 66.50% of single attempts, and right across the entire set of twenty for 47.53% of tasks. A smaller model scored 89.35% on "at least one attempt worked" and 7.50% on "all of them worked". Same system. Two questions. Wildly different answers.

This post is the practical half: how to get that second figure for your own agent in an afternoon.

Step 1: pick a task that writes

Read-only tasks will not show you anything. You want something that mutates: create a refund, reassign a ticket, amend a booking, update a customer record. The failure mode we are hunting only exists where there is state to corrupt.

Step 2: write the assertion first

Before running anything, express the correct end state as code. Two halves, and the second is the one everybody forgets.

def check(db, case):
    order = db.get_order(case.order_id)

    # what MUST be true
    assert order.status == "refunded"
    assert order.refunded_cents == case.expected_cents
    assert len(order.refunds) == 1

    # what must NOT have changed - the half people skip
    assert order.shipping_address == case.snapshot.shipping_address
    assert order.line_items == case.snapshot.line_items
    assert db.count_emails_sent(case.order_id) == 1

That second block is the whole point. The benchmark grades the same way: a run fails on wrong effects, on missing effects, and on extra effects. An agent that issues the refund and also cancels a line item has not done the job.

Step 3: run it k times from a clean slate

import statistics

K = 20
results = []

for i in range(K):
    db = reset_environment()          # a real reset, not a best effort
    run = agent.execute(case.prompt, tools=make_tools(db))
    try:
        check(db, case)
        results.append(True)
    except AssertionError as e:
        results.append(False)
        print(f"run {i}: {e}  (terminated cleanly: {run.ok})")

pass_at_1 = statistics.mean(results)          # share of attempts that worked
pass_at_k = any(results)                      # did any attempt work
pass_pow_k = all(results)                     # did every attempt work

print(f"pass@1={pass_at_1:.2%}  pass@{K}={pass_at_k}  pass^{K}={pass_pow_k}")

Run that across a suite of cases and the three aggregates fall out: the mean of pass_at_1, the share of cases where pass_at_k holds, and the share where pass_pow_k holds.

If you cannot reset the environment between runs, you do not have a test harness. You have production with a positive attitude.

Step 4: read the gap, not the score

Note the terminated cleanly flag in that loop. Print it. You will find failures where it is True - the agent finished, threw nothing, made well-formed calls, and left the data wrong. That is exactly what the benchmark authors reported, and it is why your existing observability cannot cover this: exception rates, tool-call success and latency all look healthy through it.

Three buckets will emerge from the suite:

• cases that pass every run - ship them
• cases that pass sometimes - the dangerous majority, and in the published study they were four fifths of the set for one model
• cases that never pass - route them to a human and move on, they are honest

Step 5: resist the retry reflex

The instinct on seeing a low pass^k is to wrap the call in a retry. Check whether that helps in your data before you build it. In the benchmark, attempts at the same task are plainly not independent - if they were, a model with a 66.50% per-attempt rate would clear twenty in a row about 0.03% of the time, and the measured figure is roughly sixteen hundred times that. Difficulty attaches to the task, not to the dice. Retrying a task the agent cannot really do returns the same wrong result, and in a stateful system each attempt leaves fresh side effects behind.

What to do on Monday

Add three columns to whatever dashboard you already keep: attempts passed, any attempt passed, all attempts passed. Sort your workflows by the third. Everything above the line can run unattended tonight. Everything below it needs a person, a narrower scope, or a different design.

Educational content - not financial advice.

A longer analysis, and the same measurement applied to our own published forecast record, is on our blog.