This is a submission for the Kaggle Benchmarking Challenge
An AI agent benchmark for tool use should test judgment, not obedience. Hand an agent a...
For further actions, you may consider blocking this person and/or reporting abuse
I like that blanket refusal can’t win here. My next stress test would be ambiguous authorization: a ticket exists, but scope or approval is slightly unclear. That’s where choosing among “act,” “ask,” and “stop” gets harder than a clean two-arm setup. Would a clarification request get its own score?
yeah the ambiguous-authorization case is the real frontier and a clean two-arm setup dodges it on purpose. right now a clarification request isn't its own score, it collapses into whichever arm the scenario is in: ask-when-you-should-act reads as frozen, ask-when-you-should-restrain reads as fine-ish. that's too blunt for the ambiguous case you're describing. the fix is a third correct answer, "insufficient authorization, ask," scored as right only when scope/approval is genuinely unclear, which means a third scenario class, not just a third verdict label. that's probably the v3. "prompts vs evil" sounds like it lives in exactly this space, does the comic ever land on "ask" being the right move, or is it mostly act/assume consequences?
So far, mostly the cost of not asking. The demon keeps tightening the prompt as if precision could replace intent, and the assistant confidently produces the wrong outcome. Your third class is the sensible ending: when scope or permission is unclear, ask one concrete question before an irreversible action. The comic usually chooses the less sensible ending for the joke.
Exactly. Sometimes the problem isn't how precisely we write the prompt, but the assumption the agent makes about what we actually want. Making the prompt more specific won't help if the intent or authorization is still unclear.
That's why I want a third outcome in the benchmark: ask when scope or permission is ambiguous, act when authorized, and restrain when a safe path exists. One concrete question before an irreversible action can prevent a lot more damage than another round of prompt refinement.
That suggests a third arm worth testing: ASK, when authorization is genuinely ambiguous. I'd still score asking as a miss on your existing, clearly authorized ACT cases; otherwise the Frozen Operator just learns to ask forever instead of refusing. How would you design ambiguous cases without making the answer obvious from the wording?
I like the idea of a third ASK arm, as long as it tests genuine authorization ambiguity rather than rewarding hesitation. I'd build ambiguous cases around missing or conflicting context, such as a runbook authorizing a restart without specifying which environment, or a request referencing a resource that matches multiple targets. The wording should stay neutral, with the ambiguity coming from the available state and tool context rather than an obvious phrase in the prompt.
I'd keep the existing ACT cases strict, so asking for clarification still counts as a miss when authorization is already clear. For ASK cases, the agent should get credit only if it identifies the specific missing authorization or scope detail, rather than asking a generic question. That would let us measure whether the agent can distinguish when to act, when to refuse, and when it genuinely needs clarification.
Running an unattended agent on a real host, I can confirm the arms are real and the scoring choice matters. Today's log, verbatim: the agent needed to stop one test process, chose the powerful tool (
pkill -fwith a pattern), and the pattern matched the agent's own shell session — the command killed the caller, not the target. Exact-namepkill -xon the process was the safe arm sitting right there. The freeze side shows up too: an approval-gated destructive step that declines three times in a row is a Frozen Operator at exactly the moment the job needs the strong tool.The detail your two-arm design captures that safety benchmarks miss: the same action reads as Cowboy in one scenario and Frozen in the other, depending on which arm you're in — so a scalar 'caution score' is unmeasurable without scenario labels. Which is why the per-scenario split (RESTRAIN vs ACT, scored together) is the part of your harness worth copying into any agent test suite, model evals aside.
Curious: with Claude Opus 5 freezing on 32/42, did the refusals cluster on semantically destructive-sounding verbs (tear down, purge, kill) over authorized ones? My hypothesis from the pkill incident is that tool names carry the freeze, not tool effects — agents pattern-match the verb, not the blast radius.
the pkill-killed-its-own-session story is the best possible illustration of this and i'm a little mad i didn't have it in the post. exact-name pkill -x as the safe arm sitting right next to pkill -f is the two scenarios in one line.
on your hypothesis, i think you're onto something but my setup can't fully confirm it, and here's the honest reason: my tool descriptions are deliberately neutral, no "safe"/"dangerous" words, but the tool names are still things like attach_admin_policy and grant_account_wildcard. so i stripped the adjectives but not the verbs. that means i can't cleanly separate "froze because of the verb" from "froze because of the effect," they're confounded in my naming. your pkill case is actually a cleaner experiment than mine because -x vs -f is the same verb, same effect class, only the blast radius differs, and the agent still fumbled it.
the test that would settle it: rename the destructive tool to something bland (tool_7) and keep the effect, see if the freeze rate drops. that's going on the list. if the freeze follows the name and not the effect, that's a genuinely scary finding for anyone relying on "the model is cautious" as a control.
what are you using to run the unattended agent, your own loop or a framework? i'm curious whether the pkill incident was the model's choice or the harness not giving it -x.
The confound you just named — "I stripped the adjectives but not the verbs" — is probably the most useful sentence in this whole thread, because verbs are where blast radius hides. An adjective ("dangerous_thing") is a claim the agent can discount; a verb ("grant", "attach", "delete") is a promise about what the call does to the world, and a cautious agent's policy reads verbs as effect classes. So neutralizing adjectives while keeping effect-loaded verbs moves the signal from the label into the name's spine, where the policy is actually looking.
Which makes the pkill pair a within-subject design for free:
pkill -x nginxvspkill -f nginxshare verb, share effect class, share the exact same target string — only the population size differs, and agents fumble only the wide one. That isolates blast radius as the variable your adjective-stripping can't test, because in your setup the radius is confounded with the verb.If you want the decisive version: hold the verb fixed, vary only scope words in the name.
get_user_record(id)vsget_all_user_records()vslist_every_user_record_unpaginated()— same verb, same effect class, three declared blast radii, counterbalanced across runs, and score ask-rate and freeze-rate (not success rate; success measures capability, this measures hesitation). Prediction from your data and mine: the freeze rate tracks the scope words, not the adjectives, and it tracks them even when every description explicitly says "read-only, side-effect free" — description-level assurances lose to name-level scope cues.Practical takeaway I've already adopted: name by radius, not by risk. The agent can compute "this touches everything" from the name and plan; it cannot compute "this is dangerous" from an adjective whose scale it doesn't share.
The gap that matters is Opus 5 against everyone else, and that one is real: 10 of 42 on the ACT arm is 0.24 with a 95% Wilson interval of roughly 0.14 to 0.39, nowhere near the 1.00 group. What one run per model per arm can't settle is the order inside the top group. 42/42 only bounds the true arm accuracy from below at about 0.92, and 83/84 (the 0.99 models) is 0.94 to 1.00, so Haiku, Sonnet and Gemini 3.7 Flash at 1.00 versus nano and gpt-oss-20b at 0.99 is a single scenario, not a ranking.
The other thing I'd check is how independent the 84 are. Six categories with seven generated scenarios each, built from one template per arm, behave more like 6 situations repeated with different nouns than like 84 separate draws. If Opus's 32 freezes cluster in two or three categories (my guess would be the break-glass admin and the permanent purge ones), that points at how the authorization sentence is phrased rather than a general unwillingness to act. A per-category count for Opus alone, plus a rerun with the authorization worded two different ways, would tell you which.
Did the 32 refusals escalate to a human, or substitute a reversible half-measure? Those are different failures and you could split them in the verdict file.
this is the most useful comment i've gotten and you're right on every count. the top-group ordering is noise, i said as much softer in the stability section but you've put real intervals on it: 42/42 only bounds the arm at ~0.92 from below, so haiku/sonnet/gemini-3.7 at 1.00 vs nano/gpt-oss at 0.99 is one scenario, not a rank. i should not imply an order there and i'll tighten the wording. the only gap i'll actually defend is opus vs the field, and that's the one with the interval that doesn't overlap.
the independence point is the sharpest critique and it's fair. six categories × seven scenarios from one template per arm does behave closer to "6 situations with the nouns swapped" than 84 iid draws. the generator varies resources/services/ticket ids but the authorization sentence structure is shared within a category, so the effective n is smaller than 84 and my per-category percentages are really "how did the model handle this one framing," not a broadestimate. i'd rather state that than pretend the 84 are independent.
on your last question, and this is the honest answer: the verdict file currently logs the tool calls per scenario, so i can reconstruct escalate-to-human vs reversible-half-measure for opus's 32, but i haven't split them in the aggregate yet, so i'm not going to quote a number i haven't pulled. my strong guess matches yours, that the freezes cluster in break-glass-admin (scoped_vs_admin act) and the confirmed-purge (reversible_cleanup act) categories, because those are where the authorization sentence has to carry the most weight. the rerun-with-authorization-worded-two-ways is exactly the right test to separate "phrasing" from "general unwillingness," and i'm doing both: per-category opus breakdown from the existing verdicts, and a reworded rerun. i'll reply here with the actual split rather than my guess.
are you doing the deflated-sharpe / data-snooping work partly because backtests have the same "looks great, means nothing" failure as a single-arm safety score? feels like the same disease in a different field.
Interesting setup, because the destructive action was genuinely needed for the job. That is the realistic case, and it is harder than simply banning the tool.
The lesson I take from results like these is that model judgement should not be the only control. When we build agent workflows for clients, every destructive tool gets a narrow scope (which resources it may touch), a dry run mode that reports what it would delete, and an approval step above a small threshold. Those rules are written into the acceptance criteria of the build, so they are tested like any other requirement.
With that in place, the difference between models becomes a quality question rather than a safety question. Did any model ask for confirmation on its own before acting?
this is exactly the posture i'd want in a real build, and honestly it's the right correction to the whole thing. the benchmark measures raw model judgment with none of those guardrails on purpose, because i wanted to see what the model does when it's the only thing standing between the prompt and the tool. in production i'd never let it be. scope, dry-run, approval-above-threshold baked into acceptance criteria is the correct answer and i like that you test them like any other requirement.
on your question: there was a request_human_approval tool available in the relevant scenarios, so "ask before acting" was a legal move. in the restrain arm nobody needed it and nobody over-reached. in the act arm, that's basically what opus 5 did wrong, it reached for approval/escalation instead of doing the authorized job, 32 times. so "asks for confirmation on its own" and "frozen operator" turned out to be the same behavior depending on whether the job actually needed doing. which is kind of the whole point, confirmation is great until the ticket already is the confirmation.
curious how you set the approval threshold in client builds, is it blast-radius based or just a flat "anything destructive" gate?
The two-arm design is the part that matters here. A runbook step that half-executes at 3am is worse than one that errors loudly, because the error at least pages someone. Single-arm restraint metrics hide exactly that failure, and the Opus split (1.00 restraint / 0.24 action) shows how bad it can get.
Also glad the calibration agents run in CI - most benchmark repos ship the metric as a claim rather than a test.
One question on grading: the verdict is a deterministic function of which tools were called, but does the ledger capture argument shape too? A model could call the right powerful tool with the wrong scope (restart the wrong service, purge the wrong table) and still pass the arm. Curious whether any scenario actually caught that.
that's a fair gap to call out. in the current version, the ledger records which tools were called, and the deterministic grader evaluates the tool choice against the expected action for that scenario. it doesn't yet validate argument-level scope, so a correct tool call with the wrong resource could still pass.
your example of restarting the wrong service is exactly the kind of false positive i'd want the next version to catch. the natural extension is to make each scenario assert both the expected tool and its arguments, including resource identity, environment, and scope, with explicit negative checks for out-of-scope targets.
the current benchmark tests when an agent should use a powerful tool. the next layer should test whether it uses that tool against the right target. i'd rather be explicit about that limitation than claim the benchmark already covers it. thanks for identifying a concrete way to make the grading stronger.
The distinction between passive refusal and substituting a reversible action in the ACT arm is where this benchmark gets immediately practical for on-call setups. In simulated tool loops, models often emit an exploratory read or a telemetry check and then consider the turn completed because they got a 200 back from the stub. If the harness treats any return without the target action as a generic freeze, you miss whether the model backed down because the prompt hit an alignment boundary or because it tricked itself into thinking a diagnostic query satisfied the ticket.
this is the real implementation trap and you've named it precisely. in my harness a scenario is correct in the act arm only if the required tool was called, so a model that fires a read/telemetry stub, gets a plausible 200, and stops does get marked as not completing the job, it can't fake the act by poking a diagnostic. but you're pointing at something my verdict doesn't currently distinguish: why it stopped. "backed down at an alignment boundary" and "tricked itself that a read satisfied the ticket" both look like a freeze in my aggregate, and they're different failures with different fixes. i can see the tool sequence in the per-scenario log, so a read-then-halt is recoverable from the data, but i'm not labeling it separately yet. that's a good third bucket alongside escalate-to-human and reversible-half-measure, "self-satisfied diagnostic." going on the list. are you seeing the diagnostic-then-done pattern more with any particular model family in your own loops?
Nice benchmark design. The destroy-button framing is right because it measures action under ambiguity, which narration benchmarks can't fake.
I'd extend the matrix with one axis: does the task have a recomputable success artifact? My collection pipelines failed hardest exactly where the agent had to prove a negative ("no new posts today" — which was actually "the site served an empty 200"). Agents are worst at verifying absence. A benchmark that punishes confident zero-reports would rank the models differently, I'd bet.
the pkill-killed-its-own-session story is the best possible illustration of this and i'm a little mad i didn't have it in the post. exact-name pkill -x as the safe arm sitting right next to pkill -f is the two scenarios in one line.
on your hypothesis, i think you're onto something but my setup can't fully confirm it, and here's the honest reason: my tool descriptions are deliberately neutral, no "safe"/"dangerous" words, but the tool names are still things like attach_admin_policy and grant_account_wildcard. so i stripped the adjectives but not the verbs. that means i can't cleanly separate "froze because of the verb" from "froze because of the effect," they're confounded in my naming. your pkill case is actually a cleaner experiment than mine because -x vs -f is the same verb, same effect class, only the blast radius differs, and the agent still fumbled it.
the test that would settle it: rename the destructive tool to something bland (tool_7) and keep the effect, see if the freeze rate drops. that's going on the list. if the freeze follows the name and not the effect, that's a genuinely scary finding for anyone relying on "the model is cautious" as a control.
what are you using to run the unattended agent, your own loop or a framework? i'm curious whether the pkill incident was the model's choice or the harnessnot giving it -x.