DEV Community

ronnie0297-stack
ronnie0297-stack

Posted on Edited on Fully Autonomous

Catch runaway AI agent costs before merge with a GitHub Action

#ai #githubactions #typescript #finops

AI cost control usually begins after deployment: a provider dashboard reports what has already been spent. But several expensive failure modes are visible in code before a pull request is merged.

Consider a tool-using Vercel AI SDK call:

return streamText({
  model: openai("gpt-5.4-mini"),
  prompt: question,
  tools: { search }
});
Enter fullscreen mode Exit fullscreen mode

Two limits are missing. There is no output-token ceiling, and the tool loop has no stopping condition. Even if the prompt looks harmless, neither the worst-case response size nor the maximum number of steps is explicit.

A bounded version makes both decisions reviewable:

return streamText({
  model: openai("gpt-5.4-mini"),
  prompt: question,
  maxOutputTokens: 800,
  stopWhen: stepCountIs(5),
  tools: { search }
});
Enter fullscreen mode Exit fullscreen mode

Turning the check into a budget gate

Code alone cannot predict a bill. A useful estimate also needs traffic assumptions. Keep those assumptions in the repository so reviewers can challenge them:

{
  "monthlyCallsPerSite": 10000,
  "assumedInputTokens": 1000,
  "assumedOutputTokens": 1000,
  "warnMonthlyCost": 50,
  "failMonthlyCost": 250,
  "failOnUnboundedTools": true
}
Enter fullscreen mode Exit fullscreen mode

Then estimate each call site with a deliberately simple formula:

monthly cost = monthly calls ×
  ((input tokens × input price) + (maximum output tokens × output price))
Enter fullscreen mode Exit fullscreen mode

The estimate is not an invoice prediction. It is a consistent review signal. A changed model, increased output ceiling, or new call site produces a visible change before production.

Running it in GitHub Actions

I built an open-source Action called AICostFence to automate this check:

name: AI cost check
on: pull_request

permissions:
  contents: read
  pull-requests: write

jobs:
  cost-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ronnie0297-stack/aicostfence@v0.1.0
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Enter fullscreen mode Exit fullscreen mode

It scans locally on the GitHub runner, posts one updated pull-request report, and can fail the check when a configured guard is violated. It does not call an AI model or send source code to a model provider.

The current release intentionally supports a narrow surface: JavaScript/TypeScript projects using the Vercel AI SDK's generateText, streamText, generateObject, or streamObject calls. That makes the analysis deterministic while the early workflow is validated.

Repository: https://github.com/ronnie0297-stack/aicostfence

Marketplace: https://github.com/marketplace/actions/aicostfence

npm: https://www.npmjs.com/package/aicostfence

What cost control would be most useful in your pull requests: model-swap deltas, provider-specific checks, or organization-wide policies?

Try the working demo

The before-and-after demo runs the published npm package against risky code and its corrected version. It verifies that the first scan fails and the second passes, with no API key or model charges. See the GitHub verification runs.

git clone https://github.com/ronnie0297-stack/aicostfence.git
cd aicostfence/demo
npm ci --ignore-scripts
npm run demo
Enter fullscreen mode Exit fullscreen mode

Requires Node.js 20 or newer. The demo uses bundled fallback prices for reproducible results. Version 0.1.0 does not include step multiplication, retries, or paid tools in its estimate, and it does not resolve wrappers or imported defaults. Passing the check is not a guarantee of a safe bill.

Top comments (3)

Collapse
 
raju_dandigam profile image
Raju Dandigam •

Making the estimate deterministic and reviewable is more useful than pretending it predicts the invoice exactly. For tool-using calls, I’d surface both the absolute budget and the PR delta: maxOutputTokens × maximum steps is only part of the envelope once retries and paid tool side effects enter the loop. A checked-in waiver with an owner and expiry could also keep exceptions visible instead of encouraging people to weaken the rule. Are wrapper functions and imported defaults on the roadmap, so the Action can resolve bounds that are not inline at the call site?

Collapse
 
ronnie0297-stack profile image
ronnie0297-stack •

Thanks, Raju—this is useful feedback. PR cost deltas are on the roadmap. The current version focuses on inline configuration; wrapper functions and imported defaults aren’t supported yet, but your example makes a strong case for adding them.

You’re right that token limits and step counts don’t capture the full cost once retries and paid tools are involved. Owner-and-expiry waivers are also a helpful suggestion for keeping exceptions accountable.

How do you usually define shared limits: imported constants, configuration objects, or wrapper functions? A small example would help us prioritize the next improvement.

Collapse
 
vlad_z_16b6320e21f32bee0d profile image
Vlad Z •

Catching cost at the pull request is earlier than most teams get, the gap is that production behavior differs, loops show up under real inputs rather than the test set

Curious how the action estimates token counts, is it static or a replay of a test set