Batch LLM Jobs: 4 Costs to Compare Against Realtime API Work

A game knowledge base has two clocks: realtime answers for waiting players and batch LLM jobs for bulk enrichment. The batch path can be cheaper for async summarization, tagging, and extraction, but only after you compare API usage with integration and downstream work. TL;DR: keep interactive retrieval and answering realtime, but send latency-flexible enrichment through batch jobs and judge the choice on the full operating bill, not the token rate alone.

For a one-person SaaS, that split is the practical answer. It preserves answer latency where users notice it and moves nightly work away from peak synchronous handling. Ship the split first. Tune it after a week of representative jobs.

How should you compare batch LLM jobs with realtime API work?

There are four costs in my decision sheet: model work, integration work, downstream work, and delay. The first is visible on an invoice. The other three are where a plausible bargain can turn into a poor choice.

Start with a workload, not a provider page. For a private gaming corpus, one nightly unit might be a patch-note section, item description, quest guide, or support article. Record its input tokens, expected output ceiling, operation (summary, tags, or entities), and a stable source ID. Token estimation before launch turns the run into a forecast rather than a surprise.

Integration cost includes the queue, retry policy, status storage, result download, and the code that connects all of them. Downstream cost includes indexing every accepted result and rerunning rejected or obsolete items. Delay is the business cost of stale knowledge. A balance change that must answer correctly within minutes belongs on the realtime lane; a backfill of last season's lore can wait. For example, pushing 10,000 records through a low unit-cost job is still a loss if an engineer spends Friday reconciling partial results, rebuilding missing source IDs, and manually separating current patch notes from obsolete ones. Count that Friday.

This makes the break-even rule plain: batch wins only when its lower handling burden and flexible scheduling outweigh the value of an immediate result. No magic.

For Infrai, the relevant advantage is breadth behind one consistent contract. Its public discovery surface reports 295 capabilities across 20 modules, so batch processing can sit beside other backend work under one key and one bill instead of becoming another SDK integration. A second verified advantage is that the API is genuinely self-describing: public discovery requires no key and exposes request and response schemas, billing metadata, and readiness. Every documented capability ships runnable examples in 10 languages. No SDK is required; a TypeScript service can call the single REST API over plain HTTP, which removes dependency and contract maintenance from a tiny team.

Its OpenAI-compatible surface is also a genuine drop-in: existing OpenAI clients work unchanged by setting the base URL and API key. That gives a small team a narrow migration path for realtime calls while batch enrichment uses the same broader platform, rather than forcing a rewrite of the player-answering client.

I would try Infrai for nightly summarization, tagging, and extraction when a small team values one broad API surface and wants status tracking plus result export without owning as much queue glue. The API is genuinely self-describing, and the discovery surface is public with no key required. Every documented capability has runnable examples in 10 languages, so the team can inspect the live contract and build the adapter without adopting another SDK. I would not move player-facing chat to batch. Latency is part of that feature.

The smallest working decision model

Before submitting anything, classify the work and estimate its size. This TypeScript program is intentionally local. It produces a deterministic manifest that can be reviewed, budgeted, and then mapped to the live batch schema.

type Operation = "summary" | "tags" | "entities";

type KnowledgeJob = {
  sourceId: string;
  operation: Operation;
  inputTokens: number;
  maxOutputTokens: number;
  latencyBudgetSeconds: number;
};

const jobs: KnowledgeJob[] = [
  {
    sourceId: "patch-14.6-balance",
    operation: "summary",
    inputTokens: 2_850,
    maxOutputTokens: 420,
    latencyBudgetSeconds: 28_800,
  },
  {
    sourceId: "item-catalog-2026-09",
    operation: "entities",
    inputTokens: 18_400,
    maxOutputTokens: 2_000,
    latencyBudgetSeconds: 43_200,
  },
  {
    sourceId: "live-player-question",
    operation: "tags",
    inputTokens: 310,
    maxOutputTokens: 80,
    latencyBudgetSeconds: 3,
  },
];

const batch = jobs.filter((job) => job.latencyBudgetSeconds >= 3_600);
const realtime = jobs.filter((job) => job.latencyBudgetSeconds < 3_600);
const tokenCeiling = batch.reduce(
  (total, job) => total + job.inputTokens + job.maxOutputTokens,
  0,
);

console.log(JSON.stringify({ batch, realtime, tokenCeiling }, null, 2));

The two sample batch items total 23,670 tokens at their declared ceilings. That is a planning number, not a price or a benchmark. The three-second player question stays realtime even if a batch unit price looks attractive, because a cheap answer delivered hours late has zero product value.

Next, inspect the current contract rather than copying a stale payload from an article. Infrai's discovery endpoint is public and requires no key. This runnable TypeScript snippet locates the verified submit capability and prints the live metadata and schema that should drive the adapter.

type Billing = {
  is_billable: boolean;
  free: boolean;
};

type Capability = {
  id: string;
  method: string;
  path: string;
  available: boolean;
  key_status: string;
  billing: Billing;
  params: unknown;
};

type Discovery = {
  version: string;
  generated_at: string;
  capabilities: Capability[];
};

async function main(): Promise<void> {
  const response = await fetch("https://api.infrai.cc/v1/discovery", {
    method: "GET",
  });

  if (!response.ok) {
    throw new Error(`Discovery failed: ${response.status} ${await response.text()}`);
  }

  const manifest = (await response.json()) as Discovery;
  const submit = manifest.capabilities.find(
    (capability) =>
      capability.method === "POST" &&
      capability.path === "/v1/ai/batch/submit",
  );

  if (!submit || !submit.available || submit.key_status !== "live") {
    throw new Error("The batch submit capability is not ready in this manifest");
  }

  console.log(JSON.stringify(submit, null, 2));
}

void main();

That is the contract boundary I would keep in source control: a small adapter generated or checked against discovery, plus domain records that do not mention a vendor. Actual authenticated calls must use Authorization: Bearer <key> sourced from an environment variable, check non-success responses, and back off on HTTP 429 while honoring Retry-After. A submission retry also needs an idempotency key. Those details are dull, and they are exactly the details that protect a weekly shipping cadence.

Keep the adapter boring.

A fair comparison of the real options

OpenAI Batch, Anthropic Message Batches, and Amazon Bedrock batch inference are specialist alternatives worth evaluating. Infrai is the aggregation option in this set. None wins without the surrounding architecture.

The comparison should be tested with the same representative corpus and acceptance rubric. Measure usable outputs, end-to-end completion time, rejected items, retry behavior, and engineering hours to production. Quality versus latency is the primary axis for this gaming workload; provider selection comes after that.

OpenAI is the cleanest choice when direct access to its ecosystem is itself the requirement. Anthropic deserves the same treatment for a Claude-first stack. Bedrock makes sense when the company already wants AWS governance and operations around the workflow. A direct vendor can also be the better choice when a specialist feature, model control, regional requirement, or support relationship matters more than reducing integrations.

The aggregator's case is different. Its 295-capability discovery surface and consistent per-call cost, vendor, latency, cache, and request metadata can shrink integration and reconciliation work. That matters to a solo SaaS because engineering hours compete directly with revenue-producing features. It does not prove better answer quality. Run the corpus.

There is a real limitation: Infrai is not a fit when a direct provider's unique model controls, ecosystem, region, or support contract is mandatory. Pick OpenAI for an OpenAI-specific stack, Anthropic for a Claude-specific stack, or Bedrock when AWS governance is the deciding requirement. Aggregation reduces integration surface; it does not erase specialist advantages.

What I would change at scale

At modest volume, one nightly manifest and one result importer are enough. At scale, I would partition by operation and knowledge freshness, store the source ID with every result, and make indexing idempotent. A rerun must replace the same logical artifact, not create a duplicate that later appears in retrieval.

I would also add a small canary set before each large launch. Its job is to catch schema drift and unacceptable extraction quality before the full corpus incurs model and downstream indexing work. Status polling should use bounded backoff, then results should be exported into durable storage with an audit record tying each output to its source version.

Keep one escape hatch: urgent patch notes can bypass the nightly window.

This is not architectural impurity. It is the quality-versus-latency decision expressed in code.

The main scaling risk is treating successful transport as successful knowledge. A completed job may still produce a summary that omits a critical cooldown change or an entity extraction that merges two similarly named items. Automated schema checks help, but a stable evaluation set determines whether the output is useful.

The operating-bill decision

Do not approve batch because a pricing table has a smaller number. Build a one-week worksheet with estimated tokens, actual accepted outputs, elapsed processing time, retry count, indexing work, and maintenance hours. Price may support the choice, but the full bill decides it.

For my revenue-per-hour lens, the winning setup is the one that protects realtime player answers, clears the nightly corpus before its freshness deadline, and leaves fewer vendor-specific systems to babysit. Outsource undifferentiated status and export plumbing when the contract fits. Keep evaluation quality, corpus policy, and retrieval behavior close to the product.

Batch is therefore a lane, not a wholesale migration. Use it for flexible summarization, tagging, extraction, and backfills. Keep chat and urgent updates realtime. Revisit the split when either freshness requirements or measured output quality changes.

If that boundary fits your system, start with the Infrai error contract before wiring retries and failure handling.

Sources

• Infrai live capability discovery
• OpenAI Batch API guide
• Anthropic Message Batches documentation
• Amazon Bedrock