DEV Community

Cover image for Day 26 — Why Your AI Agent Might Be Slow Even When Every Tool Is Fast
Rimsha Kanwal
Rimsha Kanwal

Posted on

Day 26 — Why Your AI Agent Might Be Slow Even When Every Tool Is Fast

Understanding Parallel Retrieval, Failure Handling, and Latency Optimization in LangGraph

When building production AI agents, developers often focus on optimizing LLM inference, reducing token usage, and choosing faster models.

But what if the biggest performance bottleneck isn't the language model?
What if it's the way your agent executes its tools?

Imagine an AI customer support agent that retrieves order information, shipping policies, and refund policies before responding to a customer.

Each retrieval operation is relatively fast.

However, executing them sequentially can introduce unnecessary latency.

This is where parallel execution, asynchronous I/O, and dependency-aware workflow design become important.

1. The Problem: Fast Tools Can Still Create Slow Workflows

Consider an AI support assistant built using Python, LangGraph, PostgreSQL, and a vector database.

A customer asks:

Why is my order delayed, and can I request a refund?

The assistant needs information from three independent sources:

  • Order information: PostgreSQL
  • Shipping policies: Vector database
  • Refund policies: Vector database

Suppose the retrieval times are:

Retrieval operation Latency
Order information 300 ms
Shipping policy 450 ms
Refund policy 350 ms

If executed sequentially, total retrieval time is:

300 + 450 + 350 = 1,100 ms

The agent spends 1.1 seconds retrieving data before generating its response.

But none of these retrievals needs the result of another.

So why execute them one at a time?

2. The Solution: Parallelize Independent Retrieval

Python's asyncio.gather() allows independent asynchronous operations to execute concurrently.

import asyncio

async def retrieve_context(order_id: str):
    order, shipping, refund = await asyncio.gather(
        fetch_order_details(order_id),
        retrieve_shipping_policy(),
        retrieve_refund_policy(),
    )

    return {
        "order": order,
        "shipping_policy": shipping,
        "refund_policy": refund,
    }
Enter fullscreen mode Exit fullscreen mode

Assuming the retrieval functions perform nonblocking asynchronous I/O, the three operations can progress concurrently.

Instead of adding their latencies together, total execution time approaches the duration of the slowest operation.

max(300, 450, 350) = 450 ms

Theoretical reduction:

Approximately 59% less retrieval latency.

These timings are illustrative. Actual performance depends on network latency, database connections, concurrency limits, and runtime overhead.

3. Parallel Execution in LangGraph

LangGraph supports parallel execution through a fan-out/fan-in graph architecture.

                    ┌── Order Retrieval ───┐
                    │                      │
START ── Validate ──┼── Shipping Policy ───┼── Generate Answer
                    │                      │
                    └── Refund Policy ─────┘
Enter fullscreen mode Exit fullscreen mode

After validation, the three retrieval nodes can execute in the same graph super-step.

The downstream response-generation node can combine their results once the required branches complete.

Each branch should write to a distinct state field or use an appropriate reducer when multiple branches update the same field.

The objective is to parallelize independent operations without introducing inconsistent application state.

4. Failure Scenario: What If One Parallel Tool Fails?

Parallel execution improves performance, but it also changes how failures must be handled.

Imagine three retrieval operations running concurrently:

Operation Execution result
Order information Success — 300 ms
Shipping policy Success — 450 ms
Refund policy Timeout after 2,000 ms

What should happen now?

A basic asyncio.gather() call propagates the first exception to its caller by default.

Importantly, when one operation raises an exception, the remaining operations are not automatically canceled by gather().

This means a production workflow needs an explicit failure-handling strategy.

For example:

import asyncio

async def retrieve_context(order_id: str):
    results = await asyncio.gather(
        fetch_order_details(order_id),
        retrieve_shipping_policy(),
        retrieve_refund_policy(),
        return_exceptions=True,
    )

    order, shipping, refund = results

    if isinstance(order, BaseException):
        raise RuntimeError(
            "Unable to retrieve order information"
        ) from order

    if isinstance(shipping, BaseException):
        shipping = None

    if isinstance(refund, BaseException):
        refund = None

    return {
        "order": order,
        "shipping_policy": shipping,
        "refund_policy": refund,
    }
Enter fullscreen mode Exit fullscreen mode

This example demonstrates partial-result handling. It assumes that the retrieval functions have appropriate timeouts and do not suppress cancellation.

But returning None does not mean the missing information can be ignored.

If the refund policy cannot be retrieved, the agent should not invent refund eligibility rules.

Instead, it can explain that the order information is available but refund eligibility could not be verified.

A faster AI agent is not necessarily a more reliable AI agent.

A reliable agent must distinguish between missing evidence and evidence that supports an answer.

Production considerations

A more robust design should also include:

  • Per-tool timeouts and bounded retries.
  • Explicit classification of required versus optional retrievals.
  • Structured error results and tracing.
  • Concurrency limits to prevent resource exhaustion.
  • Safe fallback behavior when essential evidence is unavailable.

For LangGraph workflows, failure recovery must additionally account for checkpoint and super-step execution semantics.

5. Benchmark: Sequential vs. Parallel Retrieval

To evaluate the performance difference, consider a controlled experiment using three simulated independent I/O operations.

Use the same simulated delays in both implementations:

  • Order retrieval: 300 ms
  • Shipping policy: 450 ms
  • Refund policy: 350 ms

Sequential implementation

import asyncio
import time

async def fetch_order():
    await asyncio.sleep(0.300)
    return {"order_id": "ORD-1003"}

async def fetch_shipping():
    await asyncio.sleep(0.450)
    return {"shipping": "delayed"}

async def fetch_refund():
    await asyncio.sleep(0.350)
    return {"refund_policy": "available"}

async def sequential():
    order = await fetch_order()
    shipping = await fetch_shipping()
    refund = await fetch_refund()

    return order, shipping, refund
Enter fullscreen mode Exit fullscreen mode

Parallel implementation

async def parallel():
    return await asyncio.gather(
        fetch_order(),
        fetch_shipping(),
        fetch_refund(),
    )
Enter fullscreen mode Exit fullscreen mode

Benchmark runner

import statistics

async def benchmark(fn, iterations=100):
    timings = []

    for _ in range(iterations):
        start = time.perf_counter()

        await fn()

        elapsed = time.perf_counter() - start
        timings.append(elapsed * 1000)

    timings.sort()

    return {
        "median_ms": statistics.median(timings),
        "p95_ms": timings[
            int(0.95 * (len(timings) - 1))
        ],
    }

async def main():
    print("Sequential:", await benchmark(sequential))
    print("Parallel:", await benchmark(parallel))

asyncio.run(main())
Enter fullscreen mode Exit fullscreen mode

This benchmark measures isolated coroutine execution using simulated I/O delays. It does not measure actual PostgreSQL queries, vector searches, LangGraph scheduling, or LLM generation.

Expected theoretical comparison

Metric Sequential Parallel
Retrieval latency ~1,100 ms ~450 ms
Theoretical speedup 1.0× ~2.44×
Latency reduction — ~59%

These are theoretical expectations, not measured benchmark results. The script above can be executed to obtain actual median and p95 measurements in a specific runtime environment.

For a production benchmark, replace simulated sleeps with real retrieval operations and evaluate under multiple concurrent-user loads.

Also measure end-to-end response latency, error rates, and resource utilization. Improving retrieval latency alone may have a smaller effect on overall response time.

6. Why Parallelization Isn't Always the Right Choice

Consider a different workflow:

  1. Retrieve an order.
  2. Determine the customer's country from that order.
  3. Retrieve country-specific refund policies.

Here, the third operation depends on the first.

Executing them blindly in parallel would be incorrect because the refund policy search requires information that is not yet available.

A better workflow is:

Retrieve Order
      |
      v
Identify Country
      |
      v
Retrieve Applicable Refund Policy
      |
      v
Generate Response
Enter fullscreen mode Exit fullscreen mode

The key engineering principle is:

Parallelize independent work. Preserve necessary dependencies.

Even independent operations can become slower under excessive concurrency if they compete for limited database connections, CPU, memory, or external API capacity.

7. What Developers Should Actually Measure

When optimizing an AI workflow, retrieval latency alone is not enough.

Relevant metrics include:

  • p50 latency: Typical execution time.
  • p95 latency: Time below which 95% of executions complete.
  • End-to-end latency: Total time until the user receives the completed response.
  • Failure rate: Percentage of requests that encounter errors or timeouts.
  • Concurrency utilization: How many operations run simultaneously.
  • Resource saturation: Whether connection pools, external APIs, or infrastructure become bottlenecks.

A good optimization should improve latency without compromising reliability, answer quality, or operational stability.

Final Takeaway

Production AI engineering is not just about making language models faster.

It is also about eliminating unnecessary waiting between independent operations.

Parallel retrieval can significantly reduce response latency, especially in workflows that call multiple independent databases, APIs, or retrieval services.

However, uncontrolled concurrency introduces its own risks: partial failures, resource contention, rate limits, and inconsistent state updates.

The goal is not to execute every tool simultaneously.

The goal is to design an execution graph that respects dependencies while avoiding unnecessary sequential work.

Sequential execution preserves dependencies. Parallel execution eliminates unnecessary waiting. Reliable engineering requires both.

And that distinction becomes critical when an AI agent moves from a prototype to a production system.

Top comments (2)

Collapse
 
rimsha_kanwal profile image
Rimsha Kanwal •

Research References

Python — Coroutines and Tasks
docs.python.org/3/library/asyncio-...

LangGraph — Graph API and parallel execution
docs.langchain.com/oss/python/lang...

LangChain — RunnableParallel
reference.langchain.com/python/lan...

LangGraph — Durable execution and fault tolerance
docs.langchain.com/oss/python/lang...

Collapse
 
glenallen profile image
Glen Allen •

One thing I'd add is that optimizing the critical path matters just as much as parallelizing independent calls. Even after retrieval becomes faster, unnecessary processing between nodes, repeated context transformations, or waiting for non-essential results can still increase end-to-end latency. Looking at the complete execution trace can help identify where the agent is actually spending its time.