Agent Decision Latency Budget: Where Time Goes in Every AI Agent Request

When an AI agent takes 8 seconds to respond, users don’t care which stage was slow. But as an engineer building agent infrastructure, you need to know exactly where those milliseconds go.

Every agent request is a pipeline: receive the request, retrieve context, call the LLM, maybe execute tools, format a response. Each stage has a time cost. Understanding those costs — and where to invest optimization effort — is the difference between agents that feel instant and agents that feel broken.

Anatomy of an Agent Request

A typical agent request follows this path:

User Query
  → Request parsing & routing         (1-5ms)
  → Context retrieval                  (1-500ms)  ← highly variable
  → Prompt assembly                    (1-10ms)
  → LLM inference                      (200-2000ms)
  → Tool execution (if needed)         (10-5000ms)
  → Response formatting                (1-10ms)
  → Delivery to user                   (5-50ms)
──────────────────────────────────────────────────
  Total                                (220-7575ms)

The range is enormous — 220ms to 7.5 seconds — and that’s for a single agent turn. Multi-step agents that call tools and loop back to the LLM multiply this.

Let’s break down each stage.

Stage 1: LLM Inference (200-2000ms)

This is usually the largest fixed cost. The time depends on:

• Model size — GPT-4o: 300-1500ms, GPT-4o-mini: 150-500ms, Claude Haiku: 100-400ms
• Input token count — More context = more processing time
• Output token count — Streaming helps perceived latency but total time still scales with output length
• Provider load — API latency varies by time of day and demand

You have limited control here. The main levers are:

1. Choose the right model size — Not every agent call needs GPT-4. Routing simple queries to smaller models saves 500-1000ms.
2. Minimize input tokens — Every unnecessary token in the context window adds latency. Be surgical about what context you include.
3. Set max_tokens — Cap output length for structured responses.
4. Use streaming — First token appears in 100-300ms even if the full response takes 1500ms.

Stage 2: Context Retrieval (1-500ms)

This is where architecture decisions create orders-of-magnitude differences. The same “look up customer data” operation ranges from 1ms to 30 seconds depending on how you build it.

Option A: Direct Database Query (10-100ms)

SELECT * FROM customers WHERE id = 'cust_123';

Latency: 10-100ms for a simple indexed lookup.

Problem: Every agent request hits your production database. At 100 requests/second, that’s manageable. At 10,000 requests/second, you’re competing with your application’s transactional workload. Connection pooling helps, but you’re fundamentally coupling agent traffic to operational database load.

Option B: Cache Lookup (1-5ms)

customer = redis.get(f"customer:{customer_id}")

Latency: 1-5ms for a Redis or Memcached lookup.

Problem: Cache invalidation. When the source data changes, how quickly does the cache update? If you’re using TTL-based expiration, you have a staleness window. If you’re using CDC-based cache sync, updates propagate in near real-time.

Option C: Vector Search (10-50ms)

results = vector_store.query(
    vector=embed(user_query),
    top_k=5,
    filter={"tenant_id": tenant_id}
)

Latency: 10-50ms for approximate nearest neighbor search.

Best for: Semantic lookups where you don’t know the exact ID. Product recommendations, knowledge base search, similar ticket retrieval. See building streaming embedding pipelines for keeping vector stores current.

Option D: Warehouse Query (500ms-30,000ms)

-- Snowflake / BigQuery
SELECT * FROM analytics.customer_360
WHERE customer_id = 'cust_123';

Latency: 500ms on a warm warehouse, 5-30 seconds on a cold one.

Problem: This is a non-starter for interactive agents. Warehouses are built for analytical throughput — scanning billions of rows for dashboards — not for point queries with millisecond SLAs. Even with result caching, cold-start latency is unpredictable.

Option E: Pre-Computed Feature Store (5-15ms)

features = feature_store.get_online_features(
    entity_key={"customer_id": "cust_123"},
    feature_names=["lifetime_value", "churn_risk", "last_order_days_ago"]
)

Latency: 5-15ms for pre-computed feature lookups.

Best for: ML features and derived metrics. The computation happens during streaming feature computation, and serving is just a key-value lookup.

Retrieval Method Comparison

Method	p50 Latency	p99 Latency	Freshness	Load on Source
Direct DB query	15ms	100ms	Real-time	High
Redis cache (TTL)	2ms	5ms	Seconds-minutes	None
Redis cache (CDC)	2ms	5ms	Sub-second	None
Vector search	20ms	50ms	Depends on pipeline	None
Warehouse query	2,000ms	30,000ms	Hours (batch)	None
Feature store	8ms	15ms	Sub-second (streaming)	None

The takeaway: for interactive agents, you need a serving layer between your source databases and the agent. Direct queries work at low scale but don’t survive growth.

Stage 3: Tool Execution (10-5000ms)

When an agent decides to call a tool — check inventory, send an email, query an API — execution time is wildly variable:

• Internal API call: 10-200ms
• External API call: 100-2000ms
• Database write: 10-100ms
• Multi-step workflow: 1000-5000ms

The key optimization is parallelizing tool calls when possible. If an agent needs both customer data and order history, fetch them concurrently:

import asyncio

async def execute_tools(tool_calls: list) -> list:
    tasks = [execute_single_tool(call) for call in tool_calls]
    return await asyncio.gather(*tasks)

Modern LLM APIs support parallel tool calling natively. GPT-4o and Claude can return multiple tool calls in a single response, letting you execute them concurrently.

The Math of 10K Agent Requests Per Second

Let’s work through what it takes to handle 10,000 agent requests per second — a realistic target for a customer-facing agent in a large application.

LLM inference:

10,000 req/s × 500ms average = 5,000 concurrent requests
At $0.01 per request (GPT-4o-mini) = $100/second = $8.6M/year

That cost is unsustainable with large models. You need a tiered approach:

• 80% of requests → small model (GPT-4o-mini, Haiku): 150ms, $0.001/req
• 15% of requests → medium model (GPT-4o, Sonnet): 500ms, $0.01/req
• 5% of requests → large model (GPT-4, Opus): 1500ms, $0.05/req

Blended cost: ~$0.004/request = $40/second = $3.5M/year

Still expensive. This is why many teams use fine-tuned smaller models or local inference for high-volume agents.

Context retrieval at 10K req/s:

Direct DB: 10,000 queries/second on your production DB — probably not
Redis: 10,000 gets/second — trivial (Redis handles 100K+/s)
Vector search: 10,000 queries/second — needs a beefy cluster
Feature store: 10,000 lookups/second — designed for this

Redis and feature stores are the only retrieval methods that handle this load without dedicated infrastructure scaling.

Latency Budget Templates

Interactive Chat Agent (Target: < 2 seconds)

Context retrieval:    50ms  (cache or feature store)
Prompt assembly:      10ms
LLM inference:       800ms  (GPT-4o-mini, streaming)
Tool execution:      200ms  (one fast API call)
Response formatting:  10ms
Network overhead:     30ms
─────────────────────────
Total:             1,100ms
Buffer:              900ms

Background Processing Agent (Target: < 30 seconds)

Context retrieval:   200ms  (multiple sources, including vector search)
Prompt assembly:      20ms
LLM inference:     3,000ms  (GPT-4o with large context)
Tool execution:    5,000ms  (multi-step workflow)
Second LLM call:   2,000ms  (verification step)
Response formatting:  50ms
─────────────────────────
Total:            10,270ms
Buffer:           19,730ms

Real-Time Decision Agent (Target: < 500ms)

Context retrieval:    10ms  (Redis cache, CDC-synced)
Prompt assembly:       5ms
LLM inference:       200ms  (fine-tuned small model or local)
Response formatting:   5ms
Network overhead:     20ms
─────────────────────────
Total:               240ms
Buffer:              260ms

The real-time decision agent is the hardest to build. It requires pre-computed context (no time for retrieval-heavy lookups), a small/local model, and no tool execution in the critical path.

Where to Optimize First

Rank your optimization effort by impact:

1. Context retrieval method — Switching from warehouse queries to cache lookups saves 1-30 seconds. Biggest bang for effort.
2. Model selection and routing — Using a smaller model for simple queries saves 300-1000ms per request and cuts costs.
3. Prompt engineering — Trimming unnecessary context from prompts saves LLM processing time proportional to token reduction.
4. Parallel tool execution — Executing independent tools concurrently instead of sequentially saves the sum of all but the slowest call.
5. Caching LLM responses — For repeated queries with identical context, cache the response. Hit rate varies but can be 10-30% for structured queries.

Measuring Your Latency Budget

Instrument every stage independently. Here’s a minimal tracing approach:

import time
from dataclasses import dataclass

@dataclass
class LatencyTrace:
    context_retrieval_ms: float = 0
    prompt_assembly_ms: float = 0
    llm_inference_ms: float = 0
    tool_execution_ms: float = 0
    total_ms: float = 0

def trace_agent_request(query: str) -> tuple:
    trace = LatencyTrace()
    start = time.monotonic()

    t0 = time.monotonic()
    context = retrieve_context(query)
    trace.context_retrieval_ms = (time.monotonic() - t0) * 1000

    t0 = time.monotonic()
    prompt = assemble_prompt(query, context)
    trace.prompt_assembly_ms = (time.monotonic() - t0) * 1000

    t0 = time.monotonic()
    response = call_llm(prompt)
    trace.llm_inference_ms = (time.monotonic() - t0) * 1000

    if response.tool_calls:
        t0 = time.monotonic()
        tool_results = execute_tools(response.tool_calls)
        trace.tool_execution_ms = (time.monotonic() - t0) * 1000

    trace.total_ms = (time.monotonic() - start) * 1000
    return response, trace

Log these traces for every request. After a week of data, you’ll know exactly where your time goes and where optimization will have the most impact.

Keeping Context Retrieval Under 50ms

The 50ms threshold matters because it’s the point where context retrieval becomes negligible compared to LLM inference. If your context lookup takes 500ms and LLM inference takes 500ms, optimizing retrieval gives you a 50% improvement. If retrieval takes 5ms, there’s nothing left to optimize — LLM inference dominates.

To hit sub-50ms retrieval consistently:

• Pre-compute and cache — Don’t compute derived data at query time. Use streaming pipelines to keep materialized views or caches current.
• Co-locate data and compute — Put your cache in the same region as your agent inference. Cross-region lookups add 50-150ms.
• Denormalize for reads — An agent shouldn’t join three tables at query time. Denormalize during the streaming transform stage.
• Index for your access patterns — If agents always look up by customer_id, make sure that’s a primary key or partition key in your serving layer.

Designing Agent Infrastructure Around Latency

The latency budget isn’t just a measurement exercise — it should drive your architecture decisions. If you need a 500ms total response time, warehouse queries are off the table before you write a line of code. If you need 10K requests per second, direct database queries are off the table too.

The pattern that works for most production agent systems: CDC captures changes from source databases, streaming transforms pre-compute the context agents need, and a fast serving layer (Redis, feature store, or vector store) handles point lookups at query time. The heavy lifting happens before the agent request, not during it.

---

Ready to get agent context retrieval under 50ms? Streamkap streams database changes to Redis, vector stores, and feature stores in real time, so your agents never wait on stale data. Start a free trial or learn more about real-time data for AI agents.