Your AI Agent Is a Liability: The Production Checklist That Separates $500/Day Contractors from $2,000/Day Architects

The agent graveyard is filling up fast

Forty percent of enterprise agentic AI projects will be cancelled or decommissioned by the end of 2027. That is not my estimate — that is Gartner's, published this quarter. The number tracks with what I see on the ground: every month, a new prospect calls me to rescue an agent deployment that passed the demo but failed production.

The failure pattern is so consistent it is almost scripted. A team builds an impressive prototype in two weeks. The agent calls tools, summarizes documents, sends emails, updates CRMs. The demo gets a standing ovation. Then someone deploys it to real users with real data and real edge cases, and everything collapses.

The agent loops for 47 turns on a question it should have escalated in 3. The token bill hits $14,000 in a week. A hallucinated action triggers a compliance incident. The security team discovers the agent has production database credentials with no audit trail. The project gets shelved. The budget gets frozen. The team gets reassigned.

I have seen this happen at banks, manufacturers, consultancies, and SaaS companies. The model is never the problem. The engineering around the model is always the problem. And the gap between teams that get this right and teams that do not is the gap between a contractor billing €400/day to maintain a chatbot and an architect billing €1,500+/day to ship autonomous systems that run unsupervised.

This post is the checklist I use for every engagement. It is the distillation of every production agent I have shipped and every failed agent I have autopsied. If you are building agents — or hiring someone to build them — this is the minimum viable standard.

Why agents fail differently than other software

Traditional software is deterministic. You write a function, it returns the same output for the same input, and testing is a matter of covering the input space. An AI agent is fundamentally different in three ways that make production engineering harder:

1. Non-deterministic execution paths. The same user query can produce different tool-calling sequences on different runs. You cannot write unit tests the way you would for a REST API. You need evaluation frameworks that measure outcome quality across distributions, not exact output matching.

2. Unbounded resource consumption. A poorly constrained agent can loop indefinitely, calling expensive APIs on every iteration. Without explicit turn limits, token budgets, and timeout enforcement, a single adversarial or ambiguous query can consume your entire monthly LLM budget in an afternoon.

3. Action authority without accountability. An agent that can write to a database, send an email, or trigger an API call is operating with the authority of a human employee — but without the judgment, the context, or the legal accountability. Every action an agent takes in production needs an audit trail, a rollback mechanism, and a clear escalation path.

These three properties — non-determinism, unbounded cost, and unsupervised authority — are why the standard software engineering playbook is necessary but not sufficient. You need an additional layer of controls that most teams skip because they were never required for traditional applications.

The 14-point production checklist

I do not ship an agent to production without every item on this list addressed. Some are engineering controls. Some are architectural decisions. Some are governance requirements. All of them are non-negotiable.

Tier 1: Survival (your agent will crash without these)

1. Turn limits and token budgets

Every agent loop needs a hard ceiling. I set two:

• Max turns per execution: Typically 8–15 for task-specific agents, 25–40 for research agents. If the agent has not resolved the task within the limit, it escalates to a human or returns a structured "incomplete" response.
• Max tokens per execution: A dollar-denominated budget per run. For a customer-facing agent, this is usually $0.10–$0.50 per interaction. For a back-office automation agent, $1–$5 per task.

from dataclasses import dataclass
 
@dataclass
class AgentBudget:
    max_turns: int = 12
    max_input_tokens: int = 100_000
    max_output_tokens: int = 10_000
    max_cost_usd: float = 0.50
    timeout_seconds: int = 120
 
    def is_exceeded(self, turns: int, input_tokens: int,
                    output_tokens: int, cost: float, elapsed: float) -> str | None:
        if turns >= self.max_turns:
            return f"Turn limit reached ({self.max_turns})"
        if input_tokens >= self.max_input_tokens:
            return f"Input token budget exhausted ({self.max_input_tokens:,})"
        if cost >= self.max_cost_usd:
            return f"Cost budget exhausted (${self.max_cost_usd:.2f})"
        if elapsed >= self.timeout_seconds:
            return f"Timeout ({self.timeout_seconds}s)"
        return None

Without this, you will discover your budget problem on the invoice. By then, it is too late.

2. Structured error handling and graceful degradation

Agents fail. APIs time out. Tools return unexpected formats. The model hallucinates a tool name that does not exist. The question is not whether your agent will encounter an error — it is whether it will recover gracefully or spiral into a retry loop that compounds the problem.

Every tool call gets wrapped in a structured error handler that:

• Catches the exception and classifies it (transient vs. permanent, retryable vs. fatal).
• Returns a structured error message to the agent instead of a raw stack trace.
• Limits retries to 2 attempts with exponential backoff.
• After max retries, routes to a fallback: a simpler tool, a cached response, or a human escalation.

The agent that crashes cleanly is infinitely more valuable than the agent that keeps trying and makes things worse.

3. Model routing and cost optimization

Not every agent turn requires a frontier model. I wrote about this in detail in my semantic routing post, but the principle is simple: route easy decisions to cheap models and hard decisions to expensive models.

In practice, this means:

• Intent classification and simple tool selection: A fast, cheap model (Gemini Flash, Claude Haiku, or a local SLM). Cost: $0.001–$0.01 per call.
• Complex reasoning, multi-step planning, and synthesis: A frontier model (Claude Opus, GPT-4o, Gemini Pro). Cost: $0.05–$0.50 per call.
• Structured extraction from documents: A mid-tier model with JSON mode. Cost: $0.01–$0.05 per call.

A well-routed agent costs 60–80% less than one that sends every message to the most expensive model. Multiply that across thousands of daily interactions and the savings fund the entire engineering team.

4. Timeout enforcement at every layer

This is the one that catches teams off guard. You need timeouts at four levels:

1. LLM API call timeout: 30–60 seconds. If the model has not responded, retry once, then fail.
2. Tool execution timeout: Depends on the tool. Database queries: 10 seconds. API calls: 15 seconds. File processing: 60 seconds.
3. Single turn timeout: 90 seconds. If the agent is still "thinking" after 90 seconds, something is wrong.
4. Total execution timeout: 2–5 minutes for interactive agents, 10–30 minutes for batch agents.

Without layered timeouts, a single slow API call can cascade into a user staring at a spinner for four minutes, which in an enterprise context means a support ticket, an escalation, and a conversation with the VP who approved the project.

Tier 2: Reliability (your agent will embarrass you without these)

5. Deterministic guardrails on actions

Every action the agent can take must be classified by risk level:

The classification is domain-specific and must be defined with the business stakeholder, not the engineering team. What the engineer considers "low risk" and what the compliance officer considers "low risk" are often very different things.

6. Structured output validation

The agent's output is not trusted by default. Every response that will be shown to a user or used to trigger an action passes through a validation layer:

• Schema validation: Does the output conform to the expected structure? If the agent is supposed to return a JSON object with specific fields, enforce it.
• Content validation: Does the response contain prohibited content, PII that should have been redacted, or assertions that contradict known facts?
• Confidence scoring: If the agent expresses uncertainty ("I think," "probably," "it seems"), route to a human review queue instead of presenting it as a definitive answer.

This is where context engineering earns its keep. The better the context pipeline feeding the agent, the less validation catches, and the lower the human review burden.

7. Observability: traces, not just logs

Logging the agent's final response is useless for debugging. You need full execution traces that capture:

• Every LLM call: input, output, model used, latency, token count, cost.
• Every tool call: function name, arguments, response, latency, success/failure.
• Every decision point: why the agent chose tool A over tool B, what context influenced the decision.
• The full context window at each step: what the agent "saw" when it made each decision.

I use LangSmith or Arize Phoenix for this. The trace is the medical record of the agent's execution. When something goes wrong — and it will — the trace is how you diagnose, fix, and prevent recurrence.

8. Evaluation framework (offline and online)

You cannot improve what you do not measure. Agent evaluation operates at two levels:

Offline evaluation: Before deployment, run the agent against a curated test suite of 50–200 representative queries. Measure:

• Task completion rate (did the agent accomplish what was asked?)
• Correctness (was the answer factually right?)
• Tool selection accuracy (did it use the right tools?)
• Cost per task (how much did each test case cost?)
• Latency (how long did each test case take?)

Online evaluation: In production, continuously measure:

• User satisfaction (thumbs up/down, escalation rate)
• Fallback rate (how often does the agent escalate to a human?)
• Cost per interaction (is it within budget?)
• Drift detection (are the agent's performance metrics degrading over time?)

The offline suite is your gate. If a code change causes task completion rate to drop by more than 2 points, the deployment is blocked. The online metrics are your early warning system.

Tier 3: Trust (your agent will be killed by the CISO without these)

9. Agent identity and credential management

An AI agent is a non-human actor in your system. It needs:

• A dedicated service identity — not a shared API key, not the developer's personal credentials, not "admin."
• Scoped permissions — the agent should have access to exactly the tools and data it needs, and nothing more. If the agent is a customer support agent, it does not need access to the HR database.
• Credential rotation — API keys and service tokens should rotate automatically on a schedule.

Ninety-two percent of large-enterprise CISOs lack full visibility into their AI agents' credentials. That statistic alone should terrify anyone deploying agents in a regulated industry.

10. Audit trail for every action

Every action the agent takes — every tool call, every database write, every external API request — must be recorded in an immutable audit log with:

• Timestamp
• Agent identity
• Action taken
• Input context (what information the agent had when it decided to act)
• Outcome (success/failure, response data)
• User who initiated the session (if applicable)

This is not optional. It is a regulatory requirement in financial services, healthcare, and any industry subject to the EU AI Act. But even if you are not in a regulated industry, the audit trail is how you debug production incidents, investigate complaints, and build trust with stakeholders.

11. Human escalation paths

Every agent needs a clearly defined escalation path. The agent must know when to stop trying and ask for help. I define escalation triggers as:

• Confidence below threshold: The agent is not sure about the answer.
• Risk above threshold: The requested action exceeds the agent's authority level.
• Budget exhausted: The agent has hit its turn or token limit without completing the task.
• Explicit user request: The user asks to speak to a human.

The escalation must be seamless. The human who takes over should receive the full execution trace, the user's original query, and the agent's partial work — not a blank screen with "the AI could not help."

Tier 4: Scale (your agent will not survive growth without these)

12. Rate limiting and queue management

When your agent goes from 10 users to 10,000 users, the LLM API becomes the bottleneck. You need:

• Per-user rate limits to prevent a single power user from consuming the entire quota.
• Queue management with priority tiers — premium users get lower latency, batch jobs run at off-peak hours.
• Circuit breakers that detect when the LLM provider is degraded and route to a fallback (cached responses, simpler model, or graceful "try again later" message).

13. Knowledge grounding with a knowledge graph

The agent that hallucinates is the agent that gets killed. The most effective grounding strategy I have deployed is a knowledge graph backed by Neo4j that serves as the agent's "source of truth."

Instead of stuffing the context window with raw documents and hoping the model finds the right answer, the agent queries the knowledge graph for structured, validated, timestamped facts. The graph knows what is true, what was true, and what has changed. The model reasons over structured knowledge instead of noisy text.

This is the architecture I describe in detail in my GraphRAG post and my graph algorithms post. It is the single most impactful change you can make to agent reliability.

14. Versioned deployments and rollback

Agent deployments are not one-shot. You will iterate on prompts, tools, guardrails, and model selections continuously. Every deployment must be:

• Versioned: Every combination of system prompt + tool definitions + model selection + guardrail configuration is a versioned artifact.
• A/B testable: New versions are rolled out to a percentage of traffic before going to 100%.
• Rollback-ready: If the new version degrades any key metric, you can revert to the previous version in under 60 seconds.

This is standard practice for web applications. It is exotic territory for most AI teams. That gap is why agents break in production.

The architecture in one diagram

Here is how these 14 points fit together in the production stack I deploy:

┌─────────────────────────────────────────────────┐
│                   User Interface                 │
│              (Chat, API, Automation)             │
└─────────────────────┬───────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────┐
│              Gateway / Rate Limiter              │
│          (Auth, Rate Limits, Queue Mgmt)         │
└─────────────────────┬───────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────┐
│              Semantic Router                     │
│     (Intent → Model Selection → Cost Tier)       │
└─────────────────────┬───────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────┐
│              Agent Orchestrator                  │
│     (LangGraph · Budget · Timeouts · Turns)      │
│                                                  │
│  ┌──────────┐  ┌──────────┐  ┌───────────────┐  │
│  │ Tools    │  │ Knowledge│  │ Guardrails    │  │
│  │ (Scoped) │  │ Graph    │  │ (Validation)  │  │
│  └──────────┘  └──────────┘  └───────────────┘  │
└─────────────────────┬───────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────┐
│              Observability Layer                 │
│    (Traces · Evals · Cost · Drift · Alerts)      │
└─────────────────────┬───────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────┐
│              Audit & Governance                  │
│   (Immutable Log · Escalation · Compliance)      │
└─────────────────────────────────────────────────┘

Every layer has a single responsibility. Every layer is independently testable. Every layer can be swapped without affecting the others. This is not over-engineering — it is the minimum viable architecture for an agent that will run unsupervised in production.

The contractor vs. architect gap

Here is the uncomfortable truth about the AI engineering market in mid-2026: there is a bimodal distribution of rates, and the dividing line is exactly this checklist.

The €400–500/day contractor builds the agent. They wire up LangChain or LangGraph, connect a few tools, write a system prompt, and deploy. The demo works. The POC impresses. Then the agent hits production and the problems start. The contractor does not know how to fix them because they have never shipped an agent at scale. They patch symptoms. They add retry logic. They increase the token limit. The costs spiral. The project stalls.

The €1,000–2,000/day architect builds the system around the agent. They design the budget enforcement, the semantic routing, the guardrail pipeline, the observability stack, the audit trail, the escalation paths, and the deployment pipeline. They have seen the failure modes because they have operated agents in production. They do not build agents — they build agent-grade infrastructure.

The model is a commodity. The engineering is the differentiator. The companies that understand this hire architects. The companies that do not hire contractors, burn through their budget, and then hire architects to clean up the mess.

If you are an AI engineer reading this, the path from the first category to the second is not more certifications or more side projects. It is production experience with production consequences. Ship an agent. Watch it break. Fix it. Watch it break differently. Fix it again. Build the infrastructure that prevents it from breaking. That infrastructure is the checklist above.

The bottom line

The agent graveyard is filling up because the industry treated agent deployment like a model selection problem. It is not. It is a systems engineering problem with model selection as one component among fourteen.

The 14-point checklist is not a suggestion. It is the minimum viable standard for any AI agent that will interact with real users, real data, and real money. Skip any of the Tier 1 items and your agent will crash. Skip any of the Tier 2 items and your agent will embarrass you. Skip any of the Tier 3 items and your CISO will kill it. Skip any of the Tier 4 items and it will not survive its first thousand users.

The teams that follow this checklist ship agents that run unsupervised, stay within budget, and compound value over time. The teams that skip it join the 40% cancellation statistic.

Choose your category.

Building agents that need to survive production? I architect agent-grade infrastructure for enterprises — from budget enforcement and knowledge graph grounding to full observability and governance. One checklist. Zero compromises. Let's talk.