Agent SDK: Production Deployment Patterns

Why Production Deployment Matters for Agents

Here's the thing most agent tutorials skip: local development and production are fundamentally different environments. Your laptop has unlimited retries, instant feedback loops, and the ability to restart everything if something goes sideways. Production has users, SLAs, competing workloads, and exactly one chance to get it right.

Agents add specific challenges on top of traditional service deployment. They're stateful—they maintain conversation history, maintain internal reasoning, and make autonomous decisions that depend on external context. They call external APIs, some of which timeout or fail unpredictably. They generate text, which means variable latency and variable output sizes. And they orchestrate other services, which means failure modes that are hard to predict when they're all integrated together.

But here's what most people miss: agents also have longer execution times than traditional services. A simple REST API returns in 200ms. An agent might think for 5, 10, even 30 seconds while calling Claude, processing the response, deciding on the next action, calling tools, and iterating. This changes everything about how you deploy.

Long execution times mean:

• Connection pooling matters more: You can't just create a new connection per request; you'll exhaust your connection limits.
• Request timeouts need to be generous: Your load balancer can't timeout at 30s if your agent legitimately takes 25s.
• Memory pressure is higher: Multiple concurrent requests = multiple agent reasoning contexts in memory simultaneously.
• Graceful shutdown is critical: Draining in-flight requests takes actual time, not milliseconds.

We're going to walk through containerization, health checks, scaling, secrets rotation, and blue-green deployment strategies that handle these realities. Think of this as the "operations manual" for production agents.

Containerization: Building Your Agent Image

Docker isn't fancy—it's just a portable, reproducible environment. But that simplicity is exactly why it matters in production. You want the exact same code, dependencies, and runtime between your laptop and your production servers. Container standardization eliminates the classic "it works on my machine" problem that plagues software teams.

The power of containers is reproducibility at scale. You build a container image once, test it thoroughly, then deploy that exact same image to hundreds of servers. There's no "environment drift" where production servers slowly diverge from each other. Every server runs exactly the same image. This uniformity is powerful and prevents entire classes of production bugs.

Furthermore, containers enable rapid deployment and rollback. Building, pushing, and deploying a new container image takes minutes. If something goes wrong, rolling back to the previous image takes seconds. This speed is valuable for rapid iteration and error recovery.

Container security is also important. A containerized application runs in isolation from the host system. If your application is compromised, the damage is confined to that container. The host system and other containers are protected. This isolation is defensive architecture—if one thing goes wrong, it doesn't cascade.

The foundation of containerization is understanding that your agent application needs a reproducible build process. Every time you rebuild, you want identical binary output. This means pinning versions, managing dependencies carefully, and avoiding environment-specific configurations in your image. Containers are meant to be ephemeral—they should start, run, stop, and be replaced without data loss or state issues.

When building your container, the multi-stage approach is critical. Development dependencies like TypeScript compilers, linters, and test runners shouldn't ship to production. They add hundreds of megabytes to your image and expand your attack surface. A builder stage contains all the tools needed to prepare your application; the final stage contains only what's needed to run it. This architectural pattern keeps production images lean and fast to deploy.

The security principle of running as a non-root user is often overlooked but crucial. If your container is compromised, the attacker gets whatever privileges the running user has. By running as a regular nodejs user instead of root, you're limiting the damage they can do. This is defense-in-depth—it won't stop a determined attacker, but it raises the bar significantly.

Docker layer caching is a performance multiplier. If you install dependencies before copying source code, Docker caches that layer. When you change only source code (not dependencies), Docker reuses the cached layer instead of reinstalling everything. On your next deployment, you save 2-3 minutes of build time. For teams deploying dozens of times per day, this compounds dramatically.

Health checks at the container level are your first line of defense. Docker can probe your container periodically and restart it if it becomes unresponsive. This isn't perfect—a truly stuck container might still appear healthy—but it catches obvious failure modes. The key is keeping the health check lightweight. A simple HTTP GET shouldn't require complex logic or external calls.

Building Smaller Images

Image size matters more than most developers realize. Smaller images mean faster downloads, less disk space, faster startup times (less data to decompress), and generally better performance in constrained environments. Every megabyte adds up, especially when you're deploying to hundreds of containers.

Removing npm cache saves about 50MB per image. Cleaning up build artifacts saves another 50-100MB. Using distroless base images (which contain only the Node runtime, no OS utilities) saves another 100-150MB compared to Alpine-based images. For large-scale deployments, these optimizations reduce network transfer time by half and storage costs significantly.

The tradeoff is debuggability. A distroless image contains no bash, no curl, no debugging tools. This is actually a feature in production (less to go wrong, fewer attack vectors) but makes troubleshooting harder. Solution: use a larger debugging image for dev/staging, and smaller production images for production. Your CI pipeline builds both.

Health Checks and Monitoring: Knowing When Things Break

A service that responds to traffic isn't the same as a service that's actually working. Your agent might be in a weird state—memory leak from unclosed connections, stuck in an infinite reasoning loop, or waiting on an external API that's in a degraded state. The burden is on you to detect these conditions and alert your team before users notice.

Health checks solve this by separating concerns into three levels: liveness, readiness, and comprehensive health. Liveness answers "is the process alive?" Readiness answers "should the load balancer send traffic here?" Health answers "what's the detailed status?" These are fundamentally different questions with different implications.

The liveness probe is ultra-lightweight because it runs frequently (every 10 seconds, potentially). It's just checking if the Node process is still running. If it fails three times in a row, the orchestrator (Kubernetes, Docker Swarm, CloudRun) restarts the container. This is useful for catching processes that hang or deadlock.

The readiness probe is more sophisticated. It checks dependencies, memory usage, queue depth, and external service connectivity. If it fails, the orchestrator drains traffic from that instance but doesn't restart it. This prevents cascading failures. If all instances become unready, your service is unhealthy system-wide, which triggers alerting.

The full health endpoint is the most expensive. It runs less frequently (every 30-60 seconds) and returns detailed diagnostics. Your monitoring system scrapes this and surfaces metrics to your dashboards. This is where you can afford more complex checks like "verify database connectivity" or "check queue processing rate."

Queue age tracking is specifically important for agents. A queue of 10 tasks is fine if they're fresh—maybe one is being processed, nine are waiting a few seconds. But those same 10 tasks sitting for 5 minutes means something is stuck. Maybe Claude API calls are timing out repeatedly, maybe your agent is in an infinite loop, maybe a downstream service is overloaded. Queue age gives you early warning that something has gone wrong before you start seeing user complaints.

Memory monitoring prevents out-of-memory crashes. If an agent holds 50MB per request and you get 10 concurrent requests, that's 500MB. Add Node's own overhead and garbage collection, and you're easily at 1GB of memory per instance. Running near your memory limits means garbage collection pauses get longer, which means requests get slower, which causes more requests to queue, which uses more memory. It's a death spiral. Monitoring memory and draining traffic before you hit the limit prevents this.

Health Check Gotchas

Timeout misconfiguration is the most common mistake. If your health check timeout is shorter than the actual time it takes to run, you trigger false positives. The container restarts even though it's actually healthy. This is especially true with /health which might take 5-10 seconds if it checks external databases or caches.

Cascade failures are subtler. If your health check depends on Redis and Redis goes down, every instance becomes unhealthy at once. Your orchestrator might restart all of them simultaneously, creating a thundering herd effect. Better to report "degraded" with reduced functionality (maybe caching is disabled but core features work) than "unhealthy" and trigger a full cascade.

Connection leaks in health checks are sneaky. If you create a new database connection in every health check and never close it, you'll exhaust your connection pool within minutes. Health checks are a common place for this because developers don't think as carefully about cleanup in monitoring code. Reuse connections or use connection pooling.

Scaling Agents: Multiple Instances and Load Balancing

Now you have one agent service running. But production traffic varies—3 AM might see 2 requests per minute; 2 PM might see 200. You need multiple instances and intelligent distribution.

Here's the crucial detail about agent scaling: agents are mostly stateless from the HTTP perspective, but conversation history lives somewhere. You have three patterns, each with tradeoffs.

Pattern 1: Shared State (Recommended for most cases)

Agent state lives in Redis or a database. Requests go to any instance; state is retrieved from the shared store. This is simple to scale—add more instances and they all access the same state. The downside is network latency (every request involves Redis round trips) and potential consistency issues if you're not careful with concurrency.

Redis is particularly good here because it's fast, supports key expiration (automatic cleanup), and handles concurrent access safely. Conversation histories can grow large, so you need cleanup strategy—maybe keep only the last 100 messages per conversation, or auto-expire after 24 hours.

Pattern 2: Sticky Sessions (For stateful agent reasoning)

If your agent has complex internal state that's expensive to recompute (like a large language model context window that's been carefully constructed), sticky sessions might be better. All requests from one user go to the same instance. If instance-2 crashes, that user's session is lost, but 95% of users continue working.

The tradeoff is that sticky sessions prevent horizontal scaling. You can't easily add a new instance because some users are always pinned to old ones. Also, you need circuit breaker logic to handle instance failures gracefully.

Pattern 3: Distributed Workers (For async task processing)

Agents process long-running tasks asynchronously. A request submits a task, returns immediately with a task ID, and the client polls for results. This is the most scalable pattern because HTTP workers are separate from agent workers. You can scale each independently.

The tradeoff is complexity and eventual consistency. The client doesn't get an answer immediately; they need to poll. This works well for non-interactive use cases (batch processing, background jobs) but less well for chat interfaces where users expect immediate responses.

Kubernetes Deployment (Putting It Together)

Kubernetes orchestrates all of this. You define a deployment with 3 replicas; Kubernetes ensures 3 instances are running. When you deploy a new version, Kubernetes uses rolling update strategy: spin up new pod, wait for readiness probe to pass, drain traffic from old pod, terminate old pod. Zero downtime.

The resource requests and limits are critical. Requests tell Kubernetes "this pod needs at least 256MB memory and 250m CPU to function." Limits tell it "if this pod uses more than 512MB or 500m CPU, kill it." Kubernetes uses requests for scheduling decisions—if your node has only 512MB free and your request is 256MB, Kubernetes fits 2 pods. If a pod goes over limits, Kubernetes kills it. This prevents one runaway agent from starving others.

For agents, be generous with resource limits. An agent sitting in Claude's API call for 30 seconds might hold 50-100MB in memory waiting for the response. Undersizing leads to OOM kills, which leads to cascading failures.

Secrets Management and API Key Rotation

You're calling Claude's API. Your ANTHROPIC_API_KEY is the crown jewel. Commit it to git? Someone on the internet will find it in 10 minutes and rack up charges.

The pattern is: never store secrets in code, configs, or even Docker images. Retrieve them from a secrets manager at runtime. AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, or even HashiCorp Vault all follow this pattern. Your Kubernetes cluster has a secrets API built in.

The key insight is caching. You can't call Secrets Manager for every request—that's too much latency and too much cost. So you cache the secret with a TTL (time-to-live). After an hour, you refresh. This balances latency against rotation speed.

Rotating API Keys Without Downtime

What if your key is compromised? You need to rotate it—swap the old key for a new one—without interrupting service. The pattern is a grace period. When you rotate, you maintain both old and new keys for a few minutes. In-flight requests use the primary key; if one fails, they try the backup key. After the grace period, old requests should be done, so you discard the backup and delete the old key from your provider.

This requires careful implementation. You need to track which key is active, when the rotation started, and provide fallback logic. The grace period should be longer than your max agent execution time.

Blue-Green Deployment: Zero-Downtime Updates

You've built a better version of your agent. It has smarter prompt engineering, faster reasoning, fewer hallucinations. You want to deploy it without dropping a single request.

Blue-green deployment means running two complete production environments: one handles traffic (blue), one is idle (green). To deploy: deploy to green, test green, switch traffic to green, blue becomes idle backup. If something goes wrong, switch traffic back. Instant rollback.

This is dramatically simpler than rolling updates because there's no "in-between" state where some requests hit old code and some hit new. Either all traffic is on blue or all traffic is on green. This simplicity is worth the extra infrastructure cost (running two environments simultaneously).

The orchestration is straightforward with Kubernetes: you have two deployments and a service that points to one or the other. To switch, you patch the service selector. Takes 10 seconds.

Monitoring and Observability

You have three instances, Redis, an API you're calling. When latency spikes or errors increase, where do you look?

Structured logging is your foundation. Instead of logging "got message", log JSON with context: {"timestamp": "2026-03-17T14:23:45Z", "level": "info", "message": "processed request", "conversationId": "abc123", "duration": 2341}. Your logging platform parses this JSON and you can now query "show me all requests for conversationId=abc123" or "show me requests slower than 5 seconds."

Metrics are the companion to logging. Count requests, track latency percentiles (p95, p99), measure token usage, count errors. Export to Prometheus, CloudWatch, or Datadog. Set up alerts: error rate > 5% for 5 minutes = page someone. P95 latency > 5s = scale up.

The combination of structured logs and metrics gives you the observability you need. Logs tell you "what happened" (specific request failed with specific error). Metrics tell you "is this a trend?" (error rate spiked 5 minutes ago).

Testing Before Production

You've built production deployment infrastructure. Does it work? You need tests that actually deploy and verify.

Smoke tests are your starting point. Deploy to staging, run tests that verify: can you make a request, does health check pass, can you process a chat request end-to-end. These are simple but catch obvious failures.

Load tests verify scaling. Simulate 100 concurrent requests, measure latency distribution, verify no requests fail. This reveals bottlenecks early.

Chaos tests deliberately break things. Stop Redis, verify service is degraded but not down. Kill a Kubernetes pod, verify traffic fails over to another. These test your resilience assumptions.

Advanced: Circuit Breakers and Fallback Strategies

Real production systems need resilience patterns that go beyond health checks. When external dependencies fail (Claude API times out, Redis goes down, your database is slow), your system should gracefully degrade rather than fail completely.

Circuit breaker pattern prevents cascading failures. If Claude API responses timeout, you start failing requests quickly instead of waiting for timeouts. After a threshold of failures, you open the circuit—stop trying, return a cached response or a degraded answer. After a waiting period, you try again cautiously. This pattern is essential for agent systems that make many external calls.

Implement circuit breakers for every external dependency: Claude API, Redis, database, any tool services you call. A single flaky external service shouldn't bring down your entire agent system.

Fallback strategies are equally important. If Claude API is down, maybe you have a cached response from a previous execution. Maybe you have a simpler, faster model you can call (Claude Haiku instead of Opus). Maybe you can queue the request and process it later. Building these fallbacks into your agent architecture means graceful degradation instead of failure.

Disaster Recovery and Backup Strategies

Production systems need backup and disaster recovery plans. What happens if your production database is corrupted? What happens if Redis cache is lost? What happens if a cloud region goes down?

For agents, conversation history is often your most critical data. Back it up regularly. Use database replication (primary-replica setup) so you have a standby replica. Use automated failover so if the primary dies, the replica takes over instantly.

For long-running agents, consider checkpointing. Every 5 minutes, save the agent's current state (conversation history, partial reasoning, pending tasks) to durable storage. If the agent crashes mid-reasoning, it can resume from the checkpoint instead of starting over. This saves compute cost and provides better user experience.

Test your disaster recovery plans. It's not a disaster recovery plan if you've never tested it. Run quarterly DR tests where you simulate failures and verify your team can recover within your RTO (Recovery Time Objective).

Cost Optimization for Agent Systems

Agent systems are compute-intensive. Claude API calls are expensive. Long-running requests consume resources. As your system scales, costs can spiral.

Implement caching strategies. If the same conversation or query pattern repeats, cache the Claude response and reuse it. This dramatically reduces costs for conversational systems where similar queries repeat.

Implement cost tracking. Every request should track its cost (API calls made, tokens consumed). Aggregate daily, weekly, monthly. Set budgets and alerts if spending exceeds expectations. Empower teams to see the cost impact of their features.

Optimize your prompts. Longer prompts consume more tokens. Better-engineered prompts get better responses with fewer iterations. Invest in prompt engineering that reduces token consumption. Every token you can eliminate saves money.

Understanding Agent Latency Characteristics

Agent latency is different from traditional service latency. A REST API's latency is deterministic—it takes roughly the same time every time. Agent latency is variable and dependent on many factors: Claude API response time (variable), number of reasoning steps (variable), external API calls (variable), data processing (variable).

This variability is fine for background jobs but problematic for interactive systems. If your users expect responses within 5 seconds but your agent sometimes takes 20 seconds, they'll have a poor experience.

Strategies for managing latency: implement timeouts so requests never take arbitrarily long, break complex reasoning into multiple steps where the first step responds quickly, use streaming responses so users see partial results while full reasoning happens, run non-critical reasoning asynchronously and deliver partial answers immediately.

Monitoring Agent-Specific Metrics

Standard service metrics (request count, latency, errors) are necessary but insufficient for agents. You need agent-specific metrics.

Token consumption per request: tracks cost and efficiency. If token consumption is trending up, your prompts are getting worse or your reasoning is less efficient.

Reasoning step count: how many iterations does each request take? If trending up, your agent is getting less efficient or the problems are getting harder.

External API call success rate: how often do tool calls succeed? If trending down, your tools are failing more frequently.

User satisfaction metrics: if you have user feedback, track it. An agent that's technically efficient but produces unhelpful responses is still failing.

Security Considerations for Production Agents

Agents have unique security challenges. They call external APIs, they reason about user data, they make autonomous decisions. Security is critical.

Never include secrets in prompts. If your agent includes API keys or auth tokens in Claude messages, those secrets are sent to Anthropic. Use environment variables and only reference keys by name, never by value.

Validate all tool outputs. An agent tool might be compromised or return malicious data. Never trust tool outputs—validate, sanitize, and use safely.

Implement rate limiting and abuse detection. If an agent starts making thousands of requests per second, detect and block it before it runs up huge charges.

Audit all agent decisions. For consequential decisions (payments, account changes, notifications), log them so you can audit and investigate issues.

Wrapping Up: The Production Checklist

Before you deploy your agent to production, verify:

• Container image builds and runs locally
• Health checks pass and report useful information
• Multiple instances can run simultaneously without conflicts
• Secrets are never committed or logged
• API key rotation can happen without downtime
• Blue-green deployment switches traffic cleanly
• Logging produces structured, searchable records
• Metrics track latency, errors, and queue depth
• Tests pass against the actual deployed service
• Circuit breakers prevent cascading failures
• Disaster recovery plan exists and is tested
• Cost tracking is in place and alerts configured
• Agent-specific metrics are monitored
• Security practices are documented

Production isn't scary if you plan for failure. Assume something will break—a network call will timeout, an API will go down, your container will crash. The patterns in this article are designed so that when those things happen, you know about them, your system degrades gracefully, and you can fix things without waking up at 3 AM.

Your goal isn't perfection. Your goal is graceful degradation, observable failures, and rapid recovery. Build systems that fail visibly, not silently. Monitor everything. Prepare for disasters. Then deploy with confidence.

The Operational Reality: Why Development Isn't Production

Here's something they don't teach in computer science programs: development and production are fundamentally different environments. Your laptop has unlimited retries, instant feedback loops, and you personally watching when things break. Production has users, SLAs, competing workloads, and the harsh reality of entropy. Things that work great on your machine fail mysteriously in production because they hit edge cases you never tested.

Agent systems amplify this problem. They're inherently complex—multiple external calls, variable latency, stateful reasoning, autonomous decisions. A bug that appears once per thousand requests in dev might appear ten times per day under production load. Timeout configurations that work for your test traffic might thrash under real traffic patterns.

The painful truth: you cannot test production in development. You can get close, but not exact. This is why operational patterns matter so much. You can't prevent all failures—but you can architect so that when failures happen, you detect them, respond to them, and recover from them gracefully.

The Latency Surprise: Why Agent Timing Is Different

One thing that catches teams off guard: agent latency is fundamentally different from normal service latency. A REST API endpoint returns in 100-300 milliseconds typically. An agent might be genuinely thinking for 5, 10, or 30 seconds while iterating with Claude, calling tools, processing responses. This isn't slowness. This is the nature of agent reasoning.

Your infrastructure needs to accommodate this. Load balancers configured for 30-second timeouts will silently drop agent requests. Request queues will back up because agents take longer. Kubernetes pod disruption budgets will kill agents mid-reasoning.

More subtly, this changes how you should architect for latency-sensitive use cases. If your users expect responses within 5 seconds, but your agent takes 20 seconds, you have an architecture problem. Some teams solve this with streaming—start returning partial results while the agent continues reasoning. Others use asynchronous patterns—return immediately with a task ID, let the user poll for results. Others accept that some use cases just aren't amenable to agent processing and use simpler systems.

The key insight: don't force agent latency into a synchronous request-response model. Understand your latency requirements upfront and architect accordingly.

Resource Isolation and Blast Radius Control

When things break in production—and they will—you want failures isolated. One crashing agent shouldn't bring down the entire service. One spike in API usage shouldn't starve legitimate traffic.

This is why resource limits matter. Set memory limits on your pods. If one agent holds 2GB, the pod dies rather than exhausting all memory and killing siblings. Set CPU limits. If one agent gets stuck in infinite reasoning, it's throttled rather than starving other requests.

Set concurrency limits explicitly. If you allow 1000 concurrent agent threads but your API quota supports 100, you'll thrash—each thread competing for scarce API calls, context switching overhead overwhelming actual work. Better to explicitly queue and process 100 at a time, getting real throughput instead of illusion of parallelism.

Implement circuit breakers not just for external APIs, but for your own resources. If memory usage exceeds 80%, stop accepting new requests. Queue them in an external system, process them when capacity returns. This is painful—users see queueing—but less painful than cascading failures.

Building for Observability From Day One

You cannot fix what you cannot see. Build observability into every layer from the start. Structured logging (JSON with context) lets you query logs like a database. Metrics (request counts, latencies, errors) reveal trends. Distributed tracing follows requests through multiple services. None of this is optional if you want to operate confidently.

The key is building observability as part of your system architecture, not bolting it on later. Every agent should log structured events: request start, API calls made, reasoning iterations, final result, any errors. Not printf debugging—structured events that your logging system can index and query.

Your metrics should be comprehensive but not overwhelming. Core four: requests, latency, errors, saturation. Then add agent-specific metrics: tokens per request, reasoning iterations, tool call success rates. Track costs—API calls are expensive. If token consumption is trending up, investigate why.

Distributed tracing shows the end-to-end path. A user request comes in, gets routed to an agent, the agent calls Claude, Claude returns, the agent calls your database, database returns, agent processes, response goes back. Trace the whole thing. When latency spikes, see which service is slow. When failures cascade, see where they originated.

The Reality of Team Handoff

Eventually, you're going to hand your system off to an operations team (or on-call teammates) to run. No matter how well you document it, they won't understand it as deeply as you do. This is where operational patterns become critical.

Write runbooks. Specific, step-by-step guides for common incidents. "Latency spike? Check these three metrics. If metric A is high, do X. If metric B is high, do Y." Runbooks are boring, but they let people unfamiliar with your system respond effectively to incidents.

Make your system defensible. Don't require expert knowledge to keep it running. Good defaults, clear alerts, obvious failure modes. If something breaks, it should be obvious that it's broken (not a subtle degradation), obvious what's breaking (look at the error message and know what to fix), and obvious how to fix it (your runbook tells you).

Document decisions. Why did you choose Kubernetes over CloudRun? Why did you cap concurrency at 20? When on-call engineers understand the "why" behind decisions, they can make good operational choices when unexpected situations arise.

The Economics of Reliability

There's a classic tradeoff between reliability and cost. Higher reliability costs money—redundancy, more comprehensive monitoring, better infrastructure. Eventually you hit diminishing returns where 99.99% availability costs 10x more than 99%.

But reliability has business value. Every minute of downtime costs money—users can't use the system, you lose revenue, you lose trust. So there's a break-even point where reliability investment pays for itself in reduced downtime costs.

Figure out your break-even point. What's the cost per minute of downtime? What's the cost of the infrastructure to reduce downtime by 1 minute per year? If the cost of infrastructure is less than the cost of downtime, invest. If it's more, accept some downtime.

This isn't cynical—it's realistic. Not every system needs five-nines availability. Some systems can tolerate occasional downtime. But you should make that decision consciously, with economics in mind, not accidentally.

The Peace of Mind Moment

There's a specific moment when production becomes less scary. It's when you deploy a change, watch metrics tick up, watch your alerts trigger if anything goes wrong, watch your team respond calmly because they've seen the patterns before. You know your system will handle problems because you've built it to.

That peace of mind is worth the effort of building proper operational patterns. The setup cost is real, but the ongoing confidence is priceless. You'll ship more, iterate faster, and sleep better knowing your systems are operating as intended.

-iNet