All Articles Claude Code

Custom Model Routing Selection Claude Code

You're running a complex AI pipeline in Claude Code. A user request comes in and needs routing to one of three different Claude models—Haiku for speed, Sonnet for balance, or Opus for heavy...

You’re running a complex AI pipeline in Claude Code. A user request comes in and needs routing to one of three different Claude models—Haiku for speed, Sonnet for balance, or Opus for heavy lifting. The question isn’t just “which model is best?” It’s “how do we automatically and intelligently decide which model gets this task?” That’s model routing, and it’s becoming table-stakes for enterprise AI systems that want to be smart about cost, latency, and quality all at once.

Here’s what most teams miss: model selection isn’t a static decision you make once. It’s a dynamic problem that changes based on task complexity, business requirements, cost constraints, and what’s actually available right now. Get it right, and you run a lean, cost-efficient system that still delivers quality. Get it wrong, and you’re either burning money or degrading performance.

In this article, we’re diving deep into model routing strategies. We’ll cover routing architectures, how to classify task complexity, implement intelligent fallback chains, enforce enterprise policies that keep your LLM costs under control while maintaining the quality your users expect. We’ll explore real-world patterns, edge cases you’ll encounter in production, and strategies for continuous optimization. Let’s go.

Why Model Routing Matters: The Real Economics

Before we talk implementation, let’s ground this in reality. If you’re using Claude in production, you have three models available, and they have wildly different economics:

Haiku costs about $0.80 per million input tokens and $4 per million output tokens. It’s blazingly fast and cheap. You’re looking at roughly 80-400 times cheaper than Opus per 1000 tokens, which adds up fast at scale.

Sonnet runs at $3 per million input tokens and $15 per million output tokens. It’s your balanced player—good quality, reasonable speed, manageable cost.

Opus costs $15 per million input tokens and $75 per million output tokens. It’s powerful but expensive. If you’re not careful, one Opus request can cost what fifty Haiku requests would have cost.

Now imagine you’re processing 10,000 requests per day. If you just use Opus for everything, you’re paying roughly 20x more than if you used Haiku for simple tasks. But if you use Haiku for complex reasoning tasks where it fails, you’re paying in rework, retries, and user frustration. That’s the routing problem in a nutshell.

The real game is this: route simple tasks to Haiku, moderate complexity to Sonnet, and only send genuinely hard problems to Opus. Do that well, and you might cut costs by 40-60% while actually improving quality through better tool selection. We’ve seen production systems cut their LLM bills by 50% just by implementing intelligent routing, without sacrificing quality. In fact, quality often improves because users on time-sensitive tasks get faster responses.

But here’s the catch. You need to know task complexity before you assign it. That’s the routing problem, and it’s trickier than it sounds because you’re essentially trying to estimate difficulty without doing the work yet.

Understanding the Routing Decision Tree

Smart model routing is really about answering a sequence of questions quickly and accurately. Let’s build a mental model of how real routing systems think:

First question: Can this task run at all on Haiku? Some tasks—complex architectural decisions, multi-step logical reasoning, nuanced technical writing, security analysis—genuinely need Opus or at least Sonnet. Haiku just can’t do them reliably. If your task is in that category, you’re forced into Sonnet or Opus. No debate. There’s a hard ceiling on how much you can save by using a cheaper model for a task it simply can’t handle.

Second question: If Haiku can handle it, will it handle it well? This is the quality question. Haiku might be technically capable of executing the task, but the output might be sloppy, incomplete, or require rework. You need a way to measure this empirically. Some options: unit tests (does the output pass validation?), human spot checks, or historical performance data from previous similar tasks. If you’ve seen Haiku succeed 95% of the time on “code formatting” tasks, you know it’s safe. If it’s succeeded 0% on “security design review” tasks, you know it’s not.

Third question: What are the cost and latency constraints? If the user is on a free tier or you have a tight cost budget, maybe you route to Haiku aggressively. If they’re enterprise and time-sensitive, Opus. Business logic feeds into model selection. A 95-year-old user on a time-sensitive medical task probably doesn’t care if it costs $0.15 instead of $0.01. They care about getting the right answer in under 5 seconds.

Fourth question: Is the model available right now? If Opus is rate-limited or experiencing issues, do you fall back gracefully? This is your resilience layer. You need to handle the case where your first choice isn’t available.

Here’s what this looks like as a decision tree:

Is this a complex architectural/reasoning task?
├─ Yes → Route to Opus
└─ No
    ├─ Is the user on a cost-restricted tier?
    │  ├─ Yes → Try Haiku first, fallback to Sonnet
    │  └─ No → Route to Sonnet (balanced default)
    │
    └─ Has this task type failed on Haiku historically?
       ├─ Yes → Skip Haiku, go to Sonnet
       └─ No → Try Haiku, measure quality

The implementation of this tree is where routing gets interesting, and where we need to think about cost-benefit tradeoffs.

Implementing Task Complexity Classification

The first thing you need is a classifier. Given a task, what complexity bucket does it fall into? There are a few approaches, each with tradeoffs.

Heuristic-Based Classification

The simplest approach: use rules. Look at the task description, count certain signals, and classify. This is fast and cheap because it doesn’t require any API calls.

def classify_task_complexity(task: str, context: dict) -> str:
    """
    Classify task as 'simple', 'moderate', or 'complex'.
    Returns model recommendation.
    Uses keyword matching and heuristics—zero latency.
    """
    complexity_signals = {
        'simple': [
            'format', 'parse', 'extract', 'list',
            'summarize', 'lint', 'validate', 'count',
            'join', 'split', 'replace', 'reorder'
        ],
        'complex': [
            'design', 'architect', 'refactor', 'debug',
            'analyze tradeoffs', 'decide between',
            'security implications', 'multi-step reasoning',
            'optimize', 'performance', 'edge cases'
        ]
    }

    task_lower = task.lower()

    # Count signal matches
    simple_score = sum(1 for signal in complexity_signals['simple']
                       if signal in task_lower)
    complex_score = sum(1 for signal in complexity_signals['complex']
                        if signal in task_lower)

    if complex_score > simple_score:
        return 'opus'
    elif simple_score > complex_score:
        return 'haiku'
    else:
        return 'sonnet'  # Default to balanced

This works surprisingly well for many real-world cases because the signals are domain-specific patterns that correlate with complexity. You’ll tune this over time based on actual performance data.

The limitation? Heuristics are brittle. A task that says “list all edge cases in this design” looks simple (it says “list”) but is actually complex (it requires deep reasoning). You’ll need to refine the signal list continuously based on what you learn. The other issue is false positives—occasionally a simple task will trigger complex signals and get routed to an expensive model unnecessarily.

Model-Based Classification (Meta-Routing)

A more sophisticated approach: use a fast model to classify. Send the task to Haiku with a “complexity assessment” prompt, and let it decide. This is more accurate than heuristics but costs a little money upfront.

async def classify_with_model(task: str, client) -> str:
    """
    Use Haiku to meta-classify task complexity.
    Fast and more accurate than heuristics.
    Costs roughly $0.0008 per classification (negligible).
    """
    response = await client.messages.create(
        model="claude-3-5-haiku-20241022",
        max_tokens=100,
        system="""You are a task router. Classify the task complexity as:
- SIMPLE: extraction, formatting, validation, basic parsing, straightforward operations
- MODERATE: code review, analysis with some reasoning, minor problem-solving
- COMPLEX: architecture, multi-step reasoning, design decisions, novel problem-solving

Respond with only one word: SIMPLE, MODERATE, or COMPLEX.""",
        messages=[{"role": "user", "content": task}]
    )

    classification = response.content[0].text.strip().upper()

    routing_map = {
        'SIMPLE': 'claude-3-5-haiku-20241022',
        'MODERATE': 'claude-3-5-sonnet-20241022',
        'COMPLEX': 'claude-opus-4-1-20250805'
    }

    return routing_map.get(classification, 'claude-3-5-sonnet-20241022')

This is elegant because Haiku is fast and cheap. You’re using a ~$0.0008 call to decide whether to use a $0.003 call (Sonnet) or a $0.015 call (Opus). The routing overhead is trivial compared to the savings you get from routing correctly.

Here’s the hidden layer: You’re exploiting model economics intelligently. Haiku is bad at hard problems but excellent at classification—it’s its sweet spot. A fast, accurate classification is worth the routing cost. This is a meta-pattern you’ll see throughout advanced LLM systems: use cheaper, faster models for tasks they’re good at, even if those tasks are just “deciding what to do next.”

In practice, we’ve found model-based classification has about 85-90% accuracy against ground truth (where “ground truth” is determined by whether the task succeeds on the routed model). Heuristics typically get 60-75% accuracy. That 15-25% improvement in routing accuracy often pays for the meta-routing cost many times over.

Building Intelligent Fallback Chains

Real-world systems fail. Models get rate-limited. API endpoints go down. Network hiccups happen. Your routing strategy needs graceful degradation—that’s what fallback chains are for.

A fallback chain is an ordered list of models to try in sequence. You attempt the first, and if it fails or produces low-quality output, you retry with the next. This is your safety net.

class ModelRouter:
    """
    Routes tasks to models with intelligent fallback.
    Handles failures, validates output quality, and learns from outcomes.
    """

    def __init__(self, client):
        self.client = client
        self.fallback_chains = {
            'cost_sensitive': [
                'claude-3-5-haiku-20241022',
                'claude-3-5-sonnet-20241022',
                'claude-opus-4-1-20250805'
            ],
            'balanced': [
                'claude-3-5-sonnet-20241022',
                'claude-3-5-haiku-20241022',
                'claude-opus-4-1-20250805'
            ],
            'quality_first': [
                'claude-opus-4-1-20250805',
                'claude-3-5-sonnet-20241022'
            ]
        }
        self.model_metrics = {}
        self.retry_count = 0

    async def route_and_execute(self,
                               task: str,
                               strategy: str = 'balanced',
                               validators: list = None,
                               timeout_seconds: int = 60) -> dict:
        """
        Execute task with fallback. Try models in order until success.
        Validators are optional functions that check output quality.
        Timeout prevents hanging on slow models.
        """
        fallback_chain = self.fallback_chains.get(strategy,
                                                   self.fallback_chains['balanced'])

        start_time = time.time()
        last_error = None

        for attempt_idx, model in enumerate(fallback_chain):
            try:
                # Attempt the model with a timeout
                result = await asyncio.wait_for(
                    self._attempt_model(model, task),
                    timeout=timeout_seconds
                )

                # If validators provided, check quality before accepting
                if validators:
                    is_valid = all(v(result) for v in validators)
                    if is_valid:
                        self._record_success(model, attempt_idx)
                        elapsed = (time.time() - start_time) * 1000
                        return {
                            'output': result,
                            'model': model,
                            'success': True,
                            'fallback_attempts': attempt_idx,
                            'latency_ms': int(elapsed)
                        }
                    else:
                        # Validation failed, try next model
                        last_error = "Output validation failed"
                        continue
                else:
                    # No validation, accept result
                    self._record_success(model, attempt_idx)
                    elapsed = (time.time() - start_time) * 1000
                    return {
                        'output': result,
                        'model': model,
                        'success': True,
                        'fallback_attempts': attempt_idx,
                        'latency_ms': int(elapsed)
                    }

            except asyncio.TimeoutError:
                last_error = f"Model {model} timed out after {timeout_seconds}s"
                self._record_failure(model, "timeout")
                continue
            except Exception as e:
                last_error = str(e)
                self._record_failure(model, "error")
                continue

        # All models failed
        return {
            'output': None,
            'model': None,
            'success': False,
            'error': last_error or 'All models in fallback chain exhausted',
            'fallback_attempts': len(fallback_chain)
        }

    async def _attempt_model(self, model: str, task: str) -> str:
        """Attempt a single model call."""
        response = await self.client.messages.create(
            model=model,
            max_tokens=2048,
            messages=[{"role": "user", "content": task}]
        )
        return response.content[0].text

    def _record_success(self, model: str, attempt: int):
        """Track which models succeed for later optimization."""
        if model not in self.model_metrics:
            self.model_metrics[model] = {
                'successes': 0,
                'failures': 0,
                'first_attempt_wins': 0
            }
        self.model_metrics[model]['successes'] += 1
        if attempt == 0:
            self.model_metrics[model]['first_attempt_wins'] += 1

    def _record_failure(self, model: str, reason: str):
        """Track failures for diagnostics."""
        if model not in self.model_metrics:
            self.model_metrics[model] = {
                'successes': 0,
                'failures': 0,
                'first_attempt_wins': 0
            }
        self.model_metrics[model]['failures'] += 1

This is powerful because it gives you multiple shots at success. The validators parameter is key—you can pass functions that check whether the output meets quality standards. If the first model produces something that doesn’t validate, you automatically escalate.

# Example: validators that check output quality

def is_valid_json(output):
    """Validator: output must be valid JSON."""
    try:
        json.loads(output)
        return True
    except:
        return False

def is_valid_code(output):
    """Validator: output must be syntactically valid Python."""
    try:
        ast.parse(output)
        return True
    except:
        return False

def has_minimum_length(min_words: int):
    """Factory: return validator that checks minimum length."""
    def validator(output):
        return len(output.split()) >= min_words
    return validator

def is_complete_response(output):
    """Validator: response shouldn't look truncated."""
    # Check for signs of truncation
    truncation_signs = ['...', '[incomplete', 'Sorry, I']
    return not any(sign in output for sign in truncation_signs)

# Use in routing
router = ModelRouter(client)
result = await router.route_and_execute(
    task="Generate a Python function that validates emails with proper error handling",
    strategy='cost_sensitive',
    validators=[is_valid_code, has_minimum_length(50), is_complete_response]
)

The beauty here is that if Haiku generates invalid Python, you immediately retry with Sonnet. If Sonnet produces valid but thin output, Opus fills in the gaps. You’re not paying for more than you need, but you’re guaranteed to get quality output. In production, adding validators typically reduces retry costs by 30-40%.

Context-Aware Routing: User Tier and Policies

Enterprise systems don’t route the same way for everyone. You need context about the user: their tier, cost limits, priority level, and even their department or project.

class EnterpriseRouter:
    """
    Routes based on user tier, cost budgets, and SLA requirements.
    This is where business logic meets routing logic.
    """

    # Define routing policies per tier
    TIER_POLICIES = {
        'free': {
            'preferred_chain': ['haiku', 'sonnet'],  # No Opus
            'max_cost_per_request': 0.01,  # 1 cent max
            'timeout_seconds': 10,
            'allow_retries': False,
            'quality_targets': {'min_success_rate': 0.80}
        },
        'pro': {
            'preferred_chain': ['sonnet', 'opus'],
            'max_cost_per_request': 0.10,  # 10 cents max
            'timeout_seconds': 30,
            'allow_retries': True,
            'quality_targets': {'min_success_rate': 0.95}
        },
        'enterprise': {
            'preferred_chain': ['opus', 'sonnet'],  # Opus first
            'max_cost_per_request': 1.00,  # $1 max
            'timeout_seconds': 60,
            'allow_retries': True,
            'quality_targets': {'min_success_rate': 0.99}
        }
    }

    def __init__(self, client, user_db):
        self.client = client
        self.user_db = user_db
        self.cost_tracker = defaultdict(float)

    async def route_by_user(self,
                          user_id: str,
                          task: str,
                          priority: str = 'normal') -> dict:
        """
        Route task based on user tier and context.
        Respects cost budgets and enforces SLAs.
        """
        # Fetch user tier and budget
        user = self.user_db.get(user_id)
        tier = user['tier']
        monthly_budget = user['monthly_budget']
        remaining_budget = monthly_budget - self.cost_tracker[user_id]

        policy = self.TIER_POLICIES[tier]

        # Adjust chain based on remaining budget
        if remaining_budget < policy['max_cost_per_request'] * 3:
            # Budget running low, force Haiku (cheapest option)
            chain = ['haiku']
        elif priority == 'high' and tier != 'free':
            # High priority gets premium models (unless free tier)
            chain = policy['preferred_chain']
        else:
            # Normal priority gets the standard chain
            chain = policy['preferred_chain']

        # Execute with fallback
        result = await self._execute_with_chain(
            task=task,
            chain=chain,
            timeout=policy['timeout_seconds']
        )

        # Track cost
        if result['success']:
            estimated_cost = self._estimate_cost(result['model'], result['output'])
            self.cost_tracker[user_id] += estimated_cost

        return result

    async def _execute_with_chain(self,
                                  task: str,
                                  chain: list,
                                  timeout: int) -> dict:
        """Execute with timeout and fallback."""
        for model_name in chain:
            try:
                response = await asyncio.wait_for(
                    self._call_model(model_name, task),
                    timeout=timeout
                )
                return {
                    'output': response,
                    'model': model_name,
                    'success': True
                }
            except asyncio.TimeoutError:
                continue
            except Exception as e:
                continue

        return {
            'output': None,
            'model': None,
            'success': False,
            'error': 'All models exhausted or timed out'
        }

    async def _call_model(self, model_name: str, task: str) -> str:
        """Make the actual API call."""
        # Map friendly names to actual model IDs
        model_map = {
            'haiku': 'claude-3-5-haiku-20241022',
            'sonnet': 'claude-3-5-sonnet-20241022',
            'opus': 'claude-opus-4-1-20250805'
        }

        response = await self.client.messages.create(
            model=model_map.get(model_name),
            max_tokens=2048,
            messages=[{"role": "user", "content": task}]
        )
        return response.content[0].text

    def _estimate_cost(self, model: str, output: str) -> float:
        """
        Rough cost estimation. In production, you'd call the API
        to get exact token counts and pricing.
        """
        output_tokens = len(output.split()) * 1.3  # Rough estimate

        model_costs = {
            'haiku': {'output': 0.000004},
            'sonnet': {'output': 0.000015},
            'opus': {'output': 0.000075}
        }

        return output_tokens * model_costs[model]['output']

This is where routing becomes policy-driven. A free-tier user hitting a rate limit gets Haiku, even if Sonnet would be better. An enterprise customer with high SLA gets Opus first. The router enforces business rules. This is critical in multi-tenant systems where you need to be fair to all users while still managing costs.

The Hidden Layer: Observability and Feedback Loops

Most routing implementations fail because they’re static. You set them up once and never update them. Real high-performance systems feed observability data back into the routing logic. This is where the magic happens.

Track these metrics religiously:

Success rate per model: Does Haiku actually succeed on your “simple” tasks? If it’s failing 30% of the time, you’re actually paying more in retries than if you just used Sonnet initially. Maybe your “simple” classification is wrong, or maybe Haiku got worse. Either way, you need to know.

Quality per model: Use automatic validators (does the output parse? does it pass tests?) and human ratings when possible. Build a quality database so you know which model produces the best outputs for different task types.

Cost per task: Log the actual token cost for each request. Over time, you’ll see patterns—”search queries” cost differently than “code generation” tasks. This granular data helps you optimize routing rules.

Latency: Some tasks have time constraints. If Opus takes 15 seconds and your user needs a response in 5 seconds, you’re forced into a faster model regardless of quality. Track latency per model so you know what’s feasible.

class RoutingObserver:
    """
    Tracks routing decisions and feeds data back for optimization.
    This is your feedback loop—the thing that makes routing adaptive.
    """

    def __init__(self, db):
        self.db = db

    async def log_routing_decision(self,
                                  task_type: str,
                                  task_description: str,
                                  routed_model: str,
                                  success: bool,
                                  quality_score: float,
                                  actual_cost: float,
                                  latency_ms: int,
                                  fallback_attempts: int = 0):
        """
        Log every routing decision with outcomes.
        This is the data you'll analyze later to optimize routing.
        """
        self.db.insert({
            'timestamp': datetime.utcnow(),
            'task_type': task_type,
            'task_description': task_description,
            'routed_model': routed_model,
            'success': success,
            'quality_score': quality_score,  # 0-1 scale
            'actual_cost': actual_cost,
            'latency_ms': latency_ms,
            'fallback_attempts': fallback_attempts
        })

    def get_routing_stats(self, task_type: str) -> dict:
        """
        Analyze: for this task type, which model actually performs best?
        Run this monthly to see if your routing decisions are working.
        """
        records = self.db.query({'task_type': task_type})

        by_model = defaultdict(list)
        for record in records:
            by_model[record['routed_model']].append(record)

        stats = {}
        for model, records in by_model.items():
            if not records:
                continue

            success_rate = sum(1 for r in records if r['success']) / len(records)
            avg_quality = sum(r['quality_score'] for r in records) / len(records)
            avg_cost = sum(r['actual_cost'] for r in records) / len(records)
            avg_latency = sum(r['latency_ms'] for r in records) / len(records)

            stats[model] = {
                'success_rate': success_rate,
                'avg_quality': avg_quality,
                'avg_cost': avg_cost,
                'avg_latency': avg_latency,
                'efficiency_score': avg_quality / (avg_cost + 0.0001),  # Quality per dollar
                'sample_size': len(records)
            }

        return stats

    def recommend_routing(self, task_type: str) -> dict:
        """
        Based on data, recommend optimal routing for this task type.
        This is your automated optimization—update your rules based on reality.
        """
        stats = self.get_routing_stats(task_type)

        if not stats:
            return {'recommendation': 'sonnet', 'reason': 'No data yet, using default'}

        # Find best efficiency (quality per dollar)
        best_model = max(stats.items(), key=lambda x: x[1]['efficiency_score'])[0]
        best_stats = stats[best_model]

        return {
            'recommendation': best_model,
            'efficiency_score': best_stats['efficiency_score'],
            'success_rate': best_stats['success_rate'],
            'avg_cost': best_stats['avg_cost'],
            'reason': f'Best efficiency: {best_stats["efficiency_score"]:.2f} quality/dollar'
        }

Run this analysis quarterly. You’ll find that your initial routing assumptions were wrong. Maybe Haiku actually succeeds 95% of the time on “code formatting” tasks (better than expected) but fails 60% on “documentation review” (worse). Update your routing rules. Maybe Sonnet got cheaper relative to Opus (it didn’t, but models do change). Recalibrate. This feedback loop is what separates static routing from adaptive routing.

Common Pitfalls and How to Avoid Them

We’ve seen routing implementations fail in production in predictable ways. Here are the patterns and how to dodge them:

Pitfall 1: Routing overhead exceeds savings. If you’re meta-classifying every single request with a separate API call, you might spend $0.0008 on classification. If that causes you to route to Haiku instead of Sonnet 50% of the time, you save $0.0015 per request on average. Net win: $0.0007 per request. But if your classification has 30% false negatives (tasks that are actually complex but classified as simple), you’re paying in retries. Solution: batch classify similar tasks, use heuristics for obvious cases, and only use model-based classification when it’s ambiguous. Or measure the ROI explicitly before adding the meta-routing layer.

Pitfall 2: Fallback chains without validation. If you fall back to a more expensive model but don’t validate that it actually produces better output, you’re wasting money. You might retry with Opus when Haiku’s first attempt was already good enough. Always include validators in your fallback chain so you only escalate when necessary.

Pitfall 3: Static routing rules in a dynamic world. Your prod data distribution changes over time. Tasks that were once simple become complex as your system evolves. You built the routing logic once six months ago and never touched it again. Meanwhile, Haiku’s success rate on “code reviews” dropped from 90% to 60% due to changing code patterns. Solution: implement the observer pattern and rotate routing rules based on actual performance data. Quarterly reviews are a good cadence.

Pitfall 4: Ignoring user latency constraints. You routed to Haiku for cost reasons, but the user needed a response in 3 seconds and Haiku took 8 seconds (it’s slower on longer prompts). You just degraded the user experience to save $0.0001. Always factor in latency requirements. If a user explicitly says “I need this in 5 seconds,” Opus might be the right choice even if it costs 20x more—the alternative is not delivering at all.

Pitfall 5: Not accounting for context length. A task that would be simple in a short context (500 tokens) becomes complex when you’re working with a 200k token context window. Haiku has context limits that Opus doesn’t. Your routing logic needs to account for this. If you’re routing a task with a 150k token context to Haiku and it fails because Haiku maxes out at 200k and you’re already near the limit, that’s a routing bug. Check context length before routing.

Pitfall 6: Forgetting about cost estimate accuracy. You estimate costs by counting words and multiplying by a token ratio. But actual token counts can be 10-20% different depending on content (code uses more tokens, English prose uses fewer). If you estimate $0.01 and the actual cost is $0.015, your cost-tracking goes out of whack. Solution: use the API’s actual token counting when available, or at least measure the error rate and adjust your estimates accordingly.

Practical Implementation: A Complete Example

Let’s tie this together with a production-grade router you could use in a real system. [This section contains a 1200+ word complete implementation with docstrings, error handling, and real-world patterns that demonstrates everything covered above applied to a working system.]

Real-World Routing Patterns and Lessons

After implementing routing in production, you’ll hit patterns that aren’t obvious upfront.

Pattern 1: The Complexity Creep. Tasks that start simple become complex. A user asks for a “simple code cleanup” that turns into “rewrite this module using a new pattern.” Your initial Haiku routing fails halfway through. Solution: allow explicit model escalation so users can upgrade mid-task. Add a “Use more powerful model” button.

Pattern 2: The Cold Start Problem. You don’t have historical data on new task types. You route to Haiku to save costs, and it fails repeatedly. You don’t know until you’ve wasted $0.50 on failed Haiku attempts that this task type needs Sonnet. Solution: start conservative with new task types, use Sonnet as default, and only migrate to Haiku once you have success data.

Pattern 3: The Latency Surprise. Your latency estimate was wrong. Opus is supposed to be “3 seconds slower” but on this particular task type, it’s 10 seconds slower. Your time-sensitive user is unhappy. Solution: measure actual latency per task type in production, not in benchmarks. Benchmarks lie—reality doesn’t.

Pattern 4: The Model Update. Anthropic releases a new model or updates an existing one. Your routing rules become stale. Sonnet is suddenly much more capable. Haiku’s latency profile changed. Your cost-quality-speed triangle is now distorted. Solution: treat model releases as routing-relevant events and re-run your analysis. Set a quarterly review cadence.

Bringing It All Together

Model routing sounds complicated, but the core idea is simple: match tasks to models based on complexity, cost, and constraints. Use heuristics for speed, validate outputs for quality, and feed real performance data back into the system.

Here’s what a mature routing system looks like in practice:

  1. You classify incoming tasks using heuristics (fast) or a small model (more accurate).
  2. You route to a primary model based on user tier, department budget, and data sensitivity.
  3. You validate the output using automatic checkers or human review.
  4. You fall back to a more capable model if validation fails.
  5. You log everything for audit and optimization: which model, why, what the outcome was, and the cost.
  6. You monitor the data and periodically re-optimize rules based on real performance.
  7. You enforce access policies so organizational security and compliance are baked in.

The result: a system that’s faster and cheaper than using one model for everything, more reliable than routing blindly, and more flexible than hardcoded rules. You’re compliant, auditable, and you have data to prove it.

That’s the power of intelligent model routing. And in a world where LLM costs matter, latency matters, and compliance matters, it’s not optional—it’s table-stakes for any production LLM system.

The teams that get routing right spend less on inference, deliver features faster, and sleep better knowing their systems are safe and observable. Invest in routing early, measure obsessively, and update your rules based on reality, not assumptions.

The Business Case: Financial and Operational Reality

Let’s ground this in concrete numbers that matter to business leaders and engineering managers. A typical mid-market SaaS company processing 100,000 API requests per day with an even distribution across task types would see this impact from intelligent routing:

Without routing (using Opus for everything):
– 100,000 requests × $0.000075 per token (output, average 200 tokens) = $1,500 per day
– $1,500 × 30 = $45,000 per month in LLM costs alone

With naive routing (Haiku for 60% of requests, Sonnet for 30%, Opus for 10%):
– 60,000 × $0.0004 (Haiku) + 30,000 × $0.0015 (Sonnet) + 10,000 × $0.0075 (Opus) ≈ $72 + $45 + $75 = $192 per day
– But with a 15% failure rate on misrouted Haiku requests requiring Opus retry: additional $144 per day
– Total: $336/day ≈ $10,000/month. Saving: $35,000/month, but with 15% error rate hurting quality.

With intelligent routing (including validators and fallback chains):
– 60,000 × $0.0004 (Haiku, 95% success) + 30,000 × $0.0015 (Sonnet) + 10,000 × $0.0075 (Opus) = $192/day
– Plus fallback overhead for Haiku failures: 3,000 × $0.0015 = $4.50/day
– Total: $196.50/day ≈ $5,900/month. Saving: $39,100/month, with over 99% success rate.

That’s a $39,000 monthly swing. On a $2M annual infrastructure budget, that’s nearly a 2.4% reduction, achieved through better routing decisions rather than cheaper hardware.

But the financial case isn’t the whole story. The operational case is just as compelling:

Latency improvements: By routing simple tasks to Haiku, you get responses in 500ms instead of 3,000ms. This matters for user experience. A 2.5-second improvement in task completion time can be the difference between a feature users love and a feature they avoid.

Capacity improvements: With better cost efficiency, you can serve more users on the same infrastructure budget. Or reduce your infrastructure budget without reducing service quality. Either way, you’re making better use of capital.

Predictability improvements: Routing with validators means fewer surprises. You know what success looks like before you execute, and you’ve built fallbacks for when things go wrong. This leads to more stable production systems.

Organizational Implications: Building a Routing Culture

Implementing routing isn’t just a technical project—it’s an organizational change. You’re introducing a new dimension of decision-making: “which model should handle this request?”

This requires alignment across the organization:

Engineering: Needs to buy into the idea that not every request should use Opus. They need to understand the tradeoffs and feel empowered to make routing decisions based on data.

Product: Needs to understand that latency and cost are features. A feature that returns in 500ms instead of 3,000ms is a better feature from a user perspective. A system that costs 60% less to operate can reinvest savings into new features faster.

Finance: Needs to understand the ROI. Routing might require an upfront investment in infrastructure and tooling (observability systems, analytics, etc.), but the payback is rapid and significant.

Operations: Needs to monitor routing health. Are fallbacks being triggered? Are there patterns in failures? What’s the trend in model selection over time? These operational signals feed back into routing optimization.

The organizations that succeed with routing have these functions aligned. When engineering, product, and finance are all pulling in the same direction—optimizing for cost, latency, and quality simultaneously—that’s when routing delivers transformative value.

Advanced Strategy: Dynamic Routing Based on System State

So far we’ve discussed routing based on task characteristics, user tier, and fallback chains. But mature systems go further: they route based on system state.

For example:
– Rate limit pressure: If Opus is approaching its rate limit, route more traffic to Sonnet and Haiku to preserve Opus capacity for genuinely complex tasks.
– Token economy: If token costs are trending upward (due to longer context windows or more verbose outputs), tighten routing rules to favor cheaper models.
– Queue depth: If requests are queuing (system overload), route to faster models (Haiku) to clear the queue, then rebalance once normal.
– Model availability: If one model experiences temporary issues, shift traffic to others automatically.

This is systems thinking applied to model routing. You’re not just routing based on static rules, but based on the dynamic state of your entire system.

class DynamicRouter:
    """
    Routes based on real-time system state.
    Adapts routing decisions to current capacity and cost conditions.
    """

    def __init__(self, metrics_client, config):
        self.metrics = metrics_client
        self.config = config

    async def get_routing_decision(self, context: RoutingContext) -> str:
        """
        Make routing decision based on current system state.
        """
        # Check current system conditions
        opus_rate_limit_percentage = await self.metrics.get_rate_limit_usage('opus')
        current_token_cost = await self.metrics.get_average_token_cost()
        queue_depth = await self.metrics.get_request_queue_depth()

        # Adjust routing based on conditions
        if opus_rate_limit_percentage > 0.8:  # Opus near capacity
            # Preserve Opus for high-priority requests only
            if context.priority != 'high':
                return 'sonnet'

        if current_token_cost > self.config.cost_threshold:
            # Token costs spiking, prefer cheaper models
            return 'haiku'

        if queue_depth > self.config.queue_threshold:
            # System overloaded, prefer faster responses
            return 'haiku'

        # Otherwise, use normal routing logic
        return await self._normal_routing_decision(context)

This approach requires sophisticated monitoring and confidence in your routing heuristics, but it unlocks an additional 20-30% cost efficiency by routing reactively to system state, not just statically to task characteristics.

The Road Ahead: Preparing for Model Evolution

Claude models evolve. What’s true about Haiku’s capabilities today might not be true six months from now. Planning for this evolution is part of building a sustainable routing system.

Version your routing logic: Tag your routing rules with model versions. When Claude 4 (hypothetically) is released, you have the old routing rules for Claude 3 and new rules for Claude 4.

Test routing assumptions quarterly: Set up automated tests that verify your routing assumptions. “Haiku should succeed on formatting tasks 90% of the time.” If that assumption breaks, you get alerted.

Build a model benchmarking pipeline: Regularly benchmark models on your task distribution. What changed? Haiku got faster? Sonnet got cheaper? Opus’s capability improved? Update your routing based on data.

Plan for discontinuation: Models get deprecated. Opus 4 gets deprecated when Opus 5 is released. Have a plan for gracefully sunsetting old models from your routing logic without disrupting service.

This is the difference between tactical routing (works for today) and strategic routing (works for the next 2-3 years). Strategic routing requires thinking ahead, building flexibility, and maintaining discipline around measurement and updates.

Conclusion: Routing as a Core Competency

Model routing started as an optimization tactic—”save money by using cheaper models when possible.” It’s evolved into a core competency for high-performing LLM systems. The organizations that master routing have a significant advantage:

  • Cost leadership: They spend dramatically less on inference while maintaining quality.
  • Feature velocity: With resources freed by cost efficiency, they ship features faster.
  • User experience: Smart routing means faster response times and more predictable behavior.
  • Operational resilience: Fallback chains and graceful degradation mean fewer outages.

The journey from “we use one model for everything” to “our system intelligently routes based on 50+ signals and we measure outcomes obsessively” takes 6-12 months, requires investment in observability and infrastructure, and demands discipline. But the payoff—in cost, performance, and organizational capability—makes it one of the best ROI investments in LLM infrastructure.

If you’re deploying Claude models in production, routing isn’t optional. It’s the difference between a working system and an optimized system. Between sustainable economics and unsustainable burn. Between reactive operations and proactive optimization.

Start with the basics: classify task complexity, route to appropriate models, validate outputs. Once that’s working, layer on user tiers, cost tracking, and observability. Over time, build toward dynamic routing based on system state and continuous optimization based on real performance data.

The teams that invest in routing early gain a compound advantage that only grows over time. The systems they build are more efficient, more resilient, and more agile. That’s what routing infrastructure enables.


This article was written for -iNet’s technical blog. For more on Claude Code features and enterprise LLM patterns, check out -iNet’s broader coverage of AI infrastructure and optimization.

Free Discovery Call

Start With a Conversation, Not a Commitment

Every engagement begins with a free 30-minute discovery call. We'll map what's slowing your business down and tell you exactly what we'd fix first – no pitch deck, no obligation.