The fundamental architectural decision

Every AI system in production starts with a question that seems simple but has profound implications: where do you run inference. This decision conditions costs, latency, privacy, scalability and operational complexity throughout the system's lifetime.

Most teams start with an API (OpenAI, Anthropic, Google) because it's the fastest. And that's fine for prototypes. The problem comes when the prototype grows, costs skyrocket, privacy requirements change, or latency becomes critical. Then it's time to rethink the architecture, and doing it late is expensive.

This module teaches you to make that decision with data, not intuition. We'll build a decision framework you can apply to any AI project.

The three real options

In 2026, deployment options for LLMs boil down to three categories:

• Commercial API: OpenAI, Anthropic, Google, DeepSeek. You pay per token. Zero own infrastructure.
• Self-hosted on own GPU: vLLM, TGI, Ollama on your hardware (Hetzner, AWS, on-premise). Fixed monthly cost.
• Hybrid: self-hosted for predictable loads + API for spikes and models you can't host.

The hybrid option is what most serious teams end up adopting. But to get there, you need to understand the first two in depth.

API vs self-hosted: comparative analysis

Let's compare both options across the 6 dimensions that matter:

1. Cost

API: variable cost per token. Predictable if your volume is stable, dangerous if it grows. A model like Claude Sonnet costs ~3 USD per million input tokens and ~15 USD per million output tokens. If you process 100M tokens per month, we're talking about 1,500+ USD in inference alone.

Self-hosted: fixed cost. A Hetzner GEX44 server with an A100 GPU (48GB) costs ~130 EUR/month. You process unlimited tokens. But you need an engineer to maintain it.

2. Latency

API: typical latency 200-800ms for the first token (TTFT), depending on the model and load. You don't control spikes. You don't control the queue. You're at the provider's mercy.

Self-hosted: predictable latency. With well-configured vLLM, TTFT of 50-150ms for 7-27B models. You control the queue, you control the batching, you control everything.

3. Privacy and compliance

API: your data leaves your infrastructure. For regulated sectors (banking, healthcare, defense, ENS Alto), this can be an absolute no-go. Some providers offer data residency in the EU, but reading the fine print is mandatory.

Self-hosted: data never leaves. GDPR, ENS, NIS2, DORA compliance is trivial from the LLM inference perspective.

4. Model quality

API: access to the best models on the market (GPT-4o, Claude Opus, Gemini Ultra). Open-source models don't match them yet in complex reasoning tasks.

Self-hosted: Qwen3.5-27B, Llama 3.3 70B and Phi-4 are excellent for many tasks, but they don't compete with frontier models in complex multi-step reasoning.

5. Scalability

API: you scale by calling more. No practical limits (except rate limits you can negotiate). You don't manage GPUs, you don't manage queues.

Self-hosted: you scale by buying or renting more GPUs. Provisioning time: hours or days. Requires capacity planning.

6. Operational complexity

API: an API key and an HTTP client. Operational complexity close to zero. If the API goes down, it's not your problem (but you can't do anything either).

Self-hosted: Docker, vLLM, GPU drivers, monitoring, backups, model updates, KV cache management. You need a platform team.

Breakeven calculator: when self-hosting pays off

The breakeven is the point where the fixed cost of self-hosting equals the variable cost of the API. Below it, the API is cheaper. Above it, self-hosting wins.

Simplified formula

# Breakeven calculator
# Variables
api_cost_per_1m_tokens = 3.0    # USD (input, Sonnet-type model)
api_output_per_1m = 15.0        # USD (output)
ratio_input_output = 0.7        # 70% input, 30% output typical

monthly_server_cost = 130       # EUR (Hetzner GEX44 with A100)
engineer_cost_monthly = 500     # EUR (part-time maintenance)
total_selfhost_monthly = monthly_server_cost + engineer_cost_monthly

# Weighted average cost per million tokens (API)
avg_cost_per_1m = (api_cost_per_1m_tokens * ratio_input_output +
                   api_output_per_1m * (1 - ratio_input_output))
# avg_cost_per_1m = 2.1 + 4.5 = 6.6 USD/M tokens

# Breakeven in tokens
breakeven_tokens = (total_selfhost_monthly / avg_cost_per_1m) * 1_000_000
# breakeven_tokens = (630 / 6.6) * 1M ≈ 95.4M tokens/month

print(f"Breakeven: {breakeven_tokens/1e6:.1f}M tokens/month")
print(f"If you process more than {breakeven_tokens/1e6:.1f}M tokens/month, self-hosting is cheaper")

In this example, with a server at 130 EUR/month and 500 EUR of engineer time, the breakeven is at ~95M monthly tokens. If your application processes more than that, self-hosting saves money. If it processes less, the API is more cost-effective.

Hidden variables in the calculation

The real calculation is more complex. These variables are often forgotten:

• Prompt caching: commercial APIs offer prompt caching that reduces costs by 50-90% on repetitive prompts. This moves the breakeven up.
• GPU utilization: if your server is idle 60% of the time, you're paying for capacity you don't use. The real breakeven goes up.
• Engineer opportunity cost: 500 EUR/month is optimistic. If your only ML engineer dedicates 30% of their time to maintaining infra instead of building features, the real cost is much higher.
• Redundancy: in production you need at least 2 servers (one for fallback). The fixed cost doubles.
• Batch pricing: Anthropic and OpenAI offer batch pricing at 50% off. If your workloads aren't real-time, this changes the calculation.

Latency budgets: latency planning

A latency budget is the distribution of maximum allowed time for an end-to-end request across all system components. If your SLA is "response in 3 seconds", you need to know exactly how much each part consumes.

Budget example for a RAG chatbot

# Latency budget: RAG chatbot with 3s SLA
# ----------------------------------------
# Component              Budget    P50    P99
# ----------------------------------------
# Network (client-server)  100ms    30ms   80ms
# API gateway + auth        50ms    10ms   40ms
# Embedding query           80ms    40ms   70ms
# Vector search (Qdrant)   100ms    20ms   80ms
# Reranking                150ms    50ms  120ms
# LLM TTFT                 500ms   200ms  450ms
# LLM generation          1800ms  1000ms 1600ms
# Post-processing           50ms    20ms   40ms
# ----------------------------------------
# TOTAL                   2830ms  1370ms 2480ms
# Buffer                   170ms
# ----------------------------------------
# SLA                     3000ms

The LLM consumes 77% of the budget. This is typical. The implication: optimizing the LLM (faster model, fewer output tokens, streaming) has more impact than optimizing any other component.

Techniques for meeting latency budgets

• Streaming: send tokens to the client as they are generated. The TTFT (time to first token) becomes the metric that matters, not total time.
• Smaller models: Phi-4 (14B) generates tokens 3-4x faster than a 70B. If quality is sufficient, the smaller model wins.
• Prompt compression: reducing context reduces prefill latency. Every 1000 fewer tokens in the prompt is 10-30ms less TTFT.
• Parallelization: embedding + vector search can execute in parallel if the pipeline allows it.
• Response cache: for repetitive queries (FAQ), semantic cache eliminates LLM inference entirely.

Cost modeling for AI systems

A cost model for AI has three layers: inference, support infrastructure, and human costs.

Layer 1: Inference

# Cost model: inference
# API
tokens_per_request_avg = 2500      # input + output
requests_per_day = 10000
monthly_tokens = tokens_per_request_avg * requests_per_day * 30
# = 750M tokens/month

api_cost = (monthly_tokens / 1e6) * 6.6  # weighted average
# = 4,950 USD/month

# Self-hosted (Hetzner GEX44)
server_cost = 130  # EUR/month, unlimited tokens
# Savings: 4,950 - 130 = 4,820 USD/month
# (not counting operational costs)

Layer 2: Support infrastructure

• Vector database: Qdrant self-hosted ~0 EUR (on same server) or Pinecone ~70-700 USD/month.
• Relational database: Supabase Free-Pro, 0-25 USD/month. Self-hosted PostgreSQL ~0.
• CDN and networking: Cloudflare Free covers most cases. 20 USD/month for Workers.
• Monitoring: Self-hosted Grafana ~0. Datadog 23 USD/host/month.
• Logs: Self-hosted Loki ~0. Managed Elasticsearch 100+ USD/month.

Layer 3: Human costs

The most ignored and largest cost. A senior ML engineer in Europe costs 60-100K EUR/year. If they dedicate 30% of their time to LLM infrastructure, that is 18-30K EUR/year. That is 1,500-2,500 EUR/month, more than any server.

The API vs self-hosted decision is not just technical. It is a talent allocation decision.

Infrastructure planning: from prototype to production

The transition from prototype to production is where most AI projects die. Not because of the model, but because of the infrastructure. Here is a 4-phase framework:

Phase 1: Prototype (weeks 1-4)

• Commercial API (OpenAI or Anthropic)
• SQLite or Supabase Free database
• No monitoring (logs to stdout)
• Cost: ~50 USD/month
• Goal: validate that the use case works

Phase 2: MVP (weeks 5-12)

• Commercial API with prompt caching
• Supabase Pro + local Qdrant
• Basic monitoring (latency, errors, costs)
• Cost: 200-500 USD/month
• Goal: first real users, usage metrics

Phase 3: Production (months 3-6)

• Hybrid: self-hosted for predictable loads + API for spikes
• PostgreSQL with RLS + Qdrant cluster
• OpenTelemetry + Grafana + alerts
• Cost: 500-2000 USD/month
• Goal: defined SLA, proven scalability

Phase 4: Scale (month 6+)

• Self-hosted as primary + API as fallback
• Multi-GPU, tensor parallelism
• CI/CD for models + A/B testing
• Cost: 2000-10000 USD/month
• Goal: cost per query optimization

Architecture Decision Records for AI

An ADR (Architecture Decision Record) documents a technical decision: why it was made, what alternatives were considered, and what the consequences are. For AI systems, ADRs are especially important because decisions change fast (new models every month) and consequences are expensive (migrating providers, changing models, retraining).

ADR template for AI decisions

# ADR-XXX: [Decision title]

## Status
Proposed | Accepted | Deprecated | Superseded by ADR-YYY

## Context
- What problem we are solving
- Constraints (regulatory, technical, budgetary)
- Current metrics (latency, cost, quality)

## Decision
- What we have decided
- What model/provider/architecture

## Alternatives considered
1. Option A: [description] — Discarded because [reason]
2. Option B: [description] — Discarded because [reason]

## Consequences
### Positive
- [benefit 1]
- [benefit 2]

### Negative
- [trade-off 1]
- [trade-off 2]

### Success metrics
- [KPI 1]: target value
- [KPI 2]: target value

## Review
- Review date: [when to re-evaluate this decision]
- Review trigger: [event that would force re-evaluation]

Common architecture patterns

Pattern 1: API Gateway with intelligent routing

A gateway that routes requests to different backends (commercial API or self-hosted) based on task type, priority, or available budget.

# Inference router
class InferenceRouter:
    def route(self, request: InferenceRequest) -> str:
        # Tasks with sensitive data -> self-hosted
        if request.contains_pii:
            return "vllm_local"

        # Complex reasoning tasks -> frontier API
        if request.complexity == "high":
            return "anthropic_api"

        # Simple classification tasks -> small local model
        if request.task_type == "classification":
            return "phi4_local"

        # Default -> self-hosted
        return "vllm_local"

Pattern 2: Fallback chain

Try the primary backend. If it fails (timeout, rate limit, error), fall to the next. Each level has a different timeout SLA.

# Fallback chain
backends = [
    {"name": "vllm_local", "timeout": 10, "priority": 1},
    {"name": "anthropic_api", "timeout": 30, "priority": 2},
    {"name": "openai_api", "timeout": 30, "priority": 3},
]

async def inference_with_fallback(request):
    for backend in sorted(backends, key=lambda b: b["priority"]):
        try:
            return await call_backend(
                backend["name"],
                request,
                timeout=backend["timeout"]
            )
        except (TimeoutError, RateLimitError) as e:
            log.warning(f"{backend['name']} failed: {e}")
            continue
    raise AllBackendsFailedError()

Pattern 3: Tiered model selection

Different models for different complexity levels. The cheap model filters 80% of requests; the expensive model only processes the 20% that truly need it.

# Tiered model selection
async def process_query(query: str):
    # Tier 1: quick classification with small model
    complexity = await classify_complexity(query, model="phi-4")

    if complexity == "simple":
        # Tier 2a: medium model for simple queries
        return await generate(query, model="qwen3.5-7b")
    else:
        # Tier 2b: large model for complex queries
        return await generate(query, model="qwen3.5-27b")

# Result: 80% of queries use the cheap model
# Average cost per query drops by 60%

Ejercicio practico

Puntos clave