The capstone project

This module is different from the previous ones. There is no new theory. Everything you need you have already learned in TE01-TE09. Here you integrate everything into a real project: a complete AI system deployed on your own infrastructure with end-to-end observability.

The goal is that by the end of this module you have a system you can showcase in an interview, use as a foundation for your own product, or deploy for a client. Not a prototype. A real production system.

What you will build

An LLM inference service with RAG, deployed on a GPU server, with an API gateway, monitoring, alerts, cost tracking, and automated backup. The components:

• LLM: Qwen3.5-27B AWQ served with vLLM (or Phi-4 if you don't have a 24GB+ GPU)
• RAG: vector search with Qdrant + embedding model
• API: FastAPI with authentication, rate limiting, and logging
• Observability: Prometheus + Grafana + Langfuse + alerts
• Infrastructure: Docker Compose, Caddy TLS, automated backup

System architecture

# Capstone system architecture
#
# ┌─────────────┐     ┌───────────┐     ┌──────────────┐
# │   Client    │────>│   Caddy   │────>│   FastAPI     │
# │  (HTTPS)     │<────│  (TLS)    │<────│   (API GW)    │
# └─────────────┘     └───────────┘     └──────┬───────┘
#                                              │
#                          ┌───────────────────┼───────────────────┐
#                          │                   │                   │
#                    ┌─────▼─────┐     ┌──────▼──────┐    ┌──────▼──────┐
#                    │  Qdrant   │     │    vLLM     │    │  Langfuse   │
#                    │ (vectors) │     │   (LLM)     │    │  (traces)   │
#                    └───────────┘     └─────────────┘    └─────────────┘
#                                              │
#                                     ┌────────▼────────┐
#                                     │   Prometheus    │
#                                     │   + Grafana     │
#                                     └─────────────────┘
#
# Request flow:
# 1. Client sends query via HTTPS
# 2. Caddy terminates TLS, forwards to FastAPI
# 3. FastAPI: authentication, rate limit, log
# 4. Query embedding -> Qdrant search -> relevant documents
# 5. Prompt construction (system + context + query) -> vLLM
# 6. vLLM generates response (streaming)
# 7. FastAPI records trace in Langfuse + metrics in Prometheus
# 8. Response to client

Phase 1: Base infrastructure

Prerequisite: a server with an NVIDIA GPU (Hetzner GEX44, RunPod, or your own machine with RTX 4090+). If you don't have a GPU, you can use Phi-4-mini on CPU for the mechanics, although performance will be limited.

# Project structure
capstone/
├── docker-compose.yml          # The entire stack
├── docker-compose.monitoring.yml  # Monitoring stack
├── Caddyfile                   # TLS reverse proxy
├── .env                        # Environment variables (NOT in git)
├── .env.example                # Variables template
├── app/
│   ├── main.py                 # FastAPI application
│   ├── config.py               # Configuration
│   ├── routers/
│   │   ├── chat.py             # /v1/chat endpoint
│   │   └── health.py           # Healthcheck
│   ├── services/
│   │   ├── llm.py              # vLLM client
│   │   ├── rag.py              # RAG pipeline
│   │   └── cache.py            # Semantic cache
│   ├── middleware/
│   │   ├── auth.py             # API key auth
│   │   ├── rate_limit.py       # Rate limiting
│   │   └── telemetry.py        # OpenTelemetry middleware
│   └── Dockerfile
├── prompts/
│   └── system_v1.0.txt         # Versioned system prompt
├── data/
│   └── knowledge_base/         # Documents for RAG
├── scripts/
│   ├── ingest.py               # Ingest documents into Qdrant
│   ├── benchmark.py            # Performance benchmark
│   └── backup.sh               # Automated backup
├── monitoring/
│   ├── prometheus.yml
│   ├── alerting_rules.yml
│   └── grafana/
│       └── dashboards/         # Dashboard JSONs
└── docs/
    └── ADR-001-architecture.md # Project ADR

docker-compose.yml

# docker-compose.yml
version: "3.8"

services:
  # LLM Inference
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    volumes:
      - /data/models:/models
    command: >
      --model /models/qwen3.5-27b-awq
      --port 8000
      --host 0.0.0.0
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --max-num-seqs 16
      --enable-prefix-caching
      --api-key ${VLLM_API_KEY}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 120s
    restart: unless-stopped

  # Vector Database
  qdrant:
    image: qdrant/qdrant:latest
    volumes:
      - qdrant_data:/qdrant/storage
    ports:
      - "6333:6333"
    restart: unless-stopped

  # API Gateway
  api:
    build: ./app
    env_file: .env
    depends_on:
      vllm:
        condition: service_healthy
      qdrant:
        condition: service_started
    restart: unless-stopped

  # TLS Reverse Proxy
  caddy:
    image: caddy:2
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
    depends_on:
      - api
    restart: unless-stopped

  # Semantic Cache
  redis:
    image: redis:7-alpine
    volumes:
      - redis_data:/data
    restart: unless-stopped

  # Observability
  langfuse:
    image: langfuse/langfuse:latest
    environment:
      - DATABASE_URL=postgresql://${PG_USER}:${PG_PASS}@postgres:5432/langfuse
      - NEXTAUTH_URL=${LANGFUSE_URL}
      - NEXTAUTH_SECRET=${LANGFUSE_SECRET}
    depends_on:
      - postgres
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    environment:
      - POSTGRES_USER=${PG_USER}
      - POSTGRES_PASSWORD=${PG_PASS}
      - POSTGRES_DB=langfuse
    volumes:
      - postgres_data:/var/lib/postgresql/data
    restart: unless-stopped

volumes:
  qdrant_data:
  caddy_data:
  redis_data:
  postgres_data:

Phase 2: Model deploy

# 1. Download model (do this once, outside Docker)
pip install huggingface_hub
huggingface-cli download Qwen/Qwen3.5-27B-AWQ --local-dir /data/models/qwen3.5-27b-awq

# 2. Verify checksum
sha256sum /data/models/qwen3.5-27b-awq/model-*.safetensors
# Compare with checksums on the HuggingFace page

# 3. Start vLLM
docker compose up -d vllm

# 4. Wait for it to load (30-60 seconds)
docker compose logs -f vllm
# Look for: "INFO: Application startup complete"

# 5. Verify
curl http://localhost:8000/v1/models
# {"object":"list","data":[{"id":"qwen3.5-27b-awq","object":"model",...}]}

Phase 3: API gateway and service

# app/main.py
from fastapi import FastAPI, Depends, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from contextlib import asynccontextmanager
from app.middleware.auth import verify_api_key
from app.middleware.telemetry import setup_telemetry
from app.routers import chat, health
from app.services.llm import LLMClient
from app.services.rag import RAGService
from app.services.cache import SemanticCache
from app.config import settings

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Startup
    app.state.llm = LLMClient(settings.VLLM_URL, settings.VLLM_API_KEY)
    app.state.rag = RAGService(settings.QDRANT_URL)
    app.state.cache = SemanticCache(settings.REDIS_URL)
    setup_telemetry(app)
    yield
    # Shutdown
    await app.state.cache.close()

app = FastAPI(
    title="AI Inference API",
    version="1.0.0",
    lifespan=lifespan
)

app.add_middleware(
    CORSMiddleware,
    allow_origins=["https://yourdomain.com"],
    allow_methods=["POST"],
    allow_headers=["Authorization", "Content-Type"],
)

app.include_router(health.router)
app.include_router(chat.router, dependencies=[Depends(verify_api_key)])

# app/routers/chat.py
from fastapi import APIRouter, Request
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from langfuse.decorators import observe, langfuse_context
import time

router = APIRouter()

class ChatRequest(BaseModel):
    message: str
    conversation_id: str | None = None
    stream: bool = True

class ChatResponse(BaseModel):
    response: str
    sources: list[str]
    usage: dict
    latency_ms: float

@router.post("/v1/chat")
@observe()
async def chat(request: Request, body: ChatRequest):
    start = time.perf_counter()

    # 1. Semantic cache
    cached = await request.app.state.cache.get(body.message)
    if cached:
        langfuse_context.update_current_observation(metadata={"cache": "hit"})
        return ChatResponse(
            response=cached["response"],
            sources=cached["sources"],
            usage={"cached": True},
            latency_ms=(time.perf_counter() - start) * 1000
        )

    # 2. RAG: search for relevant documents
    with langfuse_context.observe(name="rag_retrieval"):
        docs = await request.app.state.rag.search(body.message, top_k=5)
        context = "\n\n".join([d.content for d in docs])
        sources = [d.source for d in docs]

    # 3. LLM: generate response
    system_prompt = load_prompt("system_v1.0.txt")
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {body.message}"}
    ]

    with langfuse_context.observe(name="llm_generation", as_type="generation"):
        response = await request.app.state.llm.generate(messages)

    # 4. Cache the response
    await request.app.state.cache.set(
        body.message,
        {"response": response.content, "sources": sources}
    )

    # 5. Automatic score
    langfuse_context.score_current_trace(
        name="has_sources",
        value=1.0 if sources else 0.0
    )

    elapsed_ms = (time.perf_counter() - start) * 1000

    return ChatResponse(
        response=response.content,
        sources=sources,
        usage={
            "prompt_tokens": response.usage.prompt_tokens,
            "completion_tokens": response.usage.completion_tokens,
        },
        latency_ms=elapsed_ms
    )

Phase 4: Full observability

# Observability checklist:
# [x] Metrics: Prometheus scrapes vLLM (/metrics) + FastAPI (custom metrics)
# [x] Traces: Langfuse captures every request with input, output, latency, tokens
# [x] Logs: FastAPI structured logs (JSON) -> stdout -> Docker logs
# [x] Dashboards: Grafana with 3 dashboards (ops, costs, quality)

# app/middleware/telemetry.py
from prometheus_client import Counter, Histogram, make_asgi_app
from opentelemetry import trace

REQUEST_COUNT = Counter(
    "api_requests_total",
    "Total API requests",
    ["method", "endpoint", "status"]
)

REQUEST_LATENCY = Histogram(
    "api_request_duration_seconds",
    "Request latency",
    ["method", "endpoint"],
    buckets=[0.1, 0.25, 0.5, 1, 2.5, 5, 10]
)

TOKEN_COUNT = Counter(
    "api_tokens_total",
    "Total tokens processed",
    ["type"]  # input, output
)

COST_TOTAL = Counter(
    "api_cost_usd_total",
    "Total inference cost in USD"
)

CACHE_HITS = Counter(
    "api_cache_hits_total",
    "Semantic cache hits"
)

def setup_telemetry(app):
    # Mount /metrics endpoint for Prometheus
    metrics_app = make_asgi_app()
    app.mount("/metrics", metrics_app)

Phase 5: Alerts and cost tracking

# monitoring/alerting_rules.yml
groups:
  - name: capstone-alerts
    rules:
      - alert: HighLatency
        expr: histogram_quantile(0.99, rate(api_request_duration_seconds_bucket[5m])) > 5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "API P99 latency > 5s"
          action: "Check vLLM load, KV cache, and concurrency"

      - alert: HighErrorRate
        expr: >
          rate(api_requests_total{status=~"5.."}[5m]) /
          rate(api_requests_total[5m]) > 0.05
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "Error rate > 5%"

      - alert: VLLMDown
        expr: up{job="vllm"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "vLLM is not responding"
          action: "docker compose restart vllm"

      - alert: HighDailyCost
        expr: sum(increase(api_cost_usd_total[24h])) > 50
        labels:
          severity: warning
        annotations:
          summary: "Daily cost > 50 USD"

      - alert: LowCacheHitRate
        expr: >
          rate(api_cache_hits_total[1h]) /
          rate(api_requests_total[1h]) < 0.1
        for: 1h
        labels:
          severity: info
        annotations:
          summary: "Cache hit rate < 10%. Review similarity threshold."

Cost tracking dashboard

# Prometheus queries for cost dashboard in Grafana

# Accumulated cost today
# sum(increase(api_cost_usd_total[24h]))

# Cost per hour (trend)
# rate(api_cost_usd_total[1h]) * 3600

# Total tokens today (input vs output)
# sum(increase(api_tokens_total{type="input"}[24h]))
# sum(increase(api_tokens_total{type="output"}[24h]))

# Average cost per request
# rate(api_cost_usd_total[1h]) / rate(api_requests_total[1h])

# Estimated cache savings
# rate(api_cache_hits_total[24h]) * avg(api_cost_per_request)

# Monthly projection
# sum(rate(api_cost_usd_total[7d])) * 86400 * 30

Phase 6: Testing and validation

# scripts/validate_deployment.py
"""End-to-end deployment validation script."""
import httpx
import asyncio
import sys

BASE_URL = "https://yourdomain.com"
API_KEY = "your-api-key"

checks = []

async def check(name: str, test_fn):
    try:
        result = await test_fn()
        checks.append({"name": name, "status": "PASS", "detail": result})
        print(f"  PASS  {name}")
    except Exception as e:
        checks.append({"name": name, "status": "FAIL", "detail": str(e)})
        print(f"  FAIL  {name}: {e}")

async def main():
    print("Deployment validation\n" + "=" * 40)

    async with httpx.AsyncClient(timeout=30) as client:

        # 1. Health check
        await check("Health endpoint", lambda: client.get(f"{BASE_URL}/health"))

        # 2. TLS
        await check("TLS active", lambda: client.get(BASE_URL))

        # 3. Auth rejects without key
        async def test_auth():
            r = await client.post(f"{BASE_URL}/v1/chat", json={"message": "test"})
            assert r.status_code == 401, f"Expected 401, got {r.status_code}"
            return "401 OK"
        await check("Auth rejects without API key", test_auth)

        # 4. Chat works
        async def test_chat():
            r = await client.post(
                f"{BASE_URL}/v1/chat",
                json={"message": "What is machine learning?", "stream": False},
                headers={"Authorization": f"Bearer {API_KEY}"}
            )
            assert r.status_code == 200
            data = r.json()
            assert len(data["response"]) > 50
            return f"{data['usage']['completion_tokens']} tokens, {data['latency_ms']:.0f}ms"
        await check("Chat endpoint works", test_chat)

        # 5. Prometheus metrics
        await check("Prometheus metrics", lambda: client.get(f"{BASE_URL}/metrics"))

        # 6. Langfuse accessible
        await check("Langfuse UI", lambda: client.get("http://localhost:3001"))

        # 7. Grafana accessible
        await check("Grafana UI", lambda: client.get("http://localhost:3000"))

    # Resumen
    passed = sum(1 for c in checks if c["status"] == "PASS")
    total = len(checks)
    print(f"\nResult: {passed}/{total} checks passed")

    if passed < total:
        sys.exit(1)

asyncio.run(main())

Production checklist

What comes next

You have completed the AI Engineer track at IAcademy. You now have the skills to design, deploy, optimize, and maintain AI systems in production. But this is just the beginning.

Where to keep learning

• Advanced fine-tuning: RLHF, DPO, reward models. Adapting models to continuous human feedback.
• Multi-agent systems: coordinators, specialized workers, LangGraph, inter-agent communication.
• Edge AI: models on mobile devices, browsers (WebLLM), IoT. Serverless inference.
• AI security: prompt injection, jailbreaking, adversarial attacks, model security.
• Regulation: EU AI Act, NIS2, DORA. Compliance for AI systems in production.

Community and resources

• HuggingFace Hub: models, datasets, papers.
• vLLM GitHub: issues, discussions, releases.
• Langfuse Discord: observability for AI.
• MLOps Community: Slack with 20K+ professionals.
• Papers: "Attention Is All You Need", "LoRA", "vLLM: PagedAttention".