En este modulo
DevOps for AI: what changes
DevOps for traditional web services is solved: Docker, Kubernetes, CI/CD, monitoring. Each piece has mature tools and proven patterns. For AI, the same principles apply but with additional constraints: enormous Docker images (15-50GB with CUDA + PyTorch + model), GPUs as a scarce resource, slow cold starts (30-60s to load a 27B model), model storage (14-54GB needs fast NVMe storage), and specialized monitoring (VRAM, GPU utilization, temperature, KV cache).
Docker for ML: images and best practices
Multi-stage builds to reduce size. Separate model from image (mount as volume, don't copy into image). Pin exact versions. Layer caching: heavy dependencies first, code that changes frequently last. Use .dockerignore to exclude models and data. Final runtime image: ~8GB vs ~25GB without multi-stage.
Kubernetes for AI workloads
Kubernetes for AI is viable but not always necessary. For 1-3 GPUs on a single server, Docker Compose is simpler and sufficient. Kubernetes shines with 5+ services, multiple servers with heterogeneous GPUs, and auto-scaling needs. Use NVIDIA device plugin, readiness probes with generous timeouts (120s initialDelay), and PersistentVolumeClaims for model storage.
GPU scheduling
In a cluster with multiple GPUs (possibly different types), the scheduler needs to know which GPU to assign to each workload. NVIDIA device plugin for Kubernetes, node labels for heterogeneous GPUs, time-slicing for sharing GPU between pods (small models), MIG (Multi-Instance GPU) for hardware-level isolation on A100/H100.
Infrastructure as Code for AI
AI infrastructure must be reproducible. If your server crashes, you must be able to recreate everything in minutes, not days. Terraform or Pulumi for provisioning (server with GPU, firewall, volumes), cloud-init for automatic server configuration (Docker, nvidia drivers, nvidia-container-toolkit).
Infrastructure monitoring
Docker Compose monitoring stack: Prometheus + Grafana + node_exporter + nvidia_exporter + cadvisor. Critical GPU metrics: GPU utilization (60-90% is healthy), GPU memory used, GPU temperature (alert at 85C, throttling at 90C, shutdown at 95C), power draw, PCIe bandwidth.
Backup strategies
The most valuable assets aren't base models (they can be re-downloaded). They are: LoRA adapters, training datasets, eval datasets, prompts, and configurations. Automated backup script via cron at 3 AM: tar + gpg encrypt + transfer to external storage. Rotation: 30 daily + 12 monthly.
What you DON'T need to backup
Base models (Qwen3.5-27B, Llama 3.3): re-download from HuggingFace. Docker images: re-build from Dockerfile. What's critical are artifacts YOU generated: LoRA adapters, datasets, prompts, configs and metadata.
AI infrastructure security
- API keys: never in code or .env in production. Docker Swarm secrets or Kubernetes Secrets (encrypted).
- Network: vLLM not exposed directly. Always behind Caddy/Nginx with TLS. Port 8000 only accessible via localhost or internal network.
- Model poisoning: verify checksums of downloaded models. Download only from trusted sources (official HuggingFace).
- Prompt injection: validate inputs before sending to the LLM. Sanitize outputs before executing tools.
- GPU side-channels: in multi-tenant environments, shared GPUs can leak information. MIG isolates at the hardware level.
Ejercicio practico
- Write a multi-stage Dockerfile for your vLLM service. Measure final image size vs single-stage.
- Create a complete docker-compose.yml: vLLM + Caddy (TLS) + Prometheus + Grafana + node_exporter + nvidia_exporter.
- Import a Grafana dashboard with: GPU utilization, VRAM usage, temperature, and vLLM metrics (latency, throughput, KV cache).
- Write a backup script that backs up LoRA adapters, datasets, prompts and configurations. Schedule with cron.
- If using Hetzner or cloud: write the Terraform configuration for your GPU server with firewall, volume and cloud-init.
- Verify security: vLLM not publicly exposed, API key configured, TLS active, models with verified checksums.
Expected result: your LLM system can be completely destroyed and recreated from scratch (IaC + backup) in less than 30 minutes.
Puntos clave
Puntos clave from TE09
- Docker for ML: multi-stage builds, models as volumes (not in the image), exact version pins. The runtime image should be as small as possible.
- Kubernetes for AI: useful with 5+ services and multiple GPUs. For 1-3 services on 1 server, Docker Compose is simpler and sufficient.
- GPU scheduling: NVIDIA device plugin for Kubernetes, node selectors for heterogeneous GPUs, MIG for multi-tenant. GPUs are the scarce resource, not CPU.
- IaC (Terraform + cloud-init): reproduce your infrastructure in minutes. If it's not in code, it doesn't exist.
- Backup what YOU generated: LoRA adapters, datasets, prompts, configs. Base models can be re-downloaded. Encrypt before transferring (GPG + AES256).
Guia de estudio — Conceptos clave de TE09
DevOps para IA: que cambia
- Imagenes Docker enormes:una imagen con CUDA + PyTorch + modelo pesa 15-50GB. Build times de 30+ minutos. Layer caching es critico.
- GPUs como recurso escaso:no puedes escalar GPUs como CPUs. Son caras, tienen disponibilidad limitada, y requieren drivers especificos.
- Cold start lento:cargar un modelo de 27B en VRAM tarda 30-60 segundos. Los readiness probes necesitan timeouts generosos.
- Storage de modelos:modelos de 14-54GB necesitan storage rapido (NVMe) y estrategias de distribucion (pre-pull, model cache compartido).
- Monitoring especializado:ademas de CPU/RAM/disco, necesitas monitorear VRAM, utilizacion de GPU, temperatura de GPU, y KV cache.
Kubernetes para AI workloads
- Docker Compose:1-3 servicios, 1 servidor con GPU, equipo pequeno. Mas simple, menos overhead operativo.
- Kubernetes:5+ servicios, multiples servidores con GPU, auto-scaling, multi-tenant, equipo de plataforma dedicado.
- "/models/qwen3.5-27b-awq"
- "--max-model-len"
- "--gpu-memory-utilization"
- "--enable-prefix-caching"
Monitoring de infraestructura
- "9090:9090"
- "3000:3000"
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- NVIDIA_VISIBLE_DEVICES=all
Estrategias de backup
- * * /opt/scripts/backup.sh >> /var/log/backup.log 2>&1
- Que NO necesitas backupear: Modelos base (Qwen3.5-27B, Llama 3.3): se re-descargan de HuggingFace. Docker images: se re-buildan desde Dockerfile. Lo critico son los artefactos que TU generaste: LoRA adapters, datasets, prompts, configs y metadata.
Seguridad de infraestructura IA
- - API keys:nunca en codigo ni .env en produccion. Docker Swarm secrets o Kubernetes Secrets (cifrados).
- Network:vLLM no expuesto directamente. Siempre detras de Caddy/Nginx con TLS. Puerto 8000 solo accesible via localhost o red interna.
- Model poisoning:verificar checksums de modelos descargados. Descargar solo de fuentes confiables (HuggingFace oficial).
- Prompt injection:validar inputs antes de enviar al LLM. Sanitizar outputs antes de ejecutar herramientas.
- GPU side-channels:en entornos multi-tenant, GPUs compartidas pueden filtrar informacion. MIG aísla a nivel hardware.
Siguiente: TE10 - Project: Production Deploy with Observability
Track capstone: deploy a complete AI system on your own infrastructure with monitoring, alerts and cost tracking.
Ir al modulo TE10