MLOps for LLMs: what changes

Classic MLOps was designed for traditional ML models (random forest, XGBoost, small neural networks). The pipeline was: data -> training -> validation -> deploy -> monitoring. With LLMs, the pipeline changes in several fundamental ways.

Key differences from classic MLOps

• The base model isn't trained, it's chosen. In classic ML, you train your model from scratch. With LLMs, you choose a pre-trained model and adapt it (fine-tuning) or use it directly (prompting). "Training" is partially replaced by "prompt engineering".
• Prompts are code. A system prompt change modifies the system's behavior as much as a model weight change. Prompts need versioning, testing and rollback just like code.
• Models are enormous. A 27B model takes 14-54GB. You can't version it in Git. You need a specialized model registry.
• Evaluation is subjective. In classification, accuracy is objective. In text generation, "is the response good" is subjective. You need automated evals (LLM-as-judge) and human evaluation.
• Cost is variable. An API model has per-token cost. A self-hosted model has fixed cost but limited capacity. CI/CD needs to include cost metrics.

Model registry: model versioning

A model registry is a centralized store for models with metadata, versions and lineage. For LLMs, it stores: model weights (or reference), configuration, LoRA adapters, eval results, and associated prompts.

Model registry options

• MLflow Model Registry: open-source, mature, integrates with everything. Supports large models via artifact storage (S3, GCS, MinIO).
• Weights & Biases (W&B): experiment tracking + model registry. Excellent UX. Free plan for personal use.
• HuggingFace Hub: model store with Git LFS versioning. Ideal if your ecosystem is HuggingFace.
• DIY with S3/MinIO + metadata DB: the simplest. Folder with versioning + PostgreSQL table with metadata.

# DIY model registry structure (MinIO + PostgreSQL)
# Storage (MinIO)
models/
  qwen3.5-27b-soc/
    v1.0/
      model/           # Merged weights (or LoRA adapter)
      tokenizer/
      config.json
      eval_results.json
      training_config.yml
    v1.1/
      ...

# Metadata (PostgreSQL)
CREATE TABLE model_versions (
    id SERIAL PRIMARY KEY,
    model_name VARCHAR(255) NOT NULL,
    version VARCHAR(50) NOT NULL,
    base_model VARCHAR(255),
    adapter_type VARCHAR(50),           -- 'lora', 'qlora', 'full', 'none'
    training_dataset VARCHAR(255),
    training_examples INT,
    eval_score FLOAT,
    eval_dataset VARCHAR(255),
    status VARCHAR(50) DEFAULT 'staging', -- 'staging', 'canary', 'production', 'retired'
    storage_path VARCHAR(500),
    created_at TIMESTAMP DEFAULT NOW(),
    deployed_at TIMESTAMP,
    notes TEXT,
    UNIQUE(model_name, version)
);

Experiment tracking

Experiment tracking records everything that happens during training and evaluation: hyperparameters, metrics, loss curves, eval results, artifacts. Without tracking, you can't reproduce or compare experiments.

# Experiment tracking with MLflow
import mlflow

mlflow.set_experiment("soc-alert-classifier")

with mlflow.start_run(run_name="qwen27b-qlora-r16-e3"):
    # Log hyperparameters
    mlflow.log_params({
        "base_model": "Qwen/Qwen3.5-27B",
        "adapter": "qlora",
        "lora_r": 16,
        "lora_alpha": 32,
        "learning_rate": 2e-4,
        "epochs": 3,
        "batch_size": 8,
        "dataset_size": 3500,
        "max_seq_len": 4096,
    })

    # Train (your Unsloth/Axolotl code here)
    trainer.train()

    # Log training metrics
    mlflow.log_metric("train_loss_final", trainer.state.log_history[-1]["loss"])

    # Evaluate
    eval_results = evaluate_model(model, "eval_dataset.jsonl")
    mlflow.log_metrics({
        "eval_accuracy": eval_results["accuracy"],
        "eval_f1": eval_results["f1"],
        "eval_latency_p50": eval_results["latency_p50"],
        "eval_latency_p99": eval_results["latency_p99"],
    })

Prompt versioning: the forgotten component

Prompts are executable code. A system prompt change can improve or destroy system quality. They need the same versioning rigor as source code.

# Prompt versioning structure
docs/prompts/
  soc_triage/
    system_v1.0.txt       # "You are a security analyst..."
    system_v1.1.txt       # Added structured output format
    system_v2.0.txt       # Complete rewrite with chain-of-thought
    CHANGELOG.md          # What changed and why in each version
    eval_results/
      v1.0_eval.json      # Eval results with v1.0
      v1.1_eval.json
      v2.0_eval.json

CI/CD pipeline for AI models

A CI/CD pipeline for AI has 6 stages. The first 3 are CI (validation). The last 3 are CD (deployment).

# CI/CD Pipeline for AI models
# ================================

# CI Stage 1: Lint + format validation
# CI Stage 2: Automated eval (gate: eval_score >= production_score * 0.98)
# CI Stage 3: Smoke test (start vLLM, send 10 requests, verify)

# CD Stage 4: Deploy to staging
# CD Stage 5: Canary deployment (5% traffic, monitor 24h)
# CD Stage 6: Full rollout (5% -> 25% -> 50% -> 100%)

A/B testing of models

A/B testing compares two models (or two prompts) by serving real traffic to both and measuring quality and performance metrics.

# A/B testing with percentage routing
import random
from dataclasses import dataclass

@dataclass
class ModelVariant:
    name: str
    endpoint: str
    weight: float  # Traffic percentage (0.0 - 1.0)

class ABRouter:
    def __init__(self, variants: list[ModelVariant]):
        self.variants = variants

    def route(self, request_id: str) -> ModelVariant:
        # Deterministic routing based on request_id
        hash_val = hash(request_id) % 1000 / 1000
        cumulative = 0.0
        for variant in self.variants:
            cumulative += variant.weight
            if hash_val < cumulative:
                return variant
        return self.variants[-1]

Canary deployments and rollback

A canary deployment sends a small percentage of traffic to the new model. If metrics degrade, it reverts automatically.

# Canary deployment with automatic health checks
class CanaryDeployer:
    async def deploy_canary(self, new_model: str, stages=[0.05, 0.25, 0.50, 1.0]):
        for weight in stages:
            self.router.set_weight("treatment", weight)
            await asyncio.sleep(3600)  # 1 hour per stage

            metrics = await self.metrics.get_comparison("control", "treatment", window_minutes=60)

            if self.should_rollback(metrics):
                self.router.set_weight("treatment", 0.0)
                return False

        return True

    def should_rollback(self, metrics) -> bool:
        if metrics.treatment_p99 > metrics.control_p99 * 1.2:
            return True
        if metrics.treatment_error_rate > metrics.control_error_rate + 0.01:
            return True
        return False

Pipeline automation with GitHub Actions

# .github/workflows/model-deploy.yml
name: Model CI/CD Pipeline

on:
  push:
    paths:
      - 'models/**'
      - 'prompts/**'
      - 'eval/**'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Validate dataset format
        run: python scripts/validate_dataset.py eval/eval_dataset.jsonl

  evaluate:
    needs: validate
    runs-on: [self-hosted, gpu]
    steps:
      - uses: actions/checkout@v4
      - name: Run eval suite
        run: python scripts/run_evals.py --model models/latest --eval-dataset eval/eval_dataset.jsonl
      - name: Check eval gate
        run: python scripts/check_eval_gate.py --results eval_results.json --min-score 0.85

  deploy-canary:
    needs: evaluate
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Start canary
        run: python scripts/canary_deploy.py --weight 0.05 --duration 24h

Ejercicio practico

Puntos clave