Why vLLM

vLLM (Virtual Large Language Model) is the reference inference engine for LLMs in production. Created at UC Berkeley, it implements PagedAttention, which solves the biggest problem of self-hosting: efficient GPU memory management for serving multiple simultaneous requests.

Before vLLM, serving an LLM to multiple users meant wasting 60-80% of VRAM on KV cache fragmentation. vLLM reduces that waste to <5%, allowing 3-5x more concurrent requests with the same GPU.

Alternatives and when to use them

• TGI (Text Generation Inference) by HuggingFace: easier to configure, less optimized. Good for prototypes.
• Ollama: ideal for local development. Not for multi-user production.
• llama.cpp / llamafile: CPU inference. Useful for edge deployment or when you don't have a GPU.
• SGLang: emerging alternative to vLLM with better performance in certain structured generation scenarios.

For production with dedicated GPUs and multiple concurrent users, vLLM is the standard choice in 2026.

GPU requirements

The GPU is the most critical and expensive component. Choosing the right one determines which models you can serve, at what speed, and with how many concurrent users.

GPU map by model

# GPU requirements by model (with AWQ 4-bit)
# ------------------------------------------------------------------
# Model              Min VRAM   Recommended GPU       Tok/s  Concurrency
# ------------------------------------------------------------------
# Phi-4 14B AWQ         10 GB    RTX 4090 (24GB)        80     10-15
# Qwen3.5-7B AWQ         6 GB    RTX 4090 (24GB)       100     15-20
# Qwen3.5-27B AWQ       17 GB    A100-40GB              55     8-12
# Qwen3.5-27B FP16      54 GB    A100-80GB              25     4-6
# Llama 3.3 70B AWQ     38 GB    A100-80GB              30     4-8
# Llama 3.3 70B AWQ     38 GB    2x A100-40GB (TP=2)    35     6-10
# Qwen3.5-72B AWQ       40 GB    2x A100-40GB (TP=2)    28     4-6
# ------------------------------------------------------------------
# Tok/s = tokens per second per request (generation)
# Concurrency = simultaneous requests before degradation

# Typical monthly cost (rental)
# ------------------------------------------------------------------
# GPU                   Hetzner     RunPod      AWS
# ------------------------------------------------------------------
# RTX 4090              ~80 EUR     ~250 USD    N/A
# A100-40GB             ~130 EUR    ~700 USD    ~2,500 USD
# A100-80GB             ~200 EUR    ~1,100 USD  ~4,000 USD
# H100-80GB             ~350 EUR    ~2,500 USD  ~8,000 USD
# ------------------------------------------------------------------

Setup: from zero to inference

Step 1: Install vLLM

# Prerequisitos: Python 3.10+, CUDA 12.1+, NVIDIA GPU
# Verify CUDA
nvidia-smi
# Should show your GPU and CUDA version

# Install vLLM via pip
pip install vllm

# Or with conda (recommended to avoid CUDA conflicts)
conda create -n vllm python=3.11
conda activate vllm
pip install vllm

Step 2: Download model

# Option A: vLLM downloads automatically from HuggingFace
# You only need to specify the model name at startup

# Option B: Pre-download (recommended for production)
pip install huggingface_hub
huggingface-cli download Qwen/Qwen3.5-27B-AWQ --local-dir /models/qwen3.5-27b-awq

# Verify the model downloaded correctly
ls -la /models/qwen3.5-27b-awq/
# Should contain: config.json, tokenizer.json, model-*.safetensors

Step 3: Start the server

# Basic server
vllm serve Qwen/Qwen3.5-27B-AWQ \
  --port 8000 \
  --host 0.0.0.0

# Server with production options
vllm serve /models/qwen3.5-27b-awq \
  --port 8000 \
  --host 0.0.0.0 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 16 \
  --enable-prefix-caching \
  --api-key "your-secret-api-key"

# Parameters explained:
# --max-model-len: maximum context window (reduces VRAM if lowered)
# --gpu-memory-utilization: % of VRAM for vLLM (0.90 = 90%, leaves 10% for overhead)
# --max-num-seqs: maximum concurrent requests
# --enable-prefix-caching: cache common prefixes (saves compute on repetitive prompts)
# --api-key: protect the endpoint (mandatory in production)

Step 4: Verify it works

# Call the endpoint (OpenAI API compatible)
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-secret-api-key" \
  -d '{
    "model": "Qwen/Qwen3.5-27B-AWQ",
    "messages": [{"role": "user", "content": "Hello, how are you?"}],
    "max_tokens": 100,
    "temperature": 0.7
  }'

# From Python (using openai client, since vLLM is compatible)
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="your-secret-api-key"
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-27B-AWQ",
    messages=[{"role": "user", "content": "Explain PagedAttention in 3 sentences"}],
    max_tokens=200
)
print(response.choices[0].message.content)

Continuous batching: the key to throughput

Without batching, each request is processed sequentially: one user waits for the previous one to finish. With continuous batching, vLLM groups multiple requests and processes them in parallel on the GPU, leveraging compute capacity that would otherwise be idle.

How it works

The inference engine has two phases: prefill (processing the prompt) and decode (generating tokens one by one). Continuous batching allows that while one request is in the decode phase (generating token by token), another request enters the prefill phase. The GPU is never waiting.

# Without continuous batching (sequential)
# Request 1: [prefill 200ms][decode 1500ms]
# Request 2:                               [prefill 200ms][decode 1500ms]
# Throughput: 2 requests / 3.4s = 0.59 req/s

# With continuous batching
# Request 1: [prefill 200ms][decode................1500ms]
# Request 2:       [prefill 200ms][decode................1500ms]
# Request 3:              [prefill 200ms][decode................1500ms]
# Throughput: 3 requests / ~2s = 1.5 req/s (2.5x better)

# In vLLM, continuous batching is active by default
# Adjust max_num_seqs to control parallelism
# More seqs = more throughput, but more latency per individual request

Trade-off: throughput vs latency

More concurrency = more total throughput, but each individual request is slightly slower because it shares the GPU. For interactive chatbots, limit to 8-16 seqs. For batch processing, raise to 64-128.

KV cache: memory management

The KV cache stores attention results from already-processed tokens. Without cache, each new token would require reprocessing the entire previous context. The KV cache consumes the largest part of VRAM after the model weights.

PagedAttention: vLLM's innovation

Before vLLM, the KV cache was allocated as a contiguous memory block per request. This caused fragmentation: if a request reserved 8K tokens but only used 2K, the remaining 6K were wasted VRAM. PagedAttention divides the KV cache into pages (like the operating system's virtual memory), eliminating fragmentation.

# KV cache configuration in vLLM
vllm serve /models/qwen3.5-27b-awq \
  --gpu-memory-utilization 0.90 \    # 90% of VRAM available for model + KV cache
  --max-model-len 8192 \             # Limiting context reduces KV cache usage
  --enable-prefix-caching \          # Reuse KV cache between requests with common prefixes
  --block-size 16                    # KV cache page size (default: 16 tokens)

# Monitor KV cache usage
# vLLM exposes metrics at /metrics (Prometheus format)
# Key metric: vllm:gpu_cache_usage_perc
# If constantly at 95%+, you need more VRAM or reduce max_model_len

Prefix caching

If multiple requests share the same system prompt (common in chatbots), prefix caching reuses the KV cache from the shared prefix. In a chatbot with a 2000-token system prompt, this saves ~40ms of prefill per request and reduces VRAM usage.

Tensor parallelism: multi-GPU

When a model doesn't fit on a single GPU (Llama 70B FP16 needs ~140GB of VRAM), tensor parallelism splits the model across multiple GPUs. Each GPU processes a portion of each model layer, communicating via NVLink or PCIe.

# Llama 3.3 70B AWQ on 2x A100-40GB
vllm serve meta-llama/Llama-3.3-70B-AWQ \
  --tensor-parallel-size 2 \    # Split across 2 GPUs
  --port 8000 \
  --max-model-len 4096

# Requirements for tensor parallelism:
# 1. Identical GPUs (same model and VRAM)
# 2. NVLink preferred (PCIe works but is slower)
# 3. tensor-parallel-size must be a power of 2 (1, 2, 4, 8)
# 4. Model weights must be divisible by TP size

# Typical performance with TP=2 vs single GPU:
# Throughput: ~1.8x (not 2x due to communication overhead)
# Latency: ~0.95x (slightly better due to less compute per GPU)

Docker deployment for production

# docker-compose.yml for vLLM in production
version: "3.8"
services:
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - VLLM_API_KEY=${VLLM_API_KEY}
    volumes:
      - /data/models:/models
    ports:
      - "8000:8000"
    command: >
      --model /models/qwen3.5-27b-awq
      --port 8000
      --host 0.0.0.0
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --max-num-seqs 16
      --enable-prefix-caching
      --api-key ${VLLM_API_KEY}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 120s  # Large models take time to load

  # Reverse proxy with TLS
  caddy:
    image: caddy:2
    ports:
      - "443:443"
      - "80:80"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
    depends_on:
      - vllm

volumes:
  caddy_data:

# Caddyfile
llm.yourdomain.com {
    reverse_proxy vllm:8000
    header {
        Strict-Transport-Security "max-age=31536000"
        X-Content-Type-Options "nosniff"
    }
}

Throughput benchmarking

# Benchmark with official vLLM script
python -m vllm.entrypoints.openai.api_server --model /models/qwen3.5-27b-awq &

# Install benchmark tool
pip install aiohttp

# Custom benchmark script
import asyncio
import aiohttp
import time
import json

async def send_request(session, url, api_key, prompt):
    payload = {
        "model": "qwen3.5-27b-awq",
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 256,
        "temperature": 0.7
    }
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json"
    }
    start = time.perf_counter()
    async with session.post(url, json=payload, headers=headers) as resp:
        result = await resp.json()
        elapsed = time.perf_counter() - start
        tokens = result["usage"]["completion_tokens"]
        return {"latency": elapsed, "tokens": tokens, "tok_per_s": tokens / elapsed}

async def benchmark(url, api_key, num_requests=100, concurrency=10):
    prompts = [f"Explain machine learning concept number {i}" for i in range(num_requests)]
    connector = aiohttp.TCPConnector(limit=concurrency)
    async with aiohttp.ClientSession(connector=connector) as session:
        tasks = [send_request(session, url, api_key, p) for p in prompts]
        start = time.perf_counter()
        results = await asyncio.gather(*tasks)
        total_time = time.perf_counter() - start

    latencies = [r["latency"] for r in results]
    tok_per_s = [r["tok_per_s"] for r in results]

    print(f"Total requests: {num_requests}")
    print(f"Concurrency: {concurrency}")
    print(f"Total time: {total_time:.1f}s")
    print(f"Throughput: {num_requests / total_time:.1f} req/s")
    print(f"Latency P50: {sorted(latencies)[len(latencies)//2]:.2f}s")
    print(f"Latency P99: {sorted(latencies)[int(len(latencies)*0.99)]:.2f}s")
    print(f"Tokens/s P50: {sorted(tok_per_s)[len(tok_per_s)//2]:.0f}")

asyncio.run(benchmark(
    "http://localhost:8000/v1/chat/completions",
    "your-secret-api-key",
    num_requests=100,
    concurrency=10
))

Common troubleshooting

OOM (Out of Memory)

The most frequent error. Solutions: reduce --max-model-len, reduce --max-num-seqs, lower --gpu-memory-utilization to 0.85, or use a more quantized model.

Slow first request (cold start)

vLLM loads the model into VRAM at startup. A 27B model takes 30-60 seconds. Use healthcheck with a generous start_period in Docker. Don't send traffic until the healthcheck passes.

Degradation under load

If latency spikes when concurrency rises, the KV cache is full. Monitor vllm:gpu_cache_usage_perc. If it's at 95%+, reduce --max-model-len or add another GPU.

Ejercicio practico

Puntos clave