The challenge of scaling LLMs

Scaling a traditional web service means adding servers behind a load balancer. Scaling an LLM service is fundamentally different because the bottleneck is not CPU or RAM: it is GPU VRAM, which is expensive and scarce.

A web server handles 10,000 requests per second. An LLM server handles 10-50. This 3 orders of magnitude difference forces rethinking the entire scalability strategy. Every request you can avoid (cache), every token you can not generate (optimization), and every millisecond you can cut (async) has direct impact on capacity and cost.

The 3 scalability levers

• Reduce requests (caching): if 30% of queries are similar, semantic cache eliminates 30% of GPU load.
• Reduce tokens (optimization): shorter prompts, more concise outputs, more efficient models.
• Distribute load (balancing): multiple GPUs, multiple servers, intelligent routing.

Prompt caching: save compute for free

Prompt caching reuses the KV cache of prompt prefixes that repeat between requests. If your system prompt is 2000 tokens and all requests share it, the prefill of those 2000 tokens is calculated once and reused. In vLLM, enable with --enable-prefix-caching. Measured impact on a chatbot with 2000-token system prompt: 47% less TTFT latency, 33% more throughput.

Semantic cache: intelligent caching

Prompt caching works with identical prefixes. Semantic cache goes one step further: it caches responses for semantically similar queries. If one user asks "what is GDPR" and another asks "explain the general data protection regulation", semantic cache detects they are the same question and returns the cached response. Implementation uses embeddings + Redis with a similarity threshold (typically 0.92).

Load balancing for inference

With multiple vLLM instances, you need a load balancer that distributes traffic intelligently. Round-robin isn't enough: a 100-token request and a 10,000-token request don't consume the same resources. Use least-loaded strategy based on GPU metrics, or task-based routing with different models for different complexity levels.

Async patterns for high throughput

LLM calls are IO-bound (waiting for GPU response). Async patterns allow the server to process multiple requests while waiting for the GPU, maximizing throughput without adding hardware. Key patterns: request queue with worker pool, batch processing for non-interactive loads, fire-and-forget for non-critical tasks (logging, scoring, analytics).

Cost optimization

Two axes: reduce tokens (prompt compression, output truncation, context window management) and reduce cost per token (batch API at 50% discount, tiered models for 80% of simple queries, self-hosting during off-peak hours, prompt caching for 90% reduction on cached tokens).

Auto-scaling: scale on demand

Auto-scaling for LLMs is more complex than for web services. A vLLM instance takes 30-60 seconds to start (load model into VRAM). You can't scale in milliseconds. Strategies: pre-warming + conservative thresholds, warm replicas (pre-started but without traffic), scale up when GPU utilization >80%, scale down when <30% for 15 minutes.

Scalability benchmarking

Benchmark at multiple concurrency levels (1, 5, 10, 20, 50, 100) to identify the saturation point. Measure throughput (req/s), P50 latency, P99 latency and error count at each level.

Ejercicio practico

Puntos clave