Why observability for AI

An LLM in production without observability is a blind system. You don't know if responses are good, if costs are spiraling, if latency has risen, or if an agent is stuck in an infinite tool call loop. You discover it when a user complains or when the bill arrives.

AI system observability is more complex than traditional software observability for three reasons: non-deterministic outputs, variable cost per request, and complex execution chains involving multiple LLM calls and tools.

The three pillars: metrics, logs, traces

Metrics (what is happening now)

Aggregated numbers in real time. Throughput, latency, error rate, GPU usage, tokens consumed. Stored in time series (Prometheus, InfluxDB). Visualized in dashboards (Grafana).

Logs (what happened exactly)

Individual events with detail. Each request with its input, output, timestamps, errors. Stored in log systems (Loki, Elasticsearch). Searched when something fails.

Traces (how a request flowed)

The complete path of a request through all system components. Critical for multi-service and multi-agent systems. Stored in trace backends (Jaeger, Tempo). Visualized as waterfalls.

LLM-specific metrics

Beyond standard API metrics (latency, error rate, throughput), LLM systems have their own metrics: TTFT, TPS, E2E latency, GPU utilization, KV cache usage, queue depth for performance. Tokens input/output totals, cost per request, cost per user, cache hit rate for costs. Format compliance, hallucination rate, user satisfaction for quality.

OpenTelemetry for LLMs

OpenTelemetry (OTel) is the open-source standard for instrumentation. It produces metrics, logs and traces you can send to any backend. For LLMs, there are specific semantic conventions.

# Instrumentation with OpenTelemetry for LLM calls
from opentelemetry import trace, metrics

tracer = trace.get_tracer("llm-service")
meter = metrics.get_meter("llm-service")

request_counter = meter.create_counter("llm.requests.total")
token_counter = meter.create_counter("llm.tokens.total")
latency_histogram = meter.create_histogram("llm.request.duration", unit="ms")

async def llm_inference(messages: list, model: str):
    with tracer.start_as_current_span("llm.inference") as span:
        span.set_attribute("gen_ai.system", "vllm")
        span.set_attribute("gen_ai.request.model", model)

        response = await client.chat.completions.create(model=model, messages=messages, max_tokens=1024)

        request_counter.add(1, {"model": model, "status": "success"})
        token_counter.add(response.usage.prompt_tokens, {"type": "input", "model": model})
        token_counter.add(response.usage.completion_tokens, {"type": "output", "model": model})

        return response

Langfuse: native AI observability

Langfuse is an observability platform designed specifically for LLMs. Open-source, self-hostable, with direct integration for LangChain, LlamaIndex and OpenAI-compatible APIs. It offers conversation traces, scoring, prompt management, automatic cost tracking, and datasets/evals from production.

Tool call and agent tracing

In systems with agents and tool calls, a request can generate multiple LLM calls and tool invocations. Without tracing, debugging "why the agent gave an incorrect response" is impossible.

Dashboards: what to monitor

Three dashboards: Operational (SRE) with RPS, latency percentiles, error rate, GPU utilization. Costs (business) with daily/weekly/monthly cost, cost per user, tokens consumed. Quality (product) with format compliance rate, user satisfaction, hallucination rate.

Alerting: when and how

Alerts must be actionable. An alert that fires and doesn't require immediate action is noise. Critical alerts for an LLM system: P99 latency above SLA, error rate >5%, GPU VRAM almost full, abnormal daily cost, agent loop detected.

Ejercicio practico

Puntos clave