En este modulo
Multimodal landscape in 2026
AI is no longer just text. In 2026, production systems process voice, images, video and documents in integrated flows. A support assistant receives a call (audio), transcribes it (STT), analyzes screenshots of the problem (vision), generates a response (LLM), and reads it aloud (TTS). All in under 3 seconds.
For the AI engineer, this means mastering a new stack of components and knowing when to use commercial APIs and when to self-host each modality.
The 4 critical modalities
- Text-to-Speech (TTS): converting text to natural voice. From chatbot responses to podcast production.
- Speech-to-Text (STT): transcribing audio to text. From meetings to call centers.
- Vision: analyzing images and video. From invoice OCR to industrial quality inspection.
- Audio analysis: classifying, segmenting and analyzing audio beyond transcription.
Text-to-Speech: ElevenLabs vs Piper
ElevenLabs (commercial API) is the quality standard in TTS. Natural voices, voice cloning, multiple languages, emotion control. Quality is indistinguishable from human recording in most cases. Pricing from 5 USD/month (30 minutes of audio).
Piper (self-hosted, free) is an open-source TTS engine, fast, lightweight and CPU-capable. Quality is lower than ElevenLabs but sufficient for many use cases. Supports Spanish, English, and 30+ languages. Speed: ~50x realtime on CPU.
When to use each
ElevenLabs: public content (podcasts, videos, demos), voice cloning, maximum naturalness. Piper: internal notifications, IVR, alerts, sensitive data that can't leave your infra, high volume with limited budget.
Speech-to-Text: Whisper and alternatives
OpenAI's Whisper is the reference open-source STT model. Whisper Large V3 transcribes audio in 100+ languages with professional quality. It can be self-hosted with faster-whisper for optimal speed. Self-hosted processes audio at 20-30x realtime on GPU. Breakeven vs API: ~500 hours/month with A100.
Vision models: image analysis
Current multimodal models (GPT-4o, Claude Sonnet, Gemini, Qwen-VL) can analyze images directly in conversation. For specific tasks and high volume, specialized models are more efficient. Qwen2.5-VL-7B is a good open-source option for OCR, image analysis, description and visual classification.
Audio processing
Beyond transcription, audio contains valuable information: who speaks (speaker diarization with pyannote), emotional tone (emotion recognition with wav2vec2 fine-tuned), background noise, sound event classification.
Multimodal pipelines
A multimodal pipeline combines multiple modalities in a coherent flow: Audio -> STT -> LLM -> TTS -> Audio. Typical latency: STT 200-500ms, Retrieval 50-100ms, LLM 500-2000ms, TTS 200-1000ms. Total: 950-4100ms.
Real-time voice
Real-time voice conversation is the holy grail: the user speaks, the system responds with natural voice, with <1 second latency. This requires aggressive optimizations: streaming STT, streaming LLM tokens to TTS by sentence, WebSocket for bidirectional communication. Target TTFA (Time to First Audio): <800ms.
Sovereignty in multimodal
In regulated environments, each multimodal pipeline component must meet the same sovereignty constraints as the text LLM. STT: self-hosted Whisper. TTS: self-hosted Piper. Vision: self-hosted Qwen-VL. Speaker diarization: self-hosted pyannote. Total sovereign multimodal stack: ~19GB VRAM, fits on an A100-40GB alongside the text LLM if using quantized models.
Ejercicio practico
- Install faster-whisper and transcribe a 5-minute audio in your language. Measure latency and quality.
- Install Piper and generate audio from 5 sentences. Compare subjective quality with ElevenLabs (use free trial).
- Build a STT -> LLM -> TTS pipeline: record a question with your microphone, transcribe with Whisper, generate response with your LLM (TE03), synthesize with Piper.
- Measure the end-to-end pipeline latency. Identify the bottleneck.
- If you have GPU with sufficient VRAM, deploy Qwen2.5-VL-7B and analyze 10 images (screenshots, invoices, diagrams).
- Implement streaming: the LLM generates tokens and TTS synthesizes by sentences. Measure TTFA (time to first audio).
Bonus: Add speaker diarization with pyannote to a meeting recording. Combine with Whisper for transcription with speaker labels.
Puntos clave
Puntos clave from TE07
- TTS: ElevenLabs for maximum quality and public content. Piper for self-hosting, high volume and sensitive data. Both are necessary in a professional stack.
- STT: Whisper large-v3 with faster-whisper is the standard. Self-hosted processes audio at 20-30x realtime on GPU. Breakeven vs API: ~500 hours/month.
- Vision: multimodal models (Qwen-VL, GPT-4o) for flexible analysis. Specialized OCR (Tesseract, LayoutLM) for high volume and structured documents.
- Streaming is mandatory for real-time voice. LLM generates -> TTS synthesizes by sentences -> audio sent to client. Target TTFA: <800ms.
- Multimodal sovereignty: same criteria as for text. Customer audio and video don't leave your infra. Whisper + Piper + Qwen-VL fit on an A100-40GB.
Guia de estudio — Conceptos clave de TE07
Panorama multimodal en 2026
- Text-to-Speech (TTS):convertir texto en voz natural. Desde respuestas de chatbot hasta produccion de podcasts.
- Speech-to-Text (STT):transcribir audio a texto. Desde reuniones hasta call centers.
- Vision:analizar imagenes y video. Desde OCR de facturas hasta inspeccion de calidad industrial.
- Audio analysis:clasificar, segmentar y analizar audio mas alla de la transcripcion.
Text-to-Speech: ElevenLabs vs Piper
- Precios ElevenLabs: desde 5 USD/mes (30 minutos de audio). Plan Scale: 99 USD/mes (500 minutos). Para produccion con volumen, negociar plan Enterprise.
- Cuando usar cada uno: ElevenLabs: contenido publico (podcasts, videos, demos), clonacion de voz, maxima naturalidad. Piper: notificaciones internas, IVR, alertas, datos sensibles que no pueden salir de tu infra, alto volumen con presupuesto limitado.
Speech-to-Text: Whisper y alternativas
- Deepgram:API comercial, optimizada para real-time. Mas rapida que Whisper API. Precio similar.
- Assembly AI:API con speaker diarization integrada. Buena para reuniones multi-participante.
- Whisper.cpp:Whisper en C++, funciona en CPU y dispositivos edge. Para apps moviles o IoT.
Procesamiento de audio
- {turn.end:.1f}s] {speaker}")
- 5.2s] SPEAKER_00
- 12.1s] SPEAKER_01
- 18.7s] SPEAKER_00
Soberania en multimodal
- STT:Whisper self-hosted. Audio de clientes no sale de tu infraestructura.
- TTS:Piper self-hosted. ElevenLabs solo para contenido publico (no datos de cliente).
- Vision:Qwen-VL self-hosted. Imagenes con datos sensibles no se envian a APIs externas.
- Speaker diarization:pyannote self-hosted. Grabaciones de reuniones internas son datos sensibles.
- Stack soberano multimodal: STT: faster-whisper large-v3 (self-hosted, ~5GB VRAM). TTS: Piper (self-hosted, CPU). Vision: Qwen2.5-VL-7B (self-hosted, ~14GB VRAM). Total: ~19GB VRAM. Cabe en una A100-40GB junto con el LLM de texto si usas modelos cuantizados.
Siguiente: TE08 - Scalability and Performance
Your system is multimodal. Now it needs to scale: caching, load balancing, async patterns and cost optimization.
Ir al modulo TE08