Open-source landscape in 2026

The open-source model ecosystem has matured dramatically. In 2024, choosing an open-source model was a gamble: quality varied enormously, documentation was scarce and benchmarks didn't reflect real performance. In 2026, the best open-source models match or exceed 2024 commercial models in most practical tasks.

The real gap with frontier models (GPT-4o, Claude Opus, Gemini Ultra) has narrowed to a tight segment: complex multi-step reasoning, tasks requiring very broad knowledge, and high-quality creative generation. For classification, information extraction, summarization, code and tool calling, a well-configured 27B open-source model is competitive.

This radically changes the cost equation. If before you needed a 15 USD/M token API to get decent quality, now you can self-host a model that covers 80% of cases for a fixed cost.

The 4 families that matter

Qwen (Alibaba Cloud)

The Qwen family has become the default option for production self-hosting. Qwen3.5 is available in variants from 0.6B to 235B parameters. The sweet spot is Qwen3.5-27B: fits on a 24GB GPU quantized, and its performance in reasoning, code and tool calling tasks is exceptional for its size.

• Qwen3.5-7B: fast classification, embeddings, simple tasks. Fits on any modern GPU.
• Qwen3.5-27B: the workhorse. Reasoning, code, analysis, tool calling. 24GB VRAM in Q4.
• Qwen3.5-72B: near-frontier quality. Requires 2x A100 (80GB) or 1x H100.
• License: Apache 2.0. Commercial use without restrictions.

Llama (Meta)

Meta's Llama 3.3 remains relevant, especially for its fine-tuning ecosystem and tooling. Llama 3.3 70B is an excellent model for general tasks, with good multilingual capability and solid code performance.

• Llama 3.3 8B: fast, efficient. Good option for edge deployment.
• Llama 3.3 70B: competitive with GPT-4 on many tasks. Requires multi-GPU.
• Llama 3.3 405B: the largest open-source. Frontier quality but impractical for self-hosting (<8x A100).
• License: Llama Community License. Commercial use permitted, with restrictions for >700M monthly users.

Mistral (Mistral AI)

Mistral has pivoted toward commercial models, but their open-source releases remain relevant. Mistral Nemo (12B) is fast and efficient. Codestral specializes in code.

• Mistral Nemo 12B: good quality/speed balance. 128K context window.
• Codestral 22B: specialized in code. Excellent for autocompletion and generation.
• License: Apache 2.0 (Nemo). Codestral has a non-commercial license (Mistral Non-Production License).

Phi (Microsoft)

The Phi family demonstrates that well-trained small models can compete with much larger ones. Phi-4 (14B) is surprisingly capable for its size, especially in mathematical reasoning and code.

• Phi-4 14B: the best sub-15B model on the market. Fits on modest GPUs (16GB VRAM).
• Phi-4-mini 3.8B: ideal for edge, mobile, or as a classification/routing model.
• License: MIT License. The most permissive license possible.

Selection criteria: decision framework

Choosing a model isn't "whichever scores best on the benchmark". It is a multidimensional decision. These are the 7 criteria ordered by practical importance:

1. VRAM requirements

The criterion most teams ignore and that causes the most problems. If the model doesn't fit on your GPU, nothing else matters.

# VRAM estimation by precision
# Rule: ~2 bytes/parameter in FP16, ~1 byte in Q8, ~0.5 bytes in Q4

model_params = 27e9  # Qwen3.5-27B

vram_fp16 = model_params * 2 / 1e9      # 54 GB
vram_q8   = model_params * 1 / 1e9      # 27 GB
vram_q4   = model_params * 0.5 / 1e9    # 13.5 GB
vram_kv_cache = 4  # GB additional for KV cache with batch size 1

print(f"FP16: {vram_fp16 + vram_kv_cache:.0f} GB")   # 58 GB -> H100 or 2x A100
print(f"Q8:   {vram_q8 + vram_kv_cache:.0f} GB")     # 31 GB -> 1x A100-40GB tight
print(f"Q4:   {vram_q4 + vram_kv_cache:.0f} GB")     # 17.5 GB -> 1x RTX 4090 or A5000

2. Quality on YOUR specific task

General benchmarks (MMLU, HumanEval) are indicative. What matters is how the model performs on your concrete task. Create an eval dataset of 50-100 representative examples of your use case and evaluate each candidate model.

3. Inference speed (tokens/second)

A model generating 15 tokens/s is unacceptable for an interactive chatbot. 40-60 tokens/s is the comfortable range. Larger models are slower.

4. Effective context window

The announced number (128K, 1M) is not the effective window. Most models degrade their performance significantly after a certain point. Qwen3.5-27B works well up to ~32K tokens in practice, although it announces 128K.

5. Tool calling support

If your system uses agents or tools, the model needs robust tool calling (function calling) support. Not all models implement it well. Qwen3.5 and Llama 3.3 are the best at this.

6. License

Being able to download a model doesn't mean you can use it in commercial production. Review the license before integrating. More detail in the licenses section.

7. Community and support

A model with an active community has more available fine-tunes, better documentation, and bugs that get reported and fixed faster. Qwen and Llama win here by community volume.

Benchmarking: how to evaluate models

Public benchmarks are a starting point, not an answer. Here's how to interpret and complement them:

Key public benchmarks

• MMLU (Massive Multitask Language Understanding): general knowledge. Useful for knowing if the model "knows things", but doesn't predict performance on specific tasks.
• HumanEval / MBPP: code generation. HumanEval is too easy (modern models score >80%). MBPP+ is more useful.
• MT-Bench: multi-turn conversational quality. Evaluated by LLM (GPT-4 as judge). More practical than MMLU.
• BFCL (Berkeley Function Calling Leaderboard): tool calling quality. Critical if you use agents.
• Arena ELO (Chatbot Arena): blind human evaluation. The most reliable benchmark of perceived quality.

Your own benchmark: eval dataset

# Eval dataset structure
# eval_dataset.jsonl
{"input": "Classify this support ticket: 'I can't access my account'",
 "expected_output": "category: access, priority: high",
 "tags": ["classification", "support"]}

{"input": "Summarize this contract in 3 key points: [contract text]",
 "expected_output": "1. Duration: 12 months...",
 "tags": ["summary", "legal"]}

# Automated evaluation
import json

def evaluate_model(model, eval_path):
    results = []
    with open(eval_path) as f:
        for line in f:
            example = json.loads(line)
            output = model.generate(example["input"])
            score = score_output(output, example["expected_output"])
            results.append({"tags": example["tags"], "score": score})

    # Aggregate by tags
    from collections import defaultdict
    tag_scores = defaultdict(list)
    for r in results:
        for tag in r["tags"]:
            tag_scores[tag].append(r["score"])

    for tag, scores in tag_scores.items():
        avg = sum(scores) / len(scores)
        print(f"{tag}: {avg:.2%}")

Quantization: GGUF, AWQ, GPTQ

Quantization reduces the numerical precision of model weights so it takes less VRAM and is faster. The trade-off is a small quality loss. In practice, the loss is imperceptible in Q8 and acceptable in Q4 for most tasks.

Quantization formats

• GGUF: the llama.cpp format. Works on CPU and GPU. Multiple quantization levels (Q2_K to Q8_0). Ideal for local development and limited hardware.
• AWQ (Activation-aware Weight Quantization): 4-bit quantization optimized for GPU. Better quality than GPTQ at the same compression level. Supported by vLLM. Recommended for production.
• GPTQ: pioneering 4-bit quantization. Well supported but AWQ has surpassed it in quality. Still viable if your tooling only supports GPTQ.

Quality and VRAM comparison

# Qwen3.5-27B: quantization impact
# -----------------------------------------------
# Format     VRAM    MMLU    Tok/s (A100)  Use
# -----------------------------------------------
# FP16        54 GB   78.3%   25 t/s        Reference
# Q8_0 GGUF   27 GB   78.1%   35 t/s        Best possible quality
# AWQ 4-bit   14 GB   77.5%   55 t/s        Recommended production
# GPTQ 4-bit  14 GB   76.8%   50 t/s        Alternative to AWQ
# Q4_K_M GGUF 15 GB   77.2%   45 t/s        Local development
# Q2_K GGUF    8 GB   71.2%   60 t/s        Not recommended (quality)
# -----------------------------------------------
# Note: MMLU is indicative. Evaluate with your own dataset.

How to quantize a model

# Quantize with llama.cpp (GGUF)
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make

# Convert from HuggingFace to GGUF
python convert_hf_to_gguf.py \
  --outfile qwen3.5-27b-q4_k_m.gguf \
  --outtype q4_k_m \
  /path/to/Qwen3.5-27B/

# Quantize with AutoAWQ (for vLLM)
pip install autoawq
python -c "
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model = AutoAWQForCausalLM.from_pretrained('Qwen/Qwen3.5-27B')
tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen3.5-27B')

quant_config = {'zero_point': True, 'q_group_size': 128, 'w_bit': 4}
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized('Qwen3.5-27B-AWQ')
"

Model cards: what to read before using a model

A model card is the official documentation of the model. Before integrating a model into your system, read these 5 fields:

1. Training data: what data it was trained on. Important for compliance (GDPR, copyrighted data).
2. Intended use: what tasks it was designed for. A code model is not ideal for legal analysis.
3. Limitations: what it can't do. Languages with low performance, unsupported tasks, known biases.
4. Evaluation results: official benchmarks. Compare with your own evals.
5. License: what you can do with the model. Commercial production, modification, redistribution.

License comparison

Open-source model licenses are a minefield. "Open-source" doesn't always mean "use as you wish". This table summarizes the most common licenses:

# LLM model license comparison
# -------------------------------------------------------------------
# Model          License              Commercial use  Restrictions
# -------------------------------------------------------------------
# Qwen3.5         Apache 2.0            Yes            None
# Phi-4            MIT                   Yes            None
# Llama 3.3       Llama Community       Yes            >700M MAU
# Mistral Nemo    Apache 2.0            Yes            None
# Codestral       MNPL                  No             Research only
# Gemma 2         Gemma Terms           Yes            Export restrictions
# DeepSeek V3     MIT (weights)         Yes            None
# -------------------------------------------------------------------
# MNPL = Mistral Non-Production License

Selection guide by use case

Support chatbot (critical latency)

Recommendation: Qwen3.5-7B AWQ. Fast, efficient, good conversational quality. Alternative: Phi-4 if support is technical.

Document analysis (long context)

Recommendation: Qwen3.5-27B AWQ. Wide context window, good comprehension. For documents >50 pages, consider commercial API with 1M token context.

Code generation

Recommendation: Qwen3.5-27B for general generation. Codestral 22B for autocompletion (non-commercial use only). DeepSeek V3 as MIT alternative.

Classification and routing

Recommendation: Phi-4-mini 3.8B or Qwen3.5-0.6B. Small models are ideal as classifiers/routers before invoking large models.

Multi-agent system

Recommendation: Qwen3.5-27B as coordinator (good tool calling) + Phi-4 as specialized workers. Qwen has the best function calling support in open-source.

Ejercicio practico

Puntos clave