When fine-tuning vs prompt engineering

Fine-tuning is not the first option. It is the last. Before investing hours preparing datasets and training, try solving your problem with prompt engineering. The decision is pragmatic, not academic.

Use prompt engineering when

• You have fewer than 500 training examples.
• The desired output format can be described with clear instructions.
• The task is generic (summarization, translation, simple classification).
• You need to iterate quickly (change behavior without retraining).
• You use a frontier model via API (you can't effectively fine-tune GPT-4o yourself).

Use fine-tuning when

• You have 1,000+ high-quality examples and the base model doesn't achieve the desired quality with prompting.
• You need a very specific and consistent output format (JSON schema, XML, proprietary format).
• The domain is specialized and the base model lacks sufficient knowledge (medical jargon, legal, industrial sector).
• Latency matters: a fine-tuned model needs shorter prompts (no few-shot examples), reducing TTFT.
• Cost matters: a fine-tuned 7B model can match a base 70B model on your specific task.

LoRA and QLoRA: the minimum necessary theory

Full fine-tuning of a 27B model requires ~200GB of VRAM (weights + gradients + optimizer state). Impractical. LoRA (Low-Rank Adaptation) solves this by training only a small set of additional matrices (~1-5% of parameters), keeping the original weights frozen.

How LoRA works

In each attention layer of the transformer, LoRA inserts two small matrices (A and B) whose product is added to the original weights. The rank of these matrices (r) controls capacity: more rank = more trainable parameters = more adaptation capacity, but more VRAM and more overfitting risk.

# Intuition: LoRA decomposes weight updates
# W_original (4096 x 4096) = 16M parameters (frozen)
# LoRA A (4096 x r) + LoRA B (r x 4096)
# If r=16: 4096*16 + 16*4096 = 131K parameters (0.8% of original)

# QLoRA: LoRA + base model quantized to 4-bit
# The base model loads in 4-bit (saves VRAM)
# LoRA matrices train in FP16 (maintains precision)
# Result: fine-tuning a 27B model on a 24GB GPU

QLoRA: fine-tuning on accessible hardware

QLoRA combines base model quantization (4-bit NF4) with LoRA. The base model takes ~14GB for a 27B, and LoRA matrices + optimizer state add ~4-6GB. Total: ~20GB. Fits on an RTX 4090 or A100-40GB.

ChatML dataset preparation

The dataset is the most important component of fine-tuning. Mediocre models with good data outperform excellent models with bad data. Quality matters more than quantity.

ChatML format

# ChatML is the standard format for fine-tuning conversational models
# Each example is a complete conversation

# dataset.jsonl
{"conversations": [
  {"role": "system", "content": "You are a security analyst specialized in SOC alert triage."},
  {"role": "user", "content": "Alert: multiple failed login from IP 185.143.223.47 against admin user. 47 attempts in 2 minutes. Source: SIEM."},
  {"role": "assistant", "content": "## Triage\n\n**Severity:** High\n**Category:** Brute Force (T1110)\n**IOC:** 185.143.223.47\n\n### Analysis\n- 47 attempts in 2 minutes against privileged account indicates automated brute force attack\n- IP 185.143.223.47: European range, verify in threat intel\n\n### Recommended actions\n1. Block IP at perimeter firewall\n2. Verify if the account was compromised\n3. Review last 24h logs for this IP"}
]}

# Another example
{"conversations": [
  {"role": "system", "content": "You are a security analyst specialized in SOC alert triage."},
  {"role": "user", "content": "Alert: abnormal DNS traffic detected. Host 10.0.1.45 performing 3000 DNS queries to random domains in 5 minutes."},
  {"role": "assistant", "content": "## Triage\n\n**Severity:** Critical\n**Category:** DNS Tunneling / C2 (T1071.004)\n**IOC:** Host 10.0.1.45, DNS beaconing pattern\n\n### Analysis\n- 3000 queries in 5 min to random domains is a classic DNS tunneling or DGA (Domain Generation Algorithm) pattern\n- Possible data exfiltration or C2 communication\n\n### Recommended actions\n1. Isolate host 10.0.1.45 from the network immediately\n2. Capture memory and disk for forensic analysis\n3. Analyze queried domains against CTI feeds\n4. Review historical traffic for the host"}
]}

Dataset quality rules

1. Format consistency: all responses must follow the same format. If you use Markdown headers, use them always. If you use severity as a field, it should always be present.
2. Input diversity: cover a variety of scenarios. Not 500 brute force examples and 2 phishing ones.
3. Output quality: each assistant response should be a perfect example of what you want the model to produce. If you wouldn't publish it as reference, don't include it.
4. Appropriate length: neither too short (the model learns to be lazy) nor too long (the model learns to pad with filler).
5. Consistent system prompt: use the same system prompt for all examples of the same task.

How many examples do you need

# Empirical rules for dataset size
# --------------------------------------------------
# Task                    Minimum examples    Ideal
# --------------------------------------------------
# Classification (3-5 cat)       200              1000
# Custom output format           500              2000
# Specialized domain            1000              5000
# Agent with tool calling       2000             10000
# --------------------------------------------------
# Beyond 10K, diminishing returns unless
# task variability is very high

Training with Unsloth

Unsloth is the fastest tool for LLM fine-tuning. It automatically optimizes memory usage and training speed, making fine-tuning 2-5x faster than standard HuggingFace Transformers.

# Installation
pip install unsloth

# Fine-tuning script with Unsloth + QLoRA
from unsloth import FastLanguageModel
import torch

# 1. Load base model in 4-bit
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Qwen/Qwen3.5-27B",
    max_seq_length=4096,
    dtype=None,          # Auto-detect (float16 for A100, bfloat16 for H100)
    load_in_4bit=True,   # QLoRA: base model in 4-bit
)

# 2. Add LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,                       # LoRA rank (16-64 typical)
    target_modules=[            # Which layers to adapt
        "q_proj", "k_proj", "v_proj", "o_proj",  # Attention
        "gate_proj", "up_proj", "down_proj",       # FFN
    ],
    lora_alpha=32,              # Scale factor (typical: 2*r)
    lora_dropout=0.05,          # Regularization
    bias="none",
    use_gradient_checkpointing="unsloth",  # Saves VRAM
)

# 3. Prepare dataset
from datasets import load_dataset

dataset = load_dataset("json", data_files="dataset.jsonl", split="train")

def format_chat(example):
    """Converts ChatML to text format for the tokenizer."""
    messages = example["conversations"]
    text = tokenizer.apply_chat_template(messages, tokenize=False)
    return {"text": text}

dataset = dataset.map(format_chat)

# 4. Train
from trl import SFTTrainer
from transformers import TrainingArguments

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=4096,
    args=TrainingArguments(
        output_dir="./output",
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,  # Effective batch = 2*4 = 8
        warmup_steps=10,
        num_train_epochs=3,
        learning_rate=2e-4,
        fp16=not torch.cuda.is_bf16_supported(),
        bf16=torch.cuda.is_bf16_supported(),
        logging_steps=10,
        save_strategy="epoch",
        optim="adamw_8bit",  # 8-bit optimizer (saves VRAM)
    ),
)

trainer.train()

# 5. Save model
model.save_pretrained("./qwen3.5-27b-soc-lora")
tokenizer.save_pretrained("./qwen3.5-27b-soc-lora")

# 6. Merge LoRA with base model (for serving with vLLM)
model.save_pretrained_merged(
    "./qwen3.5-27b-soc-merged",
    tokenizer,
    save_method="merged_16bit",  # Merge + save in FP16
)

Training with Axolotl

Axolotl is an alternative to Unsloth with more configuration options. It is configured via YAML, which facilitates reproducibility and version control of training configurations.

# axolotl_config.yml
base_model: Qwen/Qwen3.5-27B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_4bit: true
adapter: qlora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj

datasets:
  - path: dataset.jsonl
    type: sharegpt  # ChatML format
    conversation: chatml

sequence_len: 4096
sample_packing: true  # Packs multiple short examples into one sequence

num_epochs: 3
micro_batch_size: 2
gradient_accumulation_steps: 4
learning_rate: 2e-4
lr_scheduler: cosine
warmup_steps: 10
optimizer: adamw_bnb_8bit

output_dir: ./output
save_strategy: epoch
logging_steps: 10

bf16: auto
flash_attention: true
gradient_checkpointing: true

# Run training with Axolotl
pip install axolotl
accelerate launch -m axolotl.cli.train axolotl_config.yml

# Merge LoRA after training
python -m axolotl.cli.merge_lora axolotl_config.yml --lora_model_dir ./output/checkpoint-final

Hyperparameters that matter

Of the 20+ possible hyperparameters, these 5 determine 90% of the result:

• Learning rate (2e-4 to 5e-5): too high = forgets base knowledge. Too low = doesn't learn. 2e-4 is a good starting point for QLoRA.
• LoRA rank (8, 16, 32, 64): more rank = more capacity, more VRAM, more overfit risk. 16 is the safe default. Raise to 32-64 if you have >5K examples.
• Epochs (1-5): more epochs = more overfit risk. 3 epochs is typical. With <1K examples, 1-2 epochs.
• Effective batch size (4-16): micro_batch * gradient_accumulation. Higher values stabilize training. 8 is a good default.
• Max sequence length: must cover your longest input + output. Longer = more VRAM. Don't set 8192 if your examples are 2000 tokens.

Evaluating the fine-tuned model

# Evaluation: compare base model vs fine-tuned
import json
from openai import OpenAI

def evaluate_model(client, model_name, eval_path):
    results = []
    with open(eval_path) as f:
        for line in f:
            example = json.loads(line)
            messages = example["conversations"][:-1]  # Without assistant response
            expected = example["conversations"][-1]["content"]

            response = client.chat.completions.create(
                model=model_name,
                messages=messages,
                max_tokens=1024,
                temperature=0
            )
            predicted = response.choices[0].message.content

            # Scoring (adapt to your task)
            score = compute_score(predicted, expected)
            results.append(score)

    avg_score = sum(results) / len(results)
    print(f"{model_name}: {avg_score:.2%} ({len(results)} examples)")
    return avg_score

# Evaluate base
base_score = evaluate_model(base_client, "Qwen3.5-27B-AWQ", "eval_dataset.jsonl")

# Evaluate fine-tuned
ft_score = evaluate_model(ft_client, "qwen3.5-27b-soc-merged", "eval_dataset.jsonl")

print(f"Improvement: {ft_score - base_score:+.2%}")

Common fine-tuning mistakes

1. Contaminated dataset: eval data in the train set. Inflated results. Separate before any preprocessing.
2. Overfitting: loss drops to zero, but the model can only repeat training examples. Solution: fewer epochs, more dropout, or more data.
3. Catastrophic forgetting: the model loses general capabilities after fine-tuning. Solution: use LoRA (preserves base weights), reduce learning rate, or mix general data with specific data.
4. Inconsistent format: the dataset mixes output formats. The model doesn't know which format to produce. Solution: normalize all outputs to the same format before training.
5. Low quality data: "garbage in, garbage out" applies especially to fine-tuning. 500 perfect examples beat 5000 mediocre ones.

Ejercicio practico

Puntos clave