Evaluate a Fine-Tuned LLM: Prove It Actually Got Better

What you’ll build

A repeatable evaluation pipeline that answers the only question that matters after a fine-tune: did the adapter actually make the model better at your task? You’ll get:

• A held-out eval set built from your training data (so you never grade on what you trained on)
• A perplexity sanity check (cheap, fast, and the first thing to look at)
• An LLM-as-judge scorer for open-ended outputs
• A paired A/B win rate with a confidence interval — the single most trustworthy signal
• Heuristic metrics for structured tasks (no judge required)
• A version-pinned, copy-pasteable evaluate.py that runs on the same vLLM server from tutorial #2

Prerequisites: The fine-tuned adapter from tutorial #1, a vLLM server running base + adapter from tutorial #2, Python 3.10+, and one 24 GB GPU (evaluation is inference-only).

The honest version of this tutorial: you cannot prove a model is better in the absolute sense. You can only measure performance on a sample of tasks and bound how much you can trust the number. Everything here is about making that measurement honest.

1. The core problem: loss going down ≠ the model got better

During training, the Trainer reports a loss that goes down. It feels like progress. It isn’t a reliable signal of quality, for three reasons:

1. Training loss measures token prediction, not task success. A model can lower its loss by memorizing your training examples verbatim and still be useless on a new prompt.
2. You are evaluating on the data you trained on. That’s like grading an exam you wrote the answers to. The number is meaningless for generalization.
3. Loss is a single scalar that hides what you actually care about. A model can have identical loss to another and produce wildly different, more or less useful outputs.

The fix is to evaluate on held-out data you never trained on, with a metric that matches your task. That’s the whole game.

Rule #1: Never evaluate on training data. If your eval set overlaps your training set, every number you produce is optimistic and you will ship a worse model thinking it’s better.

2. The two kinds of evaluation

Intrinsic metrics are fast and worth running, but they are a proxy. A fine-tune can improve intrinsic perplexity on in-domain text and still not help on the actual task (or vice versa). Task-specific evaluation is what you make the ship/no-ship decision on.

3. Build a held-out eval set

Split your data before you train, not after. If you train on everything and then “hold out” some, you’ve already seen it.

# split.py — run this BEFORE training
import json, random

random.seed(42)
with open("train.jsonl") as f:
    rows = [json.loads(l) for l in f if l.strip()]

random.shuffle(rows)
n_eval = max(50, int(len(rows) * 0.15))          # 15%, floor of 50
eval_rows, train_rows = rows[:n_eval], rows[n_eval:]

for path, data in [("eval.jsonl", eval_rows), ("train_split.jsonl", train_rows)]:
    with open(path, "w") as f:
        for r in data:
            f.write(json.dumps(r) + "\n")

print(f"train: {len(train_rows)}  eval: {len(eval_rows)}")

Then fine-tune on train_split.jsonl (tutorial #1) and evaluate on eval.jsonl.

How big should the eval set be? Big enough that the number is stable. For a win-rate metric, a rule of thumb: with N = 100 paired comparisons, a 95% confidence interval is roughly ±10 points, so a 60% win rate is read as “somewhere between 50% and 70%.” With N = 300, it tightens to roughly ±5.5 points. If your task is important, use 200–500 held-out examples.

Quirk: if you only have a small dataset (say 200 examples total), a 15% split gives you 30 eval examples — too few to trust. In that case, keep a dedicated eval set that was never part of training, even if it means less training data. A trustworthy 50-example eval beats a fast 20-example one.

4. Sanity check: perplexity

Perplexity is the average “surprise” per token. Lower is better. It’s the cheapest evaluation you can run and the first thing to look at — if the fine-tuned model has higher perplexity on your held-out domain text than the base, something went wrong.

# perplexity.py
import math, json
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

def perplexity(model, tokenizer, texts, max_length=1024, batch_size=1):
    total_loss, total_tokens = 0.0, 0
    for i in range(0, len(texts), batch_size):
        batch = tokenizer(
            texts[i:i+batch_size],
            return_tensors="pt", padding=True, truncation=True, max_length=max_length,
        )
        batch = {k: v.to(model.device) for k, v in batch.items()}
        with torch.inference_mode():
            out = model(**batch, labels=batch["input_ids"])
        n = (batch["input_ids"] != tokenizer.pad_token_id).sum().item()
        total_loss += out.loss.item() * n
        total_tokens += n
    return math.exp(total_loss / total_tokens)

def load(base, adapter=None):
    tok = AutoTokenizer.from_pretrained(base)
    if tok.pad_token is None:
        tok.pad_token = tok.eos_token
    m = AutoModelForCausalLM.from_pretrained(base, load_in_4bit=True, device_map="auto")
    if adapter:
        m = PeftModel.from_pretrained(m, adapter)
    return m, tok

with open("eval.jsonl") as f:
    rows = [json.loads(l) for l in f if l.strip()]
# Score the *responses* the way the model was trained to produce them
tok, _ = load("tiiuae/falcon-7b-instruct")
texts = [tok.apply_chat_template(
    [{"role":"user","content":r["instruction"]},{"role":"assistant","content":r["output"]}],
    tokenize=False) for r in rows]

base_model, base_tok = load("tiiuae/falcon-7b-instruct")
ft_model,   _        = load("tiiuae/falcon-7b-instruct", "falcon-7b-lora/final_adapter")

print(f"Base perplexity:   {perplexity(base_model, base_tok, texts):.2f}")
print(f"Fine-tuned ppl:    {perplexity(ft_model,   base_tok, texts):.2f}")

Reading it: a fine-tune should lower perplexity on in-domain held-out text (the model now finds your domain more predictable). If it goes up, the fine-tune overfit or the data/labels are noisy. Perplexity is a sanity check, not a verdict — a model can win on perplexity and still lose on your actual task.

VRAM note: this loads the 4-bit base model (~4.3 GB) and, for the fine-tuned run, the adapter on top (~0.3 GB). It fits comfortably on a 24 GB GPU. Run the two loads sequentially (as above) so you never hold two full models in memory at once.

5. Serve both models for the A/B comparison

The A/B comparison needs the base model and the fine-tuned model answering the same prompts. Tutorial #2’s server only serves the fine-tuned adapter, so start a second server for the base model. Two options:

Option A — two servers (recommended for a clean A/B):

# Terminal 1 — base model (no adapter)
vllm serve tiiuae/falcon-7b-instruct \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 2048 --dtype float16 \
    --trust-remote-code \
    --served-model-name falcon-7b

# Terminal 2 — fine-tuned (the server from tutorial #2)
vllm serve tiiuae/falcon-7b-instruct \
    --host 0.0.0.0 --port 8001 \
    --enable-lora --max-loras 1 --max-lora-rank 64 \
    --lora-modules falcon-7b-lora=./my-falcon-lora \
    --max-model-len 2048 --dtype float16 \
    --trust-remote-code \
    --served-model-name falcon-7b-lora

This needs two GPUs (or run them sequentially on one). If you only have one 24 GB GPU, run the base server, generate all base outputs, stop it, start the fine-tuned server, generate the fine-tuned outputs — gen.py below works fine pointed at whichever server is up.

Option B — one server, two names: vLLM can serve the base model and the adapter under one endpoint. The base model is the default model name; the adapter is the --lora-modules name. Point BASE_MODEL and FT_MODEL at the two names on the same VLLM_BASE_URL.

Why two clean servers matters: the whole point of the A/B is that the only difference between the two outputs is the adapter. Same base weights, same context length, same dtype, same decoding. If the base and fine-tuned runs differ in anything else (context, dtype, max_tokens), you’re measuring the difference, not the fine-tune.

6. Generate outputs from base and fine-tuned

The trustworthy comparison generates outputs from both models on the same held-out prompts, then scores them.

# gen.py — generate outputs for base and fine-tuned on the eval set
import json, os
from openai import OpenAI

# Point each at its own server (Option A) or the same server (Option B)
BASE_URL   = os.environ.get("BASE_URL",   "http://localhost:8000/v1")
FT_URL     = os.environ.get("FT_URL",     "http://localhost:8001/v1")
BASE_MODEL = os.environ.get("BASE_MODEL", "falcon-7b")        # base, no adapter
FT_MODEL   = os.environ.get("FT_MODEL",   "falcon-7b-lora")   # adapter name
SYSTEM     = "You are a helpful assistant."

base_client = OpenAI(base_url=BASE_URL, api_key="not-needed")
ft_client   = OpenAI(base_url=FT_URL,   api_key="not-needed")

def generate(client, model, prompt, max_tokens=256, temperature=0.7):
    r = client.chat.completions.create(
        model=model,
        messages=[{"role":"system","content":SYSTEM},{"role":"user","content":prompt}],
        max_tokens=max_tokens, temperature=temperature,
    )
    return r.choices[0].message.content

with open("eval.jsonl") as f:
    rows = [json.loads(l) for l in f if l.strip()]

out = []
for r in rows:
    out.append({
        "instruction": r["instruction"],
        "reference":   r.get("output", ""),
        "base":        generate(base_client, BASE_MODEL, r["instruction"]),
        "fine_tuned":  generate(ft_client,   FT_MODEL,   r["instruction"]),
    })

with open("gen_outputs.jsonl", "w") as f:
    for o in out:
        f.write(json.dumps(o) + "\n")
print(f"Generated {len(out)} pairs -> gen_outputs.jsonl")

Fairness quirk: generate both models with identical temperature, max_tokens, and system prompt. Comparing a greedy base model against a sampling fine-tuned model (or different max_tokens) injects noise that has nothing to do with the fine-tune. Keep the decoding settings the same on both sides.

7. LLM-as-judge for open-ended outputs

Your outputs are open-ended text, so there’s no exact-match to compute. The standard approach is LLM-as-judge: a (usually stronger) model grades the outputs against the instruction.

# judge.py
import os
from openai import OpenAI

JUDGE_URL   = os.environ.get("JUDGE_BASE_URL", "http://localhost:8001/v1")
JUDGE_MODEL = os.environ.get("JUDGE_MODEL", "judge-model")

judge = OpenAI(base_url=JUDGE_URL, api_key=os.environ.get("JUDGE_API_KEY", "not-needed"))

JUDGE_PROMPT = """You are an impartial judge comparing two assistant responses to the same instruction.
Judge ONLY on: (1) does it follow the instruction, (2) is it correct, (3) is it helpful and not padded.
Do NOT favor longer answers or a particular writing style.

Instruction:
{instruction}

Reference answer (ground truth, if any):
{reference}

Response A:
{a}

Response B:
{b}

Reply with exactly one token: A, B, or TIE."""

def judge_pair(instruction, reference, a, b):
    r = judge.chat.completions.create(
        model=JUDGE_MODEL,
        messages=[{"role":"user","content":JUDGE_PROMPT.format(
            instruction=instruction, reference=reference or "(none)", a=a, b=b)}],
        temperature=0.0, max_tokens=8,
    )
    t = r.choices[0].message.content.strip().upper()
    if t.startswith("A"): return "A"
    if t.startswith("B"): return "B"
    return "TIE"

What to use as the judge. Any OpenAI-compatible endpoint works (a larger model you serve with vLLM on a second port, or an API). Two honest caveats:

• Self-judging is biased. If the judge is the same model family as the candidates, it tends to favor its own style. Prefer a judge that is different from and stronger than the models being compared.
• The judge is a proxy too. It can be wrong. That’s why the next section (paired win rate with a confidence interval) matters more than any single judge’s verdict.

8. Paired A/B win rate (the trustworthy signal)

Absolute judge scores are noisy and biased. The most robust comparison is paired: for each prompt, show the judge the base output and the fine-tuned output (in a randomized order to kill position bias), and record who wins. Aggregate into a win rate with a confidence interval.

# winrate.py
import json, random, math

random.seed(0)

def wilson(k, n, z=1.96):
    """95% Wilson score interval for a proportion k/n."""
    if n == 0:
        return (0.0, 0.0)
    p = k / n
    d = 1 + z*z/n
    c = (p + z*z/(2*n)) / d
    h = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / d
    return (max(0.0, c-h), min(1.0, c+h))

# judge_pair and load gen_outputs.jsonl as in judge.py / gen.py
with open("gen_outputs.jsonl") as f:
    rows = [json.loads(l) for l in f if l.strip()]

wins = ties = losses = 0
for r in rows:
    # Randomize which side is "A" so the judge can't lean on position
    ft_first = random.random() < 0.5
    a, b = (r["fine_tuned"], r["base"]) if ft_first else (r["base"], r["fine_tuned"])
    verdict = judge_pair(r["instruction"], r["reference"], a, b)
    if verdict == "TIE":
        ties += 1
    else:
        a_won = (verdict == "A")
        ft_won = a_won if ft_first else (not a_won)
        wins   += ft_won
        losses += (not ft_won)

n = wins + losses
lo, hi = wilson(wins, n)
print(f"Decided pairs: {n}  (ties: {ties})")
print(f"Fine-tuned win rate: {wins/n:.1%}  95% CI [{lo:.1%}, {hi:.1%}]")
print(f"  base wins: {losses}  fine-tuned wins: {wins}  ties: {ties}")

How to read the result:

• The fine-tune “works” if the win rate is above 50% and the lower bound of the CI is above 50%. A 60% win rate with CI [50%, 70%] is not a confident win — the interval includes parity.
• Ties are real and common on a well-matched task. Don’t force a winner. Reporting ties honestly is what keeps the number trustworthy.
• A fine-tune that loses (win rate < 50%, CI below 50%) is a signal to go back to the data or hyperparameters — not to tweak the eval.

Why this beats a single score: it’s a relative measurement on paired examples, which cancels out a lot of the judge’s absolute bias and the task’s inherent difficulty. The confidence interval tells you how much you can trust it. That combination is the closest you get to an honest “did it get better.”

9. Heuristic metrics for structured tasks

If your task has a checkable structure (classification, extraction, fixed format), you can skip the judge entirely and measure directly. This is fully self-contained and reproducible.

# heuristic.py — example: exact match + format compliance
import json, re

with open("gen_outputs.jsonl") as f:
    rows = [json.loads(l) for l in f if l.strip()]

def norm(s): return re.sub(r"\s+", " ", s.strip().lower())

exact = format_ok = 0
for r in rows:
    out = r["fine_tuned"]
    # 1) format compliance: does it contain the required marker?
    if "ANSWER:" in out:
        format_ok += 1
    # 2) exact match against the reference (for short-answer tasks)
    if norm(out) == norm(r["reference"]):
        exact += 1

n = len(rows)
print(f"Format compliance: {format_ok/n:.1%}")
print(f"Exact match:       {exact/n:.1%}")

Swap in whatever matches your task — rouge_score for summarization, bleu for translation, f1 for entity extraction, a regex for a specific schema. The point is a deterministic metric you can re-run and that doesn’t depend on a judge’s mood.

10. Cost and VRAM

Evaluation is inference-only — no gradients, no optimizer states. The memory footprint is the same as the inference numbers in tutorial #1:

You can run base and fine-tuned generation on the same vLLM server (tutorial #2 serves both under one endpoint), so you don’t need two GPUs. If you run a local judge, serve it on a second port — or, to stay on one GPU, run the judge sequentially after generation.

Time: for N=200 examples, generation is ~2N model calls and judging is ~N calls. On a single 4090 with vLLM, expect on the order of minutes to a low number of hours depending on output length. Run your own numbers.

11. The traps that make bad fine-tunes look good

This is the part most write-ups skip. Each of these will make a worse model look better:

1. Evaluating on training data. The most common and most damaging. The model memorized the answers, so it “wins.” → Always use a held-out set (section 3).
2. Tiny eval set. N=20 gives a win-rate CI of roughly ±22 points. You can’t tell a 55% model from a 65% one. → Use 100–300+ examples (section 3).
3. Position bias in the judge. LLM judges favor whichever response is listed first. → Randomize A/B order per example (section 8).
4. Self-judging. A model grading its own family’s outputs inflates its own score. → Use a different, stronger judge (section 7).
5. Mismatched decoding settings. Comparing greedy base vs sampling fine-tuned (or different max_tokens) adds noise that isn’t the fine-tune. → Identical settings both sides (section 6).
6. Trusting a single scalar. Perplexity or one judge score can move for reasons unrelated to your task. → Use the paired win rate with a CI as the decision metric (section 8).
7. No baseline. You can’t claim improvement without measuring the base model on the same prompts. → Always A/B against base (section 8).
8. Overfitting to the eval set. If you iterate hyperparameters by repeatedly checking the same eval set, you quietly start training on it. → Keep a final, untouched holdout for the last decision.

The meta-rule: the goal of evaluation is to make a decision you can defend. Every method above exists to shrink the gap between “the number says it’s better” and “it’s actually better.”

12. Putting it together

The pieces above are one script. Save as evaluate.py and run it against your base and fine-tuned servers (section 6):

"""
evaluate.py — prove a LoRA fine-tune actually improved the model.
Pipeline: paired A/B generation -> LLM-as-judge win rate with a 95% CI.

Env:
  BASE_URL   (default http://localhost:8000/v1)  base-model server
  FT_URL     (default http://localhost:8001/v1)  fine-tuned server
  BASE_MODEL (default falcon-7b)                 base model name
  FT_MODEL   (default falcon-7b-lora)            fine-tuned adapter name
  JUDGE_URL  (default http://localhost:8002/v1)  judge server
  JUDGE_MODEL(default judge-model)               judge model name
"""
import json, os, math, random
from openai import OpenAI

random.seed(0)
BASE_URL   = os.environ.get("BASE_URL",   "http://localhost:8000/v1")
FT_URL     = os.environ.get("FT_URL",     "http://localhost:8001/v1")
BASE_MODEL = os.environ.get("BASE_MODEL", "falcon-7b")
FT_MODEL   = os.environ.get("FT_MODEL",   "falcon-7b-lora")
JUDGE_URL  = os.environ.get("JUDGE_URL",  "http://localhost:8002/v1")
JUDGE_MODEL= os.environ.get("JUDGE_MODEL","judge-model")
SYSTEM     = "You are a helpful assistant."

base_client = OpenAI(base_url=BASE_URL, api_key="not-needed")
ft_client   = OpenAI(base_url=FT_URL,   api_key="not-needed")
judge       = OpenAI(base_url=JUDGE_URL, api_key=os.environ.get("JUDGE_API_KEY", "not-needed"))

JUDGE_PROMPT = """You are an impartial judge comparing two assistant responses to the same instruction.
Judge ONLY on: (1) follows the instruction, (2) correctness, (3) helpfulness without padding.
Do NOT favor longer answers or a particular style.

Instruction:
{instruction}

Reference answer (ground truth, if any):
{reference}

Response A:
{a}

Response B:
{b}

Reply with exactly one token: A, B, or TIE."""

def generate(client, model, prompt, max_tokens=256, temperature=0.7):
    r = client.chat.completions.create(
        model=model,
        messages=[{"role":"system","content":SYSTEM},{"role":"user","content":prompt}],
        max_tokens=max_tokens, temperature=temperature)
    return r.choices[0].message.content

def judge_pair(instruction, reference, a, b):
    r = judge.chat.completions.create(
        model=JUDGE_MODEL,
        messages=[{"role":"user","content":JUDGE_PROMPT.format(
            instruction=instruction, reference=reference or "(none)", a=a, b=b)}],
        temperature=0.0, max_tokens=8)
    t = r.choices[0].message.content.strip().upper()
    return "A" if t.startswith("A") else "B" if t.startswith("B") else "TIE"

def wilson(k, n, z=1.96):
    if n == 0: return (0.0, 0.0)
    p = k/n; d = 1 + z*z/n
    c = (p + z*z/(2*n))/d
    h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
    return (max(0.0, c-h), min(1.0, c+h))

def main():
    with open("eval.jsonl") as f:
        rows = [json.loads(l) for l in f if l.strip()]
    print(f"Evaluating on {len(rows)} held-out examples\n")

    wins = losses = ties = 0
    for i, r in enumerate(rows):
        base = generate(base_client, BASE_MODEL, r["instruction"])
        ft   = generate(ft_client,   FT_MODEL,   r["instruction"])
        ft_first = random.random() < 0.5
        a, b = (ft, base) if ft_first else (base, ft)
        v = judge_pair(r["instruction"], r.get("output",""), a, b)
        if v == "TIE":
            ties += 1
        else:
            ft_won = (v == "A") if ft_first else (v == "B")
            wins   += ft_won
            losses += (not ft_won)
        if (i+1) % 25 == 0:
            print(f"  {i+1}/{len(rows)}  (W {wins} L {losses} T {ties})")

    n = wins + losses
    lo, hi = wilson(wins, n)
    print(f"\nDecided pairs: {n}   ties: {ties}")
    print(f"Fine-tuned win rate: {wins/n:.1%}   95% CI [{lo:.1%}, {hi:.1%}]")
    verdict = "IMPROVED" if lo > 0.5 else "NOT CONFIRMED (CI includes parity)"
    print(f"Verdict: {verdict}")

if __name__ == "__main__":
    main()

pip install openai
python evaluate.py

Interpreting the verdict: IMPROVED means the lower bound of the 95% CI is above 50% — a defensible win. NOT CONFIRMED means the data can’t distinguish the fine-tune from the base. In that case, the honest options are: more eval examples, a better task-aligned metric, or going back to improve the fine-tune.

13. Troubleshooting

14. What to do next

• Keep a final untouched holdout. Use a dev split for iteration and a sealed test set for the last ship/no-ship decision.
• Track eval in CI. Re-run evaluate.py on every new adapter so a regression is caught before it ships, not after.
• Add a task-specific metric. Pair the judge with a deterministic metric (exact match, F1, format compliance) for a second, independent signal.
• Report the CI, always. A win rate without a confidence interval is a number you can’t defend.
• That’s the whole cluster. You now have the loop: fine-tune → serve → evaluate. The next step is a better dataset or a harder task.

Last updated: 2026-10-17. Evaluation is inference-only; VRAM figures match the inference budget in tutorial #1. Run your own benchmarks — judge and model choices materially affect the numbers.