When to fine-tune vs. prompt vs. RAG

Teams reach for fine-tuning because it sounds like ownership of the model. Sometimes that is the right move. Often the task is 'the answer is in our documents' or 'follow these instructions', and training is the expensive way to avoid writing them down. The choice is about what kind of knowledge or behaviour you need, and whether you can tell a good output from a bad one.

Prompt when the task is an instruction

If a careful person could do the job by reading a short brief, the model can often do it from that brief. Prompts carry format, tone, the steps to follow, and the cases where the model must refuse. Few examples help when the format is unusual. Prompts are the right first tool because you can change them without a training run, and you can read them. They fail when the brief would have to contain your whole library, or when the model consistently ignores a constraint no matter how you phrase it.

RAG when the answer lives in documents

Policies, tickets, catalogues, contracts, course notes: if the truth can change without the model changing, you do not want that truth baked into weights. Retrieval puts the current passage in the prompt. You can delete a document and have it stop being quoted. You can show a person the passage the answer came from. RAG fails when the corpus does not contain the answer, when retrieval cannot find it, or when the task is not lookup at all. A model that must always emit a rigid domain format is not fixed by pasting more documents.

Fine-tune when behaviour will not stick

Fine-tuning, and in particular a small adaptation such as LoRA or another PEFT method, is for a stable behaviour or a domain format the base model does not hold after you have already tried instructions. Examples: a classification label set the model keeps blurring, a structured output it keeps breaking, a dialect of your domain it will not stay inside. You need examples you are allowed to train on, and a held-out set written before you train. If you cannot say what a wrong output looks like, you cannot tell whether the checkpoint helped.

Dataset preparation is the work

A pile of chats is not a dataset. We want examples that match the task you will actually serve: the same kind of input, the same kind of output, and a note of what must not appear. Duplicates and leaked answers in the held-out set make the eval a compliment. Personal data that should never be memorised does not belong in a training file. The base model stays available as a fallback so a bad checkpoint is a deployment choice, not a trap.

Evaluation before you pick a winner

Write the eval harness first. A handful of cases you care about, scored by a person or by a check that does not depend on the model flattering itself. Compare prompt-only, retrieval, and an adaptation on that same set. The winner is the simplest option that meets the bar you wrote down. If retrieval misses, fix retrieval before you train a model to hallucinate the missing passage. If the prompt works, do not train. Training is justified when the eval shows a gap the other two cannot close.

def choose(eval_set):
    prompt_score = run(prompt_only, eval_set)
    rag_score = run(with_retrieval, eval_set)
    if meets_bar(prompt_score):
        return "prompt"
    if meets_bar(rag_score):
        return "rag"
    adapted = train_lora(approved_train_set)
    adapt_score = run(adapted, eval_set)
    if meets_bar(adapt_score):
        return "lora"
    return "redesign the task or the data"

Combinations are normal

A production system often uses more than one. Instructions set the policy. Retrieval supplies facts that change. A small adaptation holds a format. An agent calls tools when the task is an action rather than a paragraph. The mistake is to fine-tune in order to avoid building retrieval, or to stuff the prompt in order to avoid deciding what the task is. Pick the mechanism that matches the kind of knowledge, and prove it on an eval set you trust. Simpler is the default until that set says otherwise. Training is a later step, not a badge. Do not train to conceal a document the retriever should have found. If the corpus is the source of truth, retrieval is the mechanism.

A short way to explain the choice

Ask where the truth lives. If it lives in a document that someone will edit, retrieve it. If it lives in an instruction a reviewer can read, prompt it. If it lives in a behaviour you can demonstrate with examples and the base model will not keep, adapt the model and keep the eval set as the gate. Ask who can label a wrong answer. If nobody can, you are not ready to train and you are barely ready to prompt. Ask what must never be memorised. Facts that expire, secrets, and personal records belong in retrieval or in tools, not in weights. LoRA and other PEFT methods matter because they keep the adaptation small and reversible: you can drop the adapter and return to the base model. That reversibility is part of the decision, not an implementation detail you add later. The order is always the same. Write the eval. Try the prompt. Try retrieval if the corpus has the answer. Train only when those two leave a gap you can describe. A gap you cannot show on the held-out set is not a reason to train. It is a reason to keep reading the task.