Common Mistakes When Fine-Tuning Open-Source LLMs and How to Avoid Them

Fine-tuning an open-source model can feel like a clean, controlled experiment. You pick a base checkpoint, prepare a dataset, launch a run, and watch loss fall. Then the model fails on real prompts, or the benchmark looks fine while users complain. The culprit is rarely one dramatic bug. It is usually a handful of quiet mistakes that compound. Here are the ones that cost the most time and compute, and the checks that prevent them.

Data leakage is quieter than it looks

Leakage happens when examples the model will be judged on, or close paraphrases of them, appear in training. Exact duplicates are easy to catch. Near-duplicates are not. A support-ticket dataset can contain the same issue reworded three ways; an evaluation split made randomly can put sibling rows in both training and validation. The model then looks brilliant because it has seen the answer.

• Split by source, not by row. If 12 examples come from the same conversation, document, or customer, keep them in one split.
• Deduplicate with more than string equality. Normalize whitespace and case, then use n-gram overlap or embedding similarity to flag near matches.
• Hold out a final test set before you touch the data. Do not use it for hyperparameter tuning. If you change the model based on that set, it is no longer a test set.
• Search the training text for evaluation prompts and known answers. A simple substring scan catches more than people expect.

Also verify labels. If instruction data was generated from templates, the model may learn the template rather than the task. Keep a small set of human-written examples in validation to detect that.

Overfitting hides behind falling loss

Training loss almost always falls. Validation loss can fall too while generation quality gets worse. The model starts parroting phrases from your dataset, ignoring instructions, or producing the same opening line for every prompt. That is overfitting to style and surface patterns.

Early stopping on validation loss is a start, but it is not enough. Build a task metric that reflects what you care about: exact match for extraction, pass rate for unit tests, rubric scores for summaries, or a simple human review. Check generations every few hundred steps, not only at the end. If the metric peaks and then declines while loss keeps dropping, stop.

Practical signs

• Responses become longer or more repetitive without gaining accuracy.
• The model copies rare tokens or formatting quirks from training examples.
• Validation loss improves, but your task metric stalls.
• Performance on a general instruction set drops sharply.

For small instruction datasets, many teams find that one to three epochs are enough. More passes can memorize. With LoRA, increasing rank does not fix bad data; it can make memorization easier.

Evaluation that does not match the job

A model that scores well on a public benchmark may fail your product. Benchmarks are useful for rough comparisons, but they are not a substitute for a local evaluation set. Perplexity tells you how well the model predicts text, not whether it follows instructions, calls a tool correctly, or stays within a required format.

Create the evaluation before training. Write 50 to 200 prompts that mirror real use, including edge cases and adversarial inputs. Store expected outputs or clear grading rules. Then run the same evaluation on the base model and the fine-tuned model. If the fine-tuned model does not beat the base model on your task, do not ship it.

Be careful with automated graders. Exact match punishes valid paraphrases; a language model as a judge can be inconsistent and biased toward longer answers. For high-stakes checks, use a rubric and spot-check by hand. Keep the evaluation prompts out of training.

Chat templates, masks, and tokenizer traps

Many fine-tuning failures are mechanical. The training data uses one prompt format, inference uses another. The model learns to continue the user's message instead of replying because labels were not masked. Special tokens are added or removed incorrectly. The result looks like a modeling problem but is a preprocessing problem.

Match the tokenizer and chat template to the base model. If the base model expects a system role and specific turn markers, use them during training and inference. Mask prompt tokens so loss is computed only on the completion, unless you have a reason to train on the full sequence. For multi-turn data, keep the turn boundaries intact. Test one batch: decode the input IDs and labels to confirm the model sees what you think it sees.

Hyperparameters that waste GPU hours

A high learning rate can destroy a pretrained model in a few hundred steps. A learning rate that is too low wastes hours and produces a model that has barely changed. Batch size, sequence length, and gradient accumulation interact. Long sequences can dominate memory and slow training without improving your task.

Start with a small run: a subset of data, a few hundred steps, and a clear metric. Tune learning rate first, then effective batch size. For LoRA, check which modules you are adapting; attention-only adapters may be enough for style, while other tasks need more coverage. Log throughput, memory, and step time. If a run is 10x slower than expected, fix the pipeline before scaling.

Watch for forgetting

Fine-tuning can erode general abilities. The model may become better at your task and worse at everything else. Mix in a small amount of general instruction data or replay examples, then evaluate on a broad set. If the model forgets basic formatting or reasoning, reduce epochs, lower the learning rate, or use a smaller adapter.

A pre-flight checklist before your next run

1. Freeze a test set and store it outside your training pipeline.
2. Deduplicate and group related examples before splitting.
3. Decode a sample batch and verify prompt, completion, masks, and special tokens.
4. Run the base model on your evaluation set and record results.
5. Define the one metric that decides success.
6. Launch a short run, inspect generations, and stop if the metric declines.
7. Save checkpoints and document the data version, code commit, and hyperparameters.

Most wasted compute comes from skipping these steps, not from choosing the wrong optimizer. A small, careful run will teach you more than a large, blind one. Treat data splitting, evaluation, and inspection as part of training, because they are.

Photo: Boskampi / Pixabay