Fine-Tuning Deep Dive: LoRA Before and After with a RAG Comparison

Why Dataset Quality Matters More Than Dataset Size

A fine-tuning dataset teaches the model a pattern by example, not by instruction. If your examples are inconsistent, some warm and concise, others still corporate and verbose, the model has no clean pattern to learn, and the fine-tune will produce an inconsistent result no matter how much data you feed it.

Defining the target tone precisely

Before writing a single example, write down the tone rules explicitly, this becomes your own quality checklist for every training example.

Writing paired examples

Note

Each example includes both a generic_response and a target_response, not just the target alone. Having the contrast explicit makes it far easier to audit your own dataset for consistency, and some fine-tuning approaches can use the pair directly to sharpen the contrast the model learns.

Generating enough examples efficiently

Writing 200 examples by hand is slow. Use an LLM to generate draft pairs against your tone rules, then review and correct every single one by hand, do not skip the review step.

Common Mistake

Prioritising dataset size over quality. Two hundred inconsistent examples, some following the tone rules loosely and others not at all, teach the model a blurry, unreliable pattern. A hundred and fifty tightly consistent examples, each hand-reviewed against the explicit tone rules, teach a sharp one. Quality review time is not optional here.

Guided practice

Generate and hand-review 150 to 300 training pairs covering at least 5 distinct complaint categories. Before moving to Milestone 2, randomly sample 15 of your own examples and check each one against the 5 tone rules from earlier, if any fail, fix them now, a flawed example caught after training just means retraining.

Why LoRA Instead of Full Fine-Tuning

Full fine-tuning updates every weight in the model, which is expensive, slow, and carries a much higher risk of catastrophic forgetting since nothing constrains how far the weights can drift from their original values. LoRA (Low-Rank Adaptation) freezes the base model entirely and trains a small set of additional low-rank matrices that get added to specific layers, dramatically fewer parameters to train, much faster, and far less likely to damage the model's general capability.

Setting up the fine-tuning environment

Preparing the dataset for training

Note

The ### Customer: and ### Support: markers are a simple prompt template the model learns to associate with "read a complaint, respond in the target tone." Consistency in this template between training and later inference matters, using a different format at inference time than what the model was trained on will noticeably hurt results.

Configuring the LoRA adapter

Engineering Decision

r=8 is a deliberately modest rank for a small, focused tone-shifting task. A higher rank trains more parameters and can capture more complex behaviour changes, but also increases both training cost and the risk of overfitting on a dataset this size. Start low, only increase rank if evaluation in Milestone 3 shows the adapter genuinely underfit the pattern.

Running the training loop

Common Mistake

Skipping the base-model comparison and assuming the fine-tune worked because training loss went down. A decreasing loss curve proves the model is fitting the training data, it does not prove the resulting behaviour actually generalizes to new complaints it did not see during training, that is exactly what Milestone 3 tests.

Troubleshooting scenario

After training, the fine-tuned model's responses on new test complaints still sound generic and corporate, unchanged from the base model. Before assuming the LoRA approach failed, check three things in order: did get_peft_model actually wrap the base model correctly (print_trainable_parameters should show a small nonzero trainable percentage), did the training loss actually decrease over the 3 epochs, and is the inference code loading the saved adapter weights at all, or accidentally running the base model alone.

Proving the Fine-Tune Actually Changed Behaviour

A fine-tune that only reproduces its exact training examples well but fails on new, unseen complaints has not learned the tone pattern, it has memorized the dataset. The real test is performance on complaints the model never saw during training.

Building a held-out test set

Note

Holding out test examples entirely from training is the same discipline as the train/test split from earlier in this roadmap, applied here to fine-tuning instead of classical ML. Evaluating only on training examples would be the fine-tuning equivalent of the data leakage mistake covered in Module 2, an inflated, meaningless success signal.

Running both models on the same test set

Scoring tone adherence with a rubric

Tip

Score all 20 base-model responses and all 20 fine-tuned responses with the same rubric and the same judge prompt, then compute the average score for each. This single number, average tone score before versus after, is the clearest evidence for whether the fine-tune worked.

Concept check

Before running the scoring, predict which of the 20 held-out complaints you expect the fine-tuned model to handle best, and which you expect it to still struggle with. Complaints closest in structure to your training examples should score higher, ones covering an edge case your training data under-represented should reveal the fine-tune's actual limits, not just its average performance.

Why a Narrow Fine-Tune Can Break Unrelated Capability

Catastrophic forgetting happens when fine-tuning on a narrow task degrades the model's performance on tasks it was never meant to change. A model fine-tuned exclusively on short, warm customer-support responses might start giving oddly terse or overly casual answers even to completely unrelated questions, like a general knowledge query, because its overall response style drifted, not just the specific support-tone behaviour you targeted.

Building an unrelated-task test set

Note

These prompts are deliberately chosen to have nothing to do with the fine-tuning task. If the fine-tuned model's answers to these are noticeably worse, shorter, less coherent, or oddly styled compared to the base model's answers, that is catastrophic forgetting, the fine-tune has degraded general capability while chasing the narrow tone target.

Running the comparison

Scoring for degradation

Common Mistake

Narrow dataset causing catastrophic forgetting, then not testing for it because the fine-tune "worked" on the target task. A model that nails customer-support tone but has visibly degraded at explaining photosynthesis has a real problem, one that only shows up if you deliberately test for it, it will not appear in your tone-focused evaluation from Milestone 3 at all.

Interpreting the results

If forgetting is detected, the likely causes are training for too many epochs on too narrow a dataset, or a LoRA rank set too high for the dataset size, both push the adapter to overfit specifically to the training distribution's style at the expense of general capability. The fix is typically fewer epochs, a lower rank, or a somewhat more varied training set, not abandoning LoRA for full fine-tuning, which would carry an even higher forgetting risk for the same underlying cause.

Engineering Decision

If forgetting is detected, reduce epochs or LoRA rank and retrain before considering the adapter production-ready. A tone fine-tune that quietly damages the model's ability to handle any question outside customer support is not a net improvement, even if the target metric looks good in isolation.

Guided practice

Run the forgetting test against your own fine-tuned adapter from Milestone 2. If you detect degradation, retrain with num_train_epochs=1 instead of 3, rerun both the tone evaluation and the forgetting test, and compare all four resulting scores (tone before/after, forgetting before/after) to see the tradeoff directly.

Why RAG Genuinely Does Not Apply Here

This milestone exists to make the RAG-versus-fine-tuning decision concrete rather than theoretical. RAG solves a missing-knowledge problem: the model does not have access to a fact it needs at generation time. The tone-shifting task in this project has no missing fact, the model already knows what a refund is and what a late delivery is, the only thing wrong is how it phrases the response. Retrieving more context about refund policy would not fix a response that still opens with "We regret to inform you."

Demonstrating this directly

Note

This is functionally few-shot prompting with a tone guideline document included, dressed up as "retrieval." It is worth running this comparison explicitly, because it demonstrates the real distinction: stuffing instructions into context can nudge behaviour temporarily, but it does not durably change how the model generates by default the way a fine-tune does, especially as conversations get longer and the guideline text falls out of the effective attention window.

The three-way comparison table

Build this table from your actual Milestones 3 and this milestone's RAG-style test, using your own measured numbers, not placeholders.

Engineering Decision

Fine-tune only after confirming the problem is behaviour or tone, not missing facts. This project's three-way comparison exists to make that confirmation concrete: since no version of this task ever involved a missing fact, RAG was never a candidate solution here, and the comparison table should make that gap between prompting and fine-tuning's consistency visible in real numbers, not just assert it.

Writing the final justification

Your written justification should state, using your actual measured scores: how much more consistent the fine-tuned model was than prompting alone, whether any catastrophic forgetting was detected and how it was addressed if so, and an explicit statement of why RAG was never a viable option for this specific task, tied to the definition of what RAG actually solves.

Common Mistake

Fine-tuning to fix a knowledge gap RAG would have solved faster. This project's task deliberately has no knowledge gap, but it is worth stating the inverse clearly in your write-up: if the underlying problem here had instead been "the bot gives outdated refund policy numbers," fine-tuning would have been the wrong tool entirely, that is a RAG problem, and no amount of retraining fixes a policy document that changed after training data was collected.

Capstone lab checkpoint

Complete the three-way comparison table with your real measured data, then write a one-page justification following the structure above. State explicitly what would have to change about this task for RAG or prompting alone to become the better choice instead of fine-tuning, this is what proves you understand the decision, not just the LoRA mechanics.

Final Verification

1. Dataset quality check

Sample 15 training examples at random and confirm each one follows all 5 tone rules from Milestone 1 exactly, no exceptions.

2. Training completed successfully

3. Tone improvement is measurable

Confirm the average tone score (0-5 rubric) on the 20 held-out test complaints is meaningfully higher for the fine-tuned model than for the base model, not just marginally different within noise.

4. Forgetting test passed

Confirm the fine-tuned model's answers to the 10 unrelated general-knowledge prompts show no judge-flagged degradation compared to the base model. If degradation was detected and retrained away, confirm the retrained adapter passes this check.

5. Comparison document is complete

Confirm the three-way comparison table has real measured numbers in every cell, and the written justification explicitly states why RAG does not apply to this specific task.

Quick reference

Common mistakes across the full project

Fine-tuning to fix a knowledge gap RAG would have solved faster is the single most important mistake this project's Milestone 5 exists to prevent, and the fix is confirming explicitly, as this project does, that the actual problem is behaviour or tone, not a missing fact. Prioritising dataset size over quality produces a blurry, unreliable learned pattern, and the fix is hand-reviewing every training example against explicit tone rules before it enters the dataset. Skipping the base-model comparison and trusting a decreasing training loss curve alone risks shipping an adapter that only memorized its training examples, and the fix is always evaluating on a genuinely held-out test set. Narrow datasets causing catastrophic forgetting without anyone testing for it lets a real regression ship silently, and the fix is a dedicated unrelated-task test every time, not only when something seems off. Defaulting to full fine-tuning when LoRA achieves the same result more cheaply wastes compute and increases forgetting risk for no proportional benefit, and the fix is starting with a modest LoRA rank and only escalating if evaluation shows genuine underfitting.

Tip

Keep both the base model comparison and the forgetting test results in your portfolio alongside the working adapter. A fine-tune with no evaluation evidence looks identical to one that was never properly tested, the numbers are what make this project credible to a reviewer.