A team spends three weeks fine-tuning a model on their support ticket history. The model now writes in exactly the right tone. Unfortunately, the team also expected the fine-tuned model to stay grounded in current refund policies, and it confidently invents policies that were never true, because tone was the only thing that actually needed to change, and the facts should have come from RAG instead. The three weeks were mostly wasted on the wrong tool, and the fine-tuned model now needs a second project layered on top just to stay factually grounded. This module exists to prevent that mistake, not to make you a fine-tuning specialist. As an AI engineer, fine-tuning is a tool you reach for occasionally, after RAG and prompting have already been tried and specifically failed to fix a **behavior** problem, not a knowledge problem. That is a narrower skill than training a model from scratch or running distributed multi-GPU training jobs, and this module is scoped to match, working understanding of LoRA and PEFT, not research-level mastery. > 📌 **Remember:** RAG changes what the model knows at request time. Fine-tuning changes how the model behaves, permanently, in its weights. Confusing the two is the single most expensive mistake in this space. ### Where this fits against RAG The RAG module covered giving a model access to facts it does not have. This module covers something narrower and rarer: changing the model's tone, output format, or task-specific behavior when prompting and RAG have already been tried and genuinely cannot fix it. If you have not read the RAG module yet, or have not already ruled out a prompting fix, that is the correct place to start, not here. ---
### The default assumption: you probably don't need this yet Most problems that look like they need fine-tuning are actually prompting problems or retrieval problems wearing a disguise. A model giving inconsistent formatting is often solved by a stricter system prompt and a structured output schema. A model missing facts is a RAG problem, not a fine-tuning problem, since fine-tuning bakes facts into weights that go stale the moment the underlying information changes, while RAG can be updated by editing a document. Before considering fine-tuning, the honest checklist is: has this been tried with a clearer system prompt, with few-shot examples in the prompt itself, and with retrieval if the issue is missing or outdated information? Fine-tuning is worth considering only once all three have been genuinely attempted and specifically fell short on **behavior**, not facts. ### The one distinction that decides everything | Symptom | Root Cause | Right Tool | |:---|:---|:---| | Model doesn't know your refund policy | Missing knowledge | RAG | | Model knows the policy but writes it too casually for your brand | Wrong behavior/tone | Fine-tuning | | Model's JSON output is inconsistently formatted | Weak prompt structure | Better prompt + schema | | Model ignores your company's specific internal jargon consistently, even when told | Wrong behavior, resistant to prompting | Fine-tuning | | Model gives outdated pricing information | Missing/stale knowledge | RAG | What problem are you actually solving? | +-----+-----------------------+ | | Missing or changing Output behavior is knowledge? wrong (tone/format/style)? | | v v RAG Does a better prompt or schema fix it? | +-----+-----+ | | Yes No, still fails | | v v Don't fine-tune Consider fine-tuning > 📌 **Engineering Decision:** Reach for fine-tuning only after confirming, specifically, that the problem is the model's *behavior* (tone, format, task-specific style) rather than *missing facts*, and only after prompting and retrieval have already been tried and did not fix it. Fine-tuning is not a stronger version of prompting, it solves a different category of problem entirely. ---
**Fine-tuning** is continuing the training of a pre-trained model on a smaller, task-specific dataset so its weights shift toward a narrower behavior. Fine-tuning can encode information into model weights, but it is usually a poor choice for maintaining authoritative or frequently changing knowledge. RAG keeps knowledge external and updateable, while fine-tuning is better suited to changing behavior such as tone, formatting, style, or task-specific response patterns. A useful mental model: pretraining is like a person's entire general education, fine-tuning is like a focused, intensive on-the-job training course layered on top. The person doesn't forget how to read or reason, but their habits, vocabulary, and default responses shift toward the specific job they were just trained for. Push that training too hard on too narrow a course, and they can start forgetting skills from before the course, this is exactly the catastrophic forgetting risk covered later in this module. The broader term "fine-tuning" covers several techniques, including preference optimization and continued pretraining, but the workflow this module teaches is **supervised fine-tuning**, unlike a model's original pretraining, which happens on vast unlabeled text, supervised fine-tuning uses a dataset of labeled examples, most often prompt-response pairs, and updates the model's weights so its outputs move closer to those labeled examples. ```text Pretraining (unsupervised, on vast unlabeled text) | v Base model (broad language understanding) | v Fine-tuning (supervised, on a small task-specific dataset) | v Fine-tuned model (same broad knowledge, shifted behavior) ``` ### Full fine-tuning versus PEFT, the core tradeoff **Full fine-tuning** updates every single weight in the model. It is the most expressive option, capable of the deepest behavioral change, but it requires enough GPU memory to store gradients, optimizer states, and activations for the entire model simultaneously, the same order of resource demand as the original pretraining run. For any model past a few billion parameters, this puts full fine-tuning out of reach for most teams without dedicated infrastructure. **Parameter-Efficient Fine-Tuning (PEFT)** freezes almost all of the original model's weights and trains only a small number of additional parameters. This is dramatically cheaper in memory and compute, and it carries a real secondary benefit: because the pretrained base weights remain frozen, PEFT can reduce some forms of behavioral regression on unrelated tasks. It does not eliminate that risk entirely, the resulting adapter can still distort behavior when combined with the base model, so forgetting must still be evaluated explicitly rather than assumed away. | Factor | Full Fine-Tuning | PEFT (LoRA) | |:---|:---|:---| | Weights updated | All of them | A small added subset | | Base weights | Updated | Frozen | | GPU memory needed | Very high, similar to pretraining | A fraction of full fine-tuning | | Regression risk | Generally greater capacity for broad behavioral change | Generally constrained, but still possible, always verify | | Storage per task | A full model copy per task | A small adapter file per task | | Typical AI engineer use case | Rare, deep behavioral overhaul with ample data | The default starting point | > 📌 **Engineering Decision:** Default to a PEFT method, specifically LoRA, for nearly every real fine-tuning case an AI engineer encounters. Full fine-tuning is reserved for situations demanding deep behavioral change backed by a large, high-quality dataset and real infrastructure budget, not the everyday case. ---
### Why LoRA specifically **LoRA (Low-Rank Adaptation)** is one of the most widely used PEFT methods and a strong default for many LLM fine-tuning workloads, and the reason it works is worth understanding in plain terms even without the underlying linear algebra. Instead of updating a model's full weight matrices directly, LoRA freezes them completely and represents the weight update using a low-rank decomposition, training a much smaller set of parameters while the original weights never move. Think of it like this: instead of repainting an entire building to change its color scheme, you attach a smaller, removable overlay panel that achieves the same visual effect. The building underneath is untouched, the overlay is what actually changed, and you can swap in a different overlay for a different look without repainting anything again. After training, the result is an unchanged base model plus a small **LoRA adapter**. The adapter can be dramatically smaller than a complete model copy, often representing well under 1% of the base model's parameters, depending on the rank and target modules, megabytes rather than gigabytes. At inference time, the adapter is combined with the frozen base model to produce the fine-tuned behavior. ```text Base model (frozen, unchanged) + LoRA adapter (small, trained) | v Combined fine-tuned behavior at inference ``` > **Note:** Because the base model never changes, one base model can support many different LoRA adapters, one per task or use case, swapped in and out at inference time instead of storing a full separate model copy for each task. This is a major practical advantage once a team has more than one fine-tuning use case. ### QLoRA, the memory-lean variant **QLoRA** takes LoRA further by keeping the frozen base model quantized, commonly to 4-bit precision, while training the LoRA adapter parameters themselves in higher precision on top of it, the adapter is not quantized just because the base model is. This shrinks the memory required to hold the base model during training enough to make fine-tuning a model that would otherwise need a multi-GPU cluster feasible on a single consumer or cloud GPU. ```python from peft import LoraConfig, TaskType ## r controls how expressive the adapter is - higher r means more ## trainable parameters and more capacity to change behavior, at ## the cost of a larger adapter and higher overfitting risk on ## small datasets. Start small and increase only if needed. lora_config = LoraConfig( task_type=TaskType.CAUSAL_LM, r=8, # rank of the adapter matrices lora_alpha=32, # scaling factor applied to the adapter's output lora_dropout=0.1, # regularization to reduce overfitting on small datasets target_modules=["q_proj", "v_proj"], # which layers get an adapter ) ``` > **Note:** `r` (rank) is the single most important LoRA setting to understand conceptually. A higher rank gives the adapter more capacity to represent complex behavioral changes, but also more parameters to train and a higher risk of overfitting on a small dataset. There is no universal correct value, it is a tuning parameter to sweep across a small range for your specific task, not a constant to copy from a tutorial. > **Note:** `target_modules` determines which model layers receive a LoRA adapter. The correct names depend on the model's architecture, `q_proj` and `v_proj` are common on many transformer implementations but not universal, inspect the model's actual module names rather than blindly copying a target list written for a different model. ---
### What it is and why PEFT reduces it **Catastrophic forgetting** is what happens when a model, pushed too hard on a narrow fine-tuning dataset, loses some of its ability to perform well on tasks outside that narrow focus. A model fine-tuned aggressively on refund-ticket summaries might get noticeably worse at general reasoning or at tasks completely unrelated to refunds, even though nobody intended to touch that capability. This risk is highest with full fine-tuning, since every weight is free to move, including weights that were quietly responsible for capabilities the fine-tuning dataset never exercised. PEFT methods like LoRA reduce this risk structurally, because the original weights stay frozen, only the small adapter moves, so the model's broad capabilities are far less likely to be disturbed by training on a narrow task. > 🔴 **Common Mistake:** Fine-tuning on a narrow, high-volume dataset (many examples of one specific task) and only checking that the target task improved, without testing whether general capabilities degraded. A model that scores brilliantly on the fine-tuning task but has quietly lost ground elsewhere is a regression, not a win, and it will surface in production in ways the fine-tuning evaluation never caught. ### How to actually check for it Catastrophic forgetting is not something you notice by accident, it has to be deliberately tested for, using prompts that have nothing to do with the fine-tuning task. ```python ## After fine-tuning, test with prompts unrelated to the fine-tuning ## task itself. If the fine-tuned model performs noticeably worse ## than the base model on these, that is a forgetting signal. unrelated_test_prompts = [ "Explain the difference between TCP and UDP in one paragraph.", "Write a short haiku about autumn.", "What is the capital of Australia?", ] for prompt in unrelated_test_prompts: base_output = base_model.generate(prompt) finetuned_output = finetuned_model.generate(prompt) # Compare quality manually or with an LLM-as-judge pattern, # covered in the Evaluation and Testing module, on both outputs ``` > 💡 **Practice:** Always keep a small, fixed set of general-capability prompts, unrelated to the fine-tuning task, run against both the base model and every fine-tuned checkpoint. This is the single cheapest, most reliable catastrophic forgetting check available, and it takes minutes to run. ---
This section shows the shape of a real workflow, not a from-scratch training loop. The point is recognizing each stage and what it is for, since the exact library calls shift with every framework version. 1. Select a pre-trained base model suited to the task | v 2. Prepare a small, high-quality labeled dataset (prompt-response pairs, consistently formatted) | v 3. Configure a LoRA adapter (rank, target layers, dropout) | v 4. Train the adapter only, base model weights stay frozen | v 5. Evaluate on the target task AND on unrelated prompts (catastrophic forgetting check) | v 6. Compare against the base model, decide if the gain justifies keeping the adapter in production ```python from peft import LoraConfig, get_peft_model, TaskType ## Step 3 - configure the adapter on top of an already-loaded base model lora_config = LoraConfig( task_type=TaskType.CAUSAL_LM, r=8, lora_alpha=32, lora_dropout=0.1, ) peft_model = get_peft_model(base_model, lora_config) ## Step 4 - only the adapter's parameters are trainable, the base ## model's original weights are frozen and untouched throughout trainable_params = sum(p.numel() for p in peft_model.parameters() if p.requires_grad) total_params = sum(p.numel() for p in peft_model.parameters()) print(f"Training {trainable_params:,} of {total_params:,} parameters " f"({100 * trainable_params / total_params:.2f}%)") ``` > **Note:** Depending on the model architecture, target modules, and rank, LoRA can train well below 1% of the base model's parameters. Treat the printed trainable-parameter percentage as a sanity check rather than a fixed target, if it looks close to 100 percent, the adapter configuration is likely wrong and you are accidentally close to full fine-tuning instead. ### Learning rate and training intensity, at a conceptual level Two settings matter more than any others once the adapter is configured correctly, and an AI engineer needs to recognize them even without running distributed training. **Learning rate** controls how aggressively the adapter's weights change with each training step, too high and training becomes unstable or overwrites useful behavior too fast, too low and training barely moves at all. An **epoch** is one complete pass through the training dataset, more epochs means more exposure to the same examples, which helps up to a point and then starts hurting. > 🔴 **Common Mistake:** Training for more epochs "to be safe," assuming more training time can only help. Past a certain point, additional epochs on a small dataset push the adapter toward memorizing the specific training examples rather than learning the general behavior, which is the direct mechanism behind the overfitting covered next. ### Overfitting during fine-tuning **Overfitting** happens when the adapter learns the training examples too specifically rather than learning the underlying behavior they were meant to demonstrate. The telltale sign is training performance that keeps improving while performance on examples the adapter has never seen stops improving or gets worse, which is exactly why the next section on keeping a held-out evaluation set matters so much here. The main levers that push toward or away from overfitting are the same ones already introduced: a lower learning rate and fewer epochs reduce overfitting risk, a smaller or less diverse dataset increases it, and a higher adapter rank gives the model more capacity to memorize rather than generalize. None of these has one correct setting, they are tuning parameters to check against a held-out set, not constants to set once. ### Keeping Evaluation Data Separate Never judge a fine-tune only on examples it saw during training, that measures memorization, not the behavior you actually wanted. Split your curated dataset into a training set and a held-out evaluation set before training starts, the training set teaches the behavior, the held-out set tests whether that behavior actually generalized to inputs the adapter has never encountered. Curated dataset | +-----+-----+ | | Training set Held-out evaluation set (teaches the (tests whether the behavior) behavior generalized) > 💡 **Practice:** For the lab in this module, a roughly 80/20 split between training and held-out evaluation examples is a reasonable starting point, not a universal rule. Keep this held-out set separate from the unrelated-capability regression prompts used for the catastrophic forgetting check, they are testing two different things. ### Treat adapters as versioned artifacts A LoRA adapter is not useful on its own, it only works paired with the exact base model it was trained against. A production fine-tuned system is really the combination of a base model version, an adapter, the adapter's configuration, the training dataset version, the hyperparameters used, and the evaluation results that justified shipping it, recorded together as one reproducible unit. > 📌 **Remember:** An adapter without its exact base model, configuration, and evaluation results attached is not a reproducible deployment artifact, it is a file that happens to work today. Record all five together the same way the LLMOps module treats any other deployable model artifact. ### A note on model licensing Before fine-tuning or deploying an open-weight model, verify its license actually permits your intended use, redistribution, and commercial deployment. Model weights, training datasets, and the adapters you produce from them can each carry separate licensing constraints, checking once at the start avoids finding out the hard way later. ### Data quality over data volume A smaller dataset of carefully checked, consistently formatted examples reliably outperforms a larger dataset full of noisy or inconsistent ones. This matters more for fine-tuning than almost any other AI engineering task, because every low-quality example is actively teaching the model the wrong pattern, not just failing to help. ```python ## GOOD - consistent structure, clear instruction, clean response { "prompt": "Summarize this refund request in one sentence: [ticket text]", "response": "Customer requests a refund for a delayed order." } ## BAD - inconsistent formatting, response includes irrelevant meta-text { "prompt": "hey can u summarize this ticket thing", "response": "Sure! Here's a summary: The customer wants a refund I think, based on what they said." } ``` > 🔴 **Common Mistake:** Assembling a fine-tuning dataset from whatever historical text is easiest to gather, without checking formatting consistency or filtering low-quality examples first. Training format should mirror production format exactly, including any system prompt structure, since a mismatch between how the model was trained and how it is actually called at inference time is a common and hard-to-diagnose source of quality loss. ---
A team spends three weeks fine-tuning a model on their support ticket history. The model now writes in exactly the right...
The default assumption: you probably don't need this yet Most problems that look like they need fine-tuning are actually...
Fine-tuning is continuing the training of a pre-trained model on a smaller, task-specific dataset so its weights shift t...
Why LoRA specifically LoRA (Low-Rank Adaptation) is one of the most widely used PEFT methods and a strong default for ma...
What it is and why PEFT reduces it Catastrophic forgetting is what happens when a model, pushed too hard on a narrow fin...
This section shows the shape of a real workflow, not a from-scratch training loop. The point is recognizing each stage a...
Concept checks A team's chatbot has the right facts but writes in an overly formal tone that does not match their brand ...
RAG vs fine-tuning Question RAG Fine-Tuning Model is missing facts or has outdated knowledge Yes, this is the fix No, we...
Fine-tuning to fix a knowledge gap that RAG would have solved faster and more cheaply is the most common misdiagnosis in...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.