You've built a proof-of-concept chatbot. It answers questions about your company's products reasonably well — until it hallucinates a feature that doesn't exist, forgets the pricing structure you told it about three prompts ago, and then confidently explains your refund policy using terms from a competitor's website. Your manager asks: "Should we fine-tune the model?" Your ML engineer says, "We need RAG." Your prompt engineer insists the system prompt just needs to be better.
They're all partially right, and that's the problem. The landscape of LLM customization has three major techniques — prompt engineering, retrieval-augmented generation (RAG), and fine-tuning — and in production, picking the wrong one (or failing to combine them correctly) costs you months of engineering time, tens of thousands of dollars, and a system that still doesn't behave the way you need it to. This lesson cuts through the confusion by teaching you the decision mechanics behind each approach, their internal architectures, their failure modes, and critically, how to layer them together into a coherent production system.
By the end of this lesson, you will understand not just what each technique does, but why it works at the level of model internals, data flow, and system architecture — so you can make confident design decisions without needing to run an expensive experiment every time.
What you'll learn:
This lesson assumes you are comfortable with:
You don't need a deep background in machine learning theory, but you should understand what a token is, roughly what attention mechanisms do, and why context windows are finite.
Before we compare them, we need to be precise about what each technique is doing mechanically. This is where most tutorials go wrong — they describe the techniques at a surface level and then wonder why developers make poor decisions.
Prompt engineering does not change the model. It changes what the model receives as input, and through that, steers which outputs become probable. A transformer language model generates tokens by computing probability distributions over its vocabulary at each step, conditioned on everything in the context window. When you write a better system prompt, you are not teaching the model anything — you are providing context that activates patterns the model already learned during pretraining.
This is a subtle but crucial distinction. When you tell a model "You are an expert SQL analyst. Always explain your reasoning before writing a query," you're not installing new SQL knowledge. You're placing tokens in the context that statistically correlate with the kind of careful, explanatory SQL generation you want to see. The model already associated those tokens with that behavior during training.
The ceiling of prompt engineering is therefore the ceiling of what the model already knows and can do. You can elicit better performance on tasks the model understands. You cannot teach it your internal proprietary terminology, your specific business logic, or facts that postdate its training cutoff — not reliably, anyway.
Retrieval-augmented generation is, at its core, still prompt engineering — but with a dynamic information retrieval step prepended. The model's weights don't change. What changes is that you're injecting relevant, up-to-date, specific documents into the context before asking the model to respond.
The pipeline works like this: a user query arrives, it's converted to an embedding vector, that vector is used to query a vector database for semantically similar document chunks, and those chunks are stuffed into the prompt alongside the original question. The model then uses those chunks as grounding material to generate its response.
What RAG actually solves is the knowledge gap problem — specifically, the gap between what the model learned at training time and what you need it to know right now. It does not change the model's reasoning patterns, its style, its output format, or its understanding of implicit domain conventions. It just gives it better raw material to work with.
The failure mode that trips people up: RAG can retrieve relevant documents and still produce wrong answers if the model doesn't know how to interpret those documents correctly. Inject a 10-K SEC filing into a prompt and ask a general-purpose model to extract adjusted EBITDA with specific calculation methodology — it will try, and it will probably get it wrong, not because the information wasn't there, but because the model doesn't understand your specific accounting conventions.
Fine-tuning is the only technique that actually modifies the model. During fine-tuning, you take a pretrained model and continue training it on a curated dataset of examples, updating the weights through gradient descent. This changes the probability distributions the model has over output tokens — permanently, for that model version.
Fine-tuning can teach the model things that cannot be communicated through context alone: consistent output formats across arbitrary inputs, nuanced domain conventions that are difficult to articulate in prose, a specific voice or reasoning style, and — critically — behaviors that need to generalize to inputs you haven't anticipated.
But fine-tuning has a peculiar failure mode that's worth understanding deeply: it doesn't add facts to the model the way a database stores records. When you fine-tune on examples that reference specific product names, prices, or policies, the model is learning statistical associations between tokens. This works well for teaching behavior and style. It works poorly for teaching precise facts, especially when those facts might change. A fine-tuned model that learned your Q1 pricing structure is not maintaining a price database — it's learned that certain tokens appear in certain positions, and when prices change in Q2, you need to retrain. This is why fine-tuning is often the wrong answer to "the model doesn't know our product catalog."
Rather than starting with the techniques and asking "which one should I use," start with your problem and ask these four questions in order. The answers will guide you to the right approach — or the right combination.
This is the most important distinction in the entire decision process.
A knowledge problem is when the model doesn't have access to the right information: your proprietary documents, recent events, customer-specific data, internal wikis, or anything that isn't in its training set. The solution space is RAG (for dynamic, large, or frequently updated information) or fine-tuning (for stable, deeply embedded domain knowledge that's universal to your use case).
A behavior problem is when the model has the underlying capability but isn't expressing it correctly: wrong output format, wrong tone, inconsistent reasoning style, failing to follow a specific workflow, or not applying domain conventions reliably. The solution space is prompt engineering (for simpler behavior changes) or fine-tuning (for complex, consistent behaviors that need to generalize robustly).
In practice, most production systems have both types of problems, which is why combining techniques is the norm rather than the exception. But separating them diagnostically is essential — it prevents you from fine-tuning to solve a knowledge problem (expensive, brittle, won't scale) or using RAG to solve a behavior problem (won't work at all).
If the information is static or changes slowly (once a quarter, once a year), fine-tuning it in is viable. If it changes daily, weekly, or per-user, fine-tuning is the wrong tool — you cannot retrain a model every time a product price changes. RAG shines here because you're querying a database that can be updated in real time.
The corollary: fine-tuning is not for facts, it's for patterns. Use it to teach the model how to reason about a domain, not what the current state of that domain is.
Some applications need the model to behave exactly the same way across thousands of diverse inputs — not "mostly correct" but reliably correct in a way that can be audited and depended upon. Contract analysis, clinical decision support, financial report generation, code generation in a proprietary framework — these applications need a model that has internalized the conventions of the domain deeply enough to apply them consistently without being reminded every time.
This is fine-tuning territory. Prompt engineering can achieve consistency to a point, but long, complex system prompts add latency, consume context window space, and can be "diluted" by a very long conversation or a very long retrieved document. A fine-tuned model has those conventions baked into its weights.
Fine-tuning produces a custom model. Serving that model requires either self-hosting (GPU infrastructure costs, engineering overhead) or paying for a fine-tuned model hosting tier (OpenAI's fine-tuned model endpoints are priced at a premium). RAG adds a retrieval step — typically 50 to 200ms for the vector search, plus additional latency for any re-ranking or processing you do on results. It also adds infrastructure: a vector database, an embedding service, a chunking and indexing pipeline. Prompt engineering has the lowest overhead but doesn't scale well for all use cases.
Maintenance is often the hidden cost. RAG requires you to maintain an up-to-date, well-chunked document index. Fine-tuning requires you to maintain a training dataset and retrain when things change. Complex system prompts require careful versioning and regression testing.
There's no free lunch here. Map your constraints honestly before choosing.
Assuming you've diagnosed a behavior problem that doesn't require fine-tuning, or a knowledge problem where the information fits within the context window — let's talk about what expert-level prompt engineering actually looks like.
Most developers write system prompts as a list of instructions. This works until the conversation gets long, the user's message is very detailed, or the retrieved documents are bulky — at which point the model starts "forgetting" earlier instructions. The reason is attention: later tokens in the context have stronger influence on generation than earlier ones.
Counter this with two techniques. First, put your most critical constraints at the end of the system prompt, not the beginning — closer to where generation starts. Second, use a "constraint reinforcement" pattern where key behavioral rules appear both in the system prompt and as a brief reminder in the final human turn, injected programmatically.
def build_prompt(user_query: str, retrieved_docs: list[str], persona: str) -> list[dict]:
"""
Build a prompt that keeps critical constraints close to the generation point.
"""
system_message = f"""
You are {persona}, a financial analysis assistant for Meridian Capital.
Your domain expertise:
- You analyze SEC filings, earnings transcripts, and market data
- You know Meridian's internal sector taxonomy (Technology, Healthcare, Industrials, Consumer)
- You express uncertainty explicitly using phrases like "based on available data" or "this is my interpretation"
Output format:
- Always structure responses with: Summary | Key Metrics | Risk Flags | Recommendation
- Use Meridian's standard risk scale: LOW / MEDIUM / HIGH / CRITICAL
- Cite your sources by document name and section
HARD CONSTRAINTS (never violate):
- Never make specific price predictions
- Never recommend specific securities to buy or sell
- Always note when data is older than 90 days
"""
# Retrieved context goes in the middle
context_block = "\n\n---\n\n".join([f"Document: {doc}" for doc in retrieved_docs])
# The constraint reminder goes right before generation
constraint_reminder = """
Remember: Use the structured format (Summary | Key Metrics | Risk Flags | Recommendation),
cite your sources, and flag any data older than 90 days.
"""
return [
{"role": "system", "content": system_message},
{"role": "user", "content": f"Context:\n{context_block}\n\nQuery: {user_query}\n\n{constraint_reminder}"}
]
Few-shot examples are among the most powerful prompt engineering tools, and they're dramatically underused in production. The key is that examples should be representative of edge cases, not just happy paths. If you only show the model correct examples of easy inputs, it won't know how to handle ambiguous or adversarial ones.
Structure few-shot examples to demonstrate the decision process, not just the output:
FEW_SHOT_EXAMPLES = """
Example 1 — Clear case:
Query: "What was Apex Corp's revenue growth in Q3 2023?"
Context: [Q3 2023 earnings transcript showing 12.4% YoY growth]
Response:
Summary: Apex Corp reported strong Q3 2023 revenue growth.
Key Metrics: Revenue growth YoY: 12.4% | Absolute revenue: $2.3B | Beat consensus by 1.8%
Risk Flags: MEDIUM — Growth was partially driven by one-time licensing deal ($180M),
organic growth was 4.7%.
Recommendation: Monitor Q4 for normalization of growth rate.
Sources: Q3 2023 Earnings Transcript, p.4
Example 2 — Ambiguous case (show how to handle uncertainty):
Query: "Is Apex Corp's debt load sustainable?"
Context: [Balance sheet showing $4.2B debt, partial cash flow data, no industry comparison]
Response:
Summary: Based on available data, Apex Corp's debt load appears manageable but a complete
assessment requires additional context.
Key Metrics: Total debt: $4.2B | Available cash flow data is incomplete
Risk Flags: HIGH — Insufficient data to complete debt serviceability analysis.
Missing: full FCF data, covenant details, maturity schedule.
Recommendation: Request complete debt schedule before completing assessment.
Sources: Q3 2023 Balance Sheet (data as of Sept 30, 2023 — within 90-day window)
"""
Tip: Version your few-shot examples alongside your prompts. When model behavior regresses, the first thing to check is whether an example has become misleading relative to your current use case.
A production RAG system has more moving parts than a prototype, and the gaps between them are where quality goes to die.
Most RAG tutorials chunk documents by character count or token count. In production, this is almost always wrong. Consider a legal contract: naive chunking will split a clause definition from the clause it defines, separating context that belongs together. The retrieval system will then pull half-clauses and the model will misinterpret them.
Semantic chunking — splitting on natural boundaries like paragraphs, sections, or logical units — dramatically improves retrieval quality. For structured documents, use document-aware chunking:
import re
from dataclasses import dataclass
from typing import Optional
@dataclass
class DocumentChunk:
content: str
source_document: str
section_title: Optional[str]
chunk_index: int
metadata: dict
def semantic_chunk_document(
text: str,
source_name: str,
max_chunk_tokens: int = 512,
overlap_tokens: int = 64
) -> list[DocumentChunk]:
"""
Chunk a document on section boundaries first, then by token limit.
Preserves section context in each chunk's metadata.
"""
# Split on headers (Markdown or common patterns)
section_pattern = re.compile(r'^(#{1,3}[^\n]+|[A-Z][A-Z\s]{5,}:?)\s*$', re.MULTILINE)
sections = section_pattern.split(text)
chunks = []
current_section = "Introduction"
chunk_index = 0
for segment in sections:
segment = segment.strip()
if not segment:
continue
# Detect if this segment is a section header
if section_pattern.match(segment):
current_section = segment.strip('#').strip()
continue
# Further split long sections by paragraph
paragraphs = segment.split('\n\n')
buffer = []
buffer_tokens = 0
for para in paragraphs:
para_tokens = len(para.split()) * 1.3 # rough token estimate
if buffer_tokens + para_tokens > max_chunk_tokens and buffer:
chunk_text = '\n\n'.join(buffer)
chunks.append(DocumentChunk(
content=chunk_text,
source_document=source_name,
section_title=current_section,
chunk_index=chunk_index,
metadata={"section": current_section, "source": source_name}
))
chunk_index += 1
# Overlap: keep last paragraph for context continuity
buffer = buffer[-1:] if overlap_tokens > 0 else []
buffer_tokens = len(buffer[0].split()) * 1.3 if buffer else 0
buffer.append(para)
buffer_tokens += para_tokens
if buffer:
chunks.append(DocumentChunk(
content='\n\n'.join(buffer),
source_document=source_name,
section_title=current_section,
chunk_index=chunk_index,
metadata={"section": current_section, "source": source_name}
))
chunk_index += 1
return chunks
Vanilla vector search retrieves chunks that are semantically similar to the query embedding. This works reasonably well but has two systematic failure modes:
Lexical mismatch: The user asks about "headcount reduction" but your documents consistently use "workforce optimization." Embeddings help here, but not perfectly. Hybrid retrieval — combining dense vector search with sparse keyword search (BM25) — fills this gap:
from rank_bm25 import BM25Okapi
import numpy as np
class HybridRetriever:
def __init__(self, chunks: list[DocumentChunk], embedding_model, vector_store):
self.chunks = chunks
self.embedding_model = embedding_model
self.vector_store = vector_store
# Build BM25 index over chunk content
tokenized_corpus = [chunk.content.lower().split() for chunk in chunks]
self.bm25 = BM25Okapi(tokenized_corpus)
def retrieve(
self,
query: str,
top_k: int = 10,
alpha: float = 0.6 # weight for semantic search vs. keyword
) -> list[DocumentChunk]:
"""
Hybrid retrieval: combine semantic and keyword search scores.
alpha=1.0 is pure semantic, alpha=0.0 is pure BM25.
"""
# Semantic retrieval
query_embedding = self.embedding_model.encode(query)
semantic_results = self.vector_store.query(query_embedding, top_k=top_k * 2)
semantic_scores = {r.chunk_index: r.score for r in semantic_results}
# BM25 keyword retrieval
bm25_scores_raw = self.bm25.get_scores(query.lower().split())
# Normalize BM25 scores to [0, 1]
bm25_max = bm25_scores_raw.max() or 1
bm25_scores = bm25_scores_raw / bm25_max
# Combine scores
combined = {}
all_indices = set(semantic_scores.keys()) | set(range(len(self.chunks)))
for idx in all_indices:
sem_score = semantic_scores.get(idx, 0.0)
kw_score = float(bm25_scores[idx]) if idx < len(bm25_scores) else 0.0
combined[idx] = alpha * sem_score + (1 - alpha) * kw_score
# Sort and return top-k
top_indices = sorted(combined.keys(), key=lambda i: combined[i], reverse=True)[:top_k]
return [self.chunks[i] for i in top_indices]
Missing parent context: A retrieved chunk mentions "the clause referenced in Section 4.2" but Section 4.2 is in a different chunk that wasn't retrieved. The "parent document" pattern addresses this by including the source section header and a brief summary of the broader document in every chunk's metadata, injected into the prompt alongside the content.
After retrieval, re-ranking with a cross-encoder model dramatically improves precision. Unlike bi-encoder embeddings (which encode query and document independently), a cross-encoder processes the query and document together, giving much richer relevance judgments at the cost of more compute:
from sentence_transformers import CrossEncoder
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
def rerank_chunks(
query: str,
candidates: list[DocumentChunk],
top_n: int = 5
) -> list[DocumentChunk]:
"""
Re-rank retrieved candidates using a cross-encoder.
Use after initial retrieval to improve precision.
"""
pairs = [(query, chunk.content) for chunk in candidates]
scores = reranker.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
return [chunk for chunk, score in ranked[:top_n]]
Warning: Re-ranking adds 50–150ms of latency per request depending on model size and the number of candidates. Profile this in your specific infrastructure before committing to it in every request path. For lower-latency requirements, consider caching re-ranked results for common query patterns.
Fine-tuning is justified when you have a genuine behavior problem that prompt engineering cannot reliably solve, or when you need to dramatically reduce prompt length (and thus cost and latency) by baking instructions into the model.
The strongest cases for fine-tuning:
1. Consistent specialized output formats: You need the model to output structured JSON matching a complex schema with nested objects, optional fields, and domain-specific validation rules — every single time, regardless of input complexity. Prompt engineering gets you to maybe 90-95% reliability. Fine-tuning gets you to 99%+.
2. Domain-specific reasoning patterns: A medical coding assistant needs to apply ICD-10 classification logic systematically. A legal contract reviewer needs to identify specific clause types using a firm's internal taxonomy. These aren't just output format issues — they're reasoning patterns that need to be internalized.
3. Cost and latency reduction at scale: A 4,000-token system prompt at 10 requests per second is expensive and slow. If you can encode those instructions into the model's weights, your effective prompt drops by thousands of tokens.
4. Instruction following in a specific voice: Brand voice is notoriously hard to maintain through prompting alone, especially across diverse inputs. Fine-tuning on exemplary outputs is the most reliable path to consistent voice.
The quality of your training data is far more important than the quantity. A common mistake is collecting hundreds of examples of simple, easy cases. What you actually need is:
# Structure of a fine-tuning dataset for OpenAI format
# Each example demonstrates the full desired behavior
training_examples = [
{
"messages": [
{
"role": "system",
"content": "You are a clinical documentation specialist. Extract structured data from clinical notes and output valid JSON matching the ClinicalNote schema."
},
{
"role": "user",
"content": """Extract structured data from this clinical note:
Patient presents with acute onset chest pain, 8/10 intensity, radiating to left arm,
onset 2 hours ago. PMH: HTN, Type 2 DM on metformin. BP 158/94, HR 102, RR 18.
EKG shows ST elevation in leads II, III, aVF. Troponin pending. Impression:
Probable inferior STEMI. Plan: Activate cath lab, aspirin 325mg, heparin bolus."""
},
{
"role": "assistant",
"content": """{
"chief_complaint": "acute onset chest pain",
"symptom_severity": "8/10",
"symptom_onset_hours": 2,
"radiation_pattern": "left arm",
"vital_signs": {
"blood_pressure_systolic": 158,
"blood_pressure_diastolic": 94,
"heart_rate": 102,
"respiratory_rate": 18
},
"past_medical_history": ["hypertension", "type_2_diabetes_mellitus"],
"current_medications": ["metformin"],
"diagnostic_findings": {
"ekg": "ST elevation in leads II, III, aVF",
"troponin": "pending"
},
"impression": "probable_inferior_STEMI",
"icd10_codes": ["I21.19"],
"interventions_ordered": ["cath_lab_activation", "aspirin_325mg", "heparin_bolus"],
"data_completeness": "partial",
"missing_fields": ["troponin_value", "patient_age", "patient_sex"]
}"""
}
]
}
# ... many more examples
]
Full fine-tuning of a large model requires significant GPU memory and compute. Low-Rank Adaptation (LoRA) addresses this by freezing the pretrained model weights and injecting small trainable rank decomposition matrices into each layer. In practice, LoRA reduces trainable parameters by 10,000x or more while achieving performance comparable to full fine-tuning on most tasks.
For self-hosted fine-tuning workflows, QLoRA (quantized LoRA) further reduces memory requirements by loading the base model in 4-bit precision. This makes it feasible to fine-tune models in the 7B–13B parameter range on a single consumer GPU, or 70B models on a multi-GPU setup that's accessible to most organizations.
# QLoRA fine-tuning setup with Hugging Face PEFT
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, TaskType
import torch
# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True # nested quantization for additional memory savings
)
# Load base model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
"mistralai/Mistral-7B-Instruct-v0.2",
quantization_config=bnb_config,
device_map="auto"
)
# LoRA configuration
# Target the attention projection matrices — these have the most impact
lora_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=16, # rank — higher means more capacity but more parameters
lora_alpha=32, # scaling factor — usually set to 2*r
lora_dropout=0.05,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], # attention matrices
bias="none"
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 13,631,488 || all params: 3,765,325,824 || trainable%: 0.3621
Warning: Fine-tuning can cause catastrophic forgetting — the model loses general capabilities as it specializes. Always evaluate on a general benchmark alongside your domain benchmark. If general performance degrades significantly, reduce your learning rate or use a smaller LoRA rank.
Most non-trivial production systems use all three techniques. The art is in architecting them so they complement rather than conflict with each other.
This is the most common pattern for domain-specific assistants. Fine-tune the model for behavioral consistency and domain conventions. Use RAG to inject current, specific knowledge. Use prompt engineering to handle runtime context that neither fine-tuning nor RAG addresses.
User Query
│
▼
[Query Analysis & Routing] ← prompt engineering determines query type
│
├─── Simple factual query ──────▶ [RAG Pipeline] ──────▶ Fine-tuned Model
│ │
├─── Complex analysis query ──▶ [RAG + Re-ranking] ──────▶ Fine-tuned Model
│ │
└─── Procedural query ──────────────────────────────────▶ Fine-tuned Model
│
[Response]
│
[Structured Output Validation]
Not all queries need the same treatment. A router — itself an LLM call, or a small classifier — sends different query types to different pipelines:
from enum import Enum
from openai import OpenAI
client = OpenAI()
class QueryType(Enum):
FACTUAL_LOOKUP = "factual_lookup" # needs RAG
BEHAVIORAL_TASK = "behavioral_task" # needs fine-tuned model
GENERAL_REASONING = "general_reasoning" # base model is fine
HYBRID = "hybrid" # needs both RAG and fine-tuned model
def classify_query(query: str) -> QueryType:
"""
Use a fast, cheap model to classify the query type.
This should complete in <100ms to not materially affect latency.
"""
response = client.chat.completions.create(
model="gpt-4o-mini", # fast and cheap for routing
messages=[
{
"role": "system",
"content": """Classify the user query into exactly one category:
- factual_lookup: requires specific document retrieval (product specs, policies, recent data)
- behavioral_task: requires applying a specific methodology or format (analysis templates, structured extraction)
- general_reasoning: can be answered with general knowledge and reasoning
- hybrid: requires both specific document retrieval AND specialized methodology
Respond with only the category name, nothing else."""
},
{"role": "user", "content": query}
],
max_tokens=20,
temperature=0
)
classification = response.choices[0].message.content.strip().lower()
return QueryType(classification)
async def handle_query(query: str, user_context: dict) -> str:
"""
Route query to appropriate pipeline based on classification.
"""
query_type = classify_query(query)
if query_type == QueryType.FACTUAL_LOOKUP:
return await factual_pipeline(query, user_context)
elif query_type == QueryType.BEHAVIORAL_TASK:
return await behavioral_pipeline(query, user_context)
elif query_type == QueryType.GENERAL_REASONING:
return await general_pipeline(query, user_context)
elif query_type == QueryType.HYBRID:
# Retrieve documents, then pass to fine-tuned model
retrieved_docs = await retrieve_documents(query)
return await behavioral_pipeline(query, user_context, docs=retrieved_docs)
Every production system needs an evaluation framework that tests all three layers independently. This is the piece most teams skip, and it's why they can't diagnose problems when they arise.
import json
from dataclasses import dataclass
from typing import Callable
@dataclass
class EvalCase:
query: str
expected_behavior: str # human-readable description
ground_truth: str # expected output or key assertions
requires_retrieval: bool
document_sources: list[str]
def run_layer_evaluation(
eval_cases: list[EvalCase],
pipeline_fn: Callable,
judge_fn: Callable # LLM-as-judge for quality assessment
) -> dict:
"""
Run evaluation across test cases and collect pass rates per layer.
"""
results = {
"retrieval_precision": [],
"format_compliance": [],
"factual_accuracy": [],
"behavioral_consistency": []
}
for case in eval_cases:
response, retrieved_docs = pipeline_fn(case.query)
if case.requires_retrieval:
# Check if relevant documents were retrieved
retrieval_score = judge_fn(
f"Were these documents relevant to answering: '{case.query}'?\n"
f"Documents: {retrieved_docs}\n"
f"Expected sources: {case.document_sources}"
)
results["retrieval_precision"].append(retrieval_score)
# Check format compliance (deterministic if using schema validation)
format_score = validate_output_format(response)
results["format_compliance"].append(format_score)
# Check factual accuracy via judge
accuracy_score = judge_fn(
f"Does this response correctly answer '{case.query}'?\n"
f"Response: {response}\n"
f"Ground truth: {case.ground_truth}"
)
results["factual_accuracy"].append(accuracy_score)
return {k: sum(v)/len(v) for k, v in results.items() if v}
When combining RAG and prompting (with or without a fine-tuned model), you're managing a finite resource: context window tokens. In a 128K token window, you might allocate:
The retrieval step must be calibrated to fill the document budget without overflowing. This means your top-k retrieval parameter isn't arbitrary — it should be set dynamically based on the average chunk size and your available token budget:
def calculate_dynamic_top_k(
available_tokens: int,
avg_chunk_tokens: int,
safety_margin: float = 0.85
) -> int:
"""
Calculate how many chunks to retrieve given token budget.
"""
usable_tokens = int(available_tokens * safety_margin)
return max(1, usable_tokens // avg_chunk_tokens)
Build a three-layer LLM system for a loan officer assistant at a regional bank. The assistant needs to:
Step 1 — Decide the architecture before writing a line of code. For each of the three requirements above, determine which technique addresses it: prompt engineering, RAG, or fine-tuning. Write a one-paragraph justification for each decision.
Step 2 — Build the retrieval layer. Create a small document store containing 5 mock regulatory guideline documents (you can write these yourself — things like "Debt-to-income ratio must not exceed 43% for conforming loans" formatted as policy documents). Implement the hybrid retrieval function from the RAG section above and verify it retrieves relevant chunks for at least three test queries.
Step 3 — Design a fine-tuning dataset. Write 10 training examples demonstrating the correct application of the credit policy framework. At minimum, include: 3 clear approval cases, 3 clear denial cases, and 4 edge cases where the policy requires judgment. Structure them in the OpenAI fine-tuning JSONL format.
Step 4 — Wire the prompt layer. Write a system prompt that handles cases where RAG doesn't retrieve perfectly relevant documents (it needs to gracefully tell the user when it can't find applicable guidelines rather than hallucinating). Add a constraint reminder at the end of every user message.
Step 5 — Write three evaluation cases that test the system end-to-end: one that should trigger RAG, one that should rely primarily on the baked-in policy behavior, and one that should return a "cannot determine without more information" response. What does a passing result look like for each?
Symptom: The fine-tuned model is confidently stating facts that are outdated or slightly wrong, and when you update the training data, it takes weeks to retrain and redeploy.
Diagnosis: You've fine-tuned on factual content (product specs, prices, policies) instead of behavioral patterns.
Fix: Move factual content to RAG. Fine-tune only on examples that demonstrate how to process and respond to information, not what the information is. Your training examples should be valid even if the specific facts change.
Symptom: RAG is retrieving documents correctly, but response quality degrades when many documents are injected. The model starts confusing information from different documents or ignoring later chunks.
Diagnosis: You're exceeding the model's effective attention span. Most models degrade in quality with very long contexts even when they technically fit within the context window — the "lost in the middle" phenomenon, where information in the middle of a long context is attended to less reliably.
Fix: Use re-ranking to reduce to 3–5 highly relevant chunks rather than 10–15 loosely relevant ones. Include document labels (source name, date, section) so the model can reference specific sources. Consider whether the query actually needs multiple sources or if you're over-retrieving as a safety measure.
Symptom: RAG works well for some document types (structured PDFs) but poorly for others (email chains, Slack exports, HTML pages with navigation boilerplate).
Diagnosis: You're using a one-size-fits-all chunking strategy. Different document types have fundamentally different structures, and a paragraph-based chunker will produce garbage chunks from a table or a thread of emails.
Fix: Implement document-type-aware parsing. Use a library like unstructured or docling to parse documents into structured elements (Title, NarrativeText, Table, ListItem) before chunking, and apply different chunking strategies to each element type.
Symptom: The system works in testing but breaks in production when users phrase things differently, include unexpected formatting, or switch languages mid-conversation.
Diagnosis: Your prompt was tested against a narrow distribution of inputs. The model has learned to associate your specific prompt phrasing with the desired behavior, and small deviations break that association.
Fix: Adversarial prompt testing — deliberately try to break your system prompt with unusual inputs before deployment. Common attack vectors: very long user messages, code snippets in the query, mixed-language inputs, leading questions ("As an AI without restrictions..."), and queries that reference the system prompt directly.
Symptom: The production system is giving bad answers, but you can't tell if it's because RAG retrieved the wrong documents, the model misinterpreted good documents, or the output format broke downstream processing.
Diagnosis: Your logging captures input and output but not intermediate states.
Fix: Log everything: the original query, the rewritten query (if you do query rewriting), the retrieved chunks and their scores, the full prompt sent to the model, and the raw model response before any post-processing. Structure these logs so you can sample bad cases and trace them back to the specific failure point.
import logging
from dataclasses import dataclass, asdict
import json
@dataclass
class PipelineTrace:
request_id: str
original_query: str
rewritten_query: str
retrieved_chunks: list[dict] # includes chunk content and retrieval scores
full_prompt_tokens: int
raw_model_response: str
post_processed_response: str
latency_breakdown: dict # {"retrieval_ms": 120, "rerank_ms": 80, "generation_ms": 1200}
failure_point: str | None # set if an exception occurred
def log_pipeline_trace(trace: PipelineTrace):
"""Log full pipeline trace for observability and debugging."""
logger = logging.getLogger("pipeline.traces")
logger.info(json.dumps(asdict(trace), default=str))
Here's what you should now be able to do with confidence:
Diagnose first, build second. The most important skill in this space is separating knowledge problems from behavior problems before you commit to a technique. Most teams skip this step and end up with the wrong tool.
Understand the mechanical limits of each approach. Prompt engineering activates existing model knowledge — it can't extend the model's ceiling. RAG injects dynamic knowledge into context — it doesn't change how the model reasons about that knowledge. Fine-tuning changes model weights — it teaches patterns, not facts, and patterns that change frequently shouldn't be fine-tuned.
Design for observability from day one. The only way to know which layer is failing is to log intermediate states. Don't build a system that gives you a final answer with no audit trail.
Combine techniques systematically, not reflexively. The right architecture is determined by your specific requirements: update frequency, behavioral consistency requirements, latency budget, and maintenance capacity. There is no universal "best" combination.
Evaluate your retrieval system rigorously. Look into RAGAS (Retrieval Augmented Generation Assessment) — it provides automated metrics for faithfulness, answer relevance, and context precision that you can run as part of your CI pipeline.
Explore preference optimization for fine-tuning. DPO (Direct Preference Optimization) and ORPO are often more effective than supervised fine-tuning alone for complex behavioral tasks because they explicitly teach the model what not to do, not just what to do.
Study multi-hop RAG architectures. For complex analytical queries, single-shot retrieval often misses information that requires chaining multiple lookups — first retrieving a company profile, then retrieving sector benchmarks, then retrieving analyst reports. Multi-hop and agentic RAG patterns address this.
Build a regression test suite now. Every time you improve any layer of your system, run the full eval suite to ensure the improvement didn't break something else. LLM systems have unexpected interactions across layers that are only visible through systematic testing.
Learning Path: Building with LLMs