Query Expansion in RAG: Using Hypothetical Document Embeddings and Multi-Query Retrieval to Improve Recall

Introduction

Imagine you've built a RAG (Retrieval-Augmented Generation) system for a financial services company. Users can ask questions about hundreds of internal policy documents, and the system retrieves relevant chunks to answer them. But something keeps going wrong: users ask "What happens if I miss a mortgage payment?" and the system returns nothing useful — even though the answer is clearly buried in a document titled "Delinquency Procedures and Remediation Guidelines." The problem isn't your embedding model. It isn't your chunking strategy. The problem is a vocabulary mismatch between how users ask questions and how answers are written.

This is the core retrieval recall problem in RAG systems, and it's more common than you'd think. Recall, in this context, means your system's ability to find all the relevant documents that exist — not just the ones that happen to use the same words as the user's question. Standard RAG retrieves by embedding a single query and finding semantically similar chunks. When the vocabulary gap between question and answer is large, this single-shot approach fails. Query expansion is the family of techniques designed to close that gap.

By the end of this lesson, you'll understand exactly why single-query retrieval fails in predictable ways, and you'll be able to implement two powerful solutions — Multi-Query Retrieval and Hypothetical Document Embeddings (HyDE) — using Python and LangChain. These aren't academic curiosities; they're production techniques used in real enterprise RAG systems.

What you'll learn:

• Why single-query vector search has a recall problem and when it fails most predictably
• How Multi-Query Retrieval works and how to implement it using an LLM to generate query variants
• What Hypothetical Document Embeddings (HyDE) are and the intuition behind why they improve retrieval
• How to implement both techniques end-to-end with LangChain and OpenAI
• How to combine the two approaches and evaluate whether they're actually helping

---

Prerequisites

• Basic familiarity with what RAG is (you understand the retrieve-then-generate pattern)
• Python installed with pip available
• An OpenAI API key (or access to another LLM via LangChain)
• Basic comfort with Python functions and dictionaries

Install the required packages before we start:

pip install langchain langchain-openai langchain-community chromadb tiktoken

---

Why Single-Query Retrieval Fails

Before we build solutions, we need to understand the failure mode deeply.

When you embed a user query like "What happens if I miss a mortgage payment?", you're converting that sentence into a vector — a list of numbers that represents its meaning in high-dimensional space. Your retrieval system then finds document chunks whose vectors are closest to that query vector.

The problem is that distance in embedding space reflects linguistic and semantic similarity, and professional documents are often written in a register very different from how people ask questions. The chunk containing your answer might start with: "In cases of payment delinquency exceeding 30 days, the servicer shall initiate remediation procedures as outlined in Section 4.2..."

That sentence and the original question are about the same thing, but they don't look anything alike to an embedding model. The overlap in vocabulary is minimal. The sentence structure is different. The formality level is different. The embedding distance between them may be large enough that the chunk doesn't make the top-k results.

This isn't a bug in embedding models — it's a fundamental characteristic of how they work. Embedding models are trained on text, and text in the real world has massive vocabulary variation for the same underlying concept. Query expansion acknowledges this reality and works around it instead of fighting it.

Think of it like a library card catalog. If you search for "heart attack" but the medical textbooks were indexed under "myocardial infarction," you'll come up empty. A good librarian would suggest synonymous search terms. Query expansion makes your RAG system its own librarian.

---

Technique 1: Multi-Query Retrieval

The Core Idea

Multi-Query Retrieval is conceptually simple: instead of sending one query to your vector store, you send several. Specifically, you use an LLM to rewrite your original question into multiple semantically equivalent but linguistically different versions. Each version gets embedded and retrieved independently. You then merge the results, deduplicate, and pass the combined set of chunks to your generation step.

The beauty of this approach is that each query variant has a chance to match document chunks that the others might miss. If one phrasing of the question happens to share vocabulary with the relevant document, that chunk gets retrieved — even if the original phrasing would have missed it.

Implementation

Let's build this step by step. We'll use a realistic scenario: a RAG system over a collection of HR policy documents.

First, set up your environment and a simple vector store:

import os
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.schema import Document

os.environ["OPENAI_API_KEY"] = "your-api-key-here"

# Simulate a small HR policy document corpus
hr_documents = [
    Document(
        page_content="""
        Employee Termination Procedures: When an employee's contract is terminated, 
        whether through resignation, dismissal, or redundancy, the following offboarding 
        process must be initiated within 48 hours. Access credentials must be revoked, 
        company equipment returned, and final pay calculated including accrued leave balances.
        """,
        metadata={"source": "hr_policy_v4.pdf", "section": "offboarding"}
    ),
    Document(
        page_content="""
        Remote Work Eligibility: Employees in roles classified as 'knowledge work' may 
        apply for hybrid or fully remote arrangements after completing 6 months of 
        continuous service. Requests must be submitted to the direct manager and approved 
        by HR. Performance metrics must remain within the top two quartiles.
        """,
        metadata={"source": "remote_work_policy.pdf", "section": "eligibility"}
    ),
    Document(
        page_content="""
        Annual Leave Entitlements: Full-time employees accrue 1.67 days of annual leave 
        per month of service, totalling 20 days per year. Leave must be approved by the 
        line manager at least two weeks in advance. Unused leave may be carried over up 
        to a maximum of 5 days per year.
        """,
        metadata={"source": "leave_policy.pdf", "section": "annual_leave"}
    ),
    Document(
        page_content="""
        Work From Home Guidelines: Staff working remotely are expected to maintain 
        standard business hours (9am-5pm local time), attend all scheduled video calls, 
        and ensure a professional background during client-facing meetings. Internet 
        connectivity issues must be reported to IT within 30 minutes of onset.
        """,
        metadata={"source": "remote_work_policy.pdf", "section": "wfh_guidelines"}
    ),
]

# Create vector store
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
splitter = RecursiveCharacterTextSplitter(chunk_size=400, chunk_overlap=50)
splits = splitter.split_documents(hr_documents)

vectorstore = Chroma.from_documents(splits, embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 2})

Now, let's see the vocabulary mismatch in action before we fix it:

# A query that uses casual, everyday language
casual_query = "Can I work from home?"

# Standard single-query retrieval
results = retriever.invoke(casual_query)
print(f"Single-query results for: '{casual_query}'")
for i, doc in enumerate(results):
    print(f"\nResult {i+1} (source: {doc.metadata['source']}):")
    print(doc.page_content[:150] + "...")

You might notice that this returns the WFH Guidelines chunk but misses the Remote Work Eligibility chunk — which actually contains the crucial information about who's allowed to work from home. The word "eligible" and "application process" don't appear anywhere in the casual question.

Now let's build the multi-query retriever:

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# The prompt that generates query variations
multi_query_prompt = ChatPromptTemplate.from_template("""
You are an AI assistant helping improve document retrieval. Your task is to generate 
4 different versions of the user's question that use different vocabulary and phrasing 
but ask for the same information. These variations will be used to retrieve relevant 
documents from a vector database, so linguistic diversity is important.

Return only the 4 questions, one per line, with no numbering or extra text.

Original question: {question}
""")

# Chain that generates the query variants
query_generator = (
    multi_query_prompt 
    | llm 
    | StrOutputParser() 
    | (lambda x: x.strip().split("\n"))  # Parse output into a list
)

# Test the generator
variants = query_generator.invoke({"question": "Can I work from home?"})
print("Generated query variants:")
for v in variants:
    print(f"  - {v}")

Your output will look something like:

Generated query variants:
  - What is the company policy on remote working arrangements?
  - Am I eligible for a work-from-home or hybrid schedule?
  - What are the requirements for telecommuting in this organization?
  - How do I apply for a remote work arrangement?

Notice how these variants cover vocabulary that actually appears in the policy documents: "remote working," "eligible," "arrangements," "requirements." Now let's use all of them for retrieval:

from langchain.load import dumps, loads

def get_unique_union(documents: list[list]) -> list:
    """Flatten a list of lists of documents, deduplicate by content."""
    # Serialize documents to strings for deduplication
    flattened = [dumps(doc) for sublist in documents for doc in sublist]
    unique = list(dict.fromkeys(flattened))  # Preserve order, remove dupes
    return [loads(doc) for doc in unique]

# Build the full multi-query retrieval chain
multi_query_retrieval_chain = (
    {"question": RunnablePassthrough()}
    | {
        "docs": (
            query_generator
            | retriever.map()  # Run retriever on each query variant
            | get_unique_union  # Merge and deduplicate results
        ),
        "question": RunnablePassthrough()
    }
)

results = multi_query_retrieval_chain.invoke("Can I work from home?")
print(f"\nMulti-query retrieved {len(results['docs'])} unique documents:")
for doc in results['docs']:
    print(f"  - {doc.metadata['source']} ({doc.metadata['section']})")

With multi-query retrieval, you'll now get both the WFH Guidelines and the Remote Work Eligibility chunk — giving your LLM the full picture when it generates an answer.

Tip: The retriever.map() call is a LangChain convenience that runs the same retriever over a list of inputs in parallel. This makes multi-query retrieval efficient — you're not waiting for each query to complete sequentially.

---

Technique 2: Hypothetical Document Embeddings (HyDE)

The Core Idea

HyDE takes a completely different approach that's initially counterintuitive. Instead of generating multiple versions of the question, you ask an LLM to generate a hypothetical answer — a document that looks like what the relevant chunk might look like, even if every specific fact in it is made up. Then you embed that hypothetical document and use it as your search query.

Why would this work? Think about it this way: the hypothetical answer will naturally use the vocabulary, structure, and register of documents in your corpus. It will say things like "employees are eligible after completing..." or "delinquency procedures shall be initiated..." — the kind of language that actually appears in the documents you're searching over. The embedding of this hypothetical answer will therefore be much closer in vector space to the real answer chunks than the original question's embedding would be.

This was originally described in a 2022 research paper by Gao et al., and it's one of those ideas that seems almost too clever to work — until you try it.

Important caveat: The hypothetical document might contain completely wrong facts. That's fine! We're not using it to answer the question. We're only using it as a retrieval probe — a search key. The actual answer still comes from retrieved real documents.

Implementation

hyde_prompt = ChatPromptTemplate.from_template("""
You are an expert writing content for a corporate HR policy document. 
Write a short paragraph (3-5 sentences) that would appear in an HR policy manual 
and would directly answer the following question. 

Write in formal policy language, as if this is an official document. 
The content can be approximate — you are generating this for retrieval purposes only.

Question: {question}
""")

# Chain that generates the hypothetical document
hyde_chain = hyde_prompt | llm | StrOutputParser()

# Test it
hypothetical_doc = hyde_chain.invoke({"question": "Can I work from home?"})
print("Hypothetical document generated by HyDE:")
print(hypothetical_doc)

Your output might look like:

Hypothetical document generated by HyDE:
Employees may be eligible to work from home or participate in hybrid work arrangements 
subject to approval from their line manager and the Human Resources department. 
Eligibility is typically contingent upon role type, tenure, and sustained performance 
standards. Requests for remote work arrangements should be submitted in writing, 
outlining the proposed schedule and demonstrating how key responsibilities will be met. 
All remote employees are expected to remain accessible during standard business hours 
and comply with applicable IT security guidelines.

Look at that vocabulary: "hybrid work arrangements," "eligibility," "line manager," "Human Resources department," "remote work arrangements." This synthetic text is going to embed very close to the actual policy chunks in your vector store.

Now let's wire it up as a retriever:

# Full HyDE retrieval chain
hyde_retrieval_chain = (
    {"question": RunnablePassthrough()}
    | {
        "docs": (
            hyde_chain           # Generate hypothetical document
            | retriever          # Use it as the query
        ),
        "question": RunnablePassthrough()
    }
)

results = hyde_retrieval_chain.invoke("Can I work from home?")
print(f"\nHyDE retrieved {len(results['docs'])} documents:")
for doc in results['docs']:
    print(f"  - {doc.metadata['source']} ({doc.metadata['section']})")

---

Combining Both Techniques

In practice, you can get even better recall by using both strategies together. Generate a hypothetical document and multiple query variants, run all of them through retrieval, and take the union of results.

def combined_expansion_retrieval(question: str, retriever, llm) -> list:
    """
    Use both HyDE and Multi-Query expansion, then merge results.
    Returns a deduplicated list of the most relevant document chunks.
    """
    
    # Generate hypothetical document
    hyde_doc = hyde_chain.invoke({"question": question})
    
    # Generate query variants  
    variants = query_generator.invoke({"question": question})
    
    # Combine: original question + hypothetical doc + variants
    all_queries = [question, hyde_doc] + variants
    
    # Retrieve for each query
    all_results = []
    for q in all_queries:
        docs = retriever.invoke(q)
        all_results.append(docs)
    
    # Merge and deduplicate
    return get_unique_union(all_results)

docs = combined_expansion_retrieval("Can I work from home?", retriever, llm)
print(f"Combined expansion retrieved {len(docs)} unique document chunks")

Warning: Combined expansion means more LLM calls and more retrieval calls. For a user-facing application, you'll want to profile the latency. Multi-query with 4 variants + HyDE means 6+ embedding lookups per query. This is usually acceptable for knowledge-base applications where accuracy matters more than raw speed, but build in async handling if you need to scale.

---

Hands-On Exercise

Now it's your turn to put this together. Build a simple question-answering system using query expansion and evaluate whether it improves recall over single-query retrieval.

Step 1: Create a corpus of at least 6 documents on a topic you know well — product documentation, financial regulations, medical guidelines, or any professional domain. Use the Document class as shown above and load them into a Chroma vector store.

Step 2: Write 5 test questions that use different vocabulary than the documents. For example, if your documents use "myocardial infarction," write a question that says "heart attack." If your documents use "contract termination," ask about "getting fired."

Step 3: For each test question, run three retrieval strategies:

• Single-query retrieval
• Multi-Query Retrieval (4 variants)
• HyDE

Step 4: For each question and strategy, manually inspect the results and record: Did the most relevant document chunk appear in the top-k results? Yes or No.

Step 5: Build a final answer-generation chain that uses your best retrieval strategy, formatted like this:

final_rag_prompt = ChatPromptTemplate.from_template("""
Answer the following question based only on the provided context. 
If the context doesn't contain enough information, say so.

Context:
{context}

Question: {question}

Answer:
""")

def format_docs(docs):
    return "\n\n---\n\n".join(doc.page_content for doc in docs)

final_chain = (
    {
        "context": lambda x: format_docs(
            combined_expansion_retrieval(x["question"], retriever, llm)
        ),
        "question": lambda x: x["question"]
    }
    | final_rag_prompt
    | llm
    | StrOutputParser()
)

answer = final_chain.invoke({"question": "Can I work from home?"})
print(answer)

What to observe: Notice whether the answers improve when expansion is used vs. single-query retrieval. Pay particular attention to questions where the vocabulary gap is largest.

---

Common Mistakes & Troubleshooting

"My query generator produces duplicate or nearly identical variants."

This usually means your prompt isn't being specific enough about what "different" means. Update the prompt to explicitly ask for variation in formality, technical vocabulary, and phrasing style. You can also add temperature=0.7 to the LLM to increase diversity.

"HyDE generates confident-sounding but completely wrong facts, and that concerns me."

This is expected behavior. The key mental model to maintain is: the hypothetical document is a search probe, not an answer. Your final generation step still reads from real retrieved documents. The wrong facts in the hypothetical document are irrelevant to the final answer's accuracy — they never appear in the output.

"My combined approach is too slow for production."

Move to async retrieval. LangChain's retriever.abatch() allows you to fire all queries concurrently. Also consider caching LLM outputs for common queries using LangChain's built-in caching layer (set_llm_cache).

"I'm getting the same documents regardless of which query variant I use."

This can mean two things: either your corpus is small and the relevant documents genuinely are the only relevant ones (fine), or your embeddings are so coarse that all variants map to the same region of vector space. Try a higher-quality embedding model (text-embedding-3-large vs text-embedding-3-small) or experiment with increasing k in your retriever.

"Multi-Query is retrieving too many irrelevant documents."

Generating 4-5 query variants with k=4 each can return 20 candidates before deduplication. Add a reranking step — use a cross-encoder model via the FlashrankRerank retriever or Cohere's rerank API — to score all candidates against the original query and keep only the top 5-6.

---

Summary & Next Steps

You've covered a lot of ground. Here's the mental model to take away:

Single-query RAG fails when the vocabulary of questions doesn't match the vocabulary of answers. This is the recall problem, and it's not a bug — it's a predictable consequence of how vector similarity works. Query expansion attacks this problem from two angles:

Multi-Query Retrieval diversifies the linguistic surface area of your search by generating semantically equivalent but lexically different query variants. It's like asking several people to rephrase the same question before searching.

HyDE attacks the vocabulary gap by generating a hypothetical answer in the register of your document corpus, then using that as the search probe. It's counterintuitive but effective — you're essentially asking "what would the answer look like?" and using that to find where the answer lives.

Both techniques add LLM calls to your retrieval pipeline, which has cost and latency implications. In practice, HyDE alone is often sufficient and cheaper than full multi-query expansion. But for high-stakes applications where recall matters more than cost, combining them is a reasonable choice.

Where to go next:

• Reranking: Once you have a large candidate set from expansion, learn how cross-encoder rerankers can dramatically improve precision. Look into Cohere Rerank or the open-source cross-encoder/ms-marco-MiniLM-L-6-v2 model.
• Contextual Compression: After retrieval, use an LLM to compress retrieved chunks to only the relevant sentences. This reduces noise in your context window.
• Evaluation Frameworks: Start measuring retrieval recall quantitatively using RAGAS, which provides automated metrics like context recall and context precision. You can't improve what you don't measure.
• Routing: Learn how to route different question types to different retrieval strategies — not every question needs the full expansion treatment.

Query expansion is one of the highest-leverage improvements you can make to a struggling RAG system. Now you know how it works and how to build it.