Chunking Strategies for RAG: Splitting Documents by Size, Sentence, and Semantic Meaning

Introduction

Imagine you're building a customer support bot for a software company. The bot needs to answer questions by reading through hundreds of pages of product documentation — installation guides, API references, troubleshooting articles. You load all that text into your system, and when a user asks "How do I reset my API key?", you need the bot to find the right answer instantly. Here's the problem: you can't feed 500 pages of documentation into an AI model all at once. Language models have a limited "context window" — a ceiling on how much text they can process in a single request. So you need a smarter approach.

That approach is called RAG, which stands for Retrieval-Augmented Generation. Instead of sending all your documents to the model, you split them into smaller pieces, search for the pieces most relevant to the user's question, and send only those pieces. The AI gets focused, relevant context. The user gets a better answer. The critical middle step — splitting your documents into those smaller pieces — is called chunking. How you chunk your documents is one of the most consequential decisions you'll make when building a RAG system. Chunk too broadly and you're sending noisy, unfocused context to the model. Chunk too narrowly and you lose the surrounding context that gives meaning to an answer.

By the end of this lesson, you'll understand how chunking works, why it matters so much, and how to choose the right strategy for your specific content. You'll implement three different chunking approaches from scratch and understand when each one shines.

What you'll learn:

• What chunking is, why RAG systems need it, and how chunk size affects answer quality
• How to implement fixed-size character chunking with overlapping windows
• How to split documents by sentence boundaries to preserve natural meaning
• How semantic chunking groups text by topic rather than by arbitrary size
• How to evaluate which strategy is right for your content type

Prerequisites

You should be comfortable writing Python and have a basic understanding of what a language model is. You don't need prior experience with RAG systems — we'll build intuition from the ground up. Having the sentence-transformers and scikit-learn libraries installed will help you run the later examples (pip install sentence-transformers scikit-learn).

---

What Chunking Actually Does (and Why It's Not Trivial)

Before writing any code, let's build a clear mental model.

When you build a RAG system, you store your documents in a vector database. A vector database doesn't store text as plain text — it stores text as embeddings, which are long lists of numbers that capture the meaning of the text. When a user asks a question, the system converts that question into an embedding too, then finds the stored chunks whose embeddings are most similar. Those similar chunks get passed to the language model as context for generating an answer.

Here's the key insight: the embedding for a chunk represents the meaning of the entire chunk as a single unit. If a chunk mixes together five different topics — say, installation instructions, a pricing table, a legal disclaimer, and a troubleshooting step — the resulting embedding is a blurry average of all those meanings. It probably won't be a strong match for any specific question, because it's trying to represent too many things at once.

Think of it like a filing system. If you stuff every document into a single folder labeled "Company Stuff," finding anything specific is a nightmare. But if you create focused folders — "Q3 Invoices," "Vendor Contracts," "Employee Onboarding" — retrieval becomes fast and precise. Chunking is how you create those focused units of meaning.

The three main chunking strategies address this problem in different ways:

1. Fixed-size chunking — Split by character or token count, with optional overlap
2. Sentence-based chunking — Split at natural sentence or paragraph boundaries
3. Semantic chunking — Split where the meaning of the text actually shifts

Let's build each one.

---

Strategy 1: Fixed-Size Chunking

Fixed-size chunking is the simplest approach. You pick a chunk size (say, 500 characters) and a chunk overlap (say, 100 characters), and you slide a window across your document.

The overlap is important. Without it, a sentence that straddles two chunks gets cut in half. The first chunk ends mid-thought, and the second chunk starts with no context. Overlap ensures that the boundary region appears in both adjacent chunks, so neither chunk loses critical context.

Here's a clean implementation:

def fixed_size_chunk(text: str, chunk_size: int = 500, overlap: int = 100) -> list[str]:
    """
    Split text into fixed-size chunks with overlap.
    
    Args:
        text: The document text to chunk
        chunk_size: Maximum characters per chunk
        overlap: Number of characters to repeat between adjacent chunks
    
    Returns:
        List of text chunks
    """
    chunks = []
    start = 0
    
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        
        # Move forward by (chunk_size - overlap) to create the overlap
        start += chunk_size - overlap
    
    return chunks


# Example: chunking a product documentation paragraph
doc = """
The DataSync API allows you to synchronize records between your local database and 
the cloud in real time. Authentication is handled via OAuth 2.0 tokens, which expire 
every 24 hours. To refresh your token, call the /auth/refresh endpoint with your 
client ID and secret. Rate limits apply: free-tier accounts are capped at 100 
requests per minute, while paid accounts can make up to 1,000 requests per minute. 
If you exceed these limits, the API returns a 429 status code. You should implement 
exponential backoff in your client code to handle this gracefully.
"""

chunks = fixed_size_chunk(doc, chunk_size=200, overlap=40)

for i, chunk in enumerate(chunks):
    print(f"--- Chunk {i+1} ---")
    print(chunk)
    print()

Running this gives you chunks of exactly 200 characters, with a 40-character overlap stitching them together. Let's look at what chunk 2 might contain:

--- Chunk 2 ---
 every 24 hours. To refresh your token, call the /auth/refresh endpoint with your 
client ID and secret. Rate limits apply: free-tier accounts are capped at 100 
reque

Notice that it starts partway through a sentence about token expiration — picked up from the overlap region of chunk 1. This is fixed-size chunking's biggest weakness: it has no awareness of sentence or paragraph boundaries. It'll cheerfully cut "Rate limits apply: free-tier accounts are capped at 100" in half if the numbers fall at the edge of a chunk boundary.

When to use fixed-size chunking: It works well for highly structured, uniform content like database records, log files, or code files where natural language boundaries matter less. It's also a useful baseline to benchmark against more sophisticated approaches.

Warning: For narrative text, technical documentation, or anything where sentences build on each other, fixed-size chunking often produces poor retrieval quality. A chunk that starts mid-sentence creates a misleading embedding.

---

Strategy 2: Sentence-Based Chunking

Sentence-based chunking respects the natural structure of language. Instead of counting characters blindly, you split at sentence boundaries — full stops, paragraph breaks, heading markers — and then group those sentences into chunks that stay under a target size.

This produces chunks where every unit of text is at least a complete thought. The tradeoff is that chunk sizes become variable, which is generally fine for vector databases.

import re

def sentence_chunk(text: str, max_chunk_size: int = 500, overlap_sentences: int = 1) -> list[str]:
    """
    Split text into chunks based on sentence boundaries.
    
    Args:
        text: The document text to chunk
        max_chunk_size: Soft maximum characters per chunk
        overlap_sentences: Number of sentences to repeat between chunks
    
    Returns:
        List of text chunks
    """
    # Split on sentence-ending punctuation followed by whitespace
    sentence_endings = re.compile(r'(?<=[.!?])\s+')
    sentences = sentence_endings.split(text.strip())
    
    chunks = []
    current_chunk_sentences = []
    current_length = 0
    
    for sentence in sentences:
        sentence = sentence.strip()
        if not sentence:
            continue
        
        sentence_len = len(sentence)
        
        # If adding this sentence would exceed our limit AND we already have content,
        # save the current chunk and start a new one
        if current_length + sentence_len > max_chunk_size and current_chunk_sentences:
            chunks.append(' '.join(current_chunk_sentences))
            
            # Keep the last N sentences as overlap for the next chunk
            current_chunk_sentences = current_chunk_sentences[-overlap_sentences:]
            current_length = sum(len(s) for s in current_chunk_sentences)
        
        current_chunk_sentences.append(sentence)
        current_length += sentence_len
    
    # Don't forget the final chunk
    if current_chunk_sentences:
        chunks.append(' '.join(current_chunk_sentences))
    
    return chunks


# Same documentation, now chunked by sentence
doc = """
The DataSync API allows you to synchronize records between your local database and 
the cloud in real time. Authentication is handled via OAuth 2.0 tokens, which expire 
every 24 hours. To refresh your token, call the /auth/refresh endpoint with your 
client ID and secret. Rate limits apply: free-tier accounts are capped at 100 
requests per minute, while paid accounts can make up to 1,000 requests per minute. 
If you exceed these limits, the API returns a 429 status code. You should implement 
exponential backoff in your client code to handle this gracefully.
"""

chunks = sentence_chunk(doc, max_chunk_size=300, overlap_sentences=1)

for i, chunk in enumerate(chunks):
    print(f"--- Chunk {i+1} ({len(chunk)} chars) ---")
    print(chunk)
    print()

Now your chunks look something like this:

--- Chunk 1 (298 chars) ---
The DataSync API allows you to synchronize records between your local database and 
the cloud in real time. Authentication is handled via OAuth 2.0 tokens, which expire 
every 24 hours. To refresh your token, call the /auth/refresh endpoint with your client ID and secret.

--- Chunk 2 (287 chars) ---
To refresh your token, call the /auth/refresh endpoint with your client ID and secret. 
Rate limits apply: free-tier accounts are capped at 100 requests per minute, while 
paid accounts can make up to 1,000 requests per minute.

Notice that the last sentence of chunk 1 appears again at the start of chunk 2 — that's the overlap_sentences=1 parameter doing its job. The sentence about refreshing tokens bridges the two chunks, ensuring that chunk 2 doesn't start cold.

Every chunk is now a coherent, readable unit of text. A question like "What happens if I exceed the rate limit?" will find chunk 2 (or chunk 3, depending on your exact sizes) as a strong match, because that chunk is entirely focused on rate limiting and error handling.

When to use sentence-based chunking: This is a solid default for most natural language content — documentation, articles, reports, emails, transcripts. It's easy to implement, produces interpretable chunks, and works well for most embedding models.

---

Strategy 3: Semantic Chunking

Fixed-size and sentence-based chunking are structural approaches — they use characters or punctuation as split points. Semantic chunking takes a fundamentally different approach: it uses the meaning of the text to decide where to split.

The core idea is elegant. You compute an embedding for each sentence in your document. Then you look at how similar each sentence is to the next one. When the similarity drops sharply — when the text shifts from one topic to another — you've found a natural split point.

Think about a Wikipedia article about a city. It might have sections on history, geography, economy, and culture. Within each section, consecutive sentences are topically similar. But at the boundary between "History" and "Geography," the meaning shifts significantly. Semantic chunking detects that shift and places a chunk boundary there, even if those two sections don't have explicit headers.

Here's a working implementation:

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
import re

def semantic_chunk(
    text: str,
    model_name: str = 'all-MiniLM-L6-v2',
    similarity_threshold: float = 0.5,
    min_chunk_size: int = 100
) -> list[str]:
    """
    Split text into chunks based on semantic similarity between sentences.
    
    Args:
        text: The document text to chunk
        model_name: Sentence transformer model to use for embeddings
        similarity_threshold: Similarity below this value triggers a new chunk
        min_chunk_size: Minimum characters before a split is allowed
    
    Returns:
        List of semantically coherent text chunks
    """
    # Load the embedding model
    model = SentenceTransformer(model_name)
    
    # Split into sentences
    sentence_endings = re.compile(r'(?<=[.!?])\s+')
    sentences = [s.strip() for s in sentence_endings.split(text.strip()) if s.strip()]
    
    if len(sentences) <= 1:
        return sentences
    
    # Compute embeddings for each sentence
    embeddings = model.encode(sentences)
    
    # Find split points: places where consecutive sentence similarity is low
    split_points = []
    for i in range(len(sentences) - 1):
        sim = cosine_similarity(
            embeddings[i].reshape(1, -1),
            embeddings[i + 1].reshape(1, -1)
        )[0][0]
        
        if sim < similarity_threshold:
            split_points.append(i + 1)
    
    # Build chunks from split points
    chunks = []
    current_start = 0
    
    for split_at in split_points:
        chunk_text = ' '.join(sentences[current_start:split_at])
        
        # Don't create a tiny stub chunk — absorb into previous if too small
        if len(chunk_text) >= min_chunk_size:
            chunks.append(chunk_text)
            current_start = split_at
    
    # Final chunk
    final_chunk = ' '.join(sentences[current_start:])
    if final_chunk:
        chunks.append(final_chunk)
    
    return chunks


# A document that mixes two distinct topics
mixed_doc = """
Photosynthesis is the process by which plants convert sunlight into chemical energy. 
Chlorophyll in the plant's leaves absorbs light, primarily in the red and blue wavelengths. 
This energy drives the conversion of carbon dioxide and water into glucose and oxygen. 
Without photosynthesis, virtually all life on Earth would cease to exist.

The Python programming language was created by Guido van Rossum and first released in 1991. 
It emphasizes code readability and uses indentation to define code blocks. 
Python supports multiple programming paradigms, including procedural, object-oriented, and functional styles. 
It has become one of the most popular languages for data science and machine learning.
"""

chunks = semantic_chunk(mixed_doc, similarity_threshold=0.4)

for i, chunk in enumerate(chunks):
    print(f"--- Semantic Chunk {i+1} ---")
    print(chunk)
    print()

On this mixed document, semantic chunking will correctly identify the topic boundary between photosynthesis and Python and place a split there — even though there's no blank line or heading separating them. A sentence-based chunker might split in the middle of the Python discussion, giving you one chunk about photosynthesis + the first sentence about Python, and another chunk with the rest of Python. That produces weaker embeddings for both topics.

Tuning the threshold: The similarity_threshold is the key lever. A value of 0.5 means "split whenever consecutive sentences are less than 50% cosine-similar." Lower values (0.3) create fewer, larger chunks. Higher values (0.7) create many small chunks. Start around 0.5 and evaluate on your actual content.

Warning: Semantic chunking is slower than the other strategies because it requires running an embedding model over every sentence in your document during the indexing phase. For a 10-page document this is fast. For 10,000 documents, you'll want to batch your embeddings and possibly run on GPU. Plan accordingly.

---

Choosing the Right Strategy for Your Content

Now that you can implement all three, here's how to think about which one to use:

Fixed-size chunking is your starting point when you need something working quickly, or when you're dealing with non-linguistic data (logs, structured records, code). It's also useful as a baseline to compare against. If sentence-based chunking doesn't beat fixed-size for your use case, that's valuable information.

Sentence-based chunking is the right default for most natural language documents: knowledge base articles, PDFs, transcripts, emails, reports. It's fast, produces interpretable output you can inspect, and significantly outperforms fixed-size for content where ideas live in complete sentences.

Semantic chunking shines when your documents don't have clear section headings but do shift topics — long forum posts, combined research reports, scraped web pages that mix product descriptions with reviews. It's also powerful when you're ingesting content where the structure is unknown or variable. The cost is compute time and an extra dependency.

A practical rule of thumb: if your documents have clear headings and sections, use those as primary split points first, then apply sentence-based chunking within each section. That hybrid approach often beats pure semantic chunking at a fraction of the compute cost.

---

Hands-On Exercise

Work through this exercise to solidify your understanding:

Setup: Take the following multi-section text (or substitute a real document you have access to, like a PDF you convert to text with a library like pdfplumber):

sample_text = """
Employee onboarding at Meridian Solutions begins on the first Monday of each month. 
New hires should arrive at the main reception desk by 9 AM with a valid government ID. 
HR will issue your access badge and company laptop on day one. 
You will also receive a welcome packet containing your benefits enrollment forms.

Our health insurance plan is administered through BlueCross. 
You have 30 days from your start date to enroll or waive coverage. 
Dental and vision plans are optional add-ons available at reduced group rates. 
Dependents can be added during enrollment or at qualifying life events.

The IT department will configure your laptop during the morning of day one. 
Please do not attempt to install software before the security baseline is applied. 
VPN access will be provisioned within 48 hours of your start date. 
If you need additional software licenses, submit a request through the IT portal.
"""

Your tasks:

1. Apply fixed-size chunking with chunk_size=300 and overlap=50. Count how many chunks you get and inspect whether any chunk cuts a sentence in half.

2. Apply sentence-based chunking with max_chunk_size=300 and overlap_sentences=1. Compare the number of chunks and their readability to what you got in step 1.

3. Apply semantic chunking with a threshold of 0.5. Does it correctly identify the three topic boundaries (onboarding logistics, health insurance, IT setup)? Try adjusting the threshold to 0.3 and 0.7 and observe how that changes the output.

4. For each approach, write a one-sentence description of a search query that would retrieve a relevant chunk well, and a query that might retrieve the wrong chunk. This will sharpen your intuition for where each strategy has blind spots.

---

Common Mistakes & Troubleshooting

Chunks that are too small. If your chunk size is 100 characters, many chunks will be a single sentence or less. These produce embeddings that are too narrow and too many in number, which makes retrieval slow and often noisy. A good starting range for most embedding models is 300–800 characters (roughly 50–150 words).

Chunks that are too large. If your chunks are 2,000+ characters, you're essentially feeding the model a full page of text. The embedding averages out too many ideas, retrieval quality drops, and the model's context window fills up quickly. Aim for focused, single-topic chunks.

No overlap in fixed-size chunking. If you set overlap to 0, sentences at chunk boundaries will be severed. Always use at least 10-15% of your chunk size as overlap. For a 500-character chunk, an overlap of 50-75 characters is a reasonable floor.

Not accounting for document structure. If your documents have headers (Markdown, HTML, or otherwise), strip those headers out or use them as hard split points before applying any chunking strategy. A chunk that contains only a header like ## Section 4: Pricing is nearly useless for retrieval.

Using semantic chunking on very short documents. If your document has fewer than 10 sentences, semantic chunking often produces one chunk — or triggers splits at every sentence because individual sentences vary naturally. Sentence-based chunking is almost always better for short documents.

Forgetting to clean your text first. Extra whitespace, HTML tags, header/footer boilerplate, and page numbers all dilute your chunks. A quick preprocessing step — stripping HTML, removing repeated whitespace, eliminating page number artifacts — will significantly improve chunk quality regardless of the strategy you choose.

---

Summary & Next Steps

Chunking is the invisible foundation of every RAG system. The retrieval quality you observe — whether the model gives you precise, accurate answers or vague, hallucinated ones — is directly traceable to how well your chunks represent coherent units of meaning.

Here's what you should carry forward:

• Fixed-size chunking is simple and fast, but blind to language structure. Use it as a baseline or for non-linguistic content.
• Sentence-based chunking is the practical default for most natural language documents. It's cheap to compute and produces interpretable, coherent chunks.
• Semantic chunking is the most powerful approach for documents with shifting topics, but requires embedding every sentence at index time. Tune your similarity threshold empirically.

None of these strategies is universally best. The right choice depends on your document type, your embedding model, and the nature of the questions your users ask. When in doubt, implement two approaches, evaluate retrieval quality on a small set of test questions, and let the data guide you.

Where to go next:

• Learn how to store and retrieve chunks using a vector database like Chroma or Pinecone
• Explore hierarchical chunking, which stores both sentence-level and paragraph-level chunks and retrieves at multiple granularities
• Study embedding model selection — the chunking strategy and the embedding model need to be matched; a model trained on longer passages will behave differently than one trained on short sentences
• Dive into RAG evaluation frameworks like RAGAS, which give you quantitative metrics for retrieval precision and answer faithfulness