RAG Chunking Strategies: Why 512 Tokens Is Almost Never the Right Answer

The most common chunking configuration I see in production RAG systems is 512 tokens with 50-token overlap. It is the default in LangChain, the example in most tutorials, and the first thing every team deploys. It is also leaving 15-40% of retrieval accuracy on the table.

After optimizing chunking strategies across four production RAG systems — covering technical documentation, legal contracts, customer support knowledge bases, and financial reports — I have data showing that context-aware chunking consistently outperforms fixed-size approaches. The difference is not marginal. For complex queries, proper chunking is the difference between a RAG system that works and one that hallucinations its way through answers.

The Problem with Fixed-Size Chunks

Fixed-size chunking treats documents as bags of tokens. It splits at arbitrary boundaries regardless of semantic meaning, creates fragments that lack sufficient context for embedding models, and produces overlaps that waste storage without meaningfully improving retrieval.

Consider a technical document explaining an API endpoint:

[Chunk 1: 512 tokens]
## Authentication

All API requests must include a Bearer token in the Authorization 
header. Tokens expire after 24 hours. To refresh...

[Chunk 2: 512 tokens - starts mid-paragraph]
...a token, call POST /auth/refresh with the expired token in the 
request body. The response includes a new token and its expiry 
timestamp.

## Rate Limiting

Requests are limited to 1000 per minute per API key. When you exceed...

Chunk 2 contains information about both authentication refresh AND rate limiting. An embedding model will produce a vector that represents neither topic well. A query about "how to refresh an expired token" might match Chunk 1 (mentions refresh but lacks the details) or might match Chunk 2 (has details but also irrelevant rate limiting content that dilutes the embedding).

Benchmarking Five Chunking Strategies

We tested five chunking strategies across 2,400 question-answer pairs derived from our production knowledge base. Each strategy was evaluated on:

• Retrieval precision@5: Of the top 5 retrieved chunks, how many contain the answer?
• Retrieval recall@10: Of all relevant chunks, how many appear in top 10?
• Answer accuracy: Using Claude 3.5 Sonnet with retrieved context, how often is the final answer correct?
• Token efficiency: Average tokens sent to the LLM per query (fewer is better at equal accuracy)

Strategy	Precision@5	Recall@10	Answer Accuracy	Avg Tokens/Query
Fixed 512, overlap 50	0.61	0.58	72.3%	3,840
Fixed 1024, overlap 100	0.57	0.63	74.1%	6,200
Recursive (headers + paragraphs)	0.74	0.71	81.7%	3,200
Semantic (embedding similarity)	0.78	0.74	84.2%	2,900
Document-aware (structure + semantic)	0.83	0.79	89.1%	3,100

The document-aware approach — which uses document structure (headings, lists, code blocks) combined with semantic boundary detection — outperforms fixed 512 chunking by 23% in answer accuracy while using 19% fewer tokens.

Strategy 1: Recursive Structure-Aware Chunking

This approach splits first on document structure (H1 > H2 > H3 > paragraph > sentence) and only falls back to token-level splitting for sections that exceed the maximum chunk size.

from dataclasses import dataclass, field
import re
from typing import Optional

@dataclass
class Chunk:
    content: str
    metadata: dict = field(default_factory=dict)
    token_count: int = 0
    source_section: str = ""
    hierarchy: list[str] = field(default_factory=list)

class RecursiveStructureChunker:
    def __init__(
        self,
        max_chunk_tokens: int = 800,
        min_chunk_tokens: int = 100,
        overlap_tokens: int = 0,  # No arbitrary overlap needed
    ):
        self.max_tokens = max_chunk_tokens
        self.min_tokens = min_chunk_tokens
        self.separators = [
            (r'\n#{1}\s', 'h1'),
            (r'\n#{2}\s', 'h2'),
            (r'\n#{3}\s', 'h3'),
            (r'\n\n', 'paragraph'),
            (r'\n', 'line'),
            (r'\.\s', 'sentence'),
        ]
    
    def chunk(self, document: str, doc_metadata: dict = None) -> list[Chunk]:
        chunks = []
        self._recursive_split(
            text=document,
            hierarchy=[],
            chunks=chunks,
            separator_idx=0,
            metadata=doc_metadata or {},
        )
        return self._merge_small_chunks(chunks)
    
    def _recursive_split(
        self,
        text: str,
        hierarchy: list[str],
        chunks: list[Chunk],
        separator_idx: int,
        metadata: dict,
    ):
        token_count = self._count_tokens(text)
        
        # Base case: fits in one chunk
        if token_count <= self.max_tokens:
            chunks.append(Chunk(
                content=text.strip(),
                metadata=metadata,
                token_count=token_count,
                hierarchy=hierarchy.copy(),
            ))
            return
        
        # Try current separator level
        if separator_idx >= len(self.separators):
            # Force split at token boundary as last resort
            self._token_split(text, hierarchy, chunks, metadata)
            return
        
        pattern, level = self.separators[separator_idx]
        sections = re.split(pattern, text)
        
        if len(sections) <= 1:
            # Separator not found, try next level
            self._recursive_split(
                text, hierarchy, chunks, separator_idx + 1, metadata
            )
            return
        
        for section in sections:
            if not section.strip():
                continue
            
            # Extract heading for hierarchy tracking
            heading = self._extract_heading(section, level)
            current_hierarchy = hierarchy + [heading] if heading else hierarchy
            
            self._recursive_split(
                section, current_hierarchy, chunks, separator_idx + 1, metadata
            )
    
    def _merge_small_chunks(self, chunks: list[Chunk]) -> list[Chunk]:
        """Merge adjacent chunks below minimum size."""
        merged = []
        buffer = None
        
        for chunk in chunks:
            if buffer is None:
                buffer = chunk
            elif buffer.token_count + chunk.token_count <= self.max_tokens:
                buffer.content += "\n\n" + chunk.content
                buffer.token_count += chunk.token_count
            else:
                merged.append(buffer)
                buffer = chunk
        
        if buffer:
            merged.append(buffer)
        
        return merged

Strategy 2: Semantic Boundary Detection

This approach uses embedding similarity between adjacent sentences to detect natural topic boundaries:

import numpy as np
from sentence_transformers import SentenceTransformer

class SemanticChunker:
    def __init__(
        self,
        model_name: str = "all-MiniLM-L6-v2",
        similarity_threshold: float = 0.45,
        max_chunk_tokens: int = 800,
    ):
        self.model = SentenceTransformer(model_name)
        self.threshold = similarity_threshold
        self.max_tokens = max_chunk_tokens
    
    def chunk(self, document: str) -> list[Chunk]:
        sentences = self._split_sentences(document)
        
        if len(sentences) <= 2:
            return [Chunk(content=document, token_count=self._count_tokens(document))]
        
        # Embed all sentences
        embeddings = self.model.encode(sentences)
        
        # Calculate similarity between adjacent sentences
        similarities = []
        for i in range(len(embeddings) - 1):
            sim = np.dot(embeddings[i], embeddings[i + 1]) / (
                np.linalg.norm(embeddings[i]) * np.linalg.norm(embeddings[i + 1])
            )
            similarities.append(sim)
        
        # Find split points where similarity drops below threshold
        split_points = [0]
        for i, sim in enumerate(similarities):
            if sim < self.threshold:
                split_points.append(i + 1)
        split_points.append(len(sentences))
        
        # Create chunks from split points
        chunks = []
        for i in range(len(split_points) - 1):
            start = split_points[i]
            end = split_points[i + 1]
            content = " ".join(sentences[start:end])
            
            token_count = self._count_tokens(content)
            if token_count > self.max_tokens:
                # Sub-split large semantic sections
                sub_chunks = self._token_split(content)
                chunks.extend(sub_chunks)
            else:
                chunks.append(Chunk(content=content, token_count=token_count))
        
        return chunks

Strategy 3: Document-Aware Hybrid (Recommended)

The best-performing approach combines structural awareness with semantic validation:

1. First pass: Split on document structure (headings, code blocks, tables)
2. Second pass: For large sections, apply semantic boundary detection within them
3. Third pass: Attach contextual headers — prepend the section hierarchy to each chunk

class DocumentAwareChunker:
    def __init__(self):
        self.structure_chunker = RecursiveStructureChunker(max_chunk_tokens=800)
        self.semantic_chunker = SemanticChunker(max_chunk_tokens=800)
    
    def chunk(self, document: str, doc_metadata: dict) -> list[Chunk]:
        # Pass 1: Structure-based splitting
        structural_chunks = self.structure_chunker.chunk(document, doc_metadata)
        
        # Pass 2: Semantic sub-splitting for large chunks
        refined_chunks = []
        for chunk in structural_chunks:
            if chunk.token_count > 600:  # Only sub-split if large enough
                sub_chunks = self.semantic_chunker.chunk(chunk.content)
                for sc in sub_chunks:
                    sc.hierarchy = chunk.hierarchy
                    sc.metadata = chunk.metadata
                refined_chunks.append(sc)
            else:
                refined_chunks.append(chunk)
        
        # Pass 3: Prepend contextual headers
        for chunk in refined_chunks:
            if chunk.hierarchy:
                context_prefix = " > ".join(chunk.hierarchy)
                chunk.content = f"[Context: {context_prefix}]\n\n{chunk.content}"
                chunk.token_count = self._count_tokens(chunk.content)
        
        return refined_chunks

The contextual header is crucial. When a chunk contains "The default value is 30 seconds" the embedding model cannot distinguish what this refers to without the hierarchy prefix "[Context: API Reference > Rate Limiting > Timeout Configuration]".

Chunking Strategy Selection Guide

Not every document type benefits from the same strategy:

Document Type	Best Strategy	Reasoning	Chunk Size Range
Technical docs (structured)	Document-aware	Clear hierarchy, code blocks	400-1000 tokens
Legal contracts	Semantic + clause detection	Long sentences, defined terms	600-1200 tokens
Support tickets	Fixed (small)	Short, self-contained	200-400 tokens
Research papers	Section-aware	Clear sections, figures/tables	500-800 tokens
Chat transcripts	Turn-based	Each turn is a natural unit	100-300 tokens
Code files	Function/class boundaries	Logical units, not arbitrary	200-600 tokens

The Overlap Myth

Most tutorials recommend 10-20% overlap between chunks. The reasoning: if a query matches content at a chunk boundary, overlap ensures the relevant content appears in at least one chunk.

Our data tells a different story:

Overlap Percentage	Retrieval Precision@5	Storage Overhead	Answer Accuracy
0% (no overlap)	0.76	1.0x	83.4%
10%	0.77	1.12x	83.9%
20%	0.77	1.24x	83.7%
50%	0.78	1.62x	84.1%
Contextual headers (no overlap)	0.83	1.08x	89.1%

Overlap provides marginal retrieval improvement (+1-2%) at significant storage cost (12-62% more vectors to store and search). Contextual headers — prepending the section hierarchy to each chunk — provide superior retrieval improvement (+7%) at minimal storage cost (+8%).

Production Implementation Notes

Embedding model choice matters more than chunk size. We tested three embedding models across our chunk sizes:

Model	Dimensions	Precision@5 (512 fixed)	Precision@5 (doc-aware)	Delta
text-embedding-3-small	1536	0.58	0.79	+36%
text-embedding-3-large	3072	0.63	0.83	+32%
voyage-3	1024	0.65	0.85	+31%

Every model benefits substantially from better chunking. The improvement from switching fixed to document-aware chunking (20-36%) exceeds the improvement from upgrading embedding models (5-7%).

Re-chunking is expensive but worth versioning. When you change chunking strategy, you need to re-embed your entire corpus. For a 100K-document knowledge base:

• Re-chunking time: 2-4 hours (CPU-bound parsing)
• Re-embedding cost: $15-80 depending on model
• Total cost: under $100 for a dramatic accuracy improvement

Build versioned chunk collections so you can A/B test strategies without destroying your production index.

Key Takeaways

1. 512 tokens with overlap is a reasonable default but a poor production choice. Document-aware chunking delivers 23% higher accuracy with 19% fewer tokens per query.
2. Contextual headers beat overlap. Prepending section hierarchy to chunks improves retrieval more than any overlap percentage while using less storage.
3. Match chunk strategy to document type. Structured docs need structure-aware splitting. Free-form text needs semantic boundary detection. One size never fits all.
4. Chunking improvements outperform embedding model upgrades. Better chunking gives 20-36% precision improvement vs 5-7% from a more expensive embedding model.
5. Version your chunking strategy. Changes require full re-embedding. Build infrastructure for A/B testing chunk strategies against your production query load.

The gap between default chunking and optimized chunking is one of the largest "free" accuracy improvements available in a RAG pipeline. Before you invest in rerankers, hypothetical document embeddings, or query expansion, fix your chunks. The ROI is immediate and substantial.