RAG Series (12): Advanced Chunking — Parent-Child and Contextual Retrieval

The Chunking Dilemma

There's a classic tension at the heart of RAG chunking:

• Chunks too small: Vector matching is precise, but the returned content is a fragment — lacking context, insufficient to answer the question fully
• Chunks too large: Content is complete, but semantics are too diffuse, embedding quality drops, and retrieval hit rates fall

This isn't a tuning problem. It's a structural flaw in naive chunking.

Small chunks are better for retrieval; large chunks are better for generation — these are inherently competing needs. Forcing a single chunk size to satisfy both means perpetually compromising on one.

This article covers two strategies that break out of this trap:

1. Parent-Child Chunking: Search with small chunks, return the corresponding large chunk to the LLM
2. Contextual Retrieval (Anthropic's approach): Prepend a document-level context description to each chunk before embedding, making the embedding semantically richer

Parent-Child Chunking

Core Idea

Indexing:
  Parent document (800 chars) → stored in docstore (InMemoryStore)
  ↓ split
  Child chunks (200 chars) → stored in vector index
 
Retrieval:
  query → vector search matches child chunk (precise)
  → look up corresponding parent document
  → return parent document to LLM (complete)

Retrieval uses the small chunk. The LLM receives the large chunk. Two needs, each optimized independently.

Implementation

LangChain's ParentDocumentRetriever encapsulates this logic:

from langchain_classic.retrievers import ParentDocumentRetriever
from langchain_classic.storage import InMemoryStore
from langchain_text_splitters import RecursiveCharacterTextSplitter
 
child_splitter = RecursiveCharacterTextSplitter(
    chunk_size=200,    # Small: used for vector retrieval
    chunk_overlap=20,
)
parent_splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,    # Large: returned to LLM after a match
    chunk_overlap=50,
)
 
vectorstore = Chroma(collection_name="parent_child", embedding_function=embeddings)
store = InMemoryStore()   # Stores the parent documents
 
retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
    search_kwargs={"k": 4},
)
retriever.add_documents(parent_docs)

Usage is identical to any other retriever — the child-to-parent mapping is handled internally:

docs = retriever.invoke("Which embedding model for Chinese text?")
# Returns 800-char parent documents, not 200-char child chunks

Contextual Retrieval

Core Idea

Anthropic published Contextual Retrieval in 2024 to solve a different problem: when a chunk is isolated from its document, it loses its positional and contextual meaning.

Consider a chunk like this:

"Compared to the approach mentioned earlier, this method improves accuracy by 12%."

Once it's a standalone chunk, "the approach mentioned earlier" is completely lost. The embedding sees only this sentence, with no idea what came before. Its semantic representation is severely degraded.

Contextual Retrieval's fix: use an LLM to generate a brief context description for each chunk, prepend it to the chunk content, then embed:

Original chunk:
"Compared to the approach mentioned earlier, this method improves accuracy by 12%."
 
With context prepended:
"This passage describes performance results comparing RAG hybrid search against 
standard vector retrieval, from the experimental results section of the article.
 
Compared to the approach mentioned earlier, this method improves accuracy by 12%."

The embedding sees richer information. Retrieval quality improves directly.

Prompt Design

CONTEXT_PROMPT = ChatPromptTemplate.from_messages([
    ("system", "You are a document analysis assistant."),
    ("human",
     "Here is the full document:\n\n<document>\n{doc_content}\n</document>\n\n"
     "Here is a chunk from the document:\n\n<chunk>\n{chunk_content}\n</chunk>\n\n"
     "In 1-2 sentences, describe the role and context of this chunk within the "
     "full document, to help understand its meaning. Output only the description, "
     "no prefix or label."),
])

Implementation

context_chain = CONTEXT_PROMPT | llm | StrOutputParser()
 
docs = []
for item in raw_data:
    full_content = f"Title: {item['title']}\n{item['content']}"
    chunks = splitter.split_text(full_content)
    for chunk in chunks:
        # Generate context description for this chunk
        context_desc = context_chain.invoke({
            "doc_content": full_content,
            "chunk_content": chunk,
        })
        # Prepend context to chunk before embedding
        enriched_content = f"{context_desc}\n\n{chunk}"
        docs.append(Document(page_content=enriched_content, ...))

Cost note: Each chunk requires one LLM call at index time. For 8 documents (~30–40 chunks total), this is manageable. For large knowledge bases, batch processing and cost control become important.

Experiment Design

We reuse the knowledge base (8 RAG technical documents) and test set (8 questions), comparing three strategies:

Core metrics: context_recall (is complete information retrieved?) and context_precision (is ranking quality good?).

Experimental Results

======================================================================
  RAGAS Metrics Comparison (Three Chunking Strategies)
======================================================================
 
  Metric               Naive     Parent-Child    Contextual
  ──────────────────────────────────────────────────────────
  context_recall       0.625        0.875 ◀        0.875 ◀
  context_precision    0.583        0.938 ◀        0.736
  faithfulness         0.846        0.969 ◀        0.981 ◀
  answer_relevancy     0.406        0.454          0.480 ◀
======================================================================

Reading the numbers:

• context_recall: Naive 0.625 → both reach 0.875 (+0.250) The biggest win. Naive chunking scatters a single concept across multiple small chunks, and top-4 retrieval doesn't always capture all of them. Parent-Child returns the full parent document, preserving complete context. Contextual Retrieval's semantic enrichment achieves the same through better embeddings.

• context_precision: Naive 0.583 → Parent-Child 0.938 (+0.355) Parent-Child delivers the largest jump in ranking quality. Small child chunks are semantically focused — they match precisely, and the relevant parent document consistently lands at the top. Contextual Retrieval also improves (0.736), but not as sharply.

• faithfulness: Naive 0.846 → Parent-Child 0.969 / Contextual 0.981 Better context quality cascades into lower hallucination rates. When the LLM gets richer, better-ordered context, it has less reason to improvise beyond it.

• answer_relevancy: All three similar, Contextual slightly ahead (0.480 vs 0.406) Modest improvement across the board. Not the primary lever here.

Which Strategy for Which Scenario

Practical guidance:

• For most cases, start with Parent-Child — low cost, significant improvement, straightforward to implement
• If your documents are highly logical with strong inter-paragraph dependencies (technical specs, academic papers), add Contextual Retrieval on top
• You can combine them: Parent-Child + Contextual (add context descriptions to child chunks). Best results, highest cost.

Full Code

Complete code is open-sourced at:

https://github.com/chendongqi/llm-in-action/tree/main/12-advanced-chunking

Key file:

• advanced_chunking.py — Full three-strategy comparison experiment

How to run:

git clone https://github.com/chendongqi/llm-in-action
cd 12-advanced-chunking
cp .env.example .env   # Fill in Embedding API key and LLM API key
pip install -r requirements.txt
python advanced_chunking.py

Summary

This article compared three chunking strategies through controlled experiments:

1. Naive Chunking — Simple and fast, but the search-size vs generation-size tension is unresolved. context_recall tops out at 0.625.
2. Parent-Child — Small chunks for search, large chunks for the LLM. context_recall and context_precision both improve dramatically. Best ROI of the three.
3. Contextual Retrieval — LLM-generated context descriptions enrich each chunk's embedding. Particularly effective for documents where chunk meaning depends on surrounding context.

The core insight: chunking is not just splitting — it's an information organization strategy. What retrieval needs (precise semantic matching) and what generation needs (complete context) are different questions. Answer them separately instead of compromising with a single chunk size.

References

• LangChain ParentDocumentRetriever Docs
• Anthropic Contextual Retrieval Blog
• LangChain Contextual Compression Docs