Adaptive Chunking RAG Engine

Naive fixed-size chunking is the silent killer of most RAG systems, and the word "silent" matters as much as "killer": a fixed-size splitter never throws an error, never flags a bad cut, and never tells you it just sliced a table in half. It simply walks through a document a fixed number of tokens or characters at a time and cuts, with no awareness of where one idea ends and the next begins. Tables get split across chunk boundaries so that headers land in one chunk and the rows they describe land in another. Lists lose the items that gave them context. Arguments get severed mid-thought, with a claim in one chunk and the reasoning that supports it in the next. None of this shows up as a crash or a warning in logs — it shows up later, as retrieval that quietly underperforms.

The downstream effect is that the retrieval step hands the LLM fragments instead of complete units of meaning. A chunk boundary that fell in the wrong place means the model is being asked to answer from a piece of evidence that is missing the sentence before it, the row after it, or the premise its conclusion depended on. The model still generates an answer — that is what makes the problem dangerous rather than obviously broken — but the answer is only as good as the incomplete evidence it was built from. In a system where the whole premise is that retrieval grounds the response in real source material, evidence that has been fractured mid-thought undermines that premise without ever announcing itself.

What made this particularly hard for the client's use case is that no single splitting strategy held up everywhere. Their documents varied widely in shape and density — some dominated by dense technical prose, others structured around tables or lists, others mixing formats within the same file. A fixed-size splitter tuned to behave well on one kind of document would mishandle the next, and there was no single chunk size or overlap setting that avoided fracturing meaning across all of them. What the client needed was not a better-tuned version of the same one-size-fits-all splitter, but retrieval quality that held up consistently regardless of which document shape it was applied to.

The architecture was built around one governing idea — that chunk quality determines everything retrieval and generation can achieve downstream — carried through four concrete decisions:

• Adaptive, content-aware chunking was treated as the core differentiator of the whole system, not as one configurable step among many. Rather than cutting text at fixed intervals, chunk boundaries were made to follow the document's own structure, so a table stays a table, a list stays a list, and an argument stays intact from claim to support. This was made the centerpiece deliberately: no amount of sophistication in the embedding, retrieval, or generation stages can recover meaning that was already destroyed at the chunking step, so getting this one decision right was treated as the precondition for everything else working.
• The RAG pipeline was decoupled into five distinct stages — ingest, embed, index, retrieve, and generate — each built to be tuned or swapped independently of the others. Instead of one monolithic process, each stage sits behind its own boundary, so a change to how documents are ingested, or a switch in how they're indexed, doesn't require touching the stages on either side of it. The trade-off is more upfront engineering to define clean interfaces between stages, accepted because it avoids the alternative: re-architecting the entire pipeline every time a single component needs to evolve.
• Retrieval was built on a vector database rather than a simpler in-memory or ad hoc similarity structure, specifically so that semantic search would keep performing as the corpus grows. Because the client's document collection was not a fixed, one-time set but something expected to expand, the retrieval layer needed a foundation that scales with volume rather than one that would need to be replaced once the corpus outgrew a simpler approach.
• Both the embedding model and the LLM used for generation were treated as pluggable components rather than fixed choices baked into the pipeline. Each sits behind an interface that lets it be swapped for a different model or provider without rewriting the surrounding system. The reasoning was portability: embedding models and LLMs are an area that continues to improve and change providers, and a pipeline hard-wired to one specific model would force a rebuild every time the client wanted to adopt a newer one.

• Retrieval relevance improved because chunks kept semantically complete context together instead of splitting it across boundaries. When a retrieved chunk contains a full table, a full list, or a full line of reasoning rather than a fragment of one, the LLM is working from coherent evidence instead of having to infer what was cut off before or after it. That directly addresses the original failure mode: the model answers from what the retrieval actually found, not from a partial slice of it.
• Responses became more reliable and better grounded specifically on the content types that fixed-size chunking handled worst — dense technical material, structured content like tables, and documents that mix formats within a single file. These were exactly the cases the client's document set included and exactly where a naive splitter would have fractured meaning, so the improvement showed up where it mattered most rather than uniformly across easy, prose-only content.
• The result is a retrieval foundation that adapts to differently shaped documents without requiring a separate pipeline, or repeated re-engineering, per content type. Because the chunking is content-aware and the pipeline stages are decoupled, new documents with a different structure don't need bespoke handling built for them — the same system absorbs the variation that previously would have required tuning a splitter document type by document type.