LLM-Based Chunking for RAG: Is It Worth the Cost?

A lawyer asks the firm's AI assistant when an employment contract can be ended. The assistant finds the right contract. It still answers badly: it quotes what happens after termination, but not the conditions that trigger it.

The search worked. The problem started earlier, when the contract was split into chunks (the short passages an AI assistant searches through). A fixed rule cut the clause in two, and the assistant only found one half. This look-up-then-answer setup is called RAG, short for retrieval-augmented generation.

This post explains LLM-based chunking, where an LLM (a large language model, the kind of AI behind ChatGPT) decides where to split each document. You'll see what the 2026 research says it's worth, how we run it in production, and when a simple rule is the better choice.

What is LLM-based chunking?

LLM-based chunking is a way to split documents for RAG where an AI model reads the text and decides where each chunk starts and ends. Each chunk holds one complete idea: a whole clause, a section with its heading, a table with its header. It costs more and runs slower than fixed-size splitting, and you pay that once, when documents are indexed.

Put simply:

A fixed rule splits by length. LLM-based chunking splits by meaning.

Why does chunking matter so much in RAG?

In RAG, the AI looks up passages in your own documents before it answers. It can only use the passages it finds. If a passage holds half an idea, the answer holds half an idea too.

That is why chunking quietly decides answer quality. A smarter search, such as hybrid search that also matches generated questions, helps find the right chunk. It can't repair a chunk that was cut in the wrong place.

Three common failures come from bad splits:

• A clause split in two. The conditions land in one chunk and the consequences in the next.
• A table without its header. The numbers arrive, but nothing says what the columns mean.
• A step without its explanation. The instruction is in one chunk and the reason for it in another.

What the model sees is its context, and good context engineering starts with complete passages.

Semantic chunking vs LLM-based chunking: what is the difference?

There are four common ways to split a document. They differ in who decides where the split goes.

Semantic chunking turns each sentence into numbers that capture its meaning, called embeddings. It splits where two neighbouring sentences stop being similar. It never reads the text the way a person would.

LLM-based chunking gives the text to a language model and asks it to find the natural breaks. It can also label each chunk, note its section and list the questions it answers. People also call this agentic chunking when an agent makes the choices, sometimes in several passes.

Which chunking method is best? What the 2026 benchmarks say

Two large studies this year give the most useful answer so far.

1. Tiny fixed-size chunks were among the weakest

In March 2026, researchers compared 36 chunking methods across six subject areas and five embedding models. They scored each method on how well the right passages landed in the top five search results (a measure called nDCG@5, where 1.0 is perfect).

In this study, an AI model deciding the boundaries clearly beat very small fixed-size chunks. It did not beat grouping whole paragraphs, a much cheaper method that also respects the document's structure.

2. The cost lands at indexing time

The same study found the LLM-assisted methods "introduce massive preprocessing costs" in memory and time. It also notes that this overhead is "typically amortised as a one time indexing cost". In its tests, searches over LLM-made chunks still took under 4 milliseconds.

3. Expensive methods rarely win everywhere

In August 2026, a second study tested eight chunking strategies at scale. It found that expensive methods "rarely provide consistent gains over simpler chunking". The best choice depended on the embedding model, the documents, the corpus size and the measure used. An earlier study, published at NAACL 2025, reached a similar view on similarity-based semantic chunking. Both 2026 studies are preprints on arXiv (shared before peer review), so treat them as strong evidence, not the final word.

Very small fixed-size chunks are the weak option. LLM-based chunking is a tool for hard documents, not a rule for all of them.

How does LLM-based chunking work?

Most LLM-based chunkers we have seen follow the same four steps.

1. The model reads the document. It sees the headings, sections, lists and tables, rather than a count of characters.
2. It finds the natural breaks. A new heading, the end of a clause, or a change of subject marks a boundary.
3. It builds chunks that stand alone. A section keeps its heading. A table keeps its header row.
4. It adds labels that help search. These can be a section title, a content type, a short summary or questions the chunk answers.

Some teams go one step further. Anthropic's contextual retrieval adds a short explanation of where each chunk sits in the document before it is stored, so a chunk that says "the company" still makes sense alone.

How we chunk documents in Avesta AgentOS

Our knowledge layer, IngestIQ, is one of the four modules of Avesta AgentOS. It turns PDFs, web pages and connected files into chunks that our AI co-workers search. Page grouping is a setting for each run. It is on by default for files that arrive from a connected store, and when a document is reprocessed. With it on, here is how a long PDF is chunked, in plain steps.

1. A long PDF arrives. The software marks every page, so the model always knows which page it is reading.
2. The model groups pages by topic first. It splits only where a major heading starts or the subject changes. It is asked to keep each batch to 10 pages or fewer.
3. The model chunks each batch. It keeps tables, lists and sections whole, and joins text that runs across a page break.
4. Each chunk gets labels. These are its section title, a content type such as table, list or section, and 2 to 3 questions a user might ask.
5. The software checks the page numbers. It maps each chunk back to the original pages, and rejects any page that isn't in the batch.
6. The chunks are stored for search. The questions are searchable too, alongside the text.

Why we group pages before we chunk

A 300-page report is too long to chunk in one go. Cutting it every few pages brings back the original problem at a larger size: a chapter split across two batches. So the first pass asks only one question: where does each topic start and end? Each batch is then small enough for careful chunking.

What one chunk record holds

This is an example, not a real client document.

1. The content is the whole clause, with its heading, even though it runs from page 12 to page 13.
2. The section title and content type let search filter or rank by kind of passage.
3. The pages point to the original PDF, so an answer can cite where it came from.
4. The questions are written in the words a user would use, not the contract's wording. Our post on hybrid search for RAG explains how matching on those questions works.

What we do when the model fails

Models sometimes return a broken answer or time out. A model failure should never leave a document silently half-indexed. If page grouping fails, the software falls back to fixed batches of a few pages.

A batch that fails is tried up to 3 times. If it still fails, the document is marked as failed, so nobody mistakes it for a complete one. Token use and estimated cost are logged per document for the AI steps, including page grouping and embeddings, so we can see what chunking costs.

Rules per document type

The default rules already keep tables, lists and sections whole.

Teams can add their own instructions on top, per kind of document. The IngestIQ documentation gives a legal example: keep each clause complete and preserve section numbering. This is where a team's knowledge of its own documents enters the pipeline: the team writes the rule, and the model follows it.

When is LLM-based chunking worth the cost, and when isn't it?

The benchmarks above used one collection of long documents in six subject areas. In its legal documents, grouping whole paragraphs did best with every embedding model tested. The paper doesn't describe testing PDF layouts, tables or page breaks, so treat its results as a guide, not proof for your documents. Tables that need their headers and clauses that cross a page break are the cases we built IngestIQ's chunking for.

So treat this table as our starting rule from building IngestIQ, not a benchmark result. Then test it on your own documents.

When not to use LLM-based chunking

• Your documents are short and uniform. A one-paragraph FAQ answer is already a good chunk.
• Your documents change constantly. Every edit means another model call for that document.
• You have no way to measure the result. Without a test set you can't tell if the extra cost bought anything.
• The real problem is search, not splitting. If complete chunks still rank badly, fix ranking first.

How do you test LLM-based chunking on your own documents?

Start small. Pick 50 to 100 of the documents people complain about most. Chunk them both ways: your current method and LLM-based chunking.

Ask real questions. Use questions your users have actually asked, with answers someone has checked. For India Avenue's investor assistant, we tested against 50 real questions the client supplied and marked, and every change still runs against them (as of October 2026). That set tests the whole assistant, not chunking alone, which is the point: it shows whether answers got better.

Compare three things. Did the right passage come back? Was the answer complete? What did indexing cost per document? Our guide to what AI evaluation is covers how to build the test set and score it.

If you can't measure the difference on your own documents, you can't justify the cost.

The bottom line

LLM-based chunking lets an AI model split documents at natural breaks, so each chunk holds a complete idea. In 2026 research it clearly beat tiny fixed-size chunks, but not cheaper methods that follow the document's structure. Its cost lands once, at indexing time.

Use it where structure matters most: long PDFs where tables need their headers and clauses cross page breaks. Start cheaper on clean text and short documents. Plan for model failures, check the page numbers, and test on your own questions before you switch everything. Then make sure the right chunks rank first, with hybrid search for RAG.

Which of your documents would you trust an AI assistant to quote in full today?

---

Want an AI co-worker that answers from your own documents, with the source every time? See Avesta AgentOS, where IngestIQ handles chunking, search and connectors.

Get started: hello@avestalabs.ai