AI Vocabulary · 03 Working with AI
What is Chunking?
Splitting long documents into smaller pieces before storing them for AI search. How you split decides what gets found.
Long documents don't fit well into search or the context window as a whole. Chunking cuts them into smaller pieces, for example by paragraph, section or a fixed length, before they are turned into embeddings and stored in a vector database. When a question comes in, only the most relevant chunks are retrieved.
Chunk size is a trade-off. Too small, and a chunk loses its context ("it" no longer refers to anything). Too large, and the relevant sentence is buried in noise. Good chunking often follows the document's own structure and keeps headings attached.
Why it matters for you
Many disappointing "chat with your documents" tools fail at chunking: they find a piece that mentions the topic but cuts off the actual answer. If retrieval is poor, test how documents are split before blaming the model.
Own it
Well-structured source documents, with clear headings, one topic per section and dates, chunk better with any tool. Writing for humans and for RAG turns out to be the same discipline.
Where it comes from
The word is older than computers. Psychologist George A. Miller described "chunking" in his famous 1956 paper on the limits of short-term memory: people remember more when they group information into meaningful units (Chunking in psychology). AI builders borrowed the idea for RAG, where documents are split into pieces small enough to search and fit into a model's context.
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.