Configure the Chunk Settings

Learn how to split documents into appropriate chunks to optimize information retrieval.

Table of Contents

· [What is Chunking?](#what-is-chunking)

· [Choose a Chunk Mode](#choose-a-chunk-mode)

· [Pre-process Text Before Chunking](#pre-process-text-before-chunking)

· [Enable Summary Auto-Gen](#enable-summary-auto-gen)

· [Preview Chunks](#preview-chunks)

What is Chunking?

Chunking is the process of splitting long documents into shorter text segments (called "chunks"). This is a critical step in building a Knowledge Base because:

· Chunks that are too long may contain irrelevant information, causing noise during retrieval

· Chunks that are too short may lack context, leading to incomplete answers

· Properly sized chunks result in more accurate retrieval

Two key concepts:

· Delimiter: The character or sequence where text is split. For example, \n\n splits at paragraph breaks, \n at line breaks.

📝 NOTE: Delimiters are removed during chunking. For example, using `A` as the delimiter splits `CBACD` into `CB` and `CD`. To avoid information loss, use non-content characters that don't naturally appear in your documents.

· Maximum Chunk Length: The maximum size of each chunk in characters. Text exceeding this limit is force-split regardless of delimiter settings.

Choose a Chunk Mode

ClickAI provides 4 chunking modes:

Mode Overview

Mode	Description	When to use
General	Splits by delimiter and max size. Flexible and fits most cases.	General documents, FAQs, guides
Parent-Child	Creates large chunks (parent) containing smaller chunks (child). Retrieval targets child but returns fuller parent context.	Technical docs, docs needing broader context
Paragraph	Splits by natural paragraphs.	Documents with clear paragraph structure
Full Doc	Keeps the entire document as a single chunk.	Short documents, policy documents

Quick Comparison

Criteria	General	Parent-Child	Paragraph	Full Doc
Flexibility	High	Medium	Low	Low
Broad context	Medium	High	Medium	Very High
Retrieval accuracy	High	Very High	High	Low
Suits long docs	✅	✅	✅	❌
Suits short docs	✅	❌	✅	✅

Notes on Parent-Child Mode

· Only the first 10,000 tokens are processed. Content beyond this limit will be truncated.

· The parent chunk cannot be edited once created. To modify it, you must upload a new document.

⚠️ IMPORTANT: Choosing the right chunk mode is a critical step that directly affects retrieval quality. Experiment with different modes and use the Test Retrieval feature to evaluate results.

Pre-process Text Before Chunking

ClickAI provides pre-processing options to clean text before chunking:

Replace consecutive spaces, newlines, and tabs

Automatically normalizes whitespace:

· Three or more consecutive newlines → two newlines

· Multiple spaces → single space

· Tabs, form feeds, and special Unicode spaces → regular space

Remove all URLs and email addresses

Strips all URLs and email addresses from text content.

📝 NOTE: This setting is ignored in Full Doc mode.

Enable Summary Auto-Gen

When Summary Auto-Gen is enabled, ClickAI automatically generates summaries for each chunk using an LLM. Summaries help:

· Improve retrieval when user queries differ from document language

· Add high-level information for chunks containing technical content (code, tables, logs)

· Create "semantic glue" — apply identical summaries to related chunks for grouped retrieval

💡 TIP: Summary Auto-Gen is especially useful when source documents use specialized jargon but users ask questions in everyday natural language.

Preview Chunks

After configuring chunk settings, click Preview to review results:

· See how documents are split into chunks

· Inspect content of each chunk

· Adjust configuration if results are unsatisfactory

Check chunk quality:

· Chunks too short — May lack sufficient context, leading to semantic loss and inaccurate answers

· Chunks too long — May include irrelevant information, introducing semantic noise and lowering retrieval precision

· Semantically incomplete chunks — Caused by forced chunking that cuts through sentences or paragraphs, resulting in missing or misleading content

⚠️ IMPORTANT: Always preview and check chunk quality before proceeding with indexing. Re-indexing later costs additional time and resources.

📖 Previous: [Quick Create Overview] · Next: [Index Method & Retrieval Settings]