Token chunker component¶
A component that splits texts into chunks, respecting a maximum token limit and using delimiters to find optimal breakpoints.
A Token chunker component is a text splitter that creates chunks by respecting a recommended maximum token length, using delimiters to ensure logical chunk breakpoints. It splits long texts into appropriately-sized, semantically related chunks.
Scenario¶
A Token chunker component is optional, usually placed immediately after Parser or Title chunker.
Configurations¶
Recommended chunk size¶
The recommended maximum token limit for each created chunk. The Token chunker component creates chunks at specified delimiters. If this token limit is reached before a delimiter, a chunk is created at that point.
Overlapped percent (%)¶
This defines the overlap percentage between chunks. An appropriate degree of overlap ensures semantic coherence without creating excessive, redundant tokens for the LLM.
Default: 0
Maximum: 30%
Delimiters¶
Defaults to \n. Click the right-hand Recycle bin button to remove it, or click + Add to add a delimiter.
Output¶
The global variable name for the output of the Token chunker component, which can be referenced by subsequent components in the ingestion pipeline.
Default:
chunksType:
Array<Object>