RAG Chunk Visualizer
查看文档如何切分为 RAG 分块:分块大小、重叠量、Token 统计、向量化成本与检索上下文
文档
类似于 LangChain 的 RecursiveCharacterTextSplitter:优先在空行处切分,其次是换行符、句子和词语,尽可能保留语义完整性。
预设(大小 / 重叠):
- 分块数
- 3
- 文档 Token 数
- 232
- 平均 Token / 分块
- 81
- 最小 / 最大 Token
- 46 / 126
- 待向量化 Token 数
- 242
- 重叠系数
- 1.04×
预估向量化成本: $0.000005
添加到每个提示词的上下文: 323 token
文档结构图(黄色 = 与前一分块重叠的部分)
交替显示的行便于区分相邻分块。
- #170 token280 字符0–280
# Retrieval-Augmented Generation Retrieval-Augmented Generation (RAG) gives a language model access to knowledge it was not trained on. Instead of fine-tuning, you split your documents into chunks, embed each chunk, and store the vectors in a database. ## Why chunking matters
- #2126 token503 字符255–758重叠 25 个字符
## Why chunking matters At query time the question is embedded and the closest chunks are retrieved. If a chunk is too large, it dilutes the meaning and wastes context window. If it is too small, it loses the surrounding context and the answer gets cut in half. A common starting point is 256 to 512 tokens with 10 to 20 percent overlap. Overlap repeats the end of one chunk at the start of the next, so a sentence that straddles a boundary is still found whole in at least one chunk. ## Trade-offs
- #346 token184 字符743–927重叠 15 个字符
## Trade-offs Larger overlap improves recall but increases the number of tokens you embed and store. Always measure retrieval quality on your own questions before settling on a size.
Token 计数为估算值(英文约 4 字符/Token,非拉丁字符约 1 字符/Token),并非模型分词器的精确输出。实际费用取决于提供商的最新定价。