跳转到主要内容
kitlocal.dev

RAG Chunk Visualizer

本地运行

100% 在你的浏览器中处理。输入和输出绝不会离开你的设备。

查看文档如何切分为 RAG 分块:分块大小、重叠量、Token 统计、向量化成本与检索上下文

文档

类似于 LangChain 的 RecursiveCharacterTextSplitter:优先在空行处切分,其次是换行符、句子和词语,尽可能保留语义完整性。

预设(大小 / 重叠):
分块数
3
文档 Token 数
232
平均 Token / 分块
81
最小 / 最大 Token
46 / 126
待向量化 Token 数
242
重叠系数
1.04×

预估向量化成本: $0.000005

添加到每个提示词的上下文: 323 token

文档结构图(黄色 = 与前一分块重叠的部分)

交替显示的行便于区分相邻分块。

  1. #170 token280 字符0–280
    # Retrieval-Augmented Generation
    
    Retrieval-Augmented Generation (RAG) gives a language model access to knowledge it was not trained on. Instead of fine-tuning, you split your documents into chunks, embed each chunk, and store the vectors in a database.
    
    ## Why chunking matters
    
    
  2. #2126 token503 字符255–758重叠 25 个字符
    ## Why chunking matters
    
    At query time the question is embedded and the closest chunks are retrieved. If a chunk is too large, it dilutes the meaning and wastes context window. If it is too small, it loses the surrounding context and the answer gets cut in half.
    
    A common starting point is 256 to 512 tokens with 10 to 20 percent overlap. Overlap repeats the end of one chunk at the start of the next, so a sentence that straddles a boundary is still found whole in at least one chunk.
    
    ## Trade-offs
    
    
  3. #346 token184 字符743–927重叠 15 个字符
    ## Trade-offs
    
    Larger overlap improves recall but increases the number of tokens you embed and store. Always measure retrieval quality on your own questions before settling on a size.
    

Token 计数为估算值(英文约 4 字符/Token,非拉丁字符约 1 字符/Token),并非模型分词器的精确输出。实际费用取决于提供商的最新定价。