跳至主要內容
kitlocal.dev

RAG Chunk Visualizer

本機執行

100% 在你的瀏覽器中處理。輸入與輸出絕不會離開你的裝置。

檢視文件如何為 RAG 切分成區塊:區塊大小、重疊大小、Token 計數、嵌入成本與檢索上下文

文件

類似 LangChain 的 RecursiveCharacterTextSplitter:優先在空行處切分,其次為換行、句子與單詞,使各區塊盡可能保留語意完整。

預設集(大小 / 重疊):
區塊數
3
文件 Token 數
232
平均 Token / 區塊
81
最小 / 最大 Token 數
46 / 126
待嵌入 Token 數
242
重疊係數
1.04×

預估嵌入成本: $0.000005

加入每個提示詞的上下文: 323 token

文件分佈圖(黃色 = 與前一個區塊重疊部分)

交替排列的行讓相鄰區塊更容易區分。

  1. #170 token280 字元0–280
    # Retrieval-Augmented Generation
    
    Retrieval-Augmented Generation (RAG) gives a language model access to knowledge it was not trained on. Instead of fine-tuning, you split your documents into chunks, embed each chunk, and store the vectors in a database.
    
    ## Why chunking matters
    
    
  2. #2126 token503 字元255–75825 個字元重疊
    ## Why chunking matters
    
    At query time the question is embedded and the closest chunks are retrieved. If a chunk is too large, it dilutes the meaning and wastes context window. If it is too small, it loses the surrounding context and the answer gets cut in half.
    
    A common starting point is 256 to 512 tokens with 10 to 20 percent overlap. Overlap repeats the end of one chunk at the start of the next, so a sentence that straddles a boundary is still found whole in at least one chunk.
    
    ## Trade-offs
    
    
  3. #346 token184 字元743–92715 個字元重疊
    ## Trade-offs
    
    Larger overlap improves recall but increases the number of tokens you embed and store. Always measure retrieval quality on your own questions before settling on a size.
    

Token 計數為估算值(英文約每 Token 4 個字元,非拉丁字元約每字元 1 個 Token),並非模型 Tokenizer 的精確輸出。實際費用取決於提供商目前的價格。