Marmalade

A chunk visualization tool for embedding model and semantic search application development. Enter a passage of text and configure your tokenizer settings to see how the text is converted into tokens.

See the deployed app for a live demo.

Example screenshot

I developed this application as a result of developing multiple retrieval and recommendation system applications around text embeddings. It can be difficult to internally visualize and communicate to others how a given passage of text will be translated into tokens, especially when chunking and windowing is involved. For many RAG (Retrieval Augmented Generation) applications, it is important to understand how the text is chunked and windowed before it is fed into the embedding model or stored in a vector database.