chunk-lite: token-aware text chunking for RAG pipelines that doesn't cut mid-sentence

Package #5 in my small open-source series, and the first one with a direct RAG/AI angle. Every RAG pipeline starts the same way: take a long document, split it into pieces small enough to embed, and hope the split points don't wreck the meaning inside each piece. A naive fixed-size split has no idea where a sentence or paragraph actually ends, so it happily cuts a chunk in half mid-thought - and a chunk that starts "...and that's why the deadline was moved" with no antecedent for "that" embeds as a blurry average of two unrelated ideas. It retrieves worse for both. chunk-lite makes the boundary-aware version the default. import { chunkText } from ' chunk-lite ' ; const chunks = chunkText ( longDocument , { maxTokens : 300 , overlapTokens : 50 , }); chunks [ 0 ]; // { text: "...", startOffset: 0, endOffset: 1180, tokenCount: 298, index: 0 } A few design notes: Chunks are packed sentence-by-sentence by default (falling back to word boundaries only when a single sentence alone exceeds maxTokens), so every chunk is a coherent unit of text, not a fragment. Overlap is configurable, not automatic - it prevents a fact from being invisible to retrieval just because it landed on a chunk boundary, but it also multiplies embedding calls and storage, so there's no universally "correct" default. Every chunk carries startOffset / endOffset into the source text, so you can always trace a retrieved chunk back to exactly where it came from - for citations, highlighting, or re-chunking later without losing the mapping. No hard dependency on any specific tokenizer. The default counter is a dependency-free ~4-chars-per-token heuristic; pass your own tokenCounter (tiktoken, a model's own endpoint, whatever) if you need exact counts. Fully typed, zero required runtime dependencies, 26 tests covering overlap correctness, sentence/paragraph boundaries, oversized-sentence fallback, and custom tokenizer injection. GitHub: https://github.com/tejas821/chunk-lite npm: npm i chunk-lite Full reasoning on chunk boundaries and the overlap trade-off is in the repo's CASE_STUDY.md. Feedback welcome, especially if you've hit a boundary case this doesn't handle well.