Text-splitters: add_start_index returns wrong position for duplicate chunks (#41190), happy to work on a fix

OSS Product Help LangChain
1

Hi everyone! I’m Sanjana, an IIT Graduate , AI/ML engineer from India working on LLM systems and RAG pipelines, and I’m starting to contribute to LangChain.

I recently reported a bug in text-splitters: text-splitters: add_start_index returns an earlier duplicate's position when a chunk is shorter than chunk_overlap · Issue #41190 · langchain-ai/langchain · GitHub

When add_start_index=True, a chunk that is shorter than chunk_overlap and also appears earlier in the document gets the start index of that earlier occurrence instead of its own position. This matters for RAG setups that rely on start_index to map chunks back to the source text.

I’d be happy to open a PR for this. Before I do, I’d love feedback from the maintainers on whether the fix should go in how the search offset is advanced, or whether there’s a preferred approach.

I’d also appreciate pointers to other areas in text-splitters or the core library where help would be useful. Thanks!

2

If the current implementation searches from the beginning for each chunk, I’d try a forward search cursor anchored at the previous match, advancing by at least one character rather than by the full chunk length so overlapping chunks remain possible. A regression case with identical text at multiple offsets and a chunk shorter than the overlap should show whether the right occurrence is selected. It may also be worth checking offsets around Unicode characters and newlines, since Python indices count code points rather than bytes.