hey
just wanted to float a gap i keep hitting while shipping langgraph rag / graphrag agents.
langsmith (and other tracers) already show the retriever span and the docs that came back.
that's useful.
what i still can't see is which of those items actually grounded the final answer.
typical run for me:
- retriever returns 10-15 chunks, or in graphrag a mix of entities + community summaries
- the model writes an answer
- the trace treats all of that as "retrieved"
- in reality 2 items left a trace in the output and the rest was just prompt noise
that's the failure mode i care about.
"we retrieved the right neighborhood" is not the same as "the answer used this node / this chunk".
this is worse with graphrag.
the context isn't one list of passages. it's graph objects. a community summary can dominate the answer while the entity subgraph you also stuffed in is unused. the trace currently can't say that.
what i am not asking:
- don't merge an external package into libs/langgraph
- don't add a retriever abstraction into langgraph core
- don't replace langsmith
what i am asking is whether you'd be open to a small convention so tools can attach attribution onto the same trace:
- every retrieved item has a stable id (chunk / entity / community)
- after generation, each id can carry attribution
score + method (lexical | graph-path | judge) + status (grounded | no_trace | unknown)
- graphrag retrievers put node/community ids in metadata, not only raw text
i already have a local, no-llm version of the lexical/graph-path scoring in a side package. happy to turn that into a docs cookbook / reference callback if the convention is the thing you want, not the package.
looking for a signal on where this should live:
a) documented metadata convention
b) a pattern on docs.langchain.com
c) something langsmith should render
if this is the wrong repo, point me and i'll move it.
happy to have a convo over this