Hello, our team is using Langsmith to evaluate our agents, and we have encountered the following issues during the process:
We added an LLM as a judge evaluator, but when mapping the agent's actual output in the prompt of this evaluator, the 'output' label only supports simple dictionary indexing.The output of our agent is a JSON string, where the final output is actually in the content of the last element of the messages list. However, in the prompt's UI, under the output label, it doesn't support an index like Python's list\[-1\]. Is there any way for me to get the last element of the list? The length of our messages list is variable.
I hope to be able to help us solve this problem. Thank you.
hey Leon, our LLM as judge evaluators actually do support negative indexing. do you mind sending me your workspace id and the project/dataset id that this evaluator is set up on?
hey，eric, thank for you reply ,I have found the source of the problem. When there is data with negative indexes in the output of the prompt, the preview cannot display the real data that I mapped; it appears empty. This misleads me into thinking that such index retrieval is not supported, but in actual evaluation work, negative indexes are valid.
As a loyal paying user of Langsmith, I hope your team can resolve this small issue. This scenario misleads me into thinking that negative indexes are not supported.
Thanks for flagging this. We can reproduce the issue and will get this fixed.
I did test out setting an evaluator with a negative index and although the preview doesn’t work, the evaluator itself does still work. To verify you can click into the “Evaluator Traces” tab and inspect that the data is being send to the LLM as a judge
