Agents run for a long period of time and have multiple interactions with users, tackling a wide variety of problems. This makes it hard to understand what’s happening in production.
Today we’re launching two new features in LangSmith, our agent engineering platform, so you can better understand your agent behavior:
• Insights Agent is a new agent that automatically categorizes agent behavior patterns in LangSmith. It is our first step towards helping LangSmith users find signal in their production traces.
• Multi-turn Evals lets you evaluate agent trajectory across complete conversations, so you can see if the agent actually accomplished user goals. These evaluations let you measure things like:
-
Semantic intent: What the user was actually trying to do.
-
Semantic outcomes: Whether the task was completed (and if not, why).
-
Agent trajectory: How the interaction unfolded, including tool calls and decisions made along the way.
Learn more from the blog!
