Hi,
We are configuring LangSmith Alerts to trigger based on a specific error threshold, filtered by error message.
However, we are receiving notifications when we shouldn’t. It seems the alert is not filtering errors correctly.
When setting up the alert, the simulation graph reflects the correct behavior: we can clearly see the difference in values when the filter is applied versus when it isn’t. But once the alert is enabled, it appears that the filter is not being applied properly.
Any ideas?
I’d like to investigate this - can you message me your organization id or workspace id? You can access them by clicking on Settings in the sidebar then clicking one of these buttons:
Hi Arian, thanks for looking into it.
Workspace ID is 68c9ba3b-7f66-42fd-99b3-7167fa082003
Great, thank you! Was able to repro and root cause. Fix should be shipping later today, will comment here when it’s live.
edit: I pushed an update - there was an issue with text normalization being inconsistently applied for the contains/does not contain filters. Please let me know if you still see your alert getting triggered incorrectly.
Hi Arian,
Thanks for looking into it. We are still experiencing the issue.
Sorry to hear that - I will continue investigating this issue today.
One detail that might help to debug: the error message has multiple lines
And we are filtering by “Error: Abort”
Best regards,
David
We noticed that just filtering by “Abort”, the alert filters correctly and doesnt’ trigger.
If the filter for “does not contain” is “Error: Abort”, then the filtering fails.
I am seeing some abort related errors on your runs that would get properly filtered by just “Abort” but not “Error: Abort” e.g.
This operation was aborted
AbortError: This operation was aborted
... stacktrace ...
which would explain what you are encountering.
