Hi team, we have set up some alerts based on metadata filters.
We are getting falsely alerted. When the alert triggers, we enter the tracing list using the same filters and no traces show up. The alert preview also shows no occurrances.
We had a similar issue with the error field filter in the past and was fixed Alert triggering and ignoring filters - #10 by davidfanggg .
Could you please take a look on the metadata field filtering?
The specific filter we are configuring is:
Thanks in advance!
Hi @davidfanggg
I’m looking into this one. ls_run_depth is a system-injected metadata key (LangSmith adds it during ingestion to mark root vs child runs) and it’s stored as a JSON number, not a string. Your filter has the numeric type (#) selected. Under the hood, alert firing and the alert chart preview + trace list go through two separate filter implementations, and one of the places they tend to disagree is exactly this: number vs string comparison on metadata values. We shipped a fix in this same area last week for a related metadata filter inconsistency, and your case looks like an adjacent edge it didn’t cover.
In the same filter if you switch the value type from # to T and set the value to “0” as text, does the alert preview start showing occurrences and the alert stops misfiring? If you can share the alert ID and the rough timestamp of a recent false positive, we can do some additional digging on our side as well.
Best,
Chad
Hi @lc-chad ,
Thanks for looking into this. We’ve tried with both T and #, also quoting the 0, but the preview always shows zero occurrences.
One example alert id is 7c412336-8ee5-43c5-a038-2334c20dcaab, which triggered 13/05 at 9:02 UTC.
Best,
David
Hi David,
My apologies for the delay here I missed your follow up. This looks like an issue on our side, so the T vs # and quoting attempts wouldn’t resolve.
ls_run_depth is a field we stamp on every run, stored as a JSON number. The logic that fires the alert and the logic behind the preview/trace list evaluate that number differently, so the alert matches while the preview and trace list show zero. That mismatch is the bug, and it’s why no combination of value type or quoting changed anything. I’ve reported this with our engineering team.
In the meantime, as a workaround, since you’re using ls_run_depth = 0 to scope the alert to top-level traces, can you try swapping that condition for the dedicated root-run filter: is_root is true. is_root is a first-class field rather than a metadata value, so both the alert and the preview should evaluate it consistently.
Best,
Chad
Ah, my fault on the naming. In the alert filter the field is labeled Is Trace, not is_root. In that Field dropdown you’ve got open, pick Is Trace, set it to is true, and remove the ls_run_depth condition. A trace in LangSmith is the root run, so that gives you the same root-only scoping without the metadata comparison.
Once it’s added, glance at the alert preview to confirm you’re seeing a non-zero count that matches what you expect, then you’re set.
Best,
Chad
