I want to alert my team through Slack when some tracing is added to the annotation queue. Is there a webhook or some kind of automation to do that? I only see alerts that are related to latency or feedback scores.
LangSmith currently doesn’t have built in webhooks specifically for annotation queue additions. The existing alerts focus on performance metrics (latency, feedback scores) rather than workflow events like traces being added to queues.
Potential workarounds:
-
Polling approach: periodically check the annotation queue via LangSmith API and compare with previous state to detect new additions
-
Custom integration: if you’re programmatically adding traces to the queue add Slack notifications at that point in your code
-
Feature request: this would be a valuable automation feature to request from the LangSmith team
The polling method using the LangSmith SDK would be your best bet currently, though it’s not as elegant as a proper webhook system. You could set up a simple script that checks queue size/contents every few minutes and posts to Slack when new items are detected.
We’re currently running offline evaluations, with traces manually added to the annotation queue. Our non-technical SMEs have requested Slack notifications when this occurs. This could be a valuable feature request to enhance LangSmith’s automation capabilities, especially with various third-party integrations (Slack, Linear, etc)