I’m facing an issue with Langsmith annotation queues, which I’m not sure is a bug, or a design oversight.
We need to be able to sometimes delete annotation queue runs which have already been labelled as completed, but this is neither possible through the UI or the API. In the API we just get a 404 whenever we try to delete a run with the status set to ‘completed’.
My question is if this is an intentional design or a bug. We’d really like to be able to delete runs, as right now if a run was mistakenly added to the queue and accidentally labelled, it’s there stuck forever, and the only way to delete it is to delete the whole annotation queue
Hello @gabrielrfg ,
Better if you create a ticket on LangChain Support Portal as well.
FYI:
@saad-supports-langch
Opened a ticket as well, hoping to get it resolved/clarified soon, thanks!
Hi @gabrielrfg
Could you share more about your typical use case for annotating and completing tasks in the queue? For example, do you regularly need to reprocess or correct completed tasks, or is this issue more about handling one-off mistakes?
Certainly, we’re using annotation queues as a way to feed annotated samples into a langsmith dataset for later evals, training…
For some reason annotation queues feeding into a dataset after samples are labelled is not a built-in feature of langsmith, so we had to create a script that goes through the queue, checks for completed runs, and adds the ones not in the dataset already.
Problem is, sometimes we may want to delete a sample from the dataset for whatever reason, or even have a corrupted sample in some way that does not fit the format of the dataset, resulting in exceptions when trying to insert into the dataset. Since in these cases insertion fails, it does not get added to the dataset, so every time our script runs, it will try again.
For now we’re mitigating this by just using a flag on the feedback format that disables the sync to dataset, but all the mistakes made in the annotation queue keep piling up because we can’t delete them.