I am trying to run experiment over a dataset in langsmith UI but it runs only 20 examples. Full experiment report also contains only 20 examples. My dataset has 100+ examples in it.
What is strange that it worked previously, I have an experiment where runs count is 112.
Using Python SDK, I’ve managed to fetch that 20-run-count experiment has fields end_time = None
Answering your possible questions:
I did not changed size of the dataset.
I picked the same subset (112 examples) for experiment.
I tried to run it again, still got only 20 run count.
My 2 evaluators were “custom code” and “LLM-as-a-judge”.
UI also looks strange while running an experiment. For outputs it says “Press Start to run your prompt over the dataset”, for my metrics it says “No feedback”. But experiment results are available.
While running an experiment there are no any errors on any requests. Checked it via network in browsers console.
Hi Vertrue - yes , we tried and still seeing only 20.
Just to make sure I proparly “open full experiments results” : I click on “Datasets & Experiments” → See the table with experiments names , find the row of “my experiment” (and the column of the “Run Count” reflects the number 18 - instead of 450) , and than click on that row , and seeing only 18 rows of experiment from dataset instead of 450.
Hey, i have the same issue here. My dataset has 27 examples, but it only runs 20. i’ve tried create a split, use base split and “All examples“ option. But stills only 20 examples to run. How can i fix it?
