# Observability & Evals **URL:** https://forum.langchain.com/c/help/langsmith/8.md [Latest](https://forum.langchain.com/latest.md) · [Categories](https://forum.langchain.com/categories.md) · [Tags](https://forum.langchain.com/tags.md) --- ## [About the Observability & Evals category](https://forum.langchain.com/t/about-the-observability-evals-category/14)
**Author:** [@crystalr](https://forum.langchain.com/u/crystalr)\ **Replies:** 0\ **Last updated:** [April 22, 2025, 7:54pm UTC](https://forum.langchain.com/t/about-the-observability-evals-category/14 "2025-04-22T19:54:02Z")
Questions, issues, and/or discussions regarding LangSmith Observability & Evaluations. --- ## [Bug with high precision results from code evaluators](https://forum.langchain.com/t/bug-with-high-precision-results-from-code-evaluators/1551)
**Author:** [@gmetzker-4c](https://forum.langchain.com/u/gmetzker-4c)\ **Replies:** 3\ **Last updated:** [October 2, 2026, 7:49am UTC](https://forum.langchain.com/t/bug-with-high-precision-results-from-code-evaluators/1551 "2026-10-02T07:49:47Z")
I’m not sure if this is the appropriate place for issue reports. If not please, redirect me and I’ll file elsewhere. Issue I have several LangSmith Experiments with Custom code evaluators. In the results display I’ve … --- ## [Introducing Langshark a local-first companion observability tool for self-hosted Langgraph deployments](https://forum.langchain.com/t/introducing-langshark-a-local-first-companion-observability-tool-for-self-hosted-langgraph-deployments/4332)
**Author:** [@stokomax](https://forum.langchain.com/u/stokomax)\ **Replies:** 1\ **Last updated:** [August 17, 2026, 12:30pm UTC](https://forum.langchain.com/t/introducing-langshark-a-local-first-companion-observability-tool-for-self-hosted-langgraph-deployments/4332 "2026-08-17T12:30:29Z")
As part of my foray into working with Langchain/Langgraph, I struggled initially with understanding the feature rich runtime environment. And not having a good understanding of the runtime artifacts hampered the effectiv… --- ## [LangSmith continues making OpenAI API calls after OpenAI billing threshold is reached](https://forum.langchain.com/t/langsmith-continues-making-openai-api-calls-after-openai-billing-threshold-is-reached/4303)
**Author:** [@Bitcot\_Kaushal](https://forum.langchain.com/u/Bitcot_Kaushal)\ **Replies:** 3\ **Last updated:** [August 7, 2026, 9:53pm UTC](https://forum.langchain.com/t/langsmith-continues-making-openai-api-calls-after-openai-billing-threshold-is-reached/4303 "2026-08-07T21:53:29Z")
Hi everyone, I’m seeing an issue where LangSmith appears to continue sending requests to the OpenAI API even after the usage threshold configured for my OpenAI API key has been reached. Issue: I configured a spending… --- ## [Traces on Claude](https://forum.langchain.com/t/traces-on-claude/4275)
**Author:** [@Gregradzio](https://forum.langchain.com/u/Gregradzio)\ **Replies:** 2\ **Last updated:** [August 3, 2026, 4:52am UTC](https://forum.langchain.com/t/traces-on-claude/4275 "2026-08-03T04:52:16Z")
I know how to make traces work with Langsmith platform when I use the langchain or deepagents but sometimes I am using things like anthropic sdk (claude -p ) or codex cli directly in my harness and I wonder is there a wa… --- ## [Getting 403 Forbidden on /runs/multipart despite valid key, correct workspace, and active billing](https://forum.langchain.com/t/getting-403-forbidden-on-runs-multipart-despite-valid-key-correct-workspace-and-active-billing/4169)
**Author:** [@kevin-seeto](https://forum.langchain.com/u/kevin-seeto)\ **Replies:** 2\ **Last updated:** [July 20, 2026, 7:40am UTC](https://forum.langchain.com/t/getting-403-forbidden-on-runs-multipart-despite-valid-key-correct-workspace-and-active-billing/4169 "2026-07-20T07:40:38Z")
Hi all, I’m trying to enable LangSmith tracing for a small personal LangChain + Claude (Anthropic) project running locally on Windows (Python 3.12, langsmith 0.10.2, langchain-anthropic). Every trace ingestion attempt f… --- ## [Experiment table not showing cost breakdown & own metadata](https://forum.langchain.com/t/experiment-table-not-showing-cost-breakdown-own-metadata/3975)
**Author:** [@alexk](https://forum.langchain.com/u/alexk)\ **Replies:** 1\ **Last updated:** [June 18, 2026, 3:59pm UTC](https://forum.langchain.com/t/experiment-table-not-showing-cost-breakdown-own-metadata/3975 "2026-06-18T15:59:17Z")
I’m running my experiments locally and send cost data to LangSmith along with my experiment results, without having it compute cost for me. I see that my cost breakdown is present in the run’s metadata when I open it in… --- ## [Trace-to-Fix: how are you actually improving RAG/agents after observability flags issues?](https://forum.langchain.com/t/trace-to-fix-how-are-you-actually-improving-rag-agents-after-observability-flags-issues/3289)
**Author:** [@kamran-rapidfireAI](https://forum.langchain.com/u/kamran-rapidfireAI)\ **Replies:** 3\ **Last updated:** [June 15, 2026, 1:34pm UTC](https://forum.langchain.com/t/trace-to-fix-how-are-you-actually-improving-rag-agents-after-observability-flags-issues/3289 "2026-06-15T13:34:19Z")
I’ve been looking at the agent/LLM observability space lately (LangSmith, Arize, Braintrust, Datadog LLM Observability, etc.). Traces are great at showing what failed and where it failed. What I’m still curious about is… --- ## [LangSmith cost/token attribution differs between LangGraph v2 streaming and v3 \`astream\_events\` with Anthropic prompt caching](https://forum.langchain.com/t/langsmith-cost-token-attribution-differs-between-langgraph-v2-streaming-and-v3-astream-events-with-anthropic-prompt-caching/3899)
**Author:** [@govardhan-06](https://forum.langchain.com/u/govardhan-06)\ **Replies:** 1\ **Last updated:** [June 10, 2026, 3:50pm UTC](https://forum.langchain.com/t/langsmith-cost-token-attribution-differs-between-langgraph-v2-streaming-and-v3-astream-events-with-anthropic-prompt-caching/3899 "2026-06-10T15:50:11Z")
Hi LangSmith team, I’m seeing confusing LangSmith cost/token reporting when using DeepAgents streaming with Anthropic prompt caching. Setup: DeepAgents: 0.6.8 Model: claude-sonnet-4-6 Provider: Anthropic via ChatAnth… --- ## [How can I verify my runs have been queued for deletion via the API?](https://forum.langchain.com/t/how-can-i-verify-my-runs-have-been-queued-for-deletion-via-the-api/3849)
**Author:** [@benlammers](https://forum.langchain.com/u/benlammers)\ **Replies:** 3\ **Last updated:** [June 2, 2026, 6:23pm UTC](https://forum.langchain.com/t/how-can-i-verify-my-runs-have-been-queued-for-deletion-via-the-api/3849 "2026-06-02T18:23:57Z")
I am using the /api/v1/runs/delete endpoint to queue up specific traces / runs for deletion by passing session\_id and applicable metadata filters. The response is very opaque simply indicating that runs have been queued … --- ## [\[Feature\]: MetacognitiveGate node — source-aware instruction checking before tool execution](https://forum.langchain.com/t/feature-metacognitivegate-node-source-aware-instruction-checking-before-tool-execution/3821)
**Author:** [@pypl0](https://forum.langchain.com/u/pypl0)\ **Replies:** 2\ **Last updated:** [June 2, 2026, 6:08am UTC](https://forum.langchain.com/t/feature-metacognitivegate-node-source-aware-instruction-checking-before-tool-execution/3821 "2026-06-02T06:08:01Z")
Summary A reusable LangGraph node that enforces source-aware instruction checking before any tool call executes — blocking indirect prompt injection at the graph level rather than the prompt level. Problem LangGraph a… --- ## [Is there a limit on runs deleted by /api/v1/runs/delete when using filtered metadata](https://forum.langchain.com/t/is-there-a-limit-on-runs-deleted-by-api-v1-runs-delete-when-using-filtered-metadata/3820)
**Author:** [@benlammers](https://forum.langchain.com/u/benlammers)\ **Replies:** 2\ **Last updated:** [June 1, 2026, 6:14pm UTC](https://forum.langchain.com/t/is-there-a-limit-on-runs-deleted-by-api-v1-runs-delete-when-using-filtered-metadata/3820 "2026-06-01T18:14:23Z")
When using the /api/v1/runs/delete is there a limit of how many runs can be matched when passing metadata. For example if my metadata is an organization identifier and that matches 1000s of runs, will they all be queued… --- ## [Access to Langsmith Engine](https://forum.langchain.com/t/access-to-langsmith-engine/3819)
**Author:** [@asfandkhan-andercore](https://forum.langchain.com/u/asfandkhan-andercore)\ **Replies:** 1\ **Last updated:** [May 29, 2026, 4:54pm UTC](https://forum.langchain.com/t/access-to-langsmith-engine/3819 "2026-05-29T16:54:00Z")
Hi, how can i get access to Langsmith Engine? Thanks --- ## [How to delete traces or runs?](https://forum.langchain.com/t/how-to-delete-traces-or-runs/199)
**Author:** [@joe](https://forum.langchain.com/u/joe)\ **Replies:** 10\ **Last updated:** [May 28, 2026, 2:21pm UTC](https://forum.langchain.com/t/how-to-delete-traces-or-runs/199 "2026-05-28T14:21:44Z")
When a user deletes their account I want to delete all the user data on langsmith. I have run ids stored in my db. How do I delete all traces/runs of that user based on these run ids? --- ## [Getting 404 when clicking "Connect Github" on the Issues/Langsmith Engine tab](https://forum.langchain.com/t/getting-404-when-clicking-connect-github-on-the-issues-langsmith-engine-tab/3726)
**Author:** [@hbradlow](https://forum.langchain.com/u/hbradlow)\ **Replies:** 1\ **Last updated:** [May 14, 2026, 2:46am UTC](https://forum.langchain.com/t/getting-404-when-clicking-connect-github-on-the-issues-langsmith-engine-tab/3726 "2026-05-14T02:46:34Z")
Hi! I would love to try out the Langsmith Engine but when we try to connect our Github we lead to a 404 page. It seems to be linking to “https://github.com/apps/langsmith-forge/installations/new?state=…” -Henry --- ## [Trouble understanding and editing experiment summary evaluators feedbacks](https://forum.langchain.com/t/trouble-understanding-and-editing-experiment-summary-evaluators-feedbacks/3628)
**Author:** [@kuzanna](https://forum.langchain.com/u/kuzanna)\ **Replies:** 3\ **Last updated:** [May 13, 2026, 9:40am UTC](https://forum.langchain.com/t/trouble-understanding-and-editing-experiment-summary-evaluators-feedbacks/3628 "2026-05-13T09:40:57Z")
Hi! I’m running into two related issues with experiment feedback in LangSmith and wanted to report them. Issue context I ran an experiment with a summary evaluator multiple times (had errors during the runs, so I neede… --- ## [Run link 404s — v2 endpoint can't find run that v1 just returned](https://forum.langchain.com/t/run-link-404s-v2-endpoint-cant-find-run-that-v1-just-returned/3710)
**Author:** [@sauravj](https://forum.langchain.com/u/sauravj)\ **Replies:** 3\ **Last updated:** [May 12, 2026, 3:02pm UTC](https://forum.langchain.com/t/run-link-404s-v2-endpoint-cant-find-run-that-v1-just-returned/3710 "2026-05-12T15:02:23Z")
Summary: The LangSmith trace viewer 404s for trace links that were generated from a successful client.read\_run(run\_id) call. The /v1 endpoint continues to return the run, but the /v2 endpoint fails with "run not found: r… --- ## [Question about LangSmith Trace Search](https://forum.langchain.com/t/question-about-langsmith-trace-search/3684)
**Author:** [@user-steve](https://forum.langchain.com/u/user-steve)\ **Replies:** 4\ **Last updated:** [May 8, 2026, 7:46am UTC](https://forum.langchain.com/t/question-about-langsmith-trace-search/3684 "2026-05-08T07:46:26Z")
Hi team, I’m testing trace search in LangSmith and noticed that some traces do not appear when using metadata filters together with sorting options. Is this expected behaviour, or am I missing a setting? Thanks --- ## [Runs visiblty problem](https://forum.langchain.com/t/runs-visiblty-problem/3416)
**Author:** [@echrem](https://forum.langchain.com/u/echrem)\ **Replies:** 1\ **Last updated:** [May 5, 2026, 4:03pm UTC](https://forum.langchain.com/t/runs-visiblty-problem/3416 "2026-05-05T16:03:38Z")
there was no problem till today i couldnt see my runs only check in threads page but also in there i couldnt look whats inside the agents! --- ## [Alert triggering and ignoring filters](https://forum.langchain.com/t/alert-triggering-and-ignoring-filters/2340)
**Author:** [@davidfanggg](https://forum.langchain.com/u/davidfanggg)\ **Replies:** 9\ **Last updated:** [April 29, 2026, 10:37am UTC](https://forum.langchain.com/t/alert-triggering-and-ignoring-filters/2340 "2026-04-29T10:37:54Z")
Hi, We are configuring LangSmith Alerts to trigger based on a specific error threshold, filtered by error message. However, we are receiving notifications when we shouldn’t. It seems the alert is not filtering errors c… --- ## [Enable OpenTelemetry traces without sending data to LangSmith](https://forum.langchain.com/t/enable-opentelemetry-traces-without-sending-data-to-langsmith/3428)
**Author:** [@liurui-1](https://forum.langchain.com/u/liurui-1)\ **Replies:** 2\ **Last updated:** [April 18, 2026, 4:30am UTC](https://forum.langchain.com/t/enable-opentelemetry-traces-without-sending-data-to-langsmith/3428 "2026-04-18T04:30:45Z")
Hello, I want to send traces and metrics (if it has) of my LangChain apps to the back-end OpenTelemetry Collector along with other apps. This time I do not want to install a local LangSmith server. Is it possible? --- ## [\[Proposal\] Solving "Silent Failures" with a Causal Precedence Evaluator for Agent Trajectories](https://forum.langchain.com/t/proposal-solving-silent-failures-with-a-causal-precedence-evaluator-for-agent-trajectories/3351)
**Author:** [@ZC502](https://forum.langchain.com/u/ZC502)\ **Replies:** 0\ **Last updated:** [April 9, 2026, 8:16am UTC](https://forum.langchain.com/t/proposal-solving-silent-failures-with-a-causal-precedence-evaluator-for-agent-trajectories/3351 "2026-04-09T08:16:55Z")
Hi team & @hwchase17 — I’ve been exploring LangSmith’s trajectory evaluation docs, especially the distinction between: exact / strict trajectory matching unordered / any-order matching LLM-as-judge over the full trajec… --- ## [\[FEATURE-REQUEST\] Ability to add builtin tools in Prompt Playground/Evaluations](https://forum.langchain.com/t/feature-request-ability-to-add-builtin-tools-in-prompt-playground-evaluations/3301)
**Author:** [@gmetzker-4c](https://forum.langchain.com/u/gmetzker-4c)\ **Replies:** 0\ **Last updated:** [April 1, 2026, 6:44pm UTC](https://forum.langchain.com/t/feature-request-ability-to-add-builtin-tools-in-prompt-playground-evaluations/3301 "2026-04-01T18:44:38Z")
I’m using the Langsmith UI, to run Evaluations. I would like the ability to specify the use of built-in (provider) tools , along with the OutputSchema. In AWS Bed rock, its easy I just need to add this to the root of t… --- ## [How are teams handling evals when agent pipelines span multiple LangSmith projects?](https://forum.langchain.com/t/how-are-teams-handling-evals-when-agent-pipelines-span-multiple-langsmith-projects/3300)
**Author:** [@shyam](https://forum.langchain.com/u/shyam)\ **Replies:** 1\ **Last updated:** [April 1, 2026, 6:05pm UTC](https://forum.langchain.com/t/how-are-teams-handling-evals-when-agent-pipelines-span-multiple-langsmith-projects/3300 "2026-04-01T18:05:01Z")
Hi all, I’m researching a gap in multi agent observability and evals. In orgs where different teams each own their own LangSmith project, it seems like traces, evals, and debugging can become fragmented. This can make r… --- ## [Passing variable to multi-turn conversation eval prompt, coming as empty during execution](https://forum.langchain.com/t/passing-variable-to-multi-turn-conversation-eval-prompt-coming-as-empty-during-execution/2762)
**Author:** [@guna-20](https://forum.langchain.com/u/guna-20)\ **Replies:** 3\ **Last updated:** [March 9, 2026, 11:33pm UTC](https://forum.langchain.com/t/passing-variable-to-multi-turn-conversation-eval-prompt-coming-as-empty-during-execution/2762 "2026-03-09T23:33:42Z")
I’m trying to predict the sentiment of a thread. But while passing the thread history (conversation) to the prompt, it displays \[‘all\_message\_objects’, ‘all\_messages’, ‘first\_human\_last\_ai’, ‘human\_ai\_pairs’\] but all va… --- ## [Reasoning\_effort parameter not working with GPT-5.2 in playground](https://forum.langchain.com/t/reasoning-effort-parameter-not-working-with-gpt-5-2-in-playground/3069)
**Author:** [@baptgr](https://forum.langchain.com/u/baptgr)\ **Replies:** 2\ **Last updated:** [March 9, 2026, 11:03pm UTC](https://forum.langchain.com/t/reasoning-effort-parameter-not-working-with-gpt-5-2-in-playground/3069 "2026-03-09T23:03:23Z")
I’m getting this error when trying to use reasoning\_effort: medium with GPT-5.2: TypeError("AsyncResponses.create() got an unexpected keyword argument 'reasoning\_effort'") The SDK doesn’t recognize this parameter even … --- ## [How to enable LangSmith tracing in my project?](https://forum.langchain.com/t/how-to-enable-langsmith-tracing-in-my-project/3077)
**Author:** [@Najiya](https://forum.langchain.com/u/Najiya)\ **Replies:** 5\ **Last updated:** [March 6, 2026, 10:15am UTC](https://forum.langchain.com/t/how-to-enable-langsmith-tracing-in-my-project/3077 "2026-03-06T10:15:24Z")
Can you explain how to enable LangSmith tracing in my project? Usually, what I do is create a .env file with the following variables: LANGSMITH\_TRACING=true LANGSMITH\_ENDPOINT=https://api.smith.langchain.com LANGSMITH\_… --- ## [How to delete a langsmith account](https://forum.langchain.com/t/how-to-delete-a-langsmith-account/1463)
**Author:** [@lovanc](https://forum.langchain.com/u/lovanc)\ **Replies:** 3\ **Last updated:** [February 26, 2026, 7:25pm UTC](https://forum.langchain.com/t/how-to-delete-a-langsmith-account/1463 "2026-02-26T19:25:56Z")
My organization has invited me to Langsmith however the redirect from the email to browser seems to have picked up a alternate google profile. Each time I try to accept the invite it logs me into Langsmith with the alter… --- ## [Langsmith experiment trace as empty Lambda](https://forum.langchain.com/t/langsmith-experiment-trace-as-empty-lambda/2920)
**Author:** [@hesamzkr](https://forum.langchain.com/u/hesamzkr)\ **Replies:** 1\ **Last updated:** [February 9, 2026, 11:29pm UTC](https://forum.langchain.com/t/langsmith-experiment-trace-as-empty-lambda/2920 "2026-02-09T23:29:03Z")
Hi, I’m creating experiments on datasets using the langgraph studio sdk and it creates experiments but these are the traces from it. what’s going wrong that this is happening? I have all of these env variables set: LAN… --- ## [Bug: Langsmith Dataset examples wiped from 'latest'](https://forum.langchain.com/t/bug-langsmith-dataset-examples-wiped-from-latest/2861)
**Author:** [@hesamzkr](https://forum.langchain.com/u/hesamzkr)\ **Replies:** 5\ **Last updated:** [February 4, 2026, 7:50pm UTC](https://forum.langchain.com/t/bug-langsmith-dataset-examples-wiped-from-latest/2861 "2026-02-04T19:50:31Z")
Hi I think there’s a bug in langsmith most of our datasets with the ‘latest’ have zero examples but it doesn’t show any deletions or etc. it seems like a porting to the latest issue. [Next page](https://forum.langchain.com/c/help/langsmith/8.md?page=1)