Extending LangSmith Auto Cost Tracking to Include Agent API Calls

LangSmith Product Help
, ,
1

Hi Everyone,

I’m using LangSmith for evaluating agent behavior through custom evaluators. However, I’m encountering an issue when applying LangSmith’s auto cost tracking method.

The auto cost tracking currently only includes the cost of custom evaluators, but I need to track additional costs, specifically for API calls used by the agent . Is there a method to extend or modify the auto cost tracking to include the cost of API calls alongside the evaluator costs?

Any suggestions on how to resolve this issue would be greatly appreciated.

2

You can send tokens (cost) directly: Cost tracking - Docs by LangChain using the Langsmith API and update the metadata.

Example


from langchain.agents import create_agent

from langsmith import traceable, get_current_run_tree
from langchain_core.callbacks import get_usage_metadata_callback

def run_agent_with_tracking(agent, input_data: dict):
    """Run agent and ensure all costs are tracked."""

    with get_usage_metadata_callback() as usage_cb:
        result = agent.invoke(input_data, config={"callbacks": [usage_cb]})

    # Aggregate all usage and attach to the parent run
    total_input = sum(
        u.get("input_tokens", 0)
        for u in usage_cb.usage_metadata.values()
    )

    total_output = sum(
        u.get("output_tokens", 0)
        for u in usage_cb.usage_metadata.values()
    )

    # Attach aggregated usage to the traceable run
    # You can set your per token cost here and modify
    # this as per your needs.
    run = get_current_run_tree()
    run.set(
        usage_metadata={
            "input_tokens": total_input,
            "output_tokens": total_output,
            "total_tokens": total_input + total_output,
        }
    )

    return result


agent = create_agent(
    model="gpt-4o-mini-2024-07-18",
    tools=[],
    system_prompt="You are a helpful assistant",
)

result = run_agent_with_tracking(
    agent,
    {"messages": [{"role": "user", "content": "What is the weather in Tokyo?"}]},
)