Hi
I’ve made some progress on an LangSmith integration for our Go LLM abstraction but have a few questions:
- My original implementation relied on picking OTEL fields and structuring the payload manually. It was difficult to format the messages correctly (having LangSmith recognize different content parts, for example), so I opted to send the entire request payloads. I do the same for responses as well. This was before reading the OTEL converter that @angus sent my way; is this recommended / discouraged?
// Example: Our OpenAI implementation
var params *responses.ResponseNewParams
paramsJSON, err := json.Marshal(params)
...
toolsJSON, err := json.Marshal(params.Tools)
...
invokeCtx, span := p.Instrumentor.InstrumentProviderInvoke(invokeCtx, "openai", config.Model.String(), paramsJSON, toolsJSON)
func (o *OTELInstrumentor) InstrumentProviderInvoke(ctx context.Context, provider string, model string, bodyBytes []byte, toolsBytes []byte) (context.Context, trace.Span) {
ctx, span := o.tracer.Start(ctx, fmt.Sprintf("invoke.%s.%s", provider, model),
trace.WithAttributes(
LangsmithSpanKindKey.String(LangSmithSpanKindLLM.String()), // langsmith.span.kind
GenAIOperationNameKey.String(GenAIOperationNameChat.String()), // gen_ai.operation.name
GenAISystemKey.String(provider), // gen_ai.system
GenAIRequestModelKey.String(model), // gen_ai.request.model
GenAIPromptKey.String(string(bodyBytes)), // gen_ai.prompt
ToolsDefinitionKey.String(string(toolsBytes)), // tools
),
trace.WithSpanKind(trace.SpanKindClient),
)
return ctx, span
}
func (o *OTELInstrumentor) InstrumentProviderResponse(span trace.Span, responseBytes []byte) {
span.SetAttributes(
GenAICompletionKey.String(string(responseBytes)), // gen_ai.completion
)
span.SetStatus(codes.Ok, "success")
}
func (o *OTELInstrumentor) InstrumentToolResult(span trace.Span, result any, err error) {
if err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, err.Error())
span.SetAttributes(
ToolErrorKey.String(err.Error()),
)
return
}
if result != nil {
if resultBytes, marshalErr := json.Marshal(result); marshalErr == nil {
// Truncate large results to avoid span bloat
resultStr := string(resultBytes)
span.SetAttributes(
ToolResultKey.String(resultStr), // tool.result
// Specific to LangSmith
GenAICompletionKey.String(resultStr), // gen_ai.completion
)
}
}
span.SetStatus(codes.Ok, "success")
}
- Given the implementation above, I haven’t been able to get LangSmith to parse the tools without manually specifying the tools separately from the rest of the gen_ai.prompt. This feels a little “hacky” … wondering if there’s a more elegant solution.
- Also in the realm of tools, I can’t get the tool result to be populated in the viewer unless I provide the tool result as a gen_ai.completion. Is this correct?
- This implementation seems to works in the trace viewer but once I attempt to use the playground, things start breaking (only for OpenAI, Anthropic works well). Is this due to LangSmith playground not supporting the Responses API? (Notice the prompts are lost –– the response defaults to the response of the original request)
- Much of these questions, I realize, are intended to make our implementation more robust and correct. Any other recommendations?