LangSmith OTEL: Multiple Content Parts

LangSmith Product Help Observability & Evals
1

Hi all

We’re working on a LangSmith OTEL implementation in Golang are having trouble specifying multiple parts for a single message. For example:

inputsMessages := []responses.ResponseInputItemUnionParam{
		{
			OfInputMessage: &responses.ResponseInputItemMessageParam{
				Role: string(responses.ResponseInputMessageItemRoleUser),
				Content: responses.ResponseInputMessageContentListParam{
					{
						OfInputText: &responses.ResponseInputTextParam{
							Text: "What is in this picture?",
						},
					},
					{
						OfInputImage: &responses.ResponseInputImageParam{
							ImageURL: param.NewOpt("https://cdn.creatureandcoagency.com/uploads/2014/06/Bottlenose-Dolphin-Facts-2.jpg"),
						},
					},
				},
			},
		},
	}

I’ve tried passing the entire messages array as JSON to gen_ai.prompt and sending content like so:

providerInvokeSpan.SetAttributes(attribute.Key("gen_ai.prompt.0.content").String(`[{ "type": "text", "text": "What is in this picture?" }, { "type": "input_image", "image_url": "https://cdn.creatureandcoagency.com/uploads/2014/06/Bottlenose-Dolphin-Facts-2.jpg" }]`))

Is this something the LangSmith OTEL integration supports?

I’ve also checked the Run struct in the OTEL collector but Inputs / Outputs are still weakly typed unfortunately

https://github.com/langchain-ai/langsmith-collector-proxy/blob/352bc8645a0345908ab03f0180716ece9d065d1b/internal/model/run.go#L4
2

Hi @jackHedaya super excited to hear you are building this, something I’ve wanted to do for a while!!

Here are our current supported OTEL attributes; generally we try to follow the genAI semantic conventions. Trace with OpenTelemetry | 🦜️🛠️ LangSmith

3

Looking at your example further, I should clarify that The LangSmith OTEL integration currently treats multi-part JSON content as a plain string. It doesn’t parse or recognize the multi-part structure - it just stores the entire JSON as literal text content. The converter in the langsmith-collector-proxy is pretty similar to our backend parser if that helps to serve as a reference: langsmith-collector-proxy/internal/translator/otel_converter.go at main · langchain-ai/langsmith-collector-proxy · GitHub .

If there are improvements to JSON parsing you’d like to see, lmk and I can look into updating it. Very excited to see someone work on this so would love to assist in any way I can!

4

Thank you! Saw success with using gen_ai.prompt and passing in the raw response body from OpenAI / Anthropic.

Friendly recommendation –– perhaps the integrations should follow the “parts” convention as described here?