Reporting agent using langsmith builder

LangSmith Product Help
1

hi actually I was building a reporting agent but to make that report the resources are huge too much data I am planning to use agent builder will it sustain and if you have any idea of design please let me know, design must be scalable

2

Short answer: Yes, LangSmith Fleet (formerly Agent Builder) can work for a reporting agent, but only if you treat it as an orchestrator, not a place to load huge datasets into context.

What Fleet handles well

  • Coordinating multi-step report workflows
  • Connecting to data sources via built-in tools (BigQuery, Sheets, etc.) or remote MCP servers
  • Long-running runs (no longer capped at 60s)
  • Scheduled/cron reports (Enterprise)
  • Human approval on sensitive steps via interrupt_config

What won’t scale

  • Dumping large raw datasets into the agent prompt or tool responses
  • Returning unbounded tool output into LLM context, you’ll hit context limits and quality drops fast
  • Cloud deployment payloads are capped at 25 MB (docs)

Scalable design pattern

  1. Push compute to tools, not the LLM: MCP tools should query/aggregate/filter (SQL, BigQuery, API pagination) and return summaries + pointers, not full datasets.

  2. Use subagents: e.g. data-collector → data-analyzer → report-writer. Each subagent gets a clean context window and returns concise results only (subagents pattern).

  3. Use Skills for report templates, formatting rules, and domain instructions: Loaded on demand instead of stuffing everything into the system prompt (Fleet skills).

  4. Store artifacts externally: Write charts/tables to Sheets, Docs, S3, or a file backend; the agent assembles the final narrative from summaries.

  5. Schedule + monitor: Use Fleet schedules for recurring reports and LangSmith traces for debugging.

  6. Graduate to code when needed: If volume or customization outgrows no-code, export the Fleet agent to Deep Agents and deploy via LangGraph with autoscaling (agent server scale guide).

Rule of thumb: Fleet is great for “fetch metrics for Q2 by region, compare to last quarter, draft an exec summary, post to Slack.” It’s not the right layer for “here’s 500MB of raw logs, analyze everything inline.”

3

its like multitenant system I thought the same like bringing data from lets say datadog making a report and prsenting in form of html or ppt but there are two or more customers and for each of them the creds are different and idk whether the built in mcps suport that. And I have something in mind to make orchestrator to control how to bring right customer data based on it what do you think about this ?

4

Short answer: Yes: use an orchestrator. But don’t rely on built-in Fleet MCPs for per-customer Datadog credentials. They are not designed as a multi-tenant credential vault.

What Fleet supports today

There is no built-in Datadog integration, and built-in MCPs won’t automatically swap API keys per customer.

Recommended multi-tenant design

User request ("Report for Acme, last 7 days")
        ↓
Orchestrator agent (Fleet main agent)
  - resolve customer_id
  - pick report template (Skill)
  - delegate to subagents
        ↓
Custom Datadog MCP gateway (your service)
  - fetch_metrics(customer_id, query, range)
  - looks up Acme's creds server-side
  - returns aggregated summaries only
        ↓
Report subagent
  - builds HTML / PPT from summaries

Key rules

  1. Never pass customer API keys through the LLM or agent prompt: store them in your backend (vault/DB) and inject server-side in the MCP layer.
  2. Orchestrator only routes: customer lookup, template selection, subagent delegation. Fleet subagents fit this well.
  3. Tools return summaries, not raw logs: aggregate in Datadog/your MCP before the agent sees data.
  4. Output formats:
    • PPT → Fleet has a built-in PowerPoint tool
    • HTML → generate in-agent, or write via Google Docs / Word tools, or use a sandbox/code path for richer templates (Deep Agents sandboxes)

When to stay in Fleet vs export

  • Fleet no-code works if you build a thin customer-aware MCP gateway (one endpoint, many tenants).
  • If you need strict tenant isolation, custom auth flows, or many customers at scale, export to code (Fleet → Deep Agents) and add custom_tools.py for tenant routing.

Bottom line: Orchestrator + custom MCP gateway is the right pattern. Fleet is the control plane; your backend owns customer → credentials → data fetch. Built-in MCPs alone won’t solve multi-tenant Datadog reporting.