Inaccurate results from the SQL agent tutorial

LangChain Academy
1

Followed the tutorial at LangChain: Build a SQL agent

I followed the guide and I was able to get accurate results from the chinhook database, but I got innacurate responses when I used the database I created.
For example, for chinhook.db, when I asked How many albums were done by the artist called Accept? and I got 2.

When I asked How many companies are in the database? for companies.db, I got 1000, but there are only 600 companies in the database.

I don’t know what the discrepancy is (yet)

I created a repository at langchain-practice/server at langchain-sql-agent · orimdominic/langchain-practice · GitHub for replication

2

hi @orimdominic

In your index.js you wrote:

const agent = createAgent({
  model: chatModel,
  tool: [executeSql],
});

The correct parameter name is tools (plural):

const agent = createAgent({
  model: chatModel,
  tools: [executeSql],
});

Source: createAgent type definition - CreateAgentParams.tools:

tools?: (ServerTool | ClientTool)[];

Even after fixing the tools typo, the tool would crash with a ReferenceError. Here’s why:

Your executeSql tool is defined at the module level and references db:

// Module level
const executeSql = tool(
  async ({ query }) => {
    // ...
    const result = await db.run(q);  // <-- what `db`?
  },
  // ...
);

But db is only created inside the post() function:

async function post(req, res) {
  // ...
  const db = await SqlDatabase.fromDataSourceParams({ appDataSource: datasource });
  // `db` is local to post() -- executeSql can't see it
}

The tutorial solves this with a module-level singleton pattern (see tutorial code):

let db;

async function getDb() {
  if (!db) {
    const datasource = new DataSource({ type: "sqlite", database: "./companies.db" });
    db = await SqlDatabase.fromDataSourceParams({ appDataSource: datasource });
  }
  return db;
}

Then the tool references the module-level db (or calls getDb()):

const executeSql = tool(
  async ({ query }) => {
    const q = sanitizeSqlQuery(query);
    const database = await getDb();
    const result = await database.run(q);
    return typeof result === "string" ? result : JSON.stringify(result, null, 2);
  },
  // ...
);

Bonus: use the systemPrompt parameter

The tutorial passes the system prompt as a parameter to createAgent, not via the messages array:

const agent = createAgent({
  model: chatModel,
  tools: [executeSql],
  systemPrompt: getSystemPrompt,
});

Your current approach of injecting a SystemMessage into the messages array works, but using the dedicated systemPrompt parameter is cleaner and ensures the system prompt is handled optimally by the framework (e.g., with Anthropic’s cache control support).

Source: CreateAgentParams.systemPrompt

3

Hello @pawel-twardziak !

First, I really want to appreciate you for checking it out.

You’ve made some points and it’s clear that you’ve checked my work, though your explanation doesn’t explain the discrepancy.

When I change the database source from companies.db to chinhook.db, without changing the code at all, I got accurate results. When I used companies.db with the same code, I got inaccurate results.

You’re right that my codebase is not exactly the same as that in the tutorial. I set it up that way because I have a fairly solid understanding of the programming language I used. Your explanation doesn’t explain why it works for chinhook.db but doesn’t work for companies.db.

I’ll update the codebase with these changes you have suggested and give you feedback.

4

Thank you @pawel-twardziak

The main issues were the tool→tools and the location of the db identifier.

I appreciate.

1 Like