Best practices for a long-lived, agent-managed RAG / Knowledge Base?

Talking Shop
1

I’m looking for advice from people who have built and maintained serious RAG / Knowledge Base systems over time.

I have a very large personal archive accumulated over many years around my professional interests: fintech, insurance, e-commerce, market and societal trends, marketing, business strategy, design, research papers, reports, books, presentations and articles.

Roughly 90% of the archive is PDF, with the rest mainly PowerPoint, Word documents and some video.

The archive is already organized in a filesystem using my own folder taxonomy.

A key requirement is that the existing archive must remain the source of truth and should not be modified by the KB system.

What I would like to build is roughly this:

Source archive → Librarian Agent → Knowledge Base / RAG → AI agents

The Librarian Agent would maintain a derived knowledge layer without changing the original documents.

Its responsibilities could include:

  • detecting new and modified files;

  • extracting text, structure and metadata;

  • classifying documents using my existing taxonomy plus additional semantic metadata;

  • chunking and indexing;

  • maintaining embeddings and possibly graph relationships;

  • detecting duplicates and related documents;

  • tracking provenance back to the original source;

  • identifying outdated or conflicting information;

  • periodically checking KB quality and consistency.

The resulting KB should be usable by multiple local AI models and agents when they perform research or analysis.

I am also considering a second agent:

Web / external sources → Research Agent → Human approval → Librarian Agent → Archive + KB

The Research Agent would continuously explore relevant sources — industry publications, academic papers, companies, think tanks, research organizations, etc. — and propose potentially useful material.

I would review those suggestions manually.

Only after approval would the Librarian Agent archive, classify and ingest the source.

So the overall system would remain human-supervised rather than autonomously ingesting everything it finds.

My main concern is not building a demo that can “chat with PDFs”, but creating a long-lived research library that can remain reliable over several years.

This raises a few architectural questions I’d really value practical experience on:

  • What architecture would you use today for this kind of system?

  • Would you keep the knowledge layer primarily vector-based, combine it with structured metadata, or add a knowledge graph?

  • What tools have proven reliable for incremental ingestion and synchronization with a filesystem?

  • How do you preserve strong document- and chunk-level provenance?

  • How do you handle changed or deleted source documents without slowly corrupting the index?

  • What is your preferred approach to hybrid retrieval: semantic + lexical + metadata filtering + reranking?

  • How would you separate a Librarian / curation workflow from a Research / discovery workflow?

  • Which operations would you allow an LLM agent to perform autonomously, and which would you keep deterministic?

  • How do you prevent taxonomy drift, bad automatic classifications or low-quality generated relationships from degrading the KB over time?

  • What evaluation or observability practices do you use to detect retrieval degradation?

  • Which components/frameworks would you trust today for a local-first implementation?

I’m particularly interested in systems that people have actually operated, not just proof-of-concept RAG demos.

If you’ve built something similar, I’d also love to hear what you initially designed that later turned out to be unnecessary or problematic.

2

Hello @SalvatoreLarosa , welcome to langchain community!!!

I’d treat the filesystem as the source of truth and the KB as a rebuildable derived layer. I wouldn’t let an “autonomous librarian” modify the archive directly.

The basic flow would be:

archive → scan/manifest → extract → classify → chunk/index → retrieve

Keep a manifest outside the archive with a stable source_id, file hash, version history, processing status, and parser/chunker versions. Each chunk should retain its source, version, page/slide/timestamp, and character offsets so every answer can point back to the original document.

For synchronization, use filesystem watchers for speed, but also run periodic full scans because watchers can miss events. When a file changes, create a new version and index it separately. Only make it active after extraction and validation succeed. If parsing fails, keep the previous version active. For deletions, mark the source inactive first and clean up old vectors afterward.

I’d start with structured metadata plus hybrid retrieval:

  • lexical search for names, numbers, phrases, and citations;
  • vector search for semantic similarity;
  • metadata filters for taxonomy, dates, and document types;
  • reranking of the combined results.

A knowledge graph may eventually be useful, but I would not start there. Store relationships in ordinary tables first, with provenance and confidence attached to every generated edge.

The Research Agent and Librarian should also be separate. The Research Agent can discover sources and create suggestions, but the flow should be:

web discovery → candidate queue → human approval → archive → ingestion

The LLM can suggest classifications, summaries, duplicates, entities, and possible conflicts. It should not silently create taxonomy categories, delete documents, merge sources, or publish unreviewed external material.

For a local-first setup, SQLite/Postgres can hold the manifest and metadata, with a vector store alongside it. LangChain provides useful building blocks for document processing and retrieval, while LangGraph is a good fit for durable approval workflows. The knowledge-base and human-in-the-loop docs are relevant starting points.

The things I’d postpone are the knowledge graph and fully autonomous curation. Versioning, provenance, reliable synchronization, hybrid retrieval, and a small evaluation set will matter much more in the long run.

3

Wow!! Thank you, really, you just gave me a lot to study and think harder about!

4

@SalvatoreLarosa if this helped can you mark the answer as solution. Will be grateful.