A BaseStore backed by a filesystem: which search filters do you actually use?

Talking Shop
1

Hi all,

I maintain MoobFS, an experimental, GPL-2.0 Linux kernel module: a stacked filesystem over ext4, btrfs or XFS where each memory is a file, its structured fields are typed extended attributes, and the kernel answers filter queries over them, scoped by ordinary file permissions. It was built with AI coding assistants, has one maintainer and has had no outside review, so please read everything below as a research prototype, not something to deploy.

To check that the design was not fitted to a single client, I wrote a BaseStore on top of it (namespace labels are directories, a key is a file, the value is the file’s JSON, each top-level scalar is an indexed attribute) and ran the same random operations against it and against InMemoryStore, comparing every get, search and list_namespaces. Over 18 000 operations and about 3 300 searches, with twelve filter shapes, the two agreed everywhere I compared them. Thank you for a store interface small enough that this took a day.

Three things I learnt that may be useful here:

  1. About a third of the filter conditions my test generated could not be pushed down to the index: $ne, list values, nested objects, and keys that are not simple names. The adapter evaluates those itself after reading the items.

  2. InMemoryStore raises TypeError when a range filter ($gt, $lt, …) meets an item that does not have the field, because of a float(None). My adapter never sees such items, so the two stores differ there: one raises, the other answers. I do not know if raising is intended; if it is not, I can open an issue with a minimal reproduction.

  3. InMemoryStore keeps listing a namespace after its last item is deleted. Same question: intended or not?

And the question I came to ask: in real applications, which search filters do you use? I am trying to decide what a storage layer must answer natively and what can stay in the client. Equality, ranges and prefixes on scalar fields are covered. If you rely on $ne, on membership in a list, on nested fields, or on something else, I would like to hear the concrete case.

Code and the test: GitHub - studiomoob/llmoobfs: Experimental Linux kernel module (GPL-2.0, no FUSE): a stacked filesystem layer over ext4, btrfs or XFS that adds typed xattr attributes, an in-memory index and a QUERY ioctl authorized by the caller's POSIX credentials. Intended as storage for LLM-agent memory. Research project, not for production. · GitHub (bench/client/langgraph_store.py, bench/client/langgraph-equality.py; the write-up is docs/history/0089). No latency numbers for this adapter yet, and it has not been run under LangGraph’s own test suite.

Claudio

2

Thanks for doing the differential test against InMemoryStore. That’s a useful way to pin down the contract.

From the agent-memory patterns I’ve seen, roughly in order of how much they matter:

  1. Namespace prefix scoping does most of the work. ("users", user_id, "memories"), sometimes with an extra org_id or agent_id level. Most search calls are “everything under this user’s prefix”, and list_namespaces(prefix=..., max_depth=...) is how people enumerate users or sessions. If that’s native and permission-aware, you’ve covered the common case.
  2. Equality on a small set of scalar fields: type (fact / preference / episode), status (active / superseded), source or agent_id. Superseding facts as insert-new + mark-old makes status == "active" almost universal.
  3. Ranges on time fields: valid_from / created_at with $gte/$lt for “recent” or “as of” queries. Because of your point 2, I’d write every time field on every item so the missing-field case never comes up.
  4. Semantic query= plus a metadata filter. In practice this is the main path, and the metadata filter is mostly 1–2 above. Native vector search matters more than richer filter operators.

Rarely needed: $ne (usually expressible as a positive status/type equality), and nested-field filters (people flatten what they filter on to top level). List membership (e.g. tags contains X) does come up for tag-style memories. If you add one non-scalar operator, I’d pick that one, and leave $ne and nested filters to client-side evaluation as you do now.

On your two questions: a range filter raising on a missing field looks like a bug rather than intended behaviour, and the same goes for an empty namespace still being listed. Both seem worth an issue with your minimal repro.

3

Thanks, this helps. The status == "active" example makes the common filter much more concrete. I’ll keep semantic search in user space and look at tag membership separately.

Have you seen keeping a metadata index in sync with memory files become a problem in practice, or do the existing stores usually handle it well?

I’ll check the two InMemoryStore behaviors separately before filing anything.

Thanks,
Claudio

4

In my experience the drift shows up wherever the item and its index entry are written in separate steps, and it rarely comes from the happy path. The cases that bite:

  1. Partial writes. The item is saved but the index update (often an embedding call to an external API) fails or times out. The item then exists but can’t be found by search.
  2. Deletes and overwrites. The item is removed or replaced, but its old index entries stay behind, so stale memories keep coming back in search results. For agent memory that’s worse than a miss, because the model treats them as current.
  3. Out-of-band edits. Something changes the data without going through the layer that maintains the index.

What I’d do about it:

  • Treat the item as the source of truth and the index as rebuildable. Store a version or content hash on each item next to its index entry, and re-index anything where the two differ.
  • Run a periodic reconciliation job that compares keys in the data and in the index, and fixes both directions (missing entries and orphaned ones). It’s cheap and catches the cases above.
  • Make deletes remove the index entries first, or in the same transaction where the backend allows it.

In your design the third case seems the interesting one: a process writing straight to the lower ext4/XFS bypasses your layer. Also look at crash consistency between the file write and the in-memory index update. If the index is rebuilt from the xattrs at mount time, that already gives you a clean recovery path.

5

Thanks, that’s a helpful breakdown. I checked the implementation against the cases you raised. Writes and deletes through the mount update the metadata index in the same code path. If an index update fails after a lower write, the affected domain is rebuilt before `QUERY2` answers. The index is RAM-only, so after a crash it is rebuilt from the surviving xattrs on the first query, rather than at mount time. Our client uses a temporary file, `fsync`, `rename`, and directory `fsync`; we exercised that sequence with power-loss replay.

For the separate vector cache, our prototype stores content hashes, checks candidates against the files, and reconciles after restart or event overflow. That still needs testing with a real external embedding failure.

You’re right about direct edits to the lower tree while mounted: they bypass the layer’s tracking. For now, writes go through the mount and lower-tree maintenance happens while unmounted. We tested `fanotify` on the lower ext4 filesystem as a possible change feed, but live invalidation would need more work. I’ll include these cases in the files-plus-SQLite comparison.

Thanks again,
Claudio