knowledge-agent

Problem#

Company knowledge lived in two places — Notion and Google Drive — and finding anything meant knowing where to look. Keyword search failed on paraphrased questions, and every "just ask in Slack" round-trip cost someone else's focus. The goal: ask a question in natural language, get an answer with sources, and never see a document you don't have permission to read.

That last clause is the actual problem. RAG demos ignore permissions; a production system over internal knowledge cannot.

Constraints#

• Permissions are not post-filters. Filtering retrieved chunks after the fact leaks information through the answer even when the source link is hidden. Access control had to be enforced inside retrieval.
• Sources change under you. Notion pages are edited continuously; the index had to tolerate partial staleness and re-ingest incrementally.
• No answer without a source. If retrieval comes back thin, the correct output is "I don't know, here's where I looked" — not a fluent guess.
• Every change must prove itself. Chunking, embeddings, rerankers — nothing merges without beating the current pipeline on the eval set.

Architecture#

 Notion ─┐                       ┌─ ACL snapshot (per-user doc grants)
         ├─ ingest workers ──► chunk + embed ──► vector index
 Drive ──┘        │                                  │
                  └── change feed (incremental)      │
                                                     ▼
 user question ──► auth ──► hybrid retrieval ──► rerank ──► answer + sources
                    │       (dense + BM25,                      │
                    └─────── ACL filter in-query) ◄─────────────┘

Ingestion workers pull from the Notion and Drive APIs, chunk by document structure (headings, not fixed windows), and write embeddings alongside an ACL snapshot — the set of principals allowed to see each document. At query time, the user's identity resolves to their grant set and the ACL predicate is part of the retrieval query itself, so unauthorized chunks are never scored, never retrieved, never in the context window.

Retrieval is hybrid: dense vectors for paraphrase, BM25 for exact terms (project codenames, error strings). A cross-encoder reranks the merged list. The answer step is forced to cite: each claim links back to the chunk it came from, and an answer with no supporting chunk is rejected.

Results#

• Answers carry clickable source links; the "no answer without a source" rule holds by construction, not by prompt.
• Zero permission leaks across the adversarial slice of the eval set — questions specifically crafted to fish for documents the asking user cannot read.
• The eval set (a few hundred question–answer–source triples) gates CI. Two "obvious improvements" — larger chunks, a newer embedding model — were rejected because they regressed retrieval quality. That is the eval set paying rent.
• Incremental ingestion keeps the index minutes behind the sources instead of a nightly batch behind.

The longer write-ups of the individual decisions — chunking, hybrid retrieval weights, the eval methodology — are on the writing page.