Company Knowledge Base — Privacy-Safe Compilation Pipeline

Problem

The answers were in accumulated internal conversations, but nobody could find them. The usual prescription is to embed the raw conversations and retrieve over them. The trouble is that raw conversation contradicts itself, gets reversed, and drops context. Answering from unverified raw text produces confident wrong answers.

And those conversations carry personal and institutional identifiers. They could not be sent to an external LLM as-is.

What I did

I went with compilation rather than retrieval. Answers do not come from the raw text; they come from documents an LLM reads the raw text to maintain.

Pipeline: channel ingest → two de-identification layers (regex + GPU NER) → LLM compilation → wiki → agent queries

• Tuned de-identification precision in both directions. I tightened rules so identification numbers never leave in plaintext, and replaced institution names with one-way hash codes. In the other direction, I loosened rules that over-masked public information — over-masking makes the documents useless. Both directions were written down as rules.
• Added API key failover, but limited the switching signal to 403 + billing_error. Falling over on any error would either burn the spare key or mistake a real outage for normal operation.
• Decided that final review and merge stay human. NER accuracy falls short of 95%, so automating the last gate on personal data was not the right call.

Result

A wiki that answers from compiled documents is running. The most important decision in this project was drawing the line between what gets automated and what a human catches using an accuracy number, not a hunch.