Company Knowledge Base — Privacy-Safe Compilation Pipeline

Problem

The answers were in accumulated internal conversations, but nobody could find them. The usual prescription is to embed the raw conversations and retrieve over them. The trouble is that raw conversation contradicts itself, gets reversed, and drops context. Answering from unverified raw text produces confident wrong answers.

And those conversations carry identifiers for people, farms and animals. They could not be sent to an external LLM as-is.

What I did

I went with compilation rather than retrieval. Answers do not come from the raw text; they come from documents an LLM reads the raw text to maintain.

Pipeline: channel ingest → two de-identification layers (regex + GPU NER) → LLM compilation → wiki → agent queries

• Tuned de-identification precision in both directions. Plaintext identification numbers went from 181 to 0, and farm names became one-way hash codes (135 unique) instead of collapsing into a single placeholder, so documents can still tell them apart. In the other direction, I loosened rules that masked things that should stay visible — a public livestock-grading agency was being masked as a farm because its name contains “livestock”. Over-masking makes the documents useless. Both directions were written down as rules and pinned with 20 regression asserts.
• Automated compilation as a weekly batch. Manual compilation had sat stalled for two weeks, and in the meantime answers had quietly degraded to unverified raw text. Automation stops at a draft branch, and success sends an alert too.
• Decided that final review and merge stay human. NER accuracy sits at 90–95%, so automating the last gate on personal data was not the right call.

Result

The line between what gets automated and what a human catches was drawn with an accuracy number, not a hunch.

The first backlog compile (12 min · $7.34) took the wiki from 5 to 13 pages, and review found 0/6 personal-data leaks. Questions are answered by citing compiled pages, not by searching raw text. Ingest and de-identification still run daily; only the paid weekly compile is paused for now. The most important decision in this project was drawing the line between what gets automated and what a human catches using an accuracy number, not a hunch.