Internal LLM Data Agent Platform

Problem

Seeing the data required knowing SQL. So questions piled up on whoever handled data, and that person’s time became the bottleneck. The questions themselves were mostly simple — what is this metric right now, what changed since last week.

What I did

I designed, built, deployed and operated this alone.

• Put a RAG knowledge base on top of 3-tier data routing, with 730 schema documents (95 tables · 635 columns) and a loop that auto-promotes real query logs into training examples.

• Decided the architecture shift by A/B measurement. Having the LLM classify each query into a single label for routing regressed accuracy from 0.86 to 0.69, so it was discarded. Exposing each capability as a tool and letting the LLM choose matched baseline while passing 5/5 compound queries — something the old structure could not do at all. It also asked for clarification on ambiguous questions 5/6 times versus 0/6.

  So I fixed the rationale for adoption as capability expansion, not accuracy, and recorded the fact that single-query accuracy cost 5pp. If you don’t write down the downside of what you adopted, that cost comes back later as a bug with no known cause.

• Designed fail-closed de-identification. When masking has not been applied, transmission to an external LLM is blocked outright. Row-level pre-masking plus a regex backstop gives two layers, catching identifiers that column context alone would miss.

Result

Everyone in the company uses it. I widened it to four surfaces — a remote MCP gateway (SSO, so people use it from their own AI clients), a Slack bot, a REST API, and an app API.

I also added active anomaly detection that scans periodically, attaches a root-cause analysis and sends an alert. It went from a tool that waits for questions to one that speaks first.