Internal LLM Data Agent Platform

Problem

Seeing the data required knowing SQL. So questions piled up on whoever handled data, and that person’s time became the bottleneck. The questions themselves were mostly simple — what is this metric right now, what changed since last week.

What I did

I designed, built, deployed and operated this alone.

• Put a RAG knowledge base on top of 3-tier data routing (MySQL / PostgreSQL / ClickHouse + fallback). It started at MVP with 730 schema documents (95 tables · 635 columns) and has grown with more data sources, plus a loop that auto-promotes real query logs into training examples.

• Decided the architecture shift by A/B measurement. Having the LLM classify each query into a single label for routing regressed table-jaccard from 0.86 to 0.69, so it was discarded. Exposing each capability as a tool and letting the LLM choose kept execution success at 47/47, the same as before, while passing 5/5 compound queries — something the old structure could not do at all. On questions with an unresolved reference it asked back about 5/6 times (previously 0/6).

  The cost: single-query table-jaccard dropped 5–7pp, because the agent over-decomposes questions, and prompting didn’t fix it. So I fixed the rationale for adoption as capability expansion, not accuracy, and recorded that cost. If you don’t write down the downside of what you adopted, that cost comes back later as a bug with no known cause.

• Designed fail-closed de-identification. When masking has not been applied, transmission to an external LLM is blocked outright. Row-level pre-masking plus a regex backstop gives two layers, catching identifiers that column context alone would miss.

Result

If you don’t write down the downside of what you adopted, that cost comes back later as a bug with no known cause.

I widened it to four surfaces so anyone in the company can use it — a remote MCP gateway (SSO, so people use it from their own AI clients), a Slack bot, a REST API, and an app API. Active anomaly detection — a 30-minute scan that attaches a root-cause analysis to each alert — ran live, and is paused for now while I revisit thresholds and deduplication.

The biggest lesson from operating it was silent failure. When the API credit ran out, the health check only verified that a key existed, so it kept returning 200 ok — nobody noticed for 4 hours 41 minutes. The exhaustion signal also arrived as 400, not what the docs implied, so the prepared backup-key failover never fired. I deleted that unverified failover path, switched health to a real call (300 s TTL), and added a 5-minute outage alert.