Threat Intelligence Data Platform — Ingest to Serving
A TI platform run end to end by a team of three: crawling, storage, search, API and UI. This is where I learned the difference between building a model and running one as a service.
Problem
In graduate school the job ended at the model. Someone else had prepared the data, and the result went into a paper.
What I met at SecuLetter was a situation where everything before and after that was empty. Where to crawl threat information from, how to clean it, where to store it, how to search it, what to expose it through, and how to draw it on screen — none of it was decided. With a team of three, there was nobody to hand a piece to.
What I did
Ingest and storage — built and operated the crawling service for the TI product (BeautifulSoup · Lambda · ECR). Stood up a data lake on S3 + OpenSearch for AI adoption, and added a Fluent-bit log store for cloud monitoring.
ETL and dashboard (2023.08 – 2024.07) — queried the raw data the crawlers had accumulated out of OpenSearch and transformed it into shapes the visualization needed. The core of it was making the relationships between mails, URLs and files queryable — a threat shows itself as a relationship, not as a single object. The monitoring UI was built in Svelte.
Model and serving (2023.01 – 2023.07 · 70% contribution) — implemented malicious mail classification on BERT-CNN, extracting features from mail headers to judge maliciousness. Serving was harder than the model — most of the time went into wiring FastAPI together with AWS Lambda, ECS and ECR.
Platform integration (2023.08 – 2024.03 · 60% contribution) — connected FastAPI to Lambda over VPC and configured API Gateway, then cleaned up the wiring between the TI platform and Lambda·ECS·ECR·OpenSearch.
Internal tooling (2022.08 – 2022.12) — built the training-data crawler (50% contribution) and ConTI, the internal data inspection platform (80% contribution). Svelte/Tailwind + Lambda/OpenSearch/FastAPI, deployed Bitbucket → ECR.
Result
What stayed with me was running every stage, from ingest to screen, with a small team. Being able to design, deploy and operate a system alone today rests on having gone through this.
Operating figures from that period were not retained. Daily ingest volume, data lake size and model performance cannot be recovered, so rather than invent them this is written without numbers. The figures on the other projects are all reproducible measurements.
Measurement
Operating figures from that period (daily ingest volume, data lake size, model performance) were not retained. Rather than invent them, this is left blank — the numbers on the other projects are all reproducible measurements.