Data Engineering Intern

Overview

I interned on the data engineering team building the data-collection pipeline behind a Cybersecurity Small Language Model. The goal was to turn scattered, messy cybersecurity data into a clean, well-structured dataset an SLM could actually learn from — everything from figuring out what to collect, to ingesting it securely, cleaning it, and shaping it into a final training-ready format.

What We Built

The pipeline moved data through a series of clearly separated stages:

• Categorization: We started with a sheet mapping all the data into different cybersecurity subdomains, so every source had a clear home before anything was ingested
• Secure Ingestion: Python scripts pulled the data in securely and landed it in a raw_data folder, keeping the original sources intact
• Four-Step Cleaning: Each record ran through sanitization, deduplication, PII removal, and a language filter before being written to a clean_data folder
• Quality Analysis: A parallel analysis pass flagged anything unfit and routed it to a separate folder, so bad data never leaked into the final set
• Schema & Dataset: We defined a schema and followed it to assemble the final dataset as a JSONL file — consistent, structured, and ready for training
• Streamlit Frontend: The dataset was surfaced through an interactive Streamlit interface backed by an agentic LLM, making the data explorable rather than just a file on disk
• Research Paper: We documented the entire process end-to-end in a research paper covering the methodology and decisions behind the pipeline

Skills Developed

Takeaway

Owning a single stage of a real pipeline taught me how much careful, unglamorous work goes into good data — and why clean inputs matter so much for anything you train downstream. Splitting the flow into distinct, well-defined stages made the whole system easier to reason about, and writing it all up afterward forced me to understand every decision we'd made.