Security operations teams face a continuous stream of alerts. Triage requires more than plausible text: analysts need evidence, context, explicit uncertainty, and an unambiguous way to distinguish an AI recommendation from a verified security finding.
TriageLens
An alert triage pipeline that measures how often its AI is wrong.
30-second summary
TriageLens is an investigation pipeline uniting structured alert intake, threat intelligence enrichment, event correlation, detection engineering, and cloud AI triage—measuring prediction accuracy directly against independent ground truth.
The central question is not just what an LLM concludes, but how often that conclusion is wrong. Preserving failures as explicit outcomes and maintaining human analyst decision authority ensures trustworthy SOC workflows.
The challenge & the approach
The challenge
AI-generated security assessments can be fluent, plausible, and still completely wrong. Models can overlook subtle intrusion indicators, inflate benign system noise into urgent alarms, recommend dangerous host remediation, or exhibit false overconfidence unsupported by observed facts.
Furthermore, evaluating AI performance in security operations becomes fundamentally misleading if ground-truth labels leak into model prompts, or if timeouts, API errors, and model abstentions are swept under the rug instead of being accounted for as failure modes.
The approach
TriageLens strictly separates deterministic processing from AI assessment. Alerts from JSON, CSV, and EVTX are normalized, indicators extracted, threat intelligence attached, and deterministic timelines reconstructed from shared identifiers and temporal proximity.
Only sanitized evidence is transmitted to Groq for structured assessment. Predictions, independent reference labels, rule detections, and human analyst reviews are retained as four strictly separate records—ensuring complete traceability without letting AI grade its own work.
How it works
JSON/CSV/EVTX
& Enrichment
Correlation
Groq AI Triage
Sigma / YARA
Evaluation
Dashboard
System architecture
Ingests alerts from public datasets and live logs (JSON, CSV, EVTX). Validates schemas with Pydantic, extracts IOCs (IPs, domains, hashes), and strictly isolates ground-truth labels from inference payloads.
Integrates VirusTotal, AbuseIPDB, and URLhaus with rate-limiting and cache layers. Deterministic grouping binds related alerts by shared host, user, or indicator within temporal sliding windows.
Pattern-based masking scrubs organizational tokens before querying Groq (openai/gpt-oss-120b). Output schemas require structured findings, evidence references, and explicit uncertainty flags.
When ground-truth labels exist, an air-gapped evaluation layer calculates confusion-matrix metrics, Brier score, Expected Calibration Error, and Wilson confidence intervals without leakage.
Executes a curated pySigma subset and parameterized KQL-style hunting filters across telemetry, with bounded YARA-X file inspection and MITRE ATT&CK mapping alongside AI analysis.
Zero-dependency SQLite for fast local evaluation, with optional role-separated PostgreSQL/Alembic stack. FastAPI exposes type-safe endpoints to a modern React/TypeScript analyst dashboard.
Technical highlights
Structured Pydantic outputs, explicit uncertainty ratings, missing information flags, and citations linking every claim back to sanitized evidence snapshots.
Precision, recall, specificity, F1, false-positive and false-negative rates, Brier score, Expected Calibration Error, and Wilson confidence intervals.
Provider timeouts, schema validation rejections, model abstentions, and unknown labels remain completely transparent rather than defaulting to benign.
Curated Sigma rules, parameterized KQL-style hunting queries, bounded YARA-X scanning, and MITRE ATT&CK tactic/technique annotations on recorded hits.
Environment-managed credentials, pattern-based payload scrubbing, response validation gates, and independent ground-truth isolation.
Offline provider mocks, 1,330 backend tests, 88 dashboard tests, CI security checks, Bandit scanning, and automated software bills of materials (SBOMs).
Evaluation metrics & mathematical foundations
Evaluates independently labelled binary predictions. Zero-denominator conventions guard against divide-by-zero crashes. Abstentions and provider errors are excluded from binary denominators and reported distinctly.
Measures the mean squared difference between predicted disposition confidence c_i and actual outcome y_i ∈ {0, 1}. Heavily penalizes overconfident incorrect predictions.
Partitions predictions into M=5 confidence bins. Computes the weighted absolute difference between empirical accuracy and mean confidence to verify whether confidence values can be trusted.
Quantifies statistical sampling uncertainty under binomial assumptions without relying on Gaussian approximations on small sample batches, preventing premature performance claims.
Stack & engineering decisions
FastAPI provides asynchronous request handling, OpenAPI documentation, and strict schema validation for every ingestion and triage endpoint.
Why: Robust cybersecurity tooling ecosystem with predictable data validation.Static type contracts mirror backend Pydantic models for safe frontend consumption, with modular components for timelines, alerts, and evaluation charts.
Why: Prevents runtime UI failures when inspecting high-density incident records.Zero-configuration SQLite enables immediate local evaluation and test runs; Docker Compose adds role-separated PostgreSQL with Alembic migrations.
Why: Low hardware barrier for local research, scalable for multi-user SOC deployment.High-throughput cloud chat-completions API with temperature boundaries and JSON schema enforcement, removing local GPU dependencies.
Why: High tokens-per-second throughput enables responsive real-time alert triage.Pluggable intelligence clients extract IP, domain, URL, and file hash context, caching lookups to conserve quota.
Why: Prevents absent threat intelligence from being misconstrued as proof of benign intent.Provides rule-based and signature-driven detection alongside AI analysis, with mapped MITRE ATT&CK annotations.
Why: Bridges traditional detection engineering with modern LLM investigation pipelines.Portfolio code snippets
Selected presentation examples from the core pipeline engines demonstrating deterministic correlation, safe metric evaluation, bounded intake, and the REST API surface.
src/engine/timeline.py)
if host and user:
return (f"{host}:{user}", "confirmed", "shared_host_and_user")
if host:
return (f"host:{host}", "confirmed", "shared_host")
if user:
return (f"user:{user}", "confirmed", "shared_user")
if alert.extracted_indicators:
return (f"indicator:{alert.extracted_indicators[0]}",
"confirmed", "shared_indicator")
return ("ungrouped", "potential", "time_proximity_fallback")
Why it matters: Deterministic entity grouping makes the exact reason for correlation transparent. Shared identifiers provide linkage without falsely claiming proof of an incident.
src/engine/evaluation.py)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
specificity = tn / (tn + fp) if (tn + fp) > 0 else None
f1 = (2 * precision * recall) / (precision + recall) \
if (precision + recall) > 0 else 0.0
Why it matters: Scores clearly expose failure modes and prevent division-by-zero crashes. The complete evaluator reports validity, coverage, and unavailable outcomes.
src/engine/evtx.py)
from dataclasses import asdict
from src.engine.evtx import EvtxIntakeEngine
intake = EvtxIntakeEngine()
for batch in intake.ingest_evtx(r"C:\soc-data\events.evtx", batch_size=100):
print("Alerts in batch:", len(batch))
print(asdict(intake.stats))
Why it matters: Intake emits bounded batches and reports parsing errors gracefully. Prevents memory exhaustion when processing enterprise Windows event logs.
GET /health
GET /api/alerts
GET /api/timelines/{timeline_id}
GET /api/metrics
POST /api/hunt
POST /api/timelines/{timeline_id}/analyst-review
Why it matters: Interactive Swagger/OpenAPI documentation is served at /docs. Enforces typed request/response contracts for both CLI tools and web dashboard.
Outcome & honest limitations
Recorded offline verification includes 1,330 backend tests in the T5.4 environment and 88 dashboard tests in the T5.3 environment using complete offline provider mocks.
Selected six-case synthetic tests are documented in the repository, but are intentionally selective. Reviewed real-data evaluation and live acceptance of active triage policies remain an ongoing process.
MITRE ATT&CK annotation coverage is not detection effectiveness. Strict schema validation and evidence references ensure structured outputs, but do not guarantee factual truth.
Run & test workflow
Backend verification (PowerShell)
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
if (-not (Test-Path -LiteralPath .env)) { Copy-Item .env.example .env }
python -m pytest -q
Tests run with mock providers by default to avoid quota consumption and network dependency during automated test runs.
Dashboard checks
cd dashboard
npm ci
npm test
npm run build
Verifies React component mounting, Axios API client contracts, and TypeScript builds with zero compiler warnings.
Project outcome