diff --git a/.research/findings/tasks/web-001-1.md b/.research/findings/tasks/web-001-1.md new file mode 100644 index 0000000..05ca342 --- /dev/null +++ b/.research/findings/tasks/web-001-1.md @@ -0,0 +1,344 @@ +# Web Attack Detection Research Report + +**Date**: 2026-05-29 +**Status**: Research / Pre-implementation +**Scope**: Extending Mantis NIDS to cover application-layer web attack detection + +--- + +## 1. Executive Summary + +Mantis currently detects network-level attacks through two complementary engines: +- **ML engine**: Flow-level behavioral anomaly detection (~80 statistical features) +- **Suricata engine**: Signature matching via Suricata daemon over a veth mirror + +A gap analysis revealed that neither engine covers application-layer web attacks +(SQLi, XSS, LFI, web scanning) against HTTPS services effectively. + +This report documents: +1. Why flow-level ML cannot detect web attacks over HTTPS +2. What Suricata can cover today with the right ruleset +3. A proposed HTTP metadata collection pipeline for plaintext HTTP +4. A phased implementation plan + +--- + +## 2. Current Detection Capability + +### 2.1 ML Behavioral Engine + +The ML engine extracts ~80 per-flow features (packet lengths, inter-arrival times, +byte counts, bulk transfer rates, flag counts, active/idle periods) from the eBPF +packet stream and runs them through an LSTM autoencoder. + +**What it detects**: Behavioral anomalies at the flow level — port scans, DDoS +floods, beaconing, large data exfiltration. + +**What it cannot detect**: Application-layer attack payloads. A SQL injection +request (`SELECT * FROM users WHERE id=1 OR '1'='1'`) and a normal search query +have identical flow statistical signatures when sent over HTTPS. + +Feature group | Useful for web attacks? +---|--- +Packet length stats | No — HTTPS payload sizes overlap completely +IAT stats | Marginal — scanner tools may be faster, but not reliable +Bulk transfer metrics | No +TCP flag counts | No +Fwd/bwd byte ratio | No + +This is a fundamental limitation of Encrypted Traffic Analysis (ETA): without +decryption, flow-level classifiers cannot distinguish attack content from normal +HTTPS traffic. Published academic benchmarks (ISCX-URL2016, CIC-IDS2017) do not +include realistic HTTPS web attacks against the same service. + +### 2.2 Suricata Engine + +Mantis spawns a Suricata process and mirrors raw packets to it via a veth pair +(mantis-mirror / mantis-peer). Suricata reads an auto-generated YAML config and +matches alerts to rules in `RULE_PATH/*.rules`. + +Current config state: +- `app-layer.protocols.http.enabled: yes` with `detection-ports: dp: any` +- `app-layer.protocols.tls.enabled: yes` +- EVE JSON output via unix stream socket + +The HTTP app-layer dissector is **already active**. What is missing is a ruleset +that covers web attack signatures. + +--- + +## 3. Suricata + ET Open Rules (Immediate Win) + +### 3.1 Why ET Open, not OWASP CRS + +OWASP Core Rule Set (CRS) uses **ModSecurity SecRule format**, which is +incompatible with Suricata. Suricata uses Snort-style rules. + +**Emerging Threats Open (ET Open)** provides a free, Suricata-compatible ruleset +that includes dedicated web attack categories: + +Category | File | Covers +---|---|--- +Web server attacks | `emerging-web_server.rules` | SQLi, XSS, LFI, RFI, PHP injection, shell upload +Web client exploits | `emerging-web_client.rules` | Drive-by downloads, malicious JS +Exploit frameworks | `emerging-exploit.rules` | Metasploit, Shodan scan payloads +SQL injection | `emerging-sql.rules` | DB-specific injection patterns +Scanner signatures | `emerging-scan.rules` | Nikto, Nmap HTTP probes + +### 3.2 Integration + +ET Open rules are downloaded from: +`https://rules.emergingthreats.net/open/suricata-{version}/emerging.rules.tar.gz` + +The tarball contains individual `.rules` files that can be dropped directly into +`RULE_PATH/`. Suricata's existing config (`rule-files: ["*.rules"]`) picks them +up automatically — no code change required. + +**Required steps**: +1. Download and extract ET Open rules into `RULE_PATH/` +2. Test that Suricata loads without errors (check `/tmp/suricata.log`) +3. Tune noisy SIDs via the suppress config (same mechanism as current setup) + +**Limitations**: +- ET Open covers HTTP (plaintext). HTTPS payloads are encrypted; Suricata cannot + inspect them without a TLS MitM proxy (out of scope for an inline NIDS). +- Rules match known attack patterns; zero-day or obfuscated payloads evade signatures. + +--- + +## 4. HTTP Metadata Collection Pipeline + +### 4.1 Motivation + +To train a web-attack-aware ML model and to provide richer forensic data, we need +per-request HTTP metadata. This is fundamentally different from the existing flow CSV: + +| Dimension | Flow CSV (current) | HTTP request CSV (proposed) | +|---|---|---| +| Granularity | 1 row per flow | 1 row per HTTP request | +| Content | Statistical features | Parsed application-layer fields | +| HTTPS | Yes (opaque bytes) | No (plaintext only) | +| Use | Anomaly detection | Web attack classification | + +### 4.2 Data Source + +**eBPF uprobe on OpenSSL** is not viable in this deployment. Mantis operates as +an **inline NIDS between two NICs** — it sees raw network packets, not the TLS +library calls of other processes. Uprobes only work on TLS libraries loaded on +the same host. + +Plaintext HTTP traffic is captured by eBPF at the packet level and already +delivered to the ML engine via the `process_packet` path. The payload bytes are +available in the `payload` parameter of `flow_tracker.process_packet()`. + +### 4.3 Proposed CSV Schema + +| # | Field | Type | Description | +|---|---|---|---| +| 1 | `src_ip` | string | Client IP address | +| 2 | `dst_ip` | string | Server IP address | +| 3 | `src_port` | u16 | Client port (connection identifier) | +| 4 | `dst_port` | u16 | Server port (80, 8080, etc.) | +| 5 | `timestamp` | datetime | Request timestamp | +| 6 | `method` | string | HTTP method — TRACE/OPTIONS indicate scanning | +| 7 | `uri` | string | Full URI including query string; primary attack signal (SQLi, LFI, XSS) | +| 8 | `http_version` | string | HTTP/1.0 vs 1.1 — older version common in scanner tools | +| 9 | `host` | string | Host header — virtual-host confusion attacks | +| 10 | `user_agent` | string | Tool signatures: sqlmap, nikto, curl, python-requests | +| 11 | `referer` | string | Referer header — CSRF source tracing (empty string if absent) | +| 12 | `content_type` | string | Request Content-Type — multipart/form-data indicates upload attempts | +| 13 | `content_length` | u64 | Request body size — abnormally large POST suggests payload injection | +| 14 | `status_code` | u16 | Response status — 400/403/500 bursts indicate scanning | +| 15 | `response_content_type` | string | Response Content-Type | +| 16 | `response_content_length` | u64 | Response body size | + +**Excluded fields**: +- `cookie` content: contains session tokens; privacy risk. Record presence/length only if needed. +- `x-forwarded-for`: trivially forged by attackers; high noise, low signal. +- Request/response body: too large to store; `content_length` is sufficient for classification. + +### 4.4 Implementation Architecture + +``` +eBPF packet + │ + ▼ +xsk_manager.rs (existing packet pump) + │ + ├──► ML engine (flow stats, unchanged) + │ + └──► WebEngine (new, plaintext HTTP only) + │ + ▼ + TCP Reassembler (per connection buffer) + │ + ▼ + HTTP Parser (request + response pairing) + │ + ▼ + RequestLogger → per-request CSV + │ + ▼ + WebClassifier (future ML model) +``` + +### 4.5 Component Design + +#### TCP Reassembler + +Maintains a `HashMap` of per-connection byte buffers. +Handles out-of-order segments (sequence number tracking), connection teardown, +and memory limits (cap buffer at 64 KB per connection). + +Does NOT need to handle HTTPS — detect TLS by checking if first bytes are +`0x16 0x03` (TLS record type + version); skip those connections. + +#### HTTP Parser + +Minimal parser for the data we need. Full HTTP/1.x is sufficient; HTTP/2 over +plaintext (h2c) is rare in real networks and can be skipped initially. + +``` +Request: METHOD SP URI SP HTTP/version CRLF headers CRLF [body] +Response: HTTP/version SP status_code SP reason CRLF headers CRLF [body] +``` + +Pair requests to responses by connection + request order (HTTP/1.1 pipelining +is sequential on a single TCP connection). + +#### RequestLogger + +Analogous to the existing `TrafficLogger` but writes one row per request. +Rolling daily CSV files, same pattern as the flow CSV. + +New env var needed: `HTTP_LOG_PATH` (parallel to `CSV_RECORD_PATH`). + +New config flag: `http_logging_mode: bool` in `config.toml`. + +### 4.6 Proposed New Files + +``` +mantis/src/detection/web/ +├── mod.rs +├── engine.rs -- WebEngine (tokio task, receives packets from xsk_manager) +├── tcp_reassembler.rs -- per-connection byte stream reassembly +├── http_parser.rs -- HTTP/1.x request+response parser +├── request_features.rs-- RequestFeatures struct + to_csv_record() +└── request_logger.rs -- rolling CSV writer (reuse TrafficLogger pattern) +``` + +New model/log and model/error files: +``` +mantis/src/model/log/web.rs +mantis/src/model/error/web.rs +``` + +--- + +## 5. Web Attack ML Model (Future) + +Once HTTP request CSV data is collected from production traffic, a supervised +classifier can be trained on labelled samples (normal vs. attack). + +Suggested approach: + +1. **Feature engineering**: URI path depth, query parameter count, special char + density in URI, user-agent entropy, method frequency per source IP +2. **Model type**: Gradient boosted trees (XGBoost/LightGBM) for interpretability; + alternatively a small feedforward net via ONNX +3. **Training data**: Collect 1-2 weeks of normal HTTP traffic from the target + network. Attack samples: OWASP WebGoat / DVWA logs, public datasets + (CSIC 2010 HTTP, CAIDA) +4. **Serving**: Same `ort-tract` pattern as the existing autoencoder + +This is out of scope until the collection pipeline exists and data is available. + +--- + +## 6. Architecture Decisions + +### No WAF Integration + +A WAF (e.g., ModSecurity, Coraza) operates inline and can inspect/block HTTPS +after TLS termination at a reverse proxy. This requires the WAF to sit in front +of the protected services (separate deployment concern). Mantis is a passive NIDS +and is not the right place for WAF functionality. + +Recommendation: deploy a WAF at the application tier independently; Mantis covers +the network tier. + +### No TLS Decryption / MitM + +Decrypting HTTPS requires either a TLS MitM proxy (breaks end-to-end encryption, +requires CA cert installation on clients) or access to server private keys. Both +are out of scope for a passive NIDS. + +JA3 fingerprinting via Suricata TLS events (see TODO) provides the best +non-decryption signal for HTTPS. + +### No eBPF Uprobe for TLS + +eBPF uprobes on OpenSSL/BoringSSL could capture plaintext at the TLS library +boundary — but only for TLS connections originating or terminating on the same +host. Since Mantis is deployed as an inline NIDS bridging two NICs, it does not +run the TLS libraries of the services it monitors. + +--- + +## 7. Implementation Phases + +### Phase 1 — ET Open Rules (0 code changes, immediate) + +1. Download ET Open ruleset for installed Suricata version +2. Extract `.rules` files to `RULE_PATH/` +3. Test Suricata startup; suppress noisy SIDs via suppress config +4. Verify alerts appear in EVE socket (test with `curl` against a test URI + containing `../../etc/passwd`) + +**Deliverable**: Suricata detects common web attack patterns over plaintext HTTP. + +### Phase 2 — HTTP Metadata Collection + +1. Add `WebEngine` struct and tokio task +2. Implement `TcpReassembler` with sequence tracking and TLS skip +3. Implement `HttpParser` for request/response pairs +4. Implement `RequestLogger` (rolling CSV) +5. Wire into `xsk_manager.rs` packet pump +6. Add `http_logging_mode` config flag +7. Validate CSV output against known traffic + +**Deliverable**: Per-request CSV with method, URI, user-agent, status code fields. + +### Phase 3 — Web Attack ML Model + +1. Collect 2+ weeks of labelled HTTP traffic +2. Feature engineering on collected CSV +3. Train and evaluate classifier (target: F1 > 0.90 on hold-out) +4. Export to ONNX; load via ort-tract +5. Add `WebClassifier` component; integrate with `FusionEngine` + +**Deliverable**: ML-based web attack alerts in the unified alert stream. + +--- + +## 8. Open Questions + +1. **HTTP/2 h2c prevalence**: How much plaintext HTTP/2 is in the monitored network? + If significant, the HTTP parser must handle HPACK header compression (much more + complex). Defer until data shows it is needed. + +2. **Request-response pairing under pipelining**: HTTP/1.1 allows pipelining + (multiple requests before response). The parser must queue requests and pair + them in order. Simplification: treat connections as strictly serial for v1. + +3. **Memory cap for TCP reassembly**: 64 KB per connection × max_concurrent_connections + is the upper bound. Need to measure connection count in production to size correctly. + +4. **Minimum flow size for useful data**: Very short HTTP exchanges (< 3 requests) + may not yield enough data for reliable classification. Consider a minimum-request + threshold before emitting a row to the classifier. + +5. **Suricata HTTP alert redundancy**: Once ET Open rules are active, some web + attacks will be caught by Suricata signatures AND the future ML model. The + FusionEngine should handle this naturally (both sources fire → higher confidence + fusion alert), but tune deduplication window accordingly.