research: Add web attack detection report for claude

.research/findings/tasks/web-001-1.md

@@ -0,0 +1,344 @@
+# Web Attack Detection Research Report
+
+**Date**: 2026-05-29  
+**Status**: Research / Pre-implementation  
+**Scope**: Extending Mantis NIDS to cover application-layer web attack detection
+
+---
+
+## 1. Executive Summary
+
+Mantis currently detects network-level attacks through two complementary engines:
+- **ML engine**: Flow-level behavioral anomaly detection (~80 statistical features)
+- **Suricata engine**: Signature matching via Suricata daemon over a veth mirror
+
+A gap analysis revealed that neither engine covers application-layer web attacks
+(SQLi, XSS, LFI, web scanning) against HTTPS services effectively.
+
+This report documents:
+1. Why flow-level ML cannot detect web attacks over HTTPS
+2. What Suricata can cover today with the right ruleset
+3. A proposed HTTP metadata collection pipeline for plaintext HTTP
+4. A phased implementation plan
+
+---
+
+## 2. Current Detection Capability
+
+### 2.1 ML Behavioral Engine
+
+The ML engine extracts ~80 per-flow features (packet lengths, inter-arrival times,
+byte counts, bulk transfer rates, flag counts, active/idle periods) from the eBPF
+packet stream and runs them through an LSTM autoencoder.
+
+**What it detects**: Behavioral anomalies at the flow level — port scans, DDoS
+floods, beaconing, large data exfiltration.
+
+**What it cannot detect**: Application-layer attack payloads. A SQL injection
+request (`SELECT * FROM users WHERE id=1 OR '1'='1'`) and a normal search query
+have identical flow statistical signatures when sent over HTTPS.
+
+Feature group | Useful for web attacks?
+---|---
+Packet length stats | No — HTTPS payload sizes overlap completely
+IAT stats | Marginal — scanner tools may be faster, but not reliable
+Bulk transfer metrics | No
+TCP flag counts | No
+Fwd/bwd byte ratio | No
+
+This is a fundamental limitation of Encrypted Traffic Analysis (ETA): without
+decryption, flow-level classifiers cannot distinguish attack content from normal
+HTTPS traffic. Published academic benchmarks (ISCX-URL2016, CIC-IDS2017) do not
+include realistic HTTPS web attacks against the same service.
+
+### 2.2 Suricata Engine
+
+Mantis spawns a Suricata process and mirrors raw packets to it via a veth pair
+(mantis-mirror / mantis-peer). Suricata reads an auto-generated YAML config and
+matches alerts to rules in `RULE_PATH/*.rules`.
+
+Current config state:
+- `app-layer.protocols.http.enabled: yes` with `detection-ports: dp: any`
+- `app-layer.protocols.tls.enabled: yes`
+- EVE JSON output via unix stream socket
+
+The HTTP app-layer dissector is **already active**. What is missing is a ruleset
+that covers web attack signatures.
+
+---
+
+## 3. Suricata + ET Open Rules (Immediate Win)
+
+### 3.1 Why ET Open, not OWASP CRS
+
+OWASP Core Rule Set (CRS) uses **ModSecurity SecRule format**, which is
+incompatible with Suricata. Suricata uses Snort-style rules.
+
+**Emerging Threats Open (ET Open)** provides a free, Suricata-compatible ruleset
+that includes dedicated web attack categories:
+
+Category | File | Covers
+---|---|---
+Web server attacks | `emerging-web_server.rules` | SQLi, XSS, LFI, RFI, PHP injection, shell upload
+Web client exploits | `emerging-web_client.rules` | Drive-by downloads, malicious JS
+Exploit frameworks | `emerging-exploit.rules` | Metasploit, Shodan scan payloads
+SQL injection | `emerging-sql.rules` | DB-specific injection patterns
+Scanner signatures | `emerging-scan.rules` | Nikto, Nmap HTTP probes
+
+### 3.2 Integration
+
+ET Open rules are downloaded from:
+`https://rules.emergingthreats.net/open/suricata-{version}/emerging.rules.tar.gz`
+
+The tarball contains individual `.rules` files that can be dropped directly into
+`RULE_PATH/`. Suricata's existing config (`rule-files: ["*.rules"]`) picks them
+up automatically — no code change required.
+
+**Required steps**:
+1. Download and extract ET Open rules into `RULE_PATH/`
+2. Test that Suricata loads without errors (check `/tmp/suricata.log`)
+3. Tune noisy SIDs via the suppress config (same mechanism as current setup)
+
+**Limitations**:
+- ET Open covers HTTP (plaintext). HTTPS payloads are encrypted; Suricata cannot
+  inspect them without a TLS MitM proxy (out of scope for an inline NIDS).
+- Rules match known attack patterns; zero-day or obfuscated payloads evade signatures.
+
+---
+
+## 4. HTTP Metadata Collection Pipeline
+
+### 4.1 Motivation
+
+To train a web-attack-aware ML model and to provide richer forensic data, we need
+per-request HTTP metadata. This is fundamentally different from the existing flow CSV:
+
+| Dimension | Flow CSV (current) | HTTP request CSV (proposed) |
+|---|---|---|
+| Granularity | 1 row per flow | 1 row per HTTP request |
+| Content | Statistical features | Parsed application-layer fields |
+| HTTPS | Yes (opaque bytes) | No (plaintext only) |
+| Use | Anomaly detection | Web attack classification |
+
+### 4.2 Data Source
+
+**eBPF uprobe on OpenSSL** is not viable in this deployment. Mantis operates as
+an **inline NIDS between two NICs** — it sees raw network packets, not the TLS
+library calls of other processes. Uprobes only work on TLS libraries loaded on
+the same host.
+
+Plaintext HTTP traffic is captured by eBPF at the packet level and already
+delivered to the ML engine via the `process_packet` path. The payload bytes are
+available in the `payload` parameter of `flow_tracker.process_packet()`.
+
+### 4.3 Proposed CSV Schema
+
+| # | Field | Type | Description |
+|---|---|---|---|
+| 1 | `src_ip` | string | Client IP address |
+| 2 | `dst_ip` | string | Server IP address |
+| 3 | `src_port` | u16 | Client port (connection identifier) |
+| 4 | `dst_port` | u16 | Server port (80, 8080, etc.) |
+| 5 | `timestamp` | datetime | Request timestamp |
+| 6 | `method` | string | HTTP method — TRACE/OPTIONS indicate scanning |
+| 7 | `uri` | string | Full URI including query string; primary attack signal (SQLi, LFI, XSS) |
+| 8 | `http_version` | string | HTTP/1.0 vs 1.1 — older version common in scanner tools |
+| 9 | `host` | string | Host header — virtual-host confusion attacks |
+| 10 | `user_agent` | string | Tool signatures: sqlmap, nikto, curl, python-requests |
+| 11 | `referer` | string | Referer header — CSRF source tracing (empty string if absent) |
+| 12 | `content_type` | string | Request Content-Type — multipart/form-data indicates upload attempts |
+| 13 | `content_length` | u64 | Request body size — abnormally large POST suggests payload injection |
+| 14 | `status_code` | u16 | Response status — 400/403/500 bursts indicate scanning |
+| 15 | `response_content_type` | string | Response Content-Type |
+| 16 | `response_content_length` | u64 | Response body size |
+
+**Excluded fields**:
+- `cookie` content: contains session tokens; privacy risk. Record presence/length only if needed.
+- `x-forwarded-for`: trivially forged by attackers; high noise, low signal.
+- Request/response body: too large to store; `content_length` is sufficient for classification.
+
+### 4.4 Implementation Architecture
+
+```
+eBPF packet
+    │
+    ▼
+xsk_manager.rs  (existing packet pump)
+    │
+    ├──► ML engine (flow stats, unchanged)
+    │
+    └──► WebEngine (new, plaintext HTTP only)
+              │
+              ▼
+         TCP Reassembler (per connection buffer)
+              │
+              ▼
+         HTTP Parser (request + response pairing)
+              │
+              ▼
+         RequestLogger → per-request CSV
+              │
+              ▼
+         WebClassifier (future ML model)
+```
+
+### 4.5 Component Design
+
+#### TCP Reassembler
+
+Maintains a `HashMap<FlowKey, TcpStream>` of per-connection byte buffers.
+Handles out-of-order segments (sequence number tracking), connection teardown,
+and memory limits (cap buffer at 64 KB per connection).
+
+Does NOT need to handle HTTPS — detect TLS by checking if first bytes are
+`0x16 0x03` (TLS record type + version); skip those connections.
+
+#### HTTP Parser
+
+Minimal parser for the data we need. Full HTTP/1.x is sufficient; HTTP/2 over
+plaintext (h2c) is rare in real networks and can be skipped initially.
+
+```
+Request:  METHOD SP URI SP HTTP/version CRLF headers CRLF [body]
+Response: HTTP/version SP status_code SP reason CRLF headers CRLF [body]
+```
+
+Pair requests to responses by connection + request order (HTTP/1.1 pipelining
+is sequential on a single TCP connection).
+
+#### RequestLogger
+
+Analogous to the existing `TrafficLogger` but writes one row per request.
+Rolling daily CSV files, same pattern as the flow CSV.
+
+New env var needed: `HTTP_LOG_PATH` (parallel to `CSV_RECORD_PATH`).
+
+New config flag: `http_logging_mode: bool` in `config.toml`.
+
+### 4.6 Proposed New Files
+
+```
+mantis/src/detection/web/
+├── mod.rs
+├── engine.rs          -- WebEngine (tokio task, receives packets from xsk_manager)
+├── tcp_reassembler.rs -- per-connection byte stream reassembly
+├── http_parser.rs     -- HTTP/1.x request+response parser
+├── request_features.rs-- RequestFeatures struct + to_csv_record()
+└── request_logger.rs  -- rolling CSV writer (reuse TrafficLogger pattern)
+```
+
+New model/log and model/error files:
+```
+mantis/src/model/log/web.rs
+mantis/src/model/error/web.rs
+```
+
+---
+
+## 5. Web Attack ML Model (Future)
+
+Once HTTP request CSV data is collected from production traffic, a supervised
+classifier can be trained on labelled samples (normal vs. attack).
+
+Suggested approach:
+
+1. **Feature engineering**: URI path depth, query parameter count, special char
+   density in URI, user-agent entropy, method frequency per source IP
+2. **Model type**: Gradient boosted trees (XGBoost/LightGBM) for interpretability;
+   alternatively a small feedforward net via ONNX
+3. **Training data**: Collect 1-2 weeks of normal HTTP traffic from the target
+   network. Attack samples: OWASP WebGoat / DVWA logs, public datasets
+   (CSIC 2010 HTTP, CAIDA)
+4. **Serving**: Same `ort-tract` pattern as the existing autoencoder
+
+This is out of scope until the collection pipeline exists and data is available.
+
+---
+
+## 6. Architecture Decisions
+
+### No WAF Integration
+
+A WAF (e.g., ModSecurity, Coraza) operates inline and can inspect/block HTTPS
+after TLS termination at a reverse proxy. This requires the WAF to sit in front
+of the protected services (separate deployment concern). Mantis is a passive NIDS
+and is not the right place for WAF functionality.
+
+Recommendation: deploy a WAF at the application tier independently; Mantis covers
+the network tier.
+
+### No TLS Decryption / MitM
+
+Decrypting HTTPS requires either a TLS MitM proxy (breaks end-to-end encryption,
+requires CA cert installation on clients) or access to server private keys. Both
+are out of scope for a passive NIDS.
+
+JA3 fingerprinting via Suricata TLS events (see TODO) provides the best
+non-decryption signal for HTTPS.
+
+### No eBPF Uprobe for TLS
+
+eBPF uprobes on OpenSSL/BoringSSL could capture plaintext at the TLS library
+boundary — but only for TLS connections originating or terminating on the same
+host. Since Mantis is deployed as an inline NIDS bridging two NICs, it does not
+run the TLS libraries of the services it monitors.
+
+---
+
+## 7. Implementation Phases
+
+### Phase 1 — ET Open Rules (0 code changes, immediate)
+
+1. Download ET Open ruleset for installed Suricata version
+2. Extract `.rules` files to `RULE_PATH/`
+3. Test Suricata startup; suppress noisy SIDs via suppress config
+4. Verify alerts appear in EVE socket (test with `curl` against a test URI
+   containing `../../etc/passwd`)
+
+**Deliverable**: Suricata detects common web attack patterns over plaintext HTTP.
+
+### Phase 2 — HTTP Metadata Collection
+
+1. Add `WebEngine` struct and tokio task
+2. Implement `TcpReassembler` with sequence tracking and TLS skip
+3. Implement `HttpParser` for request/response pairs
+4. Implement `RequestLogger` (rolling CSV)
+5. Wire into `xsk_manager.rs` packet pump
+6. Add `http_logging_mode` config flag
+7. Validate CSV output against known traffic
+
+**Deliverable**: Per-request CSV with method, URI, user-agent, status code fields.
+
+### Phase 3 — Web Attack ML Model
+
+1. Collect 2+ weeks of labelled HTTP traffic
+2. Feature engineering on collected CSV
+3. Train and evaluate classifier (target: F1 > 0.90 on hold-out)
+4. Export to ONNX; load via ort-tract
+5. Add `WebClassifier` component; integrate with `FusionEngine`
+
+**Deliverable**: ML-based web attack alerts in the unified alert stream.
+
+---
+
+## 8. Open Questions
+
+1. **HTTP/2 h2c prevalence**: How much plaintext HTTP/2 is in the monitored network?
+   If significant, the HTTP parser must handle HPACK header compression (much more
+   complex). Defer until data shows it is needed.
+
+2. **Request-response pairing under pipelining**: HTTP/1.1 allows pipelining
+   (multiple requests before response). The parser must queue requests and pair
+   them in order. Simplification: treat connections as strictly serial for v1.
+
+3. **Memory cap for TCP reassembly**: 64 KB per connection × max_concurrent_connections
+   is the upper bound. Need to measure connection count in production to size correctly.
+
+4. **Minimum flow size for useful data**: Very short HTTP exchanges (< 3 requests)
+   may not yield enough data for reliable classification. Consider a minimum-request
+   threshold before emitting a row to the classifier.
+
+5. **Suricata HTTP alert redundancy**: Once ET Open rules are active, some web
+   attacks will be caught by Suricata signatures AND the future ML model. The
+   FusionEngine should handle this naturally (both sources fire → higher confidence
+   fusion alert), but tune deduplication window accordingly.