Mantis/.research/findings/tasks/web-001-1.md

Web Attack Detection Research Report

Date: 2026-05-29
Status: Research / Pre-implementation
Scope: Extending Mantis NIDS to cover application-layer web attack detection

---

1. Executive Summary

Mantis currently detects network-level attacks through two complementary engines:

• ML engine: Flow-level behavioral anomaly detection (~80 statistical features)
• Suricata engine: Signature matching via Suricata daemon over a veth mirror

A gap analysis revealed that neither engine covers application-layer web attacks (SQLi, XSS, LFI, web scanning) against HTTPS services effectively.

This report documents:

1. Why flow-level ML cannot detect web attacks over HTTPS
2. What Suricata can cover today with the right ruleset
3. A proposed HTTP metadata collection pipeline for plaintext HTTP
4. A phased implementation plan

---

2. Current Detection Capability

2.1 ML Behavioral Engine

The ML engine extracts ~80 per-flow features (packet lengths, inter-arrival times, byte counts, bulk transfer rates, flag counts, active/idle periods) from the eBPF packet stream and runs them through an LSTM autoencoder.

What it detects: Behavioral anomalies at the flow level — port scans, DDoS floods, beaconing, large data exfiltration.

What it cannot detect: Application-layer attack payloads. A SQL injection request (SELECT * FROM users WHERE id=1 OR '1'='1') and a normal search query have identical flow statistical signatures when sent over HTTPS.

Feature group	Useful for web attacks?
Packet length stats	No — HTTPS payload sizes overlap completely
IAT stats	Marginal — scanner tools may be faster, but not reliable
Bulk transfer metrics	No
TCP flag counts	No
Fwd/bwd byte ratio	No

This is a fundamental limitation of Encrypted Traffic Analysis (ETA): without decryption, flow-level classifiers cannot distinguish attack content from normal HTTPS traffic. Published academic benchmarks (ISCX-URL2016, CIC-IDS2017) do not include realistic HTTPS web attacks against the same service.

2.2 Suricata Engine

Mantis spawns a Suricata process and mirrors raw packets to it via a veth pair (mantis-mirror / mantis-peer). Suricata reads an auto-generated YAML config and matches alerts to rules in RULE_PATH/*.rules.

Current config state:

• app-layer.protocols.http.enabled: yes with detection-ports: dp: any
• app-layer.protocols.tls.enabled: yes
• EVE JSON output via unix stream socket

The HTTP app-layer dissector is already active. What is missing is a ruleset that covers web attack signatures.

---

3. Suricata + ET Open Rules (Immediate Win)

3.1 Why ET Open, not OWASP CRS

OWASP Core Rule Set (CRS) uses ModSecurity SecRule format, which is incompatible with Suricata. Suricata uses Snort-style rules.

Emerging Threats Open (ET Open) provides a free, Suricata-compatible ruleset that includes dedicated web attack categories:

Category	File	Covers
Web server attacks	emerging-web_server.rules	SQLi, XSS, LFI, RFI, PHP injection, shell upload
Web client exploits	emerging-web_client.rules	Drive-by downloads, malicious JS
Exploit frameworks	emerging-exploit.rules	Metasploit, Shodan scan payloads
SQL injection	emerging-sql.rules	DB-specific injection patterns
Scanner signatures	emerging-scan.rules	Nikto, Nmap HTTP probes

3.2 Integration

ET Open rules are downloaded from: https://rules.emergingthreats.net/open/suricata-{version}/emerging.rules.tar.gz

The tarball contains individual .rules files that can be dropped directly into RULE_PATH/. Suricata's existing config (rule-files: ["*.rules"]) picks them up automatically — no code change required.

Required steps:

1. Download and extract ET Open rules into RULE_PATH/
2. Test that Suricata loads without errors (check /tmp/suricata.log)
3. Tune noisy SIDs via the suppress config (same mechanism as current setup)

Limitations:

• ET Open covers HTTP (plaintext). HTTPS payloads are encrypted; Suricata cannot inspect them without a TLS MitM proxy (out of scope for an inline NIDS).
• Rules match known attack patterns; zero-day or obfuscated payloads evade signatures.

---

4. HTTP Metadata Collection Pipeline

4.1 Motivation

To train a web-attack-aware ML model and to provide richer forensic data, we need per-request HTTP metadata. This is fundamentally different from the existing flow CSV:

Dimension	Flow CSV (current)	HTTP request CSV (proposed)
Granularity	1 row per flow	1 row per HTTP request
Content	Statistical features	Parsed application-layer fields
HTTPS	Yes (opaque bytes)	No (plaintext only)
Use	Anomaly detection	Web attack classification

4.2 Data Source

eBPF uprobe on OpenSSL is not viable in this deployment. Mantis operates as an inline NIDS between two NICs — it sees raw network packets, not the TLS library calls of other processes. Uprobes only work on TLS libraries loaded on the same host.

Plaintext HTTP traffic is captured by eBPF at the packet level and already delivered to the ML engine via the process_packet path. The payload bytes are available in the payload parameter of flow_tracker.process_packet().

4.3 Proposed CSV Schema

#	Field	Type	Description
1	src_ip	string	Client IP address
2	dst_ip	string	Server IP address
3	src_port	u16	Client port (connection identifier)
4	dst_port	u16	Server port (80, 8080, etc.)
5	timestamp	datetime	Request timestamp
6	method	string	HTTP method — TRACE/OPTIONS indicate scanning
7	uri	string	Full URI including query string; primary attack signal (SQLi, LFI, XSS)
8	http_version	string	HTTP/1.0 vs 1.1 — older version common in scanner tools
9	host	string	Host header — virtual-host confusion attacks
10	user_agent	string	Tool signatures: sqlmap, nikto, curl, python-requests
11	referer	string	Referer header — CSRF source tracing (empty string if absent)
12	content_type	string	Request Content-Type — multipart/form-data indicates upload attempts
13	content_length	u64	Request body size — abnormally large POST suggests payload injection
14	status_code	u16	Response status — 400/403/500 bursts indicate scanning
15	response_content_type	string	Response Content-Type
16	response_content_length	u64	Response body size

Excluded fields:

• cookie content: contains session tokens; privacy risk. Record presence/length only if needed.
• x-forwarded-for: trivially forged by attackers; high noise, low signal.
• Request/response body: too large to store; content_length is sufficient for classification.

4.4 Implementation Architecture

eBPF packet
    │
    ▼
xsk_manager.rs  (existing packet pump)
    │
    ├──► ML engine (flow stats, unchanged)
    │
    └──► WebEngine (new, plaintext HTTP only)
              │
              ▼
         TCP Reassembler (per connection buffer)
              │
              ▼
         HTTP Parser (request + response pairing)
              │
              ▼
         RequestLogger → per-request CSV
              │
              ▼
         WebClassifier (future ML model)

4.5 Component Design

TCP Reassembler

Maintains a HashMap<FlowKey, TcpStream> of per-connection byte buffers. Handles out-of-order segments (sequence number tracking), connection teardown, and memory limits (cap buffer at 64 KB per connection).

Does NOT need to handle HTTPS — detect TLS by checking if first bytes are 0x16 0x03 (TLS record type + version); skip those connections.

HTTP Parser

Minimal parser for the data we need. Full HTTP/1.x is sufficient; HTTP/2 over plaintext (h2c) is rare in real networks and can be skipped initially.

Request:  METHOD SP URI SP HTTP/version CRLF headers CRLF [body]
Response: HTTP/version SP status_code SP reason CRLF headers CRLF [body]

Pair requests to responses by connection + request order (HTTP/1.1 pipelining is sequential on a single TCP connection).

RequestLogger

Analogous to the existing TrafficLogger but writes one row per request. Rolling daily CSV files, same pattern as the flow CSV.

New env var needed: HTTP_LOG_PATH (parallel to CSV_RECORD_PATH).

New config flag: http_logging_mode: bool in config.toml.

4.6 Proposed New Files

mantis/src/detection/web/
├── mod.rs
├── engine.rs          -- WebEngine (tokio task, receives packets from xsk_manager)
├── tcp_reassembler.rs -- per-connection byte stream reassembly
├── http_parser.rs     -- HTTP/1.x request+response parser
├── request_features.rs-- RequestFeatures struct + to_csv_record()
└── request_logger.rs  -- rolling CSV writer (reuse TrafficLogger pattern)

New model/log and model/error files:

mantis/src/model/log/web.rs
mantis/src/model/error/web.rs

---

5. Web Attack ML Model (Future)

Once HTTP request CSV data is collected from production traffic, a supervised classifier can be trained on labelled samples (normal vs. attack).

Suggested approach:

1. Feature engineering: URI path depth, query parameter count, special char density in URI, user-agent entropy, method frequency per source IP
2. Model type: Gradient boosted trees (XGBoost/LightGBM) for interpretability; alternatively a small feedforward net via ONNX
3. Training data: Collect 1-2 weeks of normal HTTP traffic from the target network. Attack samples: OWASP WebGoat / DVWA logs, public datasets (CSIC 2010 HTTP, CAIDA)
4. Serving: Same ort-tract pattern as the existing autoencoder

This is out of scope until the collection pipeline exists and data is available.

---

6. Architecture Decisions

No WAF Integration

A WAF (e.g., ModSecurity, Coraza) operates inline and can inspect/block HTTPS after TLS termination at a reverse proxy. This requires the WAF to sit in front of the protected services (separate deployment concern). Mantis is a passive NIDS and is not the right place for WAF functionality.

Recommendation: deploy a WAF at the application tier independently; Mantis covers the network tier.

No TLS Decryption / MitM

Decrypting HTTPS requires either a TLS MitM proxy (breaks end-to-end encryption, requires CA cert installation on clients) or access to server private keys. Both are out of scope for a passive NIDS.

JA3 fingerprinting via Suricata TLS events (see TODO) provides the best non-decryption signal for HTTPS.

No eBPF Uprobe for TLS

eBPF uprobes on OpenSSL/BoringSSL could capture plaintext at the TLS library boundary — but only for TLS connections originating or terminating on the same host. Since Mantis is deployed as an inline NIDS bridging two NICs, it does not run the TLS libraries of the services it monitors.

---

7. Implementation Phases

Phase 1 — ET Open Rules (0 code changes, immediate)

1. Download ET Open ruleset for installed Suricata version
2. Extract .rules files to RULE_PATH/
3. Test Suricata startup; suppress noisy SIDs via suppress config
4. Verify alerts appear in EVE socket (test with curl against a test URI containing ../../etc/passwd)

Deliverable: Suricata detects common web attack patterns over plaintext HTTP.

Phase 2 — HTTP Metadata Collection

1. Add WebEngine struct and tokio task
2. Implement TcpReassembler with sequence tracking and TLS skip
3. Implement HttpParser for request/response pairs
4. Implement RequestLogger (rolling CSV)
5. Wire into xsk_manager.rs packet pump
6. Add http_logging_mode config flag
7. Validate CSV output against known traffic

Deliverable: Per-request CSV with method, URI, user-agent, status code fields.

Phase 3 — Web Attack ML Model

1. Collect 2+ weeks of labelled HTTP traffic
2. Feature engineering on collected CSV
3. Train and evaluate classifier (target: F1 > 0.90 on hold-out)
4. Export to ONNX; load via ort-tract
5. Add WebClassifier component; integrate with FusionEngine

Deliverable: ML-based web attack alerts in the unified alert stream.

---

8. Open Questions

1. HTTP/2 h2c prevalence: How much plaintext HTTP/2 is in the monitored network? If significant, the HTTP parser must handle HPACK header compression (much more complex). Defer until data shows it is needed.

2. Request-response pairing under pipelining: HTTP/1.1 allows pipelining (multiple requests before response). The parser must queue requests and pair them in order. Simplification: treat connections as strictly serial for v1.

3. Memory cap for TCP reassembly: 64 KB per connection × max_concurrent_connections is the upper bound. Need to measure connection count in production to size correctly.

4. Minimum flow size for useful data: Very short HTTP exchanges (< 3 requests) may not yield enough data for reliable classification. Consider a minimum-request threshold before emitting a row to the classifier.

5. Suricata HTTP alert redundancy: Once ET Open rules are active, some web attacks will be caught by Suricata signatures AND the future ML model. The FusionEngine should handle this naturally (both sources fire → higher confidence fusion alert), but tune deduplication window accordingly.