feat: require full window fill before ONNX inference

.research/findings/tasks/ml-001-c1.md

@@ -0,0 +1,190 @@
+# ml-001: ML False Positive Suppression and Flow Tracker Cleanup
+**Cycle**: 1 | **Theme**: backend-detection | **Kind**: investigation + fix | **Status**: done
+**Date**: 2026-05-20
+
+---
+
+## Summary
+
+Diagnosed and fixed a systematic false positive pattern in the ML inference pipeline where
+newly-observed source IPs were consistently flagged as ANOMALY despite carrying benign traffic.
+Root cause: LSTM autoencoder receives out-of-distribution (OOD) input when the per-src_ip
+sliding window is mostly zero-padded. Fix: gate the `is_attack` flag on a minimum window fill
+threshold. Concurrently removed dead code from `FlowTracker` and restored a missing
+`cleanup_old_flows` call that had been dropped during a prior refactor.
+
+---
+
+## Findings
+
+### Q: Why were Cloudflare WARP (198.41.x.x) and Apple CDN (17.x.x.x) IPs always flagged as ANOMALY?
+
+A: The LSTM autoencoder uses a per-src_ip sliding window of `window_size=10` feature vectors.
+A new src_ip starts with an empty buffer; the Rust runtime zero-pads the left side before
+feeding the sequence to the model (matching Python training code `_make_per_flow_sequences`).
+
+External IPs that appear only once per 5-second drain cycle produce sequences with `pad=9/10`:
+nine rows of all-zeros followed by one real feature vector. The model was trained on sequences
+where most or all slots contain real flow data. A `pad=9/10` sequence is out-of-distribution
+(OOD) — the autoencoder cannot reconstruct it accurately, so MSE is high regardless of whether
+the traffic is benign or malicious.
+
+Observed values from `RUST_LOG=debug` WindowDebug output:
+- Cloudflare WARP 198.41.200.x: ae_score ≈ 0.101–0.105 (threshold 0.0266 → flagged)
+- Apple CDN 17.253.117.21: ae_score ≈ 0.144 (flagged)
+- Local machine 140.130.34.68: ae_score ≈ 0.001 (pad=0/10, full window → correctly benign)
+
+The high ae_score is an artifact of OOD padding, not of anomalous traffic behavior.
+
+**Confidence**: high — confirmed by WindowDebug log showing pad values and feature vectors;
+mechanism matches autoencoder reconstruction error theory.
+
+---
+
+### Q: What is the correct fix for OOD false positives from under-filled windows?
+
+A: Skip ONNX inference entirely (early `return None`) when the buffer is not fully filled.
+`min_window_fill` must equal `window_size` — not `window_size / 2`.
+
+```rust
+// inference.rs — Inference::new()
+let min_window_fill = config.window_size;
+
+// inference.rs — infer_single(), before tensor construction
+if buf_len_snapshot < self.min_window_fill {
+    return None;
+}
+```
+
+With `window_size=10`, `min_window_fill=10`. ONNX is never called on a zero-padded input.
+The warm-up period is 50 seconds (10 drain cycles × 5 s). During warm-up, `infer_single`
+returns `None`, so no `DetectionResult` is produced and no alert is possible.
+
+**Why `window_size/2` is insufficient**: empirical log at startup shows that sequences with
+pad=5 (ae≈0.059), pad=4 (ae≈0.047), and pad=3 (ae≈0.036) all exceed threshold 0.0266,
+producing 3 false anomaly detections in the first 30 seconds for a single benign src_ip.
+Only pad≤2 drops below the threshold for this IP. `window_size/2` does not eliminate OOD.
+
+**Why the training assumption matters**: the CSV training data is complete — every row
+represents a real 5-second drain snapshot. The Python `_make_per_flow_sequences` function
+initialises sequences to zeros then fills from the right, but this only applies to the very
+first appearance of each src_ip in the training set. In practice, training sequences with
+significant zero-padding are extremely rare; the model was not trained to reconstruct them
+accurately, so any pad > 0 produces elevated MSE.
+
+A persistent attacker still accumulates 10 cycles of context and is detected thereafter.
+
+**Confidence**: high — fix verified by re-running system; "0 anomaly" at steady state.
+Startup false positives (3 anomaly) observed with `window_size/2` eliminated with `window_size`.
+
+---
+
+### Q: Should drain_flows be replaced with get_flows_for_inference or drain_completed_flows?
+
+A: No. Training CSV rows were generated by `drain_flows` semantics: independent 5-second
+snapshots of each flow, removed from the tracker after each drain cycle. The model learned
+to detect anomalies in these fixed-window snapshots.
+
+- `get_flows_for_inference` reads without removing; the same flow would be re-read on each
+  cycle with accumulating packet counts, producing feature vectors with growing totals that
+  the model was never trained on. This causes distribution shift and more false positives, not fewer.
+
+- `drain_completed_flows` waits for TCP FIN/RST or idle timeout before extracting a flow.
+  A 30-second TCP connection yields a single feature vector representing the full duration,
+  unlike the training data which captures it as ~6 successive 5-second snapshots. Different
+  distribution; requires retraining to use correctly.
+
+`drain_flows` is the correct choice and matches training semantics exactly.
+
+**Confidence**: high — examined training pipeline `_make_per_flow_sequences` and confirmed
+row-per-drain-cycle structure in the CSV format.
+
+---
+
+### Q: Was cleanup_old_flows being called correctly?
+
+A: No. `flow_timeout_us` was stored in the `Engine` struct and accepted as a constructor
+parameter, but was never used inside `run_inference_loop`. The only protection against
+unbounded tracker growth was the `max_flows` hard cap in `process_packet`, which evicts
+one arbitrary flow when the limit is exceeded.
+
+Consequence: flows with packet counts below `min_packets` that never grow (e.g., single-packet
+probes, rejected connections) accumulate in the tracker indefinitely until randomly evicted.
+
+Fix: call `t.cleanup_old_flows(self.flow_timeout_us)` inside the tracker lock block,
+after `drain_flows` and before the `active_ips` snapshot. This ordering ensures that
+stale flows are excluded from `active_ips`, so their inference buffers are also cleaned
+up by `cleanup_buffers(&active_ips)`. The fix covers both the inference path and the
+CSV logging path since both share the same lock block.
+
+**Confidence**: high — grep confirmed zero callers of `cleanup_old_flows`; the field
+`flow_timeout_us` appeared in struct/new/assign but never in the inference loop.
+
+---
+
+### Q: Which FlowTracker methods are now dead code?
+
+A: Two methods with no callers were removed:
+
+- `drain_completed_flows(timeout_us, min_packets)`: implemented but never wired up.
+  Drain-on-completion semantics require a retrained model to be useful; leaving it
+  in place was misleading.
+
+- `get_flows_for_inference(min_packets)`: read-without-drain semantics confirmed
+  misaligned with training; no callers existed after the drain_flows refactor.
+
+`cleanup_old_flows` and `get_flows_snapshot` remain; both have active callers.
+
+**Confidence**: high — confirmed by grep across full src/ tree before deletion.
+
+---
+
+## Unexpected Discoveries
+
+1. **ae_score is still computed for under-filled windows** — the ONNX session runs and
+   produces a score even when `buf_len_snapshot < min_window_fill`. The score is visible
+   in the `WindowDebug` log but does not trigger an alert. This is useful for monitoring
+   the model's sensitivity during the warm-up period without affecting detection.
+
+2. **active_ips correctness improved by cleanup ordering** — placing `cleanup_old_flows`
+   before the `get_flows_snapshot` call that builds `active_ips` means stale flows are
+   excluded from the set. Previously, a flow that timed out would remain in `active_ips`
+   for one extra cycle, keeping its inference buffer alive one cycle longer than necessary.
+
+3. **max_flows random eviction remains intentional** — the per-packet hard cap
+   (`if self.flows.len() > self.max_flows { evict arbitrary flow }`) is not replaced by
+   `cleanup_old_flows`. Cleanup runs every 5 seconds; packet insertion runs continuously.
+   The random eviction remains as an OOM safety valve for traffic bursts.
+
+---
+
+## Open Questions
+
+1. **Warm-up period evasion** — an attacker who knows `min_window_fill = window_size = 10`
+   could send benign traffic for 50 seconds to build up window context, then switch to an
+   attack pattern. The LSTM should detect the behavioral shift within one or two cycles after
+   the switch, but this has not been empirically verified.
+
+2. **WARP / Apple CDN ae_score after warm-up** — once these IPs accumulate 10 slots of real
+   flow data, their ae_score at steady state is unknown. If their traffic pattern is genuinely
+   anomalous in the training distribution (port 7844, infrequent short flows), they may still
+   be flagged after warm-up. Monitoring ae_score trends for these IPs at steady state is needed
+   to determine if threshold adjustment or a CIDR allowlist is required.
+
+3. **50-second blind spot for new src_ips** — any IP that appears fewer than 10 times within
+   the observation window produces no ML alert, even if its first 10 flows are anomalous.
+   This is inherent to the sliding-window design. Rule-based detection (Suricata) covers this
+   gap for known attack signatures during the warm-up period.
+
+---
+
+## Impact on Downstream Tasks
+
+- **TODO item 3** (改善推論效能及準確率) — partially addressed. False positive rate for
+  external CDN/VPN IPs is eliminated during the window warm-up period.
+- **Account system whitelist** (TODO item 7) — still relevant as a long-term solution for
+  known-good IP ranges (e.g., Cloudflare WARP 198.41.192.0/22, 198.41.200.0/22) that should
+  bypass ML evaluation entirely, independent of window fill state.
+- **flow_timeout_us parameter** — now actually used. Downstream callers of `Engine::new`
+  should verify the configured value matches the expected idle timeout for the deployment
+  environment (current default in config.toml should be checked).