refactor/frontend-detection (#7)

* wip

* feat: Adapt detection page to UnifiedAlert API

.research/findings/tasks/fe-001-c1.md

@@ -0,0 +1,216 @@
+# fe-001: Frontend Alignment with UnifiedAlert API and Infrastructure Fixes
+**Cycle**: 1 | **Theme**: frontend + backend-infra | **Kind**: investigation + design | **Status**: done
+**Date**: 2026-05-19
+
+---
+
+## Summary
+
+Refactored three frontend files (detection page, WebSocket provider, config) to match
+the current backend `UnifiedAlert` API. Concurrently fixed two backend runtime problems:
+unbounded GeoIP blocking threads and excessive debug log output from maxminddb.
+
+---
+
+## Findings
+
+### Q: What does the backend actually send to /detection/websocket/alert?
+
+A: `UnifiedAlert` struct, serialized with `#[serde(rename_all = "snake_case")]`.
+
+```rust
+pub struct UnifiedAlert {
+    pub timestamp: u64,
+    pub flow_key: String,
+    pub src_ip: String,  pub dst_ip: String,
+    pub src_port: u16,   pub dst_port: u16,
+    pub protocol: u8,
+    pub source: AlertSource,   // "ml" | "rule" | "fusion"
+    pub severity: AlertSeverity, // "high" | "critical"
+    pub is_attack: bool,
+    pub attack_type: Option<String>,
+    pub confidence: f32,
+    pub ae_score: f32,
+    pub rule_sid: Option<u32>,
+    pub rule_msg: Option<String>,
+}
+```
+
+Key difference from old `AlertLog`:
+- `source` / `severity` are enum variants; serde serializes them as lowercase strings
+- `rf_score` and `ensemble_score` no longer exist
+- `attack_type` is `Option<String>`, can be null
+- `rule_sid` and `rule_msg` are new (populated only for rule/fusion alerts)
+
+**Confidence**: high — read from `model/ml_detection.rs` and `detection/fusion.rs` directly.
+
+---
+
+### Q: Why did the old detection.tsx crash at runtime?
+
+A: `SOURCE_CONFIG` and `SEVERITY_CONFIG` used PascalCase keys (`Ml`, `Rule`, `High`, `Critical`)
+but the backend sends lowercase (`"ml"`, `"rule"`, `"high"`, `"critical"`) due to
+`#[serde(rename_all = "snake_case")]`. Accessing `SOURCE_CONFIG[data.source]` returned
+`undefined`, causing the `bgColor` property access to throw.
+
+Fix: changed all config keys and TypeScript type aliases to lowercase.
+
+**Confidence**: high — confirmed by reading Rust source and browser error message.
+
+---
+
+### Q: Why were there 60+ OS threads spawned by GeoIP lookups?
+
+A: `statistics.rs` calls `join_all()` across all flows in each WebSocket push.
+Each flow's GeoIP lookup calls `spawn_blocking` on a cache miss. With 24 concurrent
+WebSocket connections × N flows × cache-cold start, this produced an unbounded number
+of blocking OS threads (observed ThreadId 83–148 simultaneously).
+
+Fix in `geoip.rs`: added `Arc<Semaphore>` with `MAX_CONCURRENT_DB_LOOKUPS = 8`.
+`acquire_owned()` is called before `spawn_blocking`; the `OwnedSemaphorePermit` is
+moved into the closure and held for the full duration of the blocking call.
+After fix: observed max ThreadId range 63–71 (8 concurrent threads).
+
+Cache hits and private IP short-circuit paths bypass the semaphore entirely,
+so warm-cache operation is unaffected.
+
+**Confidence**: high — measured thread count before and after; mechanism is clear.
+
+---
+
+### Q: Why does maxminddb flood the log with DEBUG lines?
+
+A: The build config in `logging.rs` sets the global tracing level to `Level::DEBUG`
+when `cfg!(debug_assertions)` is true (i.e., any non-release build). maxminddb's
+deserializer calls `tracing::debug!()` for every field decoded (~40-60 lines per IP).
+With continuous GeoIP lookups this produces thousands of log lines per second.
+
+Root cause: debug build + no crate-specific filter.
+
+Fix: added `.add_directive("maxminddb=warn".parse().expect("valid directive"))` to the
+`EnvFilter` chain. Target-specific directives take precedence over the global level,
+so maxminddb is silenced without affecting the rest of the application.
+
+For release builds (`--release`), `cfg!(debug_assertions)` is false and the level is
+already `INFO`, so the directive is redundant but harmless.
+
+**Confidence**: high — mechanism from tracing-subscriber docs; confirmed by log output change.
+
+---
+
+### Q: What config URL mismatches existed between frontend and backend?
+
+A: Six mismatches found:
+
+| Frontend (old) | Backend (actual) |
+|---|---|
+| `/control/access_control/ipv4/...` | `/ebpf/access_control/ipv4/...` |
+| `/control/access_control/ipv6/...` | `/ebpf/access_control/ipv6/...` |
+| `/statistics/websocket/ipv4/...` | `/ebpf/statistics/websocket/ipv4/...` |
+| `/statistics/websocket/ipv6/...` | `/ebpf/statistics/websocket/ipv6/...` |
+| `/health/websocket/system_health` | `/health/websocket/metrics` |
+| `/ai/websocket/alert` (key: `aiAlert`) | `/detection/websocket/alert` (key: `detectionAlert`) |
+
+Also: port changed from 59182 to 2048 in the updated config.
+
+**Confidence**: high — compared against `web/api/mod.rs` route registrations.
+
+---
+
+### Q: What is the isPausedRef pattern and why is it needed?
+
+A: When `isPaused` state is captured in a `useCallback` dependency array, any
+toggle of `isPaused` causes the callback to be recreated. If that callback is
+itself in the dependency array of a `useEffect` that sets up a WebSocket subscription,
+every pause/resume toggle tears down and re-establishes all subscriptions.
+
+Fix: sync `isPaused` to a `useRef` on every change; read the ref inside the callback
+instead of the state value. The callback then has no deps on `isPaused`, the
+subscription useEffect does not re-run, and WebSocket connections are stable.
+
+```typescript
+const isPausedRef = useRef(isPaused)
+useEffect(() => { isPausedRef.current = isPaused }, [isPaused])
+
+// processNewLogData has empty deps — reads isPausedRef.current instead
+const processNewLogData = useCallback((rawData: any) => {
+    if (isPausedRef.current) return
+    ...
+}, [])
+```
+
+**Confidence**: high — standard React pattern for stale closure avoidance.
+
+---
+
+### Q: What does the FusionEngine actually do?
+
+A: Maintains a per-flow `HashMap<String, FusionState>` keyed by `src_ip:src_port-dst_ip:dst_port`.
+
+Two modes:
+- **OR mode**: broadcasts immediately when either ML or Rule fires. If the other side
+  already has a pending entry, produces a `from_fusion` alert (both sources). Otherwise
+  produces a single-source alert and keeps the entry for potential future corroboration.
+- **AND mode**: only broadcasts when both ML and Rule have fired for the same flow key
+  within `window_secs`. Produces a `from_fusion` alert only.
+
+Entries are evicted after `window_secs` on the next `record_ml` / `record_rule` call.
+
+**Confidence**: high — read directly from `detection/fusion.rs`.
+
+---
+
+## Unexpected Discoveries
+
+1. **WebSocketProvider now uses BehaviorSubject** — the user's updated provider switched
+   from `Subject` to `BehaviorSubject`. Late subscribers (e.g., a page navigated to after
+   the first data packet) now immediately receive the last cached value. This is a behavioral
+   change: the first emission after subscribe can be `null` (the initial BehaviorSubject value)
+   and consumers must guard against it.
+
+2. **`getLatestFlowData` / `getLatestSystemHealth`** — new synchronous cache accessors on
+   the WebSocket provider. These allow components to read the latest flow data without subscribing
+   to an Observable. Currently unused by the refactored pages but available for future use.
+
+3. **GeoIP lookup warm-up window** — even with the semaphore fix, initial page load with a
+   cold cache will briefly queue up to `N_connections × N_flows` lookups; only 8 will run
+   concurrently. Visible as a brief elevated thread count after first connection. This is
+   expected and resolves as cache fills.
+
+4. **ML inference warm-up window** — `infer_single` returns `None` until the per-src_ip buffer
+   has accumulated `window_size` cycles. During this window, no ML alerts are produced even if
+   the traffic is anomalous. This is inherent to the LSTM sliding window design and not a bug.
+
+---
+
+## Open Questions
+
+1. **BehaviorSubject null guard** — should `parseAlertData` / dashboard alert handler
+   explicitly skip `null` data, or should the BehaviorSubject be initialized with a sentinel
+   that consumers can ignore? Currently `parseAlertData` returns `null` for null input (safe),
+   but the dashboard's raw handler does `if (!data || !data.is_attack) return` which is also safe.
+   Consider documenting this contract.
+
+2. **GeoIP cache size** — currently hardcoded to 10,000 entries. Under continuous high-diversity
+   traffic (many unique IPs), the cache evicts aggressively and `spawn_blocking` pressure returns.
+   May need tuning based on observed IP diversity in deployment.
+
+3. **FusionEngine OR mode correctness** — in OR mode, a pure ML alert is broadcast immediately
+   AND a `FusionState` entry is created. If the rule engine later matches the same flow, a
+   second `from_fusion` alert is broadcast. This means a single attack event can produce two
+   alerts in OR mode. Whether this is intentional needs clarification.
+
+4. **Dashboard donut charts reset** — `attackTypeCounts` and `protocolCounts` accumulate in
+   memory and are never reset. After a long session, the counts no longer reflect current
+   activity. A sliding window or TTL-based reset should be considered.
+
+---
+
+## Impact on Downstream Tasks
+
+- **TODO item 6** (完成前端 detection 頁面) — completed. Covers detection.tsx,
+  WebSocketProvider.tsx, config.ts, and dashboard.tsx.
+- **Account system** (TODO item 7) — unblocked. Frontend plumbing is now correct;
+  auth routes can be added to config.ts and the provider without WebSocket changes.
+- **Hot-reload API** — when implemented, frontend only needs a new button calling
+  `POST /api/rules/reload`; WebSocket infrastructure is already in place.

.research/state.toml

@@ -0,0 +1,36 @@
+# NetGuardia Research State
+# Updated: 2026-05-19
+
+[[epics]]
+id = "nids-v1"
+title = "NetGuardia NIDS v1 — Research Prototype"
+status = "active"
+description = """
+End-to-end NIDS combining eBPF packet capture, LSTM autoencoder ML inference,
+Suricata-compatible rule engine, and a Next.js monitoring frontend.
+Target: demonstrate joint ML+Rule detection superiority over single-method baselines.
+"""
+
+[[themes]]
+id = "backend-detection"
+epic = "nids-v1"
+status = "active"
+description = "ML pipeline, rule engine, fusion layer, and alert broadcast."
+
+[[themes]]
+id = "backend-infra"
+epic = "nids-v1"
+status = "active"
+description = "GeoIP, logging, eBPF statistics, HTTP API, WebSocket delivery."
+
+[[themes]]
+id = "frontend"
+epic = "nids-v1"
+status = "active"
+description = "Next.js dashboard, detection page, WebSocket provider, config alignment."
+
+[[themes]]
+id = "account-system"
+epic = "nids-v1"
+status = "parked"
+description = "SQLite-backed accounts, sessions, persistent whitelist/blacklist."

TODO

@@ -3,7 +3,7 @@
 3. 改善推論效能及速度，在大量資料時 [x]
 4. 實現 ML + RULE 的共用 HashMap 實現共同決策結果 [x]
 5. 代碼最佳化，檢查是否除了 build.rs 以外有無 .unwarp() and eprintln() [x]
-6. 完成前端 detection 頁面
+6. 完成前端 detection 頁面 [x]
 7. 使用 sqllite 實現帳號系統、白黑名單永久記錄
 
 [x]
@@ -18,7 +18,7 @@ $HOME_NET / $EXTERNAL_NET 區分內外網，很多規則依賴這個
 flow:established 順便過濾掉不完整連線
 
 ============================================================
-PRIORITY BACKLOG  (updated 2026-05-18)
+PRIORITY BACKLOG  (updated 2026-05-19)
 ============================================================
 
 -- MUST DO (system is not usable without these) -----------
@@ -166,4 +166,73 @@ PRIORITY BACKLOG  (updated 2026-05-18)
 [x] isdataat
     Completed 2026-05-15 (included with byte_test/jump/extract).
     ByteOp kind=3; supports negated form (!isdataat) and relative flag.
-    build.rs: parse_isdataat(); rule_engine.rs: eval_byte_ops() case 3.
+    build.rs: parse_isdataat(); rule_engine.rs: eval_byte_ops() case 3.
+
+-- INFRASTRUCTURE FIXES (2026-05-19) ----------------------
+
+[x] GeoIP unbounded thread explosion
+    Completed 2026-05-19.
+    Root cause: statistics.rs calls join_all() across all flows per WebSocket push;
+    each cache-miss triggers spawn_blocking with no concurrency cap. Observed
+    60+ simultaneous OS threads (ThreadId 83-148) under normal traffic.
+    Fix: added Arc<Semaphore>(MAX_CONCURRENT_DB_LOOKUPS=8) to GeoIpService.
+    acquire_owned() before spawn_blocking; OwnedSemaphorePermit moved into
+    closure and held until blocking call returns. Cache hits and private IP
+    paths bypass semaphore. After fix: max 8 concurrent threads at any time.
+    File: core/infrastructure/geoip.rs
+
+[x] maxminddb DEBUG log flood in development builds
+    Completed 2026-05-19.
+    Root cause: logging.rs sets global tracing level to Level::DEBUG when
+    cfg!(debug_assertions) (non-release builds). maxminddb emits ~50 debug
+    lines per IP lookup via tracing::debug!() in its deserializer.
+    Fix: added EnvFilter directive "maxminddb=warn" which takes precedence
+    over the global level. Release builds are unaffected (already INFO).
+    File: utils/logging.rs
+
+-- FRONTEND ALIGNMENT (2026-05-19) ------------------------
+
+[x] Frontend detection page — UnifiedAlert API alignment
+    Completed 2026-05-19.
+    Old detection.tsx used AlertLog type with rf_score/ensemble_score and
+    confidence-based severity (4 levels). Backend sends UnifiedAlert with
+    source (ml|rule|fusion), severity (high|critical), rule_sid, rule_msg,
+    and nullable attack_type. All serde enum variants are snake_case lowercase.
+    Files updated: .research/frontend-refactor/detection.tsx
+    Changes: new UnifiedAlert type inline, SOURCE_CONFIG/SEVERITY_CONFIG with
+    lowercase keys, dual filter (severity x source), 7 stats cards,
+    ML scores panel only for ml/fusion, rule details panel only for rule/fusion,
+    isPausedRef pattern to prevent WS re-subscription on pause toggle.
+
+[x] Frontend config URL mismatches
+    Completed 2026-05-19.
+    Six URL mismatches corrected; port updated to 2048:
+      /control/access_control/... -> /ebpf/access_control/...  (x4)
+      /statistics/websocket/...   -> /ebpf/statistics/websocket/...  (x2)
+      /health/websocket/system_health -> /health/websocket/metrics
+      websocketUrl.mlAlert (/ml/websocket/alert) -> detectionAlert (/detection/websocket/alert)
+    File: .research/frontend-refactor/config.ts
+
+[x] WebSocketProvider — naming and Subject upgrade
+    Completed 2026-05-19.
+    getAiAlertStream -> getDetectionAlertStream throughout interface, context
+    default, useCallback, and initializeWebSockets. Subject -> BehaviorSubject
+    so late subscribers receive the last cached value immediately. Added
+    getLatestFlowData() and getLatestSystemHealth() sync cache accessors.
+    File: .research/frontend-refactor/WebSocketProvider.tsx
+
+[x] Dashboard page — detection stream alignment
+    Completed 2026-05-19.
+    Removed unused AlertLog import. getAiAlertStream -> getDetectionAlertStream.
+    attack_type null guard added before updating attackTypeCounts.
+    Locale zh-TW -> en-GB for chart axis time labels.
+    File: .research/frontend-refactor/dashboard.tsx
+
+-- OPEN QUESTIONS (not blocking, worth revisiting) --------
+
+[ ] attackTypeCounts / protocolCounts in dashboard never reset
+    Accumulate for the entire session lifetime. After hours of runtime, the
+    donut charts show historical totals rather than recent activity.
+    Consider: sliding window reset (e.g., clear counts every 1 hour) or
+    limit to last N alerts.
+    File: .research/frontend-refactor/dashboard.tsx

config.toml

@@ -37,8 +37,8 @@ home_net = ["140.130.34.0/24"]
 # CPU affinity (Linux only). Uncomment and tune for your hardware.
 # Pin XSK packet threads starting from this core (one core per queue pair).
 # Example: xsk_cpu_base=0 with combined_queue_count=4 uses cores 0-3 for packets.
- xsk_cpu_base = 0
+xsk_cpu_base = 0
 #
 # Pin ML inference (ONNX spawn_blocking) to this core.
 # Example: on an 8-core machine, reserve core 7 for inference.
- ml_cpu = 7
+ml_cpu = 7

net-guardia-frontend

@@ -1 +1 @@
-Subproject commit 671d42e41aa7309988f0e610df1be88ac96dde9c
+Subproject commit f1fa091f4c1abc90049b32c36f2d339400090713

net-guardia/src/core/infrastructure/geoip.rs

@@ -3,7 +3,7 @@ use std::path::{Path, PathBuf};
 use std::sync::Arc;
 
 use maxminddb::{geoip2, MaxMindDbError, Reader};
-use tokio::sync::RwLock;
+use tokio::sync::{RwLock, Semaphore};
 use lru::LruCache;
 use std::num::NonZeroUsize;
 use tokio::task;
@@ -11,9 +11,12 @@ use tokio::task;
 use crate::model::geo_stats::GeoLocation;
 use crate::utils::ip_address;
 
+const MAX_CONCURRENT_DB_LOOKUPS: usize = 8;
+
 pub struct GeoIpService {
     reader: Arc<Reader<Vec<u8>>>,
     cache: Arc<RwLock<LruCache<IpAddr, Option<GeoLocation>>>>,
+    lookup_sem: Arc<Semaphore>,
 }
 
 impl GeoIpService {
@@ -33,6 +36,7 @@ impl GeoIpService {
         Ok(Self {
             reader: Arc::new(reader),
             cache: Arc::new(RwLock::new(LruCache::new(cache_capacity))),
+            lookup_sem: Arc::new(Semaphore::new(MAX_CONCURRENT_DB_LOOKUPS)),
         })
     }
 
@@ -55,8 +59,15 @@ impl GeoIpService {
             }
         }
 
+        let permit = self.lookup_sem.clone().acquire_owned().await
+            .map_err(|_| MaxMindDbError::InvalidDatabase {
+                message: "GeoIP semaphore closed".to_string(),
+                offset: None,
+            })?;
+
         let reader = self.reader.clone();
         let result = task::spawn_blocking(move || {
+            let _permit = permit;
             Self::lookup_from_db_blocking(&reader, ip)
         })
             .await