diff --git a/.research/findings/tasks/fe-001-c1.md b/.research/findings/tasks/fe-001-c1.md new file mode 100644 index 0000000..98af078 --- /dev/null +++ b/.research/findings/tasks/fe-001-c1.md @@ -0,0 +1,216 @@ +# fe-001: Frontend Alignment with UnifiedAlert API and Infrastructure Fixes +**Cycle**: 1 | **Theme**: frontend + backend-infra | **Kind**: investigation + design | **Status**: done +**Date**: 2026-05-19 + +--- + +## Summary + +Refactored three frontend files (detection page, WebSocket provider, config) to match +the current backend `UnifiedAlert` API. Concurrently fixed two backend runtime problems: +unbounded GeoIP blocking threads and excessive debug log output from maxminddb. + +--- + +## Findings + +### Q: What does the backend actually send to /detection/websocket/alert? + +A: `UnifiedAlert` struct, serialized with `#[serde(rename_all = "snake_case")]`. + +```rust +pub struct UnifiedAlert { + pub timestamp: u64, + pub flow_key: String, + pub src_ip: String, pub dst_ip: String, + pub src_port: u16, pub dst_port: u16, + pub protocol: u8, + pub source: AlertSource, // "ml" | "rule" | "fusion" + pub severity: AlertSeverity, // "high" | "critical" + pub is_attack: bool, + pub attack_type: Option, + pub confidence: f32, + pub ae_score: f32, + pub rule_sid: Option, + pub rule_msg: Option, +} +``` + +Key difference from old `AlertLog`: +- `source` / `severity` are enum variants; serde serializes them as lowercase strings +- `rf_score` and `ensemble_score` no longer exist +- `attack_type` is `Option`, can be null +- `rule_sid` and `rule_msg` are new (populated only for rule/fusion alerts) + +**Confidence**: high — read from `model/ml_detection.rs` and `detection/fusion.rs` directly. + +--- + +### Q: Why did the old detection.tsx crash at runtime? + +A: `SOURCE_CONFIG` and `SEVERITY_CONFIG` used PascalCase keys (`Ml`, `Rule`, `High`, `Critical`) +but the backend sends lowercase (`"ml"`, `"rule"`, `"high"`, `"critical"`) due to +`#[serde(rename_all = "snake_case")]`. Accessing `SOURCE_CONFIG[data.source]` returned +`undefined`, causing the `bgColor` property access to throw. + +Fix: changed all config keys and TypeScript type aliases to lowercase. + +**Confidence**: high — confirmed by reading Rust source and browser error message. + +--- + +### Q: Why were there 60+ OS threads spawned by GeoIP lookups? + +A: `statistics.rs` calls `join_all()` across all flows in each WebSocket push. +Each flow's GeoIP lookup calls `spawn_blocking` on a cache miss. With 24 concurrent +WebSocket connections × N flows × cache-cold start, this produced an unbounded number +of blocking OS threads (observed ThreadId 83–148 simultaneously). + +Fix in `geoip.rs`: added `Arc` with `MAX_CONCURRENT_DB_LOOKUPS = 8`. +`acquire_owned()` is called before `spawn_blocking`; the `OwnedSemaphorePermit` is +moved into the closure and held for the full duration of the blocking call. +After fix: observed max ThreadId range 63–71 (8 concurrent threads). + +Cache hits and private IP short-circuit paths bypass the semaphore entirely, +so warm-cache operation is unaffected. + +**Confidence**: high — measured thread count before and after; mechanism is clear. + +--- + +### Q: Why does maxminddb flood the log with DEBUG lines? + +A: The build config in `logging.rs` sets the global tracing level to `Level::DEBUG` +when `cfg!(debug_assertions)` is true (i.e., any non-release build). maxminddb's +deserializer calls `tracing::debug!()` for every field decoded (~40-60 lines per IP). +With continuous GeoIP lookups this produces thousands of log lines per second. + +Root cause: debug build + no crate-specific filter. + +Fix: added `.add_directive("maxminddb=warn".parse().expect("valid directive"))` to the +`EnvFilter` chain. Target-specific directives take precedence over the global level, +so maxminddb is silenced without affecting the rest of the application. + +For release builds (`--release`), `cfg!(debug_assertions)` is false and the level is +already `INFO`, so the directive is redundant but harmless. + +**Confidence**: high — mechanism from tracing-subscriber docs; confirmed by log output change. + +--- + +### Q: What config URL mismatches existed between frontend and backend? + +A: Six mismatches found: + +| Frontend (old) | Backend (actual) | +|---|---| +| `/control/access_control/ipv4/...` | `/ebpf/access_control/ipv4/...` | +| `/control/access_control/ipv6/...` | `/ebpf/access_control/ipv6/...` | +| `/statistics/websocket/ipv4/...` | `/ebpf/statistics/websocket/ipv4/...` | +| `/statistics/websocket/ipv6/...` | `/ebpf/statistics/websocket/ipv6/...` | +| `/health/websocket/system_health` | `/health/websocket/metrics` | +| `/ai/websocket/alert` (key: `aiAlert`) | `/detection/websocket/alert` (key: `detectionAlert`) | + +Also: port changed from 59182 to 2048 in the updated config. + +**Confidence**: high — compared against `web/api/mod.rs` route registrations. + +--- + +### Q: What is the isPausedRef pattern and why is it needed? + +A: When `isPaused` state is captured in a `useCallback` dependency array, any +toggle of `isPaused` causes the callback to be recreated. If that callback is +itself in the dependency array of a `useEffect` that sets up a WebSocket subscription, +every pause/resume toggle tears down and re-establishes all subscriptions. + +Fix: sync `isPaused` to a `useRef` on every change; read the ref inside the callback +instead of the state value. The callback then has no deps on `isPaused`, the +subscription useEffect does not re-run, and WebSocket connections are stable. + +```typescript +const isPausedRef = useRef(isPaused) +useEffect(() => { isPausedRef.current = isPaused }, [isPaused]) + +// processNewLogData has empty deps — reads isPausedRef.current instead +const processNewLogData = useCallback((rawData: any) => { + if (isPausedRef.current) return + ... +}, []) +``` + +**Confidence**: high — standard React pattern for stale closure avoidance. + +--- + +### Q: What does the FusionEngine actually do? + +A: Maintains a per-flow `HashMap` keyed by `src_ip:src_port-dst_ip:dst_port`. + +Two modes: +- **OR mode**: broadcasts immediately when either ML or Rule fires. If the other side + already has a pending entry, produces a `from_fusion` alert (both sources). Otherwise + produces a single-source alert and keeps the entry for potential future corroboration. +- **AND mode**: only broadcasts when both ML and Rule have fired for the same flow key + within `window_secs`. Produces a `from_fusion` alert only. + +Entries are evicted after `window_secs` on the next `record_ml` / `record_rule` call. + +**Confidence**: high — read directly from `detection/fusion.rs`. + +--- + +## Unexpected Discoveries + +1. **WebSocketProvider now uses BehaviorSubject** — the user's updated provider switched + from `Subject` to `BehaviorSubject`. Late subscribers (e.g., a page navigated to after + the first data packet) now immediately receive the last cached value. This is a behavioral + change: the first emission after subscribe can be `null` (the initial BehaviorSubject value) + and consumers must guard against it. + +2. **`getLatestFlowData` / `getLatestSystemHealth`** — new synchronous cache accessors on + the WebSocket provider. These allow components to read the latest flow data without subscribing + to an Observable. Currently unused by the refactored pages but available for future use. + +3. **GeoIP lookup warm-up window** — even with the semaphore fix, initial page load with a + cold cache will briefly queue up to `N_connections × N_flows` lookups; only 8 will run + concurrently. Visible as a brief elevated thread count after first connection. This is + expected and resolves as cache fills. + +4. **ML inference warm-up window** — `infer_single` returns `None` until the per-src_ip buffer + has accumulated `window_size` cycles. During this window, no ML alerts are produced even if + the traffic is anomalous. This is inherent to the LSTM sliding window design and not a bug. + +--- + +## Open Questions + +1. **BehaviorSubject null guard** — should `parseAlertData` / dashboard alert handler + explicitly skip `null` data, or should the BehaviorSubject be initialized with a sentinel + that consumers can ignore? Currently `parseAlertData` returns `null` for null input (safe), + but the dashboard's raw handler does `if (!data || !data.is_attack) return` which is also safe. + Consider documenting this contract. + +2. **GeoIP cache size** — currently hardcoded to 10,000 entries. Under continuous high-diversity + traffic (many unique IPs), the cache evicts aggressively and `spawn_blocking` pressure returns. + May need tuning based on observed IP diversity in deployment. + +3. **FusionEngine OR mode correctness** — in OR mode, a pure ML alert is broadcast immediately + AND a `FusionState` entry is created. If the rule engine later matches the same flow, a + second `from_fusion` alert is broadcast. This means a single attack event can produce two + alerts in OR mode. Whether this is intentional needs clarification. + +4. **Dashboard donut charts reset** — `attackTypeCounts` and `protocolCounts` accumulate in + memory and are never reset. After a long session, the counts no longer reflect current + activity. A sliding window or TTL-based reset should be considered. + +--- + +## Impact on Downstream Tasks + +- **TODO item 6** (完成前端 detection 頁面) — completed. Covers detection.tsx, + WebSocketProvider.tsx, config.ts, and dashboard.tsx. +- **Account system** (TODO item 7) — unblocked. Frontend plumbing is now correct; + auth routes can be added to config.ts and the provider without WebSocket changes. +- **Hot-reload API** — when implemented, frontend only needs a new button calling + `POST /api/rules/reload`; WebSocket infrastructure is already in place. \ No newline at end of file diff --git a/.research/state.toml b/.research/state.toml new file mode 100644 index 0000000..40f0e86 --- /dev/null +++ b/.research/state.toml @@ -0,0 +1,36 @@ +# NetGuardia Research State +# Updated: 2026-05-19 + +[[epics]] +id = "nids-v1" +title = "NetGuardia NIDS v1 — Research Prototype" +status = "active" +description = """ +End-to-end NIDS combining eBPF packet capture, LSTM autoencoder ML inference, +Suricata-compatible rule engine, and a Next.js monitoring frontend. +Target: demonstrate joint ML+Rule detection superiority over single-method baselines. +""" + +[[themes]] +id = "backend-detection" +epic = "nids-v1" +status = "active" +description = "ML pipeline, rule engine, fusion layer, and alert broadcast." + +[[themes]] +id = "backend-infra" +epic = "nids-v1" +status = "active" +description = "GeoIP, logging, eBPF statistics, HTTP API, WebSocket delivery." + +[[themes]] +id = "frontend" +epic = "nids-v1" +status = "active" +description = "Next.js dashboard, detection page, WebSocket provider, config alignment." + +[[themes]] +id = "account-system" +epic = "nids-v1" +status = "parked" +description = "SQLite-backed accounts, sessions, persistent whitelist/blacklist." \ No newline at end of file diff --git a/TODO b/TODO index b52546a..b43f77b 100644 --- a/TODO +++ b/TODO @@ -3,7 +3,7 @@ 3. 改善推論效能及速度，在大量資料時 [x] 4. 實現 ML + RULE 的共用 HashMap 實現共同決策結果 [x] 5. 代碼最佳化，檢查是否除了 build.rs 以外有無 .unwarp() and eprintln() [x] -6. 完成前端 detection 頁面 +6. 完成前端 detection 頁面 [x] 7. 使用 sqllite 實現帳號系統、白黑名單永久記錄 [x] @@ -18,7 +18,7 @@ $HOME_NET / $EXTERNAL_NET 區分內外網，很多規則依賴這個 flow:established 順便過濾掉不完整連線 ============================================================ -PRIORITY BACKLOG (updated 2026-05-18) +PRIORITY BACKLOG (updated 2026-05-19) ============================================================ -- MUST DO (system is not usable without these) ----------- @@ -166,4 +166,73 @@ PRIORITY BACKLOG (updated 2026-05-18) [x] isdataat Completed 2026-05-15 (included with byte_test/jump/extract). ByteOp kind=3; supports negated form (!isdataat) and relative flag. - build.rs: parse_isdataat(); rule_engine.rs: eval_byte_ops() case 3. \ No newline at end of file + build.rs: parse_isdataat(); rule_engine.rs: eval_byte_ops() case 3. + +-- INFRASTRUCTURE FIXES (2026-05-19) ---------------------- + +[x] GeoIP unbounded thread explosion + Completed 2026-05-19. + Root cause: statistics.rs calls join_all() across all flows per WebSocket push; + each cache-miss triggers spawn_blocking with no concurrency cap. Observed + 60+ simultaneous OS threads (ThreadId 83-148) under normal traffic. + Fix: added Arc(MAX_CONCURRENT_DB_LOOKUPS=8) to GeoIpService. + acquire_owned() before spawn_blocking; OwnedSemaphorePermit moved into + closure and held until blocking call returns. Cache hits and private IP + paths bypass semaphore. After fix: max 8 concurrent threads at any time. + File: core/infrastructure/geoip.rs + +[x] maxminddb DEBUG log flood in development builds + Completed 2026-05-19. + Root cause: logging.rs sets global tracing level to Level::DEBUG when + cfg!(debug_assertions) (non-release builds). maxminddb emits ~50 debug + lines per IP lookup via tracing::debug!() in its deserializer. + Fix: added EnvFilter directive "maxminddb=warn" which takes precedence + over the global level. Release builds are unaffected (already INFO). + File: utils/logging.rs + +-- FRONTEND ALIGNMENT (2026-05-19) ------------------------ + +[x] Frontend detection page — UnifiedAlert API alignment + Completed 2026-05-19. + Old detection.tsx used AlertLog type with rf_score/ensemble_score and + confidence-based severity (4 levels). Backend sends UnifiedAlert with + source (ml|rule|fusion), severity (high|critical), rule_sid, rule_msg, + and nullable attack_type. All serde enum variants are snake_case lowercase. + Files updated: .research/frontend-refactor/detection.tsx + Changes: new UnifiedAlert type inline, SOURCE_CONFIG/SEVERITY_CONFIG with + lowercase keys, dual filter (severity x source), 7 stats cards, + ML scores panel only for ml/fusion, rule details panel only for rule/fusion, + isPausedRef pattern to prevent WS re-subscription on pause toggle. + +[x] Frontend config URL mismatches + Completed 2026-05-19. + Six URL mismatches corrected; port updated to 2048: + /control/access_control/... -> /ebpf/access_control/... (x4) + /statistics/websocket/... -> /ebpf/statistics/websocket/... (x2) + /health/websocket/system_health -> /health/websocket/metrics + websocketUrl.mlAlert (/ml/websocket/alert) -> detectionAlert (/detection/websocket/alert) + File: .research/frontend-refactor/config.ts + +[x] WebSocketProvider — naming and Subject upgrade + Completed 2026-05-19. + getAiAlertStream -> getDetectionAlertStream throughout interface, context + default, useCallback, and initializeWebSockets. Subject -> BehaviorSubject + so late subscribers receive the last cached value immediately. Added + getLatestFlowData() and getLatestSystemHealth() sync cache accessors. + File: .research/frontend-refactor/WebSocketProvider.tsx + +[x] Dashboard page — detection stream alignment + Completed 2026-05-19. + Removed unused AlertLog import. getAiAlertStream -> getDetectionAlertStream. + attack_type null guard added before updating attackTypeCounts. + Locale zh-TW -> en-GB for chart axis time labels. + File: .research/frontend-refactor/dashboard.tsx + +-- OPEN QUESTIONS (not blocking, worth revisiting) -------- + +[ ] attackTypeCounts / protocolCounts in dashboard never reset + Accumulate for the entire session lifetime. After hours of runtime, the + donut charts show historical totals rather than recent activity. + Consider: sliding window reset (e.g., clear counts every 1 hour) or + limit to last N alerts. + File: .research/frontend-refactor/dashboard.tsx \ No newline at end of file diff --git a/config.toml b/config.toml index be43cc0..882413d 100644 --- a/config.toml +++ b/config.toml @@ -37,8 +37,8 @@ home_net = ["140.130.34.0/24"] # CPU affinity (Linux only). Uncomment and tune for your hardware. # Pin XSK packet threads starting from this core (one core per queue pair). # Example: xsk_cpu_base=0 with combined_queue_count=4 uses cores 0-3 for packets. - xsk_cpu_base = 0 +xsk_cpu_base = 0 # # Pin ML inference (ONNX spawn_blocking) to this core. # Example: on an 8-core machine, reserve core 7 for inference. - ml_cpu = 7 \ No newline at end of file +ml_cpu = 7 \ No newline at end of file diff --git a/net-guardia-frontend b/net-guardia-frontend index 671d42e..f1fa091 160000 --- a/net-guardia-frontend +++ b/net-guardia-frontend @@ -1 +1 @@ -Subproject commit 671d42e41aa7309988f0e610df1be88ac96dde9c +Subproject commit f1fa091f4c1abc90049b32c36f2d339400090713 diff --git a/net-guardia/src/core/infrastructure/geoip.rs b/net-guardia/src/core/infrastructure/geoip.rs index a1f39b1..0bca797 100644 --- a/net-guardia/src/core/infrastructure/geoip.rs +++ b/net-guardia/src/core/infrastructure/geoip.rs @@ -3,7 +3,7 @@ use std::path::{Path, PathBuf}; use std::sync::Arc; use maxminddb::{geoip2, MaxMindDbError, Reader}; -use tokio::sync::RwLock; +use tokio::sync::{RwLock, Semaphore}; use lru::LruCache; use std::num::NonZeroUsize; use tokio::task; @@ -11,9 +11,12 @@ use tokio::task; use crate::model::geo_stats::GeoLocation; use crate::utils::ip_address; +const MAX_CONCURRENT_DB_LOOKUPS: usize = 8; + pub struct GeoIpService { reader: Arc>>, cache: Arc>>>, + lookup_sem: Arc, } impl GeoIpService { @@ -33,6 +36,7 @@ impl GeoIpService { Ok(Self { reader: Arc::new(reader), cache: Arc::new(RwLock::new(LruCache::new(cache_capacity))), + lookup_sem: Arc::new(Semaphore::new(MAX_CONCURRENT_DB_LOOKUPS)), }) } @@ -55,8 +59,15 @@ impl GeoIpService { } } + let permit = self.lookup_sem.clone().acquire_owned().await + .map_err(|_| MaxMindDbError::InvalidDatabase { + message: "GeoIP semaphore closed".to_string(), + offset: None, + })?; + let reader = self.reader.clone(); let result = task::spawn_blocking(move || { + let _permit = permit; Self::lookup_from_db_blocking(&reader, ip) }) .await