mirror of
https://github.com/ParrotXray/Mantis.git
synced 2026-08-24 18:50:28 +09:00
Add research findings and update TODO for 2026-05-19 session
Research findings (fe-001-c1.md): - UnifiedAlert API payload analysis and serde snake_case behavior - Runtime crash root cause (PascalCase vs lowercase config keys) - GeoIP thread explosion diagnosis and semaphore fix rationale - maxminddb DEBUG log flood: debug build level + crate filter fix - Config URL mismatches (6 endpoints corrected) - isPausedRef pattern explanation - FusionEngine OR/AND mode behavior analysis state.toml: initial epic and theme registry TODO: - Mark item 6 (frontend detection page) as completed - Add infrastructure fix records (GeoIP semaphore, logging filter) - Add frontend alignment records (detection, config, provider, dashboard) - Add open questions (OR mode double-alert, dashboard chart reset) https://claude.ai/code/session_01Wen51749mAnwBEfh7XmvUw
This commit is contained in:
Claude
parent
e0eb92c42c
commit
e3bb5f251e
216
.research/findings/tasks/fe-001-c1.md
Normal file
216
.research/findings/tasks/fe-001-c1.md
Normal file
@ -0,0 +1,216 @@
|
||||
# fe-001: Frontend Alignment with UnifiedAlert API and Infrastructure Fixes
|
||||
**Cycle**: 1 | **Theme**: frontend + backend-infra | **Kind**: investigation + design | **Status**: done
|
||||
**Date**: 2026-05-19
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Refactored three frontend files (detection page, WebSocket provider, config) to match
|
||||
the current backend `UnifiedAlert` API. Concurrently fixed two backend runtime problems:
|
||||
unbounded GeoIP blocking threads and excessive debug log output from maxminddb.
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
### Q: What does the backend actually send to /detection/websocket/alert?
|
||||
|
||||
A: `UnifiedAlert` struct, serialized with `#[serde(rename_all = "snake_case")]`.
|
||||
|
||||
```rust
|
||||
pub struct UnifiedAlert {
|
||||
pub timestamp: u64,
|
||||
pub flow_key: String,
|
||||
pub src_ip: String, pub dst_ip: String,
|
||||
pub src_port: u16, pub dst_port: u16,
|
||||
pub protocol: u8,
|
||||
pub source: AlertSource, // "ml" | "rule" | "fusion"
|
||||
pub severity: AlertSeverity, // "high" | "critical"
|
||||
pub is_attack: bool,
|
||||
pub attack_type: Option<String>,
|
||||
pub confidence: f32,
|
||||
pub ae_score: f32,
|
||||
pub rule_sid: Option<u32>,
|
||||
pub rule_msg: Option<String>,
|
||||
}
|
||||
```
|
||||
|
||||
Key difference from old `AlertLog`:
|
||||
- `source` / `severity` are enum variants; serde serializes them as lowercase strings
|
||||
- `rf_score` and `ensemble_score` no longer exist
|
||||
- `attack_type` is `Option<String>`, can be null
|
||||
- `rule_sid` and `rule_msg` are new (populated only for rule/fusion alerts)
|
||||
|
||||
**Confidence**: high — read from `model/ml_detection.rs` and `detection/fusion.rs` directly.
|
||||
|
||||
---
|
||||
|
||||
### Q: Why did the old detection.tsx crash at runtime?
|
||||
|
||||
A: `SOURCE_CONFIG` and `SEVERITY_CONFIG` used PascalCase keys (`Ml`, `Rule`, `High`, `Critical`)
|
||||
but the backend sends lowercase (`"ml"`, `"rule"`, `"high"`, `"critical"`) due to
|
||||
`#[serde(rename_all = "snake_case")]`. Accessing `SOURCE_CONFIG[data.source]` returned
|
||||
`undefined`, causing the `bgColor` property access to throw.
|
||||
|
||||
Fix: changed all config keys and TypeScript type aliases to lowercase.
|
||||
|
||||
**Confidence**: high — confirmed by reading Rust source and browser error message.
|
||||
|
||||
---
|
||||
|
||||
### Q: Why were there 60+ OS threads spawned by GeoIP lookups?
|
||||
|
||||
A: `statistics.rs` calls `join_all()` across all flows in each WebSocket push.
|
||||
Each flow's GeoIP lookup calls `spawn_blocking` on a cache miss. With 24 concurrent
|
||||
WebSocket connections × N flows × cache-cold start, this produced an unbounded number
|
||||
of blocking OS threads (observed ThreadId 83–148 simultaneously).
|
||||
|
||||
Fix in `geoip.rs`: added `Arc<Semaphore>` with `MAX_CONCURRENT_DB_LOOKUPS = 8`.
|
||||
`acquire_owned()` is called before `spawn_blocking`; the `OwnedSemaphorePermit` is
|
||||
moved into the closure and held for the full duration of the blocking call.
|
||||
After fix: observed max ThreadId range 63–71 (8 concurrent threads).
|
||||
|
||||
Cache hits and private IP short-circuit paths bypass the semaphore entirely,
|
||||
so warm-cache operation is unaffected.
|
||||
|
||||
**Confidence**: high — measured thread count before and after; mechanism is clear.
|
||||
|
||||
---
|
||||
|
||||
### Q: Why does maxminddb flood the log with DEBUG lines?
|
||||
|
||||
A: The build config in `logging.rs` sets the global tracing level to `Level::DEBUG`
|
||||
when `cfg!(debug_assertions)` is true (i.e., any non-release build). maxminddb's
|
||||
deserializer calls `tracing::debug!()` for every field decoded (~40-60 lines per IP).
|
||||
With continuous GeoIP lookups this produces thousands of log lines per second.
|
||||
|
||||
Root cause: debug build + no crate-specific filter.
|
||||
|
||||
Fix: added `.add_directive("maxminddb=warn".parse().expect("valid directive"))` to the
|
||||
`EnvFilter` chain. Target-specific directives take precedence over the global level,
|
||||
so maxminddb is silenced without affecting the rest of the application.
|
||||
|
||||
For release builds (`--release`), `cfg!(debug_assertions)` is false and the level is
|
||||
already `INFO`, so the directive is redundant but harmless.
|
||||
|
||||
**Confidence**: high — mechanism from tracing-subscriber docs; confirmed by log output change.
|
||||
|
||||
---
|
||||
|
||||
### Q: What config URL mismatches existed between frontend and backend?
|
||||
|
||||
A: Six mismatches found:
|
||||
|
||||
| Frontend (old) | Backend (actual) |
|
||||
|---|---|
|
||||
| `/control/access_control/ipv4/...` | `/ebpf/access_control/ipv4/...` |
|
||||
| `/control/access_control/ipv6/...` | `/ebpf/access_control/ipv6/...` |
|
||||
| `/statistics/websocket/ipv4/...` | `/ebpf/statistics/websocket/ipv4/...` |
|
||||
| `/statistics/websocket/ipv6/...` | `/ebpf/statistics/websocket/ipv6/...` |
|
||||
| `/health/websocket/system_health` | `/health/websocket/metrics` |
|
||||
| `/ai/websocket/alert` (key: `aiAlert`) | `/detection/websocket/alert` (key: `detectionAlert`) |
|
||||
|
||||
Also: port changed from 59182 to 2048 in the updated config.
|
||||
|
||||
**Confidence**: high — compared against `web/api/mod.rs` route registrations.
|
||||
|
||||
---
|
||||
|
||||
### Q: What is the isPausedRef pattern and why is it needed?
|
||||
|
||||
A: When `isPaused` state is captured in a `useCallback` dependency array, any
|
||||
toggle of `isPaused` causes the callback to be recreated. If that callback is
|
||||
itself in the dependency array of a `useEffect` that sets up a WebSocket subscription,
|
||||
every pause/resume toggle tears down and re-establishes all subscriptions.
|
||||
|
||||
Fix: sync `isPaused` to a `useRef` on every change; read the ref inside the callback
|
||||
instead of the state value. The callback then has no deps on `isPaused`, the
|
||||
subscription useEffect does not re-run, and WebSocket connections are stable.
|
||||
|
||||
```typescript
|
||||
const isPausedRef = useRef(isPaused)
|
||||
useEffect(() => { isPausedRef.current = isPaused }, [isPaused])
|
||||
|
||||
// processNewLogData has empty deps — reads isPausedRef.current instead
|
||||
const processNewLogData = useCallback((rawData: any) => {
|
||||
if (isPausedRef.current) return
|
||||
...
|
||||
}, [])
|
||||
```
|
||||
|
||||
**Confidence**: high — standard React pattern for stale closure avoidance.
|
||||
|
||||
---
|
||||
|
||||
### Q: What does the FusionEngine actually do?
|
||||
|
||||
A: Maintains a per-flow `HashMap<String, FusionState>` keyed by `src_ip:src_port-dst_ip:dst_port`.
|
||||
|
||||
Two modes:
|
||||
- **OR mode**: broadcasts immediately when either ML or Rule fires. If the other side
|
||||
already has a pending entry, produces a `from_fusion` alert (both sources). Otherwise
|
||||
produces a single-source alert and keeps the entry for potential future corroboration.
|
||||
- **AND mode**: only broadcasts when both ML and Rule have fired for the same flow key
|
||||
within `window_secs`. Produces a `from_fusion` alert only.
|
||||
|
||||
Entries are evicted after `window_secs` on the next `record_ml` / `record_rule` call.
|
||||
|
||||
**Confidence**: high — read directly from `detection/fusion.rs`.
|
||||
|
||||
---
|
||||
|
||||
## Unexpected Discoveries
|
||||
|
||||
1. **WebSocketProvider now uses BehaviorSubject** — the user's updated provider switched
|
||||
from `Subject` to `BehaviorSubject`. Late subscribers (e.g., a page navigated to after
|
||||
the first data packet) now immediately receive the last cached value. This is a behavioral
|
||||
change: the first emission after subscribe can be `null` (the initial BehaviorSubject value)
|
||||
and consumers must guard against it.
|
||||
|
||||
2. **`getLatestFlowData` / `getLatestSystemHealth`** — new synchronous cache accessors on
|
||||
the WebSocket provider. These allow components to read the latest flow data without subscribing
|
||||
to an Observable. Currently unused by the refactored pages but available for future use.
|
||||
|
||||
3. **GeoIP lookup warm-up window** — even with the semaphore fix, initial page load with a
|
||||
cold cache will briefly queue up to `N_connections × N_flows` lookups; only 8 will run
|
||||
concurrently. Visible as a brief elevated thread count after first connection. This is
|
||||
expected and resolves as cache fills.
|
||||
|
||||
4. **ML inference warm-up window** — `infer_single` returns `None` until the per-src_ip buffer
|
||||
has accumulated `window_size` cycles. During this window, no ML alerts are produced even if
|
||||
the traffic is anomalous. This is inherent to the LSTM sliding window design and not a bug.
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **BehaviorSubject null guard** — should `parseAlertData` / dashboard alert handler
|
||||
explicitly skip `null` data, or should the BehaviorSubject be initialized with a sentinel
|
||||
that consumers can ignore? Currently `parseAlertData` returns `null` for null input (safe),
|
||||
but the dashboard's raw handler does `if (!data || !data.is_attack) return` which is also safe.
|
||||
Consider documenting this contract.
|
||||
|
||||
2. **GeoIP cache size** — currently hardcoded to 10,000 entries. Under continuous high-diversity
|
||||
traffic (many unique IPs), the cache evicts aggressively and `spawn_blocking` pressure returns.
|
||||
May need tuning based on observed IP diversity in deployment.
|
||||
|
||||
3. **FusionEngine OR mode correctness** — in OR mode, a pure ML alert is broadcast immediately
|
||||
AND a `FusionState` entry is created. If the rule engine later matches the same flow, a
|
||||
second `from_fusion` alert is broadcast. This means a single attack event can produce two
|
||||
alerts in OR mode. Whether this is intentional needs clarification.
|
||||
|
||||
4. **Dashboard donut charts reset** — `attackTypeCounts` and `protocolCounts` accumulate in
|
||||
memory and are never reset. After a long session, the counts no longer reflect current
|
||||
activity. A sliding window or TTL-based reset should be considered.
|
||||
|
||||
---
|
||||
|
||||
## Impact on Downstream Tasks
|
||||
|
||||
- **TODO item 6** (完成前端 detection 頁面) — completed. Covers detection.tsx,
|
||||
WebSocketProvider.tsx, config.ts, and dashboard.tsx.
|
||||
- **Account system** (TODO item 7) — unblocked. Frontend plumbing is now correct;
|
||||
auth routes can be added to config.ts and the provider without WebSocket changes.
|
||||
- **Hot-reload API** — when implemented, frontend only needs a new button calling
|
||||
`POST /api/rules/reload`; WebSocket infrastructure is already in place.
|
||||
36
.research/state.toml
Normal file
36
.research/state.toml
Normal file
@ -0,0 +1,36 @@
|
||||
# NetGuardia Research State
|
||||
# Updated: 2026-05-19
|
||||
|
||||
[[epics]]
|
||||
id = "nids-v1"
|
||||
title = "NetGuardia NIDS v1 — Research Prototype"
|
||||
status = "active"
|
||||
description = """
|
||||
End-to-end NIDS combining eBPF packet capture, LSTM autoencoder ML inference,
|
||||
Suricata-compatible rule engine, and a Next.js monitoring frontend.
|
||||
Target: demonstrate joint ML+Rule detection superiority over single-method baselines.
|
||||
"""
|
||||
|
||||
[[themes]]
|
||||
id = "backend-detection"
|
||||
epic = "nids-v1"
|
||||
status = "active"
|
||||
description = "ML pipeline, rule engine, fusion layer, and alert broadcast."
|
||||
|
||||
[[themes]]
|
||||
id = "backend-infra"
|
||||
epic = "nids-v1"
|
||||
status = "active"
|
||||
description = "GeoIP, logging, eBPF statistics, HTTP API, WebSocket delivery."
|
||||
|
||||
[[themes]]
|
||||
id = "frontend"
|
||||
epic = "nids-v1"
|
||||
status = "active"
|
||||
description = "Next.js dashboard, detection page, WebSocket provider, config alignment."
|
||||
|
||||
[[themes]]
|
||||
id = "account-system"
|
||||
epic = "nids-v1"
|
||||
status = "parked"
|
||||
description = "SQLite-backed accounts, sessions, persistent whitelist/blacklist."
|
||||
83
TODO
83
TODO
@ -3,7 +3,7 @@
|
||||
3. 改善推論效能及速度，在大量資料時 [x]
|
||||
4. 實現 ML + RULE 的共用 HashMap 實現共同決策結果 [x]
|
||||
5. 代碼最佳化，檢查是否除了 build.rs 以外有無 .unwarp() and eprintln() [x]
|
||||
6. 完成前端 detection 頁面
|
||||
6. 完成前端 detection 頁面 [x]
|
||||
7. 使用 sqllite 實現帳號系統、白黑名單永久記錄
|
||||
|
||||
[x]
|
||||
@ -18,7 +18,7 @@ $HOME_NET / $EXTERNAL_NET 區分內外網，很多規則依賴這個
|
||||
flow:established 順便過濾掉不完整連線
|
||||
|
||||
============================================================
|
||||
PRIORITY BACKLOG (updated 2026-05-18)
|
||||
PRIORITY BACKLOG (updated 2026-05-19)
|
||||
============================================================
|
||||
|
||||
-- MUST DO (system is not usable without these) -----------
|
||||
@ -166,4 +166,81 @@ PRIORITY BACKLOG (updated 2026-05-18)
|
||||
[x] isdataat
|
||||
Completed 2026-05-15 (included with byte_test/jump/extract).
|
||||
ByteOp kind=3; supports negated form (!isdataat) and relative flag.
|
||||
build.rs: parse_isdataat(); rule_engine.rs: eval_byte_ops() case 3.
|
||||
build.rs: parse_isdataat(); rule_engine.rs: eval_byte_ops() case 3.
|
||||
|
||||
-- INFRASTRUCTURE FIXES (2026-05-19) ----------------------
|
||||
|
||||
[x] GeoIP unbounded thread explosion
|
||||
Completed 2026-05-19.
|
||||
Root cause: statistics.rs calls join_all() across all flows per WebSocket push;
|
||||
each cache-miss triggers spawn_blocking with no concurrency cap. Observed
|
||||
60+ simultaneous OS threads (ThreadId 83-148) under normal traffic.
|
||||
Fix: added Arc<Semaphore>(MAX_CONCURRENT_DB_LOOKUPS=8) to GeoIpService.
|
||||
acquire_owned() before spawn_blocking; OwnedSemaphorePermit moved into
|
||||
closure and held until blocking call returns. Cache hits and private IP
|
||||
paths bypass semaphore. After fix: max 8 concurrent threads at any time.
|
||||
File: core/infrastructure/geoip.rs
|
||||
|
||||
[x] maxminddb DEBUG log flood in development builds
|
||||
Completed 2026-05-19.
|
||||
Root cause: logging.rs sets global tracing level to Level::DEBUG when
|
||||
cfg!(debug_assertions) (non-release builds). maxminddb emits ~50 debug
|
||||
lines per IP lookup via tracing::debug!() in its deserializer.
|
||||
Fix: added EnvFilter directive "maxminddb=warn" which takes precedence
|
||||
over the global level. Release builds are unaffected (already INFO).
|
||||
File: utils/logging.rs
|
||||
|
||||
-- FRONTEND ALIGNMENT (2026-05-19) ------------------------
|
||||
|
||||
[x] Frontend detection page — UnifiedAlert API alignment
|
||||
Completed 2026-05-19.
|
||||
Old detection.tsx used AlertLog type with rf_score/ensemble_score and
|
||||
confidence-based severity (4 levels). Backend sends UnifiedAlert with
|
||||
source (ml|rule|fusion), severity (high|critical), rule_sid, rule_msg,
|
||||
and nullable attack_type. All serde enum variants are snake_case lowercase.
|
||||
Files updated: .research/frontend-refactor/detection.tsx
|
||||
Changes: new UnifiedAlert type inline, SOURCE_CONFIG/SEVERITY_CONFIG with
|
||||
lowercase keys, dual filter (severity x source), 7 stats cards,
|
||||
ML scores panel only for ml/fusion, rule details panel only for rule/fusion,
|
||||
isPausedRef pattern to prevent WS re-subscription on pause toggle.
|
||||
|
||||
[x] Frontend config URL mismatches
|
||||
Completed 2026-05-19.
|
||||
Six URL mismatches corrected; port updated to 2048:
|
||||
/control/access_control/... -> /ebpf/access_control/... (x4)
|
||||
/statistics/websocket/... -> /ebpf/statistics/websocket/... (x2)
|
||||
/health/websocket/system_health -> /health/websocket/metrics
|
||||
websocketUrl.mlAlert (/ml/websocket/alert) -> detectionAlert (/detection/websocket/alert)
|
||||
File: .research/frontend-refactor/config.ts
|
||||
|
||||
[x] WebSocketProvider — naming and Subject upgrade
|
||||
Completed 2026-05-19.
|
||||
getAiAlertStream -> getDetectionAlertStream throughout interface, context
|
||||
default, useCallback, and initializeWebSockets. Subject -> BehaviorSubject
|
||||
so late subscribers receive the last cached value immediately. Added
|
||||
getLatestFlowData() and getLatestSystemHealth() sync cache accessors.
|
||||
File: .research/frontend-refactor/WebSocketProvider.tsx
|
||||
|
||||
[x] Dashboard page — detection stream alignment
|
||||
Completed 2026-05-19.
|
||||
Removed unused AlertLog import. getAiAlertStream -> getDetectionAlertStream.
|
||||
attack_type null guard added before updating attackTypeCounts.
|
||||
Locale zh-TW -> en-GB for chart axis time labels.
|
||||
File: .research/frontend-refactor/dashboard.tsx
|
||||
|
||||
-- OPEN QUESTIONS (not blocking, worth revisiting) --------
|
||||
|
||||
[ ] FusionEngine OR mode produces two alerts per attack event
|
||||
In OR mode, a pure ML alert is broadcast immediately, AND a FusionState
|
||||
entry is kept. When rule engine later matches the same flow, a second
|
||||
from_fusion alert is emitted. Single attack = two frontend notifications.
|
||||
Decide: is this intentional (more verbose), or should OR mode suppress the
|
||||
initial single-source alert and wait window_secs for possible corroboration?
|
||||
File: detection/fusion.rs record_ml() / record_rule()
|
||||
|
||||
[ ] attackTypeCounts / protocolCounts in dashboard never reset
|
||||
Accumulate for the entire session lifetime. After hours of runtime, the
|
||||
donut charts show historical totals rather than recent activity.
|
||||
Consider: sliding window reset (e.g., clear counts every 1 hour) or
|
||||
limit to last N alerts.
|
||||
File: .research/frontend-refactor/dashboard.tsx
|
||||
Loading…
x
Reference in New Issue
Block a user