Mantis/TODO

1. .csv 一開始文件格式為 traffic-yyyy-oo-zz.csv, 換天後直接變日期創新檔，例如今天啟動今天為 traffic-2026-05-11.csv 隔天變成 traffic-2026-05-12.csv [x]
2. will change tract-onnx engine to ort-tract engine [x]
3. 改善推論效能及速度，在大量資料時 [x]
4. 實現 ML + RULE 的共用 HashMap 實現共同決策結果 [x]
5. 代碼最佳化，檢查是否除了 build.rs 以外有無 .unwarp() and eprintln() [x]
6. 完成前端 detection 頁面 [x]
7. 使用 sqllite 實現帳號系統、白黑名單永久記錄

[x]
Suricata
提升偵測準確率：
pcre 很多規則依賴正則，沒有會漏掉很多威脅
flow:established 減少掃握手封包的無意義處理
Flowbits 複雜攻擊鏈的偵測（例如先偵測掃描再偵測滲透）

減少假陽性：
$HOME_NET / $EXTERNAL_NET 區分內外網，很多規則依賴這個
flow:established 順便過濾掉不完整連線

============================================================
PRIORITY BACKLOG  (updated 2026-05-19)
============================================================

-- MUST DO (system is not usable without these) -----------

[x] byte_test / byte_jump / byte_extract
    Completed 2026-05-15.
    build.rs: parse_byte_test/jump/extract -> SQLite byte_ops table.
    rule_engine.rs: eval_byte_ops() -> eval_byte_test/jump, read_bytes().
    Supports: relative, endian, string mode (dec/hex/oct), bitmask, vars.

[x] app-layer-protocol: keyword
    Completed 2026-05-15.
    build.rs: extract_alproto() parses keyword -> (proto_id, negated).
    rule_engine.rs: pkt.proto as u8 == sig.alproto filter before chain.
    Protocols mapped: http(1) http2(2) tls(3) dns(4) ssh(5) smtp(6)
                      ftp(7) mqtt(8) quic(9).

[x] ML + Rule fusion decision layer
    Core differentiator of the project. Currently ML and rule
    engine produce independent alerts with no cross-awareness.
    Design: shared per-flow HashMap; two fusion modes:
      AND      - alert only when both agree
      OR       - alert when either fires (with source tag)
    Files: detection/ml/engine.rs, detection/rule/rule_engine.rs,
           new detection/fusion.rs, model/ml_detection.rs

[x] threshold: complete three modes
    Completed 2026-05-17.
    build.rs: extract_threshold() parses type/track/count/seconds -> threshold_entries table.
    rule_engine.rs: ThresholdKey (BySrc/ByDst/ByBoth/ByRule); ThresholdState;
    parking_lot::Mutex<HashMap<(sig_idx, key), state>>; should_alert() mirrors
    Suricata ThresholdCheckUpdate() exactly:
      threshold (3) - fire on every Nth hit; reset after fire; no alert on window expiry
      limit (1)     - fire on first N hits per window; alert after window reset
      both (2)      - fire exactly on Nth hit (==, not >=); silence for N+1...; fire on expiry if count==1
    Track: TRACK_DST=1 TRACK_SRC=2 TRACK_RULE=3 TRACK_BOTH=5 TRACK_FLOW=6 (matches Suricata header)

-- SHOULD DO (runtime management) ------------------------

[ ] Account system + persistent list DB  (updated 2026-05-18)
    Two SQLite files, separated by concern:

    static/db/rules.db  (rule-related, already exists)
      suppress   (id, sid INTEGER, track INTEGER, ip_net TEXT, comment TEXT, created_at)
        - track: 1=by_src  2=by_dst  4=by_either  (Suricata values)
        - already populated by build.rs; API writes go here; RuleEngine reloads on change

    static/db/app.db  (account + network list, new)
      accounts   (id, username, password_hash, role, created_at)
        - role: "admin" | "viewer"
        - password_hash: argon2 or bcrypt
      sessions   (token TEXT PK, account_id, expires_at)
        - token: 32-byte random hex, expires in 24 h
      whitelist  (id, ip_net TEXT, comment TEXT, created_at)
        - packets from whitelisted CIDRs skip rule + ML evaluation entirely
      blacklist  (id, ip_net TEXT, comment TEXT, action TEXT, created_at)
        - action: "alert" | "drop"
        - packets from blacklisted CIDRs auto-alert without rule evaluation
    Migration: CREATE TABLE IF NOT EXISTS at startup in AppServices::init().
    File: core/infrastructure/app_db.rs (new)

[ ] HTTP API - accounts / suppress / whitelist / blacklist / rules
    All routes require session token in Authorization: Bearer <token> header
    except POST /api/auth/login.

    Auth:
      POST   /api/auth/login               - { username, password } -> { token, expires_at }
      POST   /api/auth/logout              - invalidate current token
      GET    /api/auth/me                  - current account info

    Accounts (admin only):
      GET    /api/accounts                 - list accounts
      POST   /api/accounts                 - create account { username, password, role }
      PUT    /api/accounts/:id/password    - change password
      DELETE /api/accounts/:id             - delete account

    Suppress:
      GET    /api/suppress                 - list all entries
      POST   /api/suppress                 - add { sid, track, ip_net, comment }
      DELETE /api/suppress/:id             - remove entry
      (runtime effect: RuleEngine reloads suppress table after write)

    Whitelist:
      GET    /api/whitelist                - list entries
      POST   /api/whitelist                - add { ip_net, comment }
      DELETE /api/whitelist/:id            - remove

    Blacklist:
      GET    /api/blacklist                - list entries
      POST   /api/blacklist                - add { ip_net, action, comment }
      DELETE /api/blacklist/:id            - remove

    Rules:
      GET    /api/rules                    - list loaded rules (sid, msg, enabled)
      POST   /api/rules/reload             - trigger hot-reload

    System:
      POST   /api/system/restart           - graceful restart

    Files: web/routes/auth.rs, web/routes/accounts.rs, web/routes/lists.rs,
           web/routes/rules.rs, web/routes/system.rs
    Middleware: web/middleware/auth.rs (token extractor + role check)

[ ] Hot-reload: rules + suppress without process restart
    Trigger: POST /api/rules/reload  OR  SIGHUP signal.
    Steps:
      1. build.rs logic extracted into a runtime fn rebuild_rules_db(rules_dir) -> Result<()>
         that re-parses *.rules files and rewrites rules.db in a temp path.
      2. RuleEngine::reload(new_db_path) acquires a write lock, swaps BlockDatabase +
         sigs vec + SuppressList atomically; old engine dropped after swap.
      3. ML engine is NOT restarted (model/config unchanged).
      4. In-flight threshold states are preserved across reload to avoid counter reset.
    Files: detection/rule/rule_engine.rs (add reload()), build.rs (extract parse logic),
           core/infrastructure/app_services.rs (wire SIGHUP handler via tokio::signal).

[ ] Graceful restart
    Trigger: POST /api/system/restart  OR  SIGTERM + re-exec.
    Steps:
      1. Drain in-flight packets: wait up to 2 s for flow tracker to flush.
      2. Detach eBPF programs (XDP unload) cleanly.
      3. Flush CSV writer and close traffic log.
      4. tokio runtime shutdown, then re-exec self via std::process::Command.
    Why re-exec instead of just reload: eBPF object files must be re-loaded from disk
    when kernel map layouts change; a hot-reload cannot cover this case.
    Files: core/ebpf/ (detach helpers), main.rs (signal handler + re-exec path).

-- SHOULD DO (usable but blind spots remain) --------------

[x] QUIC parser
    Completed 2026-05-16. Full port of Suricata rust/src/quic/parser.rs
    and rust/src/quic/frames.rs.
    RFC QUIC v1/v2: HKDF key derivation + AES-128-GCM Initial decryption
    + TLS ClientHello SNI extraction from CRYPTO frames.
    gQUIC Q043-Q046: plaintext Initial frames; SNI from CHLO tag-value
    structure (StreamTag::Sni 0x534e4900) in STREAM frames (0x80+).
    File: app_layer/quic.rs; integrated in mod.rs + rule_engine.rs.

[x] suppress list
    Completed 2026-05-15.
    Suricata suppress.conf format: gen_id N, sig_id N, track by_src|by_dst, ip addr.
    File: detection/rule/suppress.rs; loaded at startup; filters results
    in rule_engine.rs after matching (results.retain(|m| !suppress.is_suppressed(...))).
    Placeholder: static/db/suppress.conf (empty, add entries as needed).

[x] isdataat
    Completed 2026-05-15 (included with byte_test/jump/extract).
    ByteOp kind=3; supports negated form (!isdataat) and relative flag.
    build.rs: parse_isdataat(); rule_engine.rs: eval_byte_ops() case 3.

-- INFRASTRUCTURE FIXES (2026-05-19) ----------------------

[x] GeoIP unbounded thread explosion
    Completed 2026-05-19.
    Root cause: statistics.rs calls join_all() across all flows per WebSocket push;
    each cache-miss triggers spawn_blocking with no concurrency cap. Observed
    60+ simultaneous OS threads (ThreadId 83-148) under normal traffic.
    Fix: added Arc<Semaphore>(MAX_CONCURRENT_DB_LOOKUPS=8) to GeoIpService.
    acquire_owned() before spawn_blocking; OwnedSemaphorePermit moved into
    closure and held until blocking call returns. Cache hits and private IP
    paths bypass semaphore. After fix: max 8 concurrent threads at any time.
    File: core/infrastructure/geoip.rs

[x] maxminddb DEBUG log flood in development builds
    Completed 2026-05-19.
    Root cause: logging.rs sets global tracing level to Level::DEBUG when
    cfg!(debug_assertions) (non-release builds). maxminddb emits ~50 debug
    lines per IP lookup via tracing::debug!() in its deserializer.
    Fix: added EnvFilter directive "maxminddb=warn" which takes precedence
    over the global level. Release builds are unaffected (already INFO).
    File: utils/logging.rs

-- FRONTEND ALIGNMENT (2026-05-19) ------------------------

[x] Frontend detection page — UnifiedAlert API alignment
    Completed 2026-05-19.
    Old detection.tsx used AlertLog type with rf_score/ensemble_score and
    confidence-based severity (4 levels). Backend sends UnifiedAlert with
    source (ml|rule|fusion), severity (high|critical), rule_sid, rule_msg,
    and nullable attack_type. All serde enum variants are snake_case lowercase.
    Files updated: .research/frontend-refactor/detection.tsx
    Changes: new UnifiedAlert type inline, SOURCE_CONFIG/SEVERITY_CONFIG with
    lowercase keys, dual filter (severity x source), 7 stats cards,
    ML scores panel only for ml/fusion, rule details panel only for rule/fusion,
    isPausedRef pattern to prevent WS re-subscription on pause toggle.

[x] Frontend config URL mismatches
    Completed 2026-05-19.
    Six URL mismatches corrected; port updated to 2048:
      /control/access_control/... -> /ebpf/access_control/...  (x4)
      /statistics/websocket/...   -> /ebpf/statistics/websocket/...  (x2)
      /health/websocket/system_health -> /health/websocket/metrics
      websocketUrl.mlAlert (/ml/websocket/alert) -> detectionAlert (/detection/websocket/alert)
    File: .research/frontend-refactor/config.ts

[x] WebSocketProvider — naming and Subject upgrade
    Completed 2026-05-19.
    getAiAlertStream -> getDetectionAlertStream throughout interface, context
    default, useCallback, and initializeWebSockets. Subject -> BehaviorSubject
    so late subscribers receive the last cached value immediately. Added
    getLatestFlowData() and getLatestSystemHealth() sync cache accessors.
    File: .research/frontend-refactor/WebSocketProvider.tsx

[x] Dashboard page — detection stream alignment
    Completed 2026-05-19.
    Removed unused AlertLog import. getAiAlertStream -> getDetectionAlertStream.
    attack_type null guard added before updating attackTypeCounts.
    Locale zh-TW -> en-GB for chart axis time labels.
    File: .research/frontend-refactor/dashboard.tsx

-- OPEN QUESTIONS (not blocking, worth revisiting) --------
[ ] attackTypeCounts / protocolCounts in dashboard never reset
    Accumulate for the entire session lifetime. After hours of runtime, the
    donut charts show historical totals rather than recent activity.
    Consider: sliding window reset (e.g., clear counts every 1 hour) or
    limit to last N alerts.
    File: .research/frontend-refactor/dashboard.tsx