fix(stress): Keep the SSD and USB jobs alive for the whole soak

qfx5252-stress-usb was started with --runtime 60. It finished a minute
into an overnight run and the USB path sat idle for the rest of the
night - nothing failed, nothing said so, and the log looked complete.
SSD had the same shape at --runtime 3600. Both are now 86400, and the
BMC DDR memtester drops its loop count so it runs until stopped.

86400 is 24h, not forever. Past that SSD and USB die quietly and the
only clue is the shorter bgctl list Script B already prints each round.
Said so in the version history, the release notes and ARCHITECTURE.

Adds blanton_multiphase_margin.sh: the multiphase controllers on both
switch boards over the native i2c buses (SWB0 on i2c-13/15, SWB1 on
-14/-16), 19 rails each. high3/low3/comboA/comboB set voltages and print
one line per rail; save commits them and prints Multiphase Save All,
because that list has no per-rail mapping to report against.

Two hazards it carries: save is STORE_USER_ALL and permanent, and these
are NOT the LTC2980s margin.sh drives - the two tools reach the same
rails from opposite sides with no interlock between them.

The emitted commands were diffed against tools/Multiphase_*.txt: argv
identical for all five. The rail-to-netname mapping was cross-checked
against the bus numbers rather than trusted by line position - blocks
1..19 are all bus 13/15 and 20..38 all 14/16, matching the SWB0/SWB1
split of Multiphase_Netname.

blanton_traffic_linespeed v0.7.0 appends PatternRandom=yes to every tx
including ce0, and the defaults become tx 200 length=324. A fixed
pattern can sit on a benign bit sequence for a whole run; a changing one
is what exposes a weak lane. -r no forces fixed, -r "" omits the
argument for an SDK that will not take it. Bytes per burst went 51200 ->
64800, so absolute counter totals do not carry across this build - the
pair cross-check that decides PASS/FAIL does.

Script A -> V1.1.5.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
This commit is contained in:
2026-09-01 11:04:45 +08:00
co-authored by Claude Opus 5
parent db44f72352
commit 215fed6e95
13 changed files with 2061 additions and 20 deletions
+13 -3
View File
@@ -135,6 +135,8 @@ Blanton_TTL_Script/
│ ├── 100G_PRBS.txt # 100G PRBS 原始指令(port_prbs_monitor.sh 的來源)
│ ├── TR518.txt # 內建封包測試 tr 518 → blanton_tr518.sh
│ ├── BER_Test.txt # 全機 PRBS BER 指令 → blanton_ber.sh
│ ├── Multiphase_{high3,low3,comboA,comboB,Save}.txt # → blanton_multiphase_margin.sh
│ ├── Multiphase_Netname # 38 條軌的 net 名稱(區塊順序對應)
│ ├── 100G_Traffic.txt # ce0 VLAN 138 打流 → blanton_traffic_linespeed.sh
│ ├── fan_ctrl.txt # MAX31790 unbind/bind + fan-speed-control.sh 30
│ └── gen_channel_map.sh # settings/*.conf → docs/LTC2980_channel_map.csv(需 gawk
@@ -190,6 +192,7 @@ Blanton_TTL_Script/
│ # eth0→TARGET_10G、eth1→TARGET_1G,兩張卡同時持續 ping
├── port_prbs_monitor.sh # 100G uplink PRBS 測試(bcmcmd phy diag
├── blanton_ber.sh # 全機 SerDes BER 掃描(phydiag prbs / prbsstat
├── blanton_multiphase_margin.sh # 多相 VRM margin(原生 i2c38 條軌)
├── nfc_polling.sh # NFC tag 背景輪詢 + PASS/FAIL 統計
├── blantons_nfc_validate.py # NFC/NCI 驗證工具(廠商提供,未修改)
├── blanton_tr518.sh # TR518 封包測試,量功耗前的負載來源
@@ -357,17 +360,18 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc
| 模組 | 進入點 | 功能 |
|------|--------|------|
| **Script APre-test** | `1_Blanton_Script_A.ttl` V1.1.4 | root 登入 → `date -s` 對時 → 清 job log → **收掉前一輪殘留**`hw-test-session finish``bgctl reset --yes``bgctl stop --all``bgctl list` 確認清空)→ `hw-test-session start`**設定風扇轉速****等資料面就緒**`wait_init.ttl`)→ DUT 清單(boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe)→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary``lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping****NFC 基線**start → 30s → stop → report)→ 100G uplink 設定與狀態 → **一輪流量基線**`show uptime` |
| **Script BStress** | `2_Blanton_Script_B.ttl` | `mlucas-avx2 -cpu 0:15` + `bgctl run``memtester` / SSD / USB 壓力 + BMC DDR`bmc-manager run memtester`)→ BMC USB net test(探測 cdc_ncm 介面,`systemd-run` 掛 4 小時 ping)→ 啟動 mgmt ping 監控 → **啟動 NFC 輪詢**`jobs` / `bgctl list` → traffic(依 `SWB_UNIT`)→ `while 1` 每輪 PMON + `bgctl list`/`jobs` + `pause 60`**V1.1.4 起 margin 不在迴圈內**|
| **Script APre-test** | `1_Blanton_Script_A.ttl` V1.1.5 | root 登入 → `date -s` 對時 → 清 job log → **收掉前一輪殘留**`hw-test-session finish``bgctl reset --yes``bgctl stop --all``bgctl list` 確認清空)→ `hw-test-session start`**設定風扇轉速****等資料面就緒**`wait_init.ttl`)→ DUT 清單(boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe)→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary``lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping****NFC 基線**start → 30s → stop → report)→ 100G uplink 設定與狀態 → **一輪流量基線**`show uptime` |
| **Script BStress** | `2_Blanton_Script_B.ttl` | `mlucas-avx2 -cpu 0:15` + `bgctl run``memtester` / SSD / USB 壓力v1.1.5 起 SSD/USB `--runtime 86400`、BMC memtester 不帶次數 = 跑到被停) + BMC DDR`bmc-manager run memtester`)→ BMC USB net test(探測 cdc_ncm 介面,`systemd-run` 掛 4 小時 ping)→ 啟動 mgmt ping 監控 → **啟動 NFC 輪詢**`jobs` / `bgctl list` → traffic(依 `SWB_UNIT`)→ `while 1` 每輪 PMON + `bgctl list`/`jobs` + `pause 60`**V1.1.4 起 margin 不在迴圈內**|
| **Script CPost-test** | `3_Blanton_Script_C.ttl` | `kill $(jobs -p)` + `bgctl stop --all` / `reset` + 停 BMC USB unit、mgmt ping 與 **NFC 輪詢** → PMON + dmesg → 7 組 AER + `ras-mc-ctl` → BMC USB `journalctl``ip -s link show eth0/eth1``cat` mlucas log → **NFC `report`** → traffic `stop` + `report`**BER 掃描**(依 `SWB_UNIT`)→ NVMe 健康 → `show reboot-cause`**margin 掃描**`EN_Margin`V1.1.4 由 Script B 移來)→ `uptime`**job log 複製到 USB(帶時戳)**`exit` + wait『Script done』→ `hw-test-session finish` |
| **Script 4Thermal / Safety** | `4_Blanton_Script_thermal_safety.ttl` | **獨立腳本,不接 A→B→C。** Script A 式的環境建立(登入 / session / 風扇 / `wait_init` / DUT 清單 / BMC / PCIe 清除 / `pcie_bus` / PMON / dmesg)→ **滿載**`mlucas-avx2 -cpu 0:15` 背景 + `blanton_tr518.sh start`)→ `while 1`PMON + `TH6_SWB0/1_power_readback.sh` + `bgctl list` / `jobs``pause 60`。**沒有收尾段** —— 中斷後 DUT 仍在 `port cd lb=mac` / `l2 learn off` / `test mode nr=yes`,要手動 `blanton_tr518.sh stop` |
| **SWB loopback 流量** | `blanton_traffic_linespeed <sub> [-u 0\|1\|all]` | `init`VLAN 30..137,成對 `cdN`/`cdN+32`)→ `clear``start``tx 100 length=512`)→ `stop``report`per-pair TX/RX 交叉比對 PASS/FAIL);另有 `ps`link/speed)與 `run`(一條龍)。**V0.6.0 起含 100G uplink `ce0`**VLAN 138):它沒有 loopback 對象,靠 **ingress mirror 打回自己**維持循環,`report` 最後一列就是它,底下再接一份跨 unit summary |
| **SWB loopback 流量** | `blanton_traffic_linespeed <sub> [-u 0\|1\|all]` | `init`VLAN 30..137,成對 `cdN`/`cdN+32`)→ `clear``start``tx 200 length=324 PatternRandom=yes`v0.7.0 起)→ `stop``report`per-pair TX/RX 交叉比對 PASS/FAIL);另有 `ps`link/speed)與 `run`(一條龍)。**V0.6.0 起含 100G uplink `ce0`**VLAN 138):它沒有 loopback 對象,靠 **ingress mirror 打回自己**維持循環,`report` 最後一列就是它,底下再接一份跨 unit summary |
| **FPGA 暫存器存取** | `cb_fpga <fn> <addr> [data]``fpga_rescan` | pcimem 打 `resource0` + BAR 相對 offset32-bit wordBDF 於 source 時自動偵測;`pmc/icb/swb0/swb1_fpga` 走 VSPI 窗口 |
| **CB I2C / PMBus** | `cb_i2c_init/scan/read/write``cb_pmbus_read/write/linear16` | FPGA F3 OpenCores master**讀用 Repeated START**SMBus/PMBus 裝置必需) |
| **ICB / PDB 存取** | `vi2c_init/scan/read/write``vi2c_scan_all` | CB F3 VI2C proxy~3.3 ms/reg,取代 VSPI 的 ~18 ms/reg(後者會踩 SMBus 2535 ms timeout |
| **溫度** | `temp_cb` / `temp_icb` / `temp_all` | TMP7512-bit 左靠齊 ×0.0625)、TMP432(含 RANGE bit 的 64 偏移、remote1/2 |
| **電源資料** | `pwr_data` / `pwr_data_pdb` / `pwr_pdb_id` | SWB VRM 全軌 Vin/Vout/Iout/Temp 對齊表;MP29816 解析度**執行期從 `MFR_VOUT_SCALE_LOOP` 讀**,不寫死 |
| **電壓 Margin** | `margin_init/status/set/apply_profile/save` + `*_all.sh` | 9 顆 LTC2980CB CONN13、SWB0/1 CONN13~16)批次 high/low margin 與狀態掃描。`VNOM` 為 Blanton 實測中位數(2026-08-30);SWB 的 profile 是 per-CONN,套錯 CONN 會 margin 到錯的軌且不報錯。v2.6.0 起 SWB 路徑真正接上共用 cb_i2c 後端,並有 `margin_swb_info/probe/scan/reset/sem` 除錯包裝 |
| **Multiphase VRM margin** | `blanton_multiphase_margin.sh {high3\|low3\|comboA\|comboB\|save} [-u 0\|1\|all]` | 兩片 SWB 的多相控制器(**不是** `margin.sh` 管的 LTC2980),走原生 i2c busSWB0 = `/dev/i2c-13`+`-15`、SWB1 = `-14`+`-16`。每片 19 條軌、共 38 條,逐條印 `SWB0_0V9_ANLG0 - 0.930V Configure Done``save` 沒有 per-rail 對應,只印 `Multiphase Save All`。⚠️ `save``STORE_USER_ALL`0x15TH6 core 用 0x17**永久寫 NVM**;⚠️ 與 `margin.sh` 從兩側摸同一批軌,**不可同時用** |
| **Margin 寫 NVM** | `margin_save` / `margin_save all` / `margin_save blanton` | ⚠️ 永久寫入。`0x15` 送 PMBus **send-byte**(不帶 data byte)。`all` = 依 `settings/` 走 9 顆、跑完還原原本載入的板子;`blanton` = 本平台捷徑,18 顆全走原生 i2c bus(`MARGIN_BLANTON_STORE` 表),不需 `margin_init`。每筆之間 `MARGIN_STORE_SETTLE`0.5s),工具**不 poll busy bit** |
| **管理網路 ping** | `mgmt_ping_monitor.sh {start\|stop\|status\|fg\|tail\|clear\|summary}` | 一輪 = 10G leg + 1G leg,各自 `ip address/route replace``ping -I <if>` 打自己的對端;兩張卡都保持 up。每 leg 記 `ip -s link show` 與該卡的 TX/RX packet delta,另出一行可 grep 的 `RESULT` |
| **BMC 監控** | `bmc_monitor.sh {start\|stop\|status\|fg\|tail\|clear}` | 由 host 端 `bmc-manager run` 週期取樣 `free -m`,另加一組 **I2C 寫入/讀回圖樣測試**bus 1、addr 0x41、offset 0x00C0,寫 `55AA55AA` 讀回、再寫 `AA55AA55` 讀回)。B 啟動、C 停止並 `cat log/bmc_poll.log` |
@@ -804,6 +808,12 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值,
**`sleep 120` 的方向還沒驗證** —— 若 `tr 518` 的流量是有時限的(參數含 `runtime=1`
`CheckDataTime=200`),等 120 秒可能量到已經降下來的值,要上機比對 0s / 30s / 120s 三次;
TTL 尚未呼叫它。
- **`--runtime 86400` 是 24 小時的上限,不是「永遠」**:超過一天的 soak,SSD 與 USB 壓力會在
第 24 小時安靜結束,剩下的時間那兩項是閒置的。Script B 每輪都印 `bgctl list`,數量變少會看得到,
但**沒有任何東西會主動報警**。真的要跑更久,得把數字改大或加一段重啟 job 的邏輯。
- **`blanton_multiphase_margin.sh``margin.sh` 沒有互斥機制**:兩支從不同側(原生 i2c vs
LTC2980 PMBus)摸同一批軌,各自都假設自己是唯一的寫入者。目前只靠文件與 help 提醒,
沒有任何執行期檢查。
- **Script 4 沒有收尾段**`while 1` 之後就是檔案結尾,中斷時 DUT 停在 `port cd lb=mac` /
`l2 learn off` / `test mode nr=yes`,而且 `mlucas-avx2` 還在背景跑。要手動
`blanton_tr518.sh stop` + `kill`,否則下一輪任何測試都在被污染的狀態上開始