Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
  etwenandClaude Opus 5
|
5e1e4a6a8f | ||
|
etwenandClaude Opus 5
|
a31c186142 | ||
|
etwenandClaude Opus 5
|
1b8d5a8e91 | ||
|
etwenandClaude Opus 5
|
cac37583e7 | ||
|
etwenandClaude Opus 5
|
802ee4d467 | ||
|
etwenandClaude Opus 5
|
166b53983f | ||
|
etwenandClaude Opus 5
|
ef7cbdffd4 | ||
|
etwenandClaude Opus 5
|
ae8941a844 | ||
|
etwenandClaude Opus 5
|
a6a12cdf2c | ||
|
etwenandClaude Opus 5
|
e38d24c775 | ||
|
etwenandClaude Opus 5
|
9a11c1d5bb | ||
|
etwenandClaude Opus 5
|
cbef5865ec | ||
|
etwenandClaude Opus 5
|
8714bbb773 | ||
|
etwenandClaude Opus 5
|
d8bf1f7a41 | ||
|
etwenandClaude Opus 5
|
da39122c43 | ||
|
etwenandClaude Opus 5
|
b66ae47a58 | ||
|
etwenandClaude Opus 5
|
d065db7d71 |
@@ -12,6 +12,11 @@ For_AI/
|
||||
!**/logs/.gitkeep
|
||||
!**/Logs/.gitkeep
|
||||
|
||||
# monitor 腳本的執行期產物（原始擷取、pid、狀態）
|
||||
*.raw
|
||||
*.pid
|
||||
*.state
|
||||
|
||||
# 打包輸出：產出不進 git，打包工具本身要進
|
||||
publish/*
|
||||
!publish/.gitkeep
|
||||
|
||||
+81
-34
@@ -37,8 +37,12 @@ LTC2980 margin 控制器。TTL 只負責「按順序 `sendln` 並把畫面收進
|
||||
| 電源 margin | LTC2980（= 2 × LTC2977）over PMBus，LINEAR16 編碼 |
|
||||
| 錯誤計數來源 | Linux PCIe AER sysfs（`aer_dev_*`）、EDAC sysfs（`dimm_ce/ue_count`） |
|
||||
| 平台監控 | SONiC `show platform *`（pmon container，含 `leak status` / `leak channels`） |
|
||||
| BMC 存取 | host 端 `bmc-manager run "<cmd>"`（不需對 BMC 開 SSH，繞開其 busybox 工具鏈） |
|
||||
| 壓力測試 | `mlucas-avx2`（AMD CPU）、`~/hammer/tools/stress_{mem,ssd,usb}.py` |
|
||||
| BMC 存取 | host 端 `bmc-manager run "<cmd>"`（不需對 BMC 開 SSH，繞開其 busybox 工具鏈）；`bmc-first-enroll` / `bmc-manager version\|status` |
|
||||
| 背景工作管理 | `bgctl run/list/stop --all/reset`（平台工具）；長時間 ping 用 `systemd-run --unit=` 掛成 transient unit |
|
||||
| 管理網路 | iproute2（`ip link set` / `ip address replace` / `ip route replace ... metric`）+ `ping -I` |
|
||||
| 測試 session | `hw-test-session start\|log\|status\|finish`；job log 收在 `/host/hw-eval/current/jobs/` |
|
||||
| 風扇控制 | max31790 driver rebind（`/sys/bus/i2c/drivers/max31790/{bind,unbind}`）+ `fan-speed-control.sh <%>` |
|
||||
| 壓力測試 | `mlucas-avx2`（AMD CPU）、`bgctl run` 排程的 `memtester` / `qfx5252-stress-ssd` / `qfx5252-stress-usb` |
|
||||
| 資料面流量 | `bcmcmd`（Broadcom drivshell）VLAN loopback + `tx` burst，switch unit 0 / 1 |
|
||||
| 流量報表 | DUT 上 awk（`blanton_traffic_linespeed report`）；離線 `python3 tools/bcm_mibpair_report_V1.1.0.py` |
|
||||
| 版本控制 | Git（NAS + Gitea 私有 remote；**不推 GitHub，含客戶 NDA 資料**） |
|
||||
@@ -148,8 +152,8 @@ Blanton_TTL_Script/
|
||||
├── utils/ # 可 include 的取數片段（TTL，無狀態、只 wait+sendln）
|
||||
│ ├── pcie_bus.ttl # lspci -tvvv / -vv
|
||||
│ ├── setup_pmon.ttl # show platform summary/fan/temperature/psustatus/voltage/
|
||||
│ │ # current/ssdhealth/leak status/leak channels（9 項）
|
||||
│ ├── show_dmesg.ttl # date + dmesg grep error/fail/warning
|
||||
│ │ # current/ssdhealth/leak status/leak channels + TPM（11 項）
|
||||
│ ├── show_dmesg.ttl # dmesg -T 合併 error 正則 + i2c 過濾 + dmesg -C 清空
|
||||
│ ├── show_margin_status.ttl # source margin_status_all.sh（掃 9 顆 LTC2980）
|
||||
│ ├── kill_all_process.ttl # kill $(jobs -p)
|
||||
│ ├── wait_init.ttl # 輪詢 show platform temperature 直到 pmon 起來
|
||||
@@ -172,9 +176,11 @@ Blanton_TTL_Script/
|
||||
├── blanton_pwr_data.sh # SWB VRM + PDB brick 的 Vin/Vout/Iout/Temp 表
|
||||
├── blanton_traffic_linespeed.sh # SWB loopback 線速流量（bcmcmd unit 0/1）
|
||||
│ # init/clear/show/start/stop/ps/report/run
|
||||
├── bmc_monitor.sh # BMC 監控：週期跑 free -m，寫輪替 log
|
||||
├── bmc_monitor_ddr.sh # BMC DDR 壓力：memtester，逾時 3600s
|
||||
│ # ↑ 兩支都要 chmod +x 才能跑（見 Key Constraints）
|
||||
├── mgmt_ping_monitor.sh # 10G/1G 管理網路 ping 監控（iproute2）
|
||||
│ # eth0→TARGET_10G、eth1→TARGET_1G，兩張卡都保持 up
|
||||
├── bmc_monitor.sh # BMC 監控：free -m + I2C 寫入/讀回圖樣測試
|
||||
├── bmc_monitor_ddr.sh # ⚠️ 已停用：B/C 內呼叫處已註解，DDR 壓力改走 bgctl + bmc-manager
|
||||
│ # ↑ 三支都要 chmod +x 才能跑（見 Key Constraints）
|
||||
└── LTC2980_Margin_Script/ # 電壓 margin 子專案（另有獨立 repo）
|
||||
├── margin.sh # margin_init/status/set/apply_profile/save
|
||||
├── settings/*.conf # 一個 .conf = 一顆 LTC2980（CB + SWB0/1 × CONN13~16）
|
||||
@@ -195,15 +201,20 @@ Blanton_TTL_Script/
|
||||
### 1. `config.ttl` — 全域測試參數（TTL 變數）
|
||||
|
||||
```ini
|
||||
strTestcase = "Margin" ; 測項名，進 log 檔名
|
||||
project_name = "Blanton" ; 專案名，進 log 檔名
|
||||
EN_Margin = 1 ; 1=跑 margin 掃描, 0=跳過
|
||||
strTestcase = "ENV" ; 測項名（ENV/EMC/Margin...），進 log 檔名
|
||||
EN_Margin = 0 ; 1=跑 margin 掃描, 0=跳過
|
||||
EN_log = 1 ; 1=logopen 存檔, 0=不存
|
||||
FAN_SPEED = 30 ; 傳給 fan-speed-control.sh 的風扇轉速 (%)
|
||||
SWB_UNIT0 = 1 ; 這台有 switch unit 0（流量與 wait_init 都依此）
|
||||
SWB_UNIT1 = 1 ; 這台有 switch unit 1
|
||||
prompt_login = "sonic login:"
|
||||
prompt_sonic = "admin@sonic:~$"
|
||||
prompt_sonic_root = "root@sonic:~#"
|
||||
```
|
||||
|
||||
> `SWB_UNIT0` / `SWB_UNIT1` 兩個都是 0 時，`wait_init.ttl` 直接 bypass、A/B/C 的流量段整段跳過。
|
||||
|
||||
Log 檔名規則：`<mdir>\Logs\<project_name>_<strTestcase>_<YYYYmmdd-HHMMSS>.log`
|
||||
|
||||
### 2. `settings/<board>.conf` — 一顆 LTC2980（16 channel）
|
||||
@@ -290,9 +301,9 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc
|
||||
|
||||
| 模組 | 進入點 | 功能 |
|
||||
|------|--------|------|
|
||||
| **Script A（Pre-test）** | `1_Blanton_Script_A.ttl` V1.0.3 | root 登入 → `date -s` 對時 → `show boot` → **等 pmon 起來**（`wait_init.ttl`）→ BMC 版本 → source 六個 bash 工具 → 清/讀 7 組 PCIe AER → `lspci` → margin 全掃 → PMON 九項 → dmesg → 100G port 狀態 → **一輪流量基線** → `show uptime` |
|
||||
| **Script B（Stress）** | `2_Blanton_Script_B.ttl` | `mlucas-avx2 -cpu 0:15` + 5 份 `stress_mem.py` + `stress_ssd.py` + `stress_usb.py`（**共 8 個 job**）→ 啟動兩支 BMC monitor → `jobs` → 100G port 狀態 → traffic `ps/init/clear/show/start`（**兩個 unit**）→ `while 1` 每輪完整 PMON + margin |
|
||||
| **Script C（Post-test）** | `3_Blanton_Script_C.ttl` | `kill $(jobs -p)` → PMON + dmesg → 7 組 AER → 停兩支 BMC monitor 並 `cat` 其 log → `cat` mlucas / stress_ssd log → traffic `stop` + `report`（兩個 unit）→ `show reboot-cause` / `show uptime` → 完成 messagebox |
|
||||
| **Script A（Pre-test）** | `1_Blanton_Script_A.ttl` V1.0.7 | root 登入 → `date -s` 對時 → 清 job log → `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**（`wait_init.ttl`）→ DUT 清單（boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe）→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → 100G uplink 設定與狀態 → **一輪流量基線** → `show uptime` |
|
||||
| **Script B（Stress）** | `2_Blanton_Script_B.ttl` | `mlucas-avx2 -cpu 0:15` + `bgctl run` 的 `memtester` / SSD / USB 壓力 + BMC DDR（`bmc-manager run memtester`）→ BMC USB net test（探測 cdc_ncm 介面，`systemd-run` 掛 4 小時 ping）→ 啟動 mgmt ping 監控 → `jobs` / `bgctl list` → traffic（依 `SWB_UNIT`）→ `while 1` 每輪完整 PMON + margin |
|
||||
| **Script C（Post-test）** | `3_Blanton_Script_C.ttl` | `kill $(jobs -p)` + `bgctl stop --all` / `reset` + 停 BMC USB unit 與 mgmt ping → PMON + dmesg → 7 組 AER + `ras-mc-ctl` → BMC USB `journalctl` → `ip -s link show eth0/eth1` → `cat` mlucas log → traffic `stop` + `report` → NVMe 健康 → `show reboot-cause` / `uptime` → **job log 複製到 USB（帶時戳）** → `hw-test-session finish` |
|
||||
| **SWB loopback 流量** | `blanton_traffic_linespeed <sub> [-u 0\|1\|all]` | `init`（VLAN 30..137，成對 `cdN`/`cdN+32`）→ `clear` → `start`（`tx 100 length=512`）→ `stop` → `report`（per-pair TX/RX 交叉比對 PASS/FAIL）；另有 `ps`（link/speed）與 `run`（一條龍） |
|
||||
| **FPGA 暫存器存取** | `cb_fpga <fn> <addr> [data]`、`fpga_rescan` | pcimem 打 `resource0` + BAR 相對 offset，32-bit word；BDF 於 source 時自動偵測；`pmc/icb/swb0/swb1_fpga` 走 VSPI 窗口 |
|
||||
| **CB I2C / PMBus** | `cb_i2c_init/scan/read/write`、`cb_pmbus_read/write/linear16` | FPGA F3 OpenCores master，**讀用 Repeated START**（SMBus/PMBus 裝置必需） |
|
||||
@@ -300,7 +311,9 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc
|
||||
| **溫度** | `temp_cb` / `temp_icb` / `temp_all` | TMP75（12-bit 左靠齊 ×0.0625）、TMP432（含 RANGE bit 的 −64 偏移、remote1/2） |
|
||||
| **電源資料** | `pwr_data` / `pwr_data_pdb` / `pwr_pdb_id` | SWB VRM 全軌 Vin/Vout/Iout/Temp 對齊表；MP29816 解析度**執行期從 `MFR_VOUT_SCALE_LOOP` 讀**，不寫死 |
|
||||
| **電壓 Margin** | `margin_init/status/set/apply_profile/save` + `*_all.sh` | 9 顆 LTC2980（CB CONN13、SWB0/1 CONN13~16）批次 high/low margin 與狀態掃描。v2.6.0 起 SWB 路徑真正接上共用 cb_i2c 後端，並有 `margin_swb_info/probe/scan/reset/sem` 除錯包裝 |
|
||||
| **BMC 監控 / 壓力** | `bmc_monitor.sh` / `bmc_monitor_ddr.sh` `{start\|stop\|status\|fg\|tail\|clear}` | 由 host 端 `bmc-manager run` 週期性打 BMC，背景 detached、log 輪替。前者跑 `free -m`（逾時 30s），後者跑 `memtester`（逾時 3600s），各自獨立 log 與 pid |
|
||||
| **管理網路 ping** | `mgmt_ping_monitor.sh {start\|stop\|status\|fg\|tail\|clear\|summary}` | 一輪 = 10G leg + 1G leg，各自 `ip address/route replace` 後 `ping -I <if>` 打自己的對端；兩張卡都保持 up。每 leg 記 `ip -s link show` 與該卡的 TX/RX packet delta，另出一行可 grep 的 `RESULT` |
|
||||
| **BMC 監控** | `bmc_monitor.sh {start\|stop\|status\|fg\|tail\|clear}` | 由 host 端 `bmc-manager run` 週期取樣 `free -m`，另加一組 **I2C 寫入/讀回圖樣測試**（bus 1、addr 0x41、offset 0x00C0，寫 `55AA55AA` 讀回、再寫 `AA55AA55` 讀回）。B 啟動、C 停止並 `cat log/bmc_poll.log` |
|
||||
| **BMC DDR 壓力** | `bmc_monitor_ddr.sh` | ⚠️ **已停用** —— B/C 內呼叫處已註解，改由 `bgctl run bmc-manager run memtester` 取代。檔案保留未刪 |
|
||||
| **打包交付** | `publish/publish.sh [-v Ver] [-f] [-z] [-n]` | 版號讀 Script A 檔頭，複製成 `publish/Script_ABC_Blanton_<Ver>/`，排除 `*.log`，可另出 zip |
|
||||
|
||||
---
|
||||
@@ -316,15 +329,20 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc
|
||||
Script A ── logopen Logs/Blanton_Margin_<ts>.log ──┐ (之後所有畫面都進這個檔)
|
||||
│ │
|
||||
├─ 登入 admin → sudo -i → root@sonic:~# │
|
||||
├─ date -s "<host 時間>" ← 讓 DUT 時戳可對照 │
|
||||
├─ date -s "<host 時間>" ← 讓 DUT 時戳可對照
|
||||
├─ rm /host/hw-eval/current/jobs/* ← 清掉上一輪的 job log
|
||||
├─ hw-test-session start / log / status
|
||||
├─ 風扇：max31790 unbind→sleep→bind（18-0020 與 25-0020 各一次）
|
||||
│ → fan-speed-control.sh <FAN_SPEED> → 回答 y 確認 │
|
||||
├─ source 6 個 bash 工具（函數進 shell） │
|
||||
│ └ 此時 blanton_fpga_pcimem.sh 印出 [INFO] CB FPGA BDF = ...（自動偵測結果）
|
||||
├─ AER 基線：7 組 EP+RP 的 correctable/nonfatal/fatal
|
||||
├─ lspci -tvvv / -vv ← 拓樸 + link speed/width
|
||||
├─ margin_status_all ← 9 顆 LTC2980 × 16ch 的 Vout 與偏差%
|
||||
├─ show platform × 9 ← fan/temp/psu/voltage/current/ssdhealth/leak status/leak channels
|
||||
├─ dmesg | grep error/fail/warning
|
||||
├─ show interfaces status Ethernet513,Ethernet514 ← 100G uplink 狀態
|
||||
├─ dmesg -T（合併 error 正則 + i2c 過濾）→ dmesg -C 清空 ← C 只會看到 soak 期間新產生的
|
||||
├─ mgmt ping：start → pause 45 → stop → cat log ← 10G/1G 各一輪
|
||||
├─ 100G uplink：lpmode off → 兩埠 startup → config save → show interfaces status
|
||||
├─ 一輪流量基線：ps → init → clear → show → start → stop → report
|
||||
└─ show uptime
|
||||
（最前面還有 wait_init.ttl：輪詢 show platform temperature 直到不再回報
|
||||
@@ -336,8 +354,10 @@ Script A ── logopen Logs/Blanton_Margin_<ts>.log ──┐ (之後所有畫
|
||||
Script B ── 同一個 log 檔續寫（logwrite 分隔線）
|
||||
├─ 背景程序：mlucas-avx2(CPU) ×1, stress_mem ×5, stress_ssd, stress_usb → 共 8 個 job
|
||||
│ （stress_hhmd / stress_pcie 已於 2026-08-17 移除，見 Status 表 ScriptB #5/#7 "KC Remove"）
|
||||
├─ BMC: bmc_monitor.sh start + bmc_monitor_ddr.sh start
|
||||
├─ jobs ← 確認全部起來了
|
||||
├─ bgctl run: memtester 1G / qfx5252-stress-ssd / qfx5252-stress-usb
|
||||
├─ BMC: bmc-manager run memtester（DDR）、cdc_ncm 介面探測 + systemd-run 4hr ping（USB）
|
||||
├─ mgmt_ping_monitor.sh start
|
||||
├─ jobs / bgctl list ← 確認全部起來了
|
||||
├─ traffic: ps → init → clear → (pause 15) → show → start （unit 0 與 1，VLAN 30..137 loopback）
|
||||
└─ while 1: setup_pmon（9 項）→ margin_status_all （soak 期間持續取樣）
|
||||
▼
|
||||
@@ -348,10 +368,14 @@ Script C
|
||||
├─ kill $(jobs -p) ← 收掉所有背景壓力
|
||||
├─ show platform × 9 + dmesg ← 收工快照
|
||||
├─ AER 再抓一次 ← 與 Script A 基線相減 = 本輪新增錯誤
|
||||
├─ BMC: 兩支 monitor stop → cat log/bmc_poll.log + log/bmc_ddr.log
|
||||
├─ cat mlucas_amm_log / log.stress_ssd ← 壓力程式自身的 pass/fail
|
||||
├─ bgctl stop --all / reset、systemctl stop BMC USB unit、mgmt ping stop
|
||||
├─ journalctl BMC USB unit、ip -s link show eth0/eth1、ras-mc-ctl --summary
|
||||
├─ cat mlucas_amm_log ← CPU 壓力程式自身的 pass/fail
|
||||
├─ nvme smart-log / smartctl -x /dev/nvme0
|
||||
├─ traffic: stop → report ← per-pair TX/RX 交叉比對，PASS/FAIL/NA（exit 2 = 有 FAIL/NA）
|
||||
└─ show reboot-cause / show uptime ← soak 期間有沒有意外重開，一眼看得到
|
||||
├─ show reboot-cause / show uptime ← soak 期間有沒有意外重開，一眼看得到
|
||||
├─ cp -r /host/hw-eval/current/jobs/ /mnt/usb/jobs-<YYYYmmdd>_<HHMM>
|
||||
└─ hw-test-session finish
|
||||
▼
|
||||
[messagebox "GOOD JOB! Test Case DONE"] → 人工檢查 log
|
||||
```
|
||||
@@ -409,12 +433,21 @@ pwr_data
|
||||
⚠️ 但 `.gitattributes` **不會回頭重寫既有工作區檔案** —— 早於它 clone 出來的目錄，
|
||||
需 `git rm --cached -r . && git reset --hard` 或刪掉目錄重新 checkout 才會生效，
|
||||
而且 `git status` 乾淨**不代表**磁碟上是 LF（`--renormalize` 只在進 index 的路上轉換）。
|
||||
13. **`bmc_monitor*.sh` 需要執行權**：它們是本專案第一組「被執行」而非被 `source` 的腳本，
|
||||
13. **`bmc_monitor*.sh` / `mgmt_ping_monitor.sh` 需要執行權**：它們是本專案「被執行」而非被 `source` 的腳本，
|
||||
而 `do_start` 內部是 `setsid "$SCRIPT_PATH" __daemon`，**即使用 `bash x.sh start` 呼叫
|
||||
也仍需 exec bit**。git 記錄為 `100644`，且 Windows → 隨身碟 → DUT 的路徑無法攜帶權限位元，
|
||||
所以佈署後要先 `chmod +x ~/Blanton_Script/bmc_monitor*.sh`，否則 Script B 的 start 會
|
||||
`Permission denied`、Script C 的 `cat` 撲空（BMC 段全空但不會報錯）。
|
||||
14. **客戶 NDA**：`docs/` 內含客戶 FPGA 規格與 schematic 衍生資訊，**本 repo 只推 NAS + Gitea，
|
||||
14. **TTL 的 `timeout` 是全域的，設了就要還原**：`timeout = N` 之後**每一個** `wait` 都被限制在 N 秒，
|
||||
逾時的 `wait` 會直接返回而沒吃到 prompt，於是下一個指令在前一個還沒跑完就送出，之後整輪錯開一拍。
|
||||
`wait_init.ttl` 設 60 後還原 0；Script B 的 cdc_ncm 探測設 15，在 `:skip_ping`（兩條路徑的匯流點）還原。
|
||||
加新的 `timeout` 時務必配一個 `timeout = 0`。
|
||||
15. **兩張管理網卡同網段時會 ARP flux**：`mgmt_ping_monitor.sh` 讓 eth0/eth1 同時 up，若兩者在同一個
|
||||
/24（本專案預設 `192.168.1.99` / `.101`），對端 ARP 可能由任一張卡回應，回包落在沒送封包的那張，
|
||||
量到的數字不能歸屬。`ARP_STRICT=1` 會設 `arp_ignore=1` / `arp_announce=2` 擋掉；
|
||||
另外每個 leg 的 `RESULT` 行帶 `nic_tx` / `nic_rx`（該卡自己的 packet delta），
|
||||
**ping 成功但 `nic_tx` 接近 0 就代表封包從另一張卡出去了**。兩張卡在不同網段時可設 0。
|
||||
16. **客戶 NDA**：`docs/` 內含客戶 FPGA 規格與 schematic 衍生資訊，**本 repo 只推 NAS + Gitea，
|
||||
絕不推 GitHub**。
|
||||
|
||||
---
|
||||
@@ -554,13 +587,12 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值，
|
||||
**目標：** 壓力程序起得來、起不來看得出來，soak 迴圈有時戳、可控週期。
|
||||
|
||||
**包含：**
|
||||
- [ ] 部署 `~/hammer/tools/` 前置檢查：新增 `utils/check_hammer_tools.ttl`
|
||||
（`ls -l ~/hammer/tools/` + `ls ~/hammer/tools/amd/mlucas-avx2`），缺檔就別往下跑
|
||||
- [ ] 前置檢查：`~/hammer/tools/amd/mlucas-avx2` 仍是唯一還依賴 hammer 的項目，缺檔就別往下跑
|
||||
- [ ] 壓力程序 log 加時戳，避免多輪覆蓋：
|
||||
`mlucas_amm_log` → `~/logs/mlucas_amm_log_$(date +%Y%m%d-%H%M%S).log`（Status 表 ScriptB #2 的既定寫法）
|
||||
- [ ] `jobs` 之後加驗證：期望 **8** 個背景 job（1 mlucas + 5 mem + ssd + usb），
|
||||
數量不符時 `messagebox` 提示
|
||||
（`stress_hhmd` / `stress_pcie` 已於 2026-08-17 由 KC 移除，原本的 10 改為 8）
|
||||
- [x] 壓力層改用平台的 `bgctl`（`memtester` / `qfx5252-stress-ssd` / `qfx5252-stress-usb`），
|
||||
取代 `~/hammer/tools/stress_*.py`；Script B 以 `jobs` + `bgctl list` 兩者並列確認
|
||||
- [ ] `bgctl list` 之後加數量驗證，不符時 `messagebox` 提示（目前只印出來給人看）
|
||||
- [ ] soak 迴圈改為「每輪先 `date`、跑完 `pause <間隔>`」，
|
||||
把 `EN_LoopInterval`（預設 600 秒 = 10 min）放進 `config.ttl`，兌現 Status 表的「Get data every 10mins」
|
||||
**← 目前 `while 1` 仍無 `pause`，實際間隔取決於指令執行時間，不是 10 分鐘**
|
||||
@@ -621,9 +653,13 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值，
|
||||
UE > 0 或 fatal/nonfatal > 0 直接標 FAIL
|
||||
- [ ] Margin 表解析：把 9 顆 × 16ch 的 Vout / 偏差% 收成 CSV，標出超出 ±(profile 設定值 + 容差) 的軌
|
||||
- [ ] 溫度 / 風扇 / 電源趨勢：soak 期間逐輪取樣 → CSV（可直接畫圖）
|
||||
- [ ] Stress 判定：`mlucas_amm_log` 與 `log.stress_ssd` 的錯誤關鍵字掃描
|
||||
- [ ] Stress 判定：`mlucas_amm_log` 與 `bgctl` 各工作的 log
|
||||
（`/host/hw-eval/.../qfx5252-stress-{ssd,usb}.log`）的錯誤關鍵字掃描
|
||||
- [ ] Traffic 判定：**直接複用 `tools/bcm_mibpair_report_V1.1.0.py`**（它已能吃 console log 的
|
||||
`show c` 段落並輸出 PASS/FAIL），不要重寫一套解析
|
||||
- [ ] 管理網路判定：抓 `mgmt_ping_monitor.sh` 的 `RESULT` 行（已是單行可 grep 格式），
|
||||
並檢查 `nic_tx`/`nic_rx` 不為 0 —— 否則 PASS 也不能歸屬到那張卡
|
||||
- [ ] NVMe / EDAC：`nvme smart-log`、`smartctl -x`、`ras-mc-ctl --summary` 的前後差值
|
||||
- [ ] 產出 `Logs/<run>_summary.md`：一頁 PASS/FAIL + 每個子系統一行結論
|
||||
|
||||
**驗收條件：** 拿一份既有 log（如 `Blanton_Margin_20260814-164122.log`）跑解析器，
|
||||
@@ -676,8 +712,19 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值，
|
||||
文件已加過期警告但內文未動。第 1~4 章不受影響。
|
||||
- **`wait_init.ttl` 沒有逾時出口**：若 pmon 一直起不來，`result` 既非 1 也非 2，迴圈會無限重試，
|
||||
Script A 既不往下走也不報錯。可加最大重試次數 + `messagebox` 提示。
|
||||
- **`bmc_monitor*.sh` 的執行權**：目前需人工 `chmod +x`。可在 Script B 起它們之前補一行
|
||||
`chmod +x`（比照既有的 `chmod +x ~/hammer/tools/amd/mlucas-avx2`），或在 publish 時處理。
|
||||
- **`bmc_monitor_ddr.sh` 的壓力強度**：`memtester 500M 1` 配 `INTERVAL_SEC=5` 等於整個 soak
|
||||
近乎不間斷壓 BMC 記憶體，而 BMC 同時還要服務 host 的 `show platform` 查詢。
|
||||
若 PMON 取數變慢，先調大 `INTERVAL_SEC` 或縮小 memtester 的量。
|
||||
- **腳本的執行權**：`mgmt_ping_monitor.sh` 與 `bmc_monitor.sh`（以及已停用的 `bmc_monitor_ddr.sh`）需人工 `chmod +x`。
|
||||
可在 Script B 起它之前補一行（比照既有的 `chmod +x ~/hammer/tools/amd/mlucas-avx2`），或在 publish 時處理。
|
||||
- **`bmc_monitor_ddr.sh` 已停用**：B/C 內的呼叫已註解，DDR 壓力改由 `bgctl run bmc-manager run memtester`
|
||||
取代。檔案還留在 repo，待確認不再需要後刪除，或在檔頭標註停用。
|
||||
（`bmc_monitor.sh` 已於 2026-08-25 重新啟用，並加上 I2C 寫入/讀回圖樣測試。）
|
||||
- **`rm /host/hw-eval/current/jobs/*` 清不掉子目錄**：Script A 用的是不帶 `-r` 的 `rm`，但 Script C
|
||||
是 `cp -r` 那個目錄，代表裡面可能有子目錄。清不乾淨的話，上一輪的 job log 會被當成這輪的一起
|
||||
複製到 USB。要確認 `bgctl` 的產出結構，必要時改成 `rm -rf .../jobs/*`。
|
||||
- **風扇確認提示是無限等待**：`wait "Set all configured fan channels to"` 之後才 `sendln "y"`；
|
||||
若該版 `fan-speed-control.sh` 不再詢問（或字串改了），這個 `wait` 會永遠等下去，Script A 停在那裡
|
||||
且不報錯。加個 `timeout` + 分支會比較保險（記得還原 `timeout = 0`）。
|
||||
- **USB 掛載點未檢查**：Script C 直接 `cp -r ... /mnt/usb/`，`/mnt/usb` 沒掛載時只會失敗一行，
|
||||
整輪測試的 job log 就沒被帶出來。可在複製前先 `mountpoint -q /mnt/usb` 判斷。
|
||||
- **Script C 沒有倒出 `mgmt_ping.log`**：C 會 `stop` 管理網路監控並印 `ip -s link show`，
|
||||
但沒有 `cat ~/Blanton_Script/log/mgmt_ping.log`，所以整個 soak 期間累積的 per-leg `RESULT`
|
||||
留在 DUT 上、沒進 master log。Script A 那輪基線有 `cat`。
|
||||
|
||||
@@ -9,9 +9,13 @@ DUT 端的取數能力由一組 **bash 工具庫** 提供（CB FPGA `pcimem` →
|
||||
|
||||
完整架構見 [ARCHITECTURE.md](ARCHITECTURE.md)。
|
||||
|
||||
**目前狀態（2026-08-21）**：Script A/B/C 為 **V1.0.3**，涵蓋 PCIe AER / DDR EDAC / PMON /
|
||||
電壓 margin（9 顆 LTC2980）/ SWB loopback 線速流量（兩個 switch unit）/ BMC 監控與 DDR 壓力。
|
||||
SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。交付用 `./publish/publish.sh` 打包。
|
||||
**目前狀態（2026-08-25）**：Script A 為 **V1.0.7**（A 集中記錄 A/B/C 三支的變更）。涵蓋
|
||||
PCIe AER / DDR EDAC / PMON+TPM / 電壓 margin（9 顆 LTC2980）/ SWB loopback 線速流量 /
|
||||
10G+1G 管理網路 ping / BMC DDR 與 USB / NVMe 健康，壓力層走平台的 `bgctl`。
|
||||
switch unit 數量由 `config.ttl` 的 `SWB_UNIT0` / `SWB_UNIT1` 宣告，風扇轉速由 `FAN_SPEED` 指定。
|
||||
測試 job log 收在 DUT 的 `/host/hw-eval/current/jobs/`，Script C 結束時帶時戳複製到 `/mnt/usb/`。
|
||||
最後一次發布是 **V1.0.5**；SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。
|
||||
交付用 `./publish/publish.sh` 打包。
|
||||
|
||||
## 技術棧
|
||||
|
||||
@@ -24,6 +28,9 @@ SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。交付用 `./publish
|
||||
- **資料面流量**：`bcmcmd`（Broadcom drivshell）SWB loopback，VLAN 30..137 成對 `cdN`/`cdN+32`，
|
||||
`tx 100 length=512`；判定讀 ASIC `MIB_TPOK`/`MIB_RPOK` 做 per-pair 雙向交叉比對
|
||||
- **BMC**：host 端 `bmc-manager run "<cmd>"`，不需對 BMC 開 SSH（繞開其 busybox 工具鏈）
|
||||
- **背景工作**：平台的 `bgctl run/list/stop --all/reset`；長時間 ping 用 `systemd-run --unit=` 掛 transient unit
|
||||
- **管理網路**：iproute2（`ip link set` / `address replace` / `route replace ... metric`）+ `ping -I`
|
||||
- **風扇**：max31790 driver rebind + `fan-speed-control.sh <%>`（轉速讀 `config.ttl` 的 `FAN_SPEED`）
|
||||
- **判定資料源**：PCIe AER sysfs（`aer_dev_*`）、EDAC sysfs（`dimm_{ce,ue}_count`）、SONiC `show platform *`
|
||||
- **無編譯步驟**：TTL 與 bash 都直譯執行
|
||||
|
||||
@@ -70,13 +77,21 @@ blanton_traffic_linespeed report -f <log> # 離線重解一份存下來的 cons
|
||||
# 離線版報表（開發機上跑，選項更多）
|
||||
python3 tools/bcm_mibpair_report_V1.1.0.py <drivshell.log> -a
|
||||
|
||||
# ── BMC 監控 / 壓力（需先 chmod +x！）────────────────
|
||||
chmod +x ~/Blanton_Script/bmc_monitor*.sh # 佈署後必做一次
|
||||
~/Blanton_Script/bmc_monitor.sh start # free -m 取樣，log/bmc_poll.log
|
||||
~/Blanton_Script/bmc_monitor_ddr.sh start # memtester，log/bmc_ddr.log
|
||||
~/Blanton_Script/bmc_monitor.sh status # pid / log 路徑 / log 大小
|
||||
~/Blanton_Script/bmc_monitor.sh stop
|
||||
# ⚠️ tail 子命令是 tail -f，會卡住不返回；要倒 log 請直接 cat log/*.log
|
||||
# ── 管理網路 ping 監控（需先 chmod +x！）──────────────
|
||||
chmod +x ~/Blanton_Script/*.sh # 佈署後必做一次
|
||||
~/Blanton_Script/mgmt_ping_monitor.sh fg # 前景試一輪（約 30 秒），確認 IP/對端/網卡
|
||||
~/Blanton_Script/mgmt_ping_monitor.sh start # 背景，log/mgmt_ping.log（start 會先清空）
|
||||
~/Blanton_Script/mgmt_ping_monitor.sh status
|
||||
~/Blanton_Script/mgmt_ping_monitor.sh stop
|
||||
~/Blanton_Script/mgmt_ping_monitor.sh summary # 只看每段的 RESULT 行
|
||||
# ⚠️ tail 子命令是 tail -f，會卡住不返回；TTL 裡要倒 log 請用 cat
|
||||
# bmc_monitor.sh start/stop # free -m + BMC I2C 寫入/讀回圖樣測試，log/bmc_poll.log
|
||||
# ⚠️ bmc_monitor_ddr.sh 已停用（DDR 壓力改走 bgctl + bmc-manager），檔案仍在
|
||||
|
||||
# ── 背景壓力（平台工具）──────────────────────────────
|
||||
bgctl run /usr/sbin/memtester 1G 100
|
||||
bgctl list
|
||||
bgctl stop --all ; bgctl reset --yes
|
||||
|
||||
# ── 各工具的 help ───────────────────────────────────
|
||||
blanton_fpga_help ; blanton_cb_i2c_help ; vi2c_help
|
||||
@@ -166,13 +181,25 @@ Tera Term 連 COM port (115200-8-N-1)
|
||||
`.gitattributes` 已擋住 git 這條路徑，但**不會回頭重寫既有工作區檔案**：早於它 clone 的目錄要
|
||||
`git rm --cached -r . && git reset --hard` 才會生效，而且 **`git status` 乾淨不代表磁碟上是 LF**
|
||||
（`--renormalize` 只在進 index 的路上轉換）。驗證要看 byte：`grep -rlU $'\r' <dir>`
|
||||
- ⚠️ **`bmc_monitor*.sh` 需要 exec bit**，且**不是改用 `bash x.sh start` 就能繞過** ——
|
||||
- ⚠️ **TTL 的 `timeout` 是全域的**：`timeout = N` 之後**每一個** `wait` 都被限制在 N 秒。逾時的 `wait`
|
||||
直接返回而沒吃到 prompt，下一個指令就在前一個還沒跑完時送出，之後整輪錯開一拍。**設了一定要
|
||||
配一個 `timeout = 0` 還原**（`wait_init.ttl` 60→0；Script B 的 cdc_ncm 探測 15→0 在 `:skip_ping`）
|
||||
- ⚠️ **兩張管理網卡同網段會 ARP flux**：`mgmt_ping_monitor.sh` 讓 eth0/eth1 同時 up，同一個 /24 下
|
||||
對端 ARP 可能由任一張卡回應，回包落在沒送封包的那張。`ARP_STRICT=1` 擋掉；判讀時看 `RESULT` 行的
|
||||
`nic_tx`/`nic_rx`（該卡自己的 packet delta）—— **ping 成功但 `nic_tx` 接近 0 = 封包從另一張卡出去的**
|
||||
- ⚠️ **`ip -s link` 的欄位順序是 `bytes packets errors ...`**，要 packet 數取的是**第 2 欄**不是第 1 欄
|
||||
- ⚠️ **`mgmt_ping_monitor.sh` / `bmc_monitor*.sh` 需要 exec bit**，且**不是改用 `bash x.sh start` 就能繞過** ——
|
||||
`do_start` 內部是 `setsid "$SCRIPT_PATH" __daemon`，直接執行該路徑。git 記錄是 `100644`，
|
||||
Windows → 隨身碟 → DUT 也帶不動權限位元，所以佈署後要 `chmod +x ~/Blanton_Script/bmc_monitor*.sh`。
|
||||
沒做的話 Script B 的 start 會 Permission denied、Script C 的 `cat` 撲空，而且**不會報錯**，
|
||||
只會得到一份 BMC 段全空的 log
|
||||
- ⚠️ **`bmc_monitor*.sh` 的 `tail` 子命令是 `tail -n 50 -f`，永遠不返回**。TTL 裡絕對不要呼叫它
|
||||
沒做的話 start 會 Permission denied、後面的 `cat` 撲空，而且**不會報錯**，只會得到一份全空的 log
|
||||
- ⚠️ **這幾支 monitor 的 `tail` 子命令是 `tail -n 50 -f`，永遠不返回**。TTL 裡絕對不要呼叫它
|
||||
（Script C 曾因此卡死，`18ce7a1` 改成直接 `cat log/*.log`）
|
||||
- ⚠️ **背景 monitor 的 `start` 只等 1 秒就返回**，不是跑完一輪。Script A 要做一輪基線的話，
|
||||
`start` 和 `stop` 中間必須 `pause`（目前是 45 秒，一輪約 30 秒）—— 少了它 log 只會有 START banner
|
||||
- ⚠️ **風扇那段設完會 `wait "Set all configured fan channels to"` 等確認提示再送 `y`**。該提示沒出現
|
||||
（版本改了、字串換了）就會永遠卡住且不報錯 —— 現場看起來像 Script A 當掉
|
||||
- ⚠️ **`show_dmesg.ttl` 結尾會 `dmesg -C` 清空 kernel ring buffer**。這是刻意的（讓 Script C 只看到
|
||||
soak 期間新產生的訊息），但代表事後在 DUT 上 `dmesg` 撈不到舊訊息 —— 內容只存在 master log 裡
|
||||
- ⚠️ **DUT 的 awk 對 `%d` 會夾到 INT32**（2147483647）。線速流量的計數器是 ~10¹² 量級，
|
||||
所以 awk 裡輸出大數一律用 `%.0f`（double 域，精確到 2⁵³）。`blanton_traffic_linespeed.sh`
|
||||
V0.4.1 已修；寫新的 awk 報表時要記得同一件事
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
# v1.0.7 — Management-link ping test, and the fans start where you set them
|
||||
|
||||
## ✨ New features
|
||||
|
||||
**10G / 1G management port ping test**
|
||||
* New `mgmt_ping_monitor.sh` exercises both management NICs in one loop: it brings each up with iproute2, waits for the link to settle, then pings that NIC's own target — `eth0` → `192.168.1.30`, `eth1` → `192.168.1.31`.
|
||||
* Runs detached like the other monitors: `start` / `stop` / `status` / `fg` / `summary`. `start` clears the previous log, so one run means one log.
|
||||
* Every leg writes a single greppable line — `RESULT 10G eth0 -> 192.168.1.30 tx=10 rx=10 loss=0% rtt_avg=1.743ms nic_tx=10 nic_rx=10 PASS` — so a whole soak can be read with `summary` instead of scrolling.
|
||||
* `nic_tx` / `nic_rx` are that interface's own counter delta across the burst. If a ping passes but they stay near zero, the traffic left on the *other* NIC and the number does not belong to this one.
|
||||
* Script A runs one round as a baseline; Script B leaves it running through the soak.
|
||||
|
||||
**Fan speed is set at the start of every run**
|
||||
* `FAN_SPEED` in `config.ttl` (default 30) is applied by Script A, so a run no longer inherits whatever the previous test left the fans doing.
|
||||
* Both MAX31790 controllers are rebound first, which clears a controller left in an odd state.
|
||||
|
||||
**Stress load moved onto the platform's own job runner**
|
||||
* CPU, DDR, SSD and USB stress now go through `bgctl`, replacing the `~/hammer/tools/stress_*.py` scripts. `bgctl list` shows what is running; Script C stops everything with `bgctl stop --all`.
|
||||
* BMC DDR stress runs through `bmc-manager`, and a new BMC USB test discovers the `cdc_ncm` interface and pings across it for four hours under `systemd-run`.
|
||||
|
||||
**More of the DUT on record**
|
||||
* Script A now captures the boot image, `show version`, firmware status, system EEPROM, SSD health, TPM version and NVMe SMART data before the test starts.
|
||||
* PCIe AER now sits next to `ras-mc-ctl --summary`, and Script C closes with NVMe health so a disk that degraded during the soak is visible.
|
||||
* `dmesg` capture uses human-readable timestamps and one combined error pattern, then clears the ring buffer — so Script C's dmesg shows only what the soak produced, not everything since boot.
|
||||
* Each run opens with `hw-test-session start` and closes with `finish`; Script C copies the job logs to `/mnt/usb/jobs-<date>_<time>` so they leave the DUT with the run they belong to.
|
||||
|
||||
## 🐛 Bug fixes
|
||||
|
||||
**Script A no longer hangs waiting for ports that are already up**
|
||||
* The readiness gate required *more than* 216 ports up, but a fully cabled unit reports exactly 216 — 108 loopback pairs times two. Script A sat in the poll loop forever on the very state it was waiting for. It now proceeds at 216.
|
||||
* The comments in `wait_init.ttl` still described the old "more than" rule; they now match the code, so the off-by-one cannot be reintroduced by reading the wrong line.
|
||||
|
||||
**The macro no longer runs ahead of the DUT during the soak**
|
||||
* Script B set a 15-second wait cap for the `cdc_ncm` probe and never restored it. In Tera Term that cap is global, so it stayed in force for everything after — including the traffic setup, which walks 108 VLANs per switch unit and takes far longer. A timed-out wait returns without the prompt, leaving every later command issued a step early.
|
||||
|
||||
**Fan controller is no longer bound twice**
|
||||
* One of the two MAX31790 controllers received a second `bind` immediately after the first, which could only fail because the driver was already attached.
|
||||
|
||||
## 📦 Downloads
|
||||
|
||||
| File | Contents |
|
||||
|---|---|
|
||||
| `Script_ABC_Blanton_V1.0.7.zip` | The full Tera Term working directory, including `Blanton_Script/` to copy onto the DUT |
|
||||
|
||||
**After copying to the DUT:**
|
||||
|
||||
```bash
|
||||
chmod +x ~/Blanton_Script/*.sh # required -- see below
|
||||
grep -rlU $'\r' ~/Blanton_Script # expect no output
|
||||
```
|
||||
|
||||
The `chmod +x` is not optional. `mgmt_ping_monitor.sh` is executed rather than sourced and re-execs its own path to daemonise, and neither git (mode 100644) nor a Windows/USB copy carries the execute bit. Without it `start` fails with `Permission denied` and the later `cat` finds nothing — **the section ends up empty and nothing reports an error**.
|
||||
|
||||
## ⚠️ Before you run
|
||||
|
||||
* **Declare the switch population.** `SWB_UNIT0` / `SWB_UNIT1` in `config.ttl` say which switch units this DUT has. Traffic and the readiness gate both follow them: a single unit gets `-u 0` or `-u 1`, and with neither set the traffic stage is skipped instead of failing against hardware that is not there.
|
||||
* **Both management NICs stay up.** If they share a subnet, keep `ARP_STRICT=1` — otherwise the target's ARP can be answered by either NIC and a reply may arrive on the one that did not send.
|
||||
* **Management connectivity is exercised, not preserved.** Drive the run from the serial console.
|
||||
* **Two settings persist to `config_db.json`** and survive a reboot: LLDP is left disabled, and the 100G uplinks are left configured. Restore them before the DUT moves on.
|
||||
|
||||
## 🔗 Links
|
||||
|
||||
* [ARCHITECTURE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/ARCHITECTURE.md) — test flow, data models, constraints
|
||||
* [CLAUDE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/CLAUDE.md) — commands and the bench gotchas
|
||||
|
||||
**Full changelog:** [V1.0.5...V1.0.7](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/compare/V1.0.5...V1.0.7)
|
||||
@@ -0,0 +1,61 @@
|
||||
# v1.0.8 — Both management links tested at once, and ARP stops looking like packet loss
|
||||
|
||||
## ✨ New features
|
||||
|
||||
**Both management NICs are pinged simultaneously and continuously**
|
||||
* `mgmt_ping_monitor.sh` no longer alternates fixed bursts. `start` configures both NICs, then pings from each at the same time and keeps accumulating until stopped — so a soak-long run is one continuous measurement rather than a series of snapshots.
|
||||
* `stop` renders a report into the log: the last 20 entries per NIC, each one's ping statistics and PASS/FAIL, then `ip -s link show` for both interfaces.
|
||||
* Every line is timestamped, and a request that got no reply prints a marker instead of merely being absent — a drop is visible in the tail, not inferred from a gap in the sequence numbers.
|
||||
* `status` shows both pids and how many replies each NIC has received so far, which is the quick way to see one side is dead without waiting for the report.
|
||||
|
||||
**The USB stress target is detected, not assumed**
|
||||
* New `usb_target.sh` reports an inserted USB device's node, mount point or by-id path. Script B asks it for the node and builds the stress command from the answer, so a stick that enumerates as `sdb` no longer sends a write test at whatever `/dev/sda1` happens to be.
|
||||
* It excludes whatever disk backs `/` and `/host` — this platform can boot from a USB DOM, which also reports as USB — refuses to guess when several USB disks are present, and when it mounts, verifies the result is actually writable rather than trusting that `mount` succeeded.
|
||||
|
||||
**BMC I2C integrity is exercised through the soak**
|
||||
* `bmc_monitor.sh` is back in the run and now writes two complementary patterns to a BMC scratch register and reads each back. One pattern alone cannot catch a bit stuck the same way it was written; repeating the pair through the soak turns an intermittent I2C fault into something the log records rather than something the tester has to witness.
|
||||
|
||||
**Test artefacts leave the DUT with the run**
|
||||
* Script C copies `bmc_poll.log` and `mgmt_ping.log` into the job directory before it is archived to USB, so the monitors' output travels with the `bgctl` job logs.
|
||||
* DDR stress now runs continuously instead of stopping after 100 passes, matching the other soak loads.
|
||||
|
||||
## 🐛 Bug fixes
|
||||
|
||||
**Three FAILs that were not link faults**
|
||||
* A bench run reported `FAIL=3` with zero NIC errors, zero drops and sub-millisecond replies. Every failure was missing exactly `icmp_seq=1` and nothing else: once the neighbour entry for the target expires, the first echo request is spent resolving ARP and `ping` counts it as loss. A discarded warm-up ping per NIC now absorbs that, which is what lets the loss threshold stay at zero and still mean something. Raising the threshold instead would have hidden genuine single-packet loss.
|
||||
|
||||
**Job logs were being cleared before they were collected**
|
||||
* Script C ran `bgctl reset --yes` while killing processes — before the job directory is copied to USB. Moved to after the archive, so a run's own logs are collected before anything clears them.
|
||||
|
||||
**The fan controller was bound twice**
|
||||
* One of the two MAX31790 controllers received a second `bind` immediately after the first, which could only fail because the driver was already attached.
|
||||
|
||||
## 📦 Downloads
|
||||
|
||||
| File | Contents |
|
||||
|---|---|
|
||||
| `Script_ABC_Blanton_V1.0.8.zip` | The full Tera Term working directory, including `Blanton_Script/` to copy onto the DUT |
|
||||
|
||||
**After copying to the DUT:**
|
||||
|
||||
```bash
|
||||
chmod +x ~/Blanton_Script/*.sh # see below
|
||||
grep -rlU $'\r' ~/Blanton_Script # expect no output
|
||||
```
|
||||
|
||||
`mgmt_ping_monitor.sh` no longer re-execs itself, so `bash mgmt_ping_monitor.sh start` works without the execute bit. `bmc_monitor.sh` and `usb_target.sh` still need it — and neither git (mode 100644) nor a Windows/USB copy carries it. Without the `chmod`, `start` fails with `Permission denied` and the later `cat` finds nothing: **the section ends up empty and nothing reports an error.**
|
||||
|
||||
## ⚠️ Before you run
|
||||
|
||||
* **Declare the switch population.** `SWB_UNIT0` / `SWB_UNIT1` in `config.ttl` say which switch units this DUT has. Traffic and the readiness gate both follow them; with neither set the traffic stage is skipped rather than failing against hardware that is not there.
|
||||
* **Set the fan speed you want.** `FAN_SPEED` in `config.ttl` is applied at the start of every run.
|
||||
* **Management connectivity is in use, not preserved.** Both NICs are up and pinging — drive the run from the serial console.
|
||||
* **If the two management NICs share a subnet, keep `ARP_STRICT=1`.** Otherwise the target's ARP can be answered by either NIC and a reply may arrive on the one that did not send. Check `nic_tx` / `nic_rx` on the RESULT line: they are that interface's own counter delta.
|
||||
* **Two settings persist to `config_db.json`** and survive a reboot: LLDP is left disabled and the 100G uplinks are left configured. Restore them before the DUT moves on.
|
||||
|
||||
## 🔗 Links
|
||||
|
||||
* [ARCHITECTURE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/ARCHITECTURE.md) — test flow, data models, constraints
|
||||
* [CLAUDE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/CLAUDE.md) — commands and the bench gotchas
|
||||
|
||||
**Full changelog:** [V1.0.7...V1.0.8](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/compare/V1.0.7...V1.0.8)
|
||||
@@ -1,7 +1,7 @@
|
||||
; =============================================================================
|
||||
; Script A for Blanton
|
||||
; Version : V1.0.3
|
||||
; Date : 2026-08-21
|
||||
; Version : V1.0.8
|
||||
; Date : 2026-08-25
|
||||
; Author : ETWen
|
||||
; =============================================================================
|
||||
; Version History:
|
||||
@@ -16,6 +16,48 @@
|
||||
; ScriptB/C Add Traffic Test
|
||||
; V1.0.3 2026-08-21 utils/wait_init.ttl Created
|
||||
; ScriptA Add Wait DUT Ready
|
||||
; V1.0.4 2026-08-21 utils/wait_init.ttl Wait on bcmcmd port-up, count (both units > 216)
|
||||
; utils/wait_init.ttl Add WT_UNIT0/WT_UNIT1, skip absent units
|
||||
; ScriptA Add EEPROM info (hpe-eeprom-tlv --bus 4 --addr 0x50)
|
||||
; ScriptA Remove empty Check History block
|
||||
; ScriptA/B Disable lldp + config save before traffic (silent MAC)
|
||||
; ScriptB Add sfputil lpmode off Ethernet513/514
|
||||
; V1.0.5 2026-08-21 config.ttl Add SWB_UNIT0/SWB_UNIT1
|
||||
; utils/wait_init.ttl Follow config.ttl SWB_UNIT0/SWB_UNIT1
|
||||
; ScriptA/B/C Traffic follows SWB_UNIT: -u 0 / -u 1 / skip
|
||||
; V1.0.6 2026-08-24 utils/show_dmesg.ttl Edited dmesg grep i2c & clear event
|
||||
; ScriptA HW Test Session
|
||||
; ScriptC HW Test Session Finish
|
||||
; ScriptA bmc-first-enroll, bmc version/status
|
||||
; ScriptB Stress test DDR, SSD, USB, BMC DDR, BMC USB Stress test
|
||||
; ScriptC BMC USB Stress test result
|
||||
; ScriptA DUT Info
|
||||
; ScriptAC 10G/1G MGMT ping
|
||||
; ScriptA Edited 100G Port Status
|
||||
; ScriptA, utils/setup_pmon.ttl Add TPM
|
||||
; ScriptAC NVME Error Info
|
||||
; ScriptAC Add PCIe Summary
|
||||
; V1.0.7 2026-08-25 config.ttl Add FAN_SPEED
|
||||
; ScriptA Add Fan Ctrl (max31790 rebind + fan-speed-control.sh)
|
||||
; ScriptA Clear /host/hw-eval/current/jobs before the run
|
||||
; ScriptC Copy stress jobs log to USB, timestamped
|
||||
; utils/wait_init.ttl WT_INTERVAL 10 -> 60, fix WT_MIN comments
|
||||
; ScriptC Disable BMC USB journalctl dump
|
||||
; V1.0.8 2026-08-25 Blanton_Script/bmc_monitor.sh Add BMC I2C write/read-back pattern test
|
||||
; ScriptB Re-enable bmc_monitor.sh start
|
||||
; ScriptC Re-enable bmc_monitor.sh stop + cat bmc_poll.log
|
||||
; ScriptC Copy bmc_poll.log + mgmt_ping.log into jobs/ before the USB archive
|
||||
; Blanton_Script/usb_target.sh Created - auto-detect USB device node / mount point
|
||||
; ScriptB USB stress uses the detected node, not a hard-coded /dev/sda1
|
||||
; ScriptB DDR memtester runs continuously (drop the 100-pass limit)
|
||||
; ScriptC Move bgctl reset --yes after the USB archive
|
||||
; Blanton_Script/mgmt_ping_monitor.sh V3.0.0 - both NICs ping simultaneously, accumulating
|
||||
; + ARP warm-up so a stale neighbour entry is not counted as loss
|
||||
; + stop renders last-N per NIC, stats, and both ip -s link show
|
||||
; .gitignore Ignore monitor *.raw / *.pid / *.state
|
||||
; ScriptA MGMT ping baseline 45s -> 30s, clear the log afterwards
|
||||
; ScriptB Show mgmt_ping status after start
|
||||
; ScriptC Show mgmt_ping status before stop, cat the report after
|
||||
; =============================================================================
|
||||
include "config.ttl"
|
||||
|
||||
@@ -50,17 +92,75 @@ gettime timestr
|
||||
sprintf2 cmd 'sudo date -s "%s %s"' datestr timestr
|
||||
sendln cmd
|
||||
|
||||
; ========== Show Build Name ==========
|
||||
;Stress log clear
|
||||
wait prompt_sonic_root
|
||||
sendln "show boot"
|
||||
sendln "rm /host/hw-eval/current/jobs/*"
|
||||
|
||||
; ========== HW Test Session ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "hw-test-session start"
|
||||
wait prompt_sonic_root
|
||||
sendln "hw-test-session log"
|
||||
wait prompt_sonic_root
|
||||
sendln "hw-test-session status"
|
||||
|
||||
; ========== FAN SPEED ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "echo 18-0020 > /sys/bus/i2c/drivers/max31790/unbind"
|
||||
wait prompt_sonic_root
|
||||
sendln "sleep .5"
|
||||
wait prompt_sonic_root
|
||||
sendln "echo 18-0020 > /sys/bus/i2c/drivers/max31790/bind"
|
||||
wait prompt_sonic_root
|
||||
sendln "echo 25-0020 > /sys/bus/i2c/drivers/max31790/unbind"
|
||||
wait prompt_sonic_root
|
||||
sendln "sleep .5"
|
||||
wait prompt_sonic_root
|
||||
sendln "echo 25-0020 > /sys/bus/i2c/drivers/max31790/bind"
|
||||
|
||||
wait prompt_sonic_root
|
||||
sprintf2 fan_ctrl_cmd 'fan-speed-control.sh %d' FAN_SPEED
|
||||
sendln fan_ctrl_cmd
|
||||
|
||||
wait "Set all configured fan channels to"
|
||||
sendln "y"
|
||||
|
||||
|
||||
; ========== Wait DUT Init ==========
|
||||
wait prompt_sonic_root
|
||||
include "utils/wait_init.ttl"
|
||||
|
||||
; ========== BMC version ==========
|
||||
; ========== DUT Info ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "bmc-manager run 'cat /etc/issue'"
|
||||
sendln "show boot"
|
||||
wait prompt_sonic_root
|
||||
sendln "show version"
|
||||
wait prompt_sonic_root
|
||||
sendln "fwutil show status"
|
||||
wait prompt_sonic_root
|
||||
sendln "show system-memory"
|
||||
wait prompt_sonic_root
|
||||
sendln "show service"
|
||||
wait prompt_sonic_root
|
||||
sendln "show platform syseeprom"
|
||||
wait prompt_sonic_root
|
||||
sendln "show platform ssdhealth"
|
||||
wait prompt_sonic_root
|
||||
sendln "qfx5252-tpm-version"
|
||||
wait prompt_sonic_root
|
||||
sendln "/usr/sbin/nvme smart-log -H /dev/nvme0"
|
||||
wait prompt_sonic_root
|
||||
sendln "/usr/sbin/smartctl -x /dev/nvme0"
|
||||
|
||||
; ========== BMC version ==========
|
||||
;wait prompt_sonic_root
|
||||
;sendln "bmc-manager run 'cat /etc/issue'"
|
||||
wait prompt_sonic_root
|
||||
sendln "bmc-first-enroll"
|
||||
wait prompt_sonic_root
|
||||
sendln "bmc-manager version"
|
||||
wait prompt_sonic_root
|
||||
sendln "bmc-manager status"
|
||||
|
||||
|
||||
; ========== Load Shell Script ==========
|
||||
wait prompt_sonic_root
|
||||
@@ -85,14 +185,13 @@ include "utils/show_pcie_error_reg_DDR.ttl"
|
||||
include "utils/show_pcie_error_reg_I210.ttl"
|
||||
include "utils/show_pcie_error_reg_SSD.ttl"
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln "/usr/sbin/ras-mc-ctl --summary"
|
||||
|
||||
|
||||
; ========== Check PCIE tree + PCIE link status + GET PCIE bandwidth Test ==========
|
||||
include "utils/pcie_bus.ttl"
|
||||
|
||||
; ========== Check History ==========
|
||||
;wait prompt_sonic_root
|
||||
;sendln "command"
|
||||
|
||||
; ========== Check Margin ==========
|
||||
if EN_Margin = 1 then
|
||||
include "utils/show_margin_status.ttl"
|
||||
@@ -101,33 +200,100 @@ endif
|
||||
; ========== TAKE DATA ==========
|
||||
include "utils/setup_pmon.ttl"
|
||||
include "utils/show_dmesg.ttl"
|
||||
wait prompt_sonic_root
|
||||
|
||||
;include "utils/xxx.ttl"
|
||||
;wait prompt_sonic_root
|
||||
;flushrecv
|
||||
|
||||
; ========== 10G/1G MGMT Ping Test ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "./Blanton_Script/mgmt_ping_monitor.sh start"
|
||||
wait prompt_sonic_root
|
||||
pause 30
|
||||
sendln "~/Blanton_Script/mgmt_ping_monitor.sh stop"
|
||||
wait prompt_sonic_root
|
||||
sendln "cat ~/Blanton_Script/log/mgmt_ping.log"
|
||||
wait prompt_sonic_root
|
||||
sendln "~/Blanton_Script/mgmt_ping_monitor.sh clear"
|
||||
|
||||
|
||||
; ========== 100G Port Status ==========
|
||||
; Lpmode Disable
|
||||
wait prompt_sonic_root
|
||||
sendln "sudo sfputil lpmode off Ethernet513"
|
||||
wait prompt_sonic_root
|
||||
sendln "sudo sfputil lpmode off Ethernet514"
|
||||
wait prompt_sonic_root
|
||||
sendln "config interface -n asic0 startup Ethernet513"
|
||||
wait prompt_sonic_root
|
||||
sendln "config interface -n asic1 startup Ethernet514"
|
||||
wait prompt_sonic_root
|
||||
sendln "config save -y"
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln "show interfaces status Ethernet513,Ethernet514"
|
||||
|
||||
; ========== First Traffic Test ==========
|
||||
; Silent MAC
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed ps"
|
||||
sendln "config feature state lldp disabled"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed init"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed clear"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed show"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed start"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed stop"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed report"
|
||||
sendln "config save -y"
|
||||
|
||||
if SWB_UNIT0 = 1 && SWB_UNIT1 = 1 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed ps"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed init"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed clear"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed show"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed start"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed stop"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed report"
|
||||
elseif SWB_UNIT0 = 1 && SWB_UNIT1 = 0 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed ps -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed init -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed clear -u 0"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed show -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed start -u 0"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed stop -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed report -u 0"
|
||||
elseif SWB_UNIT0 = 0 && SWB_UNIT1 = 1 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed ps -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed init -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed clear -u 1"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed show -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed start -u 1"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed stop -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed report -u 1"
|
||||
else
|
||||
|
||||
endif
|
||||
|
||||
; ========== Show Power on uptime ==========
|
||||
wait prompt_sonic_root
|
||||
|
||||
@@ -20,55 +20,138 @@ sendln 'chmod +x ~/hammer/tools/amd/mlucas-avx2'
|
||||
wait prompt_sonic_root
|
||||
sendln "~/hammer/tools/amd/mlucas-avx2 -s l -cpu 0:15 > ~/hammer/tools/amd/mlucas_amm_log 2>&1 &"
|
||||
|
||||
; ========== BMC Stress Test ==========
|
||||
; ========== BMC DDR Stress Test ==========
|
||||
;wait prompt_sonic_root
|
||||
;sendln "./Blanton_Script/bmc_monitor_ddr.sh start"
|
||||
;wait prompt_sonic_root
|
||||
;sendln "./Blanton_Script/bmc_monitor.sh start"
|
||||
wait prompt_sonic_root
|
||||
sendln "./Blanton_Script/bmc_monitor_ddr.sh start"
|
||||
sendln "bgctl run bmc-manager run '/usr/bin/memtester 64M 1'"
|
||||
|
||||
; ========== BMC Monitor Test ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "./Blanton_Script/bmc_monitor.sh start"
|
||||
|
||||
|
||||
; ========== DDR MEMORY STRESS TEST ==========
|
||||
;wait prompt_sonic_root
|
||||
;sendln "bgctl run /usr/sbin/memtester 1G 100"
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_mem.py &"
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_mem.py &"
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_mem.py &"
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_mem.py &"
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_mem.py &"
|
||||
sendln "bgctl run /usr/sbin/memtester 1G" ; Continue Execture
|
||||
|
||||
; ========== SSD read/write ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_ssd.py &"
|
||||
sendln "bgctl run qfx5252-stress-ssd 1G --runtime 3600 --passes 5 --write --force --log /host/hw-eval/current/qfx5252-stress-ssd.log"
|
||||
|
||||
; ========== USB read/write ==========
|
||||
; ---- Get USB partition node -> usb_dev ----
|
||||
usb_dev = ''
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_usb.py &"
|
||||
sendln 'D=$(bash ~/Blanton_Script/usb_target.sh -o dev 2>/dev/null); echo "<<D""EV=${D:-NONE}>>"'
|
||||
waitregex '<<DEV=([^>]*)>>'
|
||||
usb_dev = groupmatchstr1
|
||||
|
||||
wait prompt_sonic_root
|
||||
sprintf2 cmd "bgctl run qfx5252-stress-usb %s 256M --runtime 60 --passes 1 --write --log /host/hw-eval/qfx5252-stress-usb.log" usb_dev
|
||||
sendln cmd
|
||||
|
||||
;wait prompt_sonic_root
|
||||
;sendln "bgctl run qfx5252-stress-usb /dev/sda1 256M --runtime 60 --passes 1 --write --log /host/hw-eval/qfx5252-stress-usb.log"
|
||||
|
||||
; ========== SHOW Background job ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "jobs"
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl list"
|
||||
|
||||
; ========== BMC USB Test ==========
|
||||
wait prompt_sonic_root
|
||||
sendln 'echo "NCMIF=$(ls -d /sys/bus/usb/drivers/cdc_ncm/*/net/* 2>/dev/null | head -n 1 | xargs -r basename)"'
|
||||
timeout = 15
|
||||
waitregex 'NCMIF=[A-Za-z0-9_-]+'
|
||||
if result = 0 then
|
||||
messagebox 'cdc_ncm interface not found. Ping test skipped.' 'ERROR'
|
||||
goto skip_ping
|
||||
endif
|
||||
strlen matchstr
|
||||
cut_len = result - 6 ; 'NCMIF=' = 6 chars
|
||||
strcopy matchstr 7 cut_len ncm_if ; -> ncm_if = "eth2" / "eth3"
|
||||
|
||||
sprintf2 cmd 'ip -br link show %s' ncm_if
|
||||
wait prompt_sonic_root
|
||||
sendln cmd
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln 'systemctl stop qfx5252-bmc-usb-net-test.service 2>/dev/null || true'
|
||||
wait prompt_sonic_root
|
||||
sendln 'systemctl reset-failed qfx5252-bmc-usb-net-test.service 2>/dev/null || true'
|
||||
sprintf2 cmd 'systemd-run --unit=qfx5252-bmc-usb-net-test --property=Type=exec /usr/bin/ping -I %s -i 1 -W 2 -w 14400 192.168.200.200' ncm_if
|
||||
wait prompt_sonic_root
|
||||
sendln cmd
|
||||
|
||||
:skip_ping
|
||||
; restore the default (no cap): the traffic init below walks 108 VLANs per
|
||||
; unit and takes far longer than the 15 s set for the NCM probe above.
|
||||
timeout = 0
|
||||
|
||||
; ========== 10G/1G MGMT Ping Test ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "./Blanton_Script/mgmt_ping_monitor.sh start"
|
||||
wait prompt_sonic_root
|
||||
pause 3
|
||||
sendln "./Blanton_Script/mgmt_ping_monitor.sh status"
|
||||
|
||||
|
||||
; ========== 100G Port Status ==========
|
||||
; ========== 100G Port ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "show interfaces status Ethernet513,Ethernet514"
|
||||
|
||||
|
||||
; ========== Traffic START ==========
|
||||
; Silent MAC
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed ps"
|
||||
sendln "config feature state lldp disabled"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed init"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed clear"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed show"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed start"
|
||||
sendln "config save -y"
|
||||
|
||||
if SWB_UNIT0 = 1 && SWB_UNIT1 = 1 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed ps"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed init"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed clear"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed show"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed start"
|
||||
elseif SWB_UNIT0 = 1 && SWB_UNIT1 = 0 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed ps -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed init -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed clear -u 0"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed show -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed start -u 0"
|
||||
elseif SWB_UNIT0 = 0 && SWB_UNIT1 = 1 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed ps -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed init -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed clear -u 1"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed show -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed start -u 1"
|
||||
else
|
||||
|
||||
endif
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -12,6 +12,22 @@ pause 1
|
||||
|
||||
; ========== Kill Process ==========
|
||||
include "utils/kill_all_process.ttl"
|
||||
; bgctrl all stop
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl stop --all"
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl stop --all"
|
||||
; BMC USB Test STOP
|
||||
wait prompt_sonic_root
|
||||
sendln "systemctl stop qfx5252-bmc-usb-net-test.service"
|
||||
; 10G/1G MGMT Test STOP
|
||||
wait prompt_sonic_root
|
||||
sendln "~/Blanton_Script/mgmt_ping_monitor.sh status"
|
||||
wait prompt_sonic_root
|
||||
sendln "~/Blanton_Script/mgmt_ping_monitor.sh stop"
|
||||
wait prompt_sonic_root
|
||||
sendln "cat ~/Blanton_Script/log/mgmt_ping.log"
|
||||
|
||||
; ========== TAKE DATA ==========
|
||||
include "utils/setup_pmon.ttl"
|
||||
include "utils/show_dmesg.ttl"
|
||||
@@ -25,28 +41,66 @@ include "utils/show_pcie_error_reg_DDR.ttl"
|
||||
include "utils/show_pcie_error_reg_I210.ttl"
|
||||
include "utils/show_pcie_error_reg_SSD.ttl"
|
||||
|
||||
; ========== BMC TAKE DATA ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "./Blanton_Script/bmc_monitor_ddr.sh stop"
|
||||
sendln "/usr/sbin/ras-mc-ctl --summary"
|
||||
|
||||
; ========== BMC DDR TAKE DATA ==========
|
||||
;wait prompt_sonic_root
|
||||
;sendln "./Blanton_Script/bmc_monitor_ddr.sh stop"
|
||||
;wait prompt_sonic_root
|
||||
;sendln "./Blanton_Script/bmc_monitor.sh stop"
|
||||
;wait prompt_sonic_root
|
||||
;sendln "cat ./Blanton_Script/log/bmc_poll.log"
|
||||
;wait prompt_sonic_root
|
||||
;sendln "cat ./Blanton_Script/log/bmc_ddr.log"
|
||||
|
||||
; ========== BMC Monitor TAKE DATA ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "./Blanton_Script/bmc_monitor.sh stop"
|
||||
wait prompt_sonic_root
|
||||
sendln "cat ./Blanton_Script/log/bmc_poll.log"
|
||||
|
||||
; ========== BMC USB Test result ==========
|
||||
;wait prompt_sonic_root
|
||||
;sendln "journalctl -u qfx5252-bmc-usb-net-test.service --no-pager"
|
||||
|
||||
; ========== 10G/1G MGMT Test result ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "cat ./Blanton_Script/log/bmc_ddr.log"
|
||||
sendln "ip -s link show eth0"
|
||||
wait prompt_sonic_root
|
||||
sendln "ip -s link show eth1"
|
||||
|
||||
|
||||
; ========== CHECK Stress results ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "cat ~/hammer/tools/amd/mlucas_amm_log"
|
||||
wait prompt_sonic_root
|
||||
sendln "cat ~/hammer/tools/log.stress_ssd"
|
||||
|
||||
; ========== CHECK Traffic counters ==========
|
||||
if SWB_UNIT0 = 1 && SWB_UNIT1 = 1 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed stop"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed report"
|
||||
elseif SWB_UNIT0 = 1 && SWB_UNIT1 = 0 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed stop -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed report -u 0"
|
||||
elseif SWB_UNIT0 = 0 && SWB_UNIT1 = 1 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed stop -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed report -u 1"
|
||||
else
|
||||
|
||||
endif
|
||||
|
||||
; ========== NVME Err Info ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed stop"
|
||||
sendln "/usr/sbin/nvme smart-log -H /dev/nvme0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed report"
|
||||
sendln "/usr/sbin/smartctl -x /dev/nvme0"
|
||||
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln "show reboot-cause"
|
||||
@@ -55,4 +109,23 @@ sendln "show reboot-cause"
|
||||
wait prompt_sonic_root
|
||||
sendln "show uptime"
|
||||
|
||||
; ========== Copy Stress Log to USB ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "cp ./Blanton_Script/log/bmc_poll.log /host/hw-eval/current/jobs/"
|
||||
wait prompt_sonic_root
|
||||
sendln "cp ./Blanton_Script/log/mgmt_ping.log /host/hw-eval/current/jobs/"
|
||||
|
||||
wait prompt_sonic_root
|
||||
getdate ts_date "%Y%m%d"
|
||||
gettime ts_time "%H%M"
|
||||
sprintf2 usb_dst "/mnt/usb/jobs-%s_%s" ts_date ts_time
|
||||
sprintf2 cmd "cp -r /host/hw-eval/current/jobs/ %s" usb_dst
|
||||
sendln cmd
|
||||
|
||||
; ========== HW Test Session ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "hw-test-session finish"
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl reset --yes"
|
||||
|
||||
messagebox 'GOOD JOB! Test Case DONE' 'teraterm'
|
||||
@@ -34,6 +34,10 @@ set -u
|
||||
# Add a line to add a command; comment it out to disable it.
|
||||
COMMANDS=(
|
||||
"free -m"
|
||||
"i2ctransfer -f -y 1 w6@0x41 0x00 0xC0 0x55 0xAA 0x55 0xAA"
|
||||
"i2ctransfer -f -y 1 w2@0x41 0x00 0xC0 r4"
|
||||
"i2ctransfer -f -y 1 w6@0x41 0x00 0xC0 0xAA 0x55 0xAA 0x55"
|
||||
"i2ctransfer -f -y 1 w2@0x41 0x00 0xC0 r4"
|
||||
#"uptime"
|
||||
#"cat /proc/loadavg"
|
||||
#"cat /proc/meminfo"
|
||||
|
||||
@@ -0,0 +1,403 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# mgmt_ping_monitor.sh
|
||||
#
|
||||
# Version : V3.0.0
|
||||
# Author : ETWen
|
||||
# Date : 20260826
|
||||
#
|
||||
# Purpose : Ping from BOTH management NICs at the same time, continuously, and
|
||||
# keep accumulating until stopped:
|
||||
#
|
||||
# 10G (eth0) -> TARGET_10G
|
||||
# 1G (eth1) -> TARGET_1G
|
||||
#
|
||||
# `stop` renders a report into the log: the last TAIL_LINES entries
|
||||
# per NIC, each one's ping statistics and verdict, then
|
||||
# `ip -s link show` for both interfaces.
|
||||
#
|
||||
# Notes : - Both NICs are configured with iproute2 `replace`, which is
|
||||
# idempotent, and both are left up. On a shared subnet the target's
|
||||
# ARP can be answered by either NIC, so ARP_STRICT keeps each one
|
||||
# to its own address.
|
||||
# - ARP WARM-UP: the first packet after a neighbour entry expires is
|
||||
# spent resolving ARP and is counted as loss. On the bench this
|
||||
# produced three FAILs whose only missing packet was icmp_seq=1,
|
||||
# every time, with zero NIC errors or drops. One discarded ping
|
||||
# before the measured run removes that artefact, which is what lets
|
||||
# MAX_LOSS_PCT stay at 0 and still mean something.
|
||||
# - MANAGEMENT CONNECTIVITY IS IN USE while this runs. Drive it from
|
||||
# the serial console.
|
||||
#
|
||||
# Version History
|
||||
# V1.0.0 20260824 Initial Version (ifconfig, one NIC at a time)
|
||||
# V2.0.0 20260824 iproute2; both NICs stay up; per-NIC target;
|
||||
# ip -s link show + TX/RX delta per leg
|
||||
# V3.0.0 20260826 Both NICs ping SIMULTANEOUSLY and continuously instead
|
||||
# of alternating fixed bursts. Add the ARP warm-up.
|
||||
# `stop` renders last-N per NIC + statistics + both
|
||||
# ip -s link outputs into the log.
|
||||
###############################################################################
|
||||
|
||||
set -u
|
||||
|
||||
###############################################################################
|
||||
# User Configurable Section -- normally this is the only part you need to edit
|
||||
###############################################################################
|
||||
|
||||
# --- 10G management port ---
|
||||
IF_10G="eth0"
|
||||
IP_10G="192.168.1.99"
|
||||
TARGET_10G="192.168.1.30" # host this NIC pings
|
||||
PLEN_10G=24 # prefix length, e.g. 24 = /24
|
||||
GW_10G="" # default gateway; empty = do not touch routing
|
||||
METRIC_10G=100 # lower metric wins for off-subnet traffic
|
||||
|
||||
# --- 1G management port ---
|
||||
IF_1G="eth1"
|
||||
IP_1G="192.168.1.101"
|
||||
TARGET_1G="192.168.1.31"
|
||||
PLEN_1G=24
|
||||
GW_1G=""
|
||||
METRIC_1G=200
|
||||
|
||||
# --- ping ---
|
||||
PING_INTERVAL_SEC=1 # seconds between echo requests (ping -i)
|
||||
MAX_LOSS_PCT=0 # loss above this marks the NIC FAIL
|
||||
TAIL_LINES=20 # how many recent entries per NIC the report shows
|
||||
|
||||
# Extra ping flags. -D timestamps every line, -O prints a marker for a request
|
||||
# that got no reply, so a drop is visible in the tail instead of just missing.
|
||||
# Clear this if the platform's ping does not accept them.
|
||||
PING_EXTRA_OPTS="-D -O"
|
||||
|
||||
# One discarded ping per NIC before the measured run, to resolve ARP. Without
|
||||
# it the first packet of the run is lost to neighbour resolution and looks
|
||||
# exactly like a link fault. 0 disables.
|
||||
ARP_WARMUP=1
|
||||
ARP_WARMUP_TIMEOUT=2 # seconds to wait for the warm-up reply
|
||||
|
||||
# Both NICs stay up. If they share a subnet the target's ARP can be answered by
|
||||
# either one, so a reply may arrive on the NIC that did not send. This applies
|
||||
# arp_ignore=1 / arp_announce=2 to both. RAM only -- reverts on reboot.
|
||||
ARP_STRICT=1
|
||||
|
||||
# Prefix for the ip commands; empty because this already runs as root.
|
||||
SUDO=""
|
||||
|
||||
# Seconds to wait after bringing a link up before pinging.
|
||||
LINK_SETTLE_SEC=5
|
||||
|
||||
# Logging
|
||||
LOG_DIR="" # empty = <script directory>/log
|
||||
LOG_NAME="mgmt_ping.log"
|
||||
|
||||
# What `start` does with the log left behind by the previous run:
|
||||
# new = discard it (default -- one run, one log)
|
||||
# archive = rename it to <log>.YYYYmmdd-HHMMSS first
|
||||
# append = keep it
|
||||
START_LOG_MODE="new"
|
||||
|
||||
###############################################################################
|
||||
# Internal
|
||||
###############################################################################
|
||||
SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")"
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
[ -n "${LOG_DIR}" ] || LOG_DIR="${SCRIPT_DIR}/log"
|
||||
|
||||
LOG_FILE="${LOG_DIR}/${LOG_NAME}"
|
||||
BASE="${LOG_DIR}/${LOG_NAME%.log}"
|
||||
PID_FILE="${BASE}.pid"
|
||||
STATE_FILE="${BASE}.state"
|
||||
RAW_10G="${BASE}_${IF_10G}.raw"
|
||||
RAW_1G="${BASE}_${IF_1G}.raw"
|
||||
|
||||
ts() { date '+%Y-%m-%d %H:%M:%S'; }
|
||||
die() { printf '[ERROR] %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
# "<rx_packets> <tx_packets>" for an interface. ip -s link orders the columns
|
||||
# "bytes packets errors ...", so packets is $2, not $1.
|
||||
if_counters() {
|
||||
${SUDO} ip -s link show "$1" 2>/dev/null | awk '
|
||||
/RX:/ { getline; rx = $2 }
|
||||
/TX:/ { getline; tx = $2 }
|
||||
END { printf "%s %s", (rx == "" ? 0 : rx), (tx == "" ? 0 : tx) }'
|
||||
}
|
||||
|
||||
# Bring the NIC up and (re)apply its address and default route.
|
||||
setup_iface() {
|
||||
local iface="$1" ipaddr="$2" plen="$3" gw="$4" metric="$5"
|
||||
${SUDO} ip link set "${iface}" up 2>&1 || return 1
|
||||
${SUDO} ip address replace "${ipaddr}/${plen}" dev "${iface}" 2>&1 || return 1
|
||||
if [ -n "${gw}" ]; then
|
||||
${SUDO} ip route replace default via "${gw}" dev "${iface}" \
|
||||
metric "${metric}" 2>&1 || return 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
apply_arp_strict() {
|
||||
local i
|
||||
[ "${ARP_STRICT}" -eq 1 ] || return 0
|
||||
for i in "${IF_10G}" "${IF_1G}"; do
|
||||
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_ignore=1" 2>/dev/null
|
||||
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_announce=2" 2>/dev/null
|
||||
done
|
||||
}
|
||||
|
||||
# Resolve the neighbour so the measured run does not spend its first packet on
|
||||
# ARP. Result is deliberately discarded.
|
||||
arp_warmup() {
|
||||
local iface="$1" target="$2"
|
||||
[ "${ARP_WARMUP}" -eq 1 ] || return 0
|
||||
ping -I "${iface}" -c 1 -W "${ARP_WARMUP_TIMEOUT}" "${target}" >/dev/null 2>&1
|
||||
return 0
|
||||
}
|
||||
|
||||
# echo "pid_10g pid_1g" and return 0 when both are alive
|
||||
is_running() {
|
||||
local p0 p1
|
||||
[ -f "${PID_FILE}" ] || return 1
|
||||
p0="$(sed -n '1p' "${PID_FILE}" 2>/dev/null)"
|
||||
p1="$(sed -n '2p' "${PID_FILE}" 2>/dev/null)"
|
||||
[[ "${p0}" =~ ^[0-9]+$ ]] || return 1
|
||||
[[ "${p1}" =~ ^[0-9]+$ ]] || return 1
|
||||
kill -0 "${p0}" 2>/dev/null || kill -0 "${p1}" 2>/dev/null || return 1
|
||||
printf '%s %s' "${p0}" "${p1}"
|
||||
return 0
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Report
|
||||
###############################################################################
|
||||
|
||||
# The recent entries for one NIC: replies and, thanks to -O, the requests that
|
||||
# got none.
|
||||
tail_entries() {
|
||||
grep -aE 'bytes from|no answer|Unreachable|Time to live' "$1" 2>/dev/null \
|
||||
| tail -n "${TAIL_LINES}"
|
||||
}
|
||||
|
||||
# The trailing "--- x ping statistics ---" block.
|
||||
stats_block() {
|
||||
sed -n '/ping statistics ---/,$p' "$1" 2>/dev/null
|
||||
}
|
||||
|
||||
# render_leg <label> <iface> <target> <raw> <rx0> <tx0>
|
||||
render_leg() {
|
||||
local label="$1" iface="$2" target="$3" raw="$4" rx0="$5" tx0="$6"
|
||||
local st tx rx loss avg verdict c1 rx1 tx1 drx dtx
|
||||
|
||||
printf -- '----- %s : %s -> %s : last %s entries -----\n' \
|
||||
"${label}" "${iface}" "${target}" "${TAIL_LINES}"
|
||||
tail_entries "${raw}"
|
||||
printf '\n'
|
||||
st="$(stats_block "${raw}")"
|
||||
printf '%s\n' "${st}"
|
||||
|
||||
tx="$(printf '%s' "${st}" | sed -n 's/^\([0-9]\+\) packets transmitted.*/\1/p' | tail -1)"
|
||||
rx="$(printf '%s' "${st}" | sed -n 's/.*[, ]\([0-9]\+\) received.*/\1/p' | tail -1)"
|
||||
loss="$(printf '%s' "${st}" | sed -n 's/.*[, ]\([0-9]\+\)% packet loss.*/\1/p' | tail -1)"
|
||||
avg="$(printf '%s' "${st}" | sed -n 's|.*= [0-9.]*/\([0-9.]*\)/.*|\1|p' | tail -1)"
|
||||
[ -n "${tx}" ] || tx=0
|
||||
[ -n "${rx}" ] || rx=0
|
||||
[ -n "${loss}" ] || loss=100
|
||||
[ -n "${avg}" ] || avg="-"
|
||||
|
||||
c1="$(if_counters "${iface}")"; rx1="${c1% *}"; tx1="${c1#* }"
|
||||
drx=$(( rx1 - rx0 )); dtx=$(( tx1 - tx0 ))
|
||||
|
||||
if [ "${loss}" -le "${MAX_LOSS_PCT}" ] && [ "${tx}" -gt 0 ]; then
|
||||
verdict="PASS"
|
||||
else
|
||||
verdict="FAIL"
|
||||
fi
|
||||
|
||||
# nic_tx/nic_rx are this interface's own counter delta over the whole run.
|
||||
# A pass with nic_tx near zero means the traffic left on the other NIC.
|
||||
printf '[%s] RESULT %-3s %-6s -> %-15s tx=%s rx=%s loss=%s%% rtt_avg=%sms nic_tx=%s nic_rx=%s %s\n\n' \
|
||||
"$(ts)" "${label}" "${iface}" "${target}" "${tx}" "${rx}" "${loss}" "${avg}" \
|
||||
"${dtx}" "${drx}" "${verdict}"
|
||||
}
|
||||
|
||||
write_report() {
|
||||
local rx0_10g tx0_10g rx0_1g tx0_1g started
|
||||
started="$(sed -n '1p' "${STATE_FILE}" 2>/dev/null)"
|
||||
rx0_10g="$(sed -n '2p' "${STATE_FILE}" 2>/dev/null)"; rx0_10g="${rx0_10g:-0}"
|
||||
tx0_10g="$(sed -n '3p' "${STATE_FILE}" 2>/dev/null)"; tx0_10g="${tx0_10g:-0}"
|
||||
rx0_1g="$(sed -n '4p' "${STATE_FILE}" 2>/dev/null)"; rx0_1g="${rx0_1g:-0}"
|
||||
tx0_1g="$(sed -n '5p' "${STATE_FILE}" 2>/dev/null)"; tx0_1g="${tx0_1g:-0}"
|
||||
|
||||
{
|
||||
printf '#############################################################\n'
|
||||
printf '[%s] mgmt ping report\n' "$(ts)"
|
||||
printf ' started : %s\n' "${started:-unknown}"
|
||||
printf ' stopped : %s\n' "$(ts)"
|
||||
printf ' 10G : %s %s/%s -> %s (metric %s)\n' \
|
||||
"${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}"
|
||||
printf ' 1G : %s %s/%s -> %s (metric %s)\n' \
|
||||
"${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}"
|
||||
printf ' ping : -i %s %s, max loss %s%%\n' \
|
||||
"${PING_INTERVAL_SEC}" "${PING_EXTRA_OPTS}" "${MAX_LOSS_PCT}"
|
||||
printf '#############################################################\n\n'
|
||||
|
||||
render_leg "10G" "${IF_10G}" "${TARGET_10G}" "${RAW_10G}" "${rx0_10g}" "${tx0_10g}"
|
||||
render_leg "1G" "${IF_1G}" "${TARGET_1G}" "${RAW_1G}" "${rx0_1g}" "${tx0_1g}"
|
||||
|
||||
printf -- '--- ip -s link show %s ---\n' "${IF_10G}"
|
||||
${SUDO} ip -s link show "${IF_10G}" 2>&1
|
||||
printf '\n'
|
||||
printf -- '--- ip -s link show %s ---\n' "${IF_1G}"
|
||||
${SUDO} ip -s link show "${IF_1G}" 2>&1
|
||||
} >> "${LOG_FILE}"
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Sub-commands
|
||||
###############################################################################
|
||||
prepare_log_on_start() {
|
||||
local stamp
|
||||
case "${START_LOG_MODE}" in
|
||||
append) return 0 ;;
|
||||
archive)
|
||||
if [ -s "${LOG_FILE}" ]; then
|
||||
stamp="$(date '+%Y%m%d-%H%M%S')"
|
||||
mv -f "${LOG_FILE}" "${LOG_FILE}.${stamp}" || die "cannot archive ${LOG_FILE}"
|
||||
printf 'previous log archived: %s.%s\n' "${LOG_FILE}" "${stamp}"
|
||||
fi
|
||||
;;
|
||||
new)
|
||||
# Truncate rather than unlink, so a tail -f already attached keeps
|
||||
# following the new run.
|
||||
[ -f "${LOG_FILE}" ] && { : > "${LOG_FILE}" || die "cannot truncate ${LOG_FILE}"; }
|
||||
;;
|
||||
*) die "invalid START_LOG_MODE: ${START_LOG_MODE}" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
do_start() {
|
||||
local pids p0 p1 c
|
||||
|
||||
pids="$(is_running)" && die "already running (pids=${pids})"
|
||||
|
||||
command -v ip >/dev/null 2>&1 || die "ip (iproute2) not found"
|
||||
command -v ping >/dev/null 2>&1 || die "ping not found"
|
||||
|
||||
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
|
||||
prepare_log_on_start
|
||||
: > "${RAW_10G}"
|
||||
: > "${RAW_1G}"
|
||||
|
||||
setup_iface "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${GW_10G}" "${METRIC_10G}" \
|
||||
|| die "cannot configure ${IF_10G}"
|
||||
setup_iface "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${GW_1G}" "${METRIC_1G}" \
|
||||
|| die "cannot configure ${IF_1G}"
|
||||
apply_arp_strict
|
||||
sleep "${LINK_SETTLE_SEC}"
|
||||
|
||||
# Spend the ARP resolution here, not on the first measured packet.
|
||||
arp_warmup "${IF_10G}" "${TARGET_10G}"
|
||||
arp_warmup "${IF_1G}" "${TARGET_1G}"
|
||||
|
||||
# Counter baseline, taken after the warm-up so its packets are excluded.
|
||||
{ c="$(if_counters "${IF_10G}")"
|
||||
printf '%s\n%s\n%s\n' "$(ts)" "${c% *}" "${c#* }"
|
||||
c="$(if_counters "${IF_1G}")"
|
||||
printf '%s\n%s\n' "${c% *}" "${c#* }"
|
||||
} > "${STATE_FILE}"
|
||||
|
||||
# Both NICs ping at the same time and keep accumulating until stopped.
|
||||
nohup ping -I "${IF_10G}" -i "${PING_INTERVAL_SEC}" ${PING_EXTRA_OPTS} \
|
||||
"${TARGET_10G}" >> "${RAW_10G}" 2>&1 &
|
||||
p0=$!
|
||||
nohup ping -I "${IF_1G}" -i "${PING_INTERVAL_SEC}" ${PING_EXTRA_OPTS} \
|
||||
"${TARGET_1G}" >> "${RAW_1G}" 2>&1 &
|
||||
p1=$!
|
||||
disown "${p0}" 2>/dev/null
|
||||
disown "${p1}" 2>/dev/null
|
||||
printf '%s\n%s\n' "${p0}" "${p1}" > "${PID_FILE}"
|
||||
|
||||
sleep 1
|
||||
kill -0 "${p0}" 2>/dev/null || die "10G ping failed to start, see ${RAW_10G}"
|
||||
kill -0 "${p1}" 2>/dev/null || die "1G ping failed to start, see ${RAW_1G}"
|
||||
printf 'started (10G pid=%s, 1G pid=%s)\nlog: %s\n' "${p0}" "${p1}" "${LOG_FILE}"
|
||||
}
|
||||
|
||||
do_stop() {
|
||||
local pids p0 p1 i
|
||||
|
||||
pids="$(is_running)" || {
|
||||
printf 'not running\n'
|
||||
rm -f "${PID_FILE}"
|
||||
return 0
|
||||
}
|
||||
p0="${pids% *}"; p1="${pids#* }"
|
||||
|
||||
# SIGINT, not SIGTERM: ping prints its statistics block on interrupt, and
|
||||
# that block is what the report parses.
|
||||
kill -INT "${p0}" 2>/dev/null
|
||||
kill -INT "${p1}" 2>/dev/null
|
||||
for (( i = 0; i < 20; i++ )); do
|
||||
kill -0 "${p0}" 2>/dev/null || kill -0 "${p1}" 2>/dev/null || break
|
||||
sleep 0.5
|
||||
done
|
||||
kill -KILL "${p0}" 2>/dev/null
|
||||
kill -KILL "${p1}" 2>/dev/null
|
||||
|
||||
write_report
|
||||
rm -f "${PID_FILE}"
|
||||
printf 'stopped (10G pid=%s, 1G pid=%s)\nreport: %s\n' "${p0}" "${p1}" "${LOG_FILE}"
|
||||
}
|
||||
|
||||
do_status() {
|
||||
local pids
|
||||
if pids="$(is_running)"; then
|
||||
printf 'status : running (10G pid=%s, 1G pid=%s)\n' "${pids% *}" "${pids#* }"
|
||||
else
|
||||
printf 'status : stopped\n'
|
||||
fi
|
||||
printf 'log : %s\n' "${LOG_FILE}"
|
||||
printf '10G : %s %s/%s -> %s\n' "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}"
|
||||
printf '1G : %s %s/%s -> %s\n' "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}"
|
||||
[ -f "${RAW_10G}" ] && printf '10G recv: %s\n' "$(grep -ac 'bytes from' "${RAW_10G}")"
|
||||
[ -f "${RAW_1G}" ] && printf '1G recv: %s\n' "$(grep -ac 'bytes from' "${RAW_1G}")"
|
||||
}
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: ${SCRIPT_NAME} {start|stop|status|summary|clear}
|
||||
|
||||
start Configure both NICs, warm up ARP, then ping from BOTH at the same
|
||||
time and keep accumulating. The previous log is discarded first.
|
||||
stop Stop both pings and render the report into the log
|
||||
status Show pids, configured NICs and replies received so far
|
||||
summary Print just the RESULT lines from the log
|
||||
clear Remove the log and the raw captures (must be stopped first)
|
||||
|
||||
Log file : ${LOG_FILE}
|
||||
|
||||
The report holds, per NIC, the last ${TAIL_LINES} entries and the ping
|
||||
statistics, then "ip -s link show" for both interfaces.
|
||||
|
||||
Both NICs are used at once, so management connectivity is in play -- drive
|
||||
this from the serial console. Edit the User Configurable Section at the top
|
||||
to change interfaces, addresses, targets or the ping options.
|
||||
EOF
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Entry point
|
||||
###############################################################################
|
||||
case "${1:-}" in
|
||||
start) do_start ;;
|
||||
stop) do_stop ;;
|
||||
status) do_status ;;
|
||||
summary) grep -a 'RESULT' "${LOG_FILE}" 2>/dev/null || printf 'no results in %s\n' "${LOG_FILE}" ;;
|
||||
clear)
|
||||
is_running >/dev/null && die "still running, stop it first"
|
||||
rm -f "${LOG_FILE}" "${LOG_FILE}".[0-9]* "${RAW_10G}" "${RAW_1G}" "${STATE_FILE}"
|
||||
printf 'log cleared\n'
|
||||
;;
|
||||
-h|--help|"") usage ;;
|
||||
*) printf '[ERROR] unknown sub-command: %s\n\n' "$1" >&2; usage; exit 1 ;;
|
||||
esac
|
||||
@@ -0,0 +1,203 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# usb_target.sh
|
||||
#
|
||||
# Version : V1.1.0
|
||||
# Author : ETWen
|
||||
# Date : 20260825
|
||||
# Purpose : Auto-detect an inserted USB mass-storage device and report either
|
||||
# its device node, its mount point, or both.
|
||||
#
|
||||
# stdout : requested value(s) only, for $(...) capture
|
||||
# stderr : diagnostic messages
|
||||
#
|
||||
# Usage : usb_target.sh [-o dev|mnt|both|id] [-n] [-r seconds]
|
||||
#
|
||||
# -o dev print partition device node (e.g. /dev/sda1)
|
||||
# -o mnt print mount point (e.g. /mnt/usb) [default]
|
||||
# -o both print "<dev> <mnt>" on one line
|
||||
# -o id print stable by-id path (/dev/disk/by-id/...)
|
||||
# -n detect only, do not mount
|
||||
# -r sec udev enumeration wait, default 15
|
||||
#
|
||||
# Exit : 0 success
|
||||
# 1 no USB mass-storage found
|
||||
# 2 multiple USB disks detected (refuse to guess)
|
||||
# 3 no usable filesystem on the device
|
||||
# 4 mount failed
|
||||
# 5 mounted but not writable
|
||||
#
|
||||
# Version History
|
||||
# V1.0.0 20260825 Initial Version
|
||||
# V1.1.0 20260825 Add -o/-n/-r options; expose device node and by-id path
|
||||
###############################################################################
|
||||
set -u
|
||||
|
||||
MNT_BASE="/mnt/usb"
|
||||
RETRY=15
|
||||
OUTPUT="mnt"
|
||||
DO_MOUNT=1
|
||||
|
||||
log() { echo "[usb] $*" >&2; } # diagnostics go to stderr; stdout stays clean
|
||||
|
||||
while getopts "o:nr:h" opt; do
|
||||
case "$opt" in
|
||||
o) OUTPUT="$OPTARG" ;;
|
||||
n) DO_MOUNT=0 ;;
|
||||
r) RETRY="$OPTARG" ;;
|
||||
h) sed -n '3,30p' "$0" >&2; exit 0 ;;
|
||||
*) log "invalid option"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
shift $((OPTIND - 1))
|
||||
|
||||
case "$OUTPUT" in
|
||||
dev|mnt|both|id) ;;
|
||||
*) log "invalid -o value: $OUTPUT"; exit 1 ;;
|
||||
esac
|
||||
|
||||
# "-o dev" / "-o id" alone does not require mounting.
|
||||
[ "$OUTPUT" = "dev" ] && [ "$DO_MOUNT" = "1" ] && DO_MOUNT=0
|
||||
[ "$OUTPUT" = "id" ] && [ "$DO_MOUNT" = "1" ] && DO_MOUNT=0
|
||||
|
||||
#------------------------------------------------------------------------------
|
||||
# Identify the physical disk(s) backing rootfs / /host so they can be excluded.
|
||||
# Needed because some platforms boot from a USB DOM, which also reports TRAN=usb.
|
||||
#------------------------------------------------------------------------------
|
||||
get_system_disk() {
|
||||
local src
|
||||
for mp in /host / ; do
|
||||
src=$(findmnt -no SOURCE "$mp" 2>/dev/null) || continue
|
||||
lsblk -no PKNAME "$src" 2>/dev/null | head -1
|
||||
done | sort -u
|
||||
}
|
||||
|
||||
#------------------------------------------------------------------------------
|
||||
# List candidate USB disks (whole devices, not partitions).
|
||||
#------------------------------------------------------------------------------
|
||||
find_usb_disks() {
|
||||
local sysdisks; sysdisks=$(get_system_disk)
|
||||
lsblk -dn -o NAME,TYPE,TRAN,RM 2>/dev/null | while read -r name type tran rm; do
|
||||
[ "$type" = "disk" ] || continue
|
||||
case "$name" in loop*|ram*|dm-*|sr*|zram*) continue ;; esac
|
||||
|
||||
# Older util-linux may not expose the TRAN column; fall back to sysfs.
|
||||
if [ "$tran" != "usb" ]; then
|
||||
readlink -f "/sys/block/$name/device" 2>/dev/null | grep -q '/usb[0-9]' || continue
|
||||
[ "$rm" = "1" ] || continue
|
||||
fi
|
||||
|
||||
echo "$sysdisks" | grep -qx "$name" && { log "skip $name (system disk)"; continue; }
|
||||
echo "/dev/$name"
|
||||
done
|
||||
}
|
||||
|
||||
#------------------------------------------------------------------------------
|
||||
# Pick a mountable partition from a disk; fall back to the whole device for
|
||||
# superfloppy layouts (filesystem written directly, no partition table).
|
||||
#------------------------------------------------------------------------------
|
||||
pick_partition() {
|
||||
local disk="$1" p
|
||||
p=$(lsblk -ln -o NAME,TYPE,FSTYPE "$disk" | \
|
||||
awk '$2=="part" && $3!="" {print "/dev/"$1; exit}')
|
||||
[ -n "$p" ] && { echo "$p"; return; }
|
||||
[ -n "$(lsblk -dn -o FSTYPE "$disk")" ] && echo "$disk"
|
||||
}
|
||||
|
||||
#------------------------------------------------------------------------------
|
||||
# Resolve a device node to a stable /dev/disk/by-id path, if one exists.
|
||||
#------------------------------------------------------------------------------
|
||||
resolve_by_id() {
|
||||
local dev="$1" real link
|
||||
real=$(readlink -f "$dev")
|
||||
for link in /dev/disk/by-id/*; do
|
||||
[ -e "$link" ] || continue
|
||||
case "$link" in *-part*|*) ;; esac
|
||||
[ "$(readlink -f "$link")" = "$real" ] && { echo "$link"; return 0; }
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
#------------------------------------------------------------------------------
|
||||
# Main
|
||||
#------------------------------------------------------------------------------
|
||||
disks=""
|
||||
for i in $(seq 1 "$RETRY"); do
|
||||
disks=$(find_usb_disks)
|
||||
[ -n "$disks" ] && break
|
||||
sleep 1
|
||||
done
|
||||
|
||||
[ -z "$disks" ] && { log "no USB mass-storage found"; exit 1; }
|
||||
|
||||
n=$(echo "$disks" | wc -l)
|
||||
if [ "$n" -gt 1 ]; then
|
||||
log "multiple USB disks detected, refuse to guess:"
|
||||
log "$disks"
|
||||
exit 2
|
||||
fi
|
||||
disk="$disks"
|
||||
|
||||
part=$(pick_partition "$disk")
|
||||
[ -z "$part" ] && { log "$disk has no usable filesystem"; exit 3; }
|
||||
|
||||
fstype=$(lsblk -no FSTYPE "$part")
|
||||
log "found $part (disk=$disk fstype=$fstype)"
|
||||
|
||||
# --- device-node only: no mount needed ---------------------------------------
|
||||
if [ "$OUTPUT" = "dev" ]; then
|
||||
echo "$part"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$OUTPUT" = "id" ]; then
|
||||
if byid=$(resolve_by_id "$part"); then
|
||||
echo "$byid"
|
||||
exit 0
|
||||
fi
|
||||
log "no by-id path for $part, falling back to device node"
|
||||
echo "$part"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# --- mount point required ----------------------------------------------------
|
||||
mp=$(lsblk -no MOUNTPOINT "$part" | head -1)
|
||||
|
||||
if [ -z "$mp" ] && [ "$DO_MOUNT" = "0" ]; then
|
||||
log "$part is not mounted and -n was given"
|
||||
exit 4
|
||||
fi
|
||||
|
||||
if [ -z "$mp" ]; then
|
||||
log "mounting $part -> $MNT_BASE"
|
||||
mkdir -p "$MNT_BASE"
|
||||
|
||||
# exfat/ntfs are not always built into the platform kernel.
|
||||
case "$fstype" in
|
||||
exfat) grep -qw exfat /proc/filesystems || modprobe exfat 2>/dev/null ;;
|
||||
ntfs) grep -qw ntfs3 /proc/filesystems || modprobe ntfs3 2>/dev/null ;;
|
||||
esac
|
||||
|
||||
if ! mount -o rw,noatime "$part" "$MNT_BASE" 2>/dev/null; then
|
||||
log "mount failed (fstype=$fstype)"
|
||||
exit 4
|
||||
fi
|
||||
mp="$MNT_BASE"
|
||||
|
||||
# Verify it is actually writable (read-only media, dirty FAT, or full device).
|
||||
if ! touch "$mp/.wtest" 2>/dev/null; then
|
||||
log "mounted read-only or no space left"
|
||||
umount "$mp"
|
||||
exit 5
|
||||
fi
|
||||
rm -f "$mp/.wtest"
|
||||
else
|
||||
log "$part already mounted at $mp"
|
||||
fi
|
||||
|
||||
if [ "$OUTPUT" = "both" ]; then
|
||||
echo "$part $mp"
|
||||
else
|
||||
echo "$mp"
|
||||
fi
|
||||
exit 0
|
||||
@@ -5,8 +5,18 @@ project_name = "Blanton"
|
||||
strTestcase="ENV" ;ENV,EMC,Margin...etc
|
||||
|
||||
EN_Margin = 0 ; 1: Enable Margin Test, 0: Disable Margin Test
|
||||
|
||||
EN_log = 1 ; 1: Enable log, 0: Disable log
|
||||
|
||||
;================================================================
|
||||
; FAN_SPEED
|
||||
;================================================================
|
||||
FAN_SPEED = 30
|
||||
;================================================================
|
||||
; Switch Unit Population
|
||||
;================================================================
|
||||
SWB_UNIT0 = 1 ; 1 = this DUT has switch unit 0, wait for it ; 0 = skip
|
||||
SWB_UNIT1 = 1 ; 1 = this DUT has switch unit 1, wait for it ; 0 = skip
|
||||
|
||||
;================================================================
|
||||
; Prompt Definitions
|
||||
;================================================================
|
||||
|
||||
@@ -15,4 +15,9 @@ sendln "show platform ssdhealth"
|
||||
wait prompt_sonic_root
|
||||
sendln "show platform leak status"
|
||||
wait prompt_sonic_root
|
||||
sendln "show platform leak channels"
|
||||
sendln "show platform leak channels"
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln "tpm-dut-test fru"
|
||||
wait prompt_sonic_root
|
||||
sendln "tpm-dut-test tpm-read"
|
||||
@@ -1,8 +1,11 @@
|
||||
; All Event
|
||||
wait prompt_sonic_root
|
||||
sendln "date"
|
||||
sendln "dmesg -T | grep -Ei 'error|fail(ed|ure)?|fault|critical|panic|timeout'"
|
||||
|
||||
; i2c event
|
||||
wait prompt_sonic_root
|
||||
sendln "dmesg | grep -i error"
|
||||
sendln "dmesg -T | grep -Ei 'error|fail(ed|ure)?|fault|critical|panic|timeout' | grep -i i2c"
|
||||
|
||||
; Clear Event
|
||||
wait prompt_sonic_root
|
||||
sendln "dmesg | grep -i fail"
|
||||
wait prompt_sonic_root
|
||||
sendln "dmesg | grep -i warning"
|
||||
sendln "dmesg -C"
|
||||
|
||||
@@ -1,30 +1,105 @@
|
||||
; =============================================================================
|
||||
; File : utils/wait_init.ttl
|
||||
; Version : V1.0.0
|
||||
; Date : 2026-08-21
|
||||
; Author : ETWen
|
||||
; =============================================================================
|
||||
; Wait until "show platform temperature" is not "Thermal Not detected".
|
||||
; Enter : prompt of the previous command is NOT consumed.
|
||||
; Exit : "show platform temperature" is sent, prompt NOT consumed.
|
||||
;
|
||||
; Version History:
|
||||
; V1.0.0 2026-08-21 Initial Version
|
||||
; =============================================================================
|
||||
wt_interval = 10
|
||||
timeout = 30
|
||||
wait prompt_sonic_root
|
||||
|
||||
:WAIT_THERMAL
|
||||
sendln "show platform temperature"
|
||||
wait "Thermal Not detected" prompt_sonic_root
|
||||
if result = 2 then
|
||||
goto THERMAL_OK
|
||||
endif
|
||||
wait prompt_sonic_root
|
||||
pause wt_interval
|
||||
goto WAIT_THERMAL
|
||||
|
||||
:THERMAL_OK
|
||||
timeout = 0
|
||||
sendln "show platform temperature"
|
||||
; =============================================================================
|
||||
; File : utils/wait_init.ttl
|
||||
; Version : V3.0.0
|
||||
; Date : 2026-08-21
|
||||
; Author : ETWen
|
||||
; =============================================================================
|
||||
; Wait until every switch unit ENABLED below reports at least WT_MIN ports up:
|
||||
; bcmcmd -n <u> -c ps | grep -w up | wc -l
|
||||
;
|
||||
; The DUT is not always fully populated. Which units exist is declared ONCE in
|
||||
; config.ttl, so the traffic blocks in Script A/B/C and this wait agree:
|
||||
; both units -> SWB_UNIT0 = 1 , SWB_UNIT1 = 1
|
||||
; unit 0 only -> SWB_UNIT0 = 1 , SWB_UNIT1 = 0
|
||||
; unit 1 only -> SWB_UNIT0 = 0 , SWB_UNIT1 = 1
|
||||
; no unit -> SWB_UNIT0 = 0 , SWB_UNIT1 = 0 (bypass, returns immediately)
|
||||
;
|
||||
; Enter : prompt of the previous command is NOT consumed.
|
||||
; Exit : a command has been sent, prompt NOT consumed (caller does the wait).
|
||||
;
|
||||
; Why the "PORTS0=" marker: TTL can only test for a string, so reading a count
|
||||
; means capturing it. Wrapping the number in an echo gives waitregex a unique
|
||||
; anchor, and the ECHO of the command cannot false-match -- it reads
|
||||
; "PORTS0=$(bcmcmd ..." and the pattern requires a digit right after the "=".
|
||||
;
|
||||
; Version History:
|
||||
; V1.0.0 2026-08-21 Initial Version (waited on "show platform temperature"
|
||||
; until it stopped reporting "Thermal Not detected")
|
||||
; V2.0.0 2026-08-21 Wait on the bcmcmd port-up count of BOTH switch units
|
||||
; instead of the thermal sensors; proceed only when both
|
||||
; are greater than WT_MIN (216).
|
||||
; V3.0.0 2026-08-21 Add WT_UNIT0 / WT_UNIT1 so a partly populated DUT can
|
||||
; be declared: wait on the enabled units only, and bypass
|
||||
; entirely when neither is set.
|
||||
; V3.1.0 2026-08-21 Take the population from SWB_UNIT0 / SWB_UNIT1 in
|
||||
; config.ttl instead of local copies, so this wait and the
|
||||
; traffic blocks cannot drift apart.
|
||||
; V3.1.1 2026-08-25 WT_INTERVAL 10 -> 60. Fix the WT_MIN comments: the test
|
||||
; has been `< WT_MIN` (at least) since the threshold was
|
||||
; corrected, but the text still said "more than", which is
|
||||
; the off-by-one that once hung Script A.
|
||||
; =============================================================================
|
||||
; ---- which units to wait for: SWB_UNIT0 / SWB_UNIT1, set in config.ttl ------
|
||||
WT_MIN = 216 ; an enabled unit must report AT LEAST this many ports up
|
||||
; (216 = 108 loopback pairs x 2, i.e. every cabled port)
|
||||
WT_INTERVAL = 60 ; seconds between polls
|
||||
; -----------------------------------------------------------------------------
|
||||
|
||||
timeout = 60 ; per-wait cap; bcmcmd ps is not instant
|
||||
wait prompt_sonic_root
|
||||
|
||||
; nothing declared -> bypass
|
||||
if SWB_UNIT0 = 0 then
|
||||
if SWB_UNIT1 = 0 then
|
||||
goto NO_UNITS
|
||||
endif
|
||||
endif
|
||||
|
||||
:WAIT_PORTS
|
||||
wt_ok = 1
|
||||
|
||||
if SWB_UNIT0 = 1 then
|
||||
wt_up0 = 0
|
||||
sendln "echo PORTS0=$(bcmcmd -n 0 -c ps | grep -w up | wc -l)"
|
||||
waitregex "PORTS0=[ ]*([0-9]+)"
|
||||
if result = 1 then
|
||||
str2int wt_up0 groupmatchstr1
|
||||
endif
|
||||
wait prompt_sonic_root
|
||||
if wt_up0 < WT_MIN then
|
||||
wt_ok = 0
|
||||
endif
|
||||
endif
|
||||
|
||||
if SWB_UNIT1 = 1 then
|
||||
wt_up1 = 0
|
||||
sendln "echo PORTS1=$(bcmcmd -n 1 -c ps | grep -w up | wc -l)"
|
||||
waitregex "PORTS1=[ ]*([0-9]+)"
|
||||
if result = 1 then
|
||||
str2int wt_up1 groupmatchstr1
|
||||
endif
|
||||
wait prompt_sonic_root
|
||||
if wt_up1 < WT_MIN then
|
||||
wt_ok = 0
|
||||
endif
|
||||
endif
|
||||
|
||||
if wt_ok = 1 then
|
||||
goto PORTS_OK
|
||||
endif
|
||||
|
||||
pause WT_INTERVAL
|
||||
goto WAIT_PORTS
|
||||
|
||||
; ---- exits ------------------------------------------------------------------
|
||||
; Both leave exactly one prompt unconsumed, so the caller's next
|
||||
; "wait prompt_sonic_root" has something to match.
|
||||
:NO_UNITS
|
||||
sendln "echo wait_init:no-switch-unit-declared,bypassing-port-wait"
|
||||
goto DONE
|
||||
|
||||
:PORTS_OK
|
||||
sendln ""
|
||||
|
||||
:DONE
|
||||
timeout = 0
|
||||
|
||||
Reference in New Issue
Block a user