Compare commits
24
Commits
fd35857d80
..
V1.0.7
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ef7cbdffd4 | ||
|
|
ae8941a844 | ||
|
|
a6a12cdf2c | ||
|
|
e38d24c775 | ||
|
|
9a11c1d5bb | ||
|
|
cbef5865ec | ||
|
|
8714bbb773 | ||
|
|
d8bf1f7a41 | ||
|
|
da39122c43 | ||
|
|
b66ae47a58 | ||
|
|
d065db7d71 | ||
|
|
419b680980 | ||
|
|
4ed283d0ca | ||
|
|
491e807cf8 | ||
|
|
18ce7a1a73 | ||
|
|
d1b5cfea0b | ||
|
|
5e63e3da6b | ||
|
|
855b7a6e30 | ||
|
|
310f897067 | ||
|
|
fedf697977 | ||
|
|
ababb2c3d1 | ||
|
|
a9c3dafc22 | ||
|
|
4088a828c6 | ||
|
|
afb7b1e283 |
@@ -0,0 +1,29 @@
|
||||
# =============================================================================
|
||||
# 換行符政策
|
||||
# -----------------------------------------------------------------------------
|
||||
# 這個 repo 同時被 Windows(Tera Term 端編輯)與 SONiC/Linux(DUT 端執行)使用,
|
||||
# 所以必須明確指定,不能靠各機器的 core.autocrlf。
|
||||
#
|
||||
# 踩過的坑:margin.sh 帶著 CRLF 進 repo,scp 到 DUT 後 bash 直接掛:
|
||||
# -bash: $'\r': command not found
|
||||
# syntax error near unexpected token `$'{\r''
|
||||
# settings/*.conf 同樣危險 —— margin_init 是用 source 讀它們的,未加引號的
|
||||
# 賦值(例如 CB_I2C_CH=6)會變成 "6\r",SWB transport 會拿著壞掉的通道號去打 I2C。
|
||||
# =============================================================================
|
||||
|
||||
# 在 DUT / Linux 上執行或被 source 的,一律 LF
|
||||
*.sh text eol=lf
|
||||
*.conf text eol=lf
|
||||
*.py text eol=lf
|
||||
*.txt text eol=lf
|
||||
*.md text eol=lf
|
||||
|
||||
# Tera Term macro 只在 Windows 上跑,維持 CRLF;標成 -text 讓 git 原樣保存,
|
||||
# 避免對既有檔案產生大量純換行符的假 diff
|
||||
*.ttl -text
|
||||
|
||||
# 二進位,不要做任何轉換
|
||||
*.xlsx binary
|
||||
*.docx binary
|
||||
*.pdf binary
|
||||
*.h5 binary
|
||||
+2
-1
@@ -12,9 +12,10 @@ For_AI/
|
||||
!**/logs/.gitkeep
|
||||
!**/Logs/.gitkeep
|
||||
|
||||
# 打包輸出
|
||||
# 打包輸出:產出不進 git,打包工具本身要進
|
||||
publish/*
|
||||
!publish/.gitkeep
|
||||
!publish/publish.sh
|
||||
|
||||
# Office 暫存檔
|
||||
~$*
|
||||
|
||||
+154
-33
@@ -37,7 +37,12 @@ LTC2980 margin 控制器。TTL 只負責「按順序 `sendln` 並把畫面收進
|
||||
| 電源 margin | LTC2980(= 2 × LTC2977)over PMBus,LINEAR16 編碼 |
|
||||
| 錯誤計數來源 | Linux PCIe AER sysfs(`aer_dev_*`)、EDAC sysfs(`dimm_ce/ue_count`) |
|
||||
| 平台監控 | SONiC `show platform *`(pmon container,含 `leak status` / `leak channels`) |
|
||||
| 壓力測試 | `mlucas-avx2`(AMD CPU)、`~/hammer/tools/stress_{mem,ssd,usb}.py` |
|
||||
| BMC 存取 | host 端 `bmc-manager run "<cmd>"`(不需對 BMC 開 SSH,繞開其 busybox 工具鏈);`bmc-first-enroll` / `bmc-manager version\|status` |
|
||||
| 背景工作管理 | `bgctl run/list/stop --all/reset`(平台工具);長時間 ping 用 `systemd-run --unit=` 掛成 transient unit |
|
||||
| 管理網路 | iproute2(`ip link set` / `ip address replace` / `ip route replace ... metric`)+ `ping -I` |
|
||||
| 測試 session | `hw-test-session start\|log\|status\|finish`;job log 收在 `/host/hw-eval/current/jobs/` |
|
||||
| 風扇控制 | max31790 driver rebind(`/sys/bus/i2c/drivers/max31790/{bind,unbind}`)+ `fan-speed-control.sh <%>` |
|
||||
| 壓力測試 | `mlucas-avx2`(AMD CPU)、`bgctl run` 排程的 `memtester` / `qfx5252-stress-ssd` / `qfx5252-stress-usb` |
|
||||
| 資料面流量 | `bcmcmd`(Broadcom drivshell)VLAN loopback + `tx` burst,switch unit 0 / 1 |
|
||||
| 流量報表 | DUT 上 awk(`blanton_traffic_linespeed report`);離線 `python3 tools/bcm_mibpair_report_V1.1.0.py` |
|
||||
| 版本控制 | Git(NAS + Gitea 私有 remote;**不推 GitHub,含客戶 NDA 資料**) |
|
||||
@@ -106,7 +111,8 @@ Blanton_TTL_Script/
|
||||
│
|
||||
├── CLAUDE.md # 專案記憶 & 給 Claude 的指令
|
||||
├── ARCHITECTURE.md # 本架構文件
|
||||
├── .gitignore # secret/ + For_AI/ + logs 規則
|
||||
├── .gitignore # secret/ + For_AI/ + logs + publish 產出
|
||||
├── .gitattributes # 換行符政策:.sh/.conf 釘 LF、.ttl 保持 CRLF
|
||||
│
|
||||
├── docs/ # 規格與狀態文件(客戶提供 / 專案產出)
|
||||
│ ├── Blantons_FPGA_Registers.md # FPGA 暫存器規格(markdown 正本,查 offset 用)
|
||||
@@ -131,7 +137,9 @@ Blanton_TTL_Script/
|
||||
│
|
||||
├── For_AI/ # 🚫 gitignored — AI 協作素材(波形截圖、草稿)
|
||||
│
|
||||
├── publish/ # 可交付的打包輸出(Blanton_Script_ABC_v<ver>_<date>/ + zip)
|
||||
├── publish/
|
||||
│ ├── publish.sh # ✅ committed — 打包工具(版號讀 Script A 檔頭)
|
||||
│ └── Script_ABC_Blanton_<Ver>/ # 🚫 產出,gitignored(-z 另出同名 .zip)
|
||||
│
|
||||
└── src/
|
||||
└── Script_ABC_Blanton/ # ← Tera Term 的工作目錄(整包給測試員)
|
||||
@@ -144,10 +152,11 @@ Blanton_TTL_Script/
|
||||
├── utils/ # 可 include 的取數片段(TTL,無狀態、只 wait+sendln)
|
||||
│ ├── pcie_bus.ttl # lspci -tvvv / -vv
|
||||
│ ├── setup_pmon.ttl # show platform summary/fan/temperature/psustatus/voltage/
|
||||
│ │ # current/ssdhealth/leak status/leak channels(9 項)
|
||||
│ ├── show_dmesg.ttl # date + dmesg grep error/fail/warning
|
||||
│ │ # current/ssdhealth/leak status/leak channels + TPM(11 項)
|
||||
│ ├── show_dmesg.ttl # dmesg -T 合併 error 正則 + i2c 過濾 + dmesg -C 清空
|
||||
│ ├── show_margin_status.ttl # source margin_status_all.sh(掃 9 顆 LTC2980)
|
||||
│ ├── kill_all_process.ttl # kill $(jobs -p)
|
||||
│ ├── wait_init.ttl # 輪詢 show platform temperature 直到 pmon 起來
|
||||
│ ├── show_pcie_error_reg.ttl # 舊的合併版(已被下列 per-device 版取代)
|
||||
│ ├── show_pcie_error_reg_TH6_SWB0.ttl # EP 01:00.0 + RP 00:01.1
|
||||
│ ├── show_pcie_error_reg_TH6_SWB1.ttl # EP 02:00.0 + RP 00:01.2
|
||||
@@ -167,12 +176,20 @@ Blanton_TTL_Script/
|
||||
├── blanton_pwr_data.sh # SWB VRM + PDB brick 的 Vin/Vout/Iout/Temp 表
|
||||
├── blanton_traffic_linespeed.sh # SWB loopback 線速流量(bcmcmd unit 0/1)
|
||||
│ # init/clear/show/start/stop/ps/report/run
|
||||
├── mgmt_ping_monitor.sh # 10G/1G 管理網路 ping 監控(iproute2)
|
||||
│ # eth0→TARGET_10G、eth1→TARGET_1G,兩張卡都保持 up
|
||||
├── bmc_monitor.sh # ⚠️ 已停用:B/C 內呼叫處已註解,改走 bgctl + bmc-manager
|
||||
├── bmc_monitor_ddr.sh # ⚠️ 已停用,同上
|
||||
│ # ↑ 三支都要 chmod +x 才能跑(見 Key Constraints)
|
||||
└── LTC2980_Margin_Script/ # 電壓 margin 子專案(另有獨立 repo)
|
||||
├── margin.sh # margin_init/status/set/apply_profile/save
|
||||
├── settings/*.conf # 一個 .conf = 一顆 LTC2980(CB + SWB0/1 × CONN13~16)
|
||||
├── profiles/*.conf # 16-ch 批次組合(comboA/B、high3/5、low3/5、normal、off)
|
||||
├── script/*_all.sh # 跨 9 顆 LTC2980 的批次 status / apply / save
|
||||
├── Ref/margin_command_trace.md # 指令追蹤參考(SWB 章節已標過期,見文件內警告)
|
||||
└── logs/ # margin 操作 log
|
||||
# 註:v2.6.0 起不再內含 blanton_cb_i2c.sh / blanton_fpga_pcimem.sh 的複本,
|
||||
# 改由 _swb_find_backend 往上層 Blanton_Script/ 找共用版
|
||||
```
|
||||
|
||||
---
|
||||
@@ -184,15 +201,20 @@ Blanton_TTL_Script/
|
||||
### 1. `config.ttl` — 全域測試參數(TTL 變數)
|
||||
|
||||
```ini
|
||||
strTestcase = "Margin" ; 測項名,進 log 檔名
|
||||
project_name = "Blanton" ; 專案名,進 log 檔名
|
||||
EN_Margin = 1 ; 1=跑 margin 掃描, 0=跳過
|
||||
strTestcase = "ENV" ; 測項名(ENV/EMC/Margin...),進 log 檔名
|
||||
EN_Margin = 0 ; 1=跑 margin 掃描, 0=跳過
|
||||
EN_log = 1 ; 1=logopen 存檔, 0=不存
|
||||
FAN_SPEED = 30 ; 傳給 fan-speed-control.sh 的風扇轉速 (%)
|
||||
SWB_UNIT0 = 1 ; 這台有 switch unit 0(流量與 wait_init 都依此)
|
||||
SWB_UNIT1 = 1 ; 這台有 switch unit 1
|
||||
prompt_login = "sonic login:"
|
||||
prompt_sonic = "admin@sonic:~$"
|
||||
prompt_sonic_root = "root@sonic:~#"
|
||||
```
|
||||
|
||||
> `SWB_UNIT0` / `SWB_UNIT1` 兩個都是 0 時,`wait_init.ttl` 直接 bypass、A/B/C 的流量段整段跳過。
|
||||
|
||||
Log 檔名規則:`<mdir>\Logs\<project_name>_<strTestcase>_<YYYYmmdd-HHMMSS>.log`
|
||||
|
||||
### 2. `settings/<board>.conf` — 一顆 LTC2980(16 channel)
|
||||
@@ -208,6 +230,18 @@ Log 檔名規則:`<mdir>\Logs\<project_name>_<strTestcase>_<YYYYmmdd-HHMMSS>.l
|
||||
|
||||
Channel 定址:`ch / 8` → `CHIPS[]` index,`ch % 8` → LTC2977 PMBus PAGE(0x00~0x07)。
|
||||
|
||||
**SWB 通道與位址對應(2026-08-19 上機驗證,`ababb2c`):**
|
||||
|
||||
| FPGA I2C 通道 | 板 / 接頭 | LTC2977 位址(lo / hi) |
|
||||
|---|---|---|
|
||||
| Ch6 | SWB0 CONN14 / CONN16 | CONN14 `0x5C`,`0x5E` ; CONN16 `0x62`,`0x64` |
|
||||
| Ch8 | SWB0 CONN13 / CONN15 | CONN13 `0x5C`,`0x5E` ; CONN15 `0x62`,`0x64` |
|
||||
| Ch7 | SWB1 CONN14 / CONN16 | CONN14 `0x5C`,`0x5E` ; CONN16 `0x62`,`0x64` |
|
||||
| Ch9 | SWB1 CONN13 / CONN15 | CONN13 `0x5C`,`0x5E` ; CONN15 `0x62`,`0x64` |
|
||||
|
||||
> 同一顆 LTC2980 的兩個位址**相差 2**(7-bit,由規格書 8-bit 配對除以 2 而來),不是 1。
|
||||
> 兩張板共用一條通道,所以同通道上的位址必須互斥 —— 上表兩組(`0x5C/0x5E` 與 `0x62/0x64`)不重疊。
|
||||
|
||||
### 3. `profiles/*.conf` — 16-ch 批次 margin 組合
|
||||
|
||||
```bash
|
||||
@@ -267,16 +301,19 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc
|
||||
|
||||
| 模組 | 進入點 | 功能 |
|
||||
|------|--------|------|
|
||||
| **Script A(Pre-test)** | `1_Blanton_Script_A.ttl` | root 登入 → `mkdir ./logs` → `date -s` 對時 → source **六個** bash 工具 → 清/讀 7 組 PCIe AER → `lspci -tvvv/-vv` → margin 全掃 → PMON 九項 → dmesg 三 grep → `messagebox` 提示接 Script B |
|
||||
| **Script B(Stress)** | `2_Blanton_Script_B.ttl` | `mlucas-avx2 -cpu 0:15` + 5 份 `stress_mem.py` + `stress_ssd.py` + `stress_usb.py` 背景執行(**共 8 個 job**)→ `jobs` 確認 → traffic `ps/init/clear/show/start -u 0` → `while 1` 每輪完整 PMON + margin |
|
||||
| **Script C(Post-test)** | `3_Blanton_Script_C.ttl` | `kill $(jobs -p)` → PMON + dmesg 再抓一次 → 7 組 AER 再抓一次 → `cat` mlucas / stress_ssd log → traffic `stop` + `report -u 0` → 完成 messagebox |
|
||||
| **Script A(Pre-test)** | `1_Blanton_Script_A.ttl` V1.0.7 | root 登入 → `date -s` 對時 → 清 job log → `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**(`wait_init.ttl`)→ DUT 清單(boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe)→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → 100G uplink 設定與狀態 → **一輪流量基線** → `show uptime` |
|
||||
| **Script B(Stress)** | `2_Blanton_Script_B.ttl` | `mlucas-avx2 -cpu 0:15` + `bgctl run` 的 `memtester` / SSD / USB 壓力 + BMC DDR(`bmc-manager run memtester`)→ BMC USB net test(探測 cdc_ncm 介面,`systemd-run` 掛 4 小時 ping)→ 啟動 mgmt ping 監控 → `jobs` / `bgctl list` → traffic(依 `SWB_UNIT`)→ `while 1` 每輪完整 PMON + margin |
|
||||
| **Script C(Post-test)** | `3_Blanton_Script_C.ttl` | `kill $(jobs -p)` + `bgctl stop --all` / `reset` + 停 BMC USB unit 與 mgmt ping → PMON + dmesg → 7 組 AER + `ras-mc-ctl` → BMC USB `journalctl` → `ip -s link show eth0/eth1` → `cat` mlucas log → traffic `stop` + `report` → NVMe 健康 → `show reboot-cause` / `uptime` → **job log 複製到 USB(帶時戳)** → `hw-test-session finish` |
|
||||
| **SWB loopback 流量** | `blanton_traffic_linespeed <sub> [-u 0\|1\|all]` | `init`(VLAN 30..137,成對 `cdN`/`cdN+32`)→ `clear` → `start`(`tx 100 length=512`)→ `stop` → `report`(per-pair TX/RX 交叉比對 PASS/FAIL);另有 `ps`(link/speed)與 `run`(一條龍) |
|
||||
| **FPGA 暫存器存取** | `cb_fpga <fn> <addr> [data]`、`fpga_rescan` | pcimem 打 `resource0` + BAR 相對 offset,32-bit word;BDF 於 source 時自動偵測;`pmc/icb/swb0/swb1_fpga` 走 VSPI 窗口 |
|
||||
| **CB I2C / PMBus** | `cb_i2c_init/scan/read/write`、`cb_pmbus_read/write/linear16` | FPGA F3 OpenCores master,**讀用 Repeated START**(SMBus/PMBus 裝置必需) |
|
||||
| **ICB / PDB 存取** | `vi2c_init/scan/read/write`、`vi2c_scan_all` | CB F3 VI2C proxy,~3.3 ms/reg,取代 VSPI 的 ~18 ms/reg(後者會踩 SMBus 25–35 ms timeout) |
|
||||
| **溫度** | `temp_cb` / `temp_icb` / `temp_all` | TMP75(12-bit 左靠齊 ×0.0625)、TMP432(含 RANGE bit 的 −64 偏移、remote1/2) |
|
||||
| **電源資料** | `pwr_data` / `pwr_data_pdb` / `pwr_pdb_id` | SWB VRM 全軌 Vin/Vout/Iout/Temp 對齊表;MP29816 解析度**執行期從 `MFR_VOUT_SCALE_LOOP` 讀**,不寫死 |
|
||||
| **電壓 Margin** | `margin_init/status/set/apply_profile/save` + `*_all.sh` | 9 顆 LTC2980(CB CONN13、SWB0/1 CONN13~16)批次 high/low margin 與狀態掃描 |
|
||||
| **電壓 Margin** | `margin_init/status/set/apply_profile/save` + `*_all.sh` | 9 顆 LTC2980(CB CONN13、SWB0/1 CONN13~16)批次 high/low margin 與狀態掃描。v2.6.0 起 SWB 路徑真正接上共用 cb_i2c 後端,並有 `margin_swb_info/probe/scan/reset/sem` 除錯包裝 |
|
||||
| **管理網路 ping** | `mgmt_ping_monitor.sh {start\|stop\|status\|fg\|tail\|clear\|summary}` | 一輪 = 10G leg + 1G leg,各自 `ip address/route replace` 後 `ping -I <if>` 打自己的對端;兩張卡都保持 up。每 leg 記 `ip -s link show` 與該卡的 TX/RX packet delta,另出一行可 grep 的 `RESULT` |
|
||||
| **BMC 監控 / 壓力** | `bmc_monitor.sh` / `bmc_monitor_ddr.sh` | ⚠️ **已停用** —— B/C 內呼叫處已註解,改由 `bgctl` + `bmc-manager run memtester` 取代。檔案保留未刪 |
|
||||
| **打包交付** | `publish/publish.sh [-v Ver] [-f] [-z] [-n]` | 版號讀 Script A 檔頭,複製成 `publish/Script_ABC_Blanton_<Ver>/`,排除 `*.log`,可另出 zip |
|
||||
|
||||
---
|
||||
|
||||
@@ -291,14 +328,24 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc
|
||||
Script A ── logopen Logs/Blanton_Margin_<ts>.log ──┐ (之後所有畫面都進這個檔)
|
||||
│ │
|
||||
├─ 登入 admin → sudo -i → root@sonic:~# │
|
||||
├─ date -s "<host 時間>" ← 讓 DUT 時戳可對照 │
|
||||
├─ date -s "<host 時間>" ← 讓 DUT 時戳可對照
|
||||
├─ rm /host/hw-eval/current/jobs/* ← 清掉上一輪的 job log
|
||||
├─ hw-test-session start / log / status
|
||||
├─ 風扇:max31790 unbind→sleep→bind(18-0020 與 25-0020 各一次)
|
||||
│ → fan-speed-control.sh <FAN_SPEED> → 回答 y 確認 │
|
||||
├─ source 6 個 bash 工具(函數進 shell) │
|
||||
│ └ 此時 blanton_fpga_pcimem.sh 印出 [INFO] CB FPGA BDF = ...(自動偵測結果)
|
||||
├─ AER 基線:7 組 EP+RP 的 correctable/nonfatal/fatal
|
||||
├─ lspci -tvvv / -vv ← 拓樸 + link speed/width
|
||||
├─ margin_status_all ← 9 顆 LTC2980 × 16ch 的 Vout 與偏差%
|
||||
├─ show platform × 9 ← fan/temp/psu/voltage/current/ssdhealth/leak status/leak channels
|
||||
└─ dmesg | grep error/fail/warning
|
||||
├─ dmesg -T(合併 error 正則 + i2c 過濾)→ dmesg -C 清空 ← C 只會看到 soak 期間新產生的
|
||||
├─ mgmt ping:start → pause 45 → stop → cat log ← 10G/1G 各一輪
|
||||
├─ 100G uplink:lpmode off → 兩埠 startup → config save → show interfaces status
|
||||
├─ 一輪流量基線:ps → init → clear → show → start → stop → report
|
||||
└─ show uptime
|
||||
(最前面還有 wait_init.ttl:輪詢 show platform temperature 直到不再回報
|
||||
"Thermal Not detected",避免 pmon 還沒起來就抓基線)
|
||||
▼
|
||||
[messagebox "Start Script B"] ← 人工確認後手動跑 B
|
||||
│
|
||||
@@ -306,8 +353,11 @@ Script A ── logopen Logs/Blanton_Margin_<ts>.log ──┐ (之後所有畫
|
||||
Script B ── 同一個 log 檔續寫(logwrite 分隔線)
|
||||
├─ 背景程序:mlucas-avx2(CPU) ×1, stress_mem ×5, stress_ssd, stress_usb → 共 8 個 job
|
||||
│ (stress_hhmd / stress_pcie 已於 2026-08-17 移除,見 Status 表 ScriptB #5/#7 "KC Remove")
|
||||
├─ jobs ← 確認全部起來了
|
||||
├─ traffic: ps → init → clear → (pause 3) → show → start (unit 0,VLAN 30..137 loopback)
|
||||
├─ bgctl run: memtester 1G / qfx5252-stress-ssd / qfx5252-stress-usb
|
||||
├─ BMC: bmc-manager run memtester(DDR)、cdc_ncm 介面探測 + systemd-run 4hr ping(USB)
|
||||
├─ mgmt_ping_monitor.sh start
|
||||
├─ jobs / bgctl list ← 確認全部起來了
|
||||
├─ traffic: ps → init → clear → (pause 15) → show → start (unit 0 與 1,VLAN 30..137 loopback)
|
||||
└─ while 1: setup_pmon(9 項)→ margin_status_all (soak 期間持續取樣)
|
||||
▼
|
||||
[測試員 Ctrl-C / 停止 macro,接著跑 C]
|
||||
@@ -317,8 +367,14 @@ Script C
|
||||
├─ kill $(jobs -p) ← 收掉所有背景壓力
|
||||
├─ show platform × 9 + dmesg ← 收工快照
|
||||
├─ AER 再抓一次 ← 與 Script A 基線相減 = 本輪新增錯誤
|
||||
├─ cat mlucas_amm_log / log.stress_ssd ← 壓力程式自身的 pass/fail
|
||||
└─ traffic: stop → report ← per-pair TX/RX 交叉比對,PASS/FAIL/NA(exit 2 = 有 FAIL/NA)
|
||||
├─ bgctl stop --all / reset、systemctl stop BMC USB unit、mgmt ping stop
|
||||
├─ journalctl BMC USB unit、ip -s link show eth0/eth1、ras-mc-ctl --summary
|
||||
├─ cat mlucas_amm_log ← CPU 壓力程式自身的 pass/fail
|
||||
├─ nvme smart-log / smartctl -x /dev/nvme0
|
||||
├─ traffic: stop → report ← per-pair TX/RX 交叉比對,PASS/FAIL/NA(exit 2 = 有 FAIL/NA)
|
||||
├─ show reboot-cause / show uptime ← soak 期間有沒有意外重開,一眼看得到
|
||||
├─ cp -r /host/hw-eval/current/jobs/ /mnt/usb/jobs-<YYYYmmdd>_<HHMM>
|
||||
└─ hw-test-session finish
|
||||
▼
|
||||
[messagebox "GOOD JOB! Test Case DONE"] → 人工檢查 log
|
||||
```
|
||||
@@ -369,7 +425,28 @@ pwr_data
|
||||
否則它會吃掉 while 迴圈的 stdin,導致**只有第一個 VLAN 被設定**(V0.3.1 修的 bug)。
|
||||
11. **DUT 密碼不得進 git**:`1_Blanton_Script_A.ttl` 目前的 `sendln "YourPaSsWoRd"` 是 placeholder,
|
||||
實際帳密必須放 `secret/config_secret.ttl`(gitignored)並由 `config.ttl` include。
|
||||
12. **客戶 NDA**:`docs/` 內含客戶 FPGA 規格與 schematic 衍生資訊,**本 repo 只推 NAS + Gitea,
|
||||
12. **`.sh` / `.conf` 必須是 LF**:CRLF 的 shell script 在 SONiC 上直接掛
|
||||
(`-bash: $'\r': command not found` + `syntax error near unexpected token`),
|
||||
`settings/*.conf` 則因為是被 `source` 的,未加引號的賦值會變成 `"6\r"`(安靜地壞掉)。
|
||||
`.gitattributes` 已把 `*.sh` `*.conf` `*.py` `*.txt` `*.md` 釘成 LF、`*.ttl` 維持 CRLF。
|
||||
⚠️ 但 `.gitattributes` **不會回頭重寫既有工作區檔案** —— 早於它 clone 出來的目錄,
|
||||
需 `git rm --cached -r . && git reset --hard` 或刪掉目錄重新 checkout 才會生效,
|
||||
而且 `git status` 乾淨**不代表**磁碟上是 LF(`--renormalize` 只在進 index 的路上轉換)。
|
||||
13. **`bmc_monitor*.sh` / `mgmt_ping_monitor.sh` 需要執行權**:它們是本專案「被執行」而非被 `source` 的腳本,
|
||||
而 `do_start` 內部是 `setsid "$SCRIPT_PATH" __daemon`,**即使用 `bash x.sh start` 呼叫
|
||||
也仍需 exec bit**。git 記錄為 `100644`,且 Windows → 隨身碟 → DUT 的路徑無法攜帶權限位元,
|
||||
所以佈署後要先 `chmod +x ~/Blanton_Script/bmc_monitor*.sh`,否則 Script B 的 start 會
|
||||
`Permission denied`、Script C 的 `cat` 撲空(BMC 段全空但不會報錯)。
|
||||
14. **TTL 的 `timeout` 是全域的,設了就要還原**:`timeout = N` 之後**每一個** `wait` 都被限制在 N 秒,
|
||||
逾時的 `wait` 會直接返回而沒吃到 prompt,於是下一個指令在前一個還沒跑完就送出,之後整輪錯開一拍。
|
||||
`wait_init.ttl` 設 60 後還原 0;Script B 的 cdc_ncm 探測設 15,在 `:skip_ping`(兩條路徑的匯流點)還原。
|
||||
加新的 `timeout` 時務必配一個 `timeout = 0`。
|
||||
15. **兩張管理網卡同網段時會 ARP flux**:`mgmt_ping_monitor.sh` 讓 eth0/eth1 同時 up,若兩者在同一個
|
||||
/24(本專案預設 `192.168.1.99` / `.101`),對端 ARP 可能由任一張卡回應,回包落在沒送封包的那張,
|
||||
量到的數字不能歸屬。`ARP_STRICT=1` 會設 `arp_ignore=1` / `arp_announce=2` 擋掉;
|
||||
另外每個 leg 的 `RESULT` 行帶 `nic_tx` / `nic_rx`(該卡自己的 packet delta),
|
||||
**ping 成功但 `nic_tx` 接近 0 就代表封包從另一張卡出去了**。兩張卡在不同網段時可設 0。
|
||||
16. **客戶 NDA**:`docs/` 內含客戶 FPGA 規格與 schematic 衍生資訊,**本 repo 只推 NAS + Gitea,
|
||||
絕不推 GitHub**。
|
||||
|
||||
---
|
||||
@@ -404,6 +481,9 @@ cp secret/config_secret.ttl.example secret/config_secret.ttl
|
||||
|
||||
# ── 2. DUT 端:部署 bash 工具庫 ─────────────────────────────
|
||||
scp -r src/Script_ABC_Blanton/Blanton_Script admin@<DUT_IP>:~/
|
||||
# 佈署後務必補執行權(Windows/隨身碟路徑帶不動 exec bit):
|
||||
ssh admin@<DUT_IP> 'chmod +x ~/Blanton_Script/bmc_monitor*.sh'
|
||||
# 若曾在舊工作區編輯過,先確認沒有 CRLF:grep -rlU $'\r' ~/Blanton_Script
|
||||
ssh admin@<DUT_IP> 'sudo -i'
|
||||
# (現場無網路時改用 USB 隨身碟複製到 ~/Blanton_Script)
|
||||
|
||||
@@ -439,6 +519,19 @@ lspci -Dnn # 對照 ARCHITECTURE.md 的 PCIe 拓樸表
|
||||
8. 檢查 Logs\Blanton_Margin_<ts>.log
|
||||
```
|
||||
|
||||
**打包交付:**
|
||||
|
||||
```bash
|
||||
./publish/publish.sh # 版號讀 Script A 檔頭 → publish/Script_ABC_Blanton_V1.0.3/
|
||||
./publish/publish.sh -z # 另外產出同名 .zip
|
||||
./publish/publish.sh -n # dry run,只印會做什麼
|
||||
./publish/publish.sh -f # 覆蓋既有輸出資料夾
|
||||
```
|
||||
|
||||
版號**不是參數而是從 Script A 的 `; Version : Vx.y.z` 讀出來的**,所以資料夾名不可能跟
|
||||
測試員打開 macro 看到的版本不一致。要出新版就先改 Script A 檔頭再跑。
|
||||
輸出會排除 `*.log`(測試 log 含 DUT 序號,不隨交付外流),但保留空的 `logs/` 供 `logopen` 寫入。
|
||||
|
||||
> 註:本專案非 .NET / Blazor / EC2 架構,`aws-ec2-deploy` 與 `vbox-run-deploy` 兩份部署指南不適用,已跳過。
|
||||
|
||||
---
|
||||
@@ -493,19 +586,22 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值,
|
||||
**目標:** 壓力程序起得來、起不來看得出來,soak 迴圈有時戳、可控週期。
|
||||
|
||||
**包含:**
|
||||
- [ ] 部署 `~/hammer/tools/` 前置檢查:新增 `utils/check_hammer_tools.ttl`
|
||||
(`ls -l ~/hammer/tools/` + `ls ~/hammer/tools/amd/mlucas-avx2`),缺檔就別往下跑
|
||||
- [ ] 前置檢查:`~/hammer/tools/amd/mlucas-avx2` 仍是唯一還依賴 hammer 的項目,缺檔就別往下跑
|
||||
- [ ] 壓力程序 log 加時戳,避免多輪覆蓋:
|
||||
`mlucas_amm_log` → `~/logs/mlucas_amm_log_$(date +%Y%m%d-%H%M%S).log`(Status 表 ScriptB #2 的既定寫法)
|
||||
- [ ] `jobs` 之後加驗證:期望 **8** 個背景 job(1 mlucas + 5 mem + ssd + usb),
|
||||
數量不符時 `messagebox` 提示
|
||||
(`stress_hhmd` / `stress_pcie` 已於 2026-08-17 由 KC 移除,原本的 10 改為 8)
|
||||
- [x] 壓力層改用平台的 `bgctl`(`memtester` / `qfx5252-stress-ssd` / `qfx5252-stress-usb`),
|
||||
取代 `~/hammer/tools/stress_*.py`;Script B 以 `jobs` + `bgctl list` 兩者並列確認
|
||||
- [ ] `bgctl list` 之後加數量驗證,不符時 `messagebox` 提示(目前只印出來給人看)
|
||||
- [ ] soak 迴圈改為「每輪先 `date`、跑完 `pause <間隔>`」,
|
||||
把 `EN_LoopInterval`(預設 600 秒 = 10 min)放進 `config.ttl`,兌現 Status 表的「Get data every 10mins」
|
||||
**← 目前 `while 1` 仍無 `pause`,實際間隔取決於指令執行時間,不是 10 分鐘**
|
||||
- [x] soak 每輪補 `show platform psustatus` / `voltage` / `current`
|
||||
—— 迴圈已改為 `include "utils/setup_pmon.ttl"`,九項全收
|
||||
- [ ] 補 `; ========== CLEAR EVENT ==========`(Status 表 ScriptB #1,Owner: Alan)
|
||||
- [x] BMC 納入 soak:`bmc_monitor.sh`(`free -m` 取樣)與 `bmc_monitor_ddr.sh`(`memtester`)
|
||||
由 Script B 啟動、Script C 停止並把 log 併進 master log
|
||||
- [x] Script A 加開機資訊與就緒等待:`show boot`、BMC `/etc/issue`、`wait_init.ttl`
|
||||
- [x] Script C 加 `show reboot-cause` / `show uptime`,soak 期間意外重開看得見
|
||||
|
||||
**驗收條件:** Script B 啟動後 `jobs` 顯示 8 個 job;連續跑 3 輪,log 內三段各自帶正確時戳且間隔 ≈10 min;
|
||||
故意把 `~/hammer` 改名,Script B 會在前置檢查就停下並提示。
|
||||
@@ -530,14 +626,15 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值,
|
||||
(取代原規劃的 `utils/show_traffic_counters.ttl`)
|
||||
- [x] Script B 接上 `ps → init → clear → show → start`、Script C 接上 `stop → report`
|
||||
- [x] 原始 bcmcmd 指令表留檔:`tools/traffic_loopback_{vlan_setting,start,vlan_remove}.txt`
|
||||
- [ ] **Script A 的 TRAFFIC-SETUP-STAGE 仍是空區塊** —— 目前流量只在 B/C,A 沒有基線
|
||||
- [x] **Script A 的流量基線** —— 原本的空區塊已改成完整一輪
|
||||
(`ps → init → clear → show → start → stop → report`,含 15 秒沉澱)
|
||||
- [ ] `utils/show_interface_detail.ttl`:`show interface transceiver eeprom/dom`、
|
||||
`show interface link-training`(Status 表 ScriptC #4,仍 On-Going)
|
||||
- [ ] `utils/show_tech.ttl`:`show techsupport`(Script C 收工快照,Status 表 ScriptC #4)
|
||||
- [ ] `config.ttl` 加 `EN_Traffic` 旗標,未接 loopback 線的站點可整段跳過
|
||||
(現在 B/C 是無條件執行,沒接線會整批 bcmcmd 失敗)
|
||||
- [ ] 兩支腳本目前都寫死 `-u 0`,另一塊 switch board(unit 1)沒被測到;
|
||||
工具本身預設是 `TL_UNITS="0 1"`,要決定是照工具預設跑兩塊,還是維持只測 unit 0
|
||||
- [x] `-u 0` 已從三支腳本移除,全部沿用工具預設 `TL_UNITS="0 1"`,兩塊 switch board 都測到
|
||||
- [x] 100G uplink 狀態:`show interfaces status Ethernet513,Ethernet514`(A 與 B 各一次)
|
||||
|
||||
**驗收條件:** `EN_Traffic=1` 時,Script C 的 `report` 印出 108 對的 TX/RX 表且全 PASS;
|
||||
拔掉一條 loopback 線重跑,該對顯示 FAIL 或 NA 且 exit code 為 2;`EN_Traffic=0` 時整段不執行且不報錯。
|
||||
@@ -555,9 +652,13 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值,
|
||||
UE > 0 或 fatal/nonfatal > 0 直接標 FAIL
|
||||
- [ ] Margin 表解析:把 9 顆 × 16ch 的 Vout / 偏差% 收成 CSV,標出超出 ±(profile 設定值 + 容差) 的軌
|
||||
- [ ] 溫度 / 風扇 / 電源趨勢:soak 期間逐輪取樣 → CSV(可直接畫圖)
|
||||
- [ ] Stress 判定:`mlucas_amm_log` 與 `log.stress_ssd` 的錯誤關鍵字掃描
|
||||
- [ ] Stress 判定:`mlucas_amm_log` 與 `bgctl` 各工作的 log
|
||||
(`/host/hw-eval/.../qfx5252-stress-{ssd,usb}.log`)的錯誤關鍵字掃描
|
||||
- [ ] Traffic 判定:**直接複用 `tools/bcm_mibpair_report_V1.1.0.py`**(它已能吃 console log 的
|
||||
`show c` 段落並輸出 PASS/FAIL),不要重寫一套解析
|
||||
- [ ] 管理網路判定:抓 `mgmt_ping_monitor.sh` 的 `RESULT` 行(已是單行可 grep 格式),
|
||||
並檢查 `nic_tx`/`nic_rx` 不為 0 —— 否則 PASS 也不能歸屬到那張卡
|
||||
- [ ] NVMe / EDAC:`nvme smart-log`、`smartctl -x`、`ras-mc-ctl --summary` 的前後差值
|
||||
- [ ] 產出 `Logs/<run>_summary.md`:一頁 PASS/FAIL + 每個子系統一行結論
|
||||
|
||||
**驗收條件:** 拿一份既有 log(如 `Blanton_Margin_20260814-164122.log`)跑解析器,
|
||||
@@ -570,9 +671,9 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值,
|
||||
**目標:** 產出可直接交給現場測試員(或客戶)的一包,含操作手冊,不需要看原始碼。
|
||||
|
||||
**包含:**
|
||||
- [ ] `tools/publish.sh`:把 `src/Script_ABC_Blanton/` + 空 `Logs/` + `secret/*.example`
|
||||
打包成 `publish/Blanton_Script_ABC_v<ver>_<YYYYMMDD>/` 與同名 zip(比照
|
||||
既有的 `Blanton_Script_ABC_v1.0_20251229` 命名)
|
||||
- [x] `publish/publish.sh`:把 `src/Script_ABC_Blanton/` 打包成
|
||||
`publish/Script_ABC_Blanton_<Ver>/`(`-z` 另出 zip)。版號讀 Script A 檔頭,
|
||||
排除 `*.log` 但保留空的 `logs/`。工具本身進 git,產出被 gitignore
|
||||
- [ ] `README.md`(現場用):Tera Term 設定、A→B→C 操作順序、常見錯誤(BDF 不符、
|
||||
margin 掃不到、hammer 工具缺檔)與對應處理
|
||||
- [ ] `CHANGELOG.md`:延續三支腳本頂端的 Version History,集中成一份
|
||||
@@ -602,6 +703,26 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值,
|
||||
bash 工具,速度與可靠度都比 serial 好,TTL 只保留開機階段。
|
||||
- **多 DUT 併行**:多台 DUT 同時 soak 時,Tera Term 每台開一個 macro window;
|
||||
可考慮改成 Python(`pyserial`)統一排程與集中收 log。
|
||||
- **與 `LTC2980_Margin_Script` 獨立 repo 的同步機制**:目前 `Blanton_Script/LTC2980_Margin_Script/`
|
||||
是該 repo 的複本(少了 `ARCHITECTURE.md`/`CLAUDE.md`/`Reference/`,多了兩支 `blanton_*` 後端)。
|
||||
待評估改用 git submodule 或明確的 vendored 同步腳本。
|
||||
- ✅ **margin 子專案的重複檔** —— v2.6.0 已移除內含的 `blanton_cb_i2c.sh` /
|
||||
`blanton_fpga_pcimem.sh` 複本,改由 `_swb_find_backend` 往上層找共用版,漂移風險解除。
|
||||
與獨立 repo `LTC2980_Margin_Script` 的同步機制(submodule 或 vendored 腳本)仍待評估。
|
||||
- **`Ref/margin_command_trace.md` 需重新產生**:SWB 章節的 1223 條 `cb_pmbus_*` 指令是依
|
||||
修正前的通道/位址展開的(其中 487 條用到已不存在的 `0x5D`/`0x5F`/`0x63`/`0x65`),
|
||||
文件已加過期警告但內文未動。第 1~4 章不受影響。
|
||||
- **`wait_init.ttl` 沒有逾時出口**:若 pmon 一直起不來,`result` 既非 1 也非 2,迴圈會無限重試,
|
||||
Script A 既不往下走也不報錯。可加最大重試次數 + `messagebox` 提示。
|
||||
- **腳本的執行權**:`mgmt_ping_monitor.sh`(以及保留但已停用的 `bmc_monitor*.sh`)需人工 `chmod +x`。
|
||||
可在 Script B 起它之前補一行(比照既有的 `chmod +x ~/hammer/tools/amd/mlucas-avx2`),或在 publish 時處理。
|
||||
- **`bmc_monitor.sh` / `bmc_monitor_ddr.sh` 已停用**:B/C 內的呼叫都已註解,改由 `bgctl` +
|
||||
`bmc-manager run memtester` 取代。檔案還留在 repo,待確認不再需要後刪除,或在檔頭標註停用。
|
||||
- **`rm /host/hw-eval/current/jobs/*` 清不掉子目錄**:Script A 用的是不帶 `-r` 的 `rm`,但 Script C
|
||||
是 `cp -r` 那個目錄,代表裡面可能有子目錄。清不乾淨的話,上一輪的 job log 會被當成這輪的一起
|
||||
複製到 USB。要確認 `bgctl` 的產出結構,必要時改成 `rm -rf .../jobs/*`。
|
||||
- **風扇確認提示是無限等待**:`wait "Set all configured fan channels to"` 之後才 `sendln "y"`;
|
||||
若該版 `fan-speed-control.sh` 不再詢問(或字串改了),這個 `wait` 會永遠等下去,Script A 停在那裡
|
||||
且不報錯。加個 `timeout` + 分支會比較保險(記得還原 `timeout = 0`)。
|
||||
- **USB 掛載點未檢查**:Script C 直接 `cp -r ... /mnt/usb/`,`/mnt/usb` 沒掛載時只會失敗一行,
|
||||
整輪測試的 job log 就沒被帶出來。可在複製前先 `mountpoint -q /mnt/usb` 判斷。
|
||||
- **Script C 沒有倒出 `mgmt_ping.log`**:C 會 `stop` 管理網路監控並印 `ip -s link show`,
|
||||
但沒有 `cat ~/Blanton_Script/log/mgmt_ping.log`,所以整個 soak 期間累積的 per-leg `RESULT`
|
||||
留在 DUT 上、沒進 master log。Script A 那輪基線有 `cat`。
|
||||
|
||||
@@ -9,6 +9,14 @@ DUT 端的取數能力由一組 **bash 工具庫** 提供(CB FPGA `pcimem` →
|
||||
|
||||
完整架構見 [ARCHITECTURE.md](ARCHITECTURE.md)。
|
||||
|
||||
**目前狀態(2026-08-25)**:Script A 為 **V1.0.7**(A 集中記錄 A/B/C 三支的變更)。涵蓋
|
||||
PCIe AER / DDR EDAC / PMON+TPM / 電壓 margin(9 顆 LTC2980)/ SWB loopback 線速流量 /
|
||||
10G+1G 管理網路 ping / BMC DDR 與 USB / NVMe 健康,壓力層走平台的 `bgctl`。
|
||||
switch unit 數量由 `config.ttl` 的 `SWB_UNIT0` / `SWB_UNIT1` 宣告,風扇轉速由 `FAN_SPEED` 指定。
|
||||
測試 job log 收在 DUT 的 `/host/hw-eval/current/jobs/`,Script C 結束時帶時戳複製到 `/mnt/usb/`。
|
||||
最後一次發布是 **V1.0.5**;SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。
|
||||
交付用 `./publish/publish.sh` 打包。
|
||||
|
||||
## 技術棧
|
||||
|
||||
- **Host**:Tera Term 5.x TTL macro(`.ttl`),Windows
|
||||
@@ -19,6 +27,10 @@ DUT 端的取數能力由一組 **bash 工具庫** 提供(CB FPGA `pcimem` →
|
||||
- **Margin**:LTC2980(= 2 × LTC2977)over PMBus,LINEAR16
|
||||
- **資料面流量**:`bcmcmd`(Broadcom drivshell)SWB loopback,VLAN 30..137 成對 `cdN`/`cdN+32`,
|
||||
`tx 100 length=512`;判定讀 ASIC `MIB_TPOK`/`MIB_RPOK` 做 per-pair 雙向交叉比對
|
||||
- **BMC**:host 端 `bmc-manager run "<cmd>"`,不需對 BMC 開 SSH(繞開其 busybox 工具鏈)
|
||||
- **背景工作**:平台的 `bgctl run/list/stop --all/reset`;長時間 ping 用 `systemd-run --unit=` 掛 transient unit
|
||||
- **管理網路**:iproute2(`ip link set` / `address replace` / `route replace ... metric`)+ `ping -I`
|
||||
- **風扇**:max31790 driver rebind + `fan-speed-control.sh <%>`(轉速讀 `config.ttl` 的 `FAN_SPEED`)
|
||||
- **判定資料源**:PCIe AER sysfs(`aer_dev_*`)、EDAC sysfs(`dimm_{ce,ue}_count`)、SONiC `show platform *`
|
||||
- **無編譯步驟**:TTL 與 bash 都直譯執行
|
||||
|
||||
@@ -50,24 +62,51 @@ pwr_data # SWB VRM 全軌
|
||||
margin_init Blanton_CB_CONN13 && margin_status
|
||||
|
||||
# ── SWB loopback 流量(bcmcmd,unit 0/1)─────────────
|
||||
blanton_traffic_linespeed ps -u 0 # 對照線的 link/speed
|
||||
blanton_traffic_linespeed init -u 0 # VLAN 30..137,cdN <-> cdN+32
|
||||
blanton_traffic_linespeed clear -u 0 # clear c
|
||||
blanton_traffic_linespeed start -u 0 -tx 100 -length 512
|
||||
blanton_traffic_linespeed stop -u 0 # 移除 VLAN 成員(--destroy 連 VLAN 一起砍)
|
||||
blanton_traffic_linespeed report -u 0 # per-pair TX/RX + PASS/FAIL(exit 2 = 有 FAIL/NA)
|
||||
blanton_traffic_linespeed run -u 0 # ps → init → clear → start → report 一條龍
|
||||
blanton_traffic_linespeed init -u 0 -n # -n dry-run:只印要下的 bcmcmd,不碰 DUT
|
||||
# 不帶 -u 就是預設 TL_UNITS="0 1",兩塊 switch board 都做(TTL 三支腳本即如此)
|
||||
blanton_traffic_linespeed ps # 對照線的 link/speed
|
||||
blanton_traffic_linespeed init # VLAN 30..137,cdN <-> cdN+32
|
||||
blanton_traffic_linespeed clear # clear c
|
||||
blanton_traffic_linespeed start -tx 100 -length 512
|
||||
blanton_traffic_linespeed stop # 移除 VLAN 成員(--destroy 連 VLAN 一起砍)
|
||||
blanton_traffic_linespeed report # per-pair TX/RX + PASS/FAIL(exit 2 = 有 FAIL/NA)
|
||||
blanton_traffic_linespeed run # ps → init → clear → start → report 一條龍
|
||||
blanton_traffic_linespeed init -u 0 # 只做單一 unit 時才需要 -u
|
||||
blanton_traffic_linespeed init -n # -n dry-run:只印要下的 bcmcmd,不碰 DUT
|
||||
blanton_traffic_linespeed report -f <log> # 離線重解一份存下來的 console log
|
||||
|
||||
# 離線版報表(開發機上跑,選項更多)
|
||||
python3 tools/bcm_mibpair_report_V1.1.0.py <drivshell.log> -a
|
||||
|
||||
# ── 管理網路 ping 監控(需先 chmod +x!)──────────────
|
||||
chmod +x ~/Blanton_Script/*.sh # 佈署後必做一次
|
||||
~/Blanton_Script/mgmt_ping_monitor.sh fg # 前景試一輪(約 30 秒),確認 IP/對端/網卡
|
||||
~/Blanton_Script/mgmt_ping_monitor.sh start # 背景,log/mgmt_ping.log(start 會先清空)
|
||||
~/Blanton_Script/mgmt_ping_monitor.sh status
|
||||
~/Blanton_Script/mgmt_ping_monitor.sh stop
|
||||
~/Blanton_Script/mgmt_ping_monitor.sh summary # 只看每段的 RESULT 行
|
||||
# ⚠️ tail 子命令是 tail -f,會卡住不返回;TTL 裡要倒 log 請用 cat
|
||||
# ⚠️ bmc_monitor.sh / bmc_monitor_ddr.sh 已停用(改走 bgctl + bmc-manager),檔案仍在
|
||||
|
||||
# ── 背景壓力(平台工具)──────────────────────────────
|
||||
bgctl run /usr/sbin/memtester 1G 100
|
||||
bgctl list
|
||||
bgctl stop --all ; bgctl reset --yes
|
||||
|
||||
# ── 各工具的 help ───────────────────────────────────
|
||||
blanton_fpga_help ; blanton_cb_i2c_help ; vi2c_help
|
||||
temp_sensor_help ; pwr_data_help ; blanton_traffic_linespeed_help
|
||||
```
|
||||
|
||||
**打包交付(在開發機上):**
|
||||
|
||||
```bash
|
||||
./publish/publish.sh # 版號讀 Script A 檔頭 → publish/Script_ABC_Blanton_V1.0.3/
|
||||
./publish/publish.sh -z # 另出 zip
|
||||
./publish/publish.sh -n # dry run
|
||||
```
|
||||
|
||||
要出新版**先改 Script A 檔頭的 `; Version :`**,publish 才會用新版號建資料夾。
|
||||
|
||||
**Host 端跑一輪測試:**
|
||||
|
||||
```
|
||||
@@ -94,6 +133,10 @@ Tera Term 連 COM port (115200-8-N-1)
|
||||
若要回到「每支各自記錄」,B/C 的 History 需要補回去
|
||||
- **Commit**:Conventional Commits,**英文**。scope 用 `ttl` / `fpga` / `i2c` / `vi2c` /
|
||||
`margin` / `pwr` / `temp` / `docs`
|
||||
- **換行符**:由 `.gitattributes` 管理 —— `*.sh` `*.conf` `*.py` `*.txt` `*.md` 釘 **LF**
|
||||
(要在 SONiC 上跑/被 source),`*.ttl` 維持 **CRLF**(Tera Term 只在 Windows 跑)。
|
||||
編輯 `.ttl` 時要保留 CRLF,別讓整檔變成 LF 產生假 diff
|
||||
- **打包**:`./publish/publish.sh`,版號**讀 Script A 檔頭**不是參數 —— 先改檔頭再打包
|
||||
- **Status 表**:`docs/TTL_Script_Blanton_Status_*.xlsx` 是逐項進度的正本(OK / On-Going + Owner),
|
||||
完成一項就同步更新,別讓文件跟腳本脫節
|
||||
|
||||
@@ -130,9 +173,35 @@ Tera Term 連 COM port (115200-8-N-1)
|
||||
- ⚠️ **`stress_hhmd.py` / `stress_pcie.py` 已於 2026-08-17 由 KC 移除**(Status 表 ScriptB #5/#7)。
|
||||
Script B 現在是 **8 個背景 job**(1 mlucas + 5 mem + ssd + usb),不是 10 個。
|
||||
I2C/PCIe 的持續讀取壓力目前沒有替代品。
|
||||
- ⚠️ **Script B 的 traffic 與 soak 迴圈都寫死 `-u 0`、且迴圈沒有 `pause`**:
|
||||
只測到一塊 switch board,取樣間隔也不是 Status 表寫的 10 分鐘(取決於指令執行時間)。
|
||||
改之前先確認現場期待的是哪一種。
|
||||
- ⚠️ **soak 迴圈沒有 `pause`**:取樣間隔不是 Status 表寫的 10 分鐘,而是取決於指令執行時間。
|
||||
(`-u 0` 已於 `491e807` 移除,A/B/C 現在都跑兩個 switch unit。)
|
||||
- 🔒 **`.sh` / `.conf` 一定要 LF**。CRLF 的 shell script 在 SONiC 上直接掛(`$'\r': command not found`);
|
||||
`settings/*.conf` 更陰險 —— 它們是被 `source` 的,`CB_I2C_CH=6` 會變成 `"6\r"`,安靜地壞掉。
|
||||
`.gitattributes` 已擋住 git 這條路徑,但**不會回頭重寫既有工作區檔案**:早於它 clone 的目錄要
|
||||
`git rm --cached -r . && git reset --hard` 才會生效,而且 **`git status` 乾淨不代表磁碟上是 LF**
|
||||
(`--renormalize` 只在進 index 的路上轉換)。驗證要看 byte:`grep -rlU $'\r' <dir>`
|
||||
- ⚠️ **TTL 的 `timeout` 是全域的**:`timeout = N` 之後**每一個** `wait` 都被限制在 N 秒。逾時的 `wait`
|
||||
直接返回而沒吃到 prompt,下一個指令就在前一個還沒跑完時送出,之後整輪錯開一拍。**設了一定要
|
||||
配一個 `timeout = 0` 還原**(`wait_init.ttl` 60→0;Script B 的 cdc_ncm 探測 15→0 在 `:skip_ping`)
|
||||
- ⚠️ **兩張管理網卡同網段會 ARP flux**:`mgmt_ping_monitor.sh` 讓 eth0/eth1 同時 up,同一個 /24 下
|
||||
對端 ARP 可能由任一張卡回應,回包落在沒送封包的那張。`ARP_STRICT=1` 擋掉;判讀時看 `RESULT` 行的
|
||||
`nic_tx`/`nic_rx`(該卡自己的 packet delta)—— **ping 成功但 `nic_tx` 接近 0 = 封包從另一張卡出去的**
|
||||
- ⚠️ **`ip -s link` 的欄位順序是 `bytes packets errors ...`**,要 packet 數取的是**第 2 欄**不是第 1 欄
|
||||
- ⚠️ **`mgmt_ping_monitor.sh` / `bmc_monitor*.sh` 需要 exec bit**,且**不是改用 `bash x.sh start` 就能繞過** ——
|
||||
`do_start` 內部是 `setsid "$SCRIPT_PATH" __daemon`,直接執行該路徑。git 記錄是 `100644`,
|
||||
Windows → 隨身碟 → DUT 也帶不動權限位元,所以佈署後要 `chmod +x ~/Blanton_Script/bmc_monitor*.sh`。
|
||||
沒做的話 start 會 Permission denied、後面的 `cat` 撲空,而且**不會報錯**,只會得到一份全空的 log
|
||||
- ⚠️ **這幾支 monitor 的 `tail` 子命令是 `tail -n 50 -f`,永遠不返回**。TTL 裡絕對不要呼叫它
|
||||
(Script C 曾因此卡死,`18ce7a1` 改成直接 `cat log/*.log`)
|
||||
- ⚠️ **背景 monitor 的 `start` 只等 1 秒就返回**,不是跑完一輪。Script A 要做一輪基線的話,
|
||||
`start` 和 `stop` 中間必須 `pause`(目前是 45 秒,一輪約 30 秒)—— 少了它 log 只會有 START banner
|
||||
- ⚠️ **風扇那段設完會 `wait "Set all configured fan channels to"` 等確認提示再送 `y`**。該提示沒出現
|
||||
(版本改了、字串換了)就會永遠卡住且不報錯 —— 現場看起來像 Script A 當掉
|
||||
- ⚠️ **`show_dmesg.ttl` 結尾會 `dmesg -C` 清空 kernel ring buffer**。這是刻意的(讓 Script C 只看到
|
||||
soak 期間新產生的訊息),但代表事後在 DUT 上 `dmesg` 撈不到舊訊息 —— 內容只存在 master log 裡
|
||||
- ⚠️ **DUT 的 awk 對 `%d` 會夾到 INT32**(2147483647)。線速流量的計數器是 ~10¹² 量級,
|
||||
所以 awk 裡輸出大數一律用 `%.0f`(double 域,精確到 2⁵³)。`blanton_traffic_linespeed.sh`
|
||||
V0.4.1 已修;寫新的 awk 報表時要記得同一件事
|
||||
|
||||
## 資料夾說明
|
||||
|
||||
@@ -141,6 +210,6 @@ Tera Term 連 COM port (115200-8-N-1)
|
||||
- `tools/` — Host / 離線端:`bcm_mibpair_report_V1.1.0.py`(drivshell log → per-pair 報表)、
|
||||
`traffic_loopback_*.txt`(bcmcmd 原始指令表,`blanton_traffic_linespeed.sh` 是其 bash 版,
|
||||
**改 loopback 線路時兩邊要同步**)
|
||||
- `publish/` — 交付用打包輸出(`Blanton_Script_ABC_v<ver>_<date>/` + zip)
|
||||
- `publish/` — `publish.sh`(打包工具,**進 git**)+ `Script_ABC_Blanton_<Ver>/` 產出(**gitignored**)
|
||||
- `secret/` — 🚫 gitignored(除 `README.md` 與 `*.example`):DUT 帳密、per-unit BDF 覆寫、COM 設定
|
||||
- `For_AI/` — 🚫 gitignored:AI 協作素材(波形截圖、草稿筆記)
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
# v1.0.7 — Management-link ping test, and the fans start where you set them
|
||||
|
||||
## ✨ New features
|
||||
|
||||
**10G / 1G management port ping test**
|
||||
* New `mgmt_ping_monitor.sh` exercises both management NICs in one loop: it brings each up with iproute2, waits for the link to settle, then pings that NIC's own target — `eth0` → `192.168.1.30`, `eth1` → `192.168.1.31`.
|
||||
* Runs detached like the other monitors: `start` / `stop` / `status` / `fg` / `summary`. `start` clears the previous log, so one run means one log.
|
||||
* Every leg writes a single greppable line — `RESULT 10G eth0 -> 192.168.1.30 tx=10 rx=10 loss=0% rtt_avg=1.743ms nic_tx=10 nic_rx=10 PASS` — so a whole soak can be read with `summary` instead of scrolling.
|
||||
* `nic_tx` / `nic_rx` are that interface's own counter delta across the burst. If a ping passes but they stay near zero, the traffic left on the *other* NIC and the number does not belong to this one.
|
||||
* Script A runs one round as a baseline; Script B leaves it running through the soak.
|
||||
|
||||
**Fan speed is set at the start of every run**
|
||||
* `FAN_SPEED` in `config.ttl` (default 30) is applied by Script A, so a run no longer inherits whatever the previous test left the fans doing.
|
||||
* Both MAX31790 controllers are rebound first, which clears a controller left in an odd state.
|
||||
|
||||
**Stress load moved onto the platform's own job runner**
|
||||
* CPU, DDR, SSD and USB stress now go through `bgctl`, replacing the `~/hammer/tools/stress_*.py` scripts. `bgctl list` shows what is running; Script C stops everything with `bgctl stop --all`.
|
||||
* BMC DDR stress runs through `bmc-manager`, and a new BMC USB test discovers the `cdc_ncm` interface and pings across it for four hours under `systemd-run`.
|
||||
|
||||
**More of the DUT on record**
|
||||
* Script A now captures the boot image, `show version`, firmware status, system EEPROM, SSD health, TPM version and NVMe SMART data before the test starts.
|
||||
* PCIe AER now sits next to `ras-mc-ctl --summary`, and Script C closes with NVMe health so a disk that degraded during the soak is visible.
|
||||
* `dmesg` capture uses human-readable timestamps and one combined error pattern, then clears the ring buffer — so Script C's dmesg shows only what the soak produced, not everything since boot.
|
||||
* Each run opens with `hw-test-session start` and closes with `finish`; Script C copies the job logs to `/mnt/usb/jobs-<date>_<time>` so they leave the DUT with the run they belong to.
|
||||
|
||||
## 🐛 Bug fixes
|
||||
|
||||
**Script A no longer hangs waiting for ports that are already up**
|
||||
* The readiness gate required *more than* 216 ports up, but a fully cabled unit reports exactly 216 — 108 loopback pairs times two. Script A sat in the poll loop forever on the very state it was waiting for. It now proceeds at 216.
|
||||
* The comments in `wait_init.ttl` still described the old "more than" rule; they now match the code, so the off-by-one cannot be reintroduced by reading the wrong line.
|
||||
|
||||
**The macro no longer runs ahead of the DUT during the soak**
|
||||
* Script B set a 15-second wait cap for the `cdc_ncm` probe and never restored it. In Tera Term that cap is global, so it stayed in force for everything after — including the traffic setup, which walks 108 VLANs per switch unit and takes far longer. A timed-out wait returns without the prompt, leaving every later command issued a step early.
|
||||
|
||||
**Fan controller is no longer bound twice**
|
||||
* One of the two MAX31790 controllers received a second `bind` immediately after the first, which could only fail because the driver was already attached.
|
||||
|
||||
## 📦 Downloads
|
||||
|
||||
| File | Contents |
|
||||
|---|---|
|
||||
| `Script_ABC_Blanton_V1.0.7.zip` | The full Tera Term working directory, including `Blanton_Script/` to copy onto the DUT |
|
||||
|
||||
**After copying to the DUT:**
|
||||
|
||||
```bash
|
||||
chmod +x ~/Blanton_Script/*.sh # required -- see below
|
||||
grep -rlU $'\r' ~/Blanton_Script # expect no output
|
||||
```
|
||||
|
||||
The `chmod +x` is not optional. `mgmt_ping_monitor.sh` is executed rather than sourced and re-execs its own path to daemonise, and neither git (mode 100644) nor a Windows/USB copy carries the execute bit. Without it `start` fails with `Permission denied` and the later `cat` finds nothing — **the section ends up empty and nothing reports an error**.
|
||||
|
||||
## ⚠️ Before you run
|
||||
|
||||
* **Declare the switch population.** `SWB_UNIT0` / `SWB_UNIT1` in `config.ttl` say which switch units this DUT has. Traffic and the readiness gate both follow them: a single unit gets `-u 0` or `-u 1`, and with neither set the traffic stage is skipped instead of failing against hardware that is not there.
|
||||
* **Both management NICs stay up.** If they share a subnet, keep `ARP_STRICT=1` — otherwise the target's ARP can be answered by either NIC and a reply may arrive on the one that did not send.
|
||||
* **Management connectivity is exercised, not preserved.** Drive the run from the serial console.
|
||||
* **Two settings persist to `config_db.json`** and survive a reboot: LLDP is left disabled, and the 100G uplinks are left configured. Restore them before the DUT moves on.
|
||||
|
||||
## 🔗 Links
|
||||
|
||||
* [ARCHITECTURE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/ARCHITECTURE.md) — test flow, data models, constraints
|
||||
* [CLAUDE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/CLAUDE.md) — commands and the bench gotchas
|
||||
|
||||
**Full changelog:** [V1.0.5...V1.0.7](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/compare/V1.0.5...V1.0.7)
|
||||
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# publish.sh -- package src/Script_ABC_Blanton for delivery
|
||||
# =============================================================================
|
||||
# Version History:
|
||||
# V1.0.0 2026-08-21 Initial. Copies src/Script_ABC_Blanton to
|
||||
# publish/Script_ABC_Blanton_<Version>, taking <Version>
|
||||
# from the "; Version : Vx.y.z" header of
|
||||
# 1_Blanton_Script_A.ttl.
|
||||
# =============================================================================
|
||||
#
|
||||
# The version is READ FROM Script A rather than passed in, so the folder name
|
||||
# can never disagree with the header the tester sees when they open the macro.
|
||||
# Bump Script A's header first, then run this.
|
||||
#
|
||||
# Usage:
|
||||
# ./publish/publish.sh # publish at Script A's version
|
||||
# ./publish/publish.sh -v V1.0.4 # override the version
|
||||
# ./publish/publish.sh -f # overwrite an existing output folder
|
||||
# ./publish/publish.sh -z # also produce a .zip next to it
|
||||
# ./publish/publish.sh -n # dry run: print what would happen
|
||||
# =============================================================================
|
||||
set -u
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd -- "${SCRIPT_DIR}/.." && pwd)"
|
||||
SRC_DIR="${REPO_ROOT}/src/Script_ABC_Blanton"
|
||||
OUT_ROOT="${SCRIPT_DIR}"
|
||||
NAME="Script_ABC_Blanton"
|
||||
VER=""; FORCE=0; ZIP=0; DRY=0
|
||||
|
||||
info() { printf '[\033[34mINFO\033[0m] %s\n' "$*"; }
|
||||
ok() { printf '[\033[32m OK \033[0m] %s\n' "$*"; }
|
||||
die() { printf '[\033[31mERR \033[0m] %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
usage() { sed -n '2,26p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0; }
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-v|--version) VER="${2:-}"; shift 2 ;;
|
||||
-f|--force) FORCE=1; shift ;;
|
||||
-z|--zip) ZIP=1; shift ;;
|
||||
-n|--dry-run) DRY=1; shift ;;
|
||||
-h|--help) usage ;;
|
||||
*) die "unknown option '$1' (try -h)" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[ -d "${SRC_DIR}" ] || die "source not found: ${SRC_DIR}"
|
||||
|
||||
# --- version from Script A's header ------------------------------------------
|
||||
# The .ttl files are CRLF, so strip the carriage return or it lands in the
|
||||
# folder name and produces a directory no one can cd into.
|
||||
if [ -z "${VER}" ]; then
|
||||
A_TTL="${SRC_DIR}/1_Blanton_Script_A.ttl"
|
||||
[ -f "${A_TTL}" ] || die "cannot read version: ${A_TTL} missing"
|
||||
VER=$(grep -m1 -E '^;[[:space:]]*Version[[:space:]]*:' "${A_TTL}" \
|
||||
| tr -d '\r' | sed -E 's/.*:[[:space:]]*//; s/[[:space:]]+$//')
|
||||
[ -n "${VER}" ] || die "no '; Version : Vx.y.z' header in $(basename "${A_TTL}")"
|
||||
info "version from $(basename "${A_TTL}") : ${VER}"
|
||||
else
|
||||
info "version (overridden) : ${VER}"
|
||||
fi
|
||||
|
||||
case "${VER}" in
|
||||
V[0-9]*) ;;
|
||||
*) die "version '${VER}' does not look like Vx.y.z" ;;
|
||||
esac
|
||||
|
||||
DEST="${OUT_ROOT}/${NAME}_${VER}"
|
||||
info "source : ${SRC_DIR#${REPO_ROOT}/}"
|
||||
info "destination : ${DEST#${REPO_ROOT}/}"
|
||||
|
||||
if [ -e "${DEST}" ]; then
|
||||
[ "${FORCE}" = "1" ] || die "${DEST##*/} already exists (use -f to overwrite)"
|
||||
[ "${DRY}" = "1" ] || rm -rf -- "${DEST}"
|
||||
info "existing folder removed (-f)"
|
||||
fi
|
||||
|
||||
if [ "${DRY}" = "1" ]; then
|
||||
info "dry run: nothing copied"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# --- copy ---------------------------------------------------------------------
|
||||
# Captured console logs are per-run and often carry DUT serials, so they are
|
||||
# never shipped; the empty logs/ directory itself is kept because Script A's
|
||||
# logopen writes into it.
|
||||
mkdir -p -- "${DEST}" || die "cannot create ${DEST}"
|
||||
if command -v rsync >/dev/null 2>&1; then
|
||||
rsync -a --exclude='*.log' "${SRC_DIR}/" "${DEST}/" || die "rsync failed"
|
||||
else
|
||||
cp -a -- "${SRC_DIR}/." "${DEST}/" || die "cp failed"
|
||||
find "${DEST}" -type f -name '*.log' -delete
|
||||
fi
|
||||
|
||||
# Tera Term's logopen target, in case the source tree never had one
|
||||
mkdir -p -- "${DEST}/logs"
|
||||
|
||||
n_files=$(find "${DEST}" -type f | wc -l)
|
||||
ok "${n_files} files -> ${DEST##*/}"
|
||||
|
||||
if [ "${ZIP}" = "1" ]; then
|
||||
command -v zip >/dev/null 2>&1 || die "zip not installed"
|
||||
( cd -- "${OUT_ROOT}" && rm -f -- "${NAME}_${VER}.zip" \
|
||||
&& zip -qr "${NAME}_${VER}.zip" "${NAME}_${VER}" ) || die "zip failed"
|
||||
ok "$(du -h "${OUT_ROOT}/${NAME}_${VER}.zip" | cut -f1) -> ${NAME}_${VER}.zip"
|
||||
fi
|
||||
@@ -1,7 +1,7 @@
|
||||
; =============================================================================
|
||||
; Script A for Blanton
|
||||
; Version : V1.0.2
|
||||
; Date : 2026-08-17
|
||||
; Version : V1.0.7
|
||||
; Date : 2026-08-25
|
||||
; Author : ETWen
|
||||
; =============================================================================
|
||||
; Version History:
|
||||
@@ -14,7 +14,35 @@
|
||||
; ScriptB Remove stress_hhmd, stress_pcie
|
||||
; Blanton_Script Add blanton_traffic_linespeed.sh
|
||||
; ScriptB/C Add Traffic Test
|
||||
;
|
||||
; V1.0.3 2026-08-21 utils/wait_init.ttl Created
|
||||
; ScriptA Add Wait DUT Ready
|
||||
; V1.0.4 2026-08-21 utils/wait_init.ttl Wait on bcmcmd port-up, count (both units > 216)
|
||||
; utils/wait_init.ttl Add WT_UNIT0/WT_UNIT1, skip absent units
|
||||
; ScriptA Add EEPROM info (hpe-eeprom-tlv --bus 4 --addr 0x50)
|
||||
; ScriptA Remove empty Check History block
|
||||
; ScriptA/B Disable lldp + config save before traffic (silent MAC)
|
||||
; ScriptB Add sfputil lpmode off Ethernet513/514
|
||||
; V1.0.5 2026-08-21 config.ttl Add SWB_UNIT0/SWB_UNIT1
|
||||
; utils/wait_init.ttl Follow config.ttl SWB_UNIT0/SWB_UNIT1
|
||||
; ScriptA/B/C Traffic follows SWB_UNIT: -u 0 / -u 1 / skip
|
||||
; V1.0.6 2026-08-24 utils/show_dmesg.ttl Edited dmesg grep i2c & clear event
|
||||
; ScriptA HW Test Session
|
||||
; ScriptC HW Test Session Finish
|
||||
; ScriptA bmc-first-enroll, bmc version/status
|
||||
; ScriptB Stress test DDR, SSD, USB, BMC DDR, BMC USB Stress test
|
||||
; ScriptC BMC USB Stress test result
|
||||
; ScriptA DUT Info
|
||||
; ScriptAC 10G/1G MGMT ping
|
||||
; ScriptA Edited 100G Port Status
|
||||
; ScriptA, utils/setup_pmon.ttl Add TPM
|
||||
; ScriptAC NVME Error Info
|
||||
; ScriptAC Add PCIe Summary
|
||||
; V1.0.7 2026-08-25 config.ttl Add FAN_SPEED
|
||||
; ScriptA Add Fan Ctrl (max31790 rebind + fan-speed-control.sh)
|
||||
; ScriptA Clear /host/hw-eval/current/jobs before the run
|
||||
; ScriptC Copy stress jobs log to USB, timestamped
|
||||
; utils/wait_init.ttl WT_INTERVAL 10 -> 60, fix WT_MIN comments
|
||||
; ScriptC Disable BMC USB journalctl dump
|
||||
; =============================================================================
|
||||
include "config.ttl"
|
||||
|
||||
@@ -49,6 +77,76 @@ gettime timestr
|
||||
sprintf2 cmd 'sudo date -s "%s %s"' datestr timestr
|
||||
sendln cmd
|
||||
|
||||
;Stress log clear
|
||||
wait prompt_sonic_root
|
||||
sendln "rm /host/hw-eval/current/jobs/*"
|
||||
|
||||
; ========== HW Test Session ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "hw-test-session start"
|
||||
wait prompt_sonic_root
|
||||
sendln "hw-test-session log"
|
||||
wait prompt_sonic_root
|
||||
sendln "hw-test-session status"
|
||||
|
||||
; ========== FAN SPEED ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "echo 18-0020 > /sys/bus/i2c/drivers/max31790/unbind"
|
||||
wait prompt_sonic_root
|
||||
sendln "sleep .5"
|
||||
wait prompt_sonic_root
|
||||
sendln "echo 18-0020 > /sys/bus/i2c/drivers/max31790/bind"
|
||||
wait prompt_sonic_root
|
||||
sendln "echo 25-0020 > /sys/bus/i2c/drivers/max31790/unbind"
|
||||
wait prompt_sonic_root
|
||||
sendln "sleep .5"
|
||||
wait prompt_sonic_root
|
||||
sendln "echo 25-0020 > /sys/bus/i2c/drivers/max31790/bind"
|
||||
|
||||
wait prompt_sonic_root
|
||||
sprintf2 fan_ctrl_cmd 'fan-speed-control.sh %d' FAN_SPEED
|
||||
sendln fan_ctrl_cmd
|
||||
|
||||
wait "Set all configured fan channels to"
|
||||
sendln "y"
|
||||
|
||||
|
||||
; ========== Wait DUT Init ==========
|
||||
include "utils/wait_init.ttl"
|
||||
|
||||
; ========== DUT Info ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "show boot"
|
||||
wait prompt_sonic_root
|
||||
sendln "show version"
|
||||
wait prompt_sonic_root
|
||||
sendln "fwutil show status"
|
||||
wait prompt_sonic_root
|
||||
sendln "show system-memory"
|
||||
wait prompt_sonic_root
|
||||
sendln "show service"
|
||||
wait prompt_sonic_root
|
||||
sendln "show platform syseeprom"
|
||||
wait prompt_sonic_root
|
||||
sendln "show platform ssdhealth"
|
||||
wait prompt_sonic_root
|
||||
sendln "qfx5252-tpm-version"
|
||||
wait prompt_sonic_root
|
||||
sendln "/usr/sbin/nvme smart-log -H /dev/nvme0"
|
||||
wait prompt_sonic_root
|
||||
sendln "/usr/sbin/smartctl -x /dev/nvme0"
|
||||
|
||||
; ========== BMC version ==========
|
||||
;wait prompt_sonic_root
|
||||
;sendln "bmc-manager run 'cat /etc/issue'"
|
||||
wait prompt_sonic_root
|
||||
sendln "bmc-first-enroll"
|
||||
wait prompt_sonic_root
|
||||
sendln "bmc-manager version"
|
||||
wait prompt_sonic_root
|
||||
sendln "bmc-manager status"
|
||||
|
||||
|
||||
; ========== Load Shell Script ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "source ~/Blanton_Script/blanton_fpga_pcimem.sh"
|
||||
@@ -72,14 +170,13 @@ include "utils/show_pcie_error_reg_DDR.ttl"
|
||||
include "utils/show_pcie_error_reg_I210.ttl"
|
||||
include "utils/show_pcie_error_reg_SSD.ttl"
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln "/usr/sbin/ras-mc-ctl --summary"
|
||||
|
||||
|
||||
; ========== Check PCIE tree + PCIE link status + GET PCIE bandwidth Test ==========
|
||||
include "utils/pcie_bus.ttl"
|
||||
|
||||
; ========== Check History ==========
|
||||
;wait prompt_sonic_root
|
||||
;sendln "command"
|
||||
|
||||
; ========== Check Margin ==========
|
||||
if EN_Margin = 1 then
|
||||
include "utils/show_margin_status.ttl"
|
||||
@@ -88,16 +185,103 @@ endif
|
||||
; ========== TAKE DATA ==========
|
||||
include "utils/setup_pmon.ttl"
|
||||
include "utils/show_dmesg.ttl"
|
||||
wait prompt_sonic_root
|
||||
|
||||
;include "utils/xxx.ttl"
|
||||
;wait prompt_sonic_root
|
||||
;flushrecv
|
||||
|
||||
; ========== TRAFFIC-SETUP-STAGE ==========
|
||||
;sendln "command"
|
||||
; ========== 10G/1G MGMT Ping Test ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "./Blanton_Script/mgmt_ping_monitor.sh start"
|
||||
wait prompt_sonic_root
|
||||
pause 45
|
||||
sendln "~/Blanton_Script/mgmt_ping_monitor.sh stop"
|
||||
wait prompt_sonic_root
|
||||
sendln "cat ~/Blanton_Script/log/mgmt_ping.log"
|
||||
|
||||
|
||||
; ========== 100G Port Status ==========
|
||||
; Lpmode Disable
|
||||
wait prompt_sonic_root
|
||||
sendln "sudo sfputil lpmode off Ethernet513"
|
||||
wait prompt_sonic_root
|
||||
sendln "sudo sfputil lpmode off Ethernet514"
|
||||
wait prompt_sonic_root
|
||||
sendln "config interface -n asic0 startup Ethernet513"
|
||||
wait prompt_sonic_root
|
||||
sendln "config interface -n asic1 startup Ethernet514"
|
||||
wait prompt_sonic_root
|
||||
sendln "config save -y"
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln "show interfaces status Ethernet513,Ethernet514"
|
||||
|
||||
; ========== First Traffic Test ==========
|
||||
; Silent MAC
|
||||
wait prompt_sonic_root
|
||||
sendln "config feature state lldp disabled"
|
||||
wait prompt_sonic_root
|
||||
sendln "config save -y"
|
||||
|
||||
if SWB_UNIT0 = 1 && SWB_UNIT1 = 1 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed ps"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed init"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed clear"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed show"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed start"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed stop"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed report"
|
||||
elseif SWB_UNIT0 = 1 && SWB_UNIT1 = 0 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed ps -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed init -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed clear -u 0"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed show -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed start -u 0"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed stop -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed report -u 0"
|
||||
elseif SWB_UNIT0 = 0 && SWB_UNIT1 = 1 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed ps -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed init -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed clear -u 1"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed show -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed start -u 1"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed stop -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed report -u 1"
|
||||
else
|
||||
|
||||
endif
|
||||
|
||||
; ========== Show Power on uptime ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "show uptime"
|
||||
|
||||
|
||||
sendln ""
|
||||
wait prompt_sonic_root
|
||||
messagebox 'Start Script B' 'Script A DONE'
|
||||
@@ -20,44 +20,118 @@ sendln 'chmod +x ~/hammer/tools/amd/mlucas-avx2'
|
||||
wait prompt_sonic_root
|
||||
sendln "~/hammer/tools/amd/mlucas-avx2 -s l -cpu 0:15 > ~/hammer/tools/amd/mlucas_amm_log 2>&1 &"
|
||||
|
||||
; ========== BMC DDR Stress Test ==========
|
||||
;wait prompt_sonic_root
|
||||
;sendln "./Blanton_Script/bmc_monitor_ddr.sh start"
|
||||
;wait prompt_sonic_root
|
||||
;sendln "./Blanton_Script/bmc_monitor.sh start"
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl run bmc-manager run '/usr/bin/memtester 64M 1'"
|
||||
|
||||
; ========== DDR MEMORY STRESS TEST ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_mem.py &"
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_mem.py &"
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_mem.py &"
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_mem.py &"
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_mem.py &"
|
||||
sendln "bgctl run /usr/sbin/memtester 1G 100"
|
||||
|
||||
; ========== SSD read/write ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_ssd.py &"
|
||||
sendln "bgctl run qfx5252-stress-ssd 1G --runtime 3600 --passes 5 --write --force --log /host/hw-eval/current/qfx5252-stress-ssd.log"
|
||||
|
||||
; ========== USB read/write ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_usb.py &"
|
||||
sendln "bgctl run qfx5252-stress-usb /dev/sda1 256M --runtime 60 --passes 1 --write --log /host/hw-eval/qfx5252-stress-usb.log"
|
||||
|
||||
; ========== SHOW Background job ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "jobs"
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl list"
|
||||
|
||||
; ========== BMC USB Test ==========
|
||||
wait prompt_sonic_root
|
||||
sendln 'echo "NCMIF=$(ls -d /sys/bus/usb/drivers/cdc_ncm/*/net/* 2>/dev/null | head -n 1 | xargs -r basename)"'
|
||||
timeout = 15
|
||||
waitregex 'NCMIF=[A-Za-z0-9_-]+'
|
||||
if result = 0 then
|
||||
messagebox 'cdc_ncm interface not found. Ping test skipped.' 'ERROR'
|
||||
goto skip_ping
|
||||
endif
|
||||
strlen matchstr
|
||||
cut_len = result - 6 ; 'NCMIF=' = 6 chars
|
||||
strcopy matchstr 7 cut_len ncm_if ; -> ncm_if = "eth2" / "eth3"
|
||||
|
||||
sprintf2 cmd 'ip -br link show %s' ncm_if
|
||||
wait prompt_sonic_root
|
||||
sendln cmd
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln 'systemctl stop qfx5252-bmc-usb-net-test.service 2>/dev/null || true'
|
||||
wait prompt_sonic_root
|
||||
sendln 'systemctl reset-failed qfx5252-bmc-usb-net-test.service 2>/dev/null || true'
|
||||
sprintf2 cmd 'systemd-run --unit=qfx5252-bmc-usb-net-test --property=Type=exec /usr/bin/ping -I %s -i 1 -W 2 -w 14400 192.168.200.200' ncm_if
|
||||
wait prompt_sonic_root
|
||||
sendln cmd
|
||||
|
||||
:skip_ping
|
||||
; restore the default (no cap): the traffic init below walks 108 VLANs per
|
||||
; unit and takes far longer than the 15 s set for the NCM probe above.
|
||||
timeout = 0
|
||||
|
||||
; ========== 10G/1G MGMT Ping Test ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "./Blanton_Script/mgmt_ping_monitor.sh start"
|
||||
|
||||
|
||||
; ========== 100G Port ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "show interfaces status Ethernet513,Ethernet514"
|
||||
|
||||
; ========== Take one round of eye measurement on uplink ports ==========
|
||||
|
||||
; ========== Traffic START ==========
|
||||
; Silent MAC
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed ps -u 0"
|
||||
sendln "config feature state lldp disabled"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed init -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed clear -u 0"
|
||||
wait prompt_sonic_root
|
||||
pause 3
|
||||
sendln "blanton_traffic_linespeed show -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed start -u 0"
|
||||
sendln "config save -y"
|
||||
|
||||
if SWB_UNIT0 = 1 && SWB_UNIT1 = 1 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed ps"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed init"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed clear"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed show"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed start"
|
||||
elseif SWB_UNIT0 = 1 && SWB_UNIT1 = 0 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed ps -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed init -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed clear -u 0"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed show -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed start -u 0"
|
||||
elseif SWB_UNIT0 = 0 && SWB_UNIT1 = 1 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed ps -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed init -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed clear -u 1"
|
||||
wait prompt_sonic_root
|
||||
pause 15
|
||||
sendln "blanton_traffic_linespeed show -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed start -u 1"
|
||||
else
|
||||
|
||||
endif
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -12,6 +12,21 @@ pause 1
|
||||
|
||||
; ========== Kill Process ==========
|
||||
include "utils/kill_all_process.ttl"
|
||||
; bgctrl all stop
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl stop --all"
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl stop --all"
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl reset --yes"
|
||||
; BMC USB Test STOP
|
||||
wait prompt_sonic_root
|
||||
sendln "systemctl stop qfx5252-bmc-usb-net-test.service"
|
||||
; 10G/1G MGMT Test STOP
|
||||
wait prompt_sonic_root
|
||||
sendln "~/Blanton_Script/mgmt_ping_monitor.sh stop"
|
||||
|
||||
|
||||
; ========== TAKE DATA ==========
|
||||
include "utils/setup_pmon.ttl"
|
||||
include "utils/show_dmesg.ttl"
|
||||
@@ -25,17 +40,78 @@ include "utils/show_pcie_error_reg_DDR.ttl"
|
||||
include "utils/show_pcie_error_reg_I210.ttl"
|
||||
include "utils/show_pcie_error_reg_SSD.ttl"
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln "/usr/sbin/ras-mc-ctl --summary"
|
||||
|
||||
; ========== BMC DDR TAKE DATA ==========
|
||||
;wait prompt_sonic_root
|
||||
;sendln "./Blanton_Script/bmc_monitor_ddr.sh stop"
|
||||
;wait prompt_sonic_root
|
||||
;sendln "./Blanton_Script/bmc_monitor.sh stop"
|
||||
;wait prompt_sonic_root
|
||||
;sendln "cat ./Blanton_Script/log/bmc_poll.log"
|
||||
;wait prompt_sonic_root
|
||||
;sendln "cat ./Blanton_Script/log/bmc_ddr.log"
|
||||
|
||||
; ========== BMC USB Test result ==========
|
||||
;wait prompt_sonic_root
|
||||
;sendln "journalctl -u qfx5252-bmc-usb-net-test.service --no-pager"
|
||||
|
||||
; ========== 10G/1G MGMT Test result ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "ip -s link show eth0"
|
||||
wait prompt_sonic_root
|
||||
sendln "ip -s link show eth1"
|
||||
|
||||
|
||||
; ========== CHECK Stress results ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "cat ~/hammer/tools/amd/mlucas_amm_log"
|
||||
wait prompt_sonic_root
|
||||
sendln "cat ~/hammer/tools/log.stress_ssd"
|
||||
|
||||
; ========== CHECK Traffic counters ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed stop -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed report -u 0"
|
||||
if SWB_UNIT0 = 1 && SWB_UNIT1 = 1 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed stop"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed report"
|
||||
elseif SWB_UNIT0 = 1 && SWB_UNIT1 = 0 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed stop -u 0"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed report -u 0"
|
||||
elseif SWB_UNIT0 = 0 && SWB_UNIT1 = 1 then
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed stop -u 1"
|
||||
wait prompt_sonic_root
|
||||
sendln "blanton_traffic_linespeed report -u 1"
|
||||
else
|
||||
|
||||
endif
|
||||
|
||||
; ========== NVME Err Info ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "/usr/sbin/nvme smart-log -H /dev/nvme0"
|
||||
wait prompt_sonic_root
|
||||
sendln "/usr/sbin/smartctl -x /dev/nvme0"
|
||||
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln "show reboot-cause"
|
||||
|
||||
; ========== Show Power on uptime ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "show uptime"
|
||||
|
||||
; ========== Copy Stress Log to USB ==========
|
||||
getdate ts_date "%Y%m%d"
|
||||
gettime ts_time "%H%M"
|
||||
sprintf2 usb_dst "/mnt/usb/jobs-%s_%s" ts_date ts_time
|
||||
sprintf2 cmd "cp -r /host/hw-eval/current/jobs/ %s" usb_dst
|
||||
wait prompt_sonic_root
|
||||
sendln cmd
|
||||
|
||||
; ========== HW Test Session ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "hw-test-session finish"
|
||||
|
||||
messagebox 'GOOD JOB! Test Case DONE' 'teraterm'
|
||||
+22
@@ -1,5 +1,27 @@
|
||||
# margin.sh 實際下線指令追蹤 (Command Trace)
|
||||
|
||||
> ## ⚠️ 過期警告(2026-08-19)
|
||||
>
|
||||
> **本文件第 5~7 章的 SWB 指令追蹤已與現況不符,請勿照抄。**
|
||||
> 上機驗證後 SWB 的 channel 配對與 LTC2977 位址都改了(commit `ababb2c`),
|
||||
> 但本文件的 1223 條 `cb_pmbus_*` 指令是當時依舊值展開的,其中 487 條用到
|
||||
> 已不存在的位址(`0x5D` / `0x5F` / `0x63` / `0x65`)。
|
||||
>
|
||||
> **正解以 `settings/*.conf` 為準:**
|
||||
>
|
||||
> | Channel | 板/接頭 | LTC2977 位址 |
|
||||
> |---|---|---|
|
||||
> | Ch6 | SWB0 CONN14 / CONN16 | CONN14 `0x5C`,`0x5E` ; CONN16 `0x62`,`0x64` |
|
||||
> | Ch8 | SWB0 CONN13 / CONN15 | CONN13 `0x5C`,`0x5E` ; CONN15 `0x62`,`0x64` |
|
||||
> | Ch7 | SWB1 CONN14 / CONN16 | CONN14 `0x5C`,`0x5E` ; CONN16 `0x62`,`0x64` |
|
||||
> | Ch9 | SWB1 CONN13 / CONN15 | CONN13 `0x5C`,`0x5E` ; CONN15 `0x62`,`0x64` |
|
||||
>
|
||||
> 位址在同一顆 LTC2980 內是 **+2** 遞增(7-bit,由規格書 8-bit 配對除以 2 而來),
|
||||
> 不是舊文件寫的 +1。原先記錄的 Ch6 上 `0x5E` 相撞問題已隨這次修正消失。
|
||||
>
|
||||
> 第 1~4 章(CB / i2c 路徑、暫存器語意、LINEAR16 換算)不受影響,仍然有效。
|
||||
> 需要正確的 SWB 指令追蹤請重新產生本文件。
|
||||
|
||||
對象:`LTC2980_Margin_Script/margin.sh` v2.6.0
|
||||
產生日期:2026-08-18
|
||||
用途:紀錄每個 margin 指令在 I2C / PMBus 線上真正下了什麼,方便對照 log、
|
||||
|
||||
@@ -17,8 +17,8 @@
|
||||
# * DEBUG_MODE collision fix: blanton_fpga_pcimem.sh uses the same var
|
||||
# name, so pcimem tracing is gated by MARGIN_SWB_PCIMEM_DEBUG
|
||||
# * margin_swb_* wrappers (info/probe/scan/reset/sem) for debugging
|
||||
# Physical map: CB F3 Ch6=SWB0 CONN13/14, Ch8=SWB0 CONN15/16,
|
||||
# Ch7=SWB1 CONN13/14, Ch9=SWB1 CONN15/16
|
||||
# Physical map: CB F3 Ch6=SWB0 CONN14/16, Ch8=SWB0 CONN13/15,
|
||||
# Ch7=SWB1 CONN14/16, Ch9=SWB1 CONN13/15
|
||||
# v2.5.0 SWB margin support via a per-conf TRANSPORT layer. The SWB margin
|
||||
# boards are ALSO LTC2980 (2x LTC2977) -- same chip as CB -- just reached
|
||||
# through the CB FPGA F3 I2C master instead of a native bus. So the whole
|
||||
@@ -72,10 +72,10 @@ CB_I2C_CH=""
|
||||
# --- SWB transport backend (blanton_cb_i2c.sh + blanton_fpga_pcimem.sh) -------
|
||||
# The SWB margin boards' LTC2980s hang off the CB FPGA Function 3 I2C master.
|
||||
# Physical mapping (per bench wiring):
|
||||
# CB F3 Ch6 -> SWB0 CONN13 / CONN14
|
||||
# CB F3 Ch8 -> SWB0 CONN15 / CONN16
|
||||
# CB F3 Ch7 -> SWB1 CONN13 / CONN14
|
||||
# CB F3 Ch9 -> SWB1 CONN15 / CONN16
|
||||
# CB F3 Ch6 -> SWB0 CONN14 / CONN16
|
||||
# CB F3 Ch8 -> SWB0 CONN13 / CONN15
|
||||
# CB F3 Ch7 -> SWB1 CONN14 / CONN16
|
||||
# CB F3 Ch9 -> SWB1 CONN13 / CONN15
|
||||
# Two CONNs share one FPGA I2C channel, so the two boards on a channel MUST use
|
||||
# different LTC2977 slave addresses (the CHIPS array in each .conf).
|
||||
MARGIN_SWB_CHANNELS="${MARGIN_SWB_CHANNELS:-6 7 8 9}"
|
||||
|
||||
+1
-1
@@ -9,7 +9,7 @@
|
||||
# --- SWB transport (added): this LTC2980 sits behind the CB FPGA F3 I2C; reach
|
||||
# it with cb_pmbus, not i2cset. CB_I2C_CH = FPGA I2C channel for this CONN. ---
|
||||
TRANSPORT="swb"
|
||||
CB_I2C_CH=6 # CB F3 Ch6 = SWB0 CONN13/CONN14 (shared with CONN14, addr must differ)
|
||||
CB_I2C_CH=8 # CB F3 Ch8 = SWB0 CONN13/CONN15 (shared with CONN15, addr must differ)
|
||||
|
||||
# --- I2C Configuration ---
|
||||
|
||||
|
||||
+3
-3
@@ -9,15 +9,15 @@
|
||||
# --- SWB transport (added): this LTC2980 sits behind the CB FPGA F3 I2C; reach
|
||||
# it with cb_pmbus, not i2cset. CB_I2C_CH = FPGA I2C channel for this CONN. ---
|
||||
TRANSPORT="swb"
|
||||
CB_I2C_CH=6 # CB F3 Ch6 = SWB0 CONN13/CONN14 (shared with CONN13, addr must differ)
|
||||
CB_I2C_CH=6 # CB F3 Ch6 = SWB0 CONN14/CONN16 (shared with CONN16, addr must differ)
|
||||
|
||||
# --- I2C Configuration ---
|
||||
|
||||
# Format: "bus:address" (8 channels per entry / per LTC2977)
|
||||
CHIPS=(
|
||||
# chip1: SWB0 (CONN14)
|
||||
"0:0x5E" # CH0 ~ CH7
|
||||
"0:0x5F" # CH8 ~ CH15
|
||||
"0:0x5C" # CH0 ~ CH7
|
||||
"0:0x5E" # CH8 ~ CH15
|
||||
)
|
||||
|
||||
# Pre-commands to run before accessing each chip (empty = none)
|
||||
|
||||
+2
-2
@@ -9,7 +9,7 @@
|
||||
# --- SWB transport (added): this LTC2980 sits behind the CB FPGA F3 I2C; reach
|
||||
# it with cb_pmbus, not i2cset. CB_I2C_CH = FPGA I2C channel for this CONN. ---
|
||||
TRANSPORT="swb"
|
||||
CB_I2C_CH=8 # CB F3 Ch8 = SWB0 CONN15/CONN16 (shared with CONN16, addr must differ)
|
||||
CB_I2C_CH=8 # CB F3 Ch8 = SWB0 CONN13/CONN15 (shared with CONN13, addr must differ)
|
||||
|
||||
# --- I2C Configuration ---
|
||||
|
||||
@@ -17,7 +17,7 @@ CB_I2C_CH=8 # CB F3 Ch8 = SWB0 CONN15/CONN16 (shared with CONN16, addr
|
||||
CHIPS=(
|
||||
# chip1: SWB0 (CONN15)
|
||||
"0:0x62" # CH0 ~ CH7
|
||||
"0:0x63" # CH8 ~ CH15
|
||||
"0:0x64" # CH8 ~ CH15
|
||||
)
|
||||
|
||||
# Pre-commands to run before accessing each chip (empty = none)
|
||||
|
||||
+3
-3
@@ -9,15 +9,15 @@
|
||||
# --- SWB transport (added): this LTC2980 sits behind the CB FPGA F3 I2C; reach
|
||||
# it with cb_pmbus, not i2cset. CB_I2C_CH = FPGA I2C channel for this CONN. ---
|
||||
TRANSPORT="swb"
|
||||
CB_I2C_CH=8 # CB F3 Ch8 = SWB0 CONN15/CONN16 (shared with CONN15, addr must differ)
|
||||
CB_I2C_CH=6 # CB F3 Ch6 = SWB0 CONN14/CONN16 (shared with CONN14, addr must differ)
|
||||
|
||||
# --- I2C Configuration ---
|
||||
|
||||
# Format: "bus:address" (8 channels per entry / per LTC2977)
|
||||
CHIPS=(
|
||||
# chip1: SWB0 (CONN16)
|
||||
"0:0x64" # CH0 ~ CH7
|
||||
"0:0x65" # CH8 ~ CH15
|
||||
"0:0x62" # CH0 ~ CH7
|
||||
"0:0x64" # CH8 ~ CH15
|
||||
)
|
||||
|
||||
# Pre-commands to run before accessing each chip (empty = none)
|
||||
|
||||
+2
-2
@@ -9,7 +9,7 @@
|
||||
# --- SWB transport (added): this LTC2980 sits behind the CB FPGA F3 I2C; reach
|
||||
# it with cb_pmbus, not i2cset. CB_I2C_CH = FPGA I2C channel for this CONN. ---
|
||||
TRANSPORT="swb"
|
||||
CB_I2C_CH=7 # CB F3 Ch7 = SWB1 CONN13/CONN14 (shared with CONN14, addr must differ)
|
||||
CB_I2C_CH=9 # CB F3 Ch9 = SWB1 CONN13/CONN15 (shared with CONN15, addr must differ)
|
||||
|
||||
# --- I2C Configuration ---
|
||||
|
||||
@@ -17,7 +17,7 @@ CB_I2C_CH=7 # CB F3 Ch7 = SWB1 CONN13/CONN14 (shared with CONN14, addr
|
||||
CHIPS=(
|
||||
# chip1: SWB1 (CONN13)
|
||||
"1:0x5C" # CH0 ~ CH7
|
||||
"1:0x5D" # CH8 ~ CH15
|
||||
"1:0x5E" # CH8 ~ CH15
|
||||
)
|
||||
|
||||
# Pre-commands to run before accessing each chip (empty = none)
|
||||
|
||||
+3
-3
@@ -9,15 +9,15 @@
|
||||
# --- SWB transport (added): this LTC2980 sits behind the CB FPGA F3 I2C; reach
|
||||
# it with cb_pmbus, not i2cset. CB_I2C_CH = FPGA I2C channel for this CONN. ---
|
||||
TRANSPORT="swb"
|
||||
CB_I2C_CH=7 # CB F3 Ch7 = SWB1 CONN13/CONN14 (shared with CONN13, addr must differ)
|
||||
CB_I2C_CH=7 # CB F3 Ch7 = SWB1 CONN14/CONN16 (shared with CONN16, addr must differ)
|
||||
|
||||
# --- I2C Configuration ---
|
||||
|
||||
# Format: "bus:address" (8 channels per entry / per LTC2977)
|
||||
CHIPS=(
|
||||
# chip1: SWB1 (CONN14)
|
||||
"1:0x5E" # CH0 ~ CH7
|
||||
"1:0x5F" # CH8 ~ CH15
|
||||
"1:0x5C" # CH0 ~ CH7
|
||||
"1:0x5E" # CH8 ~ CH15
|
||||
)
|
||||
|
||||
# Pre-commands to run before accessing each chip (empty = none)
|
||||
|
||||
+2
-2
@@ -9,7 +9,7 @@
|
||||
# --- SWB transport (added): this LTC2980 sits behind the CB FPGA F3 I2C; reach
|
||||
# it with cb_pmbus, not i2cset. CB_I2C_CH = FPGA I2C channel for this CONN. ---
|
||||
TRANSPORT="swb"
|
||||
CB_I2C_CH=9 # CB F3 Ch9 = SWB1 CONN15/CONN16 (shared with CONN16, addr must differ)
|
||||
CB_I2C_CH=9 # CB F3 Ch9 = SWB1 CONN13/CONN15 (shared with CONN13, addr must differ)
|
||||
|
||||
# --- I2C Configuration ---
|
||||
|
||||
@@ -17,7 +17,7 @@ CB_I2C_CH=9 # CB F3 Ch9 = SWB1 CONN15/CONN16 (shared with CONN16, addr
|
||||
CHIPS=(
|
||||
# chip1: SWB1 (CONN15)
|
||||
"1:0x62" # CH0 ~ CH7
|
||||
"1:0x63" # CH8 ~ CH15
|
||||
"1:0x64" # CH8 ~ CH15
|
||||
)
|
||||
|
||||
# Pre-commands to run before accessing each chip (empty = none)
|
||||
|
||||
+3
-3
@@ -9,15 +9,15 @@
|
||||
# --- SWB transport (added): this LTC2980 sits behind the CB FPGA F3 I2C; reach
|
||||
# it with cb_pmbus, not i2cset. CB_I2C_CH = FPGA I2C channel for this CONN. ---
|
||||
TRANSPORT="swb"
|
||||
CB_I2C_CH=9 # CB F3 Ch9 = SWB1 CONN15/CONN16 (shared with CONN15, addr must differ)
|
||||
CB_I2C_CH=7 # CB F3 Ch7 = SWB1 CONN14/CONN16 (shared with CONN14, addr must differ)
|
||||
|
||||
# --- I2C Configuration ---
|
||||
|
||||
# Format: "bus:address" (8 channels per entry / per LTC2977)
|
||||
CHIPS=(
|
||||
# chip1: SWB1 (CONN16)
|
||||
"1:0x64" # CH0 ~ CH7
|
||||
"1:0x65" # CH8 ~ CH15
|
||||
"1:0x62" # CH0 ~ CH7
|
||||
"1:0x64" # CH8 ~ CH15
|
||||
)
|
||||
|
||||
# Pre-commands to run before accessing each chip (empty = none)
|
||||
|
||||
@@ -61,6 +61,19 @@
|
||||
# intentional difference from
|
||||
# tools/bcm_mibpair_report_V1.1.0.py, which hides NA pairs
|
||||
# unless -a is given.
|
||||
# V0.4.1 2026-08-19 Fix: report printed every counter as 2147483647 (INT32_MAX).
|
||||
# grp() rendered values with sprintf("%d", v); the DUT's awk
|
||||
# casts to a 32-bit int for %d, so any counter above 2^31-1
|
||||
# -- i.e. every real line-speed run -- saturated. Values are
|
||||
# now formatted with %.0f, which stays in awk's double domain
|
||||
# (exact to 2^53, far above any MIB counter).
|
||||
# NOTE: the PASS/FAIL verdicts were NEVER affected -- the
|
||||
# cross-check compares TX[]/RX[] doubles directly and only
|
||||
# the display path went through grp(). Past reports that said
|
||||
# PASS were right; only the numbers shown were wrong.
|
||||
# Same fix applied to the "# FAIL ..." delta line (%+d ->
|
||||
# %+.0f), which would otherwise saturate when a dead port
|
||||
# makes the delta the size of the whole counter.
|
||||
# =============================================================================
|
||||
#
|
||||
# The VLAN/port table below is copied verbatim from
|
||||
@@ -486,7 +499,7 @@ _tl_pair_report() {
|
||||
-v TSV="$TL_REPORT_TSV" \
|
||||
-v COLOR="$color" '
|
||||
function grp(v, s,out,l,i) {
|
||||
s = sprintf("%d", v)
|
||||
s = sprintf("%.0f", v)
|
||||
if (!GROUPED) return s
|
||||
out = ""; l = length(s)
|
||||
for (i = 1; i <= l; i++) {
|
||||
@@ -557,7 +570,7 @@ _tl_pair_report() {
|
||||
if (d <= TOL + 0) { v = "PASS"; npass++ }
|
||||
else {
|
||||
v = "FAIL"; nfail++
|
||||
FL[++nf] = sprintf("# FAIL %s/%s: %s.TX-%s.RX=%+d %s.RX-%s.TX=%+d", \
|
||||
FL[++nf] = sprintf("# FAIL %s/%s: %s.TX-%s.RX=%+.0f %s.RX-%s.TX=%+.0f", \
|
||||
a, b, a, b, dtx, a, b, drx)
|
||||
}
|
||||
ta = grp(TX[a]); ra = grp(RX[a]); tb = grp(TX[b]); rb = grp(RX[b])
|
||||
|
||||
@@ -0,0 +1,340 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# bmc_monitor.sh
|
||||
#
|
||||
# Version : V1.1.0
|
||||
# Author : ETWen
|
||||
# Date : 20260820
|
||||
#
|
||||
# Purpose : Periodically execute a list of commands on the BMC through
|
||||
# `bmc-manager run "<cmd>"` from the SONiC host side, and append all
|
||||
# output to a rotating log file. Designed to run detached in the
|
||||
# background so no interactive SSH session to the BMC is required.
|
||||
#
|
||||
# Notes : - Everything runs on the HOST. Nothing is deployed to the BMC, so
|
||||
# the BMC busybox toolchain (no base64 / no setsid ...) is a
|
||||
# non-issue.
|
||||
# - `bmc-manager run` prints the remote command output on stdout and
|
||||
# its own banner on stderr; the two streams are captured separately
|
||||
# so the log stays clean.
|
||||
#
|
||||
# Version History
|
||||
# V1.0.0 20260820 Initial Version
|
||||
# V1.1.0 20260820 `start` clears the previous log by default
|
||||
# (START_LOG_MODE: new / archive / append)
|
||||
###############################################################################
|
||||
|
||||
set -u
|
||||
|
||||
###############################################################################
|
||||
# User Configurable Section -- normally this is the only part you need to edit
|
||||
###############################################################################
|
||||
|
||||
# Commands executed on the BMC, one per line, in this order.
|
||||
# Add a line to add a command; comment it out to disable it.
|
||||
COMMANDS=(
|
||||
"free -m"
|
||||
#"uptime"
|
||||
#"cat /proc/loadavg"
|
||||
#"cat /proc/meminfo"
|
||||
#"df -h /tmp"
|
||||
#"dmesg | tail -n 30"
|
||||
)
|
||||
|
||||
# Seconds to wait after a full round (all COMMANDS executed) before the next one
|
||||
INTERVAL_SEC=10
|
||||
|
||||
# Seconds to wait between two commands inside the same round (0 = no wait)
|
||||
INTER_CMD_DELAY_SEC=0
|
||||
|
||||
# Per-command timeout in seconds; prevents a hung bmc-manager from freezing the
|
||||
# whole loop. Set to 0 to disable.
|
||||
CMD_TIMEOUT_SEC=30
|
||||
|
||||
# Stop after this many rounds. 0 = run until stopped manually.
|
||||
MAX_ROUNDS=0
|
||||
|
||||
# Command runner. Final invocation is:
|
||||
# "$RUNNER_BIN" "${RUNNER_ARGS[@]}" "<command string>"
|
||||
# BMC side : RUNNER_BIN="bmc-manager" ; RUNNER_ARGS=(run)
|
||||
# BMC by IP : RUNNER_BIN="bmc-manager" ; RUNNER_ARGS=(--ip 192.168.200.200 run)
|
||||
# Host side : RUNNER_BIN="bash" ; RUNNER_ARGS=(-c)
|
||||
RUNNER_BIN="bmc-manager"
|
||||
RUNNER_ARGS=(run)
|
||||
|
||||
# stderr handling of the runner (bmc-manager banner lives here)
|
||||
# 0 = log stderr only when the command fails (recommended, keeps log clean)
|
||||
# 1 = always log stderr
|
||||
LOG_STDERR=0
|
||||
|
||||
# Logging
|
||||
LOG_DIR="" # empty = <script directory>/log
|
||||
LOG_NAME="bmc_poll.log"
|
||||
MAX_LOG_SIZE_MB=100 # rotate above this size; 0 = never rotate
|
||||
MAX_LOG_KEEP=5 # keep .1 .. .N
|
||||
|
||||
# What `start` does with the log left behind by the previous run:
|
||||
# new = begin with an empty log; previous content and rotated copies are
|
||||
# discarded (default -- one run, one log)
|
||||
# archive = rename the previous log to <log>.YYYYmmdd-HHMMSS, then begin empty
|
||||
# append = keep appending to the existing log
|
||||
START_LOG_MODE="new"
|
||||
|
||||
###############################################################################
|
||||
# Internal
|
||||
###############################################################################
|
||||
SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")"
|
||||
SCRIPT_PATH="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)/${SCRIPT_NAME}"
|
||||
[ -n "${LOG_DIR}" ] || LOG_DIR="$(dirname -- "${SCRIPT_PATH}")/log"
|
||||
LOG_FILE="${LOG_DIR}/${LOG_NAME}"
|
||||
PID_FILE="${LOG_DIR}/${LOG_NAME%.log}.pid"
|
||||
|
||||
LOG_TO_FILE=0
|
||||
STOP=0
|
||||
SLEEP_PID=""
|
||||
|
||||
ts() { date '+%Y-%m-%d %H:%M:%S'; }
|
||||
die() { printf '[ERROR] %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
on_signal() {
|
||||
STOP=1
|
||||
[ -n "${SLEEP_PID}" ] && kill "${SLEEP_PID}" 2>/dev/null
|
||||
}
|
||||
|
||||
# sleep that can be aborted immediately by SIGTERM/SIGINT
|
||||
interruptible_sleep() {
|
||||
local sec="$1"
|
||||
[ "${sec}" -gt 0 ] 2>/dev/null || return 0
|
||||
sleep "${sec}" &
|
||||
SLEEP_PID=$!
|
||||
wait "${SLEEP_PID}" 2>/dev/null
|
||||
SLEEP_PID=""
|
||||
}
|
||||
|
||||
# echo the pid and return 0 when a live instance is found
|
||||
is_running() {
|
||||
local pid
|
||||
[ -f "${PID_FILE}" ] || return 1
|
||||
pid="$(cat "${PID_FILE}" 2>/dev/null)"
|
||||
[[ "${pid}" =~ ^[0-9]+$ ]] || return 1
|
||||
kill -0 "${pid}" 2>/dev/null || return 1
|
||||
grep -qa "${SCRIPT_NAME}" "/proc/${pid}/cmdline" 2>/dev/null || return 1
|
||||
printf '%s' "${pid}"
|
||||
return 0
|
||||
}
|
||||
|
||||
rotate_log_if_needed() {
|
||||
local max_bytes size i
|
||||
[ "${LOG_TO_FILE}" -eq 1 ] || return 0
|
||||
[ "${MAX_LOG_SIZE_MB}" -gt 0 ] || return 0
|
||||
[ -f "${LOG_FILE}" ] || return 0
|
||||
|
||||
max_bytes=$(( MAX_LOG_SIZE_MB * 1024 * 1024 ))
|
||||
size="$(stat -c %s "${LOG_FILE}" 2>/dev/null || echo 0)"
|
||||
[ "${size}" -ge "${max_bytes}" ] || return 0
|
||||
|
||||
for (( i = MAX_LOG_KEEP - 1; i >= 1; i-- )); do
|
||||
[ -f "${LOG_FILE}.${i}" ] && mv -f "${LOG_FILE}.${i}" "${LOG_FILE}.$(( i + 1 ))"
|
||||
done
|
||||
mv -f "${LOG_FILE}" "${LOG_FILE}.1"
|
||||
exec >>"${LOG_FILE}" 2>&1 # re-bind stdout/stderr to the new file
|
||||
printf '[%s] INFO : log rotated at %s MB\n' "$(ts)" "${MAX_LOG_SIZE_MB}"
|
||||
}
|
||||
|
||||
# Remove the rotated copies (.1 .. .N) only. Timestamped archives are kept.
|
||||
remove_rotated_logs() {
|
||||
local i
|
||||
for (( i = 1; i <= MAX_LOG_KEEP + 1; i++ )); do
|
||||
rm -f "${LOG_FILE}.${i}"
|
||||
done
|
||||
}
|
||||
|
||||
# Apply START_LOG_MODE just before a background run is launched.
|
||||
prepare_log_on_start() {
|
||||
local stamp
|
||||
case "${START_LOG_MODE}" in
|
||||
append)
|
||||
return 0
|
||||
;;
|
||||
archive)
|
||||
if [ -s "${LOG_FILE}" ]; then
|
||||
stamp="$(date '+%Y%m%d-%H%M%S')"
|
||||
mv -f "${LOG_FILE}" "${LOG_FILE}.${stamp}" \
|
||||
|| die "cannot archive ${LOG_FILE}"
|
||||
printf 'previous log archived: %s.%s\n' "${LOG_FILE}" "${stamp}"
|
||||
fi
|
||||
remove_rotated_logs
|
||||
;;
|
||||
new)
|
||||
# Truncate rather than unlink: the inode is preserved, so a `tail -f`
|
||||
# that is already attached keeps following the new run.
|
||||
if [ -f "${LOG_FILE}" ]; then
|
||||
: > "${LOG_FILE}" || die "cannot truncate ${LOG_FILE}"
|
||||
fi
|
||||
remove_rotated_logs
|
||||
;;
|
||||
*)
|
||||
die "invalid START_LOG_MODE: ${START_LOG_MODE}"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
run_one_command() {
|
||||
local cmd="$1"
|
||||
local out err rc t0 elapsed err_file
|
||||
|
||||
err_file="$(mktemp)"
|
||||
t0=${SECONDS}
|
||||
if [ "${CMD_TIMEOUT_SEC}" -gt 0 ]; then
|
||||
out="$(timeout "${CMD_TIMEOUT_SEC}" \
|
||||
"${RUNNER_BIN}" "${RUNNER_ARGS[@]}" "${cmd}" 2>"${err_file}")"
|
||||
else
|
||||
out="$("${RUNNER_BIN}" "${RUNNER_ARGS[@]}" "${cmd}" 2>"${err_file}")"
|
||||
fi
|
||||
rc=$?
|
||||
elapsed=$(( SECONDS - t0 ))
|
||||
err="$(cat "${err_file}" 2>/dev/null)"
|
||||
rm -f "${err_file}"
|
||||
|
||||
printf '===== [%s] CMD: %s | rc=%d | %ds =====\n' "$(ts)" "${cmd}" "${rc}" "${elapsed}"
|
||||
printf '%s\n' "${out}"
|
||||
if [ "${rc}" -eq 124 ]; then
|
||||
printf '[%s] WARN : timeout after %ds\n' "$(ts)" "${CMD_TIMEOUT_SEC}"
|
||||
fi
|
||||
if [ -n "${err}" ] && { [ "${LOG_STDERR}" -eq 1 ] || [ "${rc}" -ne 0 ]; }; then
|
||||
printf -- '--- stderr ---\n%s\n' "${err}"
|
||||
fi
|
||||
printf '\n'
|
||||
}
|
||||
|
||||
main_loop() {
|
||||
local round=0 cmd
|
||||
|
||||
[ "${#COMMANDS[@]}" -gt 0 ] || die "COMMANDS is empty"
|
||||
command -v "${RUNNER_BIN}" >/dev/null 2>&1 || die "runner not found: ${RUNNER_BIN}"
|
||||
|
||||
trap on_signal INT TERM
|
||||
|
||||
printf '#############################################################\n'
|
||||
printf '[%s] START: pid=%d interval=%ds timeout=%ds cmds=%d\n' \
|
||||
"$(ts)" "$$" "${INTERVAL_SEC}" "${CMD_TIMEOUT_SEC}" "${#COMMANDS[@]}"
|
||||
printf '#############################################################\n'
|
||||
|
||||
while [ "${STOP}" -eq 0 ]; do
|
||||
round=$(( round + 1 ))
|
||||
printf -- '---------- ROUND %d @ %s ----------\n' "${round}" "$(ts)"
|
||||
|
||||
for cmd in "${COMMANDS[@]}"; do
|
||||
[ "${STOP}" -eq 0 ] || break
|
||||
run_one_command "${cmd}"
|
||||
[ "${INTER_CMD_DELAY_SEC}" -gt 0 ] && interruptible_sleep "${INTER_CMD_DELAY_SEC}"
|
||||
done
|
||||
|
||||
rotate_log_if_needed
|
||||
|
||||
if [ "${MAX_ROUNDS}" -gt 0 ] && [ "${round}" -ge "${MAX_ROUNDS}" ]; then
|
||||
break
|
||||
fi
|
||||
# Fixed gap between rounds. For a fixed cadence instead, replace with:
|
||||
# interruptible_sleep $(( INTERVAL_SEC - round_elapsed ))
|
||||
[ "${STOP}" -eq 0 ] && interruptible_sleep "${INTERVAL_SEC}"
|
||||
done
|
||||
|
||||
printf '[%s] STOP : pid=%d after %d round(s)\n' "$(ts)" "$$" "${round}"
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Sub-commands
|
||||
###############################################################################
|
||||
do_start() {
|
||||
local pid
|
||||
pid="$(is_running)" && die "already running (pid=${pid})"
|
||||
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
|
||||
prepare_log_on_start
|
||||
|
||||
if command -v setsid >/dev/null 2>&1; then
|
||||
setsid "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 &
|
||||
else
|
||||
nohup "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 &
|
||||
disown 2>/dev/null
|
||||
fi
|
||||
|
||||
sleep 1
|
||||
pid="$(is_running)" || die "start failed, check ${LOG_FILE}"
|
||||
printf 'started (pid=%s)\nlog: %s\n' "${pid}" "${LOG_FILE}"
|
||||
}
|
||||
|
||||
do_daemon() {
|
||||
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
|
||||
LOG_TO_FILE=1
|
||||
exec >>"${LOG_FILE}" 2>&1
|
||||
printf '%d\n' "$$" > "${PID_FILE}"
|
||||
trap 'rm -f "${PID_FILE}"' EXIT
|
||||
main_loop
|
||||
}
|
||||
|
||||
do_stop() {
|
||||
local pid i
|
||||
pid="$(is_running)" || { printf 'not running\n'; rm -f "${PID_FILE}"; return 0; }
|
||||
|
||||
kill -TERM -"${pid}" 2>/dev/null || kill -TERM "${pid}" 2>/dev/null
|
||||
for (( i = 0; i < 20; i++ )); do
|
||||
kill -0 "${pid}" 2>/dev/null || break
|
||||
sleep 0.5
|
||||
done
|
||||
if kill -0 "${pid}" 2>/dev/null; then
|
||||
printf 'SIGTERM ignored, sending SIGKILL\n'
|
||||
kill -KILL -"${pid}" 2>/dev/null || kill -KILL "${pid}" 2>/dev/null
|
||||
fi
|
||||
rm -f "${PID_FILE}"
|
||||
printf 'stopped (pid=%s)\n' "${pid}"
|
||||
}
|
||||
|
||||
do_status() {
|
||||
local pid
|
||||
if pid="$(is_running)"; then
|
||||
printf 'status : running (pid=%s)\n' "${pid}"
|
||||
else
|
||||
printf 'status : stopped\n'
|
||||
fi
|
||||
printf 'log : %s\n' "${LOG_FILE}"
|
||||
[ -f "${LOG_FILE}" ] && printf 'size : %s\n' "$(du -h "${LOG_FILE}" | cut -f1)"
|
||||
printf 'cmds : %d, interval=%ds\n' "${#COMMANDS[@]}" "${INTERVAL_SEC}"
|
||||
}
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: ${SCRIPT_NAME} {start|stop|status|fg|tail|clear}
|
||||
|
||||
start Run detached in the background; the previous log is discarded first
|
||||
(see START_LOG_MODE)
|
||||
stop Stop the background instance
|
||||
status Show pid / log path / log size
|
||||
fg Run in the foreground, output to the terminal (for a quick check)
|
||||
tail tail -f the log file
|
||||
clear Remove the log file and all rotated copies (must be stopped first)
|
||||
|
||||
Log file : ${LOG_FILE}
|
||||
|
||||
To add a command, edit the COMMANDS array at the top of this script.
|
||||
EOF
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Entry point
|
||||
###############################################################################
|
||||
case "${1:-}" in
|
||||
start) do_start ;;
|
||||
stop) do_stop ;;
|
||||
status) do_status ;;
|
||||
fg) LOG_TO_FILE=0; main_loop ;;
|
||||
tail) tail -n 50 -f "${LOG_FILE}" ;;
|
||||
clear)
|
||||
is_running >/dev/null && die "still running, stop it first"
|
||||
rm -f "${LOG_FILE}" "${LOG_FILE}".[0-9]*
|
||||
printf 'log cleared\n'
|
||||
;;
|
||||
__daemon) do_daemon ;;
|
||||
-h|--help|"") usage ;;
|
||||
*) printf '[ERROR] unknown sub-command: %s\n\n' "$1" >&2; usage; exit 1 ;;
|
||||
esac
|
||||
@@ -0,0 +1,340 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# bmc_monitor_ddr.sh
|
||||
#
|
||||
# Version : V1.1.0
|
||||
# Author : ETWen
|
||||
# Date : 20260820
|
||||
#
|
||||
# Purpose : Periodically execute a list of commands on the BMC through
|
||||
# `bmc-manager run "<cmd>"` from the SONiC host side, and append all
|
||||
# output to a rotating log file. Designed to run detached in the
|
||||
# background so no interactive SSH session to the BMC is required.
|
||||
#
|
||||
# Notes : - Everything runs on the HOST. Nothing is deployed to the BMC, so
|
||||
# the BMC busybox toolchain (no base64 / no setsid ...) is a
|
||||
# non-issue.
|
||||
# - `bmc-manager run` prints the remote command output on stdout and
|
||||
# its own banner on stderr; the two streams are captured separately
|
||||
# so the log stays clean.
|
||||
#
|
||||
# Version History
|
||||
# V1.0.0 20260820 Initial Version
|
||||
# V1.1.0 20260820 `start` clears the previous log by default
|
||||
# (START_LOG_MODE: new / archive / append)
|
||||
###############################################################################
|
||||
|
||||
set -u
|
||||
|
||||
###############################################################################
|
||||
# User Configurable Section -- normally this is the only part you need to edit
|
||||
###############################################################################
|
||||
|
||||
# Commands executed on the BMC, one per line, in this order.
|
||||
# Add a line to add a command; comment it out to disable it.
|
||||
COMMANDS=(
|
||||
"memtester 500M 1"
|
||||
#"uptime"
|
||||
#"cat /proc/loadavg"
|
||||
#"cat /proc/meminfo"
|
||||
#"df -h /tmp"
|
||||
#"dmesg | tail -n 30"
|
||||
)
|
||||
|
||||
# Seconds to wait after a full round (all COMMANDS executed) before the next one
|
||||
INTERVAL_SEC=5
|
||||
|
||||
# Seconds to wait between two commands inside the same round (0 = no wait)
|
||||
INTER_CMD_DELAY_SEC=0
|
||||
|
||||
# Per-command timeout in seconds; prevents a hung bmc-manager from freezing the
|
||||
# whole loop. Set to 0 to disable.
|
||||
CMD_TIMEOUT_SEC=3600
|
||||
|
||||
# Stop after this many rounds. 0 = run until stopped manually.
|
||||
MAX_ROUNDS=0
|
||||
|
||||
# Command runner. Final invocation is:
|
||||
# "$RUNNER_BIN" "${RUNNER_ARGS[@]}" "<command string>"
|
||||
# BMC side : RUNNER_BIN="bmc-manager" ; RUNNER_ARGS=(run)
|
||||
# BMC by IP : RUNNER_BIN="bmc-manager" ; RUNNER_ARGS=(--ip 192.168.200.200 run)
|
||||
# Host side : RUNNER_BIN="bash" ; RUNNER_ARGS=(-c)
|
||||
RUNNER_BIN="bmc-manager"
|
||||
RUNNER_ARGS=(run)
|
||||
|
||||
# stderr handling of the runner (bmc-manager banner lives here)
|
||||
# 0 = log stderr only when the command fails (recommended, keeps log clean)
|
||||
# 1 = always log stderr
|
||||
LOG_STDERR=0
|
||||
|
||||
# Logging
|
||||
LOG_DIR="" # empty = <script directory>/log
|
||||
LOG_NAME="bmc_ddr.log"
|
||||
MAX_LOG_SIZE_MB=100 # rotate above this size; 0 = never rotate
|
||||
MAX_LOG_KEEP=5 # keep .1 .. .N
|
||||
|
||||
# What `start` does with the log left behind by the previous run:
|
||||
# new = begin with an empty log; previous content and rotated copies are
|
||||
# discarded (default -- one run, one log)
|
||||
# archive = rename the previous log to <log>.YYYYmmdd-HHMMSS, then begin empty
|
||||
# append = keep appending to the existing log
|
||||
START_LOG_MODE="new"
|
||||
|
||||
###############################################################################
|
||||
# Internal
|
||||
###############################################################################
|
||||
SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")"
|
||||
SCRIPT_PATH="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)/${SCRIPT_NAME}"
|
||||
[ -n "${LOG_DIR}" ] || LOG_DIR="$(dirname -- "${SCRIPT_PATH}")/log"
|
||||
LOG_FILE="${LOG_DIR}/${LOG_NAME}"
|
||||
PID_FILE="${LOG_DIR}/${LOG_NAME%.log}.pid"
|
||||
|
||||
LOG_TO_FILE=0
|
||||
STOP=0
|
||||
SLEEP_PID=""
|
||||
|
||||
ts() { date '+%Y-%m-%d %H:%M:%S'; }
|
||||
die() { printf '[ERROR] %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
on_signal() {
|
||||
STOP=1
|
||||
[ -n "${SLEEP_PID}" ] && kill "${SLEEP_PID}" 2>/dev/null
|
||||
}
|
||||
|
||||
# sleep that can be aborted immediately by SIGTERM/SIGINT
|
||||
interruptible_sleep() {
|
||||
local sec="$1"
|
||||
[ "${sec}" -gt 0 ] 2>/dev/null || return 0
|
||||
sleep "${sec}" &
|
||||
SLEEP_PID=$!
|
||||
wait "${SLEEP_PID}" 2>/dev/null
|
||||
SLEEP_PID=""
|
||||
}
|
||||
|
||||
# echo the pid and return 0 when a live instance is found
|
||||
is_running() {
|
||||
local pid
|
||||
[ -f "${PID_FILE}" ] || return 1
|
||||
pid="$(cat "${PID_FILE}" 2>/dev/null)"
|
||||
[[ "${pid}" =~ ^[0-9]+$ ]] || return 1
|
||||
kill -0 "${pid}" 2>/dev/null || return 1
|
||||
grep -qa "${SCRIPT_NAME}" "/proc/${pid}/cmdline" 2>/dev/null || return 1
|
||||
printf '%s' "${pid}"
|
||||
return 0
|
||||
}
|
||||
|
||||
rotate_log_if_needed() {
|
||||
local max_bytes size i
|
||||
[ "${LOG_TO_FILE}" -eq 1 ] || return 0
|
||||
[ "${MAX_LOG_SIZE_MB}" -gt 0 ] || return 0
|
||||
[ -f "${LOG_FILE}" ] || return 0
|
||||
|
||||
max_bytes=$(( MAX_LOG_SIZE_MB * 1024 * 1024 ))
|
||||
size="$(stat -c %s "${LOG_FILE}" 2>/dev/null || echo 0)"
|
||||
[ "${size}" -ge "${max_bytes}" ] || return 0
|
||||
|
||||
for (( i = MAX_LOG_KEEP - 1; i >= 1; i-- )); do
|
||||
[ -f "${LOG_FILE}.${i}" ] && mv -f "${LOG_FILE}.${i}" "${LOG_FILE}.$(( i + 1 ))"
|
||||
done
|
||||
mv -f "${LOG_FILE}" "${LOG_FILE}.1"
|
||||
exec >>"${LOG_FILE}" 2>&1 # re-bind stdout/stderr to the new file
|
||||
printf '[%s] INFO : log rotated at %s MB\n' "$(ts)" "${MAX_LOG_SIZE_MB}"
|
||||
}
|
||||
|
||||
# Remove the rotated copies (.1 .. .N) only. Timestamped archives are kept.
|
||||
remove_rotated_logs() {
|
||||
local i
|
||||
for (( i = 1; i <= MAX_LOG_KEEP + 1; i++ )); do
|
||||
rm -f "${LOG_FILE}.${i}"
|
||||
done
|
||||
}
|
||||
|
||||
# Apply START_LOG_MODE just before a background run is launched.
|
||||
prepare_log_on_start() {
|
||||
local stamp
|
||||
case "${START_LOG_MODE}" in
|
||||
append)
|
||||
return 0
|
||||
;;
|
||||
archive)
|
||||
if [ -s "${LOG_FILE}" ]; then
|
||||
stamp="$(date '+%Y%m%d-%H%M%S')"
|
||||
mv -f "${LOG_FILE}" "${LOG_FILE}.${stamp}" \
|
||||
|| die "cannot archive ${LOG_FILE}"
|
||||
printf 'previous log archived: %s.%s\n' "${LOG_FILE}" "${stamp}"
|
||||
fi
|
||||
remove_rotated_logs
|
||||
;;
|
||||
new)
|
||||
# Truncate rather than unlink: the inode is preserved, so a `tail -f`
|
||||
# that is already attached keeps following the new run.
|
||||
if [ -f "${LOG_FILE}" ]; then
|
||||
: > "${LOG_FILE}" || die "cannot truncate ${LOG_FILE}"
|
||||
fi
|
||||
remove_rotated_logs
|
||||
;;
|
||||
*)
|
||||
die "invalid START_LOG_MODE: ${START_LOG_MODE}"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
run_one_command() {
|
||||
local cmd="$1"
|
||||
local out err rc t0 elapsed err_file
|
||||
|
||||
err_file="$(mktemp)"
|
||||
t0=${SECONDS}
|
||||
if [ "${CMD_TIMEOUT_SEC}" -gt 0 ]; then
|
||||
out="$(timeout "${CMD_TIMEOUT_SEC}" \
|
||||
"${RUNNER_BIN}" "${RUNNER_ARGS[@]}" "${cmd}" 2>"${err_file}")"
|
||||
else
|
||||
out="$("${RUNNER_BIN}" "${RUNNER_ARGS[@]}" "${cmd}" 2>"${err_file}")"
|
||||
fi
|
||||
rc=$?
|
||||
elapsed=$(( SECONDS - t0 ))
|
||||
err="$(cat "${err_file}" 2>/dev/null)"
|
||||
rm -f "${err_file}"
|
||||
|
||||
printf '===== [%s] CMD: %s | rc=%d | %ds =====\n' "$(ts)" "${cmd}" "${rc}" "${elapsed}"
|
||||
printf '%s\n' "${out}"
|
||||
if [ "${rc}" -eq 124 ]; then
|
||||
printf '[%s] WARN : timeout after %ds\n' "$(ts)" "${CMD_TIMEOUT_SEC}"
|
||||
fi
|
||||
if [ -n "${err}" ] && { [ "${LOG_STDERR}" -eq 1 ] || [ "${rc}" -ne 0 ]; }; then
|
||||
printf -- '--- stderr ---\n%s\n' "${err}"
|
||||
fi
|
||||
printf '\n'
|
||||
}
|
||||
|
||||
main_loop() {
|
||||
local round=0 cmd
|
||||
|
||||
[ "${#COMMANDS[@]}" -gt 0 ] || die "COMMANDS is empty"
|
||||
command -v "${RUNNER_BIN}" >/dev/null 2>&1 || die "runner not found: ${RUNNER_BIN}"
|
||||
|
||||
trap on_signal INT TERM
|
||||
|
||||
printf '#############################################################\n'
|
||||
printf '[%s] START: pid=%d interval=%ds timeout=%ds cmds=%d\n' \
|
||||
"$(ts)" "$$" "${INTERVAL_SEC}" "${CMD_TIMEOUT_SEC}" "${#COMMANDS[@]}"
|
||||
printf '#############################################################\n'
|
||||
|
||||
while [ "${STOP}" -eq 0 ]; do
|
||||
round=$(( round + 1 ))
|
||||
printf -- '---------- ROUND %d @ %s ----------\n' "${round}" "$(ts)"
|
||||
|
||||
for cmd in "${COMMANDS[@]}"; do
|
||||
[ "${STOP}" -eq 0 ] || break
|
||||
run_one_command "${cmd}"
|
||||
[ "${INTER_CMD_DELAY_SEC}" -gt 0 ] && interruptible_sleep "${INTER_CMD_DELAY_SEC}"
|
||||
done
|
||||
|
||||
rotate_log_if_needed
|
||||
|
||||
if [ "${MAX_ROUNDS}" -gt 0 ] && [ "${round}" -ge "${MAX_ROUNDS}" ]; then
|
||||
break
|
||||
fi
|
||||
# Fixed gap between rounds. For a fixed cadence instead, replace with:
|
||||
# interruptible_sleep $(( INTERVAL_SEC - round_elapsed ))
|
||||
[ "${STOP}" -eq 0 ] && interruptible_sleep "${INTERVAL_SEC}"
|
||||
done
|
||||
|
||||
printf '[%s] STOP : pid=%d after %d round(s)\n' "$(ts)" "$$" "${round}"
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Sub-commands
|
||||
###############################################################################
|
||||
do_start() {
|
||||
local pid
|
||||
pid="$(is_running)" && die "already running (pid=${pid})"
|
||||
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
|
||||
prepare_log_on_start
|
||||
|
||||
if command -v setsid >/dev/null 2>&1; then
|
||||
setsid "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 &
|
||||
else
|
||||
nohup "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 &
|
||||
disown 2>/dev/null
|
||||
fi
|
||||
|
||||
sleep 1
|
||||
pid="$(is_running)" || die "start failed, check ${LOG_FILE}"
|
||||
printf 'started (pid=%s)\nlog: %s\n' "${pid}" "${LOG_FILE}"
|
||||
}
|
||||
|
||||
do_daemon() {
|
||||
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
|
||||
LOG_TO_FILE=1
|
||||
exec >>"${LOG_FILE}" 2>&1
|
||||
printf '%d\n' "$$" > "${PID_FILE}"
|
||||
trap 'rm -f "${PID_FILE}"' EXIT
|
||||
main_loop
|
||||
}
|
||||
|
||||
do_stop() {
|
||||
local pid i
|
||||
pid="$(is_running)" || { printf 'not running\n'; rm -f "${PID_FILE}"; return 0; }
|
||||
|
||||
kill -TERM -"${pid}" 2>/dev/null || kill -TERM "${pid}" 2>/dev/null
|
||||
for (( i = 0; i < 20; i++ )); do
|
||||
kill -0 "${pid}" 2>/dev/null || break
|
||||
sleep 0.5
|
||||
done
|
||||
if kill -0 "${pid}" 2>/dev/null; then
|
||||
printf 'SIGTERM ignored, sending SIGKILL\n'
|
||||
kill -KILL -"${pid}" 2>/dev/null || kill -KILL "${pid}" 2>/dev/null
|
||||
fi
|
||||
rm -f "${PID_FILE}"
|
||||
printf 'stopped (pid=%s)\n' "${pid}"
|
||||
}
|
||||
|
||||
do_status() {
|
||||
local pid
|
||||
if pid="$(is_running)"; then
|
||||
printf 'status : running (pid=%s)\n' "${pid}"
|
||||
else
|
||||
printf 'status : stopped\n'
|
||||
fi
|
||||
printf 'log : %s\n' "${LOG_FILE}"
|
||||
[ -f "${LOG_FILE}" ] && printf 'size : %s\n' "$(du -h "${LOG_FILE}" | cut -f1)"
|
||||
printf 'cmds : %d, interval=%ds\n' "${#COMMANDS[@]}" "${INTERVAL_SEC}"
|
||||
}
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: ${SCRIPT_NAME} {start|stop|status|fg|tail|clear}
|
||||
|
||||
start Run detached in the background; the previous log is discarded first
|
||||
(see START_LOG_MODE)
|
||||
stop Stop the background instance
|
||||
status Show pid / log path / log size
|
||||
fg Run in the foreground, output to the terminal (for a quick check)
|
||||
tail tail -f the log file
|
||||
clear Remove the log file and all rotated copies (must be stopped first)
|
||||
|
||||
Log file : ${LOG_FILE}
|
||||
|
||||
To add a command, edit the COMMANDS array at the top of this script.
|
||||
EOF
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Entry point
|
||||
###############################################################################
|
||||
case "${1:-}" in
|
||||
start) do_start ;;
|
||||
stop) do_stop ;;
|
||||
status) do_status ;;
|
||||
fg) LOG_TO_FILE=0; main_loop ;;
|
||||
tail) tail -n 50 -f "${LOG_FILE}" ;;
|
||||
clear)
|
||||
is_running >/dev/null && die "still running, stop it first"
|
||||
rm -f "${LOG_FILE}" "${LOG_FILE}".[0-9]*
|
||||
printf 'log cleared\n'
|
||||
;;
|
||||
__daemon) do_daemon ;;
|
||||
-h|--help|"") usage ;;
|
||||
*) printf '[ERROR] unknown sub-command: %s\n\n' "$1" >&2; usage; exit 1 ;;
|
||||
esac
|
||||
@@ -0,0 +1,463 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# mgmt_ping_monitor.sh
|
||||
#
|
||||
# Version : V1.0.0
|
||||
# Author : ETWen
|
||||
# Date : 20260824
|
||||
#
|
||||
# Purpose : Exercise both management NICs and ping each one's own target,
|
||||
# appending everything to a rotating log. One round is
|
||||
#
|
||||
# 10G (eth0) : ip link up + address/route replace -> settle
|
||||
# -> ping N x TARGET_10G -> ip -s link show
|
||||
# 1G (eth1) : ip link up + address/route replace -> settle
|
||||
# -> ping N x TARGET_1G -> ip -s link show
|
||||
#
|
||||
# Designed to run detached in the background, same sub-commands as
|
||||
# bmc_monitor.sh: start/stop/status/fg/tail/clear.
|
||||
#
|
||||
# Notes : - Both NICs are brought up and LEFT up. Addresses and default
|
||||
# routes are applied with iproute2 `replace`, which is idempotent:
|
||||
# re-running a round does not stack duplicate addresses or routes.
|
||||
# - The default routes carry different metrics (100 / 200), so the
|
||||
# lower one wins for off-subnet traffic. They do NOT decide which
|
||||
# NIC a ping leaves by: `ping -I <iface>` binds to the device.
|
||||
# - !! SAME-SUBNET WARNING !! With both NICs up on one subnet the
|
||||
# target ARPs for the address and Linux may answer from either
|
||||
# NIC, so replies can arrive on the wrong one and the result is
|
||||
# not attributable. ARP_STRICT=1 sets arp_ignore/arp_announce to
|
||||
# stop that. Leave it on unless the two NICs are on separate
|
||||
# subnets, as in the iproute2 example this was modelled on.
|
||||
# - Each leg logs `ip -s link show <iface>` and the TX/RX packet
|
||||
# delta across the burst, which is the direct evidence of which
|
||||
# NIC actually carried the traffic.
|
||||
# - The first pings after bringing a link up are lost to ARP, not to
|
||||
# the link: a bench capture showed 4x "Destination Host
|
||||
# Unreachable" on the first burst and 0% loss on the very next
|
||||
# one. LINK_SETTLE_SEC absorbs that; it is not packet loss.
|
||||
#
|
||||
# Version History
|
||||
# V1.0.0 20260824 Initial Version
|
||||
# V2.0.0 20260824 Switch from ifconfig to iproute2 (ip link / address
|
||||
# replace / route replace with per-NIC metric). Both
|
||||
# NICs now stay up -- no link down between legs. Add
|
||||
# `ip -s link show` per leg with a TX/RX delta, and
|
||||
# ARP_STRICT for the same-subnet case.
|
||||
###############################################################################
|
||||
|
||||
set -u
|
||||
|
||||
###############################################################################
|
||||
# User Configurable Section -- normally this is the only part you need to edit
|
||||
###############################################################################
|
||||
|
||||
# --- 10G management port ---
|
||||
IF_10G="eth0"
|
||||
IP_10G="192.168.1.99"
|
||||
TARGET_10G="192.168.1.30" # host this NIC pings
|
||||
PLEN_10G=24 # prefix length, e.g. 24 = /24
|
||||
GW_10G="" # default gateway; empty = do not touch routing
|
||||
METRIC_10G=100 # lower metric wins for off-subnet traffic
|
||||
|
||||
# --- 1G management port ---
|
||||
IF_1G="eth1"
|
||||
IP_1G="192.168.1.101"
|
||||
TARGET_1G="192.168.1.31" # host this NIC pings
|
||||
PLEN_1G=24
|
||||
GW_1G=""
|
||||
METRIC_1G=200
|
||||
|
||||
# --- ping ---
|
||||
PING_COUNT=10 # echo requests per leg
|
||||
MAX_LOSS_PCT=0 # loss above this marks the leg FAIL
|
||||
|
||||
# Both NICs stay up. If they share a subnet the target's ARP can be answered
|
||||
# by either one, so a reply may arrive on the NIC that did not send. Setting
|
||||
# this applies arp_ignore=1 / arp_announce=2 to both, which keeps each NIC to
|
||||
# its own address. RAM only -- reverts on reboot. Set 0 for separate subnets.
|
||||
ARP_STRICT=1
|
||||
|
||||
# Prefix for the ip commands; empty because the daemon already runs as root.
|
||||
SUDO=""
|
||||
|
||||
# Seconds to wait after bringing a link up before pinging, so ARP is resolved
|
||||
# and the PHY has finished autoneg. The bench needed a few seconds on the 1G.
|
||||
LINK_SETTLE_SEC=5
|
||||
|
||||
# Seconds between rounds (0 = start the next round immediately)
|
||||
INTERVAL_SEC=10
|
||||
|
||||
# Per-ping-burst timeout in seconds; stops a wedged ping from freezing the
|
||||
# loop. 0 disables. Must exceed PING_COUNT (one packet per second).
|
||||
PING_TIMEOUT_SEC=60
|
||||
|
||||
# Stop after this many rounds. 0 = run until stopped manually.
|
||||
MAX_ROUNDS=0
|
||||
|
||||
# Logging
|
||||
LOG_DIR="" # empty = <script directory>/log
|
||||
LOG_NAME="mgmt_ping.log"
|
||||
MAX_LOG_SIZE_MB=100 # rotate above this size; 0 = never rotate
|
||||
MAX_LOG_KEEP=5 # keep .1 .. .N
|
||||
|
||||
# What `start` does with the log left behind by the previous run:
|
||||
# new = begin with an empty log; previous content and rotated copies are
|
||||
# discarded (default -- one run, one log)
|
||||
# archive = rename the previous log to <log>.YYYYmmdd-HHMMSS, then begin empty
|
||||
# append = keep appending to the existing log
|
||||
START_LOG_MODE="new"
|
||||
|
||||
###############################################################################
|
||||
# Internal
|
||||
###############################################################################
|
||||
SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")"
|
||||
SCRIPT_PATH="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)/${SCRIPT_NAME}"
|
||||
[ -n "${LOG_DIR}" ] || LOG_DIR="$(dirname -- "${SCRIPT_PATH}")/log"
|
||||
LOG_FILE="${LOG_DIR}/${LOG_NAME}"
|
||||
PID_FILE="${LOG_DIR}/${LOG_NAME%.log}.pid"
|
||||
|
||||
LOG_TO_FILE=0
|
||||
STOP=0
|
||||
SLEEP_PID=""
|
||||
|
||||
ts() { date '+%Y-%m-%d %H:%M:%S'; }
|
||||
die() { printf '[ERROR] %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
on_signal() {
|
||||
STOP=1
|
||||
[ -n "${SLEEP_PID}" ] && kill "${SLEEP_PID}" 2>/dev/null
|
||||
}
|
||||
|
||||
interruptible_sleep() {
|
||||
local sec="$1"
|
||||
[ "${sec}" -gt 0 ] 2>/dev/null || return 0
|
||||
sleep "${sec}" &
|
||||
SLEEP_PID=$!
|
||||
wait "${SLEEP_PID}" 2>/dev/null
|
||||
SLEEP_PID=""
|
||||
}
|
||||
|
||||
is_running() {
|
||||
local pid
|
||||
[ -f "${PID_FILE}" ] || return 1
|
||||
pid="$(cat "${PID_FILE}" 2>/dev/null)"
|
||||
[[ "${pid}" =~ ^[0-9]+$ ]] || return 1
|
||||
kill -0 "${pid}" 2>/dev/null || return 1
|
||||
grep -qa "${SCRIPT_NAME}" "/proc/${pid}/cmdline" 2>/dev/null || return 1
|
||||
printf '%s' "${pid}"
|
||||
return 0
|
||||
}
|
||||
|
||||
rotate_log_if_needed() {
|
||||
local max_bytes size i
|
||||
[ "${LOG_TO_FILE}" -eq 1 ] || return 0
|
||||
[ "${MAX_LOG_SIZE_MB}" -gt 0 ] || return 0
|
||||
[ -f "${LOG_FILE}" ] || return 0
|
||||
|
||||
max_bytes=$(( MAX_LOG_SIZE_MB * 1024 * 1024 ))
|
||||
size="$(stat -c %s "${LOG_FILE}" 2>/dev/null || echo 0)"
|
||||
[ "${size}" -ge "${max_bytes}" ] || return 0
|
||||
|
||||
for (( i = MAX_LOG_KEEP - 1; i >= 1; i-- )); do
|
||||
[ -f "${LOG_FILE}.${i}" ] && mv -f "${LOG_FILE}.${i}" "${LOG_FILE}.$(( i + 1 ))"
|
||||
done
|
||||
mv -f "${LOG_FILE}" "${LOG_FILE}.1"
|
||||
exec >>"${LOG_FILE}" 2>&1
|
||||
printf '[%s] INFO : log rotated at %s MB\n' "$(ts)" "${MAX_LOG_SIZE_MB}"
|
||||
}
|
||||
|
||||
remove_rotated_logs() {
|
||||
local i
|
||||
for (( i = 1; i <= MAX_LOG_KEEP + 1; i++ )); do
|
||||
rm -f "${LOG_FILE}.${i}"
|
||||
done
|
||||
}
|
||||
|
||||
prepare_log_on_start() {
|
||||
local stamp
|
||||
case "${START_LOG_MODE}" in
|
||||
append)
|
||||
return 0
|
||||
;;
|
||||
archive)
|
||||
if [ -s "${LOG_FILE}" ]; then
|
||||
stamp="$(date '+%Y%m%d-%H%M%S')"
|
||||
mv -f "${LOG_FILE}" "${LOG_FILE}.${stamp}" \
|
||||
|| die "cannot archive ${LOG_FILE}"
|
||||
printf 'previous log archived: %s.%s\n' "${LOG_FILE}" "${stamp}"
|
||||
fi
|
||||
remove_rotated_logs
|
||||
;;
|
||||
new)
|
||||
# Truncate rather than unlink: the inode is preserved, so a tail -f
|
||||
# that is already attached keeps following the new run.
|
||||
if [ -f "${LOG_FILE}" ]; then
|
||||
: > "${LOG_FILE}" || die "cannot truncate ${LOG_FILE}"
|
||||
fi
|
||||
remove_rotated_logs
|
||||
;;
|
||||
*)
|
||||
die "invalid START_LOG_MODE: ${START_LOG_MODE}"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Cumulative "<rx_packets> <tx_packets>" for an interface, from ip -s link.
|
||||
# The numbers sit on the line after each RX:/TX: header, and the column order
|
||||
# is "bytes packets errors ..." -- so packets is $2, not $1.
|
||||
if_counters() {
|
||||
${SUDO} ip -s link show "$1" 2>/dev/null | awk '
|
||||
/RX:/ { getline; rx = $2 }
|
||||
/TX:/ { getline; tx = $2 }
|
||||
END { printf "%s %s", (rx == "" ? 0 : rx), (tx == "" ? 0 : tx) }'
|
||||
}
|
||||
|
||||
# Bring the NIC up and (re)apply its address and default route. `replace` is
|
||||
# idempotent, so repeated rounds do not stack duplicates.
|
||||
setup_iface() {
|
||||
local iface="$1" ipaddr="$2" plen="$3" gw="$4" metric="$5"
|
||||
|
||||
${SUDO} ip link set "${iface}" up 2>&1 || return 1
|
||||
${SUDO} ip address replace "${ipaddr}/${plen}" dev "${iface}" 2>&1 || return 1
|
||||
if [ -n "${gw}" ]; then
|
||||
${SUDO} ip route replace default via "${gw}" dev "${iface}" \
|
||||
metric "${metric}" 2>&1 || return 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# One leg: configure <if>, settle, ping its own target, report. The NIC is
|
||||
# LEFT UP -- nothing is brought down between legs.
|
||||
# Usage: ping_leg <label> <if> <ip> <plen> <gw> <metric> <target>
|
||||
ping_leg() {
|
||||
local label="$1" iface="$2" ipaddr="$3" plen="$4" gw="$5" metric="$6" target="$7"
|
||||
local out rc tx rx loss avg verdict
|
||||
local c0 c1 rx0 tx0 rx1 tx1 drx dtx
|
||||
|
||||
printf -- '----- [%s] %s : %s (%s/%s) -> %s -----\n' \
|
||||
"$(ts)" "${label}" "${iface}" "${ipaddr}" "${plen}" "${target}"
|
||||
|
||||
if ! setup_iface "${iface}" "${ipaddr}" "${plen}" "${gw}" "${metric}"; then
|
||||
printf '[%s] FAIL : cannot configure %s\n\n' "$(ts)" "${iface}"
|
||||
return 1
|
||||
fi
|
||||
interruptible_sleep "${LINK_SETTLE_SEC}"
|
||||
[ "${STOP}" -eq 0 ] || return 1
|
||||
|
||||
c0="$(if_counters "${iface}")"; rx0="${c0% *}"; tx0="${c0#* }"
|
||||
|
||||
if [ "${PING_TIMEOUT_SEC}" -gt 0 ]; then
|
||||
out="$(timeout "${PING_TIMEOUT_SEC}" ping -I "${iface}" -c "${PING_COUNT}" "${target}" 2>&1)"
|
||||
else
|
||||
out="$(ping -I "${iface}" -c "${PING_COUNT}" "${target}" 2>&1)"
|
||||
fi
|
||||
rc=$?
|
||||
printf '%s\n' "${out}"
|
||||
|
||||
c1="$(if_counters "${iface}")"; rx1="${c1% *}"; tx1="${c1#* }"
|
||||
drx=$(( rx1 - rx0 )); dtx=$(( tx1 - tx0 ))
|
||||
|
||||
printf -- '--- ip -s link show %s ---\n' "${iface}"
|
||||
${SUDO} ip -s link show "${iface}" 2>&1
|
||||
|
||||
# "10 packets transmitted, 10 received, 0% packet loss"
|
||||
tx="$(printf '%s' "${out}" | sed -n 's/^\([0-9]\+\) packets transmitted.*/\1/p' | tail -1)"
|
||||
rx="$(printf '%s' "${out}" | sed -n 's/.*[, ]\([0-9]\+\) received.*/\1/p' | tail -1)"
|
||||
loss="$(printf '%s' "${out}" | sed -n 's/.*[, ]\([0-9]\+\)% packet loss.*/\1/p' | tail -1)"
|
||||
# "rtt min/avg/max/mdev = 1.044/1.743/2.387/0.437 ms"
|
||||
avg="$(printf '%s' "${out}" | sed -n 's|.*= [0-9.]*/\([0-9.]*\)/.*|\1|p' | tail -1)"
|
||||
|
||||
[ -n "${tx}" ] || tx=0
|
||||
[ -n "${rx}" ] || rx=0
|
||||
[ -n "${loss}" ] || loss=100
|
||||
[ -n "${avg}" ] || avg="-"
|
||||
|
||||
if [ "${rc}" -eq 124 ]; then
|
||||
verdict="FAIL(timeout)"
|
||||
elif [ "${loss}" -le "${MAX_LOSS_PCT}" ]; then
|
||||
verdict="PASS"
|
||||
else
|
||||
verdict="FAIL"
|
||||
fi
|
||||
|
||||
# one greppable line per leg: grep RESULT <log>
|
||||
# nic_tx/nic_rx are this interface's own counter delta across the burst --
|
||||
# if they stay near zero while the ping succeeded, the traffic went out a
|
||||
# different NIC and the result is not attributable to this one.
|
||||
printf '[%s] RESULT %-3s %-6s -> %-15s tx=%s rx=%s loss=%s%% rtt_avg=%sms nic_tx=%s nic_rx=%s %s\n\n' \
|
||||
"$(ts)" "${label}" "${iface}" "${target}" "${tx}" "${rx}" "${loss}" "${avg}" \
|
||||
"${dtx}" "${drx}" "${verdict}"
|
||||
|
||||
[ "${verdict}" = "PASS" ]
|
||||
}
|
||||
|
||||
# arp_ignore=1 : answer ARP only for addresses configured on the incoming NIC
|
||||
# arp_announce=2: pick the source address from the outgoing NIC's own subnet
|
||||
# Both are RAM-only sysctls and revert on reboot.
|
||||
apply_arp_strict() {
|
||||
local i
|
||||
[ "${ARP_STRICT}" -eq 1 ] || return 0
|
||||
for i in "${IF_10G}" "${IF_1G}"; do
|
||||
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_ignore=1" 2>/dev/null
|
||||
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_announce=2" 2>/dev/null
|
||||
done
|
||||
printf '[%s] INFO : arp_ignore=1 arp_announce=2 applied to %s,%s\n' \
|
||||
"$(ts)" "${IF_10G}" "${IF_1G}"
|
||||
}
|
||||
|
||||
main_loop() {
|
||||
local round=0 pass=0 fail=0
|
||||
|
||||
command -v ip >/dev/null 2>&1 || die "ip (iproute2) not found"
|
||||
command -v ping >/dev/null 2>&1 || die "ping not found"
|
||||
[ "${PING_COUNT}" -gt 0 ] 2>/dev/null || die "PING_COUNT must be > 0"
|
||||
|
||||
trap on_signal INT TERM
|
||||
|
||||
printf '#############################################################\n'
|
||||
printf '[%s] START: pid=%d count=%d settle=%ds interval=%ds\n' \
|
||||
"$(ts)" "$$" "${PING_COUNT}" "${LINK_SETTLE_SEC}" "${INTERVAL_SEC}"
|
||||
printf '[%s] 10G %s %s/%s -> %s (metric %s)\n' \
|
||||
"$(ts)" "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}"
|
||||
printf '[%s] 1G %s %s/%s -> %s (metric %s)\n' \
|
||||
"$(ts)" "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}"
|
||||
printf '#############################################################\n'
|
||||
|
||||
apply_arp_strict
|
||||
|
||||
while [ "${STOP}" -eq 0 ]; do
|
||||
round=$(( round + 1 ))
|
||||
printf -- '---------- ROUND %d @ %s ----------\n' "${round}" "$(ts)"
|
||||
|
||||
if ping_leg "10G" "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${GW_10G}" \
|
||||
"${METRIC_10G}" "${TARGET_10G}"; then
|
||||
pass=$(( pass + 1 )); else fail=$(( fail + 1 ))
|
||||
fi
|
||||
[ "${STOP}" -eq 0 ] || break
|
||||
|
||||
if ping_leg "1G" "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${GW_1G}" \
|
||||
"${METRIC_1G}" "${TARGET_1G}"; then
|
||||
pass=$(( pass + 1 )); else fail=$(( fail + 1 ))
|
||||
fi
|
||||
|
||||
rotate_log_if_needed
|
||||
|
||||
if [ "${MAX_ROUNDS}" -gt 0 ] && [ "${round}" -ge "${MAX_ROUNDS}" ]; then
|
||||
break
|
||||
fi
|
||||
[ "${STOP}" -eq 0 ] && interruptible_sleep "${INTERVAL_SEC}"
|
||||
done
|
||||
|
||||
printf '[%s] STOP : pid=%d after %d round(s), legs PASS=%d FAIL=%d\n' \
|
||||
"$(ts)" "$$" "${round}" "${pass}" "${fail}"
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Sub-commands
|
||||
###############################################################################
|
||||
do_start() {
|
||||
local pid
|
||||
pid="$(is_running)" && die "already running (pid=${pid})"
|
||||
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
|
||||
prepare_log_on_start
|
||||
|
||||
if command -v setsid >/dev/null 2>&1; then
|
||||
setsid "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 &
|
||||
else
|
||||
nohup "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 &
|
||||
disown 2>/dev/null
|
||||
fi
|
||||
|
||||
sleep 1
|
||||
pid="$(is_running)" || die "start failed, check ${LOG_FILE}"
|
||||
printf 'started (pid=%s)\nlog: %s\n' "${pid}" "${LOG_FILE}"
|
||||
}
|
||||
|
||||
do_daemon() {
|
||||
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
|
||||
LOG_TO_FILE=1
|
||||
exec >>"${LOG_FILE}" 2>&1
|
||||
printf '%d\n' "$$" > "${PID_FILE}"
|
||||
trap 'rm -f "${PID_FILE}"' EXIT
|
||||
main_loop
|
||||
}
|
||||
|
||||
do_stop() {
|
||||
local pid i
|
||||
pid="$(is_running)" || { printf 'not running\n'; rm -f "${PID_FILE}"; return 0; }
|
||||
|
||||
kill -TERM -"${pid}" 2>/dev/null || kill -TERM "${pid}" 2>/dev/null
|
||||
for (( i = 0; i < 20; i++ )); do
|
||||
kill -0 "${pid}" 2>/dev/null || break
|
||||
sleep 0.5
|
||||
done
|
||||
if kill -0 "${pid}" 2>/dev/null; then
|
||||
printf 'SIGTERM ignored, sending SIGKILL\n'
|
||||
kill -KILL -"${pid}" 2>/dev/null || kill -KILL "${pid}" 2>/dev/null
|
||||
fi
|
||||
rm -f "${PID_FILE}"
|
||||
printf 'stopped (pid=%s)\n' "${pid}"
|
||||
}
|
||||
|
||||
do_status() {
|
||||
local pid
|
||||
if pid="$(is_running)"; then
|
||||
printf 'status : running (pid=%s)\n' "${pid}"
|
||||
else
|
||||
printf 'status : stopped\n'
|
||||
fi
|
||||
printf 'log : %s\n' "${LOG_FILE}"
|
||||
[ -f "${LOG_FILE}" ] && printf 'size : %s\n' "$(du -h "${LOG_FILE}" | cut -f1)"
|
||||
printf 'ping : count=%d, interval=%ds, settle=%ds\n' "${PING_COUNT}" "${INTERVAL_SEC}" "${LINK_SETTLE_SEC}"
|
||||
printf '10G : %s %s/%s -> %s (metric %s)\n' "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}"
|
||||
printf '1G : %s %s/%s -> %s (metric %s)\n' "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}"
|
||||
}
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: ${SCRIPT_NAME} {start|stop|status|fg|tail|clear|summary}
|
||||
|
||||
start Run detached in the background; the previous log is discarded first
|
||||
(see START_LOG_MODE)
|
||||
stop Stop the background instance
|
||||
status Show pid / log path / configured NICs
|
||||
fg Run in the foreground, output to the terminal (for a quick check)
|
||||
tail tail -f the log file (does NOT return -- do not call from a macro)
|
||||
clear Remove the log file and all rotated copies (must be stopped first)
|
||||
summary Print just the per-leg RESULT lines from the log
|
||||
|
||||
Log file : ${LOG_FILE}
|
||||
|
||||
One round = 10G leg then 1G leg, each pinging its own target. Both NICs are
|
||||
brought up and left up; addresses and routes are applied with
|
||||
"ip address/route replace", so repeating a round does not stack duplicates.
|
||||
|
||||
If both NICs share a subnet, keep ARP_STRICT=1 -- otherwise a reply can come
|
||||
back on the NIC that did not send it. Check nic_tx/nic_rx on the RESULT line:
|
||||
they are that interface's own counter delta across the burst.
|
||||
|
||||
Edit the User Configurable Section at the top to change interfaces, addresses,
|
||||
targets, prefix lengths, gateways, metrics or the packet count.
|
||||
EOF
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Entry point
|
||||
###############################################################################
|
||||
case "${1:-}" in
|
||||
start) do_start ;;
|
||||
stop) do_stop ;;
|
||||
status) do_status ;;
|
||||
fg) LOG_TO_FILE=0; main_loop ;;
|
||||
tail) tail -n 50 -f "${LOG_FILE}" ;;
|
||||
summary) grep -a 'RESULT' "${LOG_FILE}" 2>/dev/null || printf 'no results in %s\n' "${LOG_FILE}" ;;
|
||||
clear)
|
||||
is_running >/dev/null && die "still running, stop it first"
|
||||
rm -f "${LOG_FILE}" "${LOG_FILE}".[0-9]*
|
||||
printf 'log cleared\n'
|
||||
;;
|
||||
__daemon) do_daemon ;;
|
||||
-h|--help|"") usage ;;
|
||||
*) printf '[ERROR] unknown sub-command: %s\n\n' "$1" >&2; usage; exit 1 ;;
|
||||
esac
|
||||
@@ -1,12 +1,22 @@
|
||||
;================================================================
|
||||
; User Configuration
|
||||
;================================================================
|
||||
strTestcase="Margin"
|
||||
project_name = "Blanton"
|
||||
strTestcase="ENV" ;ENV,EMC,Margin...etc
|
||||
|
||||
EN_Margin = 1 ; 1: Enable Margin Test, 0: Disable Margin Test
|
||||
|
||||
EN_Margin = 0 ; 1: Enable Margin Test, 0: Disable Margin Test
|
||||
EN_log = 1 ; 1: Enable log, 0: Disable log
|
||||
|
||||
;================================================================
|
||||
; FAN_SPEED
|
||||
;================================================================
|
||||
FAN_SPEED = 30
|
||||
;================================================================
|
||||
; Switch Unit Population
|
||||
;================================================================
|
||||
SWB_UNIT0 = 1 ; 1 = this DUT has switch unit 0, wait for it ; 0 = skip
|
||||
SWB_UNIT1 = 1 ; 1 = this DUT has switch unit 1, wait for it ; 0 = skip
|
||||
|
||||
;================================================================
|
||||
; Prompt Definitions
|
||||
;================================================================
|
||||
|
||||
@@ -16,3 +16,8 @@ wait prompt_sonic_root
|
||||
sendln "show platform leak status"
|
||||
wait prompt_sonic_root
|
||||
sendln "show platform leak channels"
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln "tpm-dut-test fru"
|
||||
wait prompt_sonic_root
|
||||
sendln "tpm-dut-test tpm-read"
|
||||
@@ -1,8 +1,11 @@
|
||||
; All Event
|
||||
wait prompt_sonic_root
|
||||
sendln "date"
|
||||
sendln "dmesg -T | grep -Ei 'error|fail(ed|ure)?|fault|critical|panic|timeout'"
|
||||
|
||||
; i2c event
|
||||
wait prompt_sonic_root
|
||||
sendln "dmesg | grep -i error"
|
||||
sendln "dmesg -T | grep -Ei 'error|fail(ed|ure)?|fault|critical|panic|timeout' | grep -i i2c"
|
||||
|
||||
; Clear Event
|
||||
wait prompt_sonic_root
|
||||
sendln "dmesg | grep -i fail"
|
||||
wait prompt_sonic_root
|
||||
sendln "dmesg | grep -i warning"
|
||||
sendln "dmesg -C"
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
; =============================================================================
|
||||
; File : utils/wait_init.ttl
|
||||
; Version : V3.0.0
|
||||
; Date : 2026-08-21
|
||||
; Author : ETWen
|
||||
; =============================================================================
|
||||
; Wait until every switch unit ENABLED below reports at least WT_MIN ports up:
|
||||
; bcmcmd -n <u> -c ps | grep -w up | wc -l
|
||||
;
|
||||
; The DUT is not always fully populated. Which units exist is declared ONCE in
|
||||
; config.ttl, so the traffic blocks in Script A/B/C and this wait agree:
|
||||
; both units -> SWB_UNIT0 = 1 , SWB_UNIT1 = 1
|
||||
; unit 0 only -> SWB_UNIT0 = 1 , SWB_UNIT1 = 0
|
||||
; unit 1 only -> SWB_UNIT0 = 0 , SWB_UNIT1 = 1
|
||||
; no unit -> SWB_UNIT0 = 0 , SWB_UNIT1 = 0 (bypass, returns immediately)
|
||||
;
|
||||
; Enter : prompt of the previous command is NOT consumed.
|
||||
; Exit : a command has been sent, prompt NOT consumed (caller does the wait).
|
||||
;
|
||||
; Why the "PORTS0=" marker: TTL can only test for a string, so reading a count
|
||||
; means capturing it. Wrapping the number in an echo gives waitregex a unique
|
||||
; anchor, and the ECHO of the command cannot false-match -- it reads
|
||||
; "PORTS0=$(bcmcmd ..." and the pattern requires a digit right after the "=".
|
||||
;
|
||||
; Version History:
|
||||
; V1.0.0 2026-08-21 Initial Version (waited on "show platform temperature"
|
||||
; until it stopped reporting "Thermal Not detected")
|
||||
; V2.0.0 2026-08-21 Wait on the bcmcmd port-up count of BOTH switch units
|
||||
; instead of the thermal sensors; proceed only when both
|
||||
; are greater than WT_MIN (216).
|
||||
; V3.0.0 2026-08-21 Add WT_UNIT0 / WT_UNIT1 so a partly populated DUT can
|
||||
; be declared: wait on the enabled units only, and bypass
|
||||
; entirely when neither is set.
|
||||
; V3.1.0 2026-08-21 Take the population from SWB_UNIT0 / SWB_UNIT1 in
|
||||
; config.ttl instead of local copies, so this wait and the
|
||||
; traffic blocks cannot drift apart.
|
||||
; V3.1.1 2026-08-25 WT_INTERVAL 10 -> 60. Fix the WT_MIN comments: the test
|
||||
; has been `< WT_MIN` (at least) since the threshold was
|
||||
; corrected, but the text still said "more than", which is
|
||||
; the off-by-one that once hung Script A.
|
||||
; =============================================================================
|
||||
; ---- which units to wait for: SWB_UNIT0 / SWB_UNIT1, set in config.ttl ------
|
||||
WT_MIN = 216 ; an enabled unit must report AT LEAST this many ports up
|
||||
; (216 = 108 loopback pairs x 2, i.e. every cabled port)
|
||||
WT_INTERVAL = 60 ; seconds between polls
|
||||
; -----------------------------------------------------------------------------
|
||||
|
||||
timeout = 60 ; per-wait cap; bcmcmd ps is not instant
|
||||
wait prompt_sonic_root
|
||||
|
||||
; nothing declared -> bypass
|
||||
if SWB_UNIT0 = 0 then
|
||||
if SWB_UNIT1 = 0 then
|
||||
goto NO_UNITS
|
||||
endif
|
||||
endif
|
||||
|
||||
:WAIT_PORTS
|
||||
wt_ok = 1
|
||||
|
||||
if SWB_UNIT0 = 1 then
|
||||
wt_up0 = 0
|
||||
sendln "echo PORTS0=$(bcmcmd -n 0 -c ps | grep -w up | wc -l)"
|
||||
waitregex "PORTS0=[ ]*([0-9]+)"
|
||||
if result = 1 then
|
||||
str2int wt_up0 groupmatchstr1
|
||||
endif
|
||||
wait prompt_sonic_root
|
||||
if wt_up0 < WT_MIN then
|
||||
wt_ok = 0
|
||||
endif
|
||||
endif
|
||||
|
||||
if SWB_UNIT1 = 1 then
|
||||
wt_up1 = 0
|
||||
sendln "echo PORTS1=$(bcmcmd -n 1 -c ps | grep -w up | wc -l)"
|
||||
waitregex "PORTS1=[ ]*([0-9]+)"
|
||||
if result = 1 then
|
||||
str2int wt_up1 groupmatchstr1
|
||||
endif
|
||||
wait prompt_sonic_root
|
||||
if wt_up1 < WT_MIN then
|
||||
wt_ok = 0
|
||||
endif
|
||||
endif
|
||||
|
||||
if wt_ok = 1 then
|
||||
goto PORTS_OK
|
||||
endif
|
||||
|
||||
pause WT_INTERVAL
|
||||
goto WAIT_PORTS
|
||||
|
||||
; ---- exits ------------------------------------------------------------------
|
||||
; Both leave exactly one prompt unconsumed, so the caller's next
|
||||
; "wait prompt_sonic_root" has something to match.
|
||||
:NO_UNITS
|
||||
sendln "echo wait_init:no-switch-unit-declared,bypassing-port-wait"
|
||||
goto DONE
|
||||
|
||||
:PORTS_OK
|
||||
sendln ""
|
||||
|
||||
:DONE
|
||||
timeout = 0
|
||||
Reference in New Issue
Block a user