8 Commits
Author SHA1 Message Date
etwenandClaude Opus 5 15ea776b82 feat(soak): Show the background jobs each round, and stop the loop spinning
Script B's monitoring loop had no pause at all - each round started the
instant the last one ended, so an overnight soak filled the log with
near-identical samples taken seconds apart. The section was even labelled
"Get data every 10mins" while running with no delay whatsoever.

The loop now ends each round with pause 60 and prints both job lists:
bgctl list for the platform job runner, jobs for anything the macro
backgrounded in that shell. A stress job that dies mid-soak shows up as a
shorter list on the next round instead of going unnoticed until the end.

60s is still hard-coded and the Status sheet asks for 10 minutes, so the
Phase 3 item stays open - this makes the loop sane, it does not close it.

Also adds docs/LTC2980_channel_map.csv: the nine settings/*.conf flattened
into one Board,CONN,Ch,NetName,Vnom sheet, generated by
tools/gen_channel_map.sh. The CSV is output, not source. NC channels are
kept as NC/- because the channel number is also the PMBus page - dropping
the gaps would shift every channel after them.

Two things the merged table makes visible: SWB0 and SWB1 carry identical
net names and voltages (only the bus and CB_I2C_CH differ), so those are
two copies that can drift apart silently; and there are exactly 8 NC
channels, all on CONN14/CONN15.

tools/100G_PRBS.txt, TR518.txt and fan_ctrl.txt join the existing
traffic_loopback_*.txt references. TR518 is not scripted yet and cannot
share the hardware with blanton_traffic_linespeed - both drive port
loopback and L2 learning.

Script A -> V1.0.10.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-27 14:29:47 +08:00
etwenandClaude Opus 5 03d609cc1f feat(prbs): Add 100G PRBS monitor; fix the job log path and USB mount
port_prbs_monitor.sh drives PRBS on both 100G uplinks: lpmode off, arm
the pattern, then poll `phy diag <phy> prbs get` in the background until
stopped, counting a poll as PASS only when the output says "PRBS OK!".
prbsstat Ber is captured beside every poll for the record but does not
decide the verdict. `stop` runs prbsstat STOp and prbs clear before
rendering the report, so the test does not leave PRBS armed.

`start a` / `start b` runs one uplink alone, which is how a genuine port
fault is separated from the DUT not coping with two PRBS streams at
once. A port that was not run reports SKIP rather than FAIL -- deciding
on poll count alone would have made a deliberate single-port run look
like half the hardware was broken.

The port status is deliberately not printed at `start`: with PRBS armed
the link reports DOWN, which reads as a failure to anyone glancing at
the console. It appears in the report instead, after prbs clear and
labelled as the recovered state.

The calls in Script A, B and C are committed but commented out -- PRBS
is still under bring-up.

Fix: the job directory was written as /host/hw-eval/current/jobs in some
places and /host/hw-eval/jobs in others, so what Script B wrote was not
what Script C collected. Unified on /host/hw-eval/jobs.

Fix: Script C copied to /mnt/usb assuming it was mounted. It now finds
the device with usb_target.sh and mounts it first, so the logs actually
leave the DUT.

Script C also reorders its results -- stress logs, then the MGMT ping
report, then 100G status -- and cats the bgctl ssd/usb logs.

Script A -> V1.0.9. Docs and docs/release-notes/v1.0.9.md updated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-26 15:35:48 +08:00
etwenandClaude Opus 5 5e1e4a6a8f docs(release): Add v1.0.8 notes; wire mgmt ping status into B and C
Script A: the baseline ping window is 30 s rather than 45, and the log
is cleared afterwards so Script B starts from nothing.

Script B: `status` after `start`, and Script C: `status` before `stop`
plus a `cat` of the report. The status line shows both pids and the
replies received per NIC, so a dead link is visible while the soak is
still running instead of only at the end.

docs/release-notes/v1.0.8.md written in the house format.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-26 10:13:34 +08:00
etwenandClaude Opus 5 a31c186142 feat(mgmt): Ping both NICs at once, and stop counting ARP as loss
The bench run reported FAIL=3 with no NIC errors and no drops. All three
failures were on eth1 and all three were missing exactly icmp_seq=1,
with the other nine replies present and sub-millisecond. That is
neighbour resolution: once the ARP entry for the target expires, the
first echo request is spent resolving it and ping counts it as loss.
Rounds sat about 24 s apart, right on the edge of the default reachable
time, so it happened on some rounds and not others.

mgmt_ping_monitor.sh V3.0.0:

- One discarded ping per NIC before the measured run resolves ARP, so
  MAX_LOSS_PCT can stay at 0 and a FAIL means a real drop. Raising the
  threshold instead would have hidden genuine single-packet loss, and a
  longer settle would not help -- nothing triggers ARP until the ping.
- Both NICs now ping at the same time and keep accumulating until
  stopped, rather than alternating fixed bursts. Each writes its own raw
  capture.
- `stop` sends SIGINT rather than SIGTERM, because ping prints its
  statistics block on interrupt and that block is what the report
  parses, then renders into the log: the last TAIL_LINES entries per
  NIC, each one's statistics and verdict, and `ip -s link show` for both
  interfaces.
- `-D -O` are on by default so every line carries a timestamp and a
  request that got no reply is visible in the tail instead of merely
  absent. Clear PING_EXTRA_OPTS if the platform's ping rejects them.

Since the script no longer re-execs itself to daemonise, `bash
mgmt_ping_monitor.sh start` works without the execute bit.

.gitignore: the new *.raw / *.pid / *.state runtime files were outside
the *.log rule and would have been committed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-26 09:53:20 +08:00
etwenandClaude Opus 5 1b8d5a8e91 feat(usb): Detect the USB target instead of assuming /dev/sda1
New Blanton_Script/usb_target.sh reports a USB mass-storage device's
node, mount point or by-id path. It excludes whatever disk backs / and
/host -- this platform can boot from a USB DOM, which also reports
TRAN=usb -- refuses to guess when several USB disks are present, falls
back to the whole device for superfloppy media, and when mounting,
checks the result is actually writable rather than trusting that mount
succeeded.

Script B asks it for the device node and builds the stress command from
the answer, so a stick that enumerates as sdb no longer sends the write
test at whatever /dev/sda1 happens to be.

The device is carried back through an echo marker, split in the shell
literal as "<<D""EV=..." so the command echo cannot satisfy the pattern
the macro waits for -- the same false-match guard used for PORTS0= and
NCMIF=, arranged differently.

Script B: DDR memtester drops its 100-pass limit and runs until stopped,
matching the other soak loads.

Fix: Script C ran `bgctl reset --yes` while killing processes, which is
before the job logs are copied to USB. Moved to after the archive and
after hw-test-session finish, so the run's own logs are collected before
anything clears them.

Script A -> V1.0.8 history updated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-25 17:03:36 +08:00
etwenandClaude Opus 5 cac37583e7 feat(ttl): Include mgmt_ping.log in the USB archive
Script C now copies log/mgmt_ping.log into
/host/hw-eval/current/jobs/ alongside bmc_poll.log, so the management
link results leave the DUT with the run instead of staying behind. Both
monitors' logs are now covered.

Script A history: the two copies are one action in one block, so they
share a single V1.0.8 line rather than reading as separate changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-25 15:57:23 +08:00
etwenandClaude Opus 5 802ee4d467 feat(ttl): Fold bmc_poll.log into the USB archive
Script C copies log/bmc_poll.log into /host/hw-eval/current/jobs/ before
that directory is archived to USB, so the BMC sampling and I2C
read-back results leave the DUT with the run they belong to rather than
only existing in the Tera Term capture.

The wait moves ahead of the getdate/sprintf2 block; those build strings
without touching the terminal, so the prompt is still consumed exactly
once before the copy is sent.

Script A history updated under the existing V1.0.8 block -- that version
is not tagged yet, so this is the same batch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-25 15:55:05 +08:00
etwenandClaude Opus 5 166b53983f feat(bmc): Re-enable bmc_monitor with an I2C write/read-back test
bmc_monitor.sh comes back into the run with a second job beyond the
`free -m` sampling: an I2C pattern test against bus 1, address 0x41,
offset 0x00C0 -- write 55AA55AA and read it back, then write AA55AA55
and read that back. Two complementary patterns catch a stuck bit either
way round, and repeating it through the soak turns an intermittent I2C
fault into something the log shows rather than something the tester has
to catch live.

Script B starts it alongside the other stress; Script C stops it and
cats log/bmc_poll.log.

bmc_monitor_ddr.sh stays disabled -- BMC DDR stress runs through
bgctl + bmc-manager.

Docs: the "disabled" notes I added last time now applied to only half
the pair and read as wrong for bmc_monitor.sh. Split so the monitor is
documented as live with its I2C test, and only the _ddr variant is
marked disabled.

Script A -> V1.0.8.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-25 15:50:56 +08:00
18 changed files with 1648 additions and 371 deletions
+5
View File
@@ -12,6 +12,11 @@ For_AI/
!**/logs/.gitkeep !**/logs/.gitkeep
!**/Logs/.gitkeep !**/Logs/.gitkeep
# monitor 腳本的執行期產物(原始擷取、pid、狀態)
*.raw
*.pid
*.state
# 打包輸出:產出不進 git,打包工具本身要進 # 打包輸出:產出不進 git,打包工具本身要進
publish/* publish/*
!publish/.gitkeep !publish/.gitkeep
+53 -17
View File
@@ -40,8 +40,9 @@ LTC2980 margin 控制器。TTL 只負責「按順序 `sendln` 並把畫面收進
| BMC 存取 | host 端 `bmc-manager run "<cmd>"`(不需對 BMC 開 SSH,繞開其 busybox 工具鏈);`bmc-first-enroll` / `bmc-manager version\|status` | | BMC 存取 | host 端 `bmc-manager run "<cmd>"`(不需對 BMC 開 SSH,繞開其 busybox 工具鏈);`bmc-first-enroll` / `bmc-manager version\|status` |
| 背景工作管理 | `bgctl run/list/stop --all/reset`(平台工具);長時間 ping 用 `systemd-run --unit=` 掛成 transient unit | | 背景工作管理 | `bgctl run/list/stop --all/reset`(平台工具);長時間 ping 用 `systemd-run --unit=` 掛成 transient unit |
| 管理網路 | iproute2`ip link set` / `ip address replace` / `ip route replace ... metric`+ `ping -I` | | 管理網路 | iproute2`ip link set` / `ip address replace` / `ip route replace ... metric`+ `ping -I` |
| 測試 session | `hw-test-session start\|log\|status\|finish`job log 收在 `/host/hw-eval/current/jobs/` | | 測試 session | `hw-test-session start\|log\|status\|finish`job log 收在 `/host/hw-eval/jobs/` |
| 風扇控制 | max31790 driver rebind`/sys/bus/i2c/drivers/max31790/{bind,unbind}`+ `fan-speed-control.sh <%>` | | 風扇控制 | max31790 driver rebind`/sys/bus/i2c/drivers/max31790/{bind,unbind}`+ `fan-speed-control.sh <%>` |
| 100G PRBS | `bcmcmd -n <unit> -c 'dsh -c "phy diag <phy> prbs …"'`set / prbsstat STArt / get / Ber / STOp / clear |
| 壓力測試 | `mlucas-avx2`AMD CPU)、`bgctl run` 排程的 `memtester` / `qfx5252-stress-ssd` / `qfx5252-stress-usb` | | 壓力測試 | `mlucas-avx2`AMD CPU)、`bgctl run` 排程的 `memtester` / `qfx5252-stress-ssd` / `qfx5252-stress-usb` |
| 資料面流量 | `bcmcmd`Broadcom drivshellVLAN loopback + `tx` burstswitch unit 0 / 1 | | 資料面流量 | `bcmcmd`Broadcom drivshellVLAN loopback + `tx` burstswitch unit 0 / 1 |
| 流量報表 | DUT 上 awk`blanton_traffic_linespeed report`);離線 `python3 tools/bcm_mibpair_report_V1.1.0.py` | | 流量報表 | DUT 上 awk`blanton_traffic_linespeed report`);離線 `python3 tools/bcm_mibpair_report_V1.1.0.py` |
@@ -119,6 +120,7 @@ Blanton_TTL_Script/
│ ├── Blantons_FPGA_Registers_draft.docx # 同上,客戶原始 docx │ ├── Blantons_FPGA_Registers_draft.docx # 同上,客戶原始 docx
│ ├── Blantons_FPGA_Registers_Map_draft.xlsx │ ├── Blantons_FPGA_Registers_Map_draft.xlsx
│ ├── Blantons_PCIe_AER_and_DDR_EDAC_checks.pdf # AER / EDAC 檢查點依據 │ ├── Blantons_PCIe_AER_and_DDR_EDAC_checks.pdf # AER / EDAC 檢查點依據
│ ├── LTC2980_channel_map.csv # 9 個 settings/*.conf 合併:Board,CONN,Ch,NetName,Vnom144 列)
│ ├── TTL_Script_Blanton_Status_20260814.xlsx │ ├── TTL_Script_Blanton_Status_20260814.xlsx
│ └── TTL_Script_Blanton_Status_20260817.xlsx # ⭐ 最新:三支腳本逐項 StatusOK / On-Going + Owner │ └── TTL_Script_Blanton_Status_20260817.xlsx # ⭐ 最新:三支腳本逐項 StatusOK / On-Going + Owner
@@ -126,9 +128,13 @@ Blanton_TTL_Script/
│ ├── bcm_mibpair_report_V1.1.0.py # 離線解析 drivshell log → per-pair TX/RX + PASS/FAIL │ ├── bcm_mibpair_report_V1.1.0.py # 離線解析 drivshell log → per-pair TX/RX + PASS/FAIL
│ ├── traffic_loopback_vlan_setting.txt # bcmcmd VLAN 30..137 建立 + 成對 pbmcdN <-> cdN+32 │ ├── traffic_loopback_vlan_setting.txt # bcmcmd VLAN 30..137 建立 + 成對 pbmcdN <-> cdN+32
│ ├── traffic_loopback_start.txt # bcmcmd tx 100 length=512 VLantag=<vid> │ ├── traffic_loopback_start.txt # bcmcmd tx 100 length=512 VLantag=<vid>
── traffic_loopback_vlan_remove.txt # bcmcmd vlan remove <vid> pbm=... ── traffic_loopback_vlan_remove.txt # bcmcmd vlan remove <vid> pbm=...
# ↑ 三份為手動貼上用的原始指令表, # ↑ 三份為手動貼上用的原始指令表,
# blanton_traffic_linespeed.sh 是其 bash 版 # blanton_traffic_linespeed.sh 是其 bash 版
│ ├── 100G_PRBS.txt # 100G PRBS 原始指令(port_prbs_monitor.sh 的來源)
│ ├── TR518.txt # 內建封包測試 tr 518Scenario=6 Profile=2)— 尚未腳本化
│ ├── fan_ctrl.txt # MAX31790 unbind/bind + fan-speed-control.sh 30
│ └── gen_channel_map.sh # settings/*.conf → docs/LTC2980_channel_map.csv(需 gawk
├── secret/ # 🚫 gitignored — DUT 密碼、per-unit BDF、COM 設定 ├── secret/ # 🚫 gitignored — DUT 密碼、per-unit BDF、COM 設定
│ ├── README.md # ✅ committed — 用途索引 │ ├── README.md # ✅ committed — 用途索引
@@ -177,9 +183,12 @@ Blanton_TTL_Script/
├── blanton_traffic_linespeed.sh # SWB loopback 線速流量(bcmcmd unit 0/1 ├── blanton_traffic_linespeed.sh # SWB loopback 線速流量(bcmcmd unit 0/1
│ # init/clear/show/start/stop/ps/report/run │ # init/clear/show/start/stop/ps/report/run
├── mgmt_ping_monitor.sh # 10G/1G 管理網路 ping 監控(iproute2 ├── mgmt_ping_monitor.sh # 10G/1G 管理網路 ping 監控(iproute2
│ # eth0→TARGET_10G、eth1→TARGET_1G,兩張卡都保持 up │ # eth0→TARGET_10G、eth1→TARGET_1G,兩張卡同時持續 ping
├── bmc_monitor.sh # ⚠️ 已停用:B/C 內呼叫處已註解,改走 bgctl + bmc-manager ├── port_prbs_monitor.sh # 100G uplink PRBS 測試(bcmcmd phy diag
├── bmc_monitor_ddr.sh # ⚠️ 已停用,同上 # ⚠️ 已納入交付,但 A/B/C 的呼叫仍註解掉(bring-up 中)
├── usb_target.sh # 偵測 USB 裝置節點/掛載點(排除系統碟)
├── bmc_monitor.sh # BMC 監控:free -m + I2C 寫入/讀回圖樣測試
├── bmc_monitor_ddr.sh # ⚠️ 已停用:B/C 內呼叫處已註解,DDR 壓力改走 bgctl + bmc-manager
│ # ↑ 三支都要 chmod +x 才能跑(見 Key Constraints │ # ↑ 三支都要 chmod +x 才能跑(見 Key Constraints
└── LTC2980_Margin_Script/ # 電壓 margin 子專案(另有獨立 repo └── LTC2980_Margin_Script/ # 電壓 margin 子專案(另有獨立 repo
├── margin.sh # margin_init/status/set/apply_profile/save ├── margin.sh # margin_init/status/set/apply_profile/save
@@ -242,6 +251,16 @@ Channel 定址:`ch / 8` → `CHIPS[]` index`ch % 8` → LTC2977 PMBus PAGE
> 同一顆 LTC2980 的兩個位址**相差 2**7-bit,由規格書 8-bit 配對除以 2 而來),不是 1。 > 同一顆 LTC2980 的兩個位址**相差 2**7-bit,由規格書 8-bit 配對除以 2 而來),不是 1。
> 兩張板共用一條通道,所以同通道上的位址必須互斥 —— 上表兩組(`0x5C/0x5E` 與 `0x62/0x64`)不重疊。 > 兩張板共用一條通道,所以同通道上的位址必須互斥 —— 上表兩組(`0x5C/0x5E` 與 `0x62/0x64`)不重疊。
**全通道 net / 電壓對照:`docs/LTC2980_channel_map.csv`**9 個 `.conf` × 16ch = 144 列,
欄位 `Board,CONN,Ch,NetName,Vnom`)。查「某個 net 在哪片哪個 channel」比翻 9 個 `.conf` 快。
CSV 是產出物,不是正本 —— 改完 `.conf``./tools/gen_channel_map.sh` 重產,別手改 CSV。
兩件從表上看得出來、改 `.conf` 時要記得的事:
- **SWB0 與 SWB1 的 net / 電壓完全相同**,差別只在 `CHIPS[]` 的 bus`0:` vs `1:`)與 `CB_I2C_CH`
也就是說有兩份會各自漂移的重複資料 —— 改了 SWB0 沒改 SWB1,不會有任何東西擋你。
- **8 個 NC 通道**SWB0/1 CONN14 的 CH10/CH11/CH13、CONN15 的 CH10)在 CSV 裡以
`NC` / `-` 原樣保留,不是漏掉;濾掉它們會讓 page 編號對不上。
### 3. `profiles/*.conf` — 16-ch 批次 margin 組合 ### 3. `profiles/*.conf` — 16-ch 批次 margin 組合
```bash ```bash
@@ -312,7 +331,10 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc
| **電源資料** | `pwr_data` / `pwr_data_pdb` / `pwr_pdb_id` | SWB VRM 全軌 Vin/Vout/Iout/Temp 對齊表;MP29816 解析度**執行期從 `MFR_VOUT_SCALE_LOOP` 讀**,不寫死 | | **電源資料** | `pwr_data` / `pwr_data_pdb` / `pwr_pdb_id` | SWB VRM 全軌 Vin/Vout/Iout/Temp 對齊表;MP29816 解析度**執行期從 `MFR_VOUT_SCALE_LOOP` 讀**,不寫死 |
| **電壓 Margin** | `margin_init/status/set/apply_profile/save` + `*_all.sh` | 9 顆 LTC2980CB CONN13、SWB0/1 CONN13~16)批次 high/low margin 與狀態掃描。v2.6.0 起 SWB 路徑真正接上共用 cb_i2c 後端,並有 `margin_swb_info/probe/scan/reset/sem` 除錯包裝 | | **電壓 Margin** | `margin_init/status/set/apply_profile/save` + `*_all.sh` | 9 顆 LTC2980CB CONN13、SWB0/1 CONN13~16)批次 high/low margin 與狀態掃描。v2.6.0 起 SWB 路徑真正接上共用 cb_i2c 後端,並有 `margin_swb_info/probe/scan/reset/sem` 除錯包裝 |
| **管理網路 ping** | `mgmt_ping_monitor.sh {start\|stop\|status\|fg\|tail\|clear\|summary}` | 一輪 = 10G leg + 1G leg,各自 `ip address/route replace``ping -I <if>` 打自己的對端;兩張卡都保持 up。每 leg 記 `ip -s link show` 與該卡的 TX/RX packet delta,另出一行可 grep 的 `RESULT` | | **管理網路 ping** | `mgmt_ping_monitor.sh {start\|stop\|status\|fg\|tail\|clear\|summary}` | 一輪 = 10G leg + 1G leg,各自 `ip address/route replace``ping -I <if>` 打自己的對端;兩張卡都保持 up。每 leg 記 `ip -s link show` 與該卡的 TX/RX packet delta,另出一行可 grep 的 `RESULT` |
| **BMC 監控 / 壓力** | `bmc_monitor.sh` / `bmc_monitor_ddr.sh` | ⚠️ **已停用** —— B/C 內呼叫處已註解,改由 `bgctl` + `bmc-manager run memtester` 取代。檔案保留未刪 | | **BMC 監控** | `bmc_monitor.sh {start\|stop\|status\|fg\|tail\|clear}` | 由 host 端 `bmc-manager run` 週期取樣 `free -m`,另加一組 **I2C 寫入/讀回圖樣測試**bus 1、addr 0x41、offset 0x00C0,寫 `55AA55AA` 讀回、再寫 `AA55AA55` 讀回)。B 啟動、C 停止並 `cat log/bmc_poll.log` |
| **BMC DDR 壓力** | `bmc_monitor_ddr.sh` | ⚠️ **已停用** —— B/C 內呼叫處已註解,改由 `bgctl run bmc-manager run memtester` 取代。檔案保留未刪 |
| **100G PRBS** | `port_prbs_monitor.sh {start [both\|a\|b]\|stop\|status\|report\|clear}` | 兩個 uplink 各一個 worker,可同時或單獨跑。setup`lpmode off``prbs set``prbsstat STArt`)→ 週期 `prbs get`(判定)+ `prbsstat Ber`(只記錄)→ teardown`STOp` + `prbs clear`)。輸出含 `PRBS OK!` 才算 PASS。**⚠️ A/B/C 的呼叫目前註解掉** |
| **USB 目標偵測** | `usb_target.sh [-o dev\|mnt\|both\|id] [-n] [-r sec]` | 找出插入的 USB 儲存裝置,排除 `/``/host` 的底層碟(本平台可能從 USB DOM 開機),多顆時拒絕猜;掛載時會驗證真的可寫 |
| **打包交付** | `publish/publish.sh [-v Ver] [-f] [-z] [-n]` | 版號讀 Script A 檔頭,複製成 `publish/Script_ABC_Blanton_<Ver>/`,排除 `*.log`,可另出 zip | | **打包交付** | `publish/publish.sh [-v Ver] [-f] [-z] [-n]` | 版號讀 Script A 檔頭,複製成 `publish/Script_ABC_Blanton_<Ver>/`,排除 `*.log`,可另出 zip |
--- ---
@@ -329,7 +351,7 @@ Script A ── logopen Logs/Blanton_Margin_<ts>.log ──┐ (之後所有畫
│ │ │ │
├─ 登入 admin → sudo -i → root@sonic:~# │ ├─ 登入 admin → sudo -i → root@sonic:~# │
├─ date -s "<host 時間>" ← 讓 DUT 時戳可對照 ├─ date -s "<host 時間>" ← 讓 DUT 時戳可對照
├─ rm /host/hw-eval/current/jobs/* ← 清掉上一輪的 job log ├─ rm /host/hw-eval/jobs/* ← 清掉上一輪的 job log
├─ hw-test-session start / log / status ├─ hw-test-session start / log / status
├─ 風扇:max31790 unbind→sleep→bind18-0020 與 25-0020 各一次) ├─ 風扇:max31790 unbind→sleep→bind18-0020 與 25-0020 各一次)
│ → fan-speed-control.sh <FAN_SPEED> → 回答 y 確認 │ │ → fan-speed-control.sh <FAN_SPEED> → 回答 y 確認 │
@@ -373,7 +395,8 @@ Script C
├─ nvme smart-log / smartctl -x /dev/nvme0 ├─ nvme smart-log / smartctl -x /dev/nvme0
├─ traffic: stop → report ← per-pair TX/RX 交叉比對,PASS/FAIL/NAexit 2 = 有 FAIL/NA ├─ traffic: stop → report ← per-pair TX/RX 交叉比對,PASS/FAIL/NAexit 2 = 有 FAIL/NA
├─ show reboot-cause / show uptime ← soak 期間有沒有意外重開,一眼看得到 ├─ show reboot-cause / show uptime ← soak 期間有沒有意外重開,一眼看得到
├─ cp -r /host/hw-eval/current/jobs/ /mnt/usb/jobs-<YYYYmmdd>_<HHMM> ├─ usb_target.sh -o dev → sync → mount <dev> /mnt/usb
├─ cp bmc_poll.log / mgmt_ping.log → jobs/,再 cp -r jobs/ /mnt/usb/jobs-<YYYYmmdd>_<HHMM>
└─ hw-test-session finish └─ hw-test-session finish
[messagebox "GOOD JOB! Test Case DONE"] → 人工檢查 log [messagebox "GOOD JOB! Test Case DONE"] → 人工檢查 log
@@ -592,9 +615,11 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值,
- [x] 壓力層改用平台的 `bgctl``memtester` / `qfx5252-stress-ssd` / `qfx5252-stress-usb`), - [x] 壓力層改用平台的 `bgctl``memtester` / `qfx5252-stress-ssd` / `qfx5252-stress-usb`),
取代 `~/hammer/tools/stress_*.py`Script B 以 `jobs` + `bgctl list` 兩者並列確認 取代 `~/hammer/tools/stress_*.py`Script B 以 `jobs` + `bgctl list` 兩者並列確認
- [ ] `bgctl list` 之後加數量驗證,不符時 `messagebox` 提示(目前只印出來給人看) - [ ] `bgctl list` 之後加數量驗證,不符時 `messagebox` 提示(目前只印出來給人看)
**← V1.0.10 起 `bgctl list` + `jobs` 改為每輪都印,中途死掉的 job 看得到,但仍要靠人眼比對**
- [ ] soak 迴圈改為「每輪先 `date`、跑完 `pause <間隔>`」, - [ ] soak 迴圈改為「每輪先 `date`、跑完 `pause <間隔>`」,
`EN_LoopInterval`(預設 600 秒 = 10 min)放進 `config.ttl`,兌現 Status 表的「Get data every 10mins」 `EN_LoopInterval`(預設 600 秒 = 10 min)放進 `config.ttl`,兌現 Status 表的「Get data every 10mins」
**目前 `while 1` 仍無 `pause`,實際間隔取決於指令執行時間,不是 10 分鐘** **V1.0.10 已補上 `pause 60`(迴圈不再全速空轉),但 60 秒是寫死的、也還沒有每輪 `date`
距 Status 表的 10 分鐘仍有差距,故本項未結**
- [x] soak 每輪補 `show platform psustatus` / `voltage` / `current` - [x] soak 每輪補 `show platform psustatus` / `voltage` / `current`
—— 迴圈已改為 `include "utils/setup_pmon.ttl"`,九項全收 —— 迴圈已改為 `include "utils/setup_pmon.ttl"`,九項全收
- [ ]`; ========== CLEAR EVENT ==========`Status 表 ScriptB #1Owner: Alan - [ ]`; ========== CLEAR EVENT ==========`Status 表 ScriptB #1Owner: Alan
@@ -711,18 +736,29 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值,
文件已加過期警告但內文未動。第 1~4 章不受影響。 文件已加過期警告但內文未動。第 1~4 章不受影響。
- **`wait_init.ttl` 沒有逾時出口**:若 pmon 一直起不來,`result` 既非 1 也非 2,迴圈會無限重試, - **`wait_init.ttl` 沒有逾時出口**:若 pmon 一直起不來,`result` 既非 1 也非 2,迴圈會無限重試,
Script A 既不往下走也不報錯。可加最大重試次數 + `messagebox` 提示。 Script A 既不往下走也不報錯。可加最大重試次數 + `messagebox` 提示。
- **腳本的執行權**`mgmt_ping_monitor.sh`(以及保留但已停用的 `bmc_monitor*.sh`)需人工 `chmod +x` - **腳本的執行權**`mgmt_ping_monitor.sh``bmc_monitor.sh`(以及已停用的 `bmc_monitor_ddr.sh`)需人工 `chmod +x`
可在 Script B 起它之前補一行(比照既有的 `chmod +x ~/hammer/tools/amd/mlucas-avx2`),或在 publish 時處理。 可在 Script B 起它之前補一行(比照既有的 `chmod +x ~/hammer/tools/amd/mlucas-avx2`),或在 publish 時處理。
- **`bmc_monitor.sh` / `bmc_monitor_ddr.sh` 已停用**B/C 內的呼叫已註解,改由 `bgctl` + - **`bmc_monitor_ddr.sh` 已停用**B/C 內的呼叫已註解,DDR 壓力改由 `bgctl run bmc-manager run memtester`
`bmc-manager run memtester` 取代。檔案還留在 repo,待確認不再需要後刪除,或在檔頭標註停用。 取代。檔案還留在 repo,待確認不再需要後刪除,或在檔頭標註停用。
- **`rm /host/hw-eval/current/jobs/*` 清不掉子目錄**:Script A 用的是不帶 `-r``rm`,但 Script C `bmc_monitor.sh` 已於 2026-08-25 重新啟用,並加上 I2C 寫入/讀回圖樣測試。)
- **`rm /host/hw-eval/jobs/*` 清不掉子目錄**:Script A 用的是不帶 `-r``rm`,但 Script C
`cp -r` 那個目錄,代表裡面可能有子目錄。清不乾淨的話,上一輪的 job log 會被當成這輪的一起 `cp -r` 那個目錄,代表裡面可能有子目錄。清不乾淨的話,上一輪的 job log 會被當成這輪的一起
複製到 USB。要確認 `bgctl` 的產出結構,必要時改成 `rm -rf .../jobs/*` 複製到 USB。要確認 `bgctl` 的產出結構,必要時改成 `rm -rf .../jobs/*`
- **100G PRBS 尚未接進 A/B/C**`port_prbs_monitor.sh` 已隨包交付、可手動執行,但三支腳本裡的呼叫
都還註解著(bring-up 中)。要啟用時記得 PRBS 期間 link 會顯示 down`stop` 之後才會回來。
- **Script C 的 USB 掛載走 `mount` 而非 `usb_target.sh -o mnt`**:目前是 `-o dev` 取節點再自己
`mount <dev> /mnt/usb`,繞過了工具內建的 `mkdir -p`、exfat/ntfs `modprobe` 與可寫性驗證。
掛載點不存在、檔案系統模組沒載、或媒體唯讀時只會失敗一行,job log 就沒帶出來。
- **風扇確認提示是無限等待**`wait "Set all configured fan channels to"` 之後才 `sendln "y"` - **風扇確認提示是無限等待**`wait "Set all configured fan channels to"` 之後才 `sendln "y"`
若該版 `fan-speed-control.sh` 不再詢問(或字串改了),這個 `wait` 會永遠等下去,Script A 停在那裡 若該版 `fan-speed-control.sh` 不再詢問(或字串改了),這個 `wait` 會永遠等下去,Script A 停在那裡
且不報錯。加個 `timeout` + 分支會比較保險(記得還原 `timeout = 0`)。 且不報錯。加個 `timeout` + 分支會比較保險(記得還原 `timeout = 0`)。
- **USB 掛載點未檢查**Script C 直接 `cp -r ... /mnt/usb/``/mnt/usb` 沒掛載時只會失敗一行, - **TR518 內建封包測試尚未腳本化**:`tools/TR518.txt` 記下了 `tr 518 Scenario=6 Profile=2 Count=260
整輪測試的 job log 就沒被帶出來。可在複製前先 `mountpoint -q /mnt/usb` 判斷。 PktSize=324 PortList=1-259,270-565` 這組指令(含 `port all lb=mac` + `l2 learn off` 前置),
目前只能手貼。它與 `blanton_traffic_linespeed` 是兩條互斥的打流路線(都會動 loopback 與 L2 學習),
要腳本化的話得先決定兩者怎麼共存,不能同時跑。
- **`settings/` 裡 SWB0 與 SWB1 是兩份相同的資料**:8 個 `.conf` 中,SWB1 的四個除了 `CHIPS[]` 的
bus 與 `CB_I2C_CH` 之外,net name 與 `VNOM` 與 SWB0 逐欄相同(見 `docs/LTC2980_channel_map.csv`)。
改一邊忘了改另一邊不會有任何警告。可考慮改成「共用 net 定義 + 各自的 transport 覆寫」兩層檔案。
- **Script C 沒有倒出 `mgmt_ping.log`**C 會 `stop` 管理網路監控並印 `ip -s link show` - **Script C 沒有倒出 `mgmt_ping.log`**C 會 `stop` 管理網路監控並印 `ip -s link show`
但沒有 `cat ~/Blanton_Script/log/mgmt_ping.log`,所以整個 soak 期間累積的 per-leg `RESULT` 但沒有 `cat ~/Blanton_Script/log/mgmt_ping.log`,所以整個 soak 期間累積的 per-leg `RESULT`
留在 DUT 上、沒進 master log。Script A 那輪基線有 `cat`。 留在 DUT 上、沒進 master log。Script A 那輪基線有 `cat`。
+31 -6
View File
@@ -9,12 +9,14 @@ DUT 端的取數能力由一組 **bash 工具庫** 提供(CB FPGA `pcimem` →
完整架構見 [ARCHITECTURE.md](ARCHITECTURE.md)。 完整架構見 [ARCHITECTURE.md](ARCHITECTURE.md)。
**目前狀態(2026-08-25**Script A 為 **V1.0.7**(A 集中記錄 A/B/C 三支的變更)。涵蓋 **目前狀態(2026-08-27**Script A 為 **V1.0.10**(A 集中記錄 A/B/C 三支的變更)。涵蓋
PCIe AER / DDR EDAC / PMON+TPM / 電壓 margin9 顆 LTC2980/ SWB loopback 線速流量 / PCIe AER / DDR EDAC / PMON+TPM / 電壓 margin9 顆 LTC2980/ SWB loopback 線速流量 /
10G+1G 管理網路 ping / BMC DDR 與 USB / NVMe 健康,壓力層走平台的 `bgctl` 10G+1G 管理網路 ping / BMC DDR 與 USB / NVMe 健康,壓力層走平台的 `bgctl`
switch unit 數量由 `config.ttl``SWB_UNIT0` / `SWB_UNIT1` 宣告,風扇轉速由 `FAN_SPEED` 指定。 switch unit 數量由 `config.ttl``SWB_UNIT0` / `SWB_UNIT1` 宣告,風扇轉速由 `FAN_SPEED` 指定。
測試 job log 收在 DUT 的 `/host/hw-eval/current/jobs/`,Script C 結束時帶時戳複製到 `/mnt/usb/` 測試 job log 收在 DUT 的 `/host/hw-eval/jobs/`Script C 結束時掛好 USB 再帶時戳複製到 `/mnt/usb/`
最後一次發布是 **V1.0.5**;SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證 100G PRBS`port_prbs_monitor.sh`)已隨包交付,但 **A/B/C 的呼叫仍註解掉**,還在 bring-up
Script B 的 soak 迴圈每輪印 `bgctl list` + `jobs`,並 `pause 60`V1.0.10 之前完全沒有 pause)。
最後一次發布是 **V1.0.9**;SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。
交付用 `./publish/publish.sh` 打包。 交付用 `./publish/publish.sh` 打包。
## 技術棧 ## 技術棧
@@ -85,13 +87,31 @@ chmod +x ~/Blanton_Script/*.sh # 佈署後必做一次
~/Blanton_Script/mgmt_ping_monitor.sh stop ~/Blanton_Script/mgmt_ping_monitor.sh stop
~/Blanton_Script/mgmt_ping_monitor.sh summary # 只看每段的 RESULT 行 ~/Blanton_Script/mgmt_ping_monitor.sh summary # 只看每段的 RESULT 行
# ⚠️ tail 子命令是 tail -f,會卡住不返回;TTL 裡要倒 log 請用 cat # ⚠️ tail 子命令是 tail -f,會卡住不返回;TTL 裡要倒 log 請用 cat
# ⚠️ bmc_monitor.sh / bmc_monitor_ddr.sh 已停用(改走 bgctl + bmc-manager),檔案仍在 # bmc_monitor.sh start/stop # free -m + BMC I2C 寫入/讀回圖樣測試,log/bmc_poll.log
# ⚠️ bmc_monitor_ddr.sh 已停用(DDR 壓力改走 bgctl + bmc-manager),檔案仍在
# ── 100G PRBS(目前 TTL 未啟用,手動測用)────────────
bash ~/Blanton_Script/port_prbs_monitor.sh start # 兩個 port 同時
bash ~/Blanton_Script/port_prbs_monitor.sh start a # 只跑 Ethernet513
bash ~/Blanton_Script/port_prbs_monitor.sh start b # 只跑 Ethernet514
bash ~/Blanton_Script/port_prbs_monitor.sh report # PASS/FAIL 統計(沒跑的顯示 SKIP
bash ~/Blanton_Script/port_prbs_monitor.sh stop # 收尾會跑 prbsstat STOp + prbs clear
# ⚠️ PRBS 打起來時 link 會顯示 down,這是正常的 —— 所以 start 刻意不印 port status
# ── USB 目標偵測 ────────────────────────────────────
bash ~/Blanton_Script/usb_target.sh -o dev # /dev/sdX1
bash ~/Blanton_Script/usb_target.sh -o mnt # 掛好並回傳掛載點
# ── 背景壓力(平台工具)────────────────────────────── # ── 背景壓力(平台工具)──────────────────────────────
bgctl run /usr/sbin/memtester 1G 100 bgctl run /usr/sbin/memtester 1G 100
bgctl list bgctl list
bgctl stop --all ; bgctl reset --yes bgctl stop --all ; bgctl reset --yes
# ── 通道對照表(開發機上跑,改完 settings/*.conf 一定要重產)──
./tools/gen_channel_map.sh # → docs/LTC2980_channel_map.csv144 列)
./tools/gen_channel_map.sh - # 只印到 stdout
# ⚠️ 需要 gawk(用到 3 參數的 match());mawk 會直接擋下並提示
# ── 各工具的 help ─────────────────────────────────── # ── 各工具的 help ───────────────────────────────────
blanton_fpga_help ; blanton_cb_i2c_help ; vi2c_help blanton_fpga_help ; blanton_cb_i2c_help ; vi2c_help
temp_sensor_help ; pwr_data_help ; blanton_traffic_linespeed_help temp_sensor_help ; pwr_data_help ; blanton_traffic_linespeed_help
@@ -186,6 +206,8 @@ Tera Term 連 COM port (115200-8-N-1)
- ⚠️ **兩張管理網卡同網段會 ARP flux**`mgmt_ping_monitor.sh` 讓 eth0/eth1 同時 up,同一個 /24 下 - ⚠️ **兩張管理網卡同網段會 ARP flux**`mgmt_ping_monitor.sh` 讓 eth0/eth1 同時 up,同一個 /24 下
對端 ARP 可能由任一張卡回應,回包落在沒送封包的那張。`ARP_STRICT=1` 擋掉;判讀時看 `RESULT` 行的 對端 ARP 可能由任一張卡回應,回包落在沒送封包的那張。`ARP_STRICT=1` 擋掉;判讀時看 `RESULT` 行的
`nic_tx`/`nic_rx`(該卡自己的 packet delta)—— **ping 成功但 `nic_tx` 接近 0 = 封包從另一張卡出去的** `nic_tx`/`nic_rx`(該卡自己的 packet delta)—— **ping 成功但 `nic_tx` 接近 0 = 封包從另一張卡出去的**
- ⚠️ **PRBS 跑起來時 `show interfaces status` 會顯示 DOWN**,那是 link 離開正常運作模式,不是故障。
`port_prbs_monitor.sh``start` 因此刻意不印 port status(避免測試員誤判),只在 `prbs clear` 之後的報表裡印
- ⚠️ **`ip -s link` 的欄位順序是 `bytes packets errors ...`**,要 packet 數取的是**第 2 欄**不是第 1 欄 - ⚠️ **`ip -s link` 的欄位順序是 `bytes packets errors ...`**,要 packet 數取的是**第 2 欄**不是第 1 欄
- ⚠️ **`mgmt_ping_monitor.sh` / `bmc_monitor*.sh` 需要 exec bit**,且**不是改用 `bash x.sh start` 就能繞過** —— - ⚠️ **`mgmt_ping_monitor.sh` / `bmc_monitor*.sh` 需要 exec bit**,且**不是改用 `bash x.sh start` 就能繞過** ——
`do_start` 內部是 `setsid "$SCRIPT_PATH" __daemon`,直接執行該路徑。git 記錄是 `100644` `do_start` 內部是 `setsid "$SCRIPT_PATH" __daemon`,直接執行該路徑。git 記錄是 `100644`
@@ -206,10 +228,13 @@ Tera Term 連 COM port (115200-8-N-1)
## 資料夾說明 ## 資料夾說明
- `src/Script_ABC_Blanton/` — 可執行內容(Tera Term 工作目錄;內含要 scp 到 DUT 的 `Blanton_Script/` - `src/Script_ABC_Blanton/` — 可執行內容(Tera Term 工作目錄;內含要 scp 到 DUT 的 `Blanton_Script/`
- `docs/` — 客戶 FPGA 規格、AER/EDAC 檢查依據、逐項 Status 表(最新為 `*_20260817.xlsx` - `docs/` — 客戶 FPGA 規格、AER/EDAC 檢查依據、逐項 Status 表(最新為 `*_20260817.xlsx`
`LTC2980_channel_map.csv`9 個 `settings/*.conf` 合併成 Board,CONN,Ch,NetName,Vnom 144 列,
查 net 對應哪片哪個 channel 用;**改 `.conf` 後要重跑產生**
- `tools/` — Host / 離線端:`bcm_mibpair_report_V1.1.0.py`drivshell log → per-pair 報表)、 - `tools/` — Host / 離線端:`bcm_mibpair_report_V1.1.0.py`drivshell log → per-pair 報表)、
`traffic_loopback_*.txt`bcmcmd 原始指令表,`blanton_traffic_linespeed.sh` 是其 bash 版, `traffic_loopback_*.txt`bcmcmd 原始指令表,`blanton_traffic_linespeed.sh` 是其 bash 版,
**改 loopback 線路時兩邊要同步** **改 loopback 線路時兩邊要同步**`100G_PRBS.txt` / `TR518.txt` / `fan_ctrl.txt`
(上機手貼用的原始指令;PRBS 已有 `.sh`TR518 尚未腳本化)
- `publish/``publish.sh`(打包工具,**進 git**+ `Script_ABC_Blanton_<Ver>/` 產出(**gitignored** - `publish/``publish.sh`(打包工具,**進 git**+ `Script_ABC_Blanton_<Ver>/` 產出(**gitignored**
- `secret/` — 🚫 gitignored(除 `README.md``*.example`):DUT 帳密、per-unit BDF 覆寫、COM 設定 - `secret/` — 🚫 gitignored(除 `README.md``*.example`):DUT 帳密、per-unit BDF 覆寫、COM 設定
- `For_AI/` — 🚫 gitignored:AI 協作素材(波形截圖、草稿筆記) - `For_AI/` — 🚫 gitignored:AI 協作素材(波形截圖、草稿筆記)
+145
View File
@@ -0,0 +1,145 @@
Board,CONN,Ch,NetName,Vnom
CB,CONN13,CH0,V5P0_ALW,5.0
CB,CONN13,CH1,V3P3_ALW,3.3
CB,CONN13,CH2,PWR_VDD_MISC_ALW,0.75
CB,CONN13,CH3,V1P8_ALW,1.8
CB,CONN13,CH4,PWR_APU_VDDIO_SUS,1.1
CB,CONN13,CH5,PWR_VDD_MISC_RUN,0.75
CB,CONN13,CH6,PWR_APU_VDD_MEM_RUN,0.78
CB,CONN13,CH7,V0P8_PHY2_DVDD,0.8
CB,CONN13,CH8,V0P8_AVDD,0.8
CB,CONN13,CH9,V0P8_PHY,0.8
CB,CONN13,CH10,V1P8_FPGA,1.8
CB,CONN13,CH11,V2P5_FPGA,2.5
CB,CONN13,CH12,V1P1_FPGA,1.1
CB,CONN13,CH13,P5V_STBY,5.0
CB,CONN13,CH14,P3V3_STBY,3.3
CB,CONN13,CH15,V0P8_PHY2_AVDD,0.8
SWB0,CONN13,CH0,P0V75_DVDD_47,0.75
SWB0,CONN13,CH1,P0V75_AVDD_47,0.75
SWB0,CONN13,CH2,P1V5_AVDD_47,1.5
SWB0,CONN13,CH3,P0V9_AVDD_47,0.9
SWB0,CONN13,CH4,P1V8_1,1.8
SWB0,CONN13,CH5,PVDD3_3V_SYNTH_LDO_A3,3.3
SWB0,CONN13,CH6,PVDD3_3V_SYNTH_LDO_B1,3.3
SWB0,CONN13,CH7,P1V8_3,1.8
SWB0,CONN13,CH8,P1V8_DVDDIO_3,1.8
SWB0,CONN13,CH9,P0V75_DVDD_39,0.75
SWB0,CONN13,CH10,P0V75_AVDD_39,0.75
SWB0,CONN13,CH11,P1V5_AVDD_39,1.5
SWB0,CONN13,CH12,P0V9_AVDD_39,0.9
SWB0,CONN13,CH13,PVDD1V5_2,1.5
SWB0,CONN13,CH14,PVDD1V8_DUT,1.8
SWB0,CONN13,CH15,PVDD_MDIO,1.2
SWB0,CONN14,CH0,P0V75_AVDD_7,0.75
SWB0,CONN14,CH1,P1V5_AVDD_7,1.5
SWB0,CONN14,CH2,P0V9_AVDD_7,0.9
SWB0,CONN14,CH3,P1V8_5,1.8
SWB0,CONN14,CH4,P1V8_2,1.8
SWB0,CONN14,CH5,PVDD3_3V_SYNTH_LDO_B2,3.3
SWB0,CONN14,CH6,PVDD3_3V_SYNTH_LDO_A2,3.3
SWB0,CONN14,CH7,PVDD3_3V_SYNTH_LDO_A1,3.3
SWB0,CONN14,CH8,PVDD1V5_TSC,1.5
SWB0,CONN14,CH9,PVDD1V5_ANLG,1.5
SWB0,CONN14,CH10,NC,-
SWB0,CONN14,CH11,NC,-
SWB0,CONN14,CH12,PVDD1V5_1,1.5
SWB0,CONN14,CH13,NC,-
SWB0,CONN14,CH14,P1V8_DVDDIO_1,1.8
SWB0,CONN14,CH15,P0V75_DVDD_7,0.75
SWB0,CONN15,CH0,P1V5_AVDD_55,1.5
SWB0,CONN15,CH1,P0V9_AVDD_55,0.9
SWB0,CONN15,CH2,P0V75_DVDD_63,0.75
SWB0,CONN15,CH3,P0V75_AVDD_63,0.75
SWB0,CONN15,CH4,P1V5_AVDD_63,1.5
SWB0,CONN15,CH5,P0V9_AVDD_63,0.9
SWB0,CONN15,CH6,PVDD3_3V_SYNTH_LDO_B3,3.3
SWB0,CONN15,CH7,PVDD1V5_3,1.5
SWB0,CONN15,CH8,P0V85_STBY,0.85
SWB0,CONN15,CH9,P1V8_STBY,1.8
SWB0,CONN15,CH10,NC,-
SWB0,CONN15,CH11,P3V3_STBY,3.3
SWB0,CONN15,CH12,PVDD0V9_2,0.9
SWB0,CONN15,CH13,P0V75_DVDD_55,0.75
SWB0,CONN15,CH14,P0V75_AVDD_55,0.75
SWB0,CONN15,CH15,P1V8_DVDDIO_4,1.8
SWB0,CONN16,CH0,PVDD1V5_0,1.5
SWB0,CONN16,CH1,P1V8_4,1.8
SWB0,CONN16,CH2,PVDD3_3V_SYNTH_LDO_B5,3.3
SWB0,CONN16,CH3,PVDD3_3V_SYNTH_LDO_A5,3.3
SWB0,CONN16,CH4,PVDD3_3V_SYNTH_LDO_B4,3.3
SWB0,CONN16,CH5,PVDD3_3V_SYNTH_LDO_A4,3.3
SWB0,CONN16,CH6,P3V3,3.3
SWB0,CONN16,CH7,PVDD0_8V_T3,0.8
SWB0,CONN16,CH8,P0V75_DVDD_1,0.75
SWB0,CONN16,CH9,P0V75_AVDD_1,0.75
SWB0,CONN16,CH10,P1V5_AVDD_1,1.5
SWB0,CONN16,CH11,P0V9_AVDD_1,0.9
SWB0,CONN16,CH12,PVDD0V9_0,0.9
SWB0,CONN16,CH13,PVDD0V9_1,0.9
SWB0,CONN16,CH14,PVDD0V9_3,0.9
SWB0,CONN16,CH15,PVDD1V2_T3,1.2
SWB1,CONN13,CH0,P0V75_DVDD_47,0.75
SWB1,CONN13,CH1,P0V75_AVDD_47,0.75
SWB1,CONN13,CH2,P1V5_AVDD_47,1.5
SWB1,CONN13,CH3,P0V9_AVDD_47,0.9
SWB1,CONN13,CH4,P1V8_1,1.8
SWB1,CONN13,CH5,PVDD3_3V_SYNTH_LDO_A3,3.3
SWB1,CONN13,CH6,PVDD3_3V_SYNTH_LDO_B1,3.3
SWB1,CONN13,CH7,P1V8_3,1.8
SWB1,CONN13,CH8,P1V8_DVDDIO_3,1.8
SWB1,CONN13,CH9,P0V75_DVDD_39,0.75
SWB1,CONN13,CH10,P0V75_AVDD_39,0.75
SWB1,CONN13,CH11,P1V5_AVDD_39,1.5
SWB1,CONN13,CH12,P0V9_AVDD_39,0.9
SWB1,CONN13,CH13,PVDD1V5_2,1.5
SWB1,CONN13,CH14,PVDD1V8_DUT,1.8
SWB1,CONN13,CH15,PVDD_MDIO,1.2
SWB1,CONN14,CH0,P0V75_AVDD_7,0.75
SWB1,CONN14,CH1,P1V5_AVDD_7,1.5
SWB1,CONN14,CH2,P0V9_AVDD_7,0.9
SWB1,CONN14,CH3,P1V8_5,1.8
SWB1,CONN14,CH4,P1V8_2,1.8
SWB1,CONN14,CH5,PVDD3_3V_SYNTH_LDO_B2,3.3
SWB1,CONN14,CH6,PVDD3_3V_SYNTH_LDO_A2,3.3
SWB1,CONN14,CH7,PVDD3_3V_SYNTH_LDO_A1,3.3
SWB1,CONN14,CH8,PVDD1V5_TSC,1.5
SWB1,CONN14,CH9,PVDD1V5_ANLG,1.5
SWB1,CONN14,CH10,NC,-
SWB1,CONN14,CH11,NC,-
SWB1,CONN14,CH12,PVDD1V5_1,1.5
SWB1,CONN14,CH13,NC,-
SWB1,CONN14,CH14,P1V8_DVDDIO_1,1.8
SWB1,CONN14,CH15,P0V75_DVDD_7,0.75
SWB1,CONN15,CH0,P1V5_AVDD_55,1.5
SWB1,CONN15,CH1,P0V9_AVDD_55,0.9
SWB1,CONN15,CH2,P0V75_DVDD_63,0.75
SWB1,CONN15,CH3,P0V75_AVDD_63,0.75
SWB1,CONN15,CH4,P1V5_AVDD_63,1.5
SWB1,CONN15,CH5,P0V9_AVDD_63,0.9
SWB1,CONN15,CH6,PVDD3_3V_SYNTH_LDO_B3,3.3
SWB1,CONN15,CH7,PVDD1V5_3,1.5
SWB1,CONN15,CH8,P0V85_STBY,0.85
SWB1,CONN15,CH9,P1V8_STBY,1.8
SWB1,CONN15,CH10,NC,-
SWB1,CONN15,CH11,P3V3_STBY,3.3
SWB1,CONN15,CH12,PVDD0V9_2,0.9
SWB1,CONN15,CH13,P0V75_DVDD_55,0.75
SWB1,CONN15,CH14,P0V75_AVDD_55,0.75
SWB1,CONN15,CH15,P1V8_DVDDIO_4,1.8
SWB1,CONN16,CH0,PVDD1V5_0,1.5
SWB1,CONN16,CH1,P1V8_4,1.8
SWB1,CONN16,CH2,PVDD3_3V_SYNTH_LDO_B5,3.3
SWB1,CONN16,CH3,PVDD3_3V_SYNTH_LDO_A5,3.3
SWB1,CONN16,CH4,PVDD3_3V_SYNTH_LDO_B4,3.3
SWB1,CONN16,CH5,PVDD3_3V_SYNTH_LDO_A4,3.3
SWB1,CONN16,CH6,P3V3,3.3
SWB1,CONN16,CH7,PVDD0_8V_T3,0.8
SWB1,CONN16,CH8,P0V75_DVDD_1,0.75
SWB1,CONN16,CH9,P0V75_AVDD_1,0.75
SWB1,CONN16,CH10,P1V5_AVDD_1,1.5
SWB1,CONN16,CH11,P0V9_AVDD_1,0.9
SWB1,CONN16,CH12,PVDD0V9_0,0.9
SWB1,CONN16,CH13,PVDD0V9_1,0.9
SWB1,CONN16,CH14,PVDD0V9_3,0.9
SWB1,CONN16,CH15,PVDD1V2_T3,1.2
1 Board CONN Ch NetName Vnom
2 CB CONN13 CH0 V5P0_ALW 5.0
3 CB CONN13 CH1 V3P3_ALW 3.3
4 CB CONN13 CH2 PWR_VDD_MISC_ALW 0.75
5 CB CONN13 CH3 V1P8_ALW 1.8
6 CB CONN13 CH4 PWR_APU_VDDIO_SUS 1.1
7 CB CONN13 CH5 PWR_VDD_MISC_RUN 0.75
8 CB CONN13 CH6 PWR_APU_VDD_MEM_RUN 0.78
9 CB CONN13 CH7 V0P8_PHY2_DVDD 0.8
10 CB CONN13 CH8 V0P8_AVDD 0.8
11 CB CONN13 CH9 V0P8_PHY 0.8
12 CB CONN13 CH10 V1P8_FPGA 1.8
13 CB CONN13 CH11 V2P5_FPGA 2.5
14 CB CONN13 CH12 V1P1_FPGA 1.1
15 CB CONN13 CH13 P5V_STBY 5.0
16 CB CONN13 CH14 P3V3_STBY 3.3
17 CB CONN13 CH15 V0P8_PHY2_AVDD 0.8
18 SWB0 CONN13 CH0 P0V75_DVDD_47 0.75
19 SWB0 CONN13 CH1 P0V75_AVDD_47 0.75
20 SWB0 CONN13 CH2 P1V5_AVDD_47 1.5
21 SWB0 CONN13 CH3 P0V9_AVDD_47 0.9
22 SWB0 CONN13 CH4 P1V8_1 1.8
23 SWB0 CONN13 CH5 PVDD3_3V_SYNTH_LDO_A3 3.3
24 SWB0 CONN13 CH6 PVDD3_3V_SYNTH_LDO_B1 3.3
25 SWB0 CONN13 CH7 P1V8_3 1.8
26 SWB0 CONN13 CH8 P1V8_DVDDIO_3 1.8
27 SWB0 CONN13 CH9 P0V75_DVDD_39 0.75
28 SWB0 CONN13 CH10 P0V75_AVDD_39 0.75
29 SWB0 CONN13 CH11 P1V5_AVDD_39 1.5
30 SWB0 CONN13 CH12 P0V9_AVDD_39 0.9
31 SWB0 CONN13 CH13 PVDD1V5_2 1.5
32 SWB0 CONN13 CH14 PVDD1V8_DUT 1.8
33 SWB0 CONN13 CH15 PVDD_MDIO 1.2
34 SWB0 CONN14 CH0 P0V75_AVDD_7 0.75
35 SWB0 CONN14 CH1 P1V5_AVDD_7 1.5
36 SWB0 CONN14 CH2 P0V9_AVDD_7 0.9
37 SWB0 CONN14 CH3 P1V8_5 1.8
38 SWB0 CONN14 CH4 P1V8_2 1.8
39 SWB0 CONN14 CH5 PVDD3_3V_SYNTH_LDO_B2 3.3
40 SWB0 CONN14 CH6 PVDD3_3V_SYNTH_LDO_A2 3.3
41 SWB0 CONN14 CH7 PVDD3_3V_SYNTH_LDO_A1 3.3
42 SWB0 CONN14 CH8 PVDD1V5_TSC 1.5
43 SWB0 CONN14 CH9 PVDD1V5_ANLG 1.5
44 SWB0 CONN14 CH10 NC -
45 SWB0 CONN14 CH11 NC -
46 SWB0 CONN14 CH12 PVDD1V5_1 1.5
47 SWB0 CONN14 CH13 NC -
48 SWB0 CONN14 CH14 P1V8_DVDDIO_1 1.8
49 SWB0 CONN14 CH15 P0V75_DVDD_7 0.75
50 SWB0 CONN15 CH0 P1V5_AVDD_55 1.5
51 SWB0 CONN15 CH1 P0V9_AVDD_55 0.9
52 SWB0 CONN15 CH2 P0V75_DVDD_63 0.75
53 SWB0 CONN15 CH3 P0V75_AVDD_63 0.75
54 SWB0 CONN15 CH4 P1V5_AVDD_63 1.5
55 SWB0 CONN15 CH5 P0V9_AVDD_63 0.9
56 SWB0 CONN15 CH6 PVDD3_3V_SYNTH_LDO_B3 3.3
57 SWB0 CONN15 CH7 PVDD1V5_3 1.5
58 SWB0 CONN15 CH8 P0V85_STBY 0.85
59 SWB0 CONN15 CH9 P1V8_STBY 1.8
60 SWB0 CONN15 CH10 NC -
61 SWB0 CONN15 CH11 P3V3_STBY 3.3
62 SWB0 CONN15 CH12 PVDD0V9_2 0.9
63 SWB0 CONN15 CH13 P0V75_DVDD_55 0.75
64 SWB0 CONN15 CH14 P0V75_AVDD_55 0.75
65 SWB0 CONN15 CH15 P1V8_DVDDIO_4 1.8
66 SWB0 CONN16 CH0 PVDD1V5_0 1.5
67 SWB0 CONN16 CH1 P1V8_4 1.8
68 SWB0 CONN16 CH2 PVDD3_3V_SYNTH_LDO_B5 3.3
69 SWB0 CONN16 CH3 PVDD3_3V_SYNTH_LDO_A5 3.3
70 SWB0 CONN16 CH4 PVDD3_3V_SYNTH_LDO_B4 3.3
71 SWB0 CONN16 CH5 PVDD3_3V_SYNTH_LDO_A4 3.3
72 SWB0 CONN16 CH6 P3V3 3.3
73 SWB0 CONN16 CH7 PVDD0_8V_T3 0.8
74 SWB0 CONN16 CH8 P0V75_DVDD_1 0.75
75 SWB0 CONN16 CH9 P0V75_AVDD_1 0.75
76 SWB0 CONN16 CH10 P1V5_AVDD_1 1.5
77 SWB0 CONN16 CH11 P0V9_AVDD_1 0.9
78 SWB0 CONN16 CH12 PVDD0V9_0 0.9
79 SWB0 CONN16 CH13 PVDD0V9_1 0.9
80 SWB0 CONN16 CH14 PVDD0V9_3 0.9
81 SWB0 CONN16 CH15 PVDD1V2_T3 1.2
82 SWB1 CONN13 CH0 P0V75_DVDD_47 0.75
83 SWB1 CONN13 CH1 P0V75_AVDD_47 0.75
84 SWB1 CONN13 CH2 P1V5_AVDD_47 1.5
85 SWB1 CONN13 CH3 P0V9_AVDD_47 0.9
86 SWB1 CONN13 CH4 P1V8_1 1.8
87 SWB1 CONN13 CH5 PVDD3_3V_SYNTH_LDO_A3 3.3
88 SWB1 CONN13 CH6 PVDD3_3V_SYNTH_LDO_B1 3.3
89 SWB1 CONN13 CH7 P1V8_3 1.8
90 SWB1 CONN13 CH8 P1V8_DVDDIO_3 1.8
91 SWB1 CONN13 CH9 P0V75_DVDD_39 0.75
92 SWB1 CONN13 CH10 P0V75_AVDD_39 0.75
93 SWB1 CONN13 CH11 P1V5_AVDD_39 1.5
94 SWB1 CONN13 CH12 P0V9_AVDD_39 0.9
95 SWB1 CONN13 CH13 PVDD1V5_2 1.5
96 SWB1 CONN13 CH14 PVDD1V8_DUT 1.8
97 SWB1 CONN13 CH15 PVDD_MDIO 1.2
98 SWB1 CONN14 CH0 P0V75_AVDD_7 0.75
99 SWB1 CONN14 CH1 P1V5_AVDD_7 1.5
100 SWB1 CONN14 CH2 P0V9_AVDD_7 0.9
101 SWB1 CONN14 CH3 P1V8_5 1.8
102 SWB1 CONN14 CH4 P1V8_2 1.8
103 SWB1 CONN14 CH5 PVDD3_3V_SYNTH_LDO_B2 3.3
104 SWB1 CONN14 CH6 PVDD3_3V_SYNTH_LDO_A2 3.3
105 SWB1 CONN14 CH7 PVDD3_3V_SYNTH_LDO_A1 3.3
106 SWB1 CONN14 CH8 PVDD1V5_TSC 1.5
107 SWB1 CONN14 CH9 PVDD1V5_ANLG 1.5
108 SWB1 CONN14 CH10 NC -
109 SWB1 CONN14 CH11 NC -
110 SWB1 CONN14 CH12 PVDD1V5_1 1.5
111 SWB1 CONN14 CH13 NC -
112 SWB1 CONN14 CH14 P1V8_DVDDIO_1 1.8
113 SWB1 CONN14 CH15 P0V75_DVDD_7 0.75
114 SWB1 CONN15 CH0 P1V5_AVDD_55 1.5
115 SWB1 CONN15 CH1 P0V9_AVDD_55 0.9
116 SWB1 CONN15 CH2 P0V75_DVDD_63 0.75
117 SWB1 CONN15 CH3 P0V75_AVDD_63 0.75
118 SWB1 CONN15 CH4 P1V5_AVDD_63 1.5
119 SWB1 CONN15 CH5 P0V9_AVDD_63 0.9
120 SWB1 CONN15 CH6 PVDD3_3V_SYNTH_LDO_B3 3.3
121 SWB1 CONN15 CH7 PVDD1V5_3 1.5
122 SWB1 CONN15 CH8 P0V85_STBY 0.85
123 SWB1 CONN15 CH9 P1V8_STBY 1.8
124 SWB1 CONN15 CH10 NC -
125 SWB1 CONN15 CH11 P3V3_STBY 3.3
126 SWB1 CONN15 CH12 PVDD0V9_2 0.9
127 SWB1 CONN15 CH13 P0V75_DVDD_55 0.75
128 SWB1 CONN15 CH14 P0V75_AVDD_55 0.75
129 SWB1 CONN15 CH15 P1V8_DVDDIO_4 1.8
130 SWB1 CONN16 CH0 PVDD1V5_0 1.5
131 SWB1 CONN16 CH1 P1V8_4 1.8
132 SWB1 CONN16 CH2 PVDD3_3V_SYNTH_LDO_B5 3.3
133 SWB1 CONN16 CH3 PVDD3_3V_SYNTH_LDO_A5 3.3
134 SWB1 CONN16 CH4 PVDD3_3V_SYNTH_LDO_B4 3.3
135 SWB1 CONN16 CH5 PVDD3_3V_SYNTH_LDO_A4 3.3
136 SWB1 CONN16 CH6 P3V3 3.3
137 SWB1 CONN16 CH7 PVDD0_8V_T3 0.8
138 SWB1 CONN16 CH8 P0V75_DVDD_1 0.75
139 SWB1 CONN16 CH9 P0V75_AVDD_1 0.75
140 SWB1 CONN16 CH10 P1V5_AVDD_1 1.5
141 SWB1 CONN16 CH11 P0V9_AVDD_1 0.9
142 SWB1 CONN16 CH12 PVDD0V9_0 0.9
143 SWB1 CONN16 CH13 PVDD0V9_1 0.9
144 SWB1 CONN16 CH14 PVDD0V9_3 0.9
145 SWB1 CONN16 CH15 PVDD1V2_T3 1.2
+56
View File
@@ -0,0 +1,56 @@
# v1.0.10 — The soak loop stops spinning flat out, and shows you what is still running
## ✨ New features
**Every soak round now reports the background jobs**
* Script B's monitoring loop prints `bgctl list` (platform jobs) and `jobs` (the login shell's own background jobs) on every pass, right after the platform data. A stress job that dies three hours into an overnight soak now shows up as a shorter list on the next round, instead of only being noticed at the end — or not at all.
* The two lists answer different questions and are both worth having: `bgctl` knows about the platform's job runner, `jobs` knows about anything the macro backgrounded in that shell.
**A single table for all 144 margin channels**
* `docs/LTC2980_channel_map.csv` flattens the nine `settings/*.conf` files into one sheet — `Board,CONN,Ch,NetName,Vnom` — so "which board and channel is `PVDD1V5_TSC` on?" is one search instead of opening nine files.
* `tools/gen_channel_map.sh` regenerates it. **The CSV is output, not source**: edit the `.conf` files and re-run the script, never the other way round.
* Unused channels stay in the table as `NC` / `-` rather than being dropped, because the channel number is also the PMBus page (`ch / 8` picks the chip, `ch % 8` picks the page) — filtering the gaps out would shift everything after them.
**The bench command lists that these scripts came from**
* `tools/100G_PRBS.txt`, `tools/TR518.txt` and `tools/fan_ctrl.txt` join the existing `traffic_loopback_*.txt` set: the raw `bcmcmd` / sysfs sequences, kept in the form you can paste into a console when a script is misbehaving and you want to drive the hardware directly.
## 🐛 Bug fixes
**The soak loop ran as fast as the DUT could answer**
* `while 1` had no pause at all, so each round started the moment the previous one finished. Over a long soak that means constant console traffic and a log full of near-identical samples taken seconds apart. The loop now ends each round with `pause 60`.
* Note the gap this leaves: the Status sheet asks for a 10-minute sampling interval, and 60 seconds is still hard-coded rather than configurable. This makes the loop sane, it does not finish the job.
**A comment promised a cadence the code never had**
* The loop was labelled `Get data every 10mins` while running with no delay whatsoever. It now says what it does.
## 📦 Downloads
| File | Contents |
|---|---|
| `Script_ABC_Blanton_V1.0.10.zip` | The full Tera Term working directory, including `Blanton_Script/` to copy onto the DUT |
**After copying to the DUT:**
```bash
chmod +x ~/Blanton_Script/*.sh # see below
grep -rlU $'\r' ~/Blanton_Script # expect no output
```
`mgmt_ping_monitor.sh` and `port_prbs_monitor.sh` launch their workers through `bash`, so those two run without the execute bit. `bmc_monitor.sh` and `usb_target.sh` still need it, and neither git (mode 100644) nor a Windows/USB copy carries it. Without the `chmod`, `start` fails with `Permission denied` and the later `cat` finds nothing: **the section ends up empty and nothing reports an error.**
## ⚠️ Before you run
* **Declare the switch population.** `SWB_UNIT0` / `SWB_UNIT1` in `config.ttl` say which switch units this DUT has; traffic and the readiness gate both follow them.
* **Set `FAN_SPEED`.** It is applied at the start of every run.
* **Management connectivity is in use, not preserved.** Both management NICs are up and pinging — drive the run from the serial console.
* **PRBS is still not wired into the macros.** Run `port_prbs_monitor.sh` by hand if you want it; the calls in A/B/C remain commented out, unchanged from v1.0.9.
* **TR518 and the loopback traffic test cannot both run.** `tools/TR518.txt` sets `port all lb=mac` and `l2 learn off` across the whole unit, which is the same hardware `blanton_traffic_linespeed` is using. Pick one.
* **Two settings persist to `config_db.json`** and survive a reboot: LLDP is left disabled and the 100G uplinks are left configured. Restore them before the DUT moves on.
## 🔗 Links
* [ARCHITECTURE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/ARCHITECTURE.md) — test flow, data models, constraints
* [CLAUDE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/CLAUDE.md) — commands and the bench gotchas
* [LTC2980_channel_map.csv](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/docs/LTC2980_channel_map.csv) — all 144 margin channels
**Full changelog:** [V1.0.9...V1.0.10](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/compare/V1.0.9...V1.0.10)
+61
View File
@@ -0,0 +1,61 @@
# v1.0.8 — Both management links tested at once, and ARP stops looking like packet loss
## ✨ New features
**Both management NICs are pinged simultaneously and continuously**
* `mgmt_ping_monitor.sh` no longer alternates fixed bursts. `start` configures both NICs, then pings from each at the same time and keeps accumulating until stopped — so a soak-long run is one continuous measurement rather than a series of snapshots.
* `stop` renders a report into the log: the last 20 entries per NIC, each one's ping statistics and PASS/FAIL, then `ip -s link show` for both interfaces.
* Every line is timestamped, and a request that got no reply prints a marker instead of merely being absent — a drop is visible in the tail, not inferred from a gap in the sequence numbers.
* `status` shows both pids and how many replies each NIC has received so far, which is the quick way to see one side is dead without waiting for the report.
**The USB stress target is detected, not assumed**
* New `usb_target.sh` reports an inserted USB device's node, mount point or by-id path. Script B asks it for the node and builds the stress command from the answer, so a stick that enumerates as `sdb` no longer sends a write test at whatever `/dev/sda1` happens to be.
* It excludes whatever disk backs `/` and `/host` — this platform can boot from a USB DOM, which also reports as USB — refuses to guess when several USB disks are present, and when it mounts, verifies the result is actually writable rather than trusting that `mount` succeeded.
**BMC I2C integrity is exercised through the soak**
* `bmc_monitor.sh` is back in the run and now writes two complementary patterns to a BMC scratch register and reads each back. One pattern alone cannot catch a bit stuck the same way it was written; repeating the pair through the soak turns an intermittent I2C fault into something the log records rather than something the tester has to witness.
**Test artefacts leave the DUT with the run**
* Script C copies `bmc_poll.log` and `mgmt_ping.log` into the job directory before it is archived to USB, so the monitors' output travels with the `bgctl` job logs.
* DDR stress now runs continuously instead of stopping after 100 passes, matching the other soak loads.
## 🐛 Bug fixes
**Three FAILs that were not link faults**
* A bench run reported `FAIL=3` with zero NIC errors, zero drops and sub-millisecond replies. Every failure was missing exactly `icmp_seq=1` and nothing else: once the neighbour entry for the target expires, the first echo request is spent resolving ARP and `ping` counts it as loss. A discarded warm-up ping per NIC now absorbs that, which is what lets the loss threshold stay at zero and still mean something. Raising the threshold instead would have hidden genuine single-packet loss.
**Job logs were being cleared before they were collected**
* Script C ran `bgctl reset --yes` while killing processes — before the job directory is copied to USB. Moved to after the archive, so a run's own logs are collected before anything clears them.
**The fan controller was bound twice**
* One of the two MAX31790 controllers received a second `bind` immediately after the first, which could only fail because the driver was already attached.
## 📦 Downloads
| File | Contents |
|---|---|
| `Script_ABC_Blanton_V1.0.8.zip` | The full Tera Term working directory, including `Blanton_Script/` to copy onto the DUT |
**After copying to the DUT:**
```bash
chmod +x ~/Blanton_Script/*.sh # see below
grep -rlU $'\r' ~/Blanton_Script # expect no output
```
`mgmt_ping_monitor.sh` no longer re-execs itself, so `bash mgmt_ping_monitor.sh start` works without the execute bit. `bmc_monitor.sh` and `usb_target.sh` still need it — and neither git (mode 100644) nor a Windows/USB copy carries it. Without the `chmod`, `start` fails with `Permission denied` and the later `cat` finds nothing: **the section ends up empty and nothing reports an error.**
## ⚠️ Before you run
* **Declare the switch population.** `SWB_UNIT0` / `SWB_UNIT1` in `config.ttl` say which switch units this DUT has. Traffic and the readiness gate both follow them; with neither set the traffic stage is skipped rather than failing against hardware that is not there.
* **Set the fan speed you want.** `FAN_SPEED` in `config.ttl` is applied at the start of every run.
* **Management connectivity is in use, not preserved.** Both NICs are up and pinging — drive the run from the serial console.
* **If the two management NICs share a subnet, keep `ARP_STRICT=1`.** Otherwise the target's ARP can be answered by either NIC and a reply may arrive on the one that did not send. Check `nic_tx` / `nic_rx` on the RESULT line: they are that interface's own counter delta.
* **Two settings persist to `config_db.json`** and survive a reboot: LLDP is left disabled and the 100G uplinks are left configured. Restore them before the DUT moves on.
## 🔗 Links
* [ARCHITECTURE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/ARCHITECTURE.md) — test flow, data models, constraints
* [CLAUDE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/CLAUDE.md) — commands and the bench gotchas
**Full changelog:** [V1.0.7...V1.0.8](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/compare/V1.0.7...V1.0.8)
+63
View File
@@ -0,0 +1,63 @@
# v1.0.9 — 100G PRBS testing, and the job logs actually reach the USB stick
## ✨ New features
**100G uplink PRBS test**
* New `port_prbs_monitor.sh` drives a PRBS test on both 100G uplinks: it clears low-power mode, arms the pattern, then polls `phy diag <phy> prbs get` in the background until stopped. A poll counts as PASS only when the output says `PRBS OK!`; `prbsstat Ber` is captured alongside every poll for the record.
* `stop` runs `prbsstat STOp` and `prbs clear` on each port before rendering the report, so the test does not leave PRBS armed behind it.
* `report` gives the tally the bench actually wants:
```
PORT POLLS PASS FAIL RESULT
Ethernet513 42 42 0 PASS
Ethernet514 42 40 2 FAIL
```
* **One port at a time is a first-class mode.** `start a` or `start b` runs a single uplink, which is how you separate a genuine port fault from the DUT not coping with two PRBS streams at once. A port that was not run reports `SKIP`, not `FAIL`.
* Every command and its full output — setup, each poll, teardown — goes to that port's raw log.
> ⚠️ The script ships in this build and can be run by hand, but **the calls in Script A, B and C are commented out**: PRBS is still under bring-up on this platform.
**USB target detection**
* `usb_target.sh` reports an inserted USB device's node, mount point or by-id path. It excludes whatever disk backs `/` and `/host` — this platform can boot from a USB DOM, which also reports as USB — refuses to guess when several USB disks are present, and when mounting, verifies the result is writable rather than trusting that `mount` succeeded.
* Script B builds the USB stress command from the detected node instead of a hard-coded `/dev/sda1`.
## 🐛 Bug fixes
**The job logs were being written to a path nobody collected**
* Stress output, the monitors' logs and the USB archive disagreed about where the job directory lives. Everything now uses `/host/hw-eval/jobs/`, so what Script B writes is what Script C copies out.
**Script C copied to a USB that was never mounted**
* The archive step assumed `/mnt/usb` was ready. Script C now finds the device and mounts it first, so a run's logs leave the DUT instead of being written into an empty mount point.
**`show interfaces status` right after starting PRBS reads as a failure**
* With PRBS armed the link is out of normal operation and reports DOWN. `port_prbs_monitor.sh` deliberately does not print the port status at `start` — it appears in the report instead, after `prbs clear`, labelled as the recovered state.
## 📦 Downloads
| File | Contents |
|---|---|
| `Script_ABC_Blanton_V1.0.9.zip` | The full Tera Term working directory, including `Blanton_Script/` to copy onto the DUT |
**After copying to the DUT:**
```bash
chmod +x ~/Blanton_Script/*.sh # see below
grep -rlU $'\r' ~/Blanton_Script # expect no output
```
`mgmt_ping_monitor.sh` and `port_prbs_monitor.sh` launch their workers through `bash`, so those two run without the execute bit. `bmc_monitor.sh` and `usb_target.sh` still need it, and neither git (mode 100644) nor a Windows/USB copy carries it. Without the `chmod`, `start` fails with `Permission denied` and the later `cat` finds nothing: **the section ends up empty and nothing reports an error.**
## ⚠️ Before you run
* **Declare the switch population.** `SWB_UNIT0` / `SWB_UNIT1` in `config.ttl` say which switch units this DUT has; traffic and the readiness gate both follow them.
* **Set `FAN_SPEED`.** It is applied at the start of every run.
* **Management connectivity is in use, not preserved.** Both management NICs are up and pinging — drive the run from the serial console.
* **PRBS is not wired into the macros yet.** Run `port_prbs_monitor.sh` by hand if you want it; uncommenting the calls in A/B/C is not supported in this build.
* **Two settings persist to `config_db.json`** and survive a reboot: LLDP is left disabled and the 100G uplinks are left configured. Restore them before the DUT moves on.
## 🔗 Links
* [ARCHITECTURE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/ARCHITECTURE.md) — test flow, data models, constraints
* [CLAUDE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/CLAUDE.md) — commands and the bench gotchas
**Full changelog:** [V1.0.8...V1.0.9](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/compare/V1.0.8...V1.0.9)
+68 -5
View File
@@ -1,7 +1,7 @@
; ============================================================================= ; =============================================================================
; Script A for Blanton ; Script A for Blanton
; Version : V1.0.7 ; Version : V1.0.10
; Date : 2026-08-25 ; Date : 2026-08-27
; Author : ETWen ; Author : ETWen
; ============================================================================= ; =============================================================================
; Version History: ; Version History:
@@ -43,6 +43,37 @@
; ScriptC Copy stress jobs log to USB, timestamped ; ScriptC Copy stress jobs log to USB, timestamped
; utils/wait_init.ttl WT_INTERVAL 10 -> 60, fix WT_MIN comments ; utils/wait_init.ttl WT_INTERVAL 10 -> 60, fix WT_MIN comments
; ScriptC Disable BMC USB journalctl dump ; ScriptC Disable BMC USB journalctl dump
; V1.0.8 2026-08-25 Blanton_Script/bmc_monitor.sh Add BMC I2C write/read-back pattern test
; ScriptB Re-enable bmc_monitor.sh start
; ScriptC Re-enable bmc_monitor.sh stop + cat bmc_poll.log
; ScriptC Copy bmc_poll.log + mgmt_ping.log into jobs/ before the USB archive
; Blanton_Script/usb_target.sh Created - auto-detect USB device node / mount point
; ScriptB USB stress uses the detected node, not a hard-coded /dev/sda1
; ScriptB DDR memtester runs continuously (drop the 100-pass limit)
; ScriptC Move bgctl reset --yes after the USB archive
; Blanton_Script/mgmt_ping_monitor.sh V3.0.0 - both NICs ping simultaneously, accumulating
; + ARP warm-up so a stale neighbour entry is not counted as loss
; + stop renders last-N per NIC, stats, and both ip -s link show
; .gitignore Ignore monitor *.raw / *.pid / *.state
; ScriptA MGMT ping baseline 45s -> 30s, clear the log afterwards
; ScriptB Show mgmt_ping status after start
; ScriptC Show mgmt_ping status before stop, cat the report after
; V1.0.9 2026-08-26 Blanton_Script/port_prbs_monitor.sh Add 100G Port PRBS test
; start [both|a|b] - single port isolates a port fault from
; the DUT not coping with two PRBS streams at once
; !! TTL calls are commented out in A/B/C - still under bring-up
; ScriptA/B/C Job log path /host/hw-eval/current/jobs -> /host/hw-eval/jobs
; ScriptC Mount the USB (usb_target.sh) before copying the job logs
; ScriptC Reorder results: stress logs, then MGMT ping, then 100G status
; V1.0.10 2026-08-27 ScriptB Monitor loop lists the background jobs every round
; + bgctl list - platform jobs
; + jobs - shell jobs of the login shell
; + pause 60 - the loop used to run with no pause at all
; docs/LTC2980_channel_map.csv Channel / net / Vnom map built from settings/*.conf
; tools/gen_channel_map.sh Regenerates that CSV - the CSV is output, not source
; tools/100G_PRBS.txt Bench command references, kept for hand-run debug
; tools/TR518.txt TR518 = built-in packet test (swutil / bcmcmd tr 518)
; tools/fan_ctrl.txt MAX31790 rebind + fan-speed-control.sh
; ============================================================================= ; =============================================================================
include "config.ttl" include "config.ttl"
@@ -79,7 +110,7 @@ sendln cmd
;Stress log clear ;Stress log clear
wait prompt_sonic_root wait prompt_sonic_root
sendln "rm /host/hw-eval/current/jobs/*" sendln "rm /host/hw-eval/jobs/*"
; ========== HW Test Session ========== ; ========== HW Test Session ==========
wait prompt_sonic_root wait prompt_sonic_root
@@ -194,14 +225,15 @@ include "utils/show_dmesg.ttl"
wait prompt_sonic_root wait prompt_sonic_root
sendln "./Blanton_Script/mgmt_ping_monitor.sh start" sendln "./Blanton_Script/mgmt_ping_monitor.sh start"
wait prompt_sonic_root wait prompt_sonic_root
pause 45 pause 30
sendln "~/Blanton_Script/mgmt_ping_monitor.sh stop" sendln "~/Blanton_Script/mgmt_ping_monitor.sh stop"
wait prompt_sonic_root wait prompt_sonic_root
sendln "cat ~/Blanton_Script/log/mgmt_ping.log" sendln "cat ~/Blanton_Script/log/mgmt_ping.log"
wait prompt_sonic_root
sendln "~/Blanton_Script/mgmt_ping_monitor.sh clear"
; ========== 100G Port Status ========== ; ========== 100G Port Status ==========
; Lpmode Disable
wait prompt_sonic_root wait prompt_sonic_root
sendln "sudo sfputil lpmode off Ethernet513" sendln "sudo sfputil lpmode off Ethernet513"
wait prompt_sonic_root wait prompt_sonic_root
@@ -216,6 +248,37 @@ sendln "config save -y"
wait prompt_sonic_root wait prompt_sonic_root
sendln "show interfaces status Ethernet513,Ethernet514" sendln "show interfaces status Ethernet513,Ethernet514"
;=============================================================================================================
; ========== 100G PRBS ==========
;wait prompt_sonic_root
;sendln "~/Blanton_Script/port_prbs_monitor.sh start"
;wait prompt_sonic_root
;sendln "sleep 10"
;wait prompt_sonic_root
;sendln "~/Blanton_Script/port_prbs_monitor.sh status"
;wait prompt_sonic_root
;sendln "~/Blanton_Script/port_prbs_monitor.sh stop"
;wait prompt_sonic_root
;sendln "~/Blanton_Script/port_prbs_monitor.sh report"
;wait prompt_sonic_root
;sendln "~/Blanton_Script/port_prbs_monitor.sh clear"
; ========== 100G Port Status ==========
;wait prompt_sonic_root
;sendln "sudo sfputil lpmode off Ethernet513"
;wait prompt_sonic_root
;sendln "sudo sfputil lpmode off Ethernet514"
;wait prompt_sonic_root
;sendln "config interface -n asic0 startup Ethernet513"
;wait prompt_sonic_root
;sendln "config interface -n asic1 startup Ethernet514"
;wait prompt_sonic_root
;sendln "config save -y"
;wait prompt_sonic_root
;sendln "show interfaces status Ethernet513,Ethernet514"
;=============================================================================================================
; ========== First Traffic Test ========== ; ========== First Traffic Test ==========
; Silent MAC ; Silent MAC
wait prompt_sonic_root wait prompt_sonic_root
+36 -5
View File
@@ -28,17 +28,34 @@ sendln "~/hammer/tools/amd/mlucas-avx2 -s l -cpu 0:15 > ~/hammer/tools/amd/mluca
wait prompt_sonic_root wait prompt_sonic_root
sendln "bgctl run bmc-manager run '/usr/bin/memtester 64M 1'" sendln "bgctl run bmc-manager run '/usr/bin/memtester 64M 1'"
; ========== DDR MEMORY STRESS TEST ========== ; ========== BMC Monitor Test ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "bgctl run /usr/sbin/memtester 1G 100" sendln "./Blanton_Script/bmc_monitor.sh start"
; ========== DDR MEMORY STRESS TEST ==========
;wait prompt_sonic_root
;sendln "bgctl run /usr/sbin/memtester 1G 100"
wait prompt_sonic_root
sendln "bgctl run /usr/sbin/memtester 1G" ; Continue Execture
; ========== SSD read/write ========== ; ========== SSD read/write ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "bgctl run qfx5252-stress-ssd 1G --runtime 3600 --passes 5 --write --force --log /host/hw-eval/current/qfx5252-stress-ssd.log" sendln "bgctl run qfx5252-stress-ssd 1G --runtime 3600 --passes 5 --write --force --log /host/hw-eval/jobs/qfx5252-stress-ssd.log"
; ========== USB read/write ========== ; ========== USB read/write ==========
; ---- Get USB partition node -> usb_dev ----
usb_dev = ''
wait prompt_sonic_root wait prompt_sonic_root
sendln "bgctl run qfx5252-stress-usb /dev/sda1 256M --runtime 60 --passes 1 --write --log /host/hw-eval/qfx5252-stress-usb.log" sendln 'D=$(bash ~/Blanton_Script/usb_target.sh -o dev 2>/dev/null); echo "<<D""EV=${D:-NONE}>>"'
waitregex '<<DEV=([^>]*)>>'
usb_dev = groupmatchstr1
wait prompt_sonic_root
sprintf2 cmd "bgctl run qfx5252-stress-usb %s 256M --runtime 60 --passes 1 --write --log /host/hw-eval/jobs/qfx5252-stress-usb.log" usb_dev
sendln cmd
;wait prompt_sonic_root
;sendln "bgctl run qfx5252-stress-usb /dev/sda1 256M --runtime 60 --passes 1 --write --log /host/hw-eval/qfx5252-stress-usb.log"
; ========== SHOW Background job ========== ; ========== SHOW Background job ==========
wait prompt_sonic_root wait prompt_sonic_root
@@ -79,12 +96,21 @@ timeout = 0
; ========== 10G/1G MGMT Ping Test ========== ; ========== 10G/1G MGMT Ping Test ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "./Blanton_Script/mgmt_ping_monitor.sh start" sendln "./Blanton_Script/mgmt_ping_monitor.sh start"
wait prompt_sonic_root
pause 3
sendln "./Blanton_Script/mgmt_ping_monitor.sh status"
; ========== 100G Port ========== ; ========== 100G Port ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "show interfaces status Ethernet513,Ethernet514" sendln "show interfaces status Ethernet513,Ethernet514"
;=============================================================================================================
; ========== 100G PRBS ==========
;wait prompt_sonic_root
;sendln "~/Blanton_Script/port_prbs_monitor.sh start"
;=============================================================================================================
; ========== Traffic START ========== ; ========== Traffic START ==========
; Silent MAC ; Silent MAC
@@ -135,11 +161,16 @@ endif
; ========== Get data every 10mins ========== ; ========== Monitor loop: platform data + background jobs (pause 60s per round) ==========
while 1 while 1
include "utils/setup_pmon.ttl" include "utils/setup_pmon.ttl"
wait prompt_sonic_root
sendln "bgctl list"
wait prompt_sonic_root
sendln "jobs"
; ========== Check Margin ========== ; ========== Check Margin ==========
if EN_Margin = 1 then if EN_Margin = 1 then
include "utils/show_margin_status.ttl" include "utils/show_margin_status.ttl"
endif endif
pause 60
endwhile endwhile
+63 -10
View File
@@ -17,15 +17,31 @@ wait prompt_sonic_root
sendln "bgctl stop --all" sendln "bgctl stop --all"
wait prompt_sonic_root wait prompt_sonic_root
sendln "bgctl stop --all" sendln "bgctl stop --all"
wait prompt_sonic_root
sendln "bgctl reset --yes"
; BMC USB Test STOP ; BMC USB Test STOP
wait prompt_sonic_root wait prompt_sonic_root
sendln "systemctl stop qfx5252-bmc-usb-net-test.service" sendln "systemctl stop qfx5252-bmc-usb-net-test.service"
; 10G/1G MGMT Test STOP ; 10G/1G MGMT Test STOP
wait prompt_sonic_root wait prompt_sonic_root
sendln "~/Blanton_Script/mgmt_ping_monitor.sh status"
wait prompt_sonic_root
sendln "~/Blanton_Script/mgmt_ping_monitor.sh stop" sendln "~/Blanton_Script/mgmt_ping_monitor.sh stop"
;=============================================================================================================
; 100G PRBS STOP
;wait prompt_sonic_root
;sendln "~/Blanton_Script/port_prbs_monitor.sh stop"
;wait prompt_sonic_root
;sendln "sudo sfputil lpmode off Ethernet513"
;wait prompt_sonic_root
;sendln "sudo sfputil lpmode off Ethernet514"
;wait prompt_sonic_root
;sendln "config interface -n asic0 startup Ethernet513"
;wait prompt_sonic_root
;sendln "config interface -n asic1 startup Ethernet514"
;=============================================================================================================
; ========== TAKE DATA ========== ; ========== TAKE DATA ==========
include "utils/setup_pmon.ttl" include "utils/setup_pmon.ttl"
@@ -53,20 +69,36 @@ sendln "/usr/sbin/ras-mc-ctl --summary"
;wait prompt_sonic_root ;wait prompt_sonic_root
;sendln "cat ./Blanton_Script/log/bmc_ddr.log" ;sendln "cat ./Blanton_Script/log/bmc_ddr.log"
; ========== BMC Monitor TAKE DATA ==========
wait prompt_sonic_root
sendln "./Blanton_Script/bmc_monitor.sh stop"
wait prompt_sonic_root
sendln "cat ./Blanton_Script/log/bmc_poll.log"
; ========== BMC USB Test result ========== ; ========== BMC USB Test result ==========
;wait prompt_sonic_root ;wait prompt_sonic_root
;sendln "journalctl -u qfx5252-bmc-usb-net-test.service --no-pager" ;sendln "journalctl -u qfx5252-bmc-usb-net-test.service --no-pager"
; ========== 10G/1G MGMT Test result ==========
wait prompt_sonic_root
sendln "ip -s link show eth0"
wait prompt_sonic_root
sendln "ip -s link show eth1"
; ========== CHECK Stress results ========== ; ========== CHECK Stress results ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "cat ~/hammer/tools/amd/mlucas_amm_log" sendln "cat ~/hammer/tools/amd/mlucas_amm_log"
wait prompt_sonic_root
sendln "cat /host/hw-eval/jobs/qfx5252-stress-ssd.log"
wait prompt_sonic_root
sendln "cat /host/hw-eval/jobs/qfx5252-stress-usb.log"
; ========== 10G/1G MGMT Test result ==========
wait prompt_sonic_root
sendln "cat ~/Blanton_Script/log/mgmt_ping.log"
; ========== 100G Port ==========
wait prompt_sonic_root
sendln "show interfaces status Ethernet513,Ethernet514"
;=============================================================================================================
;wait prompt_sonic_root
;sendln "cat ~/Blanton_Script/log/port_prbs.log "
;=============================================================================================================
; ========== CHECK Traffic counters ========== ; ========== CHECK Traffic counters ==========
if SWB_UNIT0 = 1 && SWB_UNIT1 = 1 then if SWB_UNIT0 = 1 && SWB_UNIT1 = 1 then
@@ -102,16 +134,37 @@ sendln "show reboot-cause"
wait prompt_sonic_root wait prompt_sonic_root
sendln "show uptime" sendln "show uptime"
; ---- Get USB partition node -> usb_dev ----
usb_dev = ''
wait prompt_sonic_root
sendln 'D=$(bash ~/Blanton_Script/usb_target.sh -o dev 2>/dev/null); echo "<<D""EV=${D:-NONE}>>"'
waitregex '<<DEV=([^>]*)>>'
usb_dev = groupmatchstr1
wait prompt_sonic_root
sendln "sync"
wait prompt_sonic_root
sprintf2 cmd "mount %s /mnt/usb" usb_dev
sendln cmd
; ========== Copy Stress Log to USB ========== ; ========== Copy Stress Log to USB ==========
wait prompt_sonic_root
sendln "cp ./Blanton_Script/log/bmc_poll.log /host/hw-eval/jobs/"
wait prompt_sonic_root
sendln "cp ./Blanton_Script/log/mgmt_ping.log /host/hw-eval/jobs/"
wait prompt_sonic_root
getdate ts_date "%Y%m%d" getdate ts_date "%Y%m%d"
gettime ts_time "%H%M" gettime ts_time "%H%M"
sprintf2 usb_dst "/mnt/usb/jobs-%s_%s" ts_date ts_time sprintf2 usb_dst "/mnt/usb/jobs-%s_%s" ts_date ts_time
sprintf2 cmd "cp -r /host/hw-eval/current/jobs/ %s" usb_dst sprintf2 cmd "cp -r /host/hw-eval/jobs/ %s" usb_dst
wait prompt_sonic_root
sendln cmd sendln cmd
; ========== HW Test Session ========== ; ========== HW Test Session ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "hw-test-session finish" sendln "hw-test-session finish"
wait prompt_sonic_root
sendln "bgctl reset --yes"
messagebox 'GOOD JOB! Test Case DONE' 'teraterm' messagebox 'GOOD JOB! Test Case DONE' 'teraterm'
@@ -34,6 +34,10 @@ set -u
# Add a line to add a command; comment it out to disable it. # Add a line to add a command; comment it out to disable it.
COMMANDS=( COMMANDS=(
"free -m" "free -m"
"i2ctransfer -f -y 1 w6@0x41 0x00 0xC0 0x55 0xAA 0x55 0xAA"
"i2ctransfer -f -y 1 w2@0x41 0x00 0xC0 r4"
"i2ctransfer -f -y 1 w6@0x41 0x00 0xC0 0xAA 0x55 0xAA 0x55"
"i2ctransfer -f -y 1 w2@0x41 0x00 0xC0 r4"
#"uptime" #"uptime"
#"cat /proc/loadavg" #"cat /proc/loadavg"
#"cat /proc/meminfo" #"cat /proc/meminfo"
@@ -2,48 +2,41 @@
############################################################################### ###############################################################################
# mgmt_ping_monitor.sh # mgmt_ping_monitor.sh
# #
# Version : V1.0.0 # Version : V3.0.0
# Author : ETWen # Author : ETWen
# Date : 20260824 # Date : 20260826
# #
# Purpose : Exercise both management NICs and ping each one's own target, # Purpose : Ping from BOTH management NICs at the same time, continuously, and
# appending everything to a rotating log. One round is # keep accumulating until stopped:
# #
# 10G (eth0) : ip link up + address/route replace -> settle # 10G (eth0) -> TARGET_10G
# -> ping N x TARGET_10G -> ip -s link show # 1G (eth1) -> TARGET_1G
# 1G (eth1) : ip link up + address/route replace -> settle
# -> ping N x TARGET_1G -> ip -s link show
# #
# Designed to run detached in the background, same sub-commands as # `stop` renders a report into the log: the last TAIL_LINES entries
# bmc_monitor.sh: start/stop/status/fg/tail/clear. # per NIC, each one's ping statistics and verdict, then
# `ip -s link show` for both interfaces.
# #
# Notes : - Both NICs are brought up and LEFT up. Addresses and default # Notes : - Both NICs are configured with iproute2 `replace`, which is
# routes are applied with iproute2 `replace`, which is idempotent: # idempotent, and both are left up. On a shared subnet the target's
# re-running a round does not stack duplicate addresses or routes. # ARP can be answered by either NIC, so ARP_STRICT keeps each one
# - The default routes carry different metrics (100 / 200), so the # to its own address.
# lower one wins for off-subnet traffic. They do NOT decide which # - ARP WARM-UP: the first packet after a neighbour entry expires is
# NIC a ping leaves by: `ping -I <iface>` binds to the device. # spent resolving ARP and is counted as loss. On the bench this
# - !! SAME-SUBNET WARNING !! With both NICs up on one subnet the # produced three FAILs whose only missing packet was icmp_seq=1,
# target ARPs for the address and Linux may answer from either # every time, with zero NIC errors or drops. One discarded ping
# NIC, so replies can arrive on the wrong one and the result is # before the measured run removes that artefact, which is what lets
# not attributable. ARP_STRICT=1 sets arp_ignore/arp_announce to # MAX_LOSS_PCT stay at 0 and still mean something.
# stop that. Leave it on unless the two NICs are on separate # - MANAGEMENT CONNECTIVITY IS IN USE while this runs. Drive it from
# subnets, as in the iproute2 example this was modelled on. # the serial console.
# - Each leg logs `ip -s link show <iface>` and the TX/RX packet
# delta across the burst, which is the direct evidence of which
# NIC actually carried the traffic.
# - The first pings after bringing a link up are lost to ARP, not to
# the link: a bench capture showed 4x "Destination Host
# Unreachable" on the first burst and 0% loss on the very next
# one. LINK_SETTLE_SEC absorbs that; it is not packet loss.
# #
# Version History # Version History
# V1.0.0 20260824 Initial Version # V1.0.0 20260824 Initial Version (ifconfig, one NIC at a time)
# V2.0.0 20260824 Switch from ifconfig to iproute2 (ip link / address # V2.0.0 20260824 iproute2; both NICs stay up; per-NIC target;
# replace / route replace with per-NIC metric). Both # ip -s link show + TX/RX delta per leg
# NICs now stay up -- no link down between legs. Add # V3.0.0 20260826 Both NICs ping SIMULTANEOUSLY and continuously instead
# `ip -s link show` per leg with a TX/RX delta, and # of alternating fixed bursts. Add the ARP warm-up.
# ARP_STRICT for the same-subnet case. # `stop` renders last-N per NIC + statistics + both
# ip -s link outputs into the log.
############################################################################### ###############################################################################
set -u set -u
@@ -63,149 +56,67 @@ METRIC_10G=100 # lower metric wins for off-subnet traffic
# --- 1G management port --- # --- 1G management port ---
IF_1G="eth1" IF_1G="eth1"
IP_1G="192.168.1.101" IP_1G="192.168.1.101"
TARGET_1G="192.168.1.31" # host this NIC pings TARGET_1G="192.168.1.31"
PLEN_1G=24 PLEN_1G=24
GW_1G="" GW_1G=""
METRIC_1G=200 METRIC_1G=200
# --- ping --- # --- ping ---
PING_COUNT=10 # echo requests per leg PING_INTERVAL_SEC=1 # seconds between echo requests (ping -i)
MAX_LOSS_PCT=0 # loss above this marks the leg FAIL MAX_LOSS_PCT=0 # loss above this marks the NIC FAIL
TAIL_LINES=20 # how many recent entries per NIC the report shows
# Both NICs stay up. If they share a subnet the target's ARP can be answered # Extra ping flags. -D timestamps every line, -O prints a marker for a request
# by either one, so a reply may arrive on the NIC that did not send. Setting # that got no reply, so a drop is visible in the tail instead of just missing.
# this applies arp_ignore=1 / arp_announce=2 to both, which keeps each NIC to # Clear this if the platform's ping does not accept them.
# its own address. RAM only -- reverts on reboot. Set 0 for separate subnets. PING_EXTRA_OPTS="-D -O"
# One discarded ping per NIC before the measured run, to resolve ARP. Without
# it the first packet of the run is lost to neighbour resolution and looks
# exactly like a link fault. 0 disables.
ARP_WARMUP=1
ARP_WARMUP_TIMEOUT=2 # seconds to wait for the warm-up reply
# Both NICs stay up. If they share a subnet the target's ARP can be answered by
# either one, so a reply may arrive on the NIC that did not send. This applies
# arp_ignore=1 / arp_announce=2 to both. RAM only -- reverts on reboot.
ARP_STRICT=1 ARP_STRICT=1
# Prefix for the ip commands; empty because the daemon already runs as root. # Prefix for the ip commands; empty because this already runs as root.
SUDO="" SUDO=""
# Seconds to wait after bringing a link up before pinging, so ARP is resolved # Seconds to wait after bringing a link up before pinging.
# and the PHY has finished autoneg. The bench needed a few seconds on the 1G.
LINK_SETTLE_SEC=5 LINK_SETTLE_SEC=5
# Seconds between rounds (0 = start the next round immediately)
INTERVAL_SEC=10
# Per-ping-burst timeout in seconds; stops a wedged ping from freezing the
# loop. 0 disables. Must exceed PING_COUNT (one packet per second).
PING_TIMEOUT_SEC=60
# Stop after this many rounds. 0 = run until stopped manually.
MAX_ROUNDS=0
# Logging # Logging
LOG_DIR="" # empty = <script directory>/log LOG_DIR="" # empty = <script directory>/log
LOG_NAME="mgmt_ping.log" LOG_NAME="mgmt_ping.log"
MAX_LOG_SIZE_MB=100 # rotate above this size; 0 = never rotate
MAX_LOG_KEEP=5 # keep .1 .. .N
# What `start` does with the log left behind by the previous run: # What `start` does with the log left behind by the previous run:
# new = begin with an empty log; previous content and rotated copies are # new = discard it (default -- one run, one log)
# discarded (default -- one run, one log) # archive = rename it to <log>.YYYYmmdd-HHMMSS first
# archive = rename the previous log to <log>.YYYYmmdd-HHMMSS, then begin empty # append = keep it
# append = keep appending to the existing log
START_LOG_MODE="new" START_LOG_MODE="new"
############################################################################### ###############################################################################
# Internal # Internal
############################################################################### ###############################################################################
SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")" SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")"
SCRIPT_PATH="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)/${SCRIPT_NAME}" SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
[ -n "${LOG_DIR}" ] || LOG_DIR="$(dirname -- "${SCRIPT_PATH}")/log" [ -n "${LOG_DIR}" ] || LOG_DIR="${SCRIPT_DIR}/log"
LOG_FILE="${LOG_DIR}/${LOG_NAME}"
PID_FILE="${LOG_DIR}/${LOG_NAME%.log}.pid"
LOG_TO_FILE=0 LOG_FILE="${LOG_DIR}/${LOG_NAME}"
STOP=0 BASE="${LOG_DIR}/${LOG_NAME%.log}"
SLEEP_PID="" PID_FILE="${BASE}.pid"
STATE_FILE="${BASE}.state"
RAW_10G="${BASE}_${IF_10G}.raw"
RAW_1G="${BASE}_${IF_1G}.raw"
ts() { date '+%Y-%m-%d %H:%M:%S'; } ts() { date '+%Y-%m-%d %H:%M:%S'; }
die() { printf '[ERROR] %s\n' "$*" >&2; exit 1; } die() { printf '[ERROR] %s\n' "$*" >&2; exit 1; }
on_signal() { # "<rx_packets> <tx_packets>" for an interface. ip -s link orders the columns
STOP=1 # "bytes packets errors ...", so packets is $2, not $1.
[ -n "${SLEEP_PID}" ] && kill "${SLEEP_PID}" 2>/dev/null
}
interruptible_sleep() {
local sec="$1"
[ "${sec}" -gt 0 ] 2>/dev/null || return 0
sleep "${sec}" &
SLEEP_PID=$!
wait "${SLEEP_PID}" 2>/dev/null
SLEEP_PID=""
}
is_running() {
local pid
[ -f "${PID_FILE}" ] || return 1
pid="$(cat "${PID_FILE}" 2>/dev/null)"
[[ "${pid}" =~ ^[0-9]+$ ]] || return 1
kill -0 "${pid}" 2>/dev/null || return 1
grep -qa "${SCRIPT_NAME}" "/proc/${pid}/cmdline" 2>/dev/null || return 1
printf '%s' "${pid}"
return 0
}
rotate_log_if_needed() {
local max_bytes size i
[ "${LOG_TO_FILE}" -eq 1 ] || return 0
[ "${MAX_LOG_SIZE_MB}" -gt 0 ] || return 0
[ -f "${LOG_FILE}" ] || return 0
max_bytes=$(( MAX_LOG_SIZE_MB * 1024 * 1024 ))
size="$(stat -c %s "${LOG_FILE}" 2>/dev/null || echo 0)"
[ "${size}" -ge "${max_bytes}" ] || return 0
for (( i = MAX_LOG_KEEP - 1; i >= 1; i-- )); do
[ -f "${LOG_FILE}.${i}" ] && mv -f "${LOG_FILE}.${i}" "${LOG_FILE}.$(( i + 1 ))"
done
mv -f "${LOG_FILE}" "${LOG_FILE}.1"
exec >>"${LOG_FILE}" 2>&1
printf '[%s] INFO : log rotated at %s MB\n' "$(ts)" "${MAX_LOG_SIZE_MB}"
}
remove_rotated_logs() {
local i
for (( i = 1; i <= MAX_LOG_KEEP + 1; i++ )); do
rm -f "${LOG_FILE}.${i}"
done
}
prepare_log_on_start() {
local stamp
case "${START_LOG_MODE}" in
append)
return 0
;;
archive)
if [ -s "${LOG_FILE}" ]; then
stamp="$(date '+%Y%m%d-%H%M%S')"
mv -f "${LOG_FILE}" "${LOG_FILE}.${stamp}" \
|| die "cannot archive ${LOG_FILE}"
printf 'previous log archived: %s.%s\n' "${LOG_FILE}" "${stamp}"
fi
remove_rotated_logs
;;
new)
# Truncate rather than unlink: the inode is preserved, so a tail -f
# that is already attached keeps following the new run.
if [ -f "${LOG_FILE}" ]; then
: > "${LOG_FILE}" || die "cannot truncate ${LOG_FILE}"
fi
remove_rotated_logs
;;
*)
die "invalid START_LOG_MODE: ${START_LOG_MODE}"
;;
esac
}
# Cumulative "<rx_packets> <tx_packets>" for an interface, from ip -s link.
# The numbers sit on the line after each RX:/TX: header, and the column order
# is "bytes packets errors ..." -- so packets is $2, not $1.
if_counters() { if_counters() {
${SUDO} ip -s link show "$1" 2>/dev/null | awk ' ${SUDO} ip -s link show "$1" 2>/dev/null | awk '
/RX:/ { getline; rx = $2 } /RX:/ { getline; rx = $2 }
@@ -213,11 +124,9 @@ if_counters() {
END { printf "%s %s", (rx == "" ? 0 : rx), (tx == "" ? 0 : tx) }' END { printf "%s %s", (rx == "" ? 0 : rx), (tx == "" ? 0 : tx) }'
} }
# Bring the NIC up and (re)apply its address and default route. `replace` is # Bring the NIC up and (re)apply its address and default route.
# idempotent, so repeated rounds do not stack duplicates.
setup_iface() { setup_iface() {
local iface="$1" ipaddr="$2" plen="$3" gw="$4" metric="$5" local iface="$1" ipaddr="$2" plen="$3" gw="$4" metric="$5"
${SUDO} ip link set "${iface}" up 2>&1 || return 1 ${SUDO} ip link set "${iface}" up 2>&1 || return 1
${SUDO} ip address replace "${ipaddr}/${plen}" dev "${iface}" 2>&1 || return 1 ${SUDO} ip address replace "${ipaddr}/${plen}" dev "${iface}" 2>&1 || return 1
if [ -n "${gw}" ]; then if [ -n "${gw}" ]; then
@@ -227,74 +136,6 @@ setup_iface() {
return 0 return 0
} }
# One leg: configure <if>, settle, ping its own target, report. The NIC is
# LEFT UP -- nothing is brought down between legs.
# Usage: ping_leg <label> <if> <ip> <plen> <gw> <metric> <target>
ping_leg() {
local label="$1" iface="$2" ipaddr="$3" plen="$4" gw="$5" metric="$6" target="$7"
local out rc tx rx loss avg verdict
local c0 c1 rx0 tx0 rx1 tx1 drx dtx
printf -- '----- [%s] %s : %s (%s/%s) -> %s -----\n' \
"$(ts)" "${label}" "${iface}" "${ipaddr}" "${plen}" "${target}"
if ! setup_iface "${iface}" "${ipaddr}" "${plen}" "${gw}" "${metric}"; then
printf '[%s] FAIL : cannot configure %s\n\n' "$(ts)" "${iface}"
return 1
fi
interruptible_sleep "${LINK_SETTLE_SEC}"
[ "${STOP}" -eq 0 ] || return 1
c0="$(if_counters "${iface}")"; rx0="${c0% *}"; tx0="${c0#* }"
if [ "${PING_TIMEOUT_SEC}" -gt 0 ]; then
out="$(timeout "${PING_TIMEOUT_SEC}" ping -I "${iface}" -c "${PING_COUNT}" "${target}" 2>&1)"
else
out="$(ping -I "${iface}" -c "${PING_COUNT}" "${target}" 2>&1)"
fi
rc=$?
printf '%s\n' "${out}"
c1="$(if_counters "${iface}")"; rx1="${c1% *}"; tx1="${c1#* }"
drx=$(( rx1 - rx0 )); dtx=$(( tx1 - tx0 ))
printf -- '--- ip -s link show %s ---\n' "${iface}"
${SUDO} ip -s link show "${iface}" 2>&1
# "10 packets transmitted, 10 received, 0% packet loss"
tx="$(printf '%s' "${out}" | sed -n 's/^\([0-9]\+\) packets transmitted.*/\1/p' | tail -1)"
rx="$(printf '%s' "${out}" | sed -n 's/.*[, ]\([0-9]\+\) received.*/\1/p' | tail -1)"
loss="$(printf '%s' "${out}" | sed -n 's/.*[, ]\([0-9]\+\)% packet loss.*/\1/p' | tail -1)"
# "rtt min/avg/max/mdev = 1.044/1.743/2.387/0.437 ms"
avg="$(printf '%s' "${out}" | sed -n 's|.*= [0-9.]*/\([0-9.]*\)/.*|\1|p' | tail -1)"
[ -n "${tx}" ] || tx=0
[ -n "${rx}" ] || rx=0
[ -n "${loss}" ] || loss=100
[ -n "${avg}" ] || avg="-"
if [ "${rc}" -eq 124 ]; then
verdict="FAIL(timeout)"
elif [ "${loss}" -le "${MAX_LOSS_PCT}" ]; then
verdict="PASS"
else
verdict="FAIL"
fi
# one greppable line per leg: grep RESULT <log>
# nic_tx/nic_rx are this interface's own counter delta across the burst --
# if they stay near zero while the ping succeeded, the traffic went out a
# different NIC and the result is not attributable to this one.
printf '[%s] RESULT %-3s %-6s -> %-15s tx=%s rx=%s loss=%s%% rtt_avg=%sms nic_tx=%s nic_rx=%s %s\n\n' \
"$(ts)" "${label}" "${iface}" "${target}" "${tx}" "${rx}" "${loss}" "${avg}" \
"${dtx}" "${drx}" "${verdict}"
[ "${verdict}" = "PASS" ]
}
# arp_ignore=1 : answer ARP only for addresses configured on the incoming NIC
# arp_announce=2: pick the source address from the outgoing NIC's own subnet
# Both are RAM-only sysctls and revert on reboot.
apply_arp_strict() { apply_arp_strict() {
local i local i
[ "${ARP_STRICT}" -eq 1 ] || return 0 [ "${ARP_STRICT}" -eq 1 ] || return 0
@@ -302,143 +143,245 @@ apply_arp_strict() {
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_ignore=1" 2>/dev/null ${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_ignore=1" 2>/dev/null
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_announce=2" 2>/dev/null ${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_announce=2" 2>/dev/null
done done
printf '[%s] INFO : arp_ignore=1 arp_announce=2 applied to %s,%s\n' \
"$(ts)" "${IF_10G}" "${IF_1G}"
} }
main_loop() { # Resolve the neighbour so the measured run does not spend its first packet on
local round=0 pass=0 fail=0 # ARP. Result is deliberately discarded.
arp_warmup() {
local iface="$1" target="$2"
[ "${ARP_WARMUP}" -eq 1 ] || return 0
ping -I "${iface}" -c 1 -W "${ARP_WARMUP_TIMEOUT}" "${target}" >/dev/null 2>&1
return 0
}
command -v ip >/dev/null 2>&1 || die "ip (iproute2) not found" # echo "pid_10g pid_1g" and return 0 when both are alive
command -v ping >/dev/null 2>&1 || die "ping not found" is_running() {
[ "${PING_COUNT}" -gt 0 ] 2>/dev/null || die "PING_COUNT must be > 0" local p0 p1
[ -f "${PID_FILE}" ] || return 1
p0="$(sed -n '1p' "${PID_FILE}" 2>/dev/null)"
p1="$(sed -n '2p' "${PID_FILE}" 2>/dev/null)"
[[ "${p0}" =~ ^[0-9]+$ ]] || return 1
[[ "${p1}" =~ ^[0-9]+$ ]] || return 1
kill -0 "${p0}" 2>/dev/null || kill -0 "${p1}" 2>/dev/null || return 1
printf '%s %s' "${p0}" "${p1}"
return 0
}
trap on_signal INT TERM ###############################################################################
# Report
###############################################################################
# The recent entries for one NIC: replies and, thanks to -O, the requests that
# got none.
tail_entries() {
grep -aE 'bytes from|no answer|Unreachable|Time to live' "$1" 2>/dev/null \
| tail -n "${TAIL_LINES}"
}
# The trailing "--- x ping statistics ---" block.
stats_block() {
sed -n '/ping statistics ---/,$p' "$1" 2>/dev/null
}
# render_leg <label> <iface> <target> <raw> <rx0> <tx0>
render_leg() {
local label="$1" iface="$2" target="$3" raw="$4" rx0="$5" tx0="$6"
local st tx rx loss avg verdict c1 rx1 tx1 drx dtx
printf -- '----- %s : %s -> %s : last %s entries -----\n' \
"${label}" "${iface}" "${target}" "${TAIL_LINES}"
tail_entries "${raw}"
printf '\n'
st="$(stats_block "${raw}")"
printf '%s\n' "${st}"
tx="$(printf '%s' "${st}" | sed -n 's/^\([0-9]\+\) packets transmitted.*/\1/p' | tail -1)"
rx="$(printf '%s' "${st}" | sed -n 's/.*[, ]\([0-9]\+\) received.*/\1/p' | tail -1)"
loss="$(printf '%s' "${st}" | sed -n 's/.*[, ]\([0-9]\+\)% packet loss.*/\1/p' | tail -1)"
avg="$(printf '%s' "${st}" | sed -n 's|.*= [0-9.]*/\([0-9.]*\)/.*|\1|p' | tail -1)"
[ -n "${tx}" ] || tx=0
[ -n "${rx}" ] || rx=0
[ -n "${loss}" ] || loss=100
[ -n "${avg}" ] || avg="-"
c1="$(if_counters "${iface}")"; rx1="${c1% *}"; tx1="${c1#* }"
drx=$(( rx1 - rx0 )); dtx=$(( tx1 - tx0 ))
if [ "${loss}" -le "${MAX_LOSS_PCT}" ] && [ "${tx}" -gt 0 ]; then
verdict="PASS"
else
verdict="FAIL"
fi
# nic_tx/nic_rx are this interface's own counter delta over the whole run.
# A pass with nic_tx near zero means the traffic left on the other NIC.
printf '[%s] RESULT %-3s %-6s -> %-15s tx=%s rx=%s loss=%s%% rtt_avg=%sms nic_tx=%s nic_rx=%s %s\n\n' \
"$(ts)" "${label}" "${iface}" "${target}" "${tx}" "${rx}" "${loss}" "${avg}" \
"${dtx}" "${drx}" "${verdict}"
}
write_report() {
local rx0_10g tx0_10g rx0_1g tx0_1g started
started="$(sed -n '1p' "${STATE_FILE}" 2>/dev/null)"
rx0_10g="$(sed -n '2p' "${STATE_FILE}" 2>/dev/null)"; rx0_10g="${rx0_10g:-0}"
tx0_10g="$(sed -n '3p' "${STATE_FILE}" 2>/dev/null)"; tx0_10g="${tx0_10g:-0}"
rx0_1g="$(sed -n '4p' "${STATE_FILE}" 2>/dev/null)"; rx0_1g="${rx0_1g:-0}"
tx0_1g="$(sed -n '5p' "${STATE_FILE}" 2>/dev/null)"; tx0_1g="${tx0_1g:-0}"
{
printf '#############################################################\n' printf '#############################################################\n'
printf '[%s] START: pid=%d count=%d settle=%ds interval=%ds\n' \ printf '[%s] mgmt ping report\n' "$(ts)"
"$(ts)" "$$" "${PING_COUNT}" "${LINK_SETTLE_SEC}" "${INTERVAL_SEC}" printf ' started : %s\n' "${started:-unknown}"
printf '[%s] 10G %s %s/%s -> %s (metric %s)\n' \ printf ' stopped : %s\n' "$(ts)"
"$(ts)" "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}" printf ' 10G : %s %s/%s -> %s (metric %s)\n' \
printf '[%s] 1G %s %s/%s -> %s (metric %s)\n' \ "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}"
"$(ts)" "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}" printf ' 1G : %s %s/%s -> %s (metric %s)\n' \
printf '#############################################################\n' "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}"
printf ' ping : -i %s %s, max loss %s%%\n' \
"${PING_INTERVAL_SEC}" "${PING_EXTRA_OPTS}" "${MAX_LOSS_PCT}"
printf '#############################################################\n\n'
apply_arp_strict render_leg "10G" "${IF_10G}" "${TARGET_10G}" "${RAW_10G}" "${rx0_10g}" "${tx0_10g}"
render_leg "1G" "${IF_1G}" "${TARGET_1G}" "${RAW_1G}" "${rx0_1g}" "${tx0_1g}"
while [ "${STOP}" -eq 0 ]; do printf -- '--- ip -s link show %s ---\n' "${IF_10G}"
round=$(( round + 1 )) ${SUDO} ip -s link show "${IF_10G}" 2>&1
printf -- '---------- ROUND %d @ %s ----------\n' "${round}" "$(ts)" printf '\n'
printf -- '--- ip -s link show %s ---\n' "${IF_1G}"
if ping_leg "10G" "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${GW_10G}" \ ${SUDO} ip -s link show "${IF_1G}" 2>&1
"${METRIC_10G}" "${TARGET_10G}"; then } >> "${LOG_FILE}"
pass=$(( pass + 1 )); else fail=$(( fail + 1 ))
fi
[ "${STOP}" -eq 0 ] || break
if ping_leg "1G" "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${GW_1G}" \
"${METRIC_1G}" "${TARGET_1G}"; then
pass=$(( pass + 1 )); else fail=$(( fail + 1 ))
fi
rotate_log_if_needed
if [ "${MAX_ROUNDS}" -gt 0 ] && [ "${round}" -ge "${MAX_ROUNDS}" ]; then
break
fi
[ "${STOP}" -eq 0 ] && interruptible_sleep "${INTERVAL_SEC}"
done
printf '[%s] STOP : pid=%d after %d round(s), legs PASS=%d FAIL=%d\n' \
"$(ts)" "$$" "${round}" "${pass}" "${fail}"
} }
############################################################################### ###############################################################################
# Sub-commands # Sub-commands
############################################################################### ###############################################################################
do_start() { prepare_log_on_start() {
local pid local stamp
pid="$(is_running)" && die "already running (pid=${pid})" case "${START_LOG_MODE}" in
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}" append) return 0 ;;
prepare_log_on_start archive)
if [ -s "${LOG_FILE}" ]; then
if command -v setsid >/dev/null 2>&1; then stamp="$(date '+%Y%m%d-%H%M%S')"
setsid "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 & mv -f "${LOG_FILE}" "${LOG_FILE}.${stamp}" || die "cannot archive ${LOG_FILE}"
else printf 'previous log archived: %s.%s\n' "${LOG_FILE}" "${stamp}"
nohup "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 &
disown 2>/dev/null
fi fi
;;
sleep 1 new)
pid="$(is_running)" || die "start failed, check ${LOG_FILE}" # Truncate rather than unlink, so a tail -f already attached keeps
printf 'started (pid=%s)\nlog: %s\n' "${pid}" "${LOG_FILE}" # following the new run.
[ -f "${LOG_FILE}" ] && { : > "${LOG_FILE}" || die "cannot truncate ${LOG_FILE}"; }
;;
*) die "invalid START_LOG_MODE: ${START_LOG_MODE}" ;;
esac
} }
do_daemon() { do_start() {
local pids p0 p1 c
pids="$(is_running)" && die "already running (pids=${pids})"
command -v ip >/dev/null 2>&1 || die "ip (iproute2) not found"
command -v ping >/dev/null 2>&1 || die "ping not found"
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}" mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
LOG_TO_FILE=1 prepare_log_on_start
exec >>"${LOG_FILE}" 2>&1 : > "${RAW_10G}"
printf '%d\n' "$$" > "${PID_FILE}" : > "${RAW_1G}"
trap 'rm -f "${PID_FILE}"' EXIT
main_loop setup_iface "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${GW_10G}" "${METRIC_10G}" \
|| die "cannot configure ${IF_10G}"
setup_iface "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${GW_1G}" "${METRIC_1G}" \
|| die "cannot configure ${IF_1G}"
apply_arp_strict
sleep "${LINK_SETTLE_SEC}"
# Spend the ARP resolution here, not on the first measured packet.
arp_warmup "${IF_10G}" "${TARGET_10G}"
arp_warmup "${IF_1G}" "${TARGET_1G}"
# Counter baseline, taken after the warm-up so its packets are excluded.
{ c="$(if_counters "${IF_10G}")"
printf '%s\n%s\n%s\n' "$(ts)" "${c% *}" "${c#* }"
c="$(if_counters "${IF_1G}")"
printf '%s\n%s\n' "${c% *}" "${c#* }"
} > "${STATE_FILE}"
# Both NICs ping at the same time and keep accumulating until stopped.
nohup ping -I "${IF_10G}" -i "${PING_INTERVAL_SEC}" ${PING_EXTRA_OPTS} \
"${TARGET_10G}" >> "${RAW_10G}" 2>&1 &
p0=$!
nohup ping -I "${IF_1G}" -i "${PING_INTERVAL_SEC}" ${PING_EXTRA_OPTS} \
"${TARGET_1G}" >> "${RAW_1G}" 2>&1 &
p1=$!
disown "${p0}" 2>/dev/null
disown "${p1}" 2>/dev/null
printf '%s\n%s\n' "${p0}" "${p1}" > "${PID_FILE}"
sleep 1
kill -0 "${p0}" 2>/dev/null || die "10G ping failed to start, see ${RAW_10G}"
kill -0 "${p1}" 2>/dev/null || die "1G ping failed to start, see ${RAW_1G}"
printf 'started (10G pid=%s, 1G pid=%s)\nlog: %s\n' "${p0}" "${p1}" "${LOG_FILE}"
} }
do_stop() { do_stop() {
local pid i local pids p0 p1 i
pid="$(is_running)" || { printf 'not running\n'; rm -f "${PID_FILE}"; return 0; }
kill -TERM -"${pid}" 2>/dev/null || kill -TERM "${pid}" 2>/dev/null pids="$(is_running)" || {
printf 'not running\n'
rm -f "${PID_FILE}"
return 0
}
p0="${pids% *}"; p1="${pids#* }"
# SIGINT, not SIGTERM: ping prints its statistics block on interrupt, and
# that block is what the report parses.
kill -INT "${p0}" 2>/dev/null
kill -INT "${p1}" 2>/dev/null
for (( i = 0; i < 20; i++ )); do for (( i = 0; i < 20; i++ )); do
kill -0 "${pid}" 2>/dev/null || break kill -0 "${p0}" 2>/dev/null || kill -0 "${p1}" 2>/dev/null || break
sleep 0.5 sleep 0.5
done done
if kill -0 "${pid}" 2>/dev/null; then kill -KILL "${p0}" 2>/dev/null
printf 'SIGTERM ignored, sending SIGKILL\n' kill -KILL "${p1}" 2>/dev/null
kill -KILL -"${pid}" 2>/dev/null || kill -KILL "${pid}" 2>/dev/null
fi write_report
rm -f "${PID_FILE}" rm -f "${PID_FILE}"
printf 'stopped (pid=%s)\n' "${pid}" printf 'stopped (10G pid=%s, 1G pid=%s)\nreport: %s\n' "${p0}" "${p1}" "${LOG_FILE}"
} }
do_status() { do_status() {
local pid local pids
if pid="$(is_running)"; then if pids="$(is_running)"; then
printf 'status : running (pid=%s)\n' "${pid}" printf 'status : running (10G pid=%s, 1G pid=%s)\n' "${pids% *}" "${pids#* }"
else else
printf 'status : stopped\n' printf 'status : stopped\n'
fi fi
printf 'log : %s\n' "${LOG_FILE}" printf 'log : %s\n' "${LOG_FILE}"
[ -f "${LOG_FILE}" ] && printf 'size : %s\n' "$(du -h "${LOG_FILE}" | cut -f1)" printf '10G : %s %s/%s -> %s\n' "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}"
printf 'ping : count=%d, interval=%ds, settle=%ds\n' "${PING_COUNT}" "${INTERVAL_SEC}" "${LINK_SETTLE_SEC}" printf '1G : %s %s/%s -> %s\n' "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}"
printf '10G : %s %s/%s -> %s (metric %s)\n' "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}" [ -f "${RAW_10G}" ] && printf '10G recv: %s\n' "$(grep -ac 'bytes from' "${RAW_10G}")"
printf '1G : %s %s/%s -> %s (metric %s)\n' "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}" [ -f "${RAW_1G}" ] && printf '1G recv: %s\n' "$(grep -ac 'bytes from' "${RAW_1G}")"
} }
usage() { usage() {
cat <<EOF cat <<EOF
Usage: ${SCRIPT_NAME} {start|stop|status|fg|tail|clear|summary} Usage: ${SCRIPT_NAME} {start|stop|status|summary|clear}
start Run detached in the background; the previous log is discarded first start Configure both NICs, warm up ARP, then ping from BOTH at the same
(see START_LOG_MODE) time and keep accumulating. The previous log is discarded first.
stop Stop the background instance stop Stop both pings and render the report into the log
status Show pid / log path / configured NICs status Show pids, configured NICs and replies received so far
fg Run in the foreground, output to the terminal (for a quick check) summary Print just the RESULT lines from the log
tail tail -f the log file (does NOT return -- do not call from a macro) clear Remove the log and the raw captures (must be stopped first)
clear Remove the log file and all rotated copies (must be stopped first)
summary Print just the per-leg RESULT lines from the log
Log file : ${LOG_FILE} Log file : ${LOG_FILE}
One round = 10G leg then 1G leg, each pinging its own target. Both NICs are The report holds, per NIC, the last ${TAIL_LINES} entries and the ping
brought up and left up; addresses and routes are applied with statistics, then "ip -s link show" for both interfaces.
"ip address/route replace", so repeating a round does not stack duplicates.
If both NICs share a subnet, keep ARP_STRICT=1 -- otherwise a reply can come Both NICs are used at once, so management connectivity is in play -- drive
back on the NIC that did not send it. Check nic_tx/nic_rx on the RESULT line: this from the serial console. Edit the User Configurable Section at the top
they are that interface's own counter delta across the burst. to change interfaces, addresses, targets or the ping options.
Edit the User Configurable Section at the top to change interfaces, addresses,
targets, prefix lengths, gateways, metrics or the packet count.
EOF EOF
} }
@@ -449,15 +392,12 @@ case "${1:-}" in
start) do_start ;; start) do_start ;;
stop) do_stop ;; stop) do_stop ;;
status) do_status ;; status) do_status ;;
fg) LOG_TO_FILE=0; main_loop ;;
tail) tail -n 50 -f "${LOG_FILE}" ;;
summary) grep -a 'RESULT' "${LOG_FILE}" 2>/dev/null || printf 'no results in %s\n' "${LOG_FILE}" ;; summary) grep -a 'RESULT' "${LOG_FILE}" 2>/dev/null || printf 'no results in %s\n' "${LOG_FILE}" ;;
clear) clear)
is_running >/dev/null && die "still running, stop it first" is_running >/dev/null && die "still running, stop it first"
rm -f "${LOG_FILE}" "${LOG_FILE}".[0-9]* rm -f "${LOG_FILE}" "${LOG_FILE}".[0-9]* "${RAW_10G}" "${RAW_1G}" "${STATE_FILE}"
printf 'log cleared\n' printf 'log cleared\n'
;; ;;
__daemon) do_daemon ;;
-h|--help|"") usage ;; -h|--help|"") usage ;;
*) printf '[ERROR] unknown sub-command: %s\n\n' "$1" >&2; usage; exit 1 ;; *) printf '[ERROR] unknown sub-command: %s\n\n' "$1" >&2; usage; exit 1 ;;
esac esac
@@ -0,0 +1,431 @@
#!/bin/bash
###############################################################################
# port_prbs_monitor.sh
#
# Version : V1.0.0
# Author : ETWen
# Date : 20260826
#
# Purpose : Run a PRBS test on the two 100G uplinks AT THE SAME TIME and keep
# polling until stopped. Each port gets its own background worker:
#
# setup : sfputil lpmode off <port>
# phy diag <phy> prbs set <poly>
# phy diag <phy> prbsstat STArt Interval=<n>
# poll : phy diag <phy> prbs get <- PASS/FAIL comes from here
# phydiag <phy> prbsstat Ber <- recorded only
# stop : phydiag <phy> prbsstat STOp
# phydiag <phy> prbs clear
#
# Every command and its full output goes to that port's raw log.
# `stop` tears the test down and renders the report; `report`
# prints the PASS/FAIL tally per port.
#
# Notes : - A poll counts as PASS only when the output contains "PRBS OK!".
# bcmcmd exits 0 even when the BCM shell rejects a command, so the
# exit status cannot be used -- only the output can.
# - bcmcmd is always run with </dev/null. Without it, it inherits the
# polling loop's stdin and eats it, and the loop runs once.
#
# Version History
# V1.0.0 20260826 Initial Version
###############################################################################
set -u
###############################################################################
# User Configurable Section -- normally this is the only part you need to edit
###############################################################################
# --- port A ---
PORT_A_NAME="Ethernet513"
PORT_A_UNIT=0 # bcmcmd -n <unit>
PORT_A_PHY=268 # phy diag <phy>
# --- port B ---
PORT_B_NAME="Ethernet514"
PORT_B_UNIT=1
PORT_B_PHY=268
# Which ports `start` runs when no argument is given: both | a | b
# Override per run: start a / start b / start both
# Running one at a time is how you tell a genuine port fault from the DUT not
# coping with two PRBS streams at once.
PORTS="both"
# --- PRBS ---
PRBS_POLY="p=3" # passed to "prbs set"
PRBS_STAT_INTERVAL=5 # passed to "prbsstat STArt Interval="
POLL_INTERVAL_SEC=10 # seconds between polls
PASS_PATTERN="PRBS OK!" # a poll is PASS only if the output contains this
# Seconds to wait after the setup commands before the first poll, so the link
# has settled and the counters mean something.
SETTLE_SEC=10
# Stop after this many polls per port. 0 = run until stopped manually.
MAX_POLLS=0
# How many recent poll blocks the report shows per port.
TAIL_ENTRIES=20
# Prefix for privileged commands; empty because this already runs as root.
SUDO=""
BCMCMD="bcmcmd"
# Logging
LOG_DIR="" # empty = <script directory>/log
LOG_NAME="port_prbs.log"
# What `start` does with the log left behind by the previous run:
# new | archive | append
START_LOG_MODE="new"
###############################################################################
# Internal
###############################################################################
SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")"
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
SCRIPT_PATH="${SCRIPT_DIR}/${SCRIPT_NAME}"
[ -n "${LOG_DIR}" ] || LOG_DIR="${SCRIPT_DIR}/log"
LOG_FILE="${LOG_DIR}/${LOG_NAME}"
BASE="${LOG_DIR}/${LOG_NAME%.log}"
PID_FILE="${BASE}.pid"
STATE_FILE="${BASE}.state"
RAW_A="${BASE}_${PORT_A_NAME}.raw"
RAW_B="${BASE}_${PORT_B_NAME}.raw"
ts() { date '+%Y-%m-%d %H:%M:%S'; }
# port_tally <name> <raw> -> "<polls> <pass> <fail> <verdict>"
# A raw without a "PRBS START" line means the port was not run this session,
# which is SKIP -- reporting it as FAIL would make a deliberate single-port
# run look like half the hardware is broken.
port_tally() {
local name="$1" raw="$2" pass fail polls verdict
if [ ! -s "${raw}" ] || ! grep -aq "PRBS START ${name}" "${raw}" 2>/dev/null; then
printf '0 0 0 SKIP'; return 0
fi
pass=$(grep -ac "PRBS ${name} poll=[0-9]* PASS" "${raw}" 2>/dev/null); pass=${pass:-0}
fail=$(grep -ac "PRBS ${name} poll=[0-9]* FAIL" "${raw}" 2>/dev/null); fail=${fail:-0}
polls=$(( pass + fail ))
if [ "${polls}" -gt 0 ] && [ "${fail}" -eq 0 ]; then verdict=PASS; else verdict=FAIL; fi
printf '%s %s %s %s' "${polls}" "${pass}" "${fail}" "${verdict}"
}
die() { printf '[ERROR] %s\n' "$*" >&2; exit 1; }
# bcm <unit> <dsh command> -- output on stdout, never trusts the exit status.
# </dev/null so it cannot consume the caller's stdin.
bcm() {
local unit="$1" cmd="$2"
${SUDO} "${BCMCMD}" -n "${unit}" -c "dsh -c \"${cmd}\"" </dev/null 2>&1
}
# run_step <unit> <dsh command>
# The command and its full output are written to the LOG via stderr, because
# the worker has stderr pointed at the raw file. Only the output itself goes to
# stdout, so `out="$(run_step ...)"` captures the output without swallowing the
# log lines -- writing both to stdout would have put the whole record inside
# the variable and left the log with nothing but the poll verdicts.
run_step() {
local unit="$1" cmd="$2" out
out="$(bcm "${unit}" "${cmd}")"
{
printf '[%s] CMD : bcmcmd -n %s -c '\''dsh -c "%s"'\''\n' "$(ts)" "${unit}" "${cmd}"
printf '%s\n' "${out}"
} >&2
printf '%s' "${out}"
}
# Echo "<pid_a> <pid_b>" and return 0 while at least one worker is alive. A
# port that was not started this run is recorded as "-", so a single-port run
# is a first-class case rather than a half-broken two-port one.
is_running() {
local pa pb alive=0
[ -f "${PID_FILE}" ] || return 1
pa="$(sed -n '1p' "${PID_FILE}" 2>/dev/null)"; pa="${pa:--}"
pb="$(sed -n '2p' "${PID_FILE}" 2>/dev/null)"; pb="${pb:--}"
[[ "${pa}" =~ ^[0-9]+$ ]] && kill -0 "${pa}" 2>/dev/null && alive=1
[[ "${pb}" =~ ^[0-9]+$ ]] && kill -0 "${pb}" 2>/dev/null && alive=1
[ "${alive}" -eq 1 ] || return 1
printf '%s %s' "${pa}" "${pb}"
return 0
}
###############################################################################
# Worker -- one per port, launched by `start`
###############################################################################
WORKER_STOP=0
worker_on_signal() { WORKER_STOP=1; }
# __worker <name> <unit> <phy> <rawfile>
do_worker() {
local name="$1" unit="$2" phy="$3" raw="$4"
local n=0 pass=0 fail=0 out
exec >>"${raw}" 2>&1
trap worker_on_signal INT TERM
printf '#############################################################\n'
printf '[%s] PRBS START %s (unit %s, phy %s) poly=%s interval=%ss poll=%ss\n' \
"$(ts)" "${name}" "${unit}" "${phy}" "${PRBS_POLY}" \
"${PRBS_STAT_INTERVAL}" "${POLL_INTERVAL_SEC}"
printf '#############################################################\n'
# --- setup ---
printf '[%s] CMD : sfputil lpmode off %s\n' "$(ts)" "${name}"
${SUDO} sfputil lpmode off "${name}" </dev/null 2>&1
run_step "${unit}" "phy diag ${phy} prbs set ${PRBS_POLY}" >/dev/null
run_step "${unit}" "phy diag ${phy} prbsstat STArt Interval=${PRBS_STAT_INTERVAL}" >/dev/null
sleep "${SETTLE_SEC}"
# --- poll ---
while [ "${WORKER_STOP}" -eq 0 ]; do
n=$(( n + 1 ))
printf -- '---------- %s POLL %d @ %s ----------\n' "${name}" "${n}" "$(ts)"
out="$(run_step "${unit}" "phy diag ${phy} prbs get")"
# Ber is recorded for the log only; it does not decide the verdict.
run_step "${unit}" "phydiag ${phy} prbsstat Ber" >/dev/null
if printf '%s' "${out}" | grep -qF "${PASS_PATTERN}"; then
pass=$(( pass + 1 ))
printf '[%s] PRBS %s poll=%d PASS\n' "$(ts)" "${name}" "${n}"
else
fail=$(( fail + 1 ))
printf '[%s] PRBS %s poll=%d FAIL\n' "$(ts)" "${name}" "${n}"
fi
[ "${MAX_POLLS}" -gt 0 ] && [ "${n}" -ge "${MAX_POLLS}" ] && break
[ "${WORKER_STOP}" -eq 0 ] || break
sleep "${POLL_INTERVAL_SEC}"
done
# --- teardown ---
run_step "${unit}" "phydiag ${phy} prbsstat STOp" >/dev/null
run_step "${unit}" "phydiag ${phy} prbs clear" >/dev/null
printf '[%s] PRBS STOP %s polls=%d PASS=%d FAIL=%d\n' \
"$(ts)" "${name}" "${n}" "${pass}" "${fail}"
}
###############################################################################
# Report
###############################################################################
# render_port <name> <unit> <phy> <raw>
render_port() {
local name="$1" unit="$2" phy="$3" raw="$4"
local t polls pass fail verdict
t="$(port_tally "${name}" "${raw}")"
polls="$(printf '%s' "${t}" | awk '{print $1}')"
pass="$(printf '%s' "${t}" | awk '{print $2}')"
fail="$(printf '%s' "${t}" | awk '{print $3}')"
verdict="$(printf '%s' "${t}" | awk '{print $4}')"
if [ "${verdict}" = "SKIP" ]; then
printf -- '----- %s (unit %s, phy %s) : not run this session -----\n\n' \
"${name}" "${unit}" "${phy}"
printf '[%s] RESULT %-12s polls=%-4s PASS=%-4s FAIL=%-4s %s\n\n' \
"$(ts)" "${name}" "${polls}" "${pass}" "${fail}" "${verdict}"
return 0
fi
printf -- '----- %s (unit %s, phy %s) : last %s poll blocks -----\n' \
"${name}" "${unit}" "${phy}" "${TAIL_ENTRIES}"
grep -aE "^-{10} ${name} POLL|PRBS ${name} poll=|prbsstat Ber|^ *[0-9]+ *: " "${raw}" 2>/dev/null \
| tail -n $(( TAIL_ENTRIES * 3 ))
printf '\n'
printf '[%s] RESULT %-12s polls=%-4s PASS=%-4s FAIL=%-4s %s\n\n' \
"$(ts)" "${name}" "${polls}" "${pass}" "${fail}" "${verdict}"
}
write_report() {
local started
started="$(sed -n '1p' "${STATE_FILE}" 2>/dev/null)"
{
printf '#############################################################\n'
printf '[%s] 100G PRBS report\n' "$(ts)"
printf ' started : %s\n' "${started:-unknown}"
printf ' stopped : %s\n' "$(ts)"
printf ' %s : unit %s, phy %s\n' "${PORT_A_NAME}" "${PORT_A_UNIT}" "${PORT_A_PHY}"
printf ' %s : unit %s, phy %s\n' "${PORT_B_NAME}" "${PORT_B_UNIT}" "${PORT_B_PHY}"
printf ' poly=%s prbsstat Interval=%s poll every %ss\n' \
"${PRBS_POLY}" "${PRBS_STAT_INTERVAL}" "${POLL_INTERVAL_SEC}"
printf ' a poll is PASS only when the output contains "%s"\n' "${PASS_PATTERN}"
printf '#############################################################\n\n'
render_port "${PORT_A_NAME}" "${PORT_A_UNIT}" "${PORT_A_PHY}" "${RAW_A}"
render_port "${PORT_B_NAME}" "${PORT_B_UNIT}" "${PORT_B_PHY}" "${RAW_B}"
# Taken after prbsstat STOp + prbs clear, so this is the recovered
# state. During the test the same command would have reported DOWN.
printf -- '--- show interfaces status %s,%s (after prbsstat STOp + prbs clear) ---\n' \
"${PORT_A_NAME}" "${PORT_B_NAME}"
${SUDO} show interfaces status "${PORT_A_NAME},${PORT_B_NAME}" </dev/null 2>&1
} >> "${LOG_FILE}"
}
###############################################################################
# Sub-commands
###############################################################################
prepare_log_on_start() {
local stamp
case "${START_LOG_MODE}" in
append) return 0 ;;
archive)
if [ -s "${LOG_FILE}" ]; then
stamp="$(date '+%Y%m%d-%H%M%S')"
mv -f "${LOG_FILE}" "${LOG_FILE}.${stamp}" || die "cannot archive ${LOG_FILE}"
fi ;;
new)
[ -f "${LOG_FILE}" ] && { : > "${LOG_FILE}" || die "cannot truncate ${LOG_FILE}"; } ;;
*) die "invalid START_LOG_MODE: ${START_LOG_MODE}" ;;
esac
}
# do_start [both|a|b|<port name>]
do_start() {
local which="${1:-${PORTS}}" pids pa="-" pb="-" run_a=0 run_b=0
case "${which}" in
both|BOTH|all) run_a=1; run_b=1 ;;
a|A|"${PORT_A_NAME}") run_a=1 ;;
b|B|"${PORT_B_NAME}") run_b=1 ;;
*) die "unknown port selector '${which}' (use: both | a | b | ${PORT_A_NAME} | ${PORT_B_NAME})" ;;
esac
pids="$(is_running)" && die "already running (pids=${pids})"
command -v "${BCMCMD}" >/dev/null 2>&1 || die "${BCMCMD} not found"
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
prepare_log_on_start
# Only the selected ports are truncated. A raw with no "PRBS START" line is
# what the report uses to tell "not run this session" from "ran and failed".
[ "${run_a}" -eq 1 ] && : > "${RAW_A}"
[ "${run_b}" -eq 1 ] && : > "${RAW_B}"
printf '%s\n' "$(ts)" > "${STATE_FILE}"
# One worker per selected port; with both, they run at the same time.
# Invoked through bash so the file does not need the execute bit.
if [ "${run_a}" -eq 1 ]; then
setsid bash "${SCRIPT_PATH}" __worker \
"${PORT_A_NAME}" "${PORT_A_UNIT}" "${PORT_A_PHY}" "${RAW_A}" >/dev/null 2>&1 &
pa=$!
disown "${pa}" 2>/dev/null
fi
if [ "${run_b}" -eq 1 ]; then
setsid bash "${SCRIPT_PATH}" __worker \
"${PORT_B_NAME}" "${PORT_B_UNIT}" "${PORT_B_PHY}" "${RAW_B}" >/dev/null 2>&1 &
pb=$!
disown "${pb}" 2>/dev/null
fi
printf '%s\n%s\n' "${pa}" "${pb}" > "${PID_FILE}"
sleep 1
[ "${run_a}" -eq 1 ] && { kill -0 "${pa}" 2>/dev/null || die "${PORT_A_NAME} worker failed, see ${RAW_A}"; }
[ "${run_b}" -eq 1 ] && { kill -0 "${pb}" 2>/dev/null || die "${PORT_B_NAME} worker failed, see ${RAW_B}"; }
printf 'started (%s pid=%s, %s pid=%s)\nlog: %s\n' \
"${PORT_A_NAME}" "${pa}" "${PORT_B_NAME}" "${pb}" "${LOG_FILE}"
# Deliberately NOT printing "show interfaces status" here. With PRBS armed
# the link is out of normal operation and reports DOWN, which reads as a
# failure to anyone glancing at the console. The status is shown in the
# report instead, once PRBS has been cleared.
}
do_stop() {
local pids pa pb i
pids="$(is_running)" || { printf 'not running\n'; rm -f "${PID_FILE}"; return 0; }
pa="${pids% *}"; pb="${pids#* }"
# TERM lets each worker finish its poll and run prbsstat STOp / prbs clear.
[[ "${pa}" =~ ^[0-9]+$ ]] && kill -TERM "${pa}" 2>/dev/null
[[ "${pb}" =~ ^[0-9]+$ ]] && kill -TERM "${pb}" 2>/dev/null
for (( i = 0; i < 120; i++ )); do
kill -0 "${pa}" 2>/dev/null || kill -0 "${pb}" 2>/dev/null || break
sleep 0.5
done
if kill -0 "${pa}" 2>/dev/null || kill -0 "${pb}" 2>/dev/null; then
printf 'workers did not exit in time, sending SIGKILL (PRBS may be left running)\n' >&2
kill -KILL "${pa}" 2>/dev/null
kill -KILL "${pb}" 2>/dev/null
fi
write_report
rm -f "${PID_FILE}"
printf 'stopped\nreport: %s\n' "${LOG_FILE}"
}
do_report() {
local name raw pass fail polls verdict
printf '%-14s %-8s %-8s %-8s %s\n' PORT POLLS PASS FAIL RESULT
printf '%-14s %-8s %-8s %-8s %s\n' -------------- -------- -------- -------- ------
for spec in "${PORT_A_NAME}:${RAW_A}" "${PORT_B_NAME}:${RAW_B}"; do
name="${spec%%:*}"; raw="${spec#*:}"
set -- $(port_tally "${name}" "${raw}")
printf '%-14s %-8s %-8s %-8s %s\n' "${name}" "$1" "$2" "$3" "$4"
done
}
do_status() {
local pids
if pids="$(is_running)"; then
printf 'status : running (%s pid=%s, %s pid=%s)\n' \
"${PORT_A_NAME}" "${pids% *}" "${PORT_B_NAME}" "${pids#* }"
else
printf 'status : stopped\n'
fi
printf 'log : %s\n' "${LOG_FILE}"
do_report
}
usage() {
cat <<EOF
Usage: ${SCRIPT_NAME} {start [both|a|b]|stop|status|report|clear}
start Set up PRBS and poll in the background. With no argument it runs
\$PORTS (currently "${PORTS}"); "a" or "b" runs that port alone,
which is how you tell a genuine port fault from the DUT not coping
with two PRBS streams at once.
start both ports at the same time
start a ${PORT_A_NAME} only
start b ${PORT_B_NAME} only
Port status is NOT shown here: with PRBS armed the link reports
DOWN, which looks like a failure. See the report instead.
stop Stop polling, run prbsstat STOp + prbs clear on both ports, and
render the report into the log
report Print the PASS/FAIL tally per port
status Show worker pids plus the current tally
clear Remove the log and raw captures (must be stopped first)
Log file : ${LOG_FILE}
Raw : ${RAW_A}
${RAW_B}
A poll is PASS only when "phy diag <phy> prbs get" reports "${PASS_PATTERN}".
prbsstat Ber is captured alongside every poll but does not decide the verdict.
A port that was not run in this session reports SKIP, not FAIL.
EOF
}
###############################################################################
# Entry point
###############################################################################
case "${1:-}" in
start) shift; do_start "${1:-${PORTS}}" ;;
stop) do_stop ;;
status) do_status ;;
report) do_report ;;
clear)
is_running >/dev/null && die "still running, stop it first"
rm -f "${LOG_FILE}" "${LOG_FILE}".[0-9]* "${RAW_A}" "${RAW_B}" "${STATE_FILE}"
printf 'log cleared\n' ;;
__worker)
shift
do_worker "$1" "$2" "$3" "$4" ;;
-h|--help|"") usage ;;
*) printf '[ERROR] unknown sub-command: %s\n\n' "$1" >&2; usage; exit 1 ;;
esac
@@ -0,0 +1,203 @@
#!/bin/bash
###############################################################################
# usb_target.sh
#
# Version : V1.1.0
# Author : ETWen
# Date : 20260825
# Purpose : Auto-detect an inserted USB mass-storage device and report either
# its device node, its mount point, or both.
#
# stdout : requested value(s) only, for $(...) capture
# stderr : diagnostic messages
#
# Usage : usb_target.sh [-o dev|mnt|both|id] [-n] [-r seconds]
#
# -o dev print partition device node (e.g. /dev/sda1)
# -o mnt print mount point (e.g. /mnt/usb) [default]
# -o both print "<dev> <mnt>" on one line
# -o id print stable by-id path (/dev/disk/by-id/...)
# -n detect only, do not mount
# -r sec udev enumeration wait, default 15
#
# Exit : 0 success
# 1 no USB mass-storage found
# 2 multiple USB disks detected (refuse to guess)
# 3 no usable filesystem on the device
# 4 mount failed
# 5 mounted but not writable
#
# Version History
# V1.0.0 20260825 Initial Version
# V1.1.0 20260825 Add -o/-n/-r options; expose device node and by-id path
###############################################################################
set -u
MNT_BASE="/mnt/usb"
RETRY=15
OUTPUT="mnt"
DO_MOUNT=1
log() { echo "[usb] $*" >&2; } # diagnostics go to stderr; stdout stays clean
while getopts "o:nr:h" opt; do
case "$opt" in
o) OUTPUT="$OPTARG" ;;
n) DO_MOUNT=0 ;;
r) RETRY="$OPTARG" ;;
h) sed -n '3,30p' "$0" >&2; exit 0 ;;
*) log "invalid option"; exit 1 ;;
esac
done
shift $((OPTIND - 1))
case "$OUTPUT" in
dev|mnt|both|id) ;;
*) log "invalid -o value: $OUTPUT"; exit 1 ;;
esac
# "-o dev" / "-o id" alone does not require mounting.
[ "$OUTPUT" = "dev" ] && [ "$DO_MOUNT" = "1" ] && DO_MOUNT=0
[ "$OUTPUT" = "id" ] && [ "$DO_MOUNT" = "1" ] && DO_MOUNT=0
#------------------------------------------------------------------------------
# Identify the physical disk(s) backing rootfs / /host so they can be excluded.
# Needed because some platforms boot from a USB DOM, which also reports TRAN=usb.
#------------------------------------------------------------------------------
get_system_disk() {
local src
for mp in /host / ; do
src=$(findmnt -no SOURCE "$mp" 2>/dev/null) || continue
lsblk -no PKNAME "$src" 2>/dev/null | head -1
done | sort -u
}
#------------------------------------------------------------------------------
# List candidate USB disks (whole devices, not partitions).
#------------------------------------------------------------------------------
find_usb_disks() {
local sysdisks; sysdisks=$(get_system_disk)
lsblk -dn -o NAME,TYPE,TRAN,RM 2>/dev/null | while read -r name type tran rm; do
[ "$type" = "disk" ] || continue
case "$name" in loop*|ram*|dm-*|sr*|zram*) continue ;; esac
# Older util-linux may not expose the TRAN column; fall back to sysfs.
if [ "$tran" != "usb" ]; then
readlink -f "/sys/block/$name/device" 2>/dev/null | grep -q '/usb[0-9]' || continue
[ "$rm" = "1" ] || continue
fi
echo "$sysdisks" | grep -qx "$name" && { log "skip $name (system disk)"; continue; }
echo "/dev/$name"
done
}
#------------------------------------------------------------------------------
# Pick a mountable partition from a disk; fall back to the whole device for
# superfloppy layouts (filesystem written directly, no partition table).
#------------------------------------------------------------------------------
pick_partition() {
local disk="$1" p
p=$(lsblk -ln -o NAME,TYPE,FSTYPE "$disk" | \
awk '$2=="part" && $3!="" {print "/dev/"$1; exit}')
[ -n "$p" ] && { echo "$p"; return; }
[ -n "$(lsblk -dn -o FSTYPE "$disk")" ] && echo "$disk"
}
#------------------------------------------------------------------------------
# Resolve a device node to a stable /dev/disk/by-id path, if one exists.
#------------------------------------------------------------------------------
resolve_by_id() {
local dev="$1" real link
real=$(readlink -f "$dev")
for link in /dev/disk/by-id/*; do
[ -e "$link" ] || continue
case "$link" in *-part*|*) ;; esac
[ "$(readlink -f "$link")" = "$real" ] && { echo "$link"; return 0; }
done
return 1
}
#------------------------------------------------------------------------------
# Main
#------------------------------------------------------------------------------
disks=""
for i in $(seq 1 "$RETRY"); do
disks=$(find_usb_disks)
[ -n "$disks" ] && break
sleep 1
done
[ -z "$disks" ] && { log "no USB mass-storage found"; exit 1; }
n=$(echo "$disks" | wc -l)
if [ "$n" -gt 1 ]; then
log "multiple USB disks detected, refuse to guess:"
log "$disks"
exit 2
fi
disk="$disks"
part=$(pick_partition "$disk")
[ -z "$part" ] && { log "$disk has no usable filesystem"; exit 3; }
fstype=$(lsblk -no FSTYPE "$part")
log "found $part (disk=$disk fstype=$fstype)"
# --- device-node only: no mount needed ---------------------------------------
if [ "$OUTPUT" = "dev" ]; then
echo "$part"
exit 0
fi
if [ "$OUTPUT" = "id" ]; then
if byid=$(resolve_by_id "$part"); then
echo "$byid"
exit 0
fi
log "no by-id path for $part, falling back to device node"
echo "$part"
exit 0
fi
# --- mount point required ----------------------------------------------------
mp=$(lsblk -no MOUNTPOINT "$part" | head -1)
if [ -z "$mp" ] && [ "$DO_MOUNT" = "0" ]; then
log "$part is not mounted and -n was given"
exit 4
fi
if [ -z "$mp" ]; then
log "mounting $part -> $MNT_BASE"
mkdir -p "$MNT_BASE"
# exfat/ntfs are not always built into the platform kernel.
case "$fstype" in
exfat) grep -qw exfat /proc/filesystems || modprobe exfat 2>/dev/null ;;
ntfs) grep -qw ntfs3 /proc/filesystems || modprobe ntfs3 2>/dev/null ;;
esac
if ! mount -o rw,noatime "$part" "$MNT_BASE" 2>/dev/null; then
log "mount failed (fstype=$fstype)"
exit 4
fi
mp="$MNT_BASE"
# Verify it is actually writable (read-only media, dirty FAT, or full device).
if ! touch "$mp/.wtest" 2>/dev/null; then
log "mounted read-only or no space left"
umount "$mp"
exit 5
fi
rm -f "$mp/.wtest"
else
log "$part already mounted at $mp"
fi
if [ "$OUTPUT" = "both" ]; then
echo "$part $mp"
else
echo "$mp"
fi
exit 0
+63
View File
@@ -0,0 +1,63 @@
echo PORTS0=$(bcmcmd -n 0 -c ps | grep -w up | wc -l)
echo PORTS0=$(bcmcmd -n 1 -c ps | grep -w up | wc -l)
show interfaces status Ethernet513,Ethernet514
//100G Port 513
sudo sfputil lpmode off Ethernet513
bcmcmd -n 0 -c 'dsh -c "phy diag 268 prbs set p=3"'
bcmcmd -n 0 -c 'dsh -c "phy diag 268 prbsstat STArt Interval=5"'
bcmcmd -n 0 -c 'dsh -c "phy diag 268 prbs get"'
bcmcmd -n 0 -c 'dsh -c "phydiag 268 prbsstat Ber"'
bcmcmd -n 0 -c 'dsh -c "phydiag 268 prbsstat STOp"'
bcmcmd -n 0 -c 'dsh -c "phydiag 268 prbs clear"'
//100G Port 514
sudo sfputil lpmode off Ethernet514
bcmcmd -n 1 -c 'dsh -c "phy diag 268 prbs set p=3"'
bcmcmd -n 1 -c 'dsh -c "phy diag 268 prbsstat STArt Interval=5"'
bcmcmd -n 1 -c 'dsh -c "phy diag 268 prbs get"'
bcmcmd -n 1 -c 'dsh -c "phydiag 268 prbsstat Ber"'
bcmcmd -n 1 -c 'dsh -c "phydiag 268 prbsstat STOp"'
bcmcmd -n 1 -c 'dsh -c "phydiag 268 prbs clear"'
//100G Port 513
sudo sfputil lpmode off Ethernet513
sudo sfputil lpmode off Ethernet514
bcmcmd -n 0 -c 'dsh -c "phy diag 268 prbs set p=3"'
sleep 1
bcmcmd -n 1 -c 'dsh -c "phy diag 268 prbs set p=3"'
bcmcmd -n 0 -c 'dsh -c "phy diag 268 prbsstat STArt Interval=5"'
sleep 1
bcmcmd -n 1 -c 'dsh -c "phy diag 268 prbsstat STArt Interval=5"'
bcmcmd -n 0 -c 'dsh -c "phy diag 268 prbs get"'
sleep 1
bcmcmd -n 1 -c 'dsh -c "phy diag 268 prbs get"'
bcmcmd -n 0 -c 'dsh -c "phydiag 268 prbsstat Ber"'
sleep 1
bcmcmd -n 1 -c 'dsh -c "phydiag 268 prbsstat Ber"'
bcmcmd -n 0 -c 'dsh -c "phydiag 268 prbsstat STOp"'
sleep 1
bcmcmd -n 1 -c 'dsh -c "phydiag 268 prbsstat STOp"'
bcmcmd -n 0 -c 'dsh -c "phydiag 268 prbs clear"'
sleep 1
bcmcmd -n 1 -c 'dsh -c "phydiag 268 prbs clear"'
+32
View File
@@ -0,0 +1,32 @@
bcmcmd -n 0 -c '*:port all lb=mac'
bcmcmd -n 0 -c '*:l2 learn off'
bcmcmd -n 0 -c '*:dsh -c "test mode nr=yes"'
bcmcmd -n 0 -c '*:dsh -c "tr 518 Scenario=6 Profile=2 Count=260 PktSize=324 PortList=1-259,270-565 rm=ext CheckLinkTime=20 CheckDataTime=200 testphase=2"'
bcmcmd -n 1 -c "*:port all lb=mac"
bcmcmd -n 1 -c "*:l2 learn off"
bcmcmd -n 1 -c "*:dsh -c 'test mode nr=yes'"
bcmcmd -n 1 -c "*:dsh -c 'tr 518 Scenario=6 Profile=2 Count=260 PktSize=324 PortList=1-259,270-565 rm=ext CheckLinkTime=20 CheckDataTime=200 testphase=2'"
bcmcmd -n 0 -c "*:port all lb=mac"
sleep 1
bcmcmd -n 1 -c "*:port all lb=mac"
sleep 1
bcmcmd -n 0 -c "*:l2 learn off"
sleep 1
bcmcmd -n 1 -c "*:l2 learn off"
sleep 1
bcmcmd -n 0 -c "*:dsh -c 'test mode nr=yes'"
sleep 1
bcmcmd -n 1 -c "*:dsh -c 'test mode nr=yes'"
sleep 1
bcmcmd -n 0 -c "*:dsh -c 'tr 518 Scenario=6 Profile=2 Count=260 PktSize=324 PortList=1-259,270-565 rm=ext CheckLinkTime=20 CheckDataTime=200 testphase=2'"
sleep 1
bcmcmd -n 1 -c "*:dsh -c 'tr 518 Scenario=6 Profile=2 Count=260 PktSize=324 PortList=1-259,270-565 rm=ext CheckLinkTime=20 CheckDataTime=200 testphase=2'"
sleep 1
root@(none):/usr/local/bin# swutil *:l2 learn off
root@(none):/usr/local/bin# swutil *:dsh -c \"test mode nr=yes\"
root@(none):/usr/local/bin# swutil *:dsh -c \"tr 518 Scenario=6 Profile=2 Count=260 PktSize=324 PortList=1-259,270-565 rm=ext CheckLinkTime=20 CheckDataTime=200 testphase=2"
+7
View File
@@ -0,0 +1,7 @@
echo 18-0020 > /sys/bus/i2c/drivers/max31790/unbind
sleep .5
echo 18-0020 > /sys/bus/i2c/drivers/max31790/bind
echo 25-0020 > /sys/bus/i2c/drivers/max31790/unbind
sleep .5
echo 25-0020 > /sys/bus/i2c/drivers/max31790/bind
fan-speed-control.sh 30
+59
View File
@@ -0,0 +1,59 @@
#!/bin/bash
# =============================================================================
# gen_channel_map.sh - Build docs/LTC2980_channel_map.csv from settings/*.conf
# Version : V1.0.0
# Date : 2026-08-27
# Author : ETWen
# =============================================================================
# Runs on the DEV machine, not the DUT. Reads every LTC2980 margin settings
# file and flattens CH<n>_VNOM / CH<n>_NET into one CSV:
#
# Board,CONN,Ch,NetName,Vnom
# CB,CONN13,CH0,V5P0_ALW,5.0
#
# NC channels keep their literal NC / "-" so the page numbering still lines up
# (ch/8 -> CHIPS[] index, ch%8 -> LTC2977 PAGE). Do not filter them out.
#
# Usage:
# tools/gen_channel_map.sh # write docs/LTC2980_channel_map.csv
# tools/gen_channel_map.sh - # write to stdout
# =============================================================================
set -eu
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
SET_DIR="${REPO_ROOT}/src/Script_ABC_Blanton/Blanton_Script/LTC2980_Margin_Script/settings"
OUT="${1:-${REPO_ROOT}/docs/LTC2980_channel_map.csv}"
[ -d "${SET_DIR}" ] || { echo "settings dir not found: ${SET_DIR}" >&2; exit 1; }
# 3-argument match(str, regex, array) is a gawk extension. Under mawk it is a
# syntax error at best and an empty CSV at worst - fail loudly instead.
awk 'BEGIN { if (match("a", /a/, m) != 1) exit 1 }' 2>/dev/null \
|| { echo "need gawk (3-arg match); this awk is: $(awk --version 2>&1 | head -1)" >&2; exit 1; }
gen() {
printf 'Board,CONN,Ch,NetName,Vnom\n'
# CB first, then SWB0/SWB1 CONN13..CONN16 - matches the physical walk order.
for f in "${SET_DIR}"/Blanton_CB_CONN*.conf \
"${SET_DIR}"/Blanton_SWB0_CONN*.conf \
"${SET_DIR}"/Blanton_SWB1_CONN*.conf ; do
[ -f "${f}" ] || continue
base="$(basename "${f}" .conf)" # Blanton_SWB0_CONN13
board="$(echo "${base}" | cut -d_ -f2)" # SWB0
conn="$(echo "${base}" | cut -d_ -f3)" # CONN13
# .conf files are LF per .gitattributes, but strip CR anyway - a stray
# CR would land inside the net name and travel into the CSV unseen.
sed 's/\r$//' "${f}" | awk -v b="${board}" -v c="${conn}" '
match($0, /^CH([0-9]+)_VNOM="([^"]*)"[^"]*CH[0-9]+_NET="([^"]*)"/, m) {
printf "%s,%s,CH%s,%s,%s\n", b, c, m[1], m[3], m[2]
}'
done
}
if [ "${OUT}" = "-" ]; then
gen
else
gen > "${OUT}"
echo "wrote ${OUT} ($(($(wc -l < "${OUT}") - 1)) channels)"
fi