10 Commits
Author SHA1 Message Date
etwenandClaude Opus 5 5e1e4a6a8f docs(release): Add v1.0.8 notes; wire mgmt ping status into B and C
Script A: the baseline ping window is 30 s rather than 45, and the log
is cleared afterwards so Script B starts from nothing.

Script B: `status` after `start`, and Script C: `status` before `stop`
plus a `cat` of the report. The status line shows both pids and the
replies received per NIC, so a dead link is visible while the soak is
still running instead of only at the end.

docs/release-notes/v1.0.8.md written in the house format.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-26 10:13:34 +08:00
etwenandClaude Opus 5 a31c186142 feat(mgmt): Ping both NICs at once, and stop counting ARP as loss
The bench run reported FAIL=3 with no NIC errors and no drops. All three
failures were on eth1 and all three were missing exactly icmp_seq=1,
with the other nine replies present and sub-millisecond. That is
neighbour resolution: once the ARP entry for the target expires, the
first echo request is spent resolving it and ping counts it as loss.
Rounds sat about 24 s apart, right on the edge of the default reachable
time, so it happened on some rounds and not others.

mgmt_ping_monitor.sh V3.0.0:

- One discarded ping per NIC before the measured run resolves ARP, so
  MAX_LOSS_PCT can stay at 0 and a FAIL means a real drop. Raising the
  threshold instead would have hidden genuine single-packet loss, and a
  longer settle would not help -- nothing triggers ARP until the ping.
- Both NICs now ping at the same time and keep accumulating until
  stopped, rather than alternating fixed bursts. Each writes its own raw
  capture.
- `stop` sends SIGINT rather than SIGTERM, because ping prints its
  statistics block on interrupt and that block is what the report
  parses, then renders into the log: the last TAIL_LINES entries per
  NIC, each one's statistics and verdict, and `ip -s link show` for both
  interfaces.
- `-D -O` are on by default so every line carries a timestamp and a
  request that got no reply is visible in the tail instead of merely
  absent. Clear PING_EXTRA_OPTS if the platform's ping rejects them.

Since the script no longer re-execs itself to daemonise, `bash
mgmt_ping_monitor.sh start` works without the execute bit.

.gitignore: the new *.raw / *.pid / *.state runtime files were outside
the *.log rule and would have been committed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-26 09:53:20 +08:00
etwenandClaude Opus 5 1b8d5a8e91 feat(usb): Detect the USB target instead of assuming /dev/sda1
New Blanton_Script/usb_target.sh reports a USB mass-storage device's
node, mount point or by-id path. It excludes whatever disk backs / and
/host -- this platform can boot from a USB DOM, which also reports
TRAN=usb -- refuses to guess when several USB disks are present, falls
back to the whole device for superfloppy media, and when mounting,
checks the result is actually writable rather than trusting that mount
succeeded.

Script B asks it for the device node and builds the stress command from
the answer, so a stick that enumerates as sdb no longer sends the write
test at whatever /dev/sda1 happens to be.

The device is carried back through an echo marker, split in the shell
literal as "<<D""EV=..." so the command echo cannot satisfy the pattern
the macro waits for -- the same false-match guard used for PORTS0= and
NCMIF=, arranged differently.

Script B: DDR memtester drops its 100-pass limit and runs until stopped,
matching the other soak loads.

Fix: Script C ran `bgctl reset --yes` while killing processes, which is
before the job logs are copied to USB. Moved to after the archive and
after hw-test-session finish, so the run's own logs are collected before
anything clears them.

Script A -> V1.0.8 history updated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-25 17:03:36 +08:00
etwenandClaude Opus 5 cac37583e7 feat(ttl): Include mgmt_ping.log in the USB archive
Script C now copies log/mgmt_ping.log into
/host/hw-eval/current/jobs/ alongside bmc_poll.log, so the management
link results leave the DUT with the run instead of staying behind. Both
monitors' logs are now covered.

Script A history: the two copies are one action in one block, so they
share a single V1.0.8 line rather than reading as separate changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-25 15:57:23 +08:00
etwenandClaude Opus 5 802ee4d467 feat(ttl): Fold bmc_poll.log into the USB archive
Script C copies log/bmc_poll.log into /host/hw-eval/current/jobs/ before
that directory is archived to USB, so the BMC sampling and I2C
read-back results leave the DUT with the run they belong to rather than
only existing in the Tera Term capture.

The wait moves ahead of the getdate/sprintf2 block; those build strings
without touching the terminal, so the prompt is still consumed exactly
once before the copy is sent.

Script A history updated under the existing V1.0.8 block -- that version
is not tagged yet, so this is the same batch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-25 15:55:05 +08:00
etwenandClaude Opus 5 166b53983f feat(bmc): Re-enable bmc_monitor with an I2C write/read-back test
bmc_monitor.sh comes back into the run with a second job beyond the
`free -m` sampling: an I2C pattern test against bus 1, address 0x41,
offset 0x00C0 -- write 55AA55AA and read it back, then write AA55AA55
and read that back. Two complementary patterns catch a stuck bit either
way round, and repeating it through the soak turns an intermittent I2C
fault into something the log shows rather than something the tester has
to catch live.

Script B starts it alongside the other stress; Script C stops it and
cats log/bmc_poll.log.

bmc_monitor_ddr.sh stays disabled -- BMC DDR stress runs through
bgctl + bmc-manager.

Docs: the "disabled" notes I added last time now applied to only half
the pair and read as wrong for bmc_monitor.sh. Split so the monitor is
documented as live with its I2C test, and only the _ddr variant is
marked disabled.

Script A -> V1.0.8.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-25 15:50:56 +08:00
etwenandClaude Opus 5 ef7cbdffd4 docs(release): Add v1.0.7 notes; bump Script A header to V1.0.7
Script A carried a V1.0.7 history block while its "; Version :" header
still read V1.0.6, so publish.sh built Script_ABC_Blanton_V1.0.6 and the
folder would have disagreed with the tag. Header and date corrected --
reading the version from the header rather than a flag is what surfaced
this.

utils/wait_init.ttl V3.1.1: poll interval 10 s -> 60 s, and the WT_MIN
comments now say "at least" to match the test, which has been
`< WT_MIN` since the threshold was fixed. The stale "more than" wording
described exactly the off-by-one that once hung Script A.

Script C: the BMC USB journalctl dump is commented out.

docs/release-notes/v1.0.7.md written in the house format.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-25 14:22:22 +08:00
etwenandClaude Opus 5 ae8941a844 feat(ttl): Set fan speed in Script A, archive job logs to USB in C
config.ttl gains FAN_SPEED (30). Script A rebinds the two max31790
controllers (18-0020, 25-0020) and applies the speed with
fan-speed-control.sh, answering its confirmation prompt, so a run starts
from a known thermal state rather than whatever the last test left.

Script A also clears /host/hw-eval/current/jobs before the run, and
Script C copies that directory to /mnt/usb/jobs-<date>_<time> so the
bgctl job logs leave the DUT with the run they belong to.

Fix: 25-0020 was bound twice in a row -- the second bind can only fail,
since the driver is already attached. Removed; the two controllers now
have a matching unbind/sleep/bind each.

Script A -> V1.0.7, history covering the config.ttl and Script C changes
as well.

Docs: fan control and the job-log archive added to Tech Stack, Key
Features and Data Flow; the config.ttl example refreshed with FAN_SPEED
and the SWB_UNIT flags. Three risks recorded in Future Extensions --
`rm` without -r cannot clear the job subdirectories that Script C copies
with -r, the fan confirmation wait has no timeout and would hang
silently if the prompt ever changes, and /mnt/usb is never checked for
being mounted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-25 14:12:24 +08:00
etwenandClaude Opus 5 a6a12cdf2c docs: Sync ARCHITECTURE and CLAUDE with the bgctl and mgmt-ping work
Script A: wait 45 s between starting and stopping mgmt_ping_monitor.
`start` returns after one second, but a round is roughly 30 s -- without
the pause the log held only the START banner and no RESULT line.

Docs brought up to date with the eight commits since 419b680:

- Tech Stack gains bgctl, systemd-run, iproute2 and hw-test-session;
  the stress row moves off the hammer scripts
- Structure and Key Features cover mgmt_ping_monitor.sh, and mark
  bmc_monitor{,_ddr}.sh as disabled -- their call sites in B and C are
  commented out, though the files remain
- Data Flow rewritten for all three scripts
- Two constraints added: TTL's `timeout` is global and must be restored,
  and two management NICs on one subnet cause ARP flux (hence
  ARP_STRICT, and nic_tx/nic_rx as the evidence of which NIC sent)
- CLAUDE gotchas add the `ip -s link` column order, that a monitor's
  `start` returns in one second rather than after a round, and that
  show_dmesg now ends with `dmesg -C`
- Phase 3 ticks the move to bgctl; Phase 5 gains the mgmt-ping, NVMe and
  EDAC parsing targets. Future Extensions notes that Script C never cats
  mgmt_ping.log, so the soak's ping results stay on the DUT

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-24 23:34:33 +08:00
etwenandClaude Opus 5 e38d24c775 feat: Add mgmt_ping_monitor.sh and move stress onto bgctl
New Blanton_Script/mgmt_ping_monitor.sh: brings both management NICs up
with iproute2 (`ip link set` / `address replace` / `route replace` with
per-NIC metric) and pings each one's own target -- eth0 -> .30, eth1 ->
.31 -- appending to a rotating log. Same sub-commands as the bmc
monitors, plus `summary` for just the per-leg RESULT lines.

Both NICs stay up, which on a shared subnet lets the target's ARP be
answered by either one, so a reply can land on the NIC that did not
send. ARP_STRICT applies arp_ignore/arp_announce to prevent that, and
each leg logs `ip -s link show` with the interface's own TX/RX packet
delta across the burst as direct evidence of which NIC carried the
traffic. Note `ip -s link` orders columns "bytes packets ...", so the
packet count is the second field.

Script A: hw-test-session start/log/status, a fuller DUT inventory
(version, fwutil, syseeprom, ssdhealth, TPM, nvme smart-log, smartctl),
bmc-first-enroll and bmc-manager version/status, ras-mc-ctl summary, and
both 100G uplinks now brought up rather than 513 being left down.

Script B: stress moves to bgctl (memtester, qfx5252-stress-ssd/-usb) and
the BMC DDR load runs through bmc-manager, replacing the hammer scripts
and the bmc_monitor pair. Adds a BMC USB net test that discovers the
cdc_ncm interface and runs a 4-hour ping under systemd-run.

Script C: stops the bgctl jobs, the BMC USB unit and the mgmt ping, then
collects journalctl, per-NIC counters, ras-mc-ctl and NVMe health, and
closes the session with hw-test-session finish.

utils/show_dmesg.ttl: one combined error regex with `dmesg -T`, an
i2c-filtered view, then `dmesg -C` so Script C's capture shows only what
the soak produced. utils/setup_pmon.ttl: add the TPM FRU/read checks.

Fix: Script B set `timeout = 15` for the cdc_ncm probe and never
restored it, leaving the cap in force for everything after -- including
the traffic init, which walks 108 VLANs per unit and takes far longer
than 15 s. A timed-out wait returns without the prompt, so the macro
would have run ahead of the DUT for the rest of the soak. Reset to 0 at
:skip_ping, where both branches meet.

Script A -> V1.0.6.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-08-24 23:27:37 +08:00
15 changed files with 1132 additions and 93 deletions
+5
View File
@@ -12,6 +12,11 @@ For_AI/
!**/logs/.gitkeep !**/logs/.gitkeep
!**/Logs/.gitkeep !**/Logs/.gitkeep
# monitor 腳本的執行期產物(原始擷取、pid、狀態)
*.raw
*.pid
*.state
# 打包輸出:產出不進 git,打包工具本身要進 # 打包輸出:產出不進 git,打包工具本身要進
publish/* publish/*
!publish/.gitkeep !publish/.gitkeep
+81 -34
View File
@@ -37,8 +37,12 @@ LTC2980 margin 控制器。TTL 只負責「按順序 `sendln` 並把畫面收進
| 電源 margin | LTC2980= 2 × LTC2977over PMBusLINEAR16 編碼 | | 電源 margin | LTC2980= 2 × LTC2977over PMBusLINEAR16 編碼 |
| 錯誤計數來源 | Linux PCIe AER sysfs`aer_dev_*`)、EDAC sysfs`dimm_ce/ue_count` | | 錯誤計數來源 | Linux PCIe AER sysfs`aer_dev_*`)、EDAC sysfs`dimm_ce/ue_count` |
| 平台監控 | SONiC `show platform *`pmon container,含 `leak status` / `leak channels` | | 平台監控 | SONiC `show platform *`pmon container,含 `leak status` / `leak channels` |
| BMC 存取 | host 端 `bmc-manager run "<cmd>"`(不需對 BMC 開 SSH,繞開其 busybox 工具鏈) | | BMC 存取 | host 端 `bmc-manager run "<cmd>"`(不需對 BMC 開 SSH,繞開其 busybox 工具鏈)`bmc-first-enroll` / `bmc-manager version\|status` |
| 壓力測試 | `mlucas-avx2`AMD CPU)、`~/hammer/tools/stress_{mem,ssd,usb}.py` | | 背景工作管理 | `bgctl run/list/stop --all/reset`(平台工具);長時間 ping 用 `systemd-run --unit=` 掛成 transient unit |
| 管理網路 | iproute2`ip link set` / `ip address replace` / `ip route replace ... metric`+ `ping -I` |
| 測試 session | `hw-test-session start\|log\|status\|finish`job log 收在 `/host/hw-eval/current/jobs/` |
| 風扇控制 | max31790 driver rebind`/sys/bus/i2c/drivers/max31790/{bind,unbind}`+ `fan-speed-control.sh <%>` |
| 壓力測試 | `mlucas-avx2`AMD CPU)、`bgctl run` 排程的 `memtester` / `qfx5252-stress-ssd` / `qfx5252-stress-usb` |
| 資料面流量 | `bcmcmd`Broadcom drivshellVLAN loopback + `tx` burstswitch unit 0 / 1 | | 資料面流量 | `bcmcmd`Broadcom drivshellVLAN loopback + `tx` burstswitch unit 0 / 1 |
| 流量報表 | DUT 上 awk`blanton_traffic_linespeed report`);離線 `python3 tools/bcm_mibpair_report_V1.1.0.py` | | 流量報表 | DUT 上 awk`blanton_traffic_linespeed report`);離線 `python3 tools/bcm_mibpair_report_V1.1.0.py` |
| 版本控制 | GitNAS + Gitea 私有 remote**不推 GitHub,含客戶 NDA 資料** | | 版本控制 | GitNAS + Gitea 私有 remote**不推 GitHub,含客戶 NDA 資料** |
@@ -148,8 +152,8 @@ Blanton_TTL_Script/
├── utils/ # 可 include 的取數片段(TTL,無狀態、只 wait+sendln ├── utils/ # 可 include 的取數片段(TTL,無狀態、只 wait+sendln
│ ├── pcie_bus.ttl # lspci -tvvv / -vv │ ├── pcie_bus.ttl # lspci -tvvv / -vv
│ ├── setup_pmon.ttl # show platform summary/fan/temperature/psustatus/voltage/ │ ├── setup_pmon.ttl # show platform summary/fan/temperature/psustatus/voltage/
│ │ # current/ssdhealth/leak status/leak channels9 項) │ │ # current/ssdhealth/leak status/leak channels + TPM11 項)
│ ├── show_dmesg.ttl # date + dmesg grep error/fail/warning │ ├── show_dmesg.ttl # dmesg -T 合併 error 正則 + i2c 過濾 + dmesg -C 清空
│ ├── show_margin_status.ttl # source margin_status_all.sh(掃 9 顆 LTC2980 │ ├── show_margin_status.ttl # source margin_status_all.sh(掃 9 顆 LTC2980
│ ├── kill_all_process.ttl # kill $(jobs -p) │ ├── kill_all_process.ttl # kill $(jobs -p)
│ ├── wait_init.ttl # 輪詢 show platform temperature 直到 pmon 起來 │ ├── wait_init.ttl # 輪詢 show platform temperature 直到 pmon 起來
@@ -172,9 +176,11 @@ Blanton_TTL_Script/
├── blanton_pwr_data.sh # SWB VRM + PDB brick 的 Vin/Vout/Iout/Temp 表 ├── blanton_pwr_data.sh # SWB VRM + PDB brick 的 Vin/Vout/Iout/Temp 表
├── blanton_traffic_linespeed.sh # SWB loopback 線速流量(bcmcmd unit 0/1 ├── blanton_traffic_linespeed.sh # SWB loopback 線速流量(bcmcmd unit 0/1
│ # init/clear/show/start/stop/ps/report/run │ # init/clear/show/start/stop/ps/report/run
├── bmc_monitor.sh # BMC 監控:週期跑 free -m,寫輪替 log ├── mgmt_ping_monitor.sh # 10G/1G 管理網路 ping 監控(iproute2
├── bmc_monitor_ddr.sh # BMC DDR 壓力:memtester,逾時 3600s │ # eth0→TARGET_10G、eth1→TARGET_1G,兩張卡都保持 up
│ # ↑ 兩支都要 chmod +x 才能跑(見 Key Constraints ├── bmc_monitor.sh # BMC 監控:free -m + I2C 寫入/讀回圖樣測試
├── bmc_monitor_ddr.sh # ⚠️ 已停用:B/C 內呼叫處已註解,DDR 壓力改走 bgctl + bmc-manager
│ # ↑ 三支都要 chmod +x 才能跑(見 Key Constraints
└── LTC2980_Margin_Script/ # 電壓 margin 子專案(另有獨立 repo └── LTC2980_Margin_Script/ # 電壓 margin 子專案(另有獨立 repo
├── margin.sh # margin_init/status/set/apply_profile/save ├── margin.sh # margin_init/status/set/apply_profile/save
├── settings/*.conf # 一個 .conf = 一顆 LTC2980CB + SWB0/1 × CONN13~16 ├── settings/*.conf # 一個 .conf = 一顆 LTC2980CB + SWB0/1 × CONN13~16
@@ -195,15 +201,20 @@ Blanton_TTL_Script/
### 1. `config.ttl` — 全域測試參數(TTL 變數) ### 1. `config.ttl` — 全域測試參數(TTL 變數)
```ini ```ini
strTestcase = "Margin" ; 測項名,進 log 檔名
project_name = "Blanton" ; 專案名,進 log 檔名 project_name = "Blanton" ; 專案名,進 log 檔名
EN_Margin = 1 ; 1=跑 margin 掃描, 0=跳過 strTestcase = "ENV" ; 測項名(ENV/EMC/Margin...),進 log 檔名
EN_Margin = 0 ; 1=跑 margin 掃描, 0=跳過
EN_log = 1 ; 1=logopen 存檔, 0=不存 EN_log = 1 ; 1=logopen 存檔, 0=不存
FAN_SPEED = 30 ; 傳給 fan-speed-control.sh 的風扇轉速 (%)
SWB_UNIT0 = 1 ; 這台有 switch unit 0(流量與 wait_init 都依此)
SWB_UNIT1 = 1 ; 這台有 switch unit 1
prompt_login = "sonic login:" prompt_login = "sonic login:"
prompt_sonic = "admin@sonic:~$" prompt_sonic = "admin@sonic:~$"
prompt_sonic_root = "root@sonic:~#" prompt_sonic_root = "root@sonic:~#"
``` ```
> `SWB_UNIT0` / `SWB_UNIT1` 兩個都是 0 時,`wait_init.ttl` 直接 bypass、A/B/C 的流量段整段跳過。
Log 檔名規則:`<mdir>\Logs\<project_name>_<strTestcase>_<YYYYmmdd-HHMMSS>.log` Log 檔名規則:`<mdir>\Logs\<project_name>_<strTestcase>_<YYYYmmdd-HHMMSS>.log`
### 2. `settings/<board>.conf` — 一顆 LTC298016 channel ### 2. `settings/<board>.conf` — 一顆 LTC298016 channel
@@ -290,9 +301,9 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc
| 模組 | 進入點 | 功能 | | 模組 | 進入點 | 功能 |
|------|--------|------| |------|--------|------|
| **Script APre-test** | `1_Blanton_Script_A.ttl` V1.0.3 | root 登入 → `date -s` 對時 → `show boot`**等 pmon 起來**`wait_init.ttl`)→ BMC 版本 → source 六個 bash 工具 → 清/讀 7 組 PCIe AER → `lspci` → margin 全掃 → PMON 九項 → dmesg → 100G port 狀態 → **一輪流量基線**`show uptime` | | **Script APre-test** | `1_Blanton_Script_A.ttl` V1.0.7 | root 登入 → `date -s` 對時 → 清 job log → `hw-test-session start`**設定風扇轉速****等資料面就緒**`wait_init.ttl`)→ DUT 清單(boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe)→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary``lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → 100G uplink 設定與狀態 → **一輪流量基線**`show uptime` |
| **Script BStress** | `2_Blanton_Script_B.ttl` | `mlucas-avx2 -cpu 0:15` + 5 份 `stress_mem.py` + `stress_ssd.py` + `stress_usb.py`(**共 8 個 job**)→ 啟動兩支 BMC monitor → `jobs` → 100G port 狀態 → traffic `ps/init/clear/show/start`**兩個 unit**)→ `while 1` 每輪完整 PMON + margin | | **Script BStress** | `2_Blanton_Script_B.ttl` | `mlucas-avx2 -cpu 0:15` + `bgctl run``memtester` / SSD / USB 壓力 + BMC DDR`bmc-manager run memtester`)→ BMC USB net test(探測 cdc_ncm 介面,`systemd-run` 掛 4 小時 ping)→ 啟動 mgmt ping 監控 → `jobs` / `bgctl list` → traffic(依 `SWB_UNIT`)→ `while 1` 每輪完整 PMON + margin |
| **Script CPost-test** | `3_Blanton_Script_C.ttl` | `kill $(jobs -p)` → PMON + dmesg → 7 組 AER → 停兩支 BMC monitor 並 `cat` 其 log`cat` mlucas / stress_ssd log → traffic `stop` + `report`(兩個 unit`show reboot-cause` / `show uptime`完成 messagebox | | **Script CPost-test** | `3_Blanton_Script_C.ttl` | `kill $(jobs -p)` + `bgctl stop --all` / `reset` + 停 BMC USB unit 與 mgmt ping → PMON + dmesg → 7 組 AER + `ras-mc-ctl` → BMC USB `journalctl``ip -s link show eth0/eth1``cat` mlucas log → traffic `stop` + `report` → NVMe 健康 `show reboot-cause` / `uptime`**job log 複製到 USB(帶時戳)**`hw-test-session finish` |
| **SWB loopback 流量** | `blanton_traffic_linespeed <sub> [-u 0\|1\|all]` | `init`VLAN 30..137,成對 `cdN`/`cdN+32`)→ `clear``start``tx 100 length=512`)→ `stop``report`per-pair TX/RX 交叉比對 PASS/FAIL);另有 `ps`link/speed)與 `run`(一條龍) | | **SWB loopback 流量** | `blanton_traffic_linespeed <sub> [-u 0\|1\|all]` | `init`VLAN 30..137,成對 `cdN`/`cdN+32`)→ `clear``start``tx 100 length=512`)→ `stop``report`per-pair TX/RX 交叉比對 PASS/FAIL);另有 `ps`link/speed)與 `run`(一條龍) |
| **FPGA 暫存器存取** | `cb_fpga <fn> <addr> [data]``fpga_rescan` | pcimem 打 `resource0` + BAR 相對 offset32-bit wordBDF 於 source 時自動偵測;`pmc/icb/swb0/swb1_fpga` 走 VSPI 窗口 | | **FPGA 暫存器存取** | `cb_fpga <fn> <addr> [data]``fpga_rescan` | pcimem 打 `resource0` + BAR 相對 offset32-bit wordBDF 於 source 時自動偵測;`pmc/icb/swb0/swb1_fpga` 走 VSPI 窗口 |
| **CB I2C / PMBus** | `cb_i2c_init/scan/read/write``cb_pmbus_read/write/linear16` | FPGA F3 OpenCores master**讀用 Repeated START**SMBus/PMBus 裝置必需) | | **CB I2C / PMBus** | `cb_i2c_init/scan/read/write``cb_pmbus_read/write/linear16` | FPGA F3 OpenCores master**讀用 Repeated START**SMBus/PMBus 裝置必需) |
@@ -300,7 +311,9 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc
| **溫度** | `temp_cb` / `temp_icb` / `temp_all` | TMP7512-bit 左靠齊 ×0.0625)、TMP432(含 RANGE bit 的 64 偏移、remote1/2 | | **溫度** | `temp_cb` / `temp_icb` / `temp_all` | TMP7512-bit 左靠齊 ×0.0625)、TMP432(含 RANGE bit 的 64 偏移、remote1/2 |
| **電源資料** | `pwr_data` / `pwr_data_pdb` / `pwr_pdb_id` | SWB VRM 全軌 Vin/Vout/Iout/Temp 對齊表;MP29816 解析度**執行期從 `MFR_VOUT_SCALE_LOOP` 讀**,不寫死 | | **電源資料** | `pwr_data` / `pwr_data_pdb` / `pwr_pdb_id` | SWB VRM 全軌 Vin/Vout/Iout/Temp 對齊表;MP29816 解析度**執行期從 `MFR_VOUT_SCALE_LOOP` 讀**,不寫死 |
| **電壓 Margin** | `margin_init/status/set/apply_profile/save` + `*_all.sh` | 9 顆 LTC2980CB CONN13、SWB0/1 CONN13~16)批次 high/low margin 與狀態掃描。v2.6.0 起 SWB 路徑真正接上共用 cb_i2c 後端,並有 `margin_swb_info/probe/scan/reset/sem` 除錯包裝 | | **電壓 Margin** | `margin_init/status/set/apply_profile/save` + `*_all.sh` | 9 顆 LTC2980CB CONN13、SWB0/1 CONN13~16)批次 high/low margin 與狀態掃描。v2.6.0 起 SWB 路徑真正接上共用 cb_i2c 後端,並有 `margin_swb_info/probe/scan/reset/sem` 除錯包裝 |
| **BMC 監控 / 壓力** | `bmc_monitor.sh` / `bmc_monitor_ddr.sh` `{start\|stop\|status\|fg\|tail\|clear}` | 由 host 端 `bmc-manager run` 週期性打 BMC,背景 detached、log 輪替。前者跑 `free -m`(逾時 30s),後者跑 `memtester`(逾時 3600s),各自獨立 log 與 pid | | **管理網路 ping** | `mgmt_ping_monitor.sh {start\|stop\|status\|fg\|tail\|clear\|summary}` | 一輪 = 10G leg + 1G leg,各自 `ip address/route replace``ping -I <if>` 打自己的對端;兩張卡都保持 up。每 leg 記 `ip -s link show` 與該卡的 TX/RX packet delta,另出一行可 grep 的 `RESULT` |
| **BMC 監控** | `bmc_monitor.sh {start\|stop\|status\|fg\|tail\|clear}` | 由 host 端 `bmc-manager run` 週期取樣 `free -m`,另加一組 **I2C 寫入/讀回圖樣測試**bus 1、addr 0x41、offset 0x00C0,寫 `55AA55AA` 讀回、再寫 `AA55AA55` 讀回)。B 啟動、C 停止並 `cat log/bmc_poll.log` |
| **BMC DDR 壓力** | `bmc_monitor_ddr.sh` | ⚠️ **已停用** —— B/C 內呼叫處已註解,改由 `bgctl run bmc-manager run memtester` 取代。檔案保留未刪 |
| **打包交付** | `publish/publish.sh [-v Ver] [-f] [-z] [-n]` | 版號讀 Script A 檔頭,複製成 `publish/Script_ABC_Blanton_<Ver>/`,排除 `*.log`,可另出 zip | | **打包交付** | `publish/publish.sh [-v Ver] [-f] [-z] [-n]` | 版號讀 Script A 檔頭,複製成 `publish/Script_ABC_Blanton_<Ver>/`,排除 `*.log`,可另出 zip |
--- ---
@@ -316,15 +329,20 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc
Script A ── logopen Logs/Blanton_Margin_<ts>.log ──┐ (之後所有畫面都進這個檔) Script A ── logopen Logs/Blanton_Margin_<ts>.log ──┐ (之後所有畫面都進這個檔)
│ │ │ │
├─ 登入 admin → sudo -i → root@sonic:~# │ ├─ 登入 admin → sudo -i → root@sonic:~# │
├─ date -s "<host 時間>" ← 讓 DUT 時戳可對照 ├─ date -s "<host 時間>" ← 讓 DUT 時戳可對照
├─ rm /host/hw-eval/current/jobs/* ← 清掉上一輪的 job log
├─ hw-test-session start / log / status
├─ 風扇:max31790 unbind→sleep→bind18-0020 與 25-0020 各一次)
│ → fan-speed-control.sh <FAN_SPEED> → 回答 y 確認 │
├─ source 6 個 bash 工具(函數進 shell ├─ source 6 個 bash 工具(函數進 shell
│ └ 此時 blanton_fpga_pcimem.sh 印出 [INFO] CB FPGA BDF = ...(自動偵測結果) │ └ 此時 blanton_fpga_pcimem.sh 印出 [INFO] CB FPGA BDF = ...(自動偵測結果)
├─ AER 基線:7 組 EP+RP 的 correctable/nonfatal/fatal ├─ AER 基線:7 組 EP+RP 的 correctable/nonfatal/fatal
├─ lspci -tvvv / -vv ← 拓樸 + link speed/width ├─ lspci -tvvv / -vv ← 拓樸 + link speed/width
├─ margin_status_all ← 9 顆 LTC2980 × 16ch 的 Vout 與偏差% ├─ margin_status_all ← 9 顆 LTC2980 × 16ch 的 Vout 與偏差%
├─ show platform × 9 ← fan/temp/psu/voltage/current/ssdhealth/leak status/leak channels ├─ show platform × 9 ← fan/temp/psu/voltage/current/ssdhealth/leak status/leak channels
├─ dmesg | grep error/fail/warning ├─ dmesg -T(合併 error 正則 + i2c 過濾)→ dmesg -C 清空 ← C 只會看到 soak 期間新產生的
├─ show interfaces status Ethernet513,Ethernet514 ← 100G uplink 狀態 ├─ mgmt pingstart → pause 45 → stop → cat log ← 10G/1G 各一輪
├─ 100G uplinklpmode off → 兩埠 startup → config save → show interfaces status
├─ 一輪流量基線:ps → init → clear → show → start → stop → report ├─ 一輪流量基線:ps → init → clear → show → start → stop → report
└─ show uptime └─ show uptime
(最前面還有 wait_init.ttl:輪詢 show platform temperature 直到不再回報 (最前面還有 wait_init.ttl:輪詢 show platform temperature 直到不再回報
@@ -336,8 +354,10 @@ Script A ── logopen Logs/Blanton_Margin_<ts>.log ──┐ (之後所有畫
Script B ── 同一個 log 檔續寫(logwrite 分隔線) Script B ── 同一個 log 檔續寫(logwrite 分隔線)
├─ 背景程序:mlucas-avx2(CPU) ×1, stress_mem ×5, stress_ssd, stress_usb → 共 8 個 job ├─ 背景程序:mlucas-avx2(CPU) ×1, stress_mem ×5, stress_ssd, stress_usb → 共 8 個 job
stress_hhmd / stress_pcie 已於 2026-08-17 移除,見 Status 表 ScriptB #5/#7 "KC Remove" stress_hhmd / stress_pcie 已於 2026-08-17 移除,見 Status 表 ScriptB #5/#7 "KC Remove"
├─ BMC: bmc_monitor.sh start + bmc_monitor_ddr.sh start ├─ bgctl run: memtester 1G / qfx5252-stress-ssd / qfx5252-stress-usb
├─ jobs ← 確認全部起來了 ├─ BMC: bmc-manager run memtesterDDR)、cdc_ncm 介面探測 + systemd-run 4hr pingUSB
├─ mgmt_ping_monitor.sh start
├─ jobs / bgctl list ← 確認全部起來了
├─ traffic: ps → init → clear → (pause 15) → show → start unit 0 與 1VLAN 30..137 loopback ├─ traffic: ps → init → clear → (pause 15) → show → start unit 0 與 1VLAN 30..137 loopback
└─ while 1: setup_pmon9 項)→ margin_status_all soak 期間持續取樣) └─ while 1: setup_pmon9 項)→ margin_status_all soak 期間持續取樣)
@@ -348,10 +368,14 @@ Script C
├─ kill $(jobs -p) ← 收掉所有背景壓力 ├─ kill $(jobs -p) ← 收掉所有背景壓力
├─ show platform × 9 + dmesg ← 收工快照 ├─ show platform × 9 + dmesg ← 收工快照
├─ AER 再抓一次 ← 與 Script A 基線相減 = 本輪新增錯誤 ├─ AER 再抓一次 ← 與 Script A 基線相減 = 本輪新增錯誤
├─ BMC: 兩支 monitor stop → cat log/bmc_poll.log + log/bmc_ddr.log ├─ bgctl stop --all / reset、systemctl stop BMC USB unit、mgmt ping stop
├─ cat mlucas_amm_log / log.stress_ssd ← 壓力程式自身的 pass/fail ├─ journalctl BMC USB unit、ip -s link show eth0/eth1、ras-mc-ctl --summary
├─ cat mlucas_amm_log ← CPU 壓力程式自身的 pass/fail
├─ nvme smart-log / smartctl -x /dev/nvme0
├─ traffic: stop → report ← per-pair TX/RX 交叉比對,PASS/FAIL/NAexit 2 = 有 FAIL/NA ├─ traffic: stop → report ← per-pair TX/RX 交叉比對,PASS/FAIL/NAexit 2 = 有 FAIL/NA
─ show reboot-cause / show uptime ← soak 期間有沒有意外重開,一眼看得到 ─ show reboot-cause / show uptime ← soak 期間有沒有意外重開,一眼看得到
├─ cp -r /host/hw-eval/current/jobs/ /mnt/usb/jobs-<YYYYmmdd>_<HHMM>
└─ hw-test-session finish
[messagebox "GOOD JOB! Test Case DONE"] → 人工檢查 log [messagebox "GOOD JOB! Test Case DONE"] → 人工檢查 log
``` ```
@@ -409,12 +433,21 @@ pwr_data
⚠️ 但 `.gitattributes` **不會回頭重寫既有工作區檔案** —— 早於它 clone 出來的目錄, ⚠️ 但 `.gitattributes` **不會回頭重寫既有工作區檔案** —— 早於它 clone 出來的目錄,
`git rm --cached -r . && git reset --hard` 或刪掉目錄重新 checkout 才會生效, `git rm --cached -r . && git reset --hard` 或刪掉目錄重新 checkout 才會生效,
而且 `git status` 乾淨**不代表**磁碟上是 LF`--renormalize` 只在進 index 的路上轉換)。 而且 `git status` 乾淨**不代表**磁碟上是 LF`--renormalize` 只在進 index 的路上轉換)。
13. **`bmc_monitor*.sh` 需要執行權**:它們是本專案第一組「被執行」而非被 `source` 的腳本, 13. **`bmc_monitor*.sh` / `mgmt_ping_monitor.sh` 需要執行權**:它們是本專案「被執行」而非被 `source` 的腳本,
`do_start` 內部是 `setsid "$SCRIPT_PATH" __daemon`**即使用 `bash x.sh start` 呼叫 `do_start` 內部是 `setsid "$SCRIPT_PATH" __daemon`**即使用 `bash x.sh start` 呼叫
也仍需 exec bit**。git 記錄為 `100644`,且 Windows → 隨身碟 → DUT 的路徑無法攜帶權限位元, 也仍需 exec bit**。git 記錄為 `100644`,且 Windows → 隨身碟 → DUT 的路徑無法攜帶權限位元,
所以佈署後要先 `chmod +x ~/Blanton_Script/bmc_monitor*.sh`,否則 Script B 的 start 會 所以佈署後要先 `chmod +x ~/Blanton_Script/bmc_monitor*.sh`,否則 Script B 的 start 會
`Permission denied`、Script C 的 `cat` 撲空(BMC 段全空但不會報錯)。 `Permission denied`、Script C 的 `cat` 撲空(BMC 段全空但不會報錯)。
14. **客戶 NDA**`docs/` 內含客戶 FPGA 規格與 schematic 衍生資訊,**本 repo 只推 NAS + Gitea 14. **TTL 的 `timeout` 是全域的,設了就要還原**`timeout = N` 之後**每一個** `wait` 都被限制在 N 秒
逾時的 `wait` 會直接返回而沒吃到 prompt,於是下一個指令在前一個還沒跑完就送出,之後整輪錯開一拍。
`wait_init.ttl` 設 60 後還原 0Script B 的 cdc_ncm 探測設 15,在 `:skip_ping`(兩條路徑的匯流點)還原。
加新的 `timeout` 時務必配一個 `timeout = 0`
15. **兩張管理網卡同網段時會 ARP flux**`mgmt_ping_monitor.sh` 讓 eth0/eth1 同時 up,若兩者在同一個
/24(本專案預設 `192.168.1.99` / `.101`),對端 ARP 可能由任一張卡回應,回包落在沒送封包的那張,
量到的數字不能歸屬。`ARP_STRICT=1` 會設 `arp_ignore=1` / `arp_announce=2` 擋掉;
另外每個 leg 的 `RESULT` 行帶 `nic_tx` / `nic_rx`(該卡自己的 packet delta),
**ping 成功但 `nic_tx` 接近 0 就代表封包從另一張卡出去了**。兩張卡在不同網段時可設 0。
16. **客戶 NDA**`docs/` 內含客戶 FPGA 規格與 schematic 衍生資訊,**本 repo 只推 NAS + Gitea
絕不推 GitHub**。 絕不推 GitHub**。
--- ---
@@ -554,13 +587,12 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值,
**目標:** 壓力程序起得來、起不來看得出來,soak 迴圈有時戳、可控週期。 **目標:** 壓力程序起得來、起不來看得出來,soak 迴圈有時戳、可控週期。
**包含:** **包含:**
- [ ] 部署 `~/hammer/tools/` 前置檢查:新增 `utils/check_hammer_tools.ttl` - [ ] 前置檢查:`~/hammer/tools/amd/mlucas-avx2` 仍是唯一還依賴 hammer 的項目,缺檔就別往下跑
`ls -l ~/hammer/tools/` + `ls ~/hammer/tools/amd/mlucas-avx2`),缺檔就別往下跑
- [ ] 壓力程序 log 加時戳,避免多輪覆蓋: - [ ] 壓力程序 log 加時戳,避免多輪覆蓋:
`mlucas_amm_log``~/logs/mlucas_amm_log_$(date +%Y%m%d-%H%M%S).log`Status 表 ScriptB #2 的既定寫法) `mlucas_amm_log``~/logs/mlucas_amm_log_$(date +%Y%m%d-%H%M%S).log`Status 表 ScriptB #2 的既定寫法)
- [ ] `jobs` 之後加驗證:期望 **8** 個背景 job1 mlucas + 5 mem + ssd + usb), - [x] 壓力層改用平台的 `bgctl``memtester` / `qfx5252-stress-ssd` / `qfx5252-stress-usb`),
數量不符時 `messagebox` 提示 取代 `~/hammer/tools/stress_*.py`Script B 以 `jobs` + `bgctl list` 兩者並列確認
`stress_hhmd` / `stress_pcie` 已於 2026-08-17 由 KC 移除,原本的 10 改為 8 - [ ] `bgctl list` 之後加數量驗證,不符時 `messagebox` 提示(目前只印出來給人看
- [ ] soak 迴圈改為「每輪先 `date`、跑完 `pause <間隔>`」, - [ ] soak 迴圈改為「每輪先 `date`、跑完 `pause <間隔>`」,
`EN_LoopInterval`(預設 600 秒 = 10 min)放進 `config.ttl`,兌現 Status 表的「Get data every 10mins」 `EN_LoopInterval`(預設 600 秒 = 10 min)放進 `config.ttl`,兌現 Status 表的「Get data every 10mins」
**← 目前 `while 1` 仍無 `pause`,實際間隔取決於指令執行時間,不是 10 分鐘** **← 目前 `while 1` 仍無 `pause`,實際間隔取決於指令執行時間,不是 10 分鐘**
@@ -621,9 +653,13 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值,
UE > 0 或 fatal/nonfatal > 0 直接標 FAIL UE > 0 或 fatal/nonfatal > 0 直接標 FAIL
- [ ] Margin 表解析:把 9 顆 × 16ch 的 Vout / 偏差% 收成 CSV,標出超出 ±(profile 設定值 + 容差) 的軌 - [ ] Margin 表解析:把 9 顆 × 16ch 的 Vout / 偏差% 收成 CSV,標出超出 ±(profile 設定值 + 容差) 的軌
- [ ] 溫度 / 風扇 / 電源趨勢:soak 期間逐輪取樣 → CSV(可直接畫圖) - [ ] 溫度 / 風扇 / 電源趨勢:soak 期間逐輪取樣 → CSV(可直接畫圖)
- [ ] Stress 判定:`mlucas_amm_log``log.stress_ssd` 的錯誤關鍵字掃描 - [ ] Stress 判定:`mlucas_amm_log``bgctl` 各工作的 log
`/host/hw-eval/.../qfx5252-stress-{ssd,usb}.log`)的錯誤關鍵字掃描
- [ ] Traffic 判定:**直接複用 `tools/bcm_mibpair_report_V1.1.0.py`**(它已能吃 console log 的 - [ ] Traffic 判定:**直接複用 `tools/bcm_mibpair_report_V1.1.0.py`**(它已能吃 console log 的
`show c` 段落並輸出 PASS/FAIL),不要重寫一套解析 `show c` 段落並輸出 PASS/FAIL),不要重寫一套解析
- [ ] 管理網路判定:抓 `mgmt_ping_monitor.sh``RESULT` 行(已是單行可 grep 格式),
並檢查 `nic_tx`/`nic_rx` 不為 0 —— 否則 PASS 也不能歸屬到那張卡
- [ ] NVMe / EDAC`nvme smart-log``smartctl -x``ras-mc-ctl --summary` 的前後差值
- [ ] 產出 `Logs/<run>_summary.md`:一頁 PASS/FAIL + 每個子系統一行結論 - [ ] 產出 `Logs/<run>_summary.md`:一頁 PASS/FAIL + 每個子系統一行結論
**驗收條件:** 拿一份既有 log(如 `Blanton_Margin_20260814-164122.log`)跑解析器, **驗收條件:** 拿一份既有 log(如 `Blanton_Margin_20260814-164122.log`)跑解析器,
@@ -676,8 +712,19 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值,
文件已加過期警告但內文未動。第 1~4 章不受影響。 文件已加過期警告但內文未動。第 1~4 章不受影響。
- **`wait_init.ttl` 沒有逾時出口**:若 pmon 一直起不來,`result` 既非 1 也非 2,迴圈會無限重試, - **`wait_init.ttl` 沒有逾時出口**:若 pmon 一直起不來,`result` 既非 1 也非 2,迴圈會無限重試,
Script A 既不往下走也不報錯。可加最大重試次數 + `messagebox` 提示。 Script A 既不往下走也不報錯。可加最大重試次數 + `messagebox` 提示。
- **`bmc_monitor*.sh` 的執行權**:目前需人工 `chmod +x`。可在 Script B 起它們之前補一行 - **腳本的執行權**`mgmt_ping_monitor.sh``bmc_monitor.sh`(以及已停用的 `bmc_monitor_ddr.sh`)需人工 `chmod +x`
`chmod +x`(比照既有的 `chmod +x ~/hammer/tools/amd/mlucas-avx2`),或在 publish 時處理。 可在 Script B 起它之前補一行(比照既有的 `chmod +x ~/hammer/tools/amd/mlucas-avx2`),或在 publish 時處理。
- **`bmc_monitor_ddr.sh` 的壓力強度**`memtester 500M 1``INTERVAL_SEC=5` 等於整個 soak - **`bmc_monitor_ddr.sh` 已停用**:B/C 內的呼叫已註解,DDR 壓力改由 `bgctl run bmc-manager run memtester`
近乎不間斷壓 BMC 記憶體,而 BMC 同時還要服務 host 的 `show platform` 查詢 取代。檔案還留在 repo,待確認不再需要後刪除,或在檔頭標註停用
若 PMON 取數變慢,先調大 `INTERVAL_SEC` 或縮小 memtester 的量。 `bmc_monitor.sh` 已於 2026-08-25 重新啟用,並加上 I2C 寫入/讀回圖樣測試。)
- **`rm /host/hw-eval/current/jobs/*` 清不掉子目錄**:Script A 用的是不帶 `-r``rm`,但 Script C
`cp -r` 那個目錄,代表裡面可能有子目錄。清不乾淨的話,上一輪的 job log 會被當成這輪的一起
複製到 USB。要確認 `bgctl` 的產出結構,必要時改成 `rm -rf .../jobs/*`
- **風扇確認提示是無限等待**`wait "Set all configured fan channels to"` 之後才 `sendln "y"`
若該版 `fan-speed-control.sh` 不再詢問(或字串改了),這個 `wait` 會永遠等下去,Script A 停在那裡
且不報錯。加個 `timeout` + 分支會比較保險(記得還原 `timeout = 0`)。
- **USB 掛載點未檢查**Script C 直接 `cp -r ... /mnt/usb/``/mnt/usb` 沒掛載時只會失敗一行,
整輪測試的 job log 就沒被帶出來。可在複製前先 `mountpoint -q /mnt/usb` 判斷。
- **Script C 沒有倒出 `mgmt_ping.log`**C 會 `stop` 管理網路監控並印 `ip -s link show`
但沒有 `cat ~/Blanton_Script/log/mgmt_ping.log`,所以整個 soak 期間累積的 per-leg `RESULT`
留在 DUT 上、沒進 master log。Script A 那輪基線有 `cat`
+41 -14
View File
@@ -9,9 +9,13 @@ DUT 端的取數能力由一組 **bash 工具庫** 提供(CB FPGA `pcimem` →
完整架構見 [ARCHITECTURE.md](ARCHITECTURE.md)。 完整架構見 [ARCHITECTURE.md](ARCHITECTURE.md)。
**目前狀態(2026-08-21**Script A/B/C**V1.0.3**,涵蓋 PCIe AER / DDR EDAC / PMON / **目前狀態(2026-08-25**Script A 為 **V1.0.7**(A 集中記錄 A/B/C 三支的變更)。涵蓋
電壓 margin9 顆 LTC2980/ SWB loopback 線速流量(兩個 switch unit/ BMC 監控與 DDR 壓力。 PCIe AER / DDR EDAC / PMON+TPM / 電壓 margin9 顆 LTC2980/ SWB loopback 線速流量 /
SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。交付用 `./publish/publish.sh` 打包 10G+1G 管理網路 ping / BMC DDR 與 USB / NVMe 健康,壓力層走平台的 `bgctl`
switch unit 數量由 `config.ttl``SWB_UNIT0` / `SWB_UNIT1` 宣告,風扇轉速由 `FAN_SPEED` 指定。
測試 job log 收在 DUT 的 `/host/hw-eval/current/jobs/`,Script C 結束時帶時戳複製到 `/mnt/usb/`
最後一次發布是 **V1.0.5**;SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。
交付用 `./publish/publish.sh` 打包。
## 技術棧 ## 技術棧
@@ -24,6 +28,9 @@ SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。交付用 `./publish
- **資料面流量**`bcmcmd`Broadcom drivshellSWB loopbackVLAN 30..137 成對 `cdN`/`cdN+32` - **資料面流量**`bcmcmd`Broadcom drivshellSWB loopbackVLAN 30..137 成對 `cdN`/`cdN+32`
`tx 100 length=512`;判定讀 ASIC `MIB_TPOK`/`MIB_RPOK` 做 per-pair 雙向交叉比對 `tx 100 length=512`;判定讀 ASIC `MIB_TPOK`/`MIB_RPOK` 做 per-pair 雙向交叉比對
- **BMC**host 端 `bmc-manager run "<cmd>"`,不需對 BMC 開 SSH(繞開其 busybox 工具鏈) - **BMC**host 端 `bmc-manager run "<cmd>"`,不需對 BMC 開 SSH(繞開其 busybox 工具鏈)
- **背景工作**:平台的 `bgctl run/list/stop --all/reset`;長時間 ping 用 `systemd-run --unit=` 掛 transient unit
- **管理網路**iproute2`ip link set` / `address replace` / `route replace ... metric`+ `ping -I`
- **風扇**max31790 driver rebind + `fan-speed-control.sh <%>`(轉速讀 `config.ttl``FAN_SPEED`
- **判定資料源**PCIe AER sysfs`aer_dev_*`)、EDAC sysfs`dimm_{ce,ue}_count`)、SONiC `show platform *` - **判定資料源**PCIe AER sysfs`aer_dev_*`)、EDAC sysfs`dimm_{ce,ue}_count`)、SONiC `show platform *`
- **無編譯步驟**:TTL 與 bash 都直譯執行 - **無編譯步驟**:TTL 與 bash 都直譯執行
@@ -70,13 +77,21 @@ blanton_traffic_linespeed report -f <log> # 離線重解一份存下來的 cons
# 離線版報表(開發機上跑,選項更多) # 離線版報表(開發機上跑,選項更多)
python3 tools/bcm_mibpair_report_V1.1.0.py <drivshell.log> -a python3 tools/bcm_mibpair_report_V1.1.0.py <drivshell.log> -a
# ── BMC 監控 / 壓力(需先 chmod +x!)──────────────── # ── 管理網路 ping 監控(需先 chmod +x!)──────────────
chmod +x ~/Blanton_Script/bmc_monitor*.sh # 佈署後必做一次 chmod +x ~/Blanton_Script/*.sh # 佈署後必做一次
~/Blanton_Script/bmc_monitor.sh start # free -m 取樣,log/bmc_poll.log ~/Blanton_Script/mgmt_ping_monitor.sh fg # 前景試一輪(約 30 秒),確認 IP/對端/網卡
~/Blanton_Script/bmc_monitor_ddr.sh start # memtesterlog/bmc_ddr.log ~/Blanton_Script/mgmt_ping_monitor.sh start # 背景,log/mgmt_ping.logstart 會先清空)
~/Blanton_Script/bmc_monitor.sh status # pid / log 路徑 / log 大小 ~/Blanton_Script/mgmt_ping_monitor.sh status
~/Blanton_Script/bmc_monitor.sh stop ~/Blanton_Script/mgmt_ping_monitor.sh stop
# ⚠️ tail 子命令是 tail -f,會卡住不返回;要倒 log 請直接 cat log/*.log ~/Blanton_Script/mgmt_ping_monitor.sh summary # 只看每段的 RESULT 行
# ⚠️ tail 子命令是 tail -f,會卡住不返回;TTL 裡要倒 log 請用 cat
# bmc_monitor.sh start/stop # free -m + BMC I2C 寫入/讀回圖樣測試,log/bmc_poll.log
# ⚠️ bmc_monitor_ddr.sh 已停用(DDR 壓力改走 bgctl + bmc-manager),檔案仍在
# ── 背景壓力(平台工具)──────────────────────────────
bgctl run /usr/sbin/memtester 1G 100
bgctl list
bgctl stop --all ; bgctl reset --yes
# ── 各工具的 help ─────────────────────────────────── # ── 各工具的 help ───────────────────────────────────
blanton_fpga_help ; blanton_cb_i2c_help ; vi2c_help blanton_fpga_help ; blanton_cb_i2c_help ; vi2c_help
@@ -166,13 +181,25 @@ Tera Term 連 COM port (115200-8-N-1)
`.gitattributes` 已擋住 git 這條路徑,但**不會回頭重寫既有工作區檔案**:早於它 clone 的目錄要 `.gitattributes` 已擋住 git 這條路徑,但**不會回頭重寫既有工作區檔案**:早於它 clone 的目錄要
`git rm --cached -r . && git reset --hard` 才會生效,而且 **`git status` 乾淨不代表磁碟上是 LF** `git rm --cached -r . && git reset --hard` 才會生效,而且 **`git status` 乾淨不代表磁碟上是 LF**
`--renormalize` 只在進 index 的路上轉換)。驗證要看 byte:`grep -rlU $'\r' <dir>` `--renormalize` 只在進 index 的路上轉換)。驗證要看 byte:`grep -rlU $'\r' <dir>`
- ⚠️ **`bmc_monitor*.sh` 需要 exec bit**,且**不是改用 `bash x.sh start` 就能繞過** —— - ⚠️ **TTL 的 `timeout` 是全域的**`timeout = N` 之後**每一個** `wait` 都被限制在 N 秒。逾時的 `wait`
直接返回而沒吃到 prompt,下一個指令就在前一個還沒跑完時送出,之後整輪錯開一拍。**設了一定要
配一個 `timeout = 0` 還原**`wait_init.ttl` 60→0Script B 的 cdc_ncm 探測 15→0 在 `:skip_ping`
- ⚠️ **兩張管理網卡同網段會 ARP flux**`mgmt_ping_monitor.sh` 讓 eth0/eth1 同時 up,同一個 /24 下
對端 ARP 可能由任一張卡回應,回包落在沒送封包的那張。`ARP_STRICT=1` 擋掉;判讀時看 `RESULT` 行的
`nic_tx`/`nic_rx`(該卡自己的 packet delta)—— **ping 成功但 `nic_tx` 接近 0 = 封包從另一張卡出去的**
- ⚠️ **`ip -s link` 的欄位順序是 `bytes packets errors ...`**,要 packet 數取的是**第 2 欄**不是第 1 欄
- ⚠️ **`mgmt_ping_monitor.sh` / `bmc_monitor*.sh` 需要 exec bit**,且**不是改用 `bash x.sh start` 就能繞過** ——
`do_start` 內部是 `setsid "$SCRIPT_PATH" __daemon`,直接執行該路徑。git 記錄是 `100644` `do_start` 內部是 `setsid "$SCRIPT_PATH" __daemon`,直接執行該路徑。git 記錄是 `100644`
Windows → 隨身碟 → DUT 也帶不動權限位元,所以佈署後要 `chmod +x ~/Blanton_Script/bmc_monitor*.sh` Windows → 隨身碟 → DUT 也帶不動權限位元,所以佈署後要 `chmod +x ~/Blanton_Script/bmc_monitor*.sh`
沒做的話 Script B 的 start 會 Permission denied、Script C `cat` 撲空,而且**不會報錯** 沒做的話 start 會 Permission denied、後面`cat` 撲空,而且**不會報錯**只會得到一份全空的 log
只會得到一份 BMC 段全空的 log - ⚠️ **這幾支 monitor 的 `tail` 子命令是 `tail -n 50 -f`,永遠不返回**。TTL 裡絕對不要呼叫它
- ⚠️ **`bmc_monitor*.sh``tail` 子命令是 `tail -n 50 -f`,永遠不返回**。TTL 裡絕對不要呼叫它
Script C 曾因此卡死,`18ce7a1` 改成直接 `cat log/*.log` Script C 曾因此卡死,`18ce7a1` 改成直接 `cat log/*.log`
- ⚠️ **背景 monitor 的 `start` 只等 1 秒就返回**,不是跑完一輪。Script A 要做一輪基線的話,
`start``stop` 中間必須 `pause`(目前是 45 秒,一輪約 30 秒)—— 少了它 log 只會有 START banner
- ⚠️ **風扇那段設完會 `wait "Set all configured fan channels to"` 等確認提示再送 `y`**。該提示沒出現
(版本改了、字串換了)就會永遠卡住且不報錯 —— 現場看起來像 Script A 當掉
- ⚠️ **`show_dmesg.ttl` 結尾會 `dmesg -C` 清空 kernel ring buffer**。這是刻意的(讓 Script C 只看到
soak 期間新產生的訊息),但代表事後在 DUT 上 `dmesg` 撈不到舊訊息 —— 內容只存在 master log 裡
- ⚠️ **DUT 的 awk 對 `%d` 會夾到 INT32**(2147483647)。線速流量的計數器是 ~10¹² 量級, - ⚠️ **DUT 的 awk 對 `%d` 會夾到 INT32**(2147483647)。線速流量的計數器是 ~10¹² 量級,
所以 awk 裡輸出大數一律用 `%.0f`double 域,精確到 2⁵³)。`blanton_traffic_linespeed.sh` 所以 awk 裡輸出大數一律用 `%.0f`double 域,精確到 2⁵³)。`blanton_traffic_linespeed.sh`
V0.4.1 已修;寫新的 awk 報表時要記得同一件事 V0.4.1 已修;寫新的 awk 報表時要記得同一件事
+65
View File
@@ -0,0 +1,65 @@
# v1.0.7 — Management-link ping test, and the fans start where you set them
## ✨ New features
**10G / 1G management port ping test**
* New `mgmt_ping_monitor.sh` exercises both management NICs in one loop: it brings each up with iproute2, waits for the link to settle, then pings that NIC's own target — `eth0``192.168.1.30`, `eth1``192.168.1.31`.
* Runs detached like the other monitors: `start` / `stop` / `status` / `fg` / `summary`. `start` clears the previous log, so one run means one log.
* Every leg writes a single greppable line — `RESULT 10G eth0 -> 192.168.1.30 tx=10 rx=10 loss=0% rtt_avg=1.743ms nic_tx=10 nic_rx=10 PASS` — so a whole soak can be read with `summary` instead of scrolling.
* `nic_tx` / `nic_rx` are that interface's own counter delta across the burst. If a ping passes but they stay near zero, the traffic left on the *other* NIC and the number does not belong to this one.
* Script A runs one round as a baseline; Script B leaves it running through the soak.
**Fan speed is set at the start of every run**
* `FAN_SPEED` in `config.ttl` (default 30) is applied by Script A, so a run no longer inherits whatever the previous test left the fans doing.
* Both MAX31790 controllers are rebound first, which clears a controller left in an odd state.
**Stress load moved onto the platform's own job runner**
* CPU, DDR, SSD and USB stress now go through `bgctl`, replacing the `~/hammer/tools/stress_*.py` scripts. `bgctl list` shows what is running; Script C stops everything with `bgctl stop --all`.
* BMC DDR stress runs through `bmc-manager`, and a new BMC USB test discovers the `cdc_ncm` interface and pings across it for four hours under `systemd-run`.
**More of the DUT on record**
* Script A now captures the boot image, `show version`, firmware status, system EEPROM, SSD health, TPM version and NVMe SMART data before the test starts.
* PCIe AER now sits next to `ras-mc-ctl --summary`, and Script C closes with NVMe health so a disk that degraded during the soak is visible.
* `dmesg` capture uses human-readable timestamps and one combined error pattern, then clears the ring buffer — so Script C's dmesg shows only what the soak produced, not everything since boot.
* Each run opens with `hw-test-session start` and closes with `finish`; Script C copies the job logs to `/mnt/usb/jobs-<date>_<time>` so they leave the DUT with the run they belong to.
## 🐛 Bug fixes
**Script A no longer hangs waiting for ports that are already up**
* The readiness gate required *more than* 216 ports up, but a fully cabled unit reports exactly 216 — 108 loopback pairs times two. Script A sat in the poll loop forever on the very state it was waiting for. It now proceeds at 216.
* The comments in `wait_init.ttl` still described the old "more than" rule; they now match the code, so the off-by-one cannot be reintroduced by reading the wrong line.
**The macro no longer runs ahead of the DUT during the soak**
* Script B set a 15-second wait cap for the `cdc_ncm` probe and never restored it. In Tera Term that cap is global, so it stayed in force for everything after — including the traffic setup, which walks 108 VLANs per switch unit and takes far longer. A timed-out wait returns without the prompt, leaving every later command issued a step early.
**Fan controller is no longer bound twice**
* One of the two MAX31790 controllers received a second `bind` immediately after the first, which could only fail because the driver was already attached.
## 📦 Downloads
| File | Contents |
|---|---|
| `Script_ABC_Blanton_V1.0.7.zip` | The full Tera Term working directory, including `Blanton_Script/` to copy onto the DUT |
**After copying to the DUT:**
```bash
chmod +x ~/Blanton_Script/*.sh # required -- see below
grep -rlU $'\r' ~/Blanton_Script # expect no output
```
The `chmod +x` is not optional. `mgmt_ping_monitor.sh` is executed rather than sourced and re-execs its own path to daemonise, and neither git (mode 100644) nor a Windows/USB copy carries the execute bit. Without it `start` fails with `Permission denied` and the later `cat` finds nothing — **the section ends up empty and nothing reports an error**.
## ⚠️ Before you run
* **Declare the switch population.** `SWB_UNIT0` / `SWB_UNIT1` in `config.ttl` say which switch units this DUT has. Traffic and the readiness gate both follow them: a single unit gets `-u 0` or `-u 1`, and with neither set the traffic stage is skipped instead of failing against hardware that is not there.
* **Both management NICs stay up.** If they share a subnet, keep `ARP_STRICT=1` — otherwise the target's ARP can be answered by either NIC and a reply may arrive on the one that did not send.
* **Management connectivity is exercised, not preserved.** Drive the run from the serial console.
* **Two settings persist to `config_db.json`** and survive a reboot: LLDP is left disabled, and the 100G uplinks are left configured. Restore them before the DUT moves on.
## 🔗 Links
* [ARCHITECTURE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/ARCHITECTURE.md) — test flow, data models, constraints
* [CLAUDE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/CLAUDE.md) — commands and the bench gotchas
**Full changelog:** [V1.0.5...V1.0.7](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/compare/V1.0.5...V1.0.7)
+61
View File
@@ -0,0 +1,61 @@
# v1.0.8 — Both management links tested at once, and ARP stops looking like packet loss
## ✨ New features
**Both management NICs are pinged simultaneously and continuously**
* `mgmt_ping_monitor.sh` no longer alternates fixed bursts. `start` configures both NICs, then pings from each at the same time and keeps accumulating until stopped — so a soak-long run is one continuous measurement rather than a series of snapshots.
* `stop` renders a report into the log: the last 20 entries per NIC, each one's ping statistics and PASS/FAIL, then `ip -s link show` for both interfaces.
* Every line is timestamped, and a request that got no reply prints a marker instead of merely being absent — a drop is visible in the tail, not inferred from a gap in the sequence numbers.
* `status` shows both pids and how many replies each NIC has received so far, which is the quick way to see one side is dead without waiting for the report.
**The USB stress target is detected, not assumed**
* New `usb_target.sh` reports an inserted USB device's node, mount point or by-id path. Script B asks it for the node and builds the stress command from the answer, so a stick that enumerates as `sdb` no longer sends a write test at whatever `/dev/sda1` happens to be.
* It excludes whatever disk backs `/` and `/host` — this platform can boot from a USB DOM, which also reports as USB — refuses to guess when several USB disks are present, and when it mounts, verifies the result is actually writable rather than trusting that `mount` succeeded.
**BMC I2C integrity is exercised through the soak**
* `bmc_monitor.sh` is back in the run and now writes two complementary patterns to a BMC scratch register and reads each back. One pattern alone cannot catch a bit stuck the same way it was written; repeating the pair through the soak turns an intermittent I2C fault into something the log records rather than something the tester has to witness.
**Test artefacts leave the DUT with the run**
* Script C copies `bmc_poll.log` and `mgmt_ping.log` into the job directory before it is archived to USB, so the monitors' output travels with the `bgctl` job logs.
* DDR stress now runs continuously instead of stopping after 100 passes, matching the other soak loads.
## 🐛 Bug fixes
**Three FAILs that were not link faults**
* A bench run reported `FAIL=3` with zero NIC errors, zero drops and sub-millisecond replies. Every failure was missing exactly `icmp_seq=1` and nothing else: once the neighbour entry for the target expires, the first echo request is spent resolving ARP and `ping` counts it as loss. A discarded warm-up ping per NIC now absorbs that, which is what lets the loss threshold stay at zero and still mean something. Raising the threshold instead would have hidden genuine single-packet loss.
**Job logs were being cleared before they were collected**
* Script C ran `bgctl reset --yes` while killing processes — before the job directory is copied to USB. Moved to after the archive, so a run's own logs are collected before anything clears them.
**The fan controller was bound twice**
* One of the two MAX31790 controllers received a second `bind` immediately after the first, which could only fail because the driver was already attached.
## 📦 Downloads
| File | Contents |
|---|---|
| `Script_ABC_Blanton_V1.0.8.zip` | The full Tera Term working directory, including `Blanton_Script/` to copy onto the DUT |
**After copying to the DUT:**
```bash
chmod +x ~/Blanton_Script/*.sh # see below
grep -rlU $'\r' ~/Blanton_Script # expect no output
```
`mgmt_ping_monitor.sh` no longer re-execs itself, so `bash mgmt_ping_monitor.sh start` works without the execute bit. `bmc_monitor.sh` and `usb_target.sh` still need it — and neither git (mode 100644) nor a Windows/USB copy carries it. Without the `chmod`, `start` fails with `Permission denied` and the later `cat` finds nothing: **the section ends up empty and nothing reports an error.**
## ⚠️ Before you run
* **Declare the switch population.** `SWB_UNIT0` / `SWB_UNIT1` in `config.ttl` say which switch units this DUT has. Traffic and the readiness gate both follow them; with neither set the traffic stage is skipped rather than failing against hardware that is not there.
* **Set the fan speed you want.** `FAN_SPEED` in `config.ttl` is applied at the start of every run.
* **Management connectivity is in use, not preserved.** Both NICs are up and pinging — drive the run from the serial console.
* **If the two management NICs share a subnet, keep `ARP_STRICT=1`.** Otherwise the target's ARP can be answered by either NIC and a reply may arrive on the one that did not send. Check `nic_tx` / `nic_rx` on the RESULT line: they are that interface's own counter delta.
* **Two settings persist to `config_db.json`** and survive a reboot: LLDP is left disabled and the 100G uplinks are left configured. Restore them before the DUT moves on.
## 🔗 Links
* [ARCHITECTURE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/ARCHITECTURE.md) — test flow, data models, constraints
* [CLAUDE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/CLAUDE.md) — commands and the bench gotchas
**Full changelog:** [V1.0.7...V1.0.8](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/compare/V1.0.7...V1.0.8)
+116 -11
View File
@@ -1,7 +1,7 @@
; ============================================================================= ; =============================================================================
; Script A for Blanton ; Script A for Blanton
; Version : V1.0.5 ; Version : V1.0.8
; Date : 2026-08-21 ; Date : 2026-08-25
; Author : ETWen ; Author : ETWen
; ============================================================================= ; =============================================================================
; Version History: ; Version History:
@@ -25,6 +25,39 @@
; V1.0.5 2026-08-21 config.ttl Add SWB_UNIT0/SWB_UNIT1 ; V1.0.5 2026-08-21 config.ttl Add SWB_UNIT0/SWB_UNIT1
; utils/wait_init.ttl Follow config.ttl SWB_UNIT0/SWB_UNIT1 ; utils/wait_init.ttl Follow config.ttl SWB_UNIT0/SWB_UNIT1
; ScriptA/B/C Traffic follows SWB_UNIT: -u 0 / -u 1 / skip ; ScriptA/B/C Traffic follows SWB_UNIT: -u 0 / -u 1 / skip
; V1.0.6 2026-08-24 utils/show_dmesg.ttl Edited dmesg grep i2c & clear event
; ScriptA HW Test Session
; ScriptC HW Test Session Finish
; ScriptA bmc-first-enroll, bmc version/status
; ScriptB Stress test DDR, SSD, USB, BMC DDR, BMC USB Stress test
; ScriptC BMC USB Stress test result
; ScriptA DUT Info
; ScriptAC 10G/1G MGMT ping
; ScriptA Edited 100G Port Status
; ScriptA, utils/setup_pmon.ttl Add TPM
; ScriptAC NVME Error Info
; ScriptAC Add PCIe Summary
; V1.0.7 2026-08-25 config.ttl Add FAN_SPEED
; ScriptA Add Fan Ctrl (max31790 rebind + fan-speed-control.sh)
; ScriptA Clear /host/hw-eval/current/jobs before the run
; ScriptC Copy stress jobs log to USB, timestamped
; utils/wait_init.ttl WT_INTERVAL 10 -> 60, fix WT_MIN comments
; ScriptC Disable BMC USB journalctl dump
; V1.0.8 2026-08-25 Blanton_Script/bmc_monitor.sh Add BMC I2C write/read-back pattern test
; ScriptB Re-enable bmc_monitor.sh start
; ScriptC Re-enable bmc_monitor.sh stop + cat bmc_poll.log
; ScriptC Copy bmc_poll.log + mgmt_ping.log into jobs/ before the USB archive
; Blanton_Script/usb_target.sh Created - auto-detect USB device node / mount point
; ScriptB USB stress uses the detected node, not a hard-coded /dev/sda1
; ScriptB DDR memtester runs continuously (drop the 100-pass limit)
; ScriptC Move bgctl reset --yes after the USB archive
; Blanton_Script/mgmt_ping_monitor.sh V3.0.0 - both NICs ping simultaneously, accumulating
; + ARP warm-up so a stale neighbour entry is not counted as loss
; + stop renders last-N per NIC, stats, and both ip -s link show
; .gitignore Ignore monitor *.raw / *.pid / *.state
; ScriptA MGMT ping baseline 45s -> 30s, clear the log afterwards
; ScriptB Show mgmt_ping status after start
; ScriptC Show mgmt_ping status before stop, cat the report after
; ============================================================================= ; =============================================================================
include "config.ttl" include "config.ttl"
@@ -59,21 +92,75 @@ gettime timestr
sprintf2 cmd 'sudo date -s "%s %s"' datestr timestr sprintf2 cmd 'sudo date -s "%s %s"' datestr timestr
sendln cmd sendln cmd
; ========== Show Build Name ========== ;Stress log clear
wait prompt_sonic_root wait prompt_sonic_root
sendln "show boot" sendln "rm /host/hw-eval/current/jobs/*"
; ========== EEPROM Info ========== ; ========== HW Test Session ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "hpe-eeprom-tlv show --bus 4 --addr 0x50" sendln "hw-test-session start"
wait prompt_sonic_root
sendln "hw-test-session log"
wait prompt_sonic_root
sendln "hw-test-session status"
; ========== FAN SPEED ==========
wait prompt_sonic_root
sendln "echo 18-0020 > /sys/bus/i2c/drivers/max31790/unbind"
wait prompt_sonic_root
sendln "sleep .5"
wait prompt_sonic_root
sendln "echo 18-0020 > /sys/bus/i2c/drivers/max31790/bind"
wait prompt_sonic_root
sendln "echo 25-0020 > /sys/bus/i2c/drivers/max31790/unbind"
wait prompt_sonic_root
sendln "sleep .5"
wait prompt_sonic_root
sendln "echo 25-0020 > /sys/bus/i2c/drivers/max31790/bind"
wait prompt_sonic_root
sprintf2 fan_ctrl_cmd 'fan-speed-control.sh %d' FAN_SPEED
sendln fan_ctrl_cmd
wait "Set all configured fan channels to"
sendln "y"
; ========== Wait DUT Init ========== ; ========== Wait DUT Init ==========
wait prompt_sonic_root
include "utils/wait_init.ttl" include "utils/wait_init.ttl"
; ========== BMC version ========== ; ========== DUT Info ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "bmc-manager run 'cat /etc/issue'" sendln "show boot"
wait prompt_sonic_root
sendln "show version"
wait prompt_sonic_root
sendln "fwutil show status"
wait prompt_sonic_root
sendln "show system-memory"
wait prompt_sonic_root
sendln "show service"
wait prompt_sonic_root
sendln "show platform syseeprom"
wait prompt_sonic_root
sendln "show platform ssdhealth"
wait prompt_sonic_root
sendln "qfx5252-tpm-version"
wait prompt_sonic_root
sendln "/usr/sbin/nvme smart-log -H /dev/nvme0"
wait prompt_sonic_root
sendln "/usr/sbin/smartctl -x /dev/nvme0"
; ========== BMC version ==========
;wait prompt_sonic_root
;sendln "bmc-manager run 'cat /etc/issue'"
wait prompt_sonic_root
sendln "bmc-first-enroll"
wait prompt_sonic_root
sendln "bmc-manager version"
wait prompt_sonic_root
sendln "bmc-manager status"
; ========== Load Shell Script ========== ; ========== Load Shell Script ==========
wait prompt_sonic_root wait prompt_sonic_root
@@ -98,6 +185,9 @@ include "utils/show_pcie_error_reg_DDR.ttl"
include "utils/show_pcie_error_reg_I210.ttl" include "utils/show_pcie_error_reg_I210.ttl"
include "utils/show_pcie_error_reg_SSD.ttl" include "utils/show_pcie_error_reg_SSD.ttl"
wait prompt_sonic_root
sendln "/usr/sbin/ras-mc-ctl --summary"
; ========== Check PCIE tree + PCIE link status + GET PCIE bandwidth Test ========== ; ========== Check PCIE tree + PCIE link status + GET PCIE bandwidth Test ==========
include "utils/pcie_bus.ttl" include "utils/pcie_bus.ttl"
@@ -115,12 +205,27 @@ include "utils/show_dmesg.ttl"
;wait prompt_sonic_root ;wait prompt_sonic_root
;flushrecv ;flushrecv
; ========== 10G/1G MGMT Ping Test ==========
wait prompt_sonic_root
sendln "./Blanton_Script/mgmt_ping_monitor.sh start"
wait prompt_sonic_root
pause 30
sendln "~/Blanton_Script/mgmt_ping_monitor.sh stop"
wait prompt_sonic_root
sendln "cat ~/Blanton_Script/log/mgmt_ping.log"
wait prompt_sonic_root
sendln "~/Blanton_Script/mgmt_ping_monitor.sh clear"
; ========== 100G Port Status ========== ; ========== 100G Port Status ==========
; Lpmode Disable
wait prompt_sonic_root
sendln "sudo sfputil lpmode off Ethernet513"
wait prompt_sonic_root
sendln "sudo sfputil lpmode off Ethernet514"
wait prompt_sonic_root wait prompt_sonic_root
sendln "config interface -n asic0 startup Ethernet513" sendln "config interface -n asic0 startup Ethernet513"
wait prompt_sonic_root wait prompt_sonic_root
sendln "config interface -n asic0 shutdown Ethernet513"
wait prompt_sonic_root
sendln "config interface -n asic1 startup Ethernet514" sendln "config interface -n asic1 startup Ethernet514"
wait prompt_sonic_root wait prompt_sonic_root
sendln "config save -y" sendln "config save -y"
+63 -20
View File
@@ -20,45 +20,88 @@ sendln 'chmod +x ~/hammer/tools/amd/mlucas-avx2'
wait prompt_sonic_root wait prompt_sonic_root
sendln "~/hammer/tools/amd/mlucas-avx2 -s l -cpu 0:15 > ~/hammer/tools/amd/mlucas_amm_log 2>&1 &" sendln "~/hammer/tools/amd/mlucas-avx2 -s l -cpu 0:15 > ~/hammer/tools/amd/mlucas_amm_log 2>&1 &"
; ========== BMC Stress Test ========== ; ========== BMC DDR Stress Test ==========
;wait prompt_sonic_root
;sendln "./Blanton_Script/bmc_monitor_ddr.sh start"
;wait prompt_sonic_root
;sendln "./Blanton_Script/bmc_monitor.sh start"
wait prompt_sonic_root wait prompt_sonic_root
sendln "./Blanton_Script/bmc_monitor_ddr.sh start" sendln "bgctl run bmc-manager run '/usr/bin/memtester 64M 1'"
; ========== BMC Monitor Test ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "./Blanton_Script/bmc_monitor.sh start" sendln "./Blanton_Script/bmc_monitor.sh start"
; ========== DDR MEMORY STRESS TEST ========== ; ========== DDR MEMORY STRESS TEST ==========
;wait prompt_sonic_root
;sendln "bgctl run /usr/sbin/memtester 1G 100"
wait prompt_sonic_root wait prompt_sonic_root
sendln "python ~/hammer/tools/stress_mem.py &" sendln "bgctl run /usr/sbin/memtester 1G" ; Continue Execture
wait prompt_sonic_root
sendln "python ~/hammer/tools/stress_mem.py &"
wait prompt_sonic_root
sendln "python ~/hammer/tools/stress_mem.py &"
wait prompt_sonic_root
sendln "python ~/hammer/tools/stress_mem.py &"
wait prompt_sonic_root
sendln "python ~/hammer/tools/stress_mem.py &"
; ========== SSD read/write ========== ; ========== SSD read/write ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "python ~/hammer/tools/stress_ssd.py &" sendln "bgctl run qfx5252-stress-ssd 1G --runtime 3600 --passes 5 --write --force --log /host/hw-eval/current/qfx5252-stress-ssd.log"
; ========== USB read/write ========== ; ========== USB read/write ==========
; ---- Get USB partition node -> usb_dev ----
usb_dev = ''
wait prompt_sonic_root wait prompt_sonic_root
sendln "python ~/hammer/tools/stress_usb.py &" sendln 'D=$(bash ~/Blanton_Script/usb_target.sh -o dev 2>/dev/null); echo "<<D""EV=${D:-NONE}>>"'
waitregex '<<DEV=([^>]*)>>'
usb_dev = groupmatchstr1
wait prompt_sonic_root
sprintf2 cmd "bgctl run qfx5252-stress-usb %s 256M --runtime 60 --passes 1 --write --log /host/hw-eval/qfx5252-stress-usb.log" usb_dev
sendln cmd
;wait prompt_sonic_root
;sendln "bgctl run qfx5252-stress-usb /dev/sda1 256M --runtime 60 --passes 1 --write --log /host/hw-eval/qfx5252-stress-usb.log"
; ========== SHOW Background job ========== ; ========== SHOW Background job ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "jobs" sendln "jobs"
wait prompt_sonic_root
sendln "bgctl list"
; ========== BMC USB Test ==========
wait prompt_sonic_root
sendln 'echo "NCMIF=$(ls -d /sys/bus/usb/drivers/cdc_ncm/*/net/* 2>/dev/null | head -n 1 | xargs -r basename)"'
timeout = 15
waitregex 'NCMIF=[A-Za-z0-9_-]+'
if result = 0 then
messagebox 'cdc_ncm interface not found. Ping test skipped.' 'ERROR'
goto skip_ping
endif
strlen matchstr
cut_len = result - 6 ; 'NCMIF=' = 6 chars
strcopy matchstr 7 cut_len ncm_if ; -> ncm_if = "eth2" / "eth3"
sprintf2 cmd 'ip -br link show %s' ncm_if
wait prompt_sonic_root
sendln cmd
wait prompt_sonic_root
sendln 'systemctl stop qfx5252-bmc-usb-net-test.service 2>/dev/null || true'
wait prompt_sonic_root
sendln 'systemctl reset-failed qfx5252-bmc-usb-net-test.service 2>/dev/null || true'
sprintf2 cmd 'systemd-run --unit=qfx5252-bmc-usb-net-test --property=Type=exec /usr/bin/ping -I %s -i 1 -W 2 -w 14400 192.168.200.200' ncm_if
wait prompt_sonic_root
sendln cmd
:skip_ping
; restore the default (no cap): the traffic init below walks 108 VLANs per
; unit and takes far longer than the 15 s set for the NCM probe above.
timeout = 0
; ========== 10G/1G MGMT Ping Test ==========
wait prompt_sonic_root
sendln "./Blanton_Script/mgmt_ping_monitor.sh start"
wait prompt_sonic_root
pause 3
sendln "./Blanton_Script/mgmt_ping_monitor.sh status"
; ========== 100G Port ========== ; ========== 100G Port ==========
; Lpmode Disable
wait prompt_sonic_root
sendln "sudo sfputil lpmode off Ethernet513"
wait prompt_sonic_root
sendln "sudo sfputil lpmode off Ethernet514"
wait prompt_sonic_root wait prompt_sonic_root
sendln "show interfaces status Ethernet513,Ethernet514" sendln "show interfaces status Ethernet513,Ethernet514"
+64 -5
View File
@@ -12,6 +12,22 @@ pause 1
; ========== Kill Process ========== ; ========== Kill Process ==========
include "utils/kill_all_process.ttl" include "utils/kill_all_process.ttl"
; bgctrl all stop
wait prompt_sonic_root
sendln "bgctl stop --all"
wait prompt_sonic_root
sendln "bgctl stop --all"
; BMC USB Test STOP
wait prompt_sonic_root
sendln "systemctl stop qfx5252-bmc-usb-net-test.service"
; 10G/1G MGMT Test STOP
wait prompt_sonic_root
sendln "~/Blanton_Script/mgmt_ping_monitor.sh status"
wait prompt_sonic_root
sendln "~/Blanton_Script/mgmt_ping_monitor.sh stop"
wait prompt_sonic_root
sendln "cat ~/Blanton_Script/log/mgmt_ping.log"
; ========== TAKE DATA ========== ; ========== TAKE DATA ==========
include "utils/setup_pmon.ttl" include "utils/setup_pmon.ttl"
include "utils/show_dmesg.ttl" include "utils/show_dmesg.ttl"
@@ -25,22 +41,39 @@ include "utils/show_pcie_error_reg_DDR.ttl"
include "utils/show_pcie_error_reg_I210.ttl" include "utils/show_pcie_error_reg_I210.ttl"
include "utils/show_pcie_error_reg_SSD.ttl" include "utils/show_pcie_error_reg_SSD.ttl"
; ========== BMC TAKE DATA ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "./Blanton_Script/bmc_monitor_ddr.sh stop" sendln "/usr/sbin/ras-mc-ctl --summary"
; ========== BMC DDR TAKE DATA ==========
;wait prompt_sonic_root
;sendln "./Blanton_Script/bmc_monitor_ddr.sh stop"
;wait prompt_sonic_root
;sendln "./Blanton_Script/bmc_monitor.sh stop"
;wait prompt_sonic_root
;sendln "cat ./Blanton_Script/log/bmc_poll.log"
;wait prompt_sonic_root
;sendln "cat ./Blanton_Script/log/bmc_ddr.log"
; ========== BMC Monitor TAKE DATA ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "./Blanton_Script/bmc_monitor.sh stop" sendln "./Blanton_Script/bmc_monitor.sh stop"
wait prompt_sonic_root wait prompt_sonic_root
sendln "cat ./Blanton_Script/log/bmc_poll.log" sendln "cat ./Blanton_Script/log/bmc_poll.log"
; ========== BMC USB Test result ==========
;wait prompt_sonic_root
;sendln "journalctl -u qfx5252-bmc-usb-net-test.service --no-pager"
; ========== 10G/1G MGMT Test result ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "cat ./Blanton_Script/log/bmc_ddr.log" sendln "ip -s link show eth0"
wait prompt_sonic_root
sendln "ip -s link show eth1"
; ========== CHECK Stress results ========== ; ========== CHECK Stress results ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "cat ~/hammer/tools/amd/mlucas_amm_log" sendln "cat ~/hammer/tools/amd/mlucas_amm_log"
wait prompt_sonic_root
sendln "cat ~/hammer/tools/log.stress_ssd"
; ========== CHECK Traffic counters ========== ; ========== CHECK Traffic counters ==========
if SWB_UNIT0 = 1 && SWB_UNIT1 = 1 then if SWB_UNIT0 = 1 && SWB_UNIT1 = 1 then
@@ -62,6 +95,13 @@ else
endif endif
; ========== NVME Err Info ==========
wait prompt_sonic_root
sendln "/usr/sbin/nvme smart-log -H /dev/nvme0"
wait prompt_sonic_root
sendln "/usr/sbin/smartctl -x /dev/nvme0"
wait prompt_sonic_root wait prompt_sonic_root
sendln "show reboot-cause" sendln "show reboot-cause"
@@ -69,4 +109,23 @@ sendln "show reboot-cause"
wait prompt_sonic_root wait prompt_sonic_root
sendln "show uptime" sendln "show uptime"
; ========== Copy Stress Log to USB ==========
wait prompt_sonic_root
sendln "cp ./Blanton_Script/log/bmc_poll.log /host/hw-eval/current/jobs/"
wait prompt_sonic_root
sendln "cp ./Blanton_Script/log/mgmt_ping.log /host/hw-eval/current/jobs/"
wait prompt_sonic_root
getdate ts_date "%Y%m%d"
gettime ts_time "%H%M"
sprintf2 usb_dst "/mnt/usb/jobs-%s_%s" ts_date ts_time
sprintf2 cmd "cp -r /host/hw-eval/current/jobs/ %s" usb_dst
sendln cmd
; ========== HW Test Session ==========
wait prompt_sonic_root
sendln "hw-test-session finish"
wait prompt_sonic_root
sendln "bgctl reset --yes"
messagebox 'GOOD JOB! Test Case DONE' 'teraterm' messagebox 'GOOD JOB! Test Case DONE' 'teraterm'
@@ -34,6 +34,10 @@ set -u
# Add a line to add a command; comment it out to disable it. # Add a line to add a command; comment it out to disable it.
COMMANDS=( COMMANDS=(
"free -m" "free -m"
"i2ctransfer -f -y 1 w6@0x41 0x00 0xC0 0x55 0xAA 0x55 0xAA"
"i2ctransfer -f -y 1 w2@0x41 0x00 0xC0 r4"
"i2ctransfer -f -y 1 w6@0x41 0x00 0xC0 0xAA 0x55 0xAA 0x55"
"i2ctransfer -f -y 1 w2@0x41 0x00 0xC0 r4"
#"uptime" #"uptime"
#"cat /proc/loadavg" #"cat /proc/loadavg"
#"cat /proc/meminfo" #"cat /proc/meminfo"
@@ -0,0 +1,403 @@
#!/bin/bash
###############################################################################
# mgmt_ping_monitor.sh
#
# Version : V3.0.0
# Author : ETWen
# Date : 20260826
#
# Purpose : Ping from BOTH management NICs at the same time, continuously, and
# keep accumulating until stopped:
#
# 10G (eth0) -> TARGET_10G
# 1G (eth1) -> TARGET_1G
#
# `stop` renders a report into the log: the last TAIL_LINES entries
# per NIC, each one's ping statistics and verdict, then
# `ip -s link show` for both interfaces.
#
# Notes : - Both NICs are configured with iproute2 `replace`, which is
# idempotent, and both are left up. On a shared subnet the target's
# ARP can be answered by either NIC, so ARP_STRICT keeps each one
# to its own address.
# - ARP WARM-UP: the first packet after a neighbour entry expires is
# spent resolving ARP and is counted as loss. On the bench this
# produced three FAILs whose only missing packet was icmp_seq=1,
# every time, with zero NIC errors or drops. One discarded ping
# before the measured run removes that artefact, which is what lets
# MAX_LOSS_PCT stay at 0 and still mean something.
# - MANAGEMENT CONNECTIVITY IS IN USE while this runs. Drive it from
# the serial console.
#
# Version History
# V1.0.0 20260824 Initial Version (ifconfig, one NIC at a time)
# V2.0.0 20260824 iproute2; both NICs stay up; per-NIC target;
# ip -s link show + TX/RX delta per leg
# V3.0.0 20260826 Both NICs ping SIMULTANEOUSLY and continuously instead
# of alternating fixed bursts. Add the ARP warm-up.
# `stop` renders last-N per NIC + statistics + both
# ip -s link outputs into the log.
###############################################################################
set -u
###############################################################################
# User Configurable Section -- normally this is the only part you need to edit
###############################################################################
# --- 10G management port ---
IF_10G="eth0"
IP_10G="192.168.1.99"
TARGET_10G="192.168.1.30" # host this NIC pings
PLEN_10G=24 # prefix length, e.g. 24 = /24
GW_10G="" # default gateway; empty = do not touch routing
METRIC_10G=100 # lower metric wins for off-subnet traffic
# --- 1G management port ---
IF_1G="eth1"
IP_1G="192.168.1.101"
TARGET_1G="192.168.1.31"
PLEN_1G=24
GW_1G=""
METRIC_1G=200
# --- ping ---
PING_INTERVAL_SEC=1 # seconds between echo requests (ping -i)
MAX_LOSS_PCT=0 # loss above this marks the NIC FAIL
TAIL_LINES=20 # how many recent entries per NIC the report shows
# Extra ping flags. -D timestamps every line, -O prints a marker for a request
# that got no reply, so a drop is visible in the tail instead of just missing.
# Clear this if the platform's ping does not accept them.
PING_EXTRA_OPTS="-D -O"
# One discarded ping per NIC before the measured run, to resolve ARP. Without
# it the first packet of the run is lost to neighbour resolution and looks
# exactly like a link fault. 0 disables.
ARP_WARMUP=1
ARP_WARMUP_TIMEOUT=2 # seconds to wait for the warm-up reply
# Both NICs stay up. If they share a subnet the target's ARP can be answered by
# either one, so a reply may arrive on the NIC that did not send. This applies
# arp_ignore=1 / arp_announce=2 to both. RAM only -- reverts on reboot.
ARP_STRICT=1
# Prefix for the ip commands; empty because this already runs as root.
SUDO=""
# Seconds to wait after bringing a link up before pinging.
LINK_SETTLE_SEC=5
# Logging
LOG_DIR="" # empty = <script directory>/log
LOG_NAME="mgmt_ping.log"
# What `start` does with the log left behind by the previous run:
# new = discard it (default -- one run, one log)
# archive = rename it to <log>.YYYYmmdd-HHMMSS first
# append = keep it
START_LOG_MODE="new"
###############################################################################
# Internal
###############################################################################
SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")"
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
[ -n "${LOG_DIR}" ] || LOG_DIR="${SCRIPT_DIR}/log"
LOG_FILE="${LOG_DIR}/${LOG_NAME}"
BASE="${LOG_DIR}/${LOG_NAME%.log}"
PID_FILE="${BASE}.pid"
STATE_FILE="${BASE}.state"
RAW_10G="${BASE}_${IF_10G}.raw"
RAW_1G="${BASE}_${IF_1G}.raw"
ts() { date '+%Y-%m-%d %H:%M:%S'; }
die() { printf '[ERROR] %s\n' "$*" >&2; exit 1; }
# "<rx_packets> <tx_packets>" for an interface. ip -s link orders the columns
# "bytes packets errors ...", so packets is $2, not $1.
if_counters() {
${SUDO} ip -s link show "$1" 2>/dev/null | awk '
/RX:/ { getline; rx = $2 }
/TX:/ { getline; tx = $2 }
END { printf "%s %s", (rx == "" ? 0 : rx), (tx == "" ? 0 : tx) }'
}
# Bring the NIC up and (re)apply its address and default route.
setup_iface() {
local iface="$1" ipaddr="$2" plen="$3" gw="$4" metric="$5"
${SUDO} ip link set "${iface}" up 2>&1 || return 1
${SUDO} ip address replace "${ipaddr}/${plen}" dev "${iface}" 2>&1 || return 1
if [ -n "${gw}" ]; then
${SUDO} ip route replace default via "${gw}" dev "${iface}" \
metric "${metric}" 2>&1 || return 1
fi
return 0
}
apply_arp_strict() {
local i
[ "${ARP_STRICT}" -eq 1 ] || return 0
for i in "${IF_10G}" "${IF_1G}"; do
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_ignore=1" 2>/dev/null
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_announce=2" 2>/dev/null
done
}
# Resolve the neighbour so the measured run does not spend its first packet on
# ARP. Result is deliberately discarded.
arp_warmup() {
local iface="$1" target="$2"
[ "${ARP_WARMUP}" -eq 1 ] || return 0
ping -I "${iface}" -c 1 -W "${ARP_WARMUP_TIMEOUT}" "${target}" >/dev/null 2>&1
return 0
}
# echo "pid_10g pid_1g" and return 0 when both are alive
is_running() {
local p0 p1
[ -f "${PID_FILE}" ] || return 1
p0="$(sed -n '1p' "${PID_FILE}" 2>/dev/null)"
p1="$(sed -n '2p' "${PID_FILE}" 2>/dev/null)"
[[ "${p0}" =~ ^[0-9]+$ ]] || return 1
[[ "${p1}" =~ ^[0-9]+$ ]] || return 1
kill -0 "${p0}" 2>/dev/null || kill -0 "${p1}" 2>/dev/null || return 1
printf '%s %s' "${p0}" "${p1}"
return 0
}
###############################################################################
# Report
###############################################################################
# The recent entries for one NIC: replies and, thanks to -O, the requests that
# got none.
tail_entries() {
grep -aE 'bytes from|no answer|Unreachable|Time to live' "$1" 2>/dev/null \
| tail -n "${TAIL_LINES}"
}
# The trailing "--- x ping statistics ---" block.
stats_block() {
sed -n '/ping statistics ---/,$p' "$1" 2>/dev/null
}
# render_leg <label> <iface> <target> <raw> <rx0> <tx0>
render_leg() {
local label="$1" iface="$2" target="$3" raw="$4" rx0="$5" tx0="$6"
local st tx rx loss avg verdict c1 rx1 tx1 drx dtx
printf -- '----- %s : %s -> %s : last %s entries -----\n' \
"${label}" "${iface}" "${target}" "${TAIL_LINES}"
tail_entries "${raw}"
printf '\n'
st="$(stats_block "${raw}")"
printf '%s\n' "${st}"
tx="$(printf '%s' "${st}" | sed -n 's/^\([0-9]\+\) packets transmitted.*/\1/p' | tail -1)"
rx="$(printf '%s' "${st}" | sed -n 's/.*[, ]\([0-9]\+\) received.*/\1/p' | tail -1)"
loss="$(printf '%s' "${st}" | sed -n 's/.*[, ]\([0-9]\+\)% packet loss.*/\1/p' | tail -1)"
avg="$(printf '%s' "${st}" | sed -n 's|.*= [0-9.]*/\([0-9.]*\)/.*|\1|p' | tail -1)"
[ -n "${tx}" ] || tx=0
[ -n "${rx}" ] || rx=0
[ -n "${loss}" ] || loss=100
[ -n "${avg}" ] || avg="-"
c1="$(if_counters "${iface}")"; rx1="${c1% *}"; tx1="${c1#* }"
drx=$(( rx1 - rx0 )); dtx=$(( tx1 - tx0 ))
if [ "${loss}" -le "${MAX_LOSS_PCT}" ] && [ "${tx}" -gt 0 ]; then
verdict="PASS"
else
verdict="FAIL"
fi
# nic_tx/nic_rx are this interface's own counter delta over the whole run.
# A pass with nic_tx near zero means the traffic left on the other NIC.
printf '[%s] RESULT %-3s %-6s -> %-15s tx=%s rx=%s loss=%s%% rtt_avg=%sms nic_tx=%s nic_rx=%s %s\n\n' \
"$(ts)" "${label}" "${iface}" "${target}" "${tx}" "${rx}" "${loss}" "${avg}" \
"${dtx}" "${drx}" "${verdict}"
}
write_report() {
local rx0_10g tx0_10g rx0_1g tx0_1g started
started="$(sed -n '1p' "${STATE_FILE}" 2>/dev/null)"
rx0_10g="$(sed -n '2p' "${STATE_FILE}" 2>/dev/null)"; rx0_10g="${rx0_10g:-0}"
tx0_10g="$(sed -n '3p' "${STATE_FILE}" 2>/dev/null)"; tx0_10g="${tx0_10g:-0}"
rx0_1g="$(sed -n '4p' "${STATE_FILE}" 2>/dev/null)"; rx0_1g="${rx0_1g:-0}"
tx0_1g="$(sed -n '5p' "${STATE_FILE}" 2>/dev/null)"; tx0_1g="${tx0_1g:-0}"
{
printf '#############################################################\n'
printf '[%s] mgmt ping report\n' "$(ts)"
printf ' started : %s\n' "${started:-unknown}"
printf ' stopped : %s\n' "$(ts)"
printf ' 10G : %s %s/%s -> %s (metric %s)\n' \
"${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}"
printf ' 1G : %s %s/%s -> %s (metric %s)\n' \
"${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}"
printf ' ping : -i %s %s, max loss %s%%\n' \
"${PING_INTERVAL_SEC}" "${PING_EXTRA_OPTS}" "${MAX_LOSS_PCT}"
printf '#############################################################\n\n'
render_leg "10G" "${IF_10G}" "${TARGET_10G}" "${RAW_10G}" "${rx0_10g}" "${tx0_10g}"
render_leg "1G" "${IF_1G}" "${TARGET_1G}" "${RAW_1G}" "${rx0_1g}" "${tx0_1g}"
printf -- '--- ip -s link show %s ---\n' "${IF_10G}"
${SUDO} ip -s link show "${IF_10G}" 2>&1
printf '\n'
printf -- '--- ip -s link show %s ---\n' "${IF_1G}"
${SUDO} ip -s link show "${IF_1G}" 2>&1
} >> "${LOG_FILE}"
}
###############################################################################
# Sub-commands
###############################################################################
prepare_log_on_start() {
local stamp
case "${START_LOG_MODE}" in
append) return 0 ;;
archive)
if [ -s "${LOG_FILE}" ]; then
stamp="$(date '+%Y%m%d-%H%M%S')"
mv -f "${LOG_FILE}" "${LOG_FILE}.${stamp}" || die "cannot archive ${LOG_FILE}"
printf 'previous log archived: %s.%s\n' "${LOG_FILE}" "${stamp}"
fi
;;
new)
# Truncate rather than unlink, so a tail -f already attached keeps
# following the new run.
[ -f "${LOG_FILE}" ] && { : > "${LOG_FILE}" || die "cannot truncate ${LOG_FILE}"; }
;;
*) die "invalid START_LOG_MODE: ${START_LOG_MODE}" ;;
esac
}
do_start() {
local pids p0 p1 c
pids="$(is_running)" && die "already running (pids=${pids})"
command -v ip >/dev/null 2>&1 || die "ip (iproute2) not found"
command -v ping >/dev/null 2>&1 || die "ping not found"
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
prepare_log_on_start
: > "${RAW_10G}"
: > "${RAW_1G}"
setup_iface "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${GW_10G}" "${METRIC_10G}" \
|| die "cannot configure ${IF_10G}"
setup_iface "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${GW_1G}" "${METRIC_1G}" \
|| die "cannot configure ${IF_1G}"
apply_arp_strict
sleep "${LINK_SETTLE_SEC}"
# Spend the ARP resolution here, not on the first measured packet.
arp_warmup "${IF_10G}" "${TARGET_10G}"
arp_warmup "${IF_1G}" "${TARGET_1G}"
# Counter baseline, taken after the warm-up so its packets are excluded.
{ c="$(if_counters "${IF_10G}")"
printf '%s\n%s\n%s\n' "$(ts)" "${c% *}" "${c#* }"
c="$(if_counters "${IF_1G}")"
printf '%s\n%s\n' "${c% *}" "${c#* }"
} > "${STATE_FILE}"
# Both NICs ping at the same time and keep accumulating until stopped.
nohup ping -I "${IF_10G}" -i "${PING_INTERVAL_SEC}" ${PING_EXTRA_OPTS} \
"${TARGET_10G}" >> "${RAW_10G}" 2>&1 &
p0=$!
nohup ping -I "${IF_1G}" -i "${PING_INTERVAL_SEC}" ${PING_EXTRA_OPTS} \
"${TARGET_1G}" >> "${RAW_1G}" 2>&1 &
p1=$!
disown "${p0}" 2>/dev/null
disown "${p1}" 2>/dev/null
printf '%s\n%s\n' "${p0}" "${p1}" > "${PID_FILE}"
sleep 1
kill -0 "${p0}" 2>/dev/null || die "10G ping failed to start, see ${RAW_10G}"
kill -0 "${p1}" 2>/dev/null || die "1G ping failed to start, see ${RAW_1G}"
printf 'started (10G pid=%s, 1G pid=%s)\nlog: %s\n' "${p0}" "${p1}" "${LOG_FILE}"
}
do_stop() {
local pids p0 p1 i
pids="$(is_running)" || {
printf 'not running\n'
rm -f "${PID_FILE}"
return 0
}
p0="${pids% *}"; p1="${pids#* }"
# SIGINT, not SIGTERM: ping prints its statistics block on interrupt, and
# that block is what the report parses.
kill -INT "${p0}" 2>/dev/null
kill -INT "${p1}" 2>/dev/null
for (( i = 0; i < 20; i++ )); do
kill -0 "${p0}" 2>/dev/null || kill -0 "${p1}" 2>/dev/null || break
sleep 0.5
done
kill -KILL "${p0}" 2>/dev/null
kill -KILL "${p1}" 2>/dev/null
write_report
rm -f "${PID_FILE}"
printf 'stopped (10G pid=%s, 1G pid=%s)\nreport: %s\n' "${p0}" "${p1}" "${LOG_FILE}"
}
do_status() {
local pids
if pids="$(is_running)"; then
printf 'status : running (10G pid=%s, 1G pid=%s)\n' "${pids% *}" "${pids#* }"
else
printf 'status : stopped\n'
fi
printf 'log : %s\n' "${LOG_FILE}"
printf '10G : %s %s/%s -> %s\n' "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}"
printf '1G : %s %s/%s -> %s\n' "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}"
[ -f "${RAW_10G}" ] && printf '10G recv: %s\n' "$(grep -ac 'bytes from' "${RAW_10G}")"
[ -f "${RAW_1G}" ] && printf '1G recv: %s\n' "$(grep -ac 'bytes from' "${RAW_1G}")"
}
usage() {
cat <<EOF
Usage: ${SCRIPT_NAME} {start|stop|status|summary|clear}
start Configure both NICs, warm up ARP, then ping from BOTH at the same
time and keep accumulating. The previous log is discarded first.
stop Stop both pings and render the report into the log
status Show pids, configured NICs and replies received so far
summary Print just the RESULT lines from the log
clear Remove the log and the raw captures (must be stopped first)
Log file : ${LOG_FILE}
The report holds, per NIC, the last ${TAIL_LINES} entries and the ping
statistics, then "ip -s link show" for both interfaces.
Both NICs are used at once, so management connectivity is in play -- drive
this from the serial console. Edit the User Configurable Section at the top
to change interfaces, addresses, targets or the ping options.
EOF
}
###############################################################################
# Entry point
###############################################################################
case "${1:-}" in
start) do_start ;;
stop) do_stop ;;
status) do_status ;;
summary) grep -a 'RESULT' "${LOG_FILE}" 2>/dev/null || printf 'no results in %s\n' "${LOG_FILE}" ;;
clear)
is_running >/dev/null && die "still running, stop it first"
rm -f "${LOG_FILE}" "${LOG_FILE}".[0-9]* "${RAW_10G}" "${RAW_1G}" "${STATE_FILE}"
printf 'log cleared\n'
;;
-h|--help|"") usage ;;
*) printf '[ERROR] unknown sub-command: %s\n\n' "$1" >&2; usage; exit 1 ;;
esac
@@ -0,0 +1,203 @@
#!/bin/bash
###############################################################################
# usb_target.sh
#
# Version : V1.1.0
# Author : ETWen
# Date : 20260825
# Purpose : Auto-detect an inserted USB mass-storage device and report either
# its device node, its mount point, or both.
#
# stdout : requested value(s) only, for $(...) capture
# stderr : diagnostic messages
#
# Usage : usb_target.sh [-o dev|mnt|both|id] [-n] [-r seconds]
#
# -o dev print partition device node (e.g. /dev/sda1)
# -o mnt print mount point (e.g. /mnt/usb) [default]
# -o both print "<dev> <mnt>" on one line
# -o id print stable by-id path (/dev/disk/by-id/...)
# -n detect only, do not mount
# -r sec udev enumeration wait, default 15
#
# Exit : 0 success
# 1 no USB mass-storage found
# 2 multiple USB disks detected (refuse to guess)
# 3 no usable filesystem on the device
# 4 mount failed
# 5 mounted but not writable
#
# Version History
# V1.0.0 20260825 Initial Version
# V1.1.0 20260825 Add -o/-n/-r options; expose device node and by-id path
###############################################################################
set -u
MNT_BASE="/mnt/usb"
RETRY=15
OUTPUT="mnt"
DO_MOUNT=1
log() { echo "[usb] $*" >&2; } # diagnostics go to stderr; stdout stays clean
while getopts "o:nr:h" opt; do
case "$opt" in
o) OUTPUT="$OPTARG" ;;
n) DO_MOUNT=0 ;;
r) RETRY="$OPTARG" ;;
h) sed -n '3,30p' "$0" >&2; exit 0 ;;
*) log "invalid option"; exit 1 ;;
esac
done
shift $((OPTIND - 1))
case "$OUTPUT" in
dev|mnt|both|id) ;;
*) log "invalid -o value: $OUTPUT"; exit 1 ;;
esac
# "-o dev" / "-o id" alone does not require mounting.
[ "$OUTPUT" = "dev" ] && [ "$DO_MOUNT" = "1" ] && DO_MOUNT=0
[ "$OUTPUT" = "id" ] && [ "$DO_MOUNT" = "1" ] && DO_MOUNT=0
#------------------------------------------------------------------------------
# Identify the physical disk(s) backing rootfs / /host so they can be excluded.
# Needed because some platforms boot from a USB DOM, which also reports TRAN=usb.
#------------------------------------------------------------------------------
get_system_disk() {
local src
for mp in /host / ; do
src=$(findmnt -no SOURCE "$mp" 2>/dev/null) || continue
lsblk -no PKNAME "$src" 2>/dev/null | head -1
done | sort -u
}
#------------------------------------------------------------------------------
# List candidate USB disks (whole devices, not partitions).
#------------------------------------------------------------------------------
find_usb_disks() {
local sysdisks; sysdisks=$(get_system_disk)
lsblk -dn -o NAME,TYPE,TRAN,RM 2>/dev/null | while read -r name type tran rm; do
[ "$type" = "disk" ] || continue
case "$name" in loop*|ram*|dm-*|sr*|zram*) continue ;; esac
# Older util-linux may not expose the TRAN column; fall back to sysfs.
if [ "$tran" != "usb" ]; then
readlink -f "/sys/block/$name/device" 2>/dev/null | grep -q '/usb[0-9]' || continue
[ "$rm" = "1" ] || continue
fi
echo "$sysdisks" | grep -qx "$name" && { log "skip $name (system disk)"; continue; }
echo "/dev/$name"
done
}
#------------------------------------------------------------------------------
# Pick a mountable partition from a disk; fall back to the whole device for
# superfloppy layouts (filesystem written directly, no partition table).
#------------------------------------------------------------------------------
pick_partition() {
local disk="$1" p
p=$(lsblk -ln -o NAME,TYPE,FSTYPE "$disk" | \
awk '$2=="part" && $3!="" {print "/dev/"$1; exit}')
[ -n "$p" ] && { echo "$p"; return; }
[ -n "$(lsblk -dn -o FSTYPE "$disk")" ] && echo "$disk"
}
#------------------------------------------------------------------------------
# Resolve a device node to a stable /dev/disk/by-id path, if one exists.
#------------------------------------------------------------------------------
resolve_by_id() {
local dev="$1" real link
real=$(readlink -f "$dev")
for link in /dev/disk/by-id/*; do
[ -e "$link" ] || continue
case "$link" in *-part*|*) ;; esac
[ "$(readlink -f "$link")" = "$real" ] && { echo "$link"; return 0; }
done
return 1
}
#------------------------------------------------------------------------------
# Main
#------------------------------------------------------------------------------
disks=""
for i in $(seq 1 "$RETRY"); do
disks=$(find_usb_disks)
[ -n "$disks" ] && break
sleep 1
done
[ -z "$disks" ] && { log "no USB mass-storage found"; exit 1; }
n=$(echo "$disks" | wc -l)
if [ "$n" -gt 1 ]; then
log "multiple USB disks detected, refuse to guess:"
log "$disks"
exit 2
fi
disk="$disks"
part=$(pick_partition "$disk")
[ -z "$part" ] && { log "$disk has no usable filesystem"; exit 3; }
fstype=$(lsblk -no FSTYPE "$part")
log "found $part (disk=$disk fstype=$fstype)"
# --- device-node only: no mount needed ---------------------------------------
if [ "$OUTPUT" = "dev" ]; then
echo "$part"
exit 0
fi
if [ "$OUTPUT" = "id" ]; then
if byid=$(resolve_by_id "$part"); then
echo "$byid"
exit 0
fi
log "no by-id path for $part, falling back to device node"
echo "$part"
exit 0
fi
# --- mount point required ----------------------------------------------------
mp=$(lsblk -no MOUNTPOINT "$part" | head -1)
if [ -z "$mp" ] && [ "$DO_MOUNT" = "0" ]; then
log "$part is not mounted and -n was given"
exit 4
fi
if [ -z "$mp" ]; then
log "mounting $part -> $MNT_BASE"
mkdir -p "$MNT_BASE"
# exfat/ntfs are not always built into the platform kernel.
case "$fstype" in
exfat) grep -qw exfat /proc/filesystems || modprobe exfat 2>/dev/null ;;
ntfs) grep -qw ntfs3 /proc/filesystems || modprobe ntfs3 2>/dev/null ;;
esac
if ! mount -o rw,noatime "$part" "$MNT_BASE" 2>/dev/null; then
log "mount failed (fstype=$fstype)"
exit 4
fi
mp="$MNT_BASE"
# Verify it is actually writable (read-only media, dirty FAT, or full device).
if ! touch "$mp/.wtest" 2>/dev/null; then
log "mounted read-only or no space left"
umount "$mp"
exit 5
fi
rm -f "$mp/.wtest"
else
log "$part already mounted at $mp"
fi
if [ "$OUTPUT" = "both" ]; then
echo "$part $mp"
else
echo "$mp"
fi
exit 0
+4
View File
@@ -7,6 +7,10 @@ strTestcase="ENV" ;ENV,EMC,Margin...etc
EN_Margin = 0 ; 1: Enable Margin Test, 0: Disable Margin Test EN_Margin = 0 ; 1: Enable Margin Test, 0: Disable Margin Test
EN_log = 1 ; 1: Enable log, 0: Disable log EN_log = 1 ; 1: Enable log, 0: Disable log
;================================================================
; FAN_SPEED
;================================================================
FAN_SPEED = 30
;================================================================ ;================================================================
; Switch Unit Population ; Switch Unit Population
;================================================================ ;================================================================
+6 -1
View File
@@ -15,4 +15,9 @@ sendln "show platform ssdhealth"
wait prompt_sonic_root wait prompt_sonic_root
sendln "show platform leak status" sendln "show platform leak status"
wait prompt_sonic_root wait prompt_sonic_root
sendln "show platform leak channels" sendln "show platform leak channels"
wait prompt_sonic_root
sendln "tpm-dut-test fru"
wait prompt_sonic_root
sendln "tpm-dut-test tpm-read"
+8 -5
View File
@@ -1,8 +1,11 @@
; All Event
wait prompt_sonic_root wait prompt_sonic_root
sendln "date" sendln "dmesg -T | grep -Ei 'error|fail(ed|ure)?|fault|critical|panic|timeout'"
; i2c event
wait prompt_sonic_root wait prompt_sonic_root
sendln "dmesg | grep -i error" sendln "dmesg -T | grep -Ei 'error|fail(ed|ure)?|fault|critical|panic|timeout' | grep -i i2c"
; Clear Event
wait prompt_sonic_root wait prompt_sonic_root
sendln "dmesg | grep -i fail" sendln "dmesg -C"
wait prompt_sonic_root
sendln "dmesg | grep -i warning"
+8 -3
View File
@@ -4,7 +4,7 @@
; Date : 2026-08-21 ; Date : 2026-08-21
; Author : ETWen ; Author : ETWen
; ============================================================================= ; =============================================================================
; Wait until every switch unit ENABLED below reports more than WT_MIN ports up: ; Wait until every switch unit ENABLED below reports at least WT_MIN ports up:
; bcmcmd -n <u> -c ps | grep -w up | wc -l ; bcmcmd -n <u> -c ps | grep -w up | wc -l
; ;
; The DUT is not always fully populated. Which units exist is declared ONCE in ; The DUT is not always fully populated. Which units exist is declared ONCE in
@@ -34,10 +34,15 @@
; V3.1.0 2026-08-21 Take the population from SWB_UNIT0 / SWB_UNIT1 in ; V3.1.0 2026-08-21 Take the population from SWB_UNIT0 / SWB_UNIT1 in
; config.ttl instead of local copies, so this wait and the ; config.ttl instead of local copies, so this wait and the
; traffic blocks cannot drift apart. ; traffic blocks cannot drift apart.
; V3.1.1 2026-08-25 WT_INTERVAL 10 -> 60. Fix the WT_MIN comments: the test
; has been `< WT_MIN` (at least) since the threshold was
; corrected, but the text still said "more than", which is
; the off-by-one that once hung Script A.
; ============================================================================= ; =============================================================================
; ---- which units to wait for: SWB_UNIT0 / SWB_UNIT1, set in config.ttl ------ ; ---- which units to wait for: SWB_UNIT0 / SWB_UNIT1, set in config.ttl ------
WT_MIN = 216 ; an enabled unit must report MORE than this many ports up WT_MIN = 216 ; an enabled unit must report AT LEAST this many ports up
WT_INTERVAL = 10 ; seconds between polls ; (216 = 108 loopback pairs x 2, i.e. every cabled port)
WT_INTERVAL = 60 ; seconds between polls
; ----------------------------------------------------------------------------- ; -----------------------------------------------------------------------------
timeout = 60 ; per-wait cap; bcmcmd ps is not instant timeout = 60 ; per-wait cap; bcmcmd ps is not instant