Compare commits

...
5 Commits
Author SHA1 Message Date
etwenandClaude Opus 5 b911ccd658 docs: Record V1.2.2 in Script A's history and add the release note
Script A: four entries under V1.2.2 for blanton_ber.sh v1.1.0 and
v1.1.1, bcm_100g_dsc_cycle.sh, and tools/100G_dsc_cycle.txt; header date
to 2026-09-15.

CLAUDE.md: status to V1.2.2; the BER paragraph notes port 268 and the
per-unit group summary; a commands block for bcm_100g_dsc_cycle.sh; the
BER command block notes the three summary rows and that a zero-lane
group is a failure; and a new gotcha that 'prbs get' output is data, not
diagnostics.

ARCHITECTURE.md: Script A ref to V1.2.2; the BER row covers port 268 and
the group summary; a Key Features row for the 100G cycle test; both new
files in the tree; the git tag checkbox to V1.2.2; a known issue for the
unverified cycle script; and one recording that a completeness check
which cannot see a failure has now bitten this repo three times -
bcm_dsc_scan.sh counting its own markers, blanton_ber.sh reading lane
results as shell errors, and the power-cycle report parser turning a
100% packet loss into "no data".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-09-15 21:27:40 +08:00
etwenandClaude Opus 5 dd558a7b0f feat(dsc): Add the 100G link down/up cycle test
bcm_100g_dsc_cycle.sh tunes once - gbsyncd paused, serdes-tx-set on
sys 0..7, tx_taps per lane - then repeats: port 268 en=0, wait, en=1,
wait, phydiag 268 dsc on both units, and the retimer read back with
serdes-tx-get / serdes-rx-get for sys 0..7. One log per run under
~/dsc_log, split by "### TEST CYCLE n/N", ending in
"### DSC_CYCLE_DONE rc=<n>" for a TTL wait to key on.

Waiting for a link state polls "bcmcmd ps <port>" rather than reading
"show interfaces status". "port en=" changes the SDK, and the SONiC
table only follows once gbsyncd and orchagent have synced it - and this
test pauses gbsyncd, so the SONiC table was the wrong thing to wait on.
The state is read from the row whose first field ends in "(<port>)", and
"down" is tested as "not up" so !ena counts.

All output is ASCII English on purpose: the DUT serial console mangles
UTF-8, and a Chinese error message came back as "ERROR: -c ? 1" - an
error you cannot read is worse than none. -d echoes every command as a
grey [CMD] line, CMD_DELAY paces them, and -n is a dry run that touches
nothing and sleeps not at all. The cycle count may be given as a bare
argument, which is what people type first.

Not verified on hardware. gbsyncd is unpaused again on exit, Ctrl-C
included.

tools/100G_dsc_cycle.txt is the hand-typed bench version of the same
commands, kept the way BER_Test.txt and TR518.txt are, so the script can
be checked against what a person would actually run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-09-15 21:27:25 +08:00
etwenandClaude Opus 5 690be5db7d feat(ber): Measure port 268, and stop reading lane results as shell errors
The port list ran 256-259 straight to 270-308, so ce0(268) - the 100G
uplink - was never measured. It is now its own group:

    BER_PORTS_200G   the cabled 200G ports
    BER_PORTS_100G   268, one per unit

They go out on one phydiag command and come back reported apart. The
summary gives each unit three rows - 200G ports, 100G p268, TOTAL -
because unit 0 and unit 1 each have their own ce0(268), and a marginal
uplink lane averaged into 432 others is invisible. TOTAL is not a
hand-added sum: every lane is folded into its own group and into ALL
separately, so the parts cannot drift from the whole. A group that
reports no lane at all counts as a failure - silence on the port under
test is the easiest way to ship a green report that measured nothing.

Separately, a fix that has been needed since v1.0.0. init kept answering

    [ERR ] unit 1: the BCM shell rejected 'phydiag ... prbs get'

on runs that were working. 'prbs get' prints one line per lane, a lane
that has not locked says "PRBS Error!", and the rejection check grepped
the whole output case-insensitively for ERROR|Invalid|... - so the
measurement matched the pattern meant for shell diagnostics.

The false alarm was the smaller half. The function returned early, which
skipped the lock count that was supposed to judge those very lanes: the
operator saw an error they could not act on, and did not see which lanes
had failed to lock. Per-lane lines are now excluded from the check, the
match is case-sensitive, and a genuine rejection prints the offending
line.

Developed on the bench as blanton_ber_test.sh and folded back over this
file once proven, so there is one BER script again rather than two with
diverging port lists.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-09-15 21:27:25 +08:00
etwenandClaude Opus 5 a5b7c4757b feat(boot): Name the log and the banner after the kind of boot
EN_Cold_Warm_boot = 1 opens Logs\Blanton_Reboot_Cycle_<ts>.log and prints
a "Reboot Cycle : N" banner each round; = 2 opens
Blanton_Power_Cycle_<ts>.log and prints "Power Cycle : N". Three sites
branch on the flag now: the log name, the banner, and the boot itself.

Two things follow from it.

With EN_Cold_Warm_boot = 0 there is no longer a logopen either, so the
run that already hangs at wait "Booting in blind mode" now also leaves no
log to show where it stopped. EN_log = 1 on its own no longer produces a
log from this macro.

And anything that parses these logs has to accept both banners. The cycle
banner is the record separator the check-point and per-segment timing
reports are built on, so a parser that only greps "Reboot Cycle :" finds
zero cycles in a power-cycle log and reports nothing without complaining.

The logopen timestamp flag stays a literal 1 rather than EN_timestamp the
way Script A does it. That is the right call here - the per-segment
timing analysis is derived from those timestamps and there is no other
way to recover them - but it does mean this macro ignores
EN_timestamp = 0.

Recorded under V1.2.2 in Script A's history; the header version is
unchanged because V1.2.2 has not been tagged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-09-09 10:15:39 +08:00
etwenandClaude Opus 5 f5694345fb feat(boot): Rename Script 5 to Boot Cycle, add a cold path and stop the log growing
The macro does warm and cold now, so "Reboot" was too narrow:
5_Blanton_Script_Reboot_Cycle.ttl -> 5_Blanton_Script_Boot_Cycle.ttl.

EN_Cold_Warm_boot in config.ttl picks which:

  0  neither - for the A/B/C flow, where this macro is not used
  1  reboot (warm), what the macro always did
  2  cold, by writing the CB FPGA:
     pcimem .../0000:05:00.0/resource0 0x480 w 0xD0000

Three things about that worth knowing before anyone runs it. Setting 0 and
then running this macro hangs it: nothing reboots, and the
wait "Booting in blind mode" underneath has no timeout, so it waits for a
boot that will never come. 0x480 is ICB_CTRL_REG in
docs/Blantons_FPGA_Registers.md and that table has no bit definitions for
it, nor is 0xD0000 written down anywhere in this repo. And the BDF
0000:05:00.0 is hard-coded, while blanton_fpga_pcimem.sh auto-detects the
CB FPGA precisely because it moves per unit - on a DUT where it is not at
05:00.0 this is a raw 32-bit write to an unknown register on some other
device. The cold path has not been run on hardware.

Separately, both stress logs are now removed before each run:

  rm -rf /host/hw-eval/jobs/qfx5252-stress-{ssd,usb}.log

qfx5252-stress-* appends to its --log file and the macro cats the whole
thing afterwards, so every cycle printed back everything the cycles
before it had written. On the 199-cycle log from 2026-09-04 that was 289
lines at cycle 1 and 36,555 at cycle 199; at ~6.2 ms a line, which is
what a 115200 baud console costs, it accounts for +212 s of the +214 s
the cycle grew over the run. The stress itself is fixed at --runtime 30,
and the other eleven gaps between check points drifted by under 1.5 s
each. The DUT was never getting slower - the test was reading its own
history aloud, once per cycle.

Script A's header goes to V1.2.2 and its Version History covers all of
the above.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
2026-09-09 09:14:09 +08:00
9 changed files with 1143 additions and 47 deletions
+19 -3
View File
@@ -136,6 +136,7 @@ Blanton_TTL_Script/
│ ├── 100G_PRBS.txt # 100G PRBS 原始指令(port_prbs_monitor.sh 的來源) │ ├── 100G_PRBS.txt # 100G PRBS 原始指令(port_prbs_monitor.sh 的來源)
│ ├── TR518.txt # 內建封包測試 tr 518 → blanton_tr518.sh │ ├── TR518.txt # 內建封包測試 tr 518 → blanton_tr518.sh
│ ├── BER_Test.txt # 全機 PRBS BER 指令 → blanton_ber.sh │ ├── BER_Test.txt # 全機 PRBS BER 指令 → blanton_ber.sh
│ ├── 100G_dsc_cycle.txt # 100G tuning + link 循環 + DSC → bcm_100g_dsc_cycle.sh
│ ├── Multiphase_{high3,low3,comboA,comboB,Save}.txt # → blanton_multiphase_margin.sh │ ├── Multiphase_{high3,low3,comboA,comboB,Save}.txt # → blanton_multiphase_margin.sh
│ ├── Multiphase_Netname # 38 條軌的 net 名稱(區塊順序對應) │ ├── Multiphase_Netname # 38 條軌的 net 名稱(區塊順序對應)
│ ├── 100G_Traffic.txt # ce0 VLAN 138 打流 → blanton_traffic_linespeed.sh │ ├── 100G_Traffic.txt # ce0 VLAN 138 打流 → blanton_traffic_linespeed.sh
@@ -201,6 +202,8 @@ Blanton_TTL_Script/
├── port_prbs_monitor.sh # 100G uplink PRBS 測試(bcmcmd phy diag) ├── port_prbs_monitor.sh # 100G uplink PRBS 測試(bcmcmd phy diag)
│ # ⚠️ V1.1.9 起 A/B/C 內註解掉的呼叫已整段刪除,只剩手動執行 │ # ⚠️ V1.1.9 起 A/B/C 內註解掉的呼叫已整段刪除,只剩手動執行
├── blanton_ber.sh # 全機 SerDes BER 掃描(phydiag prbs / prbsstat) ├── blanton_ber.sh # 全機 SerDes BER 掃描(phydiag prbs / prbsstat)
│ # V1.2.2 起含 port 268(100G ce0),summary 每 unit 分 200G/100G/TOTAL
├── bcm_100g_dsc_cycle.sh # 100G link down/up 循環 + 每輪 DSC 與 retimer TX/RX 讀回(未驗證)
├── bcm_dsc_scan.sh # 全機 SerDes DSC dump(phydiag <port> dsc,~445 port × 2 unit) ├── bcm_dsc_scan.sh # 全機 SerDes DSC dump(phydiag <port> dsc,~445 port × 2 unit)
│ # 由 EN_TX_FIR 控制,A 與 C 各掃一次;單次約 10 分鐘 │ # 由 EN_TX_FIR 控制,A 與 C 各掃一次;單次約 10 分鐘
├── bcm_dsc_parse.sh # DSC raw log → <prefix>_lane.csv / _pm.csv(純 sh + awk) ├── bcm_dsc_parse.sh # DSC raw log → <prefix>_lane.csv / _pm.csv(純 sh + awk)
@@ -374,7 +377,7 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc
| 模組 | 進入點 | 功能 | | 模組 | 進入點 | 功能 |
|------|--------|------| |------|--------|------|
| **Script A(Pre-test)** | `1_Blanton_Script_A.ttl` V1.2.1 | root 登入 → `date -s` 對時 → 清 job log → **收掉前一輪殘留**(`hw-test-session finish` → `bgctl reset --yes` → `bgctl stop --all` → `bgctl list` 確認清空)→ `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**(`wait_init.ttl`)→ DUT 清單(boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe)→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → **NFC 基線**(start → 30s → stop → report)→ 100G uplink 設定與狀態 → **BER 基線掃描**(依 `SWB_UNIT`,V1.2.1 新增)→ **一輪流量基線** → **SerDes DSC 基線**(`EN_TX_FIR`)→ `show uptime` | | **Script A(Pre-test)** | `1_Blanton_Script_A.ttl` V1.2.2 | root 登入 → `date -s` 對時 → 清 job log → **收掉前一輪殘留**(`hw-test-session finish` → `bgctl reset --yes` → `bgctl stop --all` → `bgctl list` 確認清空)→ `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**(`wait_init.ttl`)→ DUT 清單(boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe)→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → **NFC 基線**(start → 30s → stop → report)→ 100G uplink 設定與狀態 → **BER 基線掃描**(依 `SWB_UNIT`,V1.2.1 新增)→ **一輪流量基線** → **SerDes DSC 基線**(`EN_TX_FIR`)→ `show uptime` |
| **Script B(Stress)** | `2_Blanton_Script_B.ttl` | **V1.1.8 起 `mlucas-avx2 -cpu 0:15` 由 `systemd-run --unit=mlucas-amm`(`Type=exec`、`Restart=always`)啟動,不再是 shell 的 `&` 背景 job** + `bgctl run` 的 `memtester` / SSD / USB 壓力(v1.1.7 起 SSD/USB 的 `--runtime`/`--passes` 讀 `config.ttl` 的 `SSD_RUNTIME`/`SSD_COUNT`、`USB_RUNTIME`/`USB_COUNT`;BMC memtester 不帶次數 = 跑到被停) + BMC DDR(`bmc-manager run memtester`)→ BMC USB net test(探測 cdc_ncm 介面,`systemd-run` 掛 `BMC_USB_RUNTIME` 秒的 ping,V1.1.8 前寫死 4 小時)→ 啟動 mgmt ping 監控 → **啟動 NFC 輪詢** → `jobs` / `systemctl is-active mlucas-amm` / `bgctl list` → traffic(依 `SWB_UNIT`)→ `while 1` 每輪 PMON + `bgctl list`/`jobs`/`systemctl is-active mlucas-amm` + `sleep 60`(**V1.1.4 起 margin 不在迴圈內**;V1.2.0 起等待走 DUT 端 `sleep` 而非 macro 端 `pause`)| | **Script B(Stress)** | `2_Blanton_Script_B.ttl` | **V1.1.8 起 `mlucas-avx2 -cpu 0:15` 由 `systemd-run --unit=mlucas-amm`(`Type=exec`、`Restart=always`)啟動,不再是 shell 的 `&` 背景 job** + `bgctl run` 的 `memtester` / SSD / USB 壓力(v1.1.7 起 SSD/USB 的 `--runtime`/`--passes` 讀 `config.ttl` 的 `SSD_RUNTIME`/`SSD_COUNT`、`USB_RUNTIME`/`USB_COUNT`;BMC memtester 不帶次數 = 跑到被停) + BMC DDR(`bmc-manager run memtester`)→ BMC USB net test(探測 cdc_ncm 介面,`systemd-run` 掛 `BMC_USB_RUNTIME` 秒的 ping,V1.1.8 前寫死 4 小時)→ 啟動 mgmt ping 監控 → **啟動 NFC 輪詢** → `jobs` / `systemctl is-active mlucas-amm` / `bgctl list` → traffic(依 `SWB_UNIT`)→ `while 1` 每輪 PMON + `bgctl list`/`jobs`/`systemctl is-active mlucas-amm` + `sleep 60`(**V1.1.4 起 margin 不在迴圈內**;V1.2.0 起等待走 DUT 端 `sleep` 而非 macro 端 `pause`)|
| **Script C(Post-test)** | `3_Blanton_Script_C.ttl` | `kill $(jobs -p)` + **`systemctl stop mlucas-amm`**(systemd unit 不是 shell job,`jobs -p` 抓不到)+ `bgctl stop --all` / `reset` + 停 BMC USB unit、mgmt ping 與 **NFC 輪詢** → PMON + dmesg → 7 組 AER + `ras-mc-ctl` → BMC USB `journalctl` **寫進 `log/bmc_usb_net.log`**(不再印在 console) → `ip -s link show eth0/eth1` → `cat` mlucas log(並 `cp` 進 job 目錄一起帶走)→ **NFC `report`** → traffic `stop` + `report` → **SerDes DSC 收尾**(`EN_TX_FIR`)→ **BER 掃描**(依 `SWB_UNIT`)→ NVMe 健康 → `show reboot-cause` → **margin 掃描**(`EN_Margin`,V1.1.4 由 Script B 移來)→ `uptime` → **job log 複製到 USB(帶時戳)** → `exit` + wait『Script done』→ `hw-test-session finish` | | **Script C(Post-test)** | `3_Blanton_Script_C.ttl` | `kill $(jobs -p)` + **`systemctl stop mlucas-amm`**(systemd unit 不是 shell job,`jobs -p` 抓不到)+ `bgctl stop --all` / `reset` + 停 BMC USB unit、mgmt ping 與 **NFC 輪詢** → PMON + dmesg → 7 組 AER + `ras-mc-ctl` → BMC USB `journalctl` **寫進 `log/bmc_usb_net.log`**(不再印在 console) → `ip -s link show eth0/eth1` → `cat` mlucas log(並 `cp` 進 job 目錄一起帶走)→ **NFC `report`** → traffic `stop` + `report` → **SerDes DSC 收尾**(`EN_TX_FIR`)→ **BER 掃描**(依 `SWB_UNIT`)→ NVMe 健康 → `show reboot-cause` → **margin 掃描**(`EN_Margin`,V1.1.4 由 Script B 移來)→ `uptime` → **job log 複製到 USB(帶時戳)** → `exit` + wait『Script done』→ `hw-test-session finish` |
| **Script 4(Thermal / Safety)** | `4_Blanton_Script_thermal_safety.ttl` | **獨立腳本,不接 A→B→C。** Script A 式的環境建立(登入 / session / 風扇 / `wait_init` / DUT 清單 / BMC / PCIe 清除 / `pcie_bus` / PMON / dmesg)→ **滿載**(`mlucas-avx2 -cpu 0:15`,V1.1.8 起與 Script B 同樣掛成 `mlucas-amm` systemd unit + `blanton_tr518.sh start`)→ `while 1`:PMON + `TH6_SWB0/1_power_readback.sh` + `bgctl list` / `jobs` / `systemctl is-active mlucas-amm`,`pause 60`。**沒有收尾段** —— 中斷後 DUT 仍在 `port cd lb=mac` / `l2 learn off` / `test mode nr=yes`,且 CPU 仍滿載,要手動 `blanton_tr518.sh stop` + `systemctl stop mlucas-amm` | | **Script 4(Thermal / Safety)** | `4_Blanton_Script_thermal_safety.ttl` | **獨立腳本,不接 A→B→C。** Script A 式的環境建立(登入 / session / 風扇 / `wait_init` / DUT 清單 / BMC / PCIe 清除 / `pcie_bus` / PMON / dmesg)→ **滿載**(`mlucas-avx2 -cpu 0:15`,V1.1.8 起與 Script B 同樣掛成 `mlucas-amm` systemd unit + `blanton_tr518.sh start`)→ `while 1`:PMON + `TH6_SWB0/1_power_readback.sh` + `bgctl list` / `jobs` / `systemctl is-active mlucas-amm`,`pause 60`。**沒有收尾段** —— 中斷後 DUT 仍在 `port cd lb=mac` / `l2 learn off` / `test mode nr=yes`,且 CPU 仍滿載,要手動 `blanton_tr518.sh stop` + `systemctl stop mlucas-amm` |
@@ -393,7 +396,8 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc
| **BMC DDR 壓力** | `bmc_monitor_ddr.sh` | ⚠️ **已停用** —— V1.1.9 起 B/C 內的註解區塊已刪除,改由 `bgctl run bmc-manager run memtester` 取代。檔案保留未刪 | | **BMC DDR 壓力** | `bmc_monitor_ddr.sh` | ⚠️ **已停用** —— V1.1.9 起 B/C 內的註解區塊已刪除,改由 `bgctl run bmc-manager run memtester` 取代。檔案保留未刪 |
| **SerDes DSC 擷取** | `bcm_dsc_scan.sh [-u 0\|1] [-f <name>] [-F] [-n] [-C]` + `bcm_dsc_parse.sh -i <log> -o <prefix>` | `bcmcmd -n <unit> -c "dsh -c 'phydiag <port> dsc'"` 掃全清單(1 個 100G + 445 個 200G port)× 2 unit,raw log 寫進 `<script_dir>/log` 再 `cp` 到 `/host/hw-eval/jobs`。`-f` 支援 `%u` / `%t`;沒帶 `%u` 時自動插 `_u0`/`_u1`。成功與否**不看 `bcmcmd` 的 exit code**,改比對 `SERDES DISPLAY DIAG DATA END` 數量 + 實際迭代筆數,結尾印 `### DSC_SCAN_DONE rc=<n>`。parse 端輸出 `_lane.csv`(每 lane:TXEQ/RXFFE/DFE/EYE/SNR)與 `_pm.csv`(每 pm_id:溫度/PLL/lane 數/統計)。**由 `EN_TX_FIR` 控制,預設關** | | **SerDes DSC 擷取** | `bcm_dsc_scan.sh [-u 0\|1] [-f <name>] [-F] [-n] [-C]` + `bcm_dsc_parse.sh -i <log> -o <prefix>` | `bcmcmd -n <unit> -c "dsh -c 'phydiag <port> dsc'"` 掃全清單(1 個 100G + 445 個 200G port)× 2 unit,raw log 寫進 `<script_dir>/log` 再 `cp` 到 `/host/hw-eval/jobs`。`-f` 支援 `%u` / `%t`;沒帶 `%u` 時自動插 `_u0`/`_u1`。成功與否**不看 `bcmcmd` 的 exit code**,改比對 `SERDES DISPLAY DIAG DATA END` 數量 + 實際迭代筆數,結尾印 `### DSC_SCAN_DONE rc=<n>`。parse 端輸出 `_lane.csv`(每 lane:TXEQ/RXFFE/DFE/EYE/SNR)與 `_pm.csv`(每 pm_id:溫度/PLL/lane 數/統計)。**由 `EN_TX_FIR` 控制,預設關** |
| **100G PRBS** | `port_prbs_monitor.sh {start [both\|a\|b]\|stop\|status\|report\|clear}` | 兩個 uplink 各一個 worker,可同時或單獨跑。setup(`lpmode off` → `prbs set` → `prbsstat STArt`)→ 週期 `prbs get`(判定)+ `prbsstat Ber`(只記錄)→ teardown(`STOp` + `prbs clear`)。輸出含 `PRBS OK!` 才算 PASS。**⚠️ V1.1.9 起 A/B/C 內的註解呼叫已刪除,只能手動跑** | | **100G PRBS** | `port_prbs_monitor.sh {start [both\|a\|b]\|stop\|status\|report\|clear}` | 兩個 uplink 各一個 worker,可同時或單獨跑。setup(`lpmode off` → `prbs set` → `prbsstat STArt`)→ 週期 `prbs get`(判定)+ `prbsstat Ber`(只記錄)→ teardown(`STOp` + `prbs clear`)。輸出含 `PRBS OK!` 才算 PASS。**⚠️ V1.1.9 起 A/B/C 內的註解呼叫已刪除,只能手動跑** |
| **SerDes BER 掃描** | `blanton_ber.sh {init\|clear\|start\|stop\|report} [-u 0\|1\|all]` | 全機每個對接 port 的 PRBS BER:`prbs set p=3` + `prbs get`(確認每條 lane 都鎖上)→ `prbsstat STArt Interval=30` →(等一個 interval)→ `prbsstat Ber` → `STOp` → `prbs clear`。`report` 一列一條 lane,**BER < 1e-6 才 PASS**,最後跨 unit 的 summary 帶最差 lane。⚠️ PRBS 期間 link 顯示 DOWN、不過流量。V1.2.1 起 Script A 與 Script C 各跑一次(A 在流量之前、C 在流量報表之後),夾住 soak | | **SerDes BER 掃描** | `blanton_ber.sh {init\|clear\|start\|stop\|report} [-u 0\|1\|all]` | 全機每個對接 port 的 PRBS BER:`prbs set p=3` + `prbs get`(確認每條 lane 都鎖上)→ `prbsstat STArt Interval=30` →(等一個 interval)→ `prbsstat Ber` → `STOp` → `prbs clear`。`report` 一列一條 lane,**BER < 1e-6 才 PASS**,最後跨 unit 的 summary 帶最差 lane。⚠️ PRBS 期間 link 顯示 DOWN、不過流量。V1.2.1 起 Script A 與 Script C 各跑一次(A 在流量之前、C 在流量報表之後),夾住 soak。**V1.2.2(腳本 v1.1.0)起清單納入 port 268(100G uplink ce0)**,summary 每個 unit 拆成 200G ports / 100G p268 / TOTAL 三列;某組零 lane 算 FAIL |
| **100G link 循環 + DSC** | `bcm_100g_dsc_cycle.sh <cycles> [-d] [-T] [-P] [-n]` | tuning(`docker pause gbsyncd*` → `serdes-tx-set sys 0..7` → 每 lane `tx_taps set`)一次,之後每輪:`port 268 en=0` → 輪詢 `bcmcmd ps 268` 等 `!ena` → `en=1` → 等 `up` → 兩個 unit 的 `phydiag 268 dsc` + `serdes-tx-get`/`serdes-rx-get sys 0..7`。log 一次執行一份(`~/dsc_log/`),每輪一段 `### TEST CYCLE n/N`,結尾 `### DSC_CYCLE_DONE rc=<n>`。**全 ASCII 輸出**(DUT console 會把 UTF-8 弄成亂碼)。⚠️ 尚未上機驗證 |
| **TR518 負載** | `blanton_tr518.sh {start\|stop} [-u 0\|1\|all]` | `port cd lb=mac` → `l2 learn off` → `test mode nr=yes` → `tr 518 ... testphase=1`,然後 settle `TR_SETTLE_SEC`(120s)再讓人量功耗。`stop` 只送 `port cd lb=none`(照 `TR518.txt`),L2 學習與 test mode **不會還原**,`TR_RESTORE_*=1` 才會 | | **TR518 負載** | `blanton_tr518.sh {start\|stop} [-u 0\|1\|all]` | `port cd lb=mac` → `l2 learn off` → `test mode nr=yes` → `tr 518 ... testphase=1`,然後 settle `TR_SETTLE_SEC`(120s)再讓人量功耗。`stop` 只送 `port cd lb=none`(照 `TR518.txt`),L2 學習與 test mode **不會還原**,`TR_RESTORE_*=1` 才會 |
| **PDB / CPB brick** | `pwr_brick.sh show [-r]` | 5 顆 brick 的 Vin / Iin / Vout / Iout / Temp(SWB0 `0x62`/`0x63`、SWB1 `0x60`/`0x61`、CPB `0x26`)。四顆 ADPM12200 **每個量各自一個 PMBus PAGE**(Vin 9、Iin 10、Vout 2、Iout 14、Temp 18),讀之前一定要先寫 PAGE —— `blanton_pwr_data.sh` 完全沒寫,所以整張 PDB 表都是 NA。換算用手冊 DIRECT 公式 `X=(1/m)(Y×10⁻ᴿ−b)`。每個值重讀到通過三道檢查為止(非 `0xFFFF`、非 `0x0000`、落在合理範圍)—— 這條匯流排**只壞高位元組**,`11.98 V` 會變成 `32.00 V`,全一過濾擋不住 | | **PDB / CPB brick** | `pwr_brick.sh show [-r]` | 5 顆 brick 的 Vin / Iin / Vout / Iout / Temp(SWB0 `0x62`/`0x63`、SWB1 `0x60`/`0x61`、CPB `0x26`)。四顆 ADPM12200 **每個量各自一個 PMBus PAGE**(Vin 9、Iin 10、Vout 2、Iout 14、Temp 18),讀之前一定要先寫 PAGE —— `blanton_pwr_data.sh` 完全沒寫,所以整張 PDB 表都是 NA。換算用手冊 DIRECT 公式 `X=(1/m)(Y×10⁻ᴿ−b)`。每個值重讀到通過三道檢查為止(非 `0xFFFF`、非 `0x0000`、落在合理範圍)—— 這條匯流排**只壞高位元組**,`11.98 V` 會變成 `32.00 V`,全一過濾擋不住 |
| **SWB 軌功耗** | `TH6_SWB{0,1}_power_readback.sh` | 讀 `show platform voltage/current` **各一次**,19 條軌 V×I 與總功耗。舊版每軌各讀一次(38 次呼叫、每軌不同時刻),總和是不同瞬間的加總 | | **SWB 軌功耗** | `TH6_SWB{0,1}_power_readback.sh` | 讀 `show platform voltage/current` **各一次**,19 條軌 V×I 與總功耗。舊版每軌各讀一次(38 次呼叫、每軌不同時刻),總和是不同瞬間的加總 |
@@ -804,7 +808,7 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值,
- [ ] `CHANGELOG.md`:延續三支腳本頂端的 Version History,集中成一份 - [ ] `CHANGELOG.md`:延續三支腳本頂端的 Version History,集中成一份
- [ ] 三支腳本的 header 版號與 `CHANGELOG.md` 對齊,`config.ttl` 加 `script_version` 並寫進 log 開頭 - [ ] 三支腳本的 header 版號與 `CHANGELOG.md` 對齊,`config.ttl` 加 `script_version` 並寫進 log 開頭
- [ ] `docs/TTL_Script_Blanton_Status_*.xlsx` 更新為全 OK,或把剩餘項目移進 Future Extensions - [ ] `docs/TTL_Script_Blanton_Status_*.xlsx` 更新為全 OK,或把剩餘項目移進 Future Extensions
- [x] 打 git tag,push `nas` + `gitea`(已至 `V1.2.1`;`publish/` 依 Script A 檔頭版號打包) - [x] 打 git tag,push `nas` + `gitea`(已至 `V1.2.2`;`publish/` 依 Script A 檔頭版號打包)
**驗收條件:** 從 zip 解壓到一台乾淨的測試 NB,照 README 操作,不看原始碼就能完成一輪 A→B→C **驗收條件:** 從 zip 解壓到一台乾淨的測試 NB,照 README 操作,不看原始碼就能完成一輪 A→B→C
並產出 summary;`publish/` 內容與 git tag 一致。 並產出 summary;`publish/` 內容與 git tag 一致。
@@ -851,6 +855,10 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值,
開啟後 Script A 與 Script C 各掃一次、整輪多約 20 分鐘;只要單邊資料可以在腳本裡改成 `-u 0`。 開啟後 Script A 與 Script C 各掃一次、整輪多約 20 分鐘;只要單邊資料可以在腳本裡改成 `-u 0`。
掃描本身是唯讀(`phydiag ... dsc` 不改 SerDes 設定),但**它與 `blanton_ber.sh` / 掃描本身是唯讀(`phydiag ... dsc` 不改 SerDes 設定),但**它與 `blanton_ber.sh` /
`blanton_traffic_linespeed` / `blanton_tr518.sh` 一樣都在動同一批 port,不要同時跑**。 `blanton_traffic_linespeed` / `blanton_tr518.sh` 一樣都在動同一批 port,不要同時跑**。
- **`bcm_100g_dsc_cycle.sh` 尚未上機驗證**:`port en=0/1` 之後 link 多久會動、以及
`docker pause gbsyncd*` 期間的影響,都要上機才知道。等待是讀 SDK(`bcmcmd ps <port>`)而不是
`show interfaces status`,所以理論上不受 gbsyncd 暫停影響,但沒實測過。
結束時(含 Ctrl-C)會自動 `docker unpause`,`RESUME_GBSYNCD=0` 可關掉。
- **Script 5(Reboot Cycle)尚未上機驗證**:`5_Blanton_Script_Reboot_Cycle.ttl` 隨 V1.1.7 交付, - **Script 5(Reboot Cycle)尚未上機驗證**:`5_Blanton_Script_Reboot_Cycle.ttl` 隨 V1.1.7 交付,
但還沒在真 DUT 上跑過一輪完整循環。兩個要現場確認的點:`wait "Booting in blind mode"` 是否 但還沒在真 DUT 上跑過一輪完整循環。兩個要現場確認的點:`wait "Booting in blind mode"` 是否
真的在每一版開機訊息裡出現(沒出現就會永遠等下去,而且沒有 `timeout` 保護);以及每輪重跑 真的在每一版開機訊息裡出現(沒出現就會永遠等下去,而且沒有 `timeout` 保護);以及每輪重跑
@@ -914,6 +922,14 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值,
- **margin 在 soak 期間不再取樣**(V1.1.4):只剩 Script A 的起始值與 Script C 的結束值。 - **margin 在 soak 期間不再取樣**(V1.1.4):只剩 Script A 的起始值與 Script C 的結束值。
漂移發生在中間就看不到了。若要恢復,比較好的做法是在 B 的迴圈內每 N 輪才掃一次, 漂移發生在中間就看不到了。若要恢復,比較好的做法是在 B 的迴圈內每 N 輪才掃一次,
而不是每輪都掃。 而不是每輪都掃。
- ✅ **`blanton_ber.sh` 不再把 lane 的量測結果當成 shell 錯誤**(v1.1.1,2026-09-15):
`prbs get` 每條 lane 一行,沒鎖上的會印 `PRBS Error!`,而錯誤樣式是拿
`grep -i 'ERROR|Invalid|...'` 掃整份輸出的 —— 於是正常的 init 會報「the BCM shell rejected」,
而且提早 return,**把「哪幾條 lane 沒鎖」這個真正有用的資訊吞掉**。現在 per-lane 行排除在
掃描外、改成大小寫敏感、真正的拒絕會把該行印出來。
⚠️ **這類「完整性檢查自己看不到失敗」的錯在這個 repo 已經出現三次**
(`bcm_dsc_scan.sh` v1.3.0 數 marker、`blanton_ber.sh` v1.1.1、以及 power cycle 報告的
ping 解析把 100% 丟包當成「沒資料」)。寫檢查時要問:**失敗長什麼樣,這段程式碼看得到嗎?**
- **`blanton_ber.sh` 的 `sleep 30` 與 `BER_INTERVAL` 是兩處要同步改的數字**:Script A(V1.2.1 起) - **`blanton_ber.sh` 的 `sleep 30` 與 `BER_INTERVAL` 是兩處要同步改的數字**:Script A(V1.2.1 起)
與 Script C 都寫死 `sendln "sleep 30"`,腳本預設 `BER_INTERVAL=30`。改一邊沒改另一邊,report 會 與 Script C 都寫死 `sendln "sleep 30"`,腳本預設 `BER_INTERVAL=30`。改一邊沒改另一邊,report 會
讀到還沒累積完一個 interval 的數字,而且不會有任何警告。現在有**三處**要一起改。 讀到還沒累積完一個 interval 的數字,而且不會有任何警告。現在有**三處**要一起改。
+20 -1
View File
@@ -9,7 +9,7 @@ DUT 端的取數能力由一組 **bash 工具庫** 提供(CB FPGA `pcimem` →
完整架構見 [ARCHITECTURE.md](ARCHITECTURE.md)。 完整架構見 [ARCHITECTURE.md](ARCHITECTURE.md)。
**目前狀態(2026-09-08)**:Script A 為 **V1.2.1**(A 集中記錄所有 TTL 的變更)。涵蓋 **目前狀態(2026-09-15)**:Script A 為 **V1.2.2**(A 集中記錄所有 TTL 的變更)。涵蓋
PCIe AER / DDR EDAC / PMON+TPM / 電壓 margin(9 顆 LTC2980)/ SWB loopback 線速流量 / PCIe AER / DDR EDAC / PMON+TPM / 電壓 margin(9 顆 LTC2980)/ SWB loopback 線速流量 /
10G+1G 管理網路 ping / BMC DDR 與 USB / NVMe 健康,壓力層走平台的 `bgctl`。 10G+1G 管理網路 ping / BMC DDR 與 USB / NVMe 健康,壓力層走平台的 `bgctl`。
switch unit 數量由 `config.ttl` 的 `SWB_UNIT0` / `SWB_UNIT1` 宣告,風扇轉速由 `FAN_SPEED` 指定 switch unit 數量由 `config.ttl` 的 `SWB_UNIT0` / `SWB_UNIT1` 宣告,風扇轉速由 `FAN_SPEED` 指定
@@ -25,6 +25,7 @@ Script B 的 soak 迴圈每輪印 `bgctl list` + `jobs` + `systemctl is-active m
`mlucas-amm` transient unit(`Restart=always`),不再是 shell 的 `&` 背景 job —— 掉線也不會停, `mlucas-amm` transient unit(`Restart=always`),不再是 shell 的 `&` 背景 job —— 掉線也不會停,
Script C 用 `systemctl stop mlucas-amm` 收掉並把 log 複製到隨身碟。 Script C 用 `systemctl stop mlucas-amm` 收掉並把 log 複製到隨身碟。
V1.1.0 新增全機 SerDes BER 掃描(`blanton_ber.sh`,依 `SWB_UNIT` 分支);**V1.2.1 起 Script A 也接了一次**, V1.1.0 新增全機 SerDes BER 掃描(`blanton_ber.sh`,依 `SWB_UNIT` 分支);**V1.2.1 起 Script A 也接了一次**,
**V1.2.2 起清單納入 port 268(100G uplink,ce0)**,summary 每個 unit 拆成「200G ports / 100G p268 / TOTAL」三列;
在第一輪流量之前取基線,與 Script C 流量報表之後的收尾掃描一起夾住 soak(同 DSC 的做法)。另有 在第一輪流量之前取基線,與 Script C 流量報表之後的收尾掃描一起夾住 soak(同 DSC 的做法)。另有
TR518 負載(`blanton_tr518.sh`,**TTL 尚未接**)、SWB 軌功耗(`TH6_SWB{0,1}_power_readback.sh`), TR518 負載(`blanton_tr518.sh`,**TTL 尚未接**)、SWB 軌功耗(`TH6_SWB{0,1}_power_readback.sh`),
線速流量也納入 100G `ce0`(VLAN 138,靠 ingress mirror 維持循環)。 線速流量也納入 100G `ce0`(VLAN 138,靠 ingress mirror 維持循環)。
@@ -152,6 +153,8 @@ sleep 30 # 至少等一個 interval 再 rep
./Blanton_Script/blanton_ber.sh report # 每 lane 一列,BER < 1e-6 才 PASS,最後 summary ./Blanton_Script/blanton_ber.sh report # 每 lane 一列,BER < 1e-6 才 PASS,最後 summary
./Blanton_Script/blanton_ber.sh stop # prbsstat STOp ./Blanton_Script/blanton_ber.sh stop # prbsstat STOp
./Blanton_Script/blanton_ber.sh clear # prbs clear(會解除 PRBS,要重測得再 init) ./Blanton_Script/blanton_ber.sh clear # prbs clear(會解除 PRBS,要重測得再 init)
# summary 每個 unit 三列:200G ports / 100G p268 / TOTAL(兩個 unit 各有自己的 ce0(268))
# ⚠️ 某一組「一條 lane 都沒回報」算 FAIL 不算 PASS —— 待測的 port 沉默不是通過
# -u 0 / -u 1 單邊;-c 1e-9 換門檻;-F 只列失敗;-f <log> 離線重解 # -u 0 / -u 1 單邊;-c 1e-9 換門檻;-F 只列失敗;-f <log> 離線重解
# ⚠️ 不能跟 blanton_traffic_linespeed / blanton_tr518 同時跑(都會接管同一批 port) # ⚠️ 不能跟 blanton_traffic_linespeed / blanton_tr518 同時跑(都會接管同一批 port)
@@ -171,6 +174,17 @@ sh ~/Blanton_Script/bcm_dsc_parse.sh -i blanton_dsc_ScriptA_u0.log -o u0
# 純 sh + awk(mawk 相容),欄位靠括號群組順序抓,SDK 改排版不會壞; # 純 sh + awk(mawk 相容),欄位靠括號群組順序抓,SDK 改排版不會壞;
# 群組數 != 9 的列標成 UNPARSED 印到 stderr,不會安靜解錯 # 群組數 != 9 的列標成 UNPARSED 印到 stderr,不會安靜解錯
# ── 100G link down/up 循環 + DSC(TTL 尚未接)────────
./Blanton_Script/bcm_100g_dsc_cycle.sh # 印說明
./Blanton_Script/bcm_100g_dsc_cycle.sh 200 # 跑 200 輪(-c 200 亦可)
./Blanton_Script/bcm_100g_dsc_cycle.sh 200 -d # 每行指令以灰色 [CMD] 印出來
./Blanton_Script/bcm_100g_dsc_cycle.sh 20 -T # 跳過 tuning
./Blanton_Script/bcm_100g_dsc_cycle.sh 1 -n # dry-run,不碰硬體也不 sleep
# log: ~/dsc_log/100G_dsc_cycle_test_<ts>.log,每輪一段 '### TEST CYCLE n/N'
# 每輪收:兩個 unit 的 phydiag dsc + serdes-tx-get / serdes-rx-get sys 0..7
# 等 link 是輪詢 'bcmcmd ps 268'(每 -p 秒,上限 -w 秒),不是 show interfaces status
# ⚠️ 尚未上機驗證;與 blanton_ber / blanton_traffic_linespeed / blanton_tr518 互斥
# ── TR518 負載 + 量功耗(TTL 尚未接)───────────────── # ── TR518 負載 + 量功耗(TTL 尚未接)─────────────────
./Blanton_Script/blanton_tr518.sh start # 兩 unit,跑完 settle 120s ./Blanton_Script/blanton_tr518.sh start # 兩 unit,跑完 settle 120s
./Blanton_Script/TH6_SWB0_power_readback.sh # 19 條軌 V×I + TOTAL POWER ./Blanton_Script/TH6_SWB0_power_readback.sh # 19 條軌 V×I + TOTAL POWER
@@ -354,6 +368,11 @@ Tera Term 連 COM port (115200-8-N-1)
- ⚠️ **兩張管理網卡同網段會 ARP flux**:`mgmt_ping_monitor.sh` 讓 eth0/eth1 同時 up,同一個 /24 下 - ⚠️ **兩張管理網卡同網段會 ARP flux**:`mgmt_ping_monitor.sh` 讓 eth0/eth1 同時 up,同一個 /24 下
對端 ARP 可能由任一張卡回應,回包落在沒送封包的那張。`ARP_STRICT=1` 擋掉;判讀時看 `RESULT` 行的 對端 ARP 可能由任一張卡回應,回包落在沒送封包的那張。`ARP_STRICT=1` 擋掉;判讀時看 `RESULT` 行的
`nic_tx`/`nic_rx`(該卡自己的 packet delta)—— **ping 成功但 `nic_tx` 接近 0 = 封包從另一張卡出去的** `nic_tx`/`nic_rx`(該卡自己的 packet delta)—— **ping 成功但 `nic_tx` 接近 0 = 封包從另一張卡出去的**
- ⚠️ **`prbs get` 印的每一行是資料,不是 shell 的錯誤訊息**:沒鎖上的 lane 會印 `PRBS Error!`,
而 `blanton_ber.sh` 從 v1.0.0 就用 `grep -i 'ERROR|Invalid|...'` 掃**整份輸出**判斷「指令被拒絕」——
於是**跑得好好的 init 會報 `the BCM shell rejected 'phydiag ... prbs get'`**,而且因為提早 return,
那段「數 PRBS OK / 總數」的檢查根本沒跑,**你看到一個假錯誤,卻看不到哪幾條 lane 真的沒鎖**。
v1.1.1 已把 per-lane 行排除在錯誤掃描外、改成大小寫敏感,並把真正的拒絕訊息印出來
- ⚠️ **BER 的 `clear` 才是把 lane 放回正常運作的動作**:V1.2.1 起 Script A 在第一輪流量**之前**跑 BER, - ⚠️ **BER 的 `clear` 才是把 lane 放回正常運作的動作**:V1.2.1 起 Script A 在第一輪流量**之前**跑 BER,
而 `blanton_ber.sh clear` 與緊接著的 `blanton_traffic_linespeed ps` 之間**沒有任何 settle**。 而 `blanton_ber.sh clear` 與緊接著的 `blanton_traffic_linespeed ps` 之間**沒有任何 settle**。
Script C 是流量報表之後才跑 BER,沒有這個順序問題。第一次上機時比對兩邊的 traffic report —— Script C 是流量報表之後才跑 BER,沒有這個順序問題。第一次上機時比對兩邊的 traffic report ——
+63
View File
@@ -0,0 +1,63 @@
# v1.2.2 — The 100G uplink joins the BER sweep, and a boot cycle that can pull the power
## ✨ New features
**The BER sweep now measures port 268, the 100G uplink**
* The port list ran `256-259` straight to `270-308`, so `ce0(268)` — the 100G uplink — was never in it. It is now its own group alongside the 200G ports.
* The summary gives **each unit three rows**: `200G ports`, `100G p268`, and `TOTAL`. Unit 0 and unit 1 each have their own `ce0(268)`, and a marginal uplink lane averaged into 432 others is invisible; on its own row it is not.
* A group that reports no lane at all now counts as a **failure**, not a pass. Silence on the port under test is the easiest way to ship a green report that measured nothing.
**Script 5 is now a boot cycle, warm or cold**
* `5_Blanton_Script_Reboot_Cycle.ttl` → `5_Blanton_Script_Boot_Cycle.ttl`, with `EN_Cold_Warm_boot` in `config.ttl` choosing: `1` = `reboot` (warm), `2` = a cold cycle by writing the CB FPGA, `0` = neither.
* The log file and the per-round banner are named after the kind of boot: `Blanton_Reboot_Cycle_<ts>.log` / `Reboot Cycle : N`, or `Blanton_Power_Cycle_<ts>.log` / `Power Cycle : N`.
* Both stress logs are removed before each run. `qfx5252-stress-*` appends to its `--log` and the macro `cat`s the whole thing, so every cycle was printing back everything the earlier cycles had written — 289 lines at cycle 1 and 36,555 by cycle 199. At roughly 6.2 ms a line on a 115200 console that accounted for **+212 s of the +214 s the cycle grew** over a 199-cycle run. The DUT was never getting slower.
**`bcm_100g_dsc_cycle.sh` — 100G link down/up cycling with SerDes capture**
* Tuning once, then N cycles of `port 268 en=0` → wait → `en=1` → wait → `phydiag 268 dsc` on both units → retimer `serdes-tx-get` / `serdes-rx-get` for `sys 0..7`. One log per run under `~/dsc_log/`, split by `### TEST CYCLE n/N`.
* Waiting is a **poll of `bcmcmd ps 268`**, not `show interfaces status`. `port en=` changes the SDK, and the SONiC table only follows once gbsyncd has synced it — and this test pauses gbsyncd.
* `-d` echoes every command as a grey `[CMD]` line before running it; `-n` is a dry run that touches nothing; the cycle count can be a bare argument (`./bcm_100g_dsc_cycle.sh 200`).
## 🐛 Bug fixes
**`blanton_ber.sh init` reported a hard error on runs that were working**
* `[ERR ] unit 1: the BCM shell rejected 'phydiag ... prbs get'` — except the command was fine. `prbs get` prints one line per lane, a lane that has not locked says `PRBS Error!`, and the rejection check grepped the **whole** output case-insensitively for `ERROR|Invalid|...`.
* Worse than the false alarm: the function returned early, so the lock count that was meant to judge those very lanes never ran. The operator saw an error they could not act on, and did **not** see which lanes had failed to lock. That check has been there since v1.0.0.
* Per-lane result lines are now excluded, the match is case-sensitive, and a genuine rejection prints the offending line.
## 🔧 Changes
* **Script A takes a BER baseline before its first traffic run** (v1.2.1), so together with Script C's closing sweep the pair brackets the soak.
* **`EN_TX_FIR`** gates the SerDes DSC scans in Script A and Script C. Default `0`: a full scan is ~10 min and both scripts do one.
## 📦 Downloads
| File | Contents |
|---|---|
| `Script_ABC_Blanton_V1.2.2.zip` | The full Tera Term working directory, including `Blanton_Script/` to copy onto the DUT |
```bash
chmod +x ~/Blanton_Script/*.sh
grep -rlU $'\r' ~/Blanton_Script # expect no output
ls ~/hammer/tools/amd/mlucas-avx2 # must already be there
```
## ⚠️ Before you run
* **Replace `config.ttl` with the macros.** `EN_Cold_Warm_boot` is new, and Tera Term aborts on an undefined variable rather than treating it as 0.
* **Do not run the boot cycle with `EN_Cold_Warm_boot = 0`.** Nothing reboots, the `wait "Booting in blind mode"` underneath has no timeout, and since v1.2.2 there is no `logopen` either — so it hangs *and* leaves no log showing where. `0` is the value for the A → B → C flow, where that macro is not used.
* **The cold path writes a raw FPGA register.** `0x480` is `ICB_CTRL_REG`; the register map carries no bit definitions for it and `0xD0000` is not documented anywhere in this repo. The BDF `0000:05:00.0` is hard-coded while `blanton_fpga_pcimem.sh` auto-detects it because it moves per unit — **check the BDF on every board before running the cold path**.
* **Anything that parses these logs must accept both banners**, `Reboot Cycle : N` and `Power Cycle : N`. A parser that greps only the first finds zero cycles in a power-cycle log and reports nothing without complaining.
* **`bcm_100g_dsc_cycle.sh` has not been run on hardware.** It pauses `gbsyncd0/1` for tuning and unpauses them on exit, Ctrl-C included; `-P` leaves them alone.
* **`sleep 30` and `BER_INTERVAL` now live in three places** — Script A, Script C and `blanton_ber.sh`.
* **BER, the DSC cycle, `blanton_traffic_linespeed` and `blanton_tr518.sh` cannot overlap.** They all take the same ports.
* **Script 4 has no teardown**: finish it with `blanton_tr518.sh stop` **and** `systemctl stop mlucas-amm`.
* **`blanton_multiphase_margin.sh save` is permanent**, and must not be mixed with `margin.sh`.
* **`READ_VIN` and `READ_IIN` are still not in the ADPM12200 coefficient table.**
* **`FAN_SPEED` is 100.**
## 🔗 Links
* [ARCHITECTURE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/ARCHITECTURE.md) — test flow, data models, constraints
* [CLAUDE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/CLAUDE.md) — commands and the bench gotchas
**Full changelog:** [V1.2.1...V1.2.2](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/compare/V1.2.1...V1.2.2)
+132 -2
View File
@@ -1,7 +1,7 @@
; ============================================================================= ; =============================================================================
; Script A for Blanton ; Script A for Blanton
; Version : V1.2.1 ; Version : V1.2.2
; Date : 2026-09-08 ; Date : 2026-09-15
; Author : ETWen ; Author : ETWen
; ============================================================================= ; =============================================================================
; Version History: ; Version History:
@@ -562,6 +562,136 @@
; concerned: once here and once inside ScriptC. Neither ; concerned: once here and once inside ScriptC. Neither
; can overlap blanton_traffic_linespeed or ; can overlap blanton_traffic_linespeed or
; blanton_tr518.sh - they all take over the same ports ; blanton_tr518.sh - they all take over the same ports
; V1.2.2 2026-09-15
; 5_.._Boot_Cycle.ttl Renamed from 5_Blanton_Script_Reboot_Cycle.ttl
; It does warm and cold now, so 'Reboot' was too narrow.
; Same macro otherwise - git records it as a rename
; 5_.._Boot_Cycle.ttl Warm or cold boot, chosen by EN_Cold_Warm_boot
; 1 = 'reboot' (warm, what the macro always did). 2 = cold,
; by writing the CB FPGA: pcimem
; /sys/bus/pci/devices/0000:05:00.0/resource0 0x480 w
; 0xD0000. 0 = neither
; !! EN_Cold_Warm_boot = 0 hangs the macro. Neither branch
; runs, so nothing reboots, and the 'wait "Booting in
; blind mode"' underneath has no timeout - cycle 1
; finishes and the macro then waits for a boot that will
; never come. 0 is the value for the A/B/C flow, where
; this macro is not used at all; do not run ScriptBoot
; with it
; !! 0x480 is ICB_CTRL_REG in
; docs/Blantons_FPGA_Registers.md, and that table
; carries no bit definitions for it. 0xD0000 is not
; written down anywhere in this repo either. Whoever
; knows what those bits do should put it in the register
; doc before someone changes the value
; !! the BDF 0000:05:00.0 is hard-coded here, while
; blanton_fpga_pcimem.sh auto-detects the CB FPGA
; precisely because it moves per unit (see the V1.0.x
; notes and Key Constraints #4). On a DUT where the FPGA
; is not at 05:00.0 this becomes a raw 32-bit write to
; an unknown register on some other device. Verify the
; BDF on every unit before running the cold path
; !! NOT verified on hardware. The warm path has 199 cycles
; behind it; the cold path has none
; 5_.._Boot_Cycle.ttl Truncate the two stress logs before each run
; rm -rf /host/hw-eval/jobs/qfx5252-stress-ssd.log and the
; usb one, added ahead of each qfx5252-stress-* call
; qfx5252-stress-* APPENDS to its --log file and the macro
; cats the whole thing afterwards, so every cycle printed
; back everything the ones before it had written
; Measured on the 199 cycle log from 2026-09-04: cycle 1
; printed 289 lines, cycle 199 printed 36,555. At ~6.2 ms a
; line - which is what a 115200 baud console costs - that
; is +212 s of the +214 s the cycle grew over the run. The
; stress itself is fixed at --runtime 30, and the other
; eleven segments between check points drifted by less than
; 1.5 s each. The DUT was never getting slower; the test
; was reading its own history aloud once per cycle
; 5_.._Boot_Cycle.ttl Name the log file and the banner after the kind of boot
; EN_Cold_Warm_boot = 1 gives
; Logs\Blanton_Reboot_Cycle_<ts>.log and a 'Reboot Cycle :
; N' banner; = 2 gives Blanton_Power_Cycle_<ts>.log and
; 'Power Cycle : N'. Three sites now branch on the flag:
; the log name, the banner and the boot itself
; !! with 0 there is no longer a logopen either, so the run
; that already hangs at wait "Booting in blind mode" now
; also leaves no log to show where it stopped. EN_log =
; 1 no longer guarantees a log in this macro -
; EN_Cold_Warm_boot has to be 1 or 2 as well
; !! anything that parses these logs has to accept BOTH
; banners. The cycle banner is the record separator the
; check-point and per-segment timing reports are built
; on, so a parser that only greps 'Reboot Cycle :' finds
; zero cycles in a power-cycle log and silently reports
; nothing at all
; The logopen timestamp flag is still a literal 1 here
; rather than EN_timestamp the way ScriptA does it. Leaving
; it on is the right call - the per-segment timing analysis
; is derived from those timestamps and there is nothing
; else to recover them from - but it does mean this macro
; ignores EN_timestamp = 0
; config.ttl Add EN_Cold_Warm_boot
; 0 = normal test (A/B/C), 1 = reboot cycle, 2 = power
; cycle. Default 0 so that opening the shared config cannot
; reboot anyone's DUT by accident
; Blanton_Script/blanton_ber.sh v1.1.0 Add port 268, the 100G uplink
; The port list jumps 256-259 -> 270-308, so 268 (ce0) was
; never measured. It is now its own group: BER_PORTS_200G
; and BER_PORTS_100G, joined for the phydiag command but
; reported apart
; The summary gives each unit three rows - 200G ports /
; 100G p268 / TOTAL - because unit 0 and unit 1 each have
; their own ce0(268). A bad uplink lane averaged into 432
; others is invisible; on its own row it is not
; TOTAL is not a hand-added sum: every lane is folded into
; its own group and into ALL separately, so the parts
; cannot drift from the whole
; !! a group that reports no lane at all is a FAILURE, not
; a pass. Silence on the port under test is the easiest
; way to ship a green report that measured nothing
; Blanton_Script/blanton_ber.sh v1.1.1 Stop reading lane results as shell errors
; init kept answering "the BCM shell rejected 'phydiag ...
; prbs get'" on runs that were working. The check grepped
; the WHOLE output case-insensitively for
; ERROR|Invalid|..., and 'prbs get' prints one line per
; lane - a lane that had not locked says PRBS Error!, which
; matched
; So a good run reported a hard error, and worse, the
; function returned before the lock-count check that was
; supposed to judge those very lanes. The operator saw a
; failure and did NOT see which lanes were unlocked
; Per-lane lines are now excluded from the check, the match
; is case-sensitive, and a genuine rejection prints the
; offending line so it can be read
; !! this was in the file from v1.0.0. Any earlier init
; that printed that error was probably fine, and any
; earlier init that passed said nothing about lanes that
; had failed to lock
; Blanton_Script/bcm_100g_dsc_cycle.sh Created v1.2.0 - 100G link down/up cycle with DSC capture
; Tuning once (gbsyncd pause, serdes-tx-set sys 0..7,
; tx_taps per lane), then N cycles of: port en=0, wait,
; port en=1, wait, phydiag dsc on both units, serdes-tx-get
; and serdes-rx-get sys 0..7. One log per run under
; ~/dsc_log, split by '### TEST CYCLE n/N'
; Waiting polls 'bcmcmd ps <port>' every -p seconds up to
; -w, and reads the state from the row whose first field
; ends in (<port>). Asking the SDK matters here: 'port en='
; changes the SDK, while 'show interfaces status' only
; follows once gbsyncd has synced it - and this test pauses
; gbsyncd
; -d prints every command as [CMD] in grey before running
; it, CMD_DELAY paces them, -n is a dry run that touches
; nothing and sleeps not at all
; !! all output is ASCII English on purpose. The DUT serial
; console mangles UTF-8: a Chinese error message came
; back as "ERROR: -c ? 1", which is worse than no
; message
; !! NOT verified on hardware
; tools/100G_dsc_cycle.txt Created - the hand-typed command list behind that script
; Same role as BER_Test.txt and TR518.txt: the bench
; version of the commands, kept so the .sh can be checked
; against what a person would actually type. Change one and
; change the other
; ============================================================================= ; =============================================================================
include "config.ttl" include "config.ttl"
@@ -8,8 +8,14 @@ gettime str_starttime "%Y%m%d-%H%M%S"
getdir mdir getdir mdir
if EN_log = 1 then if EN_log = 1 then
sprintf2 filename '%s\Logs\%s_%s_%s.log' mdir project_name "Reboot_Cycle" str_starttime if EN_Cold_Warm_boot = 1 then
logopen filename 0 1 1 1 sprintf2 filename '%s\Logs\%s_%s_%s.log' mdir project_name "Reboot_Cycle" str_starttime
logopen filename 0 1 1 1
elseif EN_Cold_Warm_boot = 2 then
sprintf2 filename '%s\Logs\%s_%s_%s.log' mdir project_name "Power_Cycle" str_starttime
logopen filename 0 1 1 1
else
endif
endif endif
cnt_cycle = 0 cnt_cycle = 0
@@ -26,9 +32,16 @@ while 1
wait prompt_sonic_root wait prompt_sonic_root
sendln "sleep 10" sendln "sleep 10"
wait prompt_sonic_root if EN_Cold_Warm_boot = 1 then
sprintf2 strTitle "echo -e '\033[1;32mReboot Cycle : %d \033[0m'" cnt_cycle wait prompt_sonic_root
sendln strTitle sprintf2 strTitle "echo -e '\033[1;32mReboot Cycle : %d \033[0m'" cnt_cycle
sendln strTitle
elseif EN_Cold_Warm_boot = 2 then
wait prompt_sonic_root
sprintf2 strTitle "echo -e '\033[1;32mPower Cycle : %d \033[0m'" cnt_cycle
sendln strTitle
else
endif
; ========== Pre Setting ========== ; ========== Pre Setting ==========
; Make log directory ; Make log directory
@@ -172,6 +185,8 @@ while 1
sprintf2 strCKPt5 "echo -e '\033[36mCheck Pt5 : SSD Test\033[0m'" sprintf2 strCKPt5 "echo -e '\033[36mCheck Pt5 : SSD Test\033[0m'"
sendln strCKPt5 sendln strCKPt5
wait prompt_sonic_root
sendln "rm -rf /host/hw-eval/jobs/qfx5252-stress-ssd.log"
wait prompt_sonic_root wait prompt_sonic_root
sendln "qfx5252-stress-ssd 1G --runtime 30 --passes 2 --write --force --log /host/hw-eval/jobs/qfx5252-stress-ssd.log" sendln "qfx5252-stress-ssd 1G --runtime 30 --passes 2 --write --force --log /host/hw-eval/jobs/qfx5252-stress-ssd.log"
wait prompt_sonic_root wait prompt_sonic_root
@@ -189,6 +204,8 @@ while 1
waitregex '<<DEV=([^>]*)>>' waitregex '<<DEV=([^>]*)>>'
usb_dev = groupmatchstr1 usb_dev = groupmatchstr1
wait prompt_sonic_root
sendln "rm -rf /host/hw-eval/jobs/qfx5252-stress-usb.log"
wait prompt_sonic_root wait prompt_sonic_root
sprintf2 cmd "qfx5252-stress-usb %s 256M --runtime 30 --passes 2 --write --log /host/hw-eval/jobs/qfx5252-stress-usb.log" usb_dev sprintf2 cmd "qfx5252-stress-usb %s 256M --runtime 30 --passes 2 --write --log /host/hw-eval/jobs/qfx5252-stress-usb.log" usb_dev
sendln cmd sendln cmd
@@ -412,8 +429,15 @@ while 1
sendln "show uptime" sendln "show uptime"
; ========== Show Power on uptime ========== ; ========== Show Power on uptime ==========
wait prompt_sonic_root if EN_Cold_Warm_boot = 1 then
sendln "reboot" wait prompt_sonic_root
sendln "reboot"
elseif EN_Cold_Warm_boot = 2 then
wait prompt_sonic_root
sendln "pcimem /sys/bus/pci/devices/0000:05:00.0/resource0 0x480 w 0xD0000"
else
endif
wait "Booting in blind mode" wait "Booting in blind mode"
endwhile endwhile
@@ -0,0 +1,608 @@
#!/bin/sh
# =============================================================================
# bcm_100g_dsc_cycle.sh
#
# Author : ETWen
# Date : 20260909
#
# Version History
# V1.0.0 20260909 Initial Version
# V1.1.0 20260909 All runtime output is ASCII English - the DUT serial
# console mangles UTF-8 (a Chinese error message came
# back as "ERROR: -c ? 1"). Cycle count may be given as
# a bare argument. Added -d to echo every command, and
# CMD_DELAY to pace them.
# V1.2.0 20260909 Wait on "bcmcmd ps <port>" instead of
# "show interfaces status". The state comes from the
# SDK, which is what "port en=" actually changes, so it
# no longer depends on gbsyncd syncing into the DB.
#
# Description:
# Link down/up cycling on the 100G uplink. After every re-link it collects a
# SerDes DSC dump and reads the retimer back, so you can see whether the
# tuned TX taps settle to the same place each time the link comes up.
#
# One run has two parts:
# 1. Tuning -- once. pause gbsyncd -> serdes-tx-set sys 0..7
# -> tx_taps set on four lanes of each unit
# 2. Cycle -- repeated N times:
# port en=0 -> wait until "ps <port>" reads !ena on both units
# port en=1 -> wait until "ps <port>" reads up on both units
# phydiag <port> dsc on each unit
# serdes-tx-get / serdes-rx-get sys 0..7
# all of it appended to one log, under a per-cycle heading
#
# Command source: tools/100G_dsc_cycle.txt (the hand-typed bench version).
# Change that file and this one together.
#
# Deployment:
# Script : /root/Blanton_Script/bcm_100g_dsc_cycle.sh
# Log : ~/dsc_log/100G_dsc_cycle_test_<YYYYmmdd_HHMMSS>.log
# One file per run, every cycle in it, split by "### TEST CYCLE n/N".
#
# Paths come from $HOME. Run as root (after sudo -i) and ~ is /root.
#
# Usage:
# ./bcm_100g_dsc_cycle.sh # print this help
# ./bcm_100g_dsc_cycle.sh 200 # 200 cycles
# ./bcm_100g_dsc_cycle.sh -c 200 # same thing
# ./bcm_100g_dsc_cycle.sh -c 200 -d # echo every command as it is issued
# ./bcm_100g_dsc_cycle.sh -c 10 -T # skip tuning, cycle only
# ./bcm_100g_dsc_cycle.sh -c 1 -n # dry run, print commands, touch nothing
#
# Exit code:
# 0 = every cycle linked down and up as expected, and all data was collected
# 1 = a cycle timed out waiting for link, or data was missing
# 2 = bad argument / path
# 3 = a required tool is missing (bcmcmd / show / blanton_barchetta_tool)
#
# The last line is always "### DSC_CYCLE_DONE rc=<n>", for a TTL wait to
# key on.
#
# Design notes:
# - POSIX sh. No bash, no Python.
# - All runtime output is plain ASCII. The DUT console is not UTF-8 clean:
# Chinese comes back as "M-^A"-style garbage, and an error you cannot read
# is worse than no error at all. Comments are English for the same reason -
# whoever debugs this on the bench is reading it through that console.
# - Link state comes from "bcmcmd -n <unit> -c 'ps <port>'". Its header is
# two stacked lines, so a column index cannot be derived from it the way
# it can from a SONiC table. Instead the data row is found by its first
# field ending in "(<port>)" - e.g. ce0(268) - and the state is the field
# after it:
# ce0(268) !ena 4 100G FD ... <- port disabled
# ce0(268) up 4 100G FD ... <- link up
# "down" is therefore tested as "not up", which covers !ena as well as an
# enabled port whose link has not come back.
# - Asking the SDK rather than "show interfaces status" matters here: "port
# en=" changes the SDK, and the SONiC table only follows once gbsyncd and
# orchagent have synced it into the DB. Since this test pauses gbsyncd,
# the SONiC table was the wrong thing to wait on.
# - bcmcmd reads stdin, and inside a while loop it will swallow the loop's
# input. blanton_traffic_linespeed.sh hit this at v0.3.1 (only the first
# VLAN was configured) and bcm_dsc_scan.sh hit it again at v1.3.0 (only
# the first port was scanned). Every external command here gets
# < /dev/null.
# - bcmcmd exits 0 even when the diag shell rejects a command, so the DSC is
# verified by counting "SERDES DISPLAY DIAG DATA END", and the retimer read
# by counting sections that actually have output under them - not by exit
# code, and not by counting headings this script prints itself.
#
# Cautions:
# - !! While gbsyncd is paused the gearbox state stops being synced into the
# DB. The wait reads the SDK directly ("ps <port>") so it is not affected,
# but anything you check by hand with "show interfaces status" during the
# run may be stale. Set PAUSE_GBSYNCD=0 (or -P) if that gets in the way.
# - !! On exit, including Ctrl-C, both gbsyncd containers are unpaused again
# so nothing is left paused. Set RESUME_GBSYNCD=0 to keep them paused.
# - !! phydiag dsc is read-only, but port en=0/1 and tx_taps set do change
# the hardware. Do not run this at the same time as blanton_ber.sh,
# blanton_traffic_linespeed or blanton_tr518.sh - they take the same ports.
# - !! NOT verified on hardware yet.
# =============================================================================
set -u
# ----------------------------------------------------------------------------
# Knobs - change these for a different DUT or port, not the flow below
# ----------------------------------------------------------------------------
PORT=268 # bcmcmd port number, not a port name
UNITS="0 1" # units to drive
SYS_LANES="0 1 2 3 4 5 6 7" # blanton_barchetta_tool sys lanes, tuning
SYS_TX="pre=-10 main=87 post=0" # TX setting applied to each of them
# tx_taps, one line per lane: "<unit> <lane> <main> <pre1> <post1>"
TX_TAPS="
0 0 31 4 13
0 1 31 4 13
0 2 27 4 13
0 3 29 4 13
1 0 29 4 13
1 1 30 4 13
1 2 31 4 13
1 3 30 4 13
"
TX_LANES="0 1 2 3 4 5 6 7" # retimer TX lanes read back each cycle
RX_LANES="0 1 2 3 4 5 6 7" # retimer RX lanes read back each cycle
SUDO="" # set to "sudo" if not running as root
PAUSE_GBSYNCD=1 # pause gbsyncd0/1 before tuning
RESUME_GBSYNCD=1 # unpause again on exit, Ctrl-C included
GBSYNCD="gbsyncd0 gbsyncd1"
LINK_WAIT=90 # give up waiting for a link state after this
POLL=2 # seconds between "ps <port>" reads while waiting
SETTLE=2 # quiet time after link up, before collecting
CMD_DELAY=0.2 # pause after every command issued to the DUT
OUTDIR="${HOME:-/root}/dsc_log"
# ----------------------------------------------------------------------------
CYCLES=0
DRYRUN=0
DOTUNE=1
DEBUG=0
RC=0
usage() {
cat >&2 << 'USAGE'
Usage: bcm_100g_dsc_cycle.sh <cycles> [options]
bcm_100g_dsc_cycle.sh -c <cycles> [options]
-c <n> how many cycles to run. May also be given as a bare argument.
-o <dir> log directory (default: ~/dsc_log)
-w <sec> how long to wait for a link state (default: 90)
-p <sec> how often to re-read the port state (default: 2)
-s <sec> quiet time after link up, before collecting (default: 2)
-d debug: echo every command as [CMD] before running it
-T skip tuning, go straight to the cycles
-P leave gbsyncd alone (no pause, no unpause)
-n dry run: print the commands, touch nothing
-h this help
One log per run:
~/dsc_log/100G_dsc_cycle_test_<YYYYmmdd_HHMMSS>.log
Each cycle is one "### TEST CYCLE n/N" section holding, in order: the "ps"
output showing the port down, the same showing it up, phydiag dsc for both
units, and the retimer serdes-tx-get / serdes-rx-get for sys 0..7.
Waiting for a port state is a poll, not a single read: "bcmcmd ps <port>" on
every unit, every -p seconds, until both agree or -w seconds have passed.
Examples:
./bcm_100g_dsc_cycle.sh 200
./bcm_100g_dsc_cycle.sh 200 -d
./bcm_100g_dsc_cycle.sh -c 20 -T -w 120
USAGE
exit 1
}
[ $# -eq 0 ] && usage
# getopts stops at the first argument that is not an option, so "200 -d" would
# leave -d unread. Parse, take the bare count, then parse whatever followed it.
# getopts inside a function reads the function's own arguments; OPTIND is
# global, so it has to be reset on each entry.
PARSED=0
parse_opts() {
OPTIND=1
while getopts "c:o:w:p:s:dTPnh" opt; do
case "$opt" in
c) CYCLES="$OPTARG" ;;
o) OUTDIR="$OPTARG" ;;
w) LINK_WAIT="$OPTARG" ;;
p) POLL="$OPTARG" ;;
s) SETTLE="$OPTARG" ;;
d) DEBUG=1 ;;
T) DOTUNE=0 ;;
P) PAUSE_GBSYNCD=0; RESUME_GBSYNCD=0 ;;
n) DRYRUN=1 ;;
*) usage ;;
esac
done
PARSED=$((OPTIND - 1))
}
parse_opts "$@"
shift "$PARSED"
# Allow the count as a bare argument: "... 200" == "... -c 200". That is what
# most people type first, and answering it with an error helps nobody.
if [ "$CYCLES" = "0" ] && [ $# -ge 1 ]; then
case "$1" in
-*) : ;; # an option; leave it for the second pass
*) CYCLES="$1"; shift ;; # anything else is meant to be the count,
esac # and the check below says so if it is not
fi
# Anything after the count is options again: "200 -d -T"
if [ $# -gt 0 ]; then
parse_opts "$@"
shift "$PARSED"
fi
if [ $# -gt 0 ]; then
echo "ERROR: unexpected argument '$1'" >&2
echo " run with no argument to see the usage" >&2
exit 2
fi
case "$CYCLES" in
''|*[!0-9]*)
echo "ERROR: cycle count must be a positive integer (got: '$CYCLES')" >&2
echo " try: $0 200 or $0 -c 200" >&2
exit 2 ;;
esac
if [ "$CYCLES" -lt 1 ]; then
echo "ERROR: cycle count must be at least 1 (got: $CYCLES)" >&2
exit 2
fi
# ----------------------------------------------------------------------------
# Colour, only when stdout is a terminal, so a redirected run stays clean
# ----------------------------------------------------------------------------
if [ -t 1 ] && [ -z "${NO_COLOR:-}" ]; then
C_INFO=$(printf '\033[1;34m') # blue
C_OK=$(printf '\033[1;32m') # green
C_WARN=$(printf '\033[1;33m') # yellow
C_ERR=$(printf '\033[1;31m') # red
C_CMD=$(printf '\033[90m') # dark grey
C_OFF=$(printf '\033[0m')
else
C_INFO=''; C_OK=''; C_WARN=''; C_ERR=''; C_CMD=''; C_OFF=''
fi
# A dry run touches nothing, so it must not sleep either - otherwise
# "-c 200 -n" sits there for as long as the real thing would.
naptime() { [ "$DRYRUN" = "1" ] && return 0; [ "${1:-0}" = "0" ] && return 0; sleep "$1"; }
info() { printf '%s[info]%s %s\n' "$C_INFO" "$C_OFF" "$*"; }
ok() { printf '%s[ OK ]%s %s\n' "$C_OK" "$C_OFF" "$*"; }
warn() { printf '%s[warn]%s %s\n' "$C_WARN" "$C_OFF" "$*" >&2; }
err() { printf '%s[ERR ]%s %s\n' "$C_ERR" "$C_OFF" "$*" >&2; }
# Both -d and -n print commands. During collection the whole block is
# redirected into a temp file, so those lines would end up in the file with
# nothing on screen. Bind fd 9 to the terminal up front and print there.
if { [ "$DEBUG" = "1" ] || [ "$DRYRUN" = "1" ]; } && ( true > /dev/tty ) 2>/dev/null; then
exec 9> /dev/tty
else
exec 9>&1
fi
# ':' is a special built-in in dash (the DUT's /bin/sh); a failed redirection
# on one kills the whole shell with no message. Hence (true > /dev/tty) in a
# subshell above - if that cannot open, only the subshell dies.
# Print a command the way you would type it: quote any argument that needs it,
# so the line can be copied back to the prompt as is.
# NOTE: sh has no local variables - every name assigned here is global. All
# helper variables are therefore prefixed. _show_cmd used to assign "tag",
# which silently overwrote the caller's cycle label and turned every later
# message into "[info] [CMD] : Wait Port Link Down".
_show_cmd() {
_sc_tag="$1"; _sc_col="$2"; shift 2
{
printf '%s%s%s' "$_sc_col" "$_sc_tag" "$C_OFF"
for a in "$@"; do
case "$a" in
''|*[!A-Za-z0-9_./=:,+-]*) printf " '%s'" "$a" ;;
*) printf ' %s' "$a" ;;
esac
done
printf '\n'
} >&9
}
# Every external command goes through here: never reads stdin, echoed when
# asked, and paced by CMD_DELAY so the DUT is not hammered.
run() {
[ "$DEBUG" = "1" ] && [ "$DRYRUN" = "0" ] && _show_cmd '[CMD]' "$C_CMD" "$@"
if [ "$DRYRUN" = "1" ]; then
_show_cmd ' +' "$C_CMD" "$@"
return 0
fi
"$@" < /dev/null
_cmd_rc=$?
naptime "$CMD_DELAY"
return $_cmd_rc
}
# Same, but discards output. Do not write "> /dev/null" at the call site or
# the [CMD] line goes with it.
runq() {
[ "$DEBUG" = "1" ] && [ "$DRYRUN" = "0" ] && _show_cmd '[CMD]' "$C_CMD" "$@"
if [ "$DRYRUN" = "1" ]; then
_show_cmd ' +' "$C_CMD" "$@"
return 0
fi
"$@" < /dev/null > /dev/null 2>&1
_cmd_rc=$?
naptime "$CMD_DELAY"
return $_cmd_rc
}
# ----------------------------------------------------------------------------
# Preflight
# ----------------------------------------------------------------------------
if [ "$DRYRUN" = "0" ]; then
command -v bcmcmd > /dev/null 2>&1 || { err "not found: bcmcmd"; exit 3; }
# -T only skips tuning; the per-cycle retimer read still needs this tool
if [ ! -x /usr/sbin/blanton_barchetta_tool ]; then
err "not found: /usr/sbin/blanton_barchetta_tool (tuning and the per-cycle retimer read both use it)"
exit 3
fi
fi
# busybox sleep may not take a fraction
if [ "$DRYRUN" = "0" ] && [ "$CMD_DELAY" != "0" ] && ! sleep "$CMD_DELAY" 2>/dev/null; then
warn "sleep '$CMD_DELAY' not supported, falling back to 1"
CMD_DELAY=1
fi
mkdir -p "$OUTDIR" 2>/dev/null || { err "cannot create outdir: $OUTDIR"; exit 2; }
[ -w "$OUTDIR" ] || { err "outdir not writable: $OUTDIR"; exit 2; }
TS=$(date '+%Y%m%d_%H%M%S')
LOG="${OUTDIR}/100G_dsc_cycle_test_${TS}.log"
: > "$LOG" || { err "cannot write $LOG"; exit 2; }
STATUS_TMP=$(mktemp) || { err "mktemp failed"; exit 2; }
CYCLE_TMP=$(mktemp) || { err "mktemp failed"; exit 2; }
# ----------------------------------------------------------------------------
# gbsyncd must go back the way it was found, or the container stays paused
# ----------------------------------------------------------------------------
PAUSED=0
resume_gbsyncd() {
[ "$PAUSED" = "1" ] || return 0
[ "$RESUME_GBSYNCD" = "1" ] || return 0
PAUSED=0
for _rg_c in $GBSYNCD; do
if runq $SUDO docker unpause "$_rg_c"; then
info "gbsyncd resumed: $_rg_c"
else
warn "docker unpause $_rg_c failed - check it by hand"
fi
done
}
on_exit() { rm -f "$STATUS_TMP" "$CYCLE_TMP"; resume_gbsyncd; }
on_int() {
echo ""
warn "interrupted by user"
resume_gbsyncd
printf '\n### DSC_CYCLE_DONE rc=130 (INTERRUPTED)\n' >> "$LOG"
printf '### DSC_CYCLE_DONE rc=130 (INTERRUPTED)\n'
rm -f "$STATUS_TMP" "$CYCLE_TMP"
exit 130
}
trap on_exit EXIT
trap on_int INT TERM
tee_log() { tee -a "$LOG"; }
# ----------------------------------------------------------------------------
# Link state. Find which field "Oper" is from the header row, then read that
# field on every data row. Returns 0 only when every interface matches.
# The table itself is left in $STATUS_TMP for the caller to print.
# ----------------------------------------------------------------------------
# $1 = "up" or "down". Runs "ps <port>" on every unit, leaves the combined
# output in $STATUS_TMP for the caller to print, and returns 0 only when every
# unit agrees.
link_is() {
_li_want="$1"
: > "$STATUS_TMP"
for _li_u in $UNITS; do
printf '\n[unit %s] bcmcmd -n %s -c "ps %s"\n' "$_li_u" "$_li_u" "$PORT" >> "$STATUS_TMP"
if [ "$DRYRUN" = "1" ]; then
_show_cmd ' +' "$C_CMD" bcmcmd -n "$_li_u" -c "ps $PORT"
printf ' ce0(%s) %s 4 100G FD (dry-run)\n' \
"$PORT" "$([ "$_li_want" = up ] && echo 'up ' || echo '!ena')" >> "$STATUS_TMP"
continue
fi
[ "$DEBUG" = "1" ] && _show_cmd '[CMD]' "$C_CMD" bcmcmd -n "$_li_u" -c "ps $PORT"
bcmcmd -n "$_li_u" -c "ps $PORT" < /dev/null >> "$STATUS_TMP" 2>&1
naptime "$CMD_DELAY"
done
# The data row is the one whose first field ends in "(<port>)"; the state
# is the field right after it. "down" is "not up", so !ena counts.
awk -v port="$PORT" -v want="$_li_want" -v units="$(echo "$UNITS" | wc -w)" '
$1 ~ ("\\(" port "\\)$") {
n++
if (want == "up") { if ($2 != "up") bad++ }
else { if ($2 == "up") bad++ }
}
END { exit (n >= units && bad == 0) ? 0 : 1 }
' "$STATUS_TMP"
}
# $1 = wanted state (Down/up), $2 = label for the messages
wait_link() {
_wl_want="$1"; _wl_tag="$2"
_wl_waited=0
while :; do
if link_is "$_wl_want"; then
{
printf '\n--- port %s after %ss ---\n' "$_wl_want" "$_wl_waited"
cat "$STATUS_TMP"
} | tee_log
ok "$_wl_tag : port $_wl_want after ${_wl_waited}s"
return 0
fi
[ "$_wl_waited" -ge "$LINK_WAIT" ] && break
naptime "$POLL"
_wl_waited=$((_wl_waited + POLL))
done
{
printf '\n--- TIMEOUT waiting for port %s (%ss) ---\n' "$_wl_want" "$LINK_WAIT"
cat "$STATUS_TMP"
} | tee_log
err "$_wl_tag : TIMEOUT waiting for port $_wl_want (${LINK_WAIT}s)"
return 1
}
port_en() {
for _pe_u in $UNITS; do
runq bcmcmd -n "$_pe_u" -c "port ${PORT} en=$1"
done
}
# ----------------------------------------------------------------------------
# Log header
# ----------------------------------------------------------------------------
{
printf '=====================================================================\n'
printf '100G DSC cycle test\n'
printf ' script : bcm_100g_dsc_cycle.sh V1.2.0\n'
printf ' started : %s\n' "$(date '+%Y-%m-%d %H:%M:%S')"
printf ' cycles : %s\n' "$CYCLES"
printf ' port : %s units : %s\n' "$PORT" "$UNITS"
printf ' tuning : %s\n' "$([ "$DOTUNE" = 1 ] && echo yes || echo 'skipped (-T)')"
printf ' tx lanes : %s\n' "$TX_LANES"
printf ' rx lanes : %s\n' "$RX_LANES"
printf ' link wait: %ss poll %ss settle %ss cmd delay %ss\n' \
"$LINK_WAIT" "$POLL" "$SETTLE" "$CMD_DELAY"
printf '=====================================================================\n'
} >> "$LOG"
info "log : $LOG"
# ----------------------------------------------------------------------------
# 1. Tuning, once
# ----------------------------------------------------------------------------
if [ "$DOTUNE" = "1" ]; then
info "Tuning"
printf '\n### TUNING %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" >> "$LOG"
if [ "$PAUSE_GBSYNCD" = "1" ]; then
for c in $GBSYNCD; do
if runq $SUDO docker pause "$c"; then
PAUSED=1
info "gbsyncd paused: $c"
else
warn "docker pause $c failed (container not running?)"
fi
done
fi
run $SUDO /usr/sbin/blanton_barchetta_tool --serdes-tx-get sys 0 2>&1 | tee_log
for l in $SYS_LANES; do
# shellcheck disable=SC2086
run $SUDO /usr/sbin/blanton_barchetta_tool --serdes-tx-set sys "$l" $SYS_TX 2>&1 | tee_log
done
echo "$TX_TAPS" | while read -r u lane main pre1 post1; do
[ -n "${u:-}" ] || continue
run bcmcmd -n "$u" -c \
" dsh -c 'phydiag ${PORT} tx_taps set lane=${lane} main=${main} pre1=${pre1} post1=${post1}' " \
2>&1 | tee_log
done
ok "Tuning done"
fi
# ----------------------------------------------------------------------------
# 2. Test cycles
# ----------------------------------------------------------------------------
n=1
while [ "$n" -le "$CYCLES" ]; do
tag="Test Cycle $n / $CYCLES"
printf '\n\n### TEST CYCLE %s/%s %s\n' "$n" "$CYCLES" \
"$(date '+%Y-%m-%d %H:%M:%S')" >> "$LOG"
# ---- Disable Port ------------------------------------------------------
info "$tag : Disable Port"
port_en 0
info "$tag : Wait Port Link Down"
wait_link down "$tag" || RC=1
# ---- Enable Port -------------------------------------------------------
info "$tag : Enable Port"
port_en 1
info "$tag : Wait Link up"
if wait_link up "$tag"; then
naptime "$SETTLE"
else
RC=1
fi
# ---- Collect -----------------------------------------------------------
# Written to a per-cycle temp file first, then appended. Appending to the
# log and then re-reading it to check would mean re-reading an ever larger
# file every round - O(n^2) over 200 cycles.
info "$tag : Collect Data"
: > "$CYCLE_TMP"
for u in $UNITS; do
{
printf '\n--- phydiag %s dsc : unit %s ---\n' "$PORT" "$u"
run bcmcmd -n "$u" -c "dsh -c 'phydiag ${PORT} dsc'" 2>&1
} >> "$CYCLE_TMP"
done
info "$tag : Read Retimer TX"
for l in $TX_LANES; do
{
printf '\n--- serdes-tx-get sys %s ---\n' "$l"
run $SUDO /usr/sbin/blanton_barchetta_tool --serdes-tx-get sys "$l" 2>&1
} >> "$CYCLE_TMP"
done
info "$tag : Read Retimer RX"
for l in $RX_LANES; do
{
printf '\n--- serdes-rx-get sys %s ---\n' "$l"
run $SUDO /usr/sbin/blanton_barchetta_tool --serdes-rx-get sys "$l" 2>&1
} >> "$CYCLE_TMP"
done
cat "$CYCLE_TMP" >> "$LOG"
if [ "$DRYRUN" = "0" ]; then
# bcmcmd exits 0 even when it failed, so count the end marker instead
got=$(grep -c 'SERDES DISPLAY DIAG DATA END' "$CYCLE_TMP" 2>/dev/null || true)
want=$(echo "$UNITS" | wc -w)
if [ "${got:-0}" -lt "$want" ]; then
err "$tag : got only ${got:-0}/${want} DSC dumps"
RC=1
fi
# The section headings are printed by this script, so counting them
# only proves the loop ran. Count sections that have output under them.
for d in tx rx; do
case "$d" in
tx) wantl="$TX_LANES" ;;
rx) wantl="$RX_LANES" ;;
esac
got_l=$(awk -v pat="--- serdes-${d}-get sys " '
index($0, pat) == 1 { if (insec && body) ok++; insec = 1; body = 0; next }
/^--- / || /^### / { if (insec && body) ok++; insec = 0; next }
insec && NF { body = 1 }
END { if (insec && body) ok++; print ok + 0 }
' "$CYCLE_TMP")
want_l=$(echo "$wantl" | wc -w)
if [ "${got_l:-0}" -lt "$want_l" ]; then
err "$tag : retimer $(echo "$d" | tr 'a-z' 'A-Z') returned data for only ${got_l:-0}/${want_l} lanes"
RC=1
fi
done
fi
n=$((n + 1))
done
resume_gbsyncd
printf '\n### DSC_CYCLE_DONE rc=%s\n' "$RC" >> "$LOG"
if [ "$RC" = "0" ]; then
ok "all $CYCLES cycles done -> $LOG"
else
err "some cycles did not complete as expected - see $LOG"
fi
printf '### DSC_CYCLE_DONE rc=%s\n' "$RC"
exit "$RC"
@@ -1,8 +1,31 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# ============================================================================= # =============================================================================
# blanton_ber.sh -- SerDes BER test over every cabled port (PRBS via phydiag) # blanton_ber.sh -- SerDes BER test, cabled ports PLUS the 100G uplink
# ============================================================================= # =============================================================================
# Version History: # Version History:
# V1.1.1 2026-09-15 Fix: "the BCM shell rejected 'phydiag ... prbs get'" on
# a run that was in fact working. The rejection check
# grepped the WHOLE output, case-insensitively, for
# ERROR|Invalid|... -- and 'prbs get' prints one line per
# lane, so a lane that had not locked ("PRBS Error!")
# matched. The command was fine; the output was data.
# Two consequences: a good run reported a hard error, and
# the lock-count check that was supposed to judge exactly
# those lanes never ran. Per-lane result lines are now
# excluded from the check, the match is case-sensitive,
# and a real rejection prints the offending line.
# The rejection check had been there since V1.0.0, so
# every earlier run carried the same blind spot.
# V1.1.0 2026-09-15 Adds port 268, the 100G uplink, which the old port list
# deliberately skipped (it jumps 256-259 -> 270-308).
# The per-lane table is unchanged; the summary now breaks
# each unit into "200G ports" and "100G p268" plus a
# TOTAL row, so the uplink cannot hide inside a 432-lane
# average. A group that reports no lane at all counts as
# a failure, not a pass - silence on the port under test
# is not a result. Developed on the bench as
# blanton_ber_test.sh and folded back over this file once
# it was proven, so there is one BER script again.
# V1.0.1 2026-08-28 Fix: every lane reported NA and the Port column showed # V1.0.1 2026-08-28 Fix: every lane reported NA and the Port column showed
# "NA". bcmcmd returns CRLF, and the BER is the last token # "NA". bcmcmd returns CRLF, and the BER is the last token
# on its line, so the CR stayed glued to the number: it # on its line, so the CR stayed glued to the number: it
@@ -19,8 +42,24 @@
# ============================================================================= # =============================================================================
# #
# This is the WHOLE-SWITCH BER sweep: every cabled port, both lanes, PRBS-31 # This is the WHOLE-SWITCH BER sweep: every cabled port, both lanes, PRBS-31
# through the SerDes. Not to be confused with port_prbs_monitor.sh, which drives # through the SerDes -- and, unlike blanton_ber.sh, the 100G uplink as well.
# the two 100G uplinks only and polls them continuously. # Not to be confused with port_prbs_monitor.sh, which drives the two 100G
# uplinks only and polls them continuously.
#
# The port list is two groups:
# BER_PORTS_200G the cabled 200G ports (432 lanes per unit)
# BER_PORTS_100G the 100G uplink, port 268 = ce0, one per unit
# They are measured in one phydiag pass, but the summary reports them
# separately per unit, plus a TOTAL:
#
# Unit GROUP LANES PASS FAIL NA WORST BER ...
# unit 0 200G ports 432 432 0 0 2.07e-08 PASS
# unit 0 100G p268 4 4 0 0 1.10e-11 PASS
# unit 0 TOTAL 436 436 0 0 2.07e-08 PASS
# unit 1 ...
#
# A 100G row of "NO DATA" means the uplink reported no lane at all, and is
# treated as a failure -- silence on the port under test is not a pass.
# #
# Order of operations (matches tools/BER_Test.txt): # Order of operations (matches tools/BER_Test.txt):
# init phydiag <ports> prbs set p=3 arm PRBS on every lane # init phydiag <ports> prbs set p=3 arm PRBS on every lane
@@ -51,7 +90,7 @@
# -u|--unit <0|1|all|0,1> bcmcmd unit(s). Default: $BER_UNITS ("0 1") # -u|--unit <0|1|all|0,1> bcmcmd unit(s). Default: $BER_UNITS ("0 1")
# -c|--criteria <ber> report: PASS threshold (default $BER_MAX) # -c|--criteria <ber> report: PASS threshold (default $BER_MAX)
# -i|--interval <sec> start: prbsstat Interval (default $BER_INTERVAL) # -i|--interval <sec> start: prbsstat Interval (default $BER_INTERVAL)
# -p|--ports <list> override the port list # -p|--ports <list> override the whole port list (both groups)
# -F|--fail-only report: list only the lanes that FAIL # -F|--fail-only report: list only the lanes that FAIL
# --tsv report: tab-separated (paste into Excel) # --tsv report: tab-separated (paste into Excel)
# --raw report: also print the raw 'prbsstat Ber' output # --raw report: also print the raw 'prbsstat Ber' output
@@ -63,7 +102,8 @@
# #
# Environment overrides: # Environment overrides:
# BCMCMD_CMD=bcmcmd BER_UNITS="0 1" BER_MAX=1e-6 BER_INTERVAL=30 # BCMCMD_CMD=bcmcmd BER_UNITS="0 1" BER_MAX=1e-6 BER_INTERVAL=30
# BER_PORTS="1-24,..." BER_POLY=p=3 BER_LOG_DIR=/tmp DEBUG_MODE=1 # BER_PORTS="1-24,..." BER_PORTS_200G=... BER_PORTS_100G=268
# BER_POLY=p=3 BER_LOG_DIR=/tmp DEBUG_MODE=1
# ============================================================================= # =============================================================================
# === Config ================================================================== # === Config ==================================================================
@@ -73,7 +113,18 @@ BER_UNITS="${BER_UNITS:-0 1}" # one DUT = two switch boards (0 and 1)
# The cabled ports, in bcm logical-port numbering. Same list as # The cabled ports, in bcm logical-port numbering. Same list as
# blanton_traffic_linespeed.sh's TL_PS_PORTS -- kept as one string so it can be # blanton_traffic_linespeed.sh's TL_PS_PORTS -- kept as one string so it can be
# pasted straight from/into a manual phydiag command. # pasted straight from/into a manual phydiag command.
BER_PORTS="${BER_PORTS:-1-24,40-96,112-168,184-240,256-259,270-308,310-327,342-380,382-399,414-452,454-471,486-524,526-543,558-565}" #
# Note the gap: the list runs 256-259 then 270-308, so 260..269 are not in it.
# 268 is ce0, the 100G uplink, and it is the reason this fork exists.
BER_PORTS_200G="${BER_PORTS_200G:-1-24,40-96,112-168,184-240,256-259,270-308,310-327,342-380,382-399,414-452,454-471,486-524,526-543,558-565}"
# The 100G uplink. One port per unit -- unit 0 and unit 1 each have their own
# ce0(268), so a two-unit run measures two of them. Comma-separated if more are
# ever added; the summary treats everything in here as the "100G" group.
BER_PORTS_100G="${BER_PORTS_100G:-268}"
# What actually goes on the phydiag command line. -p|--ports overrides the lot.
BER_PORTS="${BER_PORTS:-${BER_PORTS_200G},${BER_PORTS_100G}}"
BER_POLY="${BER_POLY:-p=3}" # passed to 'prbs set' BER_POLY="${BER_POLY:-p=3}" # passed to 'prbs set'
BER_INTERVAL="${BER_INTERVAL:-30}" # passed to 'prbsstat STArt Interval=' BER_INTERVAL="${BER_INTERVAL:-30}" # passed to 'prbsstat STArt Interval='
@@ -111,6 +162,24 @@ _ber_dbg() { [ "$DEBUG_MODE" = "1" ] && echo -e "\033[90m[DEBG] $*\033[0m" >&2;
BER_ERR_PAT="${BER_ERR_PAT:-ERROR|Failed to execute|Invalid|Syntax error|Unavailable|Unknown command}" BER_ERR_PAT="${BER_ERR_PAT:-ERROR|Failed to execute|Invalid|Syntax error|Unavailable|Unknown command}"
BER_BENIGN_PAT="${BER_BENIGN_PAT:-polling socket timeout}" BER_BENIGN_PAT="${BER_BENIGN_PAT:-polling socket timeout}"
# A per-lane result line -- " 185 : PRBS OK!..." from 'prbs get', or
# " 203[1] : 5.05e-09" from 'prbsstat Ber'. These are MEASUREMENTS, never
# shell diagnostics, and they must be kept out of the rejection check:
# a lane that did not lock prints its own wording (PRBS Error!, and similar),
# which a case-insensitive match on "ERROR" reads as the BCM shell refusing
# the command. That is what turned a perfectly good run into
# [ERR ] unit 1: the BCM shell rejected 'phydiag ... prbs get'
# while init/start/report were in fact working -- and it skipped the
# lock-count check that was supposed to judge those very lanes.
BER_LANE_LINE='^[ \t]*[0-9]+(\[[0-9]+\])?[ \t]*:'
# _ber_shell_error <output> -- echo the offending line(s), return 0 if any
_ber_shell_error() {
printf '%s\n' "$1" | tr -d '\r' \
| grep -vE "$BER_LANE_LINE" \
| grep -E "$BER_ERR_PAT"
}
# _ber_bcm <unit> <phydiag args...> -- wraps in dsh -c 'phydiag <ports> ...' # _ber_bcm <unit> <phydiag args...> -- wraps in dsh -c 'phydiag <ports> ...'
# echoes the DUT output on stdout, returns 0 ok / 1 failed # echoes the DUT output on stdout, returns 0 ok / 1 failed
_ber_bcm() { _ber_bcm() {
@@ -125,11 +194,14 @@ _ber_bcm() {
local out rc local out rc
out=$("$BCMCMD_CMD" -n "$unit" -c "$cmd" </dev/null 2>&1); rc=$? out=$("$BCMCMD_CMD" -n "$unit" -c "$cmd" </dev/null 2>&1); rc=$?
printf '%s\n' "$out" printf '%s\n' "$out"
if echo "$out" | grep -qiE "$BER_ERR_PAT"; then local why
why=$(_ber_shell_error "$out")
if [ -n "$why" ]; then
_ber_err "unit $unit: the BCM shell rejected 'phydiag ... $*'" _ber_err "unit $unit: the BCM shell rejected 'phydiag ... $*'"
printf '%s\n' "$why" | sed 's/^/ /' >&2
return 1 return 1
fi fi
if [ "$rc" -ne 0 ] && ! echo "$out" | grep -qiE "$BER_BENIGN_PAT"; then if [ "$rc" -ne 0 ] && ! printf '%s\n' "$out" | grep -qiE "$BER_BENIGN_PAT"; then
_ber_err "unit $unit: bcmcmd exited $rc on 'phydiag ... $*'" _ber_err "unit $unit: bcmcmd exited $rc on 'phydiag ... $*'"
return 1 return 1
fi fi
@@ -243,13 +315,32 @@ _ber_render() {
-v FAILONLY="$BER_FAIL_ONLY" \ -v FAILONLY="$BER_FAIL_ONLY" \
-v TSV="$BER_TSV" \ -v TSV="$BER_TSV" \
-v COLOR="$color" \ -v COLOR="$color" \
-v SIDE="$side" ' -v SIDE="$side" \
-v P100="$BER_PORTS_100G" '
function vc(v) { function vc(v) {
if (!COLOR) return v if (!COLOR) return v
if (v == "PASS") return "\033[32m" v "\033[0m" if (v == "PASS") return "\033[32m" v "\033[0m"
if (v == "FAIL") return "\033[31m" v "\033[0m" if (v == "FAIL") return "\033[31m" v "\033[0m"
return "\033[33m" v "\033[0m" return "\033[33m" v "\033[0m"
} }
# Fold one lane into a group tally. Called once for its own group and
# and once for ALL, so the TOTAL row is never a hand-added sum that can
# drift away from the parts.
function acct(g, verdict, berstr, berval, label) {
GN[g]++
if (verdict == "PASS") GP[g]++
else if (verdict == "FAIL") GF[g]++
else GNA[g]++
if (berval >= 0 && (!(g in GW) || berval > GW[g])) {
GW[g] = berval; GWS[g] = berstr; GWL[g] = label
}
}
function emit(g, lanes) {
lanes = GN[g] + 0
printf "SUM %s %d %d %d %d %s %s\n", g, lanes, GP[g] + 0, GF[g] + 0, GNA[g] + 0,
(lanes > 0 && (g in GWS)) ? GWS[g] : "-",
(lanes > 0 && (g in GWL)) ? GWL[g] : "-" > SIDE
}
BEGIN { BEGIN {
LIM = MAXBER + 0 LIM = MAXBER + 0
NC = 4 NC = 4
@@ -258,6 +349,18 @@ _ber_render() {
HDR[3] = "BER"; AL[3] = 1 HDR[3] = "BER"; AL[3] = 1
HDR[4] = "RESULT"; AL[4] = 1 HDR[4] = "RESULT"; AL[4] = 1
worst = -1; worstlane = "-" worst = -1; worstlane = "-"
# Expand the 100G list ("268" or "268,270-271") into a lookup, so a lane
# can be put in the right group by its port number alone.
np = split(P100, pp, /[ ,\t]+/)
for (i2 = 1; i2 <= np; i2++) {
if (pp[i2] == "") continue
if (split(pp[i2], rr, "-") == 2) {
for (j2 = rr[1] + 0; j2 <= rr[2] + 0; j2++) IS100[j2] = 1
} else {
IS100[pp[i2] + 0] = 1
}
}
} }
# bcmcmd hands back CRLF. On the cd/traffic report the counter is followed by # bcmcmd hands back CRLF. On the cd/traffic report the counter is followed by
# a delta, so the CR got trimmed with the rest of the line; here the BER is # a delta, so the CR got trimmed with the rest of the line; here the BER is
@@ -282,6 +385,8 @@ _ber_render() {
seen++ seen++
n++ n++
PORT[n] = port; LANE[n] = lane; BERS[n] = ber PORT[n] = port; LANE[n] = lane; BERS[n] = ber
grp = ((port + 0) in IS100) ? "P100" : "P200"
GRP[n] = grp
# a lane that reported something non-numeric has no usable measurement # a lane that reported something non-numeric has no usable measurement
if (ber ~ /^[0-9]+(\.[0-9]+)?([eE][-+]?[0-9]+)?$/) { if (ber ~ /^[0-9]+(\.[0-9]+)?([eE][-+]?[0-9]+)?$/) {
v = ber + 0 v = ber + 0
@@ -290,7 +395,10 @@ _ber_render() {
if (v > worst) { worst = v; worstlane = port "[" lane "]"; worststr = ber } if (v > worst) { worst = v; worstlane = port "[" lane "]"; worststr = ber }
} else { } else {
V[n] = "NA"; nna++ V[n] = "NA"; nna++
v = -1
} }
acct(grp, V[n], ber, v, port "[" lane "]")
acct("ALL", V[n], ber, v, port "[" lane "]")
} }
END { END {
if (seen == 0) { if (seen == 0) {
@@ -327,19 +435,32 @@ _ber_render() {
} }
print "" print ""
printf "# lanes measured : %d\n", n printf "# lanes measured : %d (200G %d, 100G %d)\n", n, GN["P200"] + 0, GN["P100"] + 0
printf "# %s %d %s %d %s %d (criteria BER < %s)\n", \ printf "# %s %d %s %d %s %d (criteria BER < %s)\n", \
vc("PASS"), npass + 0, vc("FAIL"), nfail + 0, vc("NA"), nna + 0, MAXBER vc("PASS"), npass + 0, vc("FAIL"), nfail + 0, vc("NA"), nna + 0, MAXBER
if (worst >= 0) if (worst >= 0)
printf "# worst lane : %s %s\n", worstlane, worststr printf "# worst lane : %s %s\n", worstlane, worststr
if (SIDE != "" && SIDE != "/dev/null") if (GN["P100"] + 0 == 0)
printf "SUM %d %d %d %d %s %s\n", n, npass + 0, nfail + 0, nna + 0, \ printf "# %s the 100G port list (%s) matched no lane in this capture\n", vc("NA"), P100
(worst >= 0 ? worststr : "-"), worstlane > SIDE else
printf "# 100G p%s : %s %d %s %d %s %d worst %s %s\n", P100,
vc("PASS"), GP["P100"] + 0, vc("FAIL"), GF["P100"] + 0,
vc("NA"), GNA["P100"] + 0,
("P100" in GWS) ? GWS["P100"] : "-",
("P100" in GWL) ? GWL["P100"] : "-"
if (SIDE != "" && SIDE != "/dev/null") {
emit("P200"); emit("P100"); emit("ALL")
}
exit (nfail || nna) ? 2 : 0 exit (nfail || nna) ? 2 : 0
}' "$log" }' "$log"
} }
# _ber_summary <stamp> -- one table for every unit, printed last # _ber_summary <stamp> -- one table for every unit, printed last.
#
# Each unit gets three rows: the 200G ports, the 100G uplink on its own, and a
# TOTAL. Unit 0 and unit 1 each have their own ce0(268), so the 100G row is per
# unit too -- a bad uplink on one board is what this fork exists to make
# visible, and averaged into 432 other lanes it would not be.
_ber_summary() { _ber_summary() {
local stamp="$1" unit side g r y n worst=0 local stamp="$1" unit side g r y n worst=0
g=$'\033[32m'; r=$'\033[31m'; y=$'\033[33m'; n=$'\033[0m' g=$'\033[32m'; r=$'\033[31m'; y=$'\033[33m'; n=$'\033[0m'
@@ -347,27 +468,41 @@ _ber_summary() {
echo "" echo ""
echo "===== summary : all units =====" echo "===== summary : all units ====="
printf '%-8s %8s %8s %8s %8s %-12s %-10s %s\n' \ printf '%-8s %-12s %7s %7s %7s %7s %-12s %-12s %s\n' \
"Unit" "LANES" "PASS" "FAIL" "NA" "WORST BER" "WORST LANE" "RESULT" "Unit" "GROUP" "LANES" "PASS" "FAIL" "NA" "WORST BER" "WORST LANE" "RESULT"
local tag lanes pass fail na wber wlane v any=0 local tag grp lanes pass fail na wber wlane v any=0 label
for unit in $BER_UNITS; do for unit in $BER_UNITS; do
side="$BER_LOG_DIR/berside_unit${unit}_${stamp}" side="$BER_LOG_DIR/bertside_unit${unit}_${stamp}"
[ -r "$side" ] || continue [ -r "$side" ] || continue
read -r tag lanes pass fail na wber wlane < "$side" 2>/dev/null while read -r tag grp lanes pass fail na wber wlane; do
[ "$tag" = "SUM" ] || continue [ "$tag" = "SUM" ] || continue
any=1 any=1
if [ "${fail:-0}" -gt 0 ] || [ "${na:-0}" -gt 0 ]; then case "$grp" in
v="${r}FAIL${n}"; worst=2 P200) label="200G ports" ;;
else P100) label="100G p${BER_PORTS_100G}" ;;
v="${g}PASS${n}" ALL) label="TOTAL" ;;
fi *) label="$grp" ;;
printf '%-8s %8s %8s %8s %8s %-12s %-10s %b\n' \ esac
"unit $unit" "$lanes" "$pass" "$fail" "$na" "$wber" "$wlane" "$v" # A group with no lanes is not a pass. On the 100G row it means the
# uplink never reported -- exactly the case worth flagging.
if [ "${lanes:-0}" -eq 0 ]; then
v="${y}NO DATA${n}"
[ "$grp" = "P100" ] && worst=2
elif [ "${fail:-0}" -gt 0 ] || [ "${na:-0}" -gt 0 ]; then
v="${r}FAIL${n}"; worst=2
else
v="${g}PASS${n}"
fi
printf '%-8s %-12s %7s %7s %7s %7s %-12s %-12s %b\n' \
"unit $unit" "$label" "$lanes" "$pass" "$fail" "$na" \
"$wber" "$wlane" "$v"
done < "$side"
done done
[ "$any" = "1" ] || echo " (no unit produced a result)" [ "$any" = "1" ] || echo " (no unit produced a result)"
echo "" echo ""
printf 'criteria: BER < %s\n' "$BER_MAX" printf 'criteria: BER < %s\n' "$BER_MAX"
printf '100G group: port(s) %s -- one per unit\n' "$BER_PORTS_100G"
echo "" echo ""
return "$worst" return "$worst"
} }
@@ -391,7 +526,7 @@ blanton_ber_report() {
stamp=$(date +%Y%m%d_%H%M%S 2>/dev/null || echo now) stamp=$(date +%Y%m%d_%H%M%S 2>/dev/null || echo now)
for unit in $BER_UNITS; do for unit in $BER_UNITS; do
log="$BER_LOG_DIR/prbsber_unit${unit}_${stamp}.log" log="$BER_LOG_DIR/prbsber_unit${unit}_${stamp}.log"
side="$BER_LOG_DIR/berside_unit${unit}_${stamp}" side="$BER_LOG_DIR/bertside_unit${unit}_${stamp}"
: > "$side" : > "$side"
if ! _ber_bcm "$unit" "prbsstat Ber" > "$log" 2>&1; then if ! _ber_bcm "$unit" "prbsstat Ber" > "$log" 2>&1; then
_ber_err "unit $unit: 'prbsstat Ber' failed (see $log)"; worst=1; continue _ber_err "unit $unit: 'prbsstat Ber' failed (see $log)"; worst=1; continue
@@ -409,22 +544,24 @@ blanton_ber_report() {
_ber_summary "$stamp"; rc=$? _ber_summary "$stamp"; rc=$?
[ "$rc" -gt "$worst" ] && worst=$rc [ "$rc" -gt "$worst" ] && worst=$rc
rm -f "$BER_LOG_DIR"/berside_unit*_"${stamp}" rm -f "$BER_LOG_DIR"/bertside_unit*_"${stamp}"
return "$worst" return "$worst"
} }
# === Help ==================================================================== # === Help ====================================================================
blanton_ber_help() { blanton_ber_help() {
echo -e "\033[1mblanton_ber.sh - SerDes BER test over every cabled port (PRBS via phydiag)\033[0m" echo -e "\033[1mblanton_ber.sh - SerDes BER test: cabled 200G ports + the 100G uplink\033[0m"
echo "" echo ""
echo -e " Units : \033[36m$BER_UNITS\033[0m poly: \033[36m$BER_POLY\033[0m interval: \033[36m${BER_INTERVAL}s\033[0m" echo -e " Units : \033[36m$BER_UNITS\033[0m poly: \033[36m$BER_POLY\033[0m interval: \033[36m${BER_INTERVAL}s\033[0m"
echo -e " Criteria : BER < \033[36m$BER_MAX\033[0m" echo -e " Criteria : BER < \033[36m$BER_MAX\033[0m"
echo -e " Ports : \033[36m$BER_PORTS\033[0m" echo -e " 200G : \033[36m$BER_PORTS_200G\033[0m"
echo -e " 100G : \033[36m$BER_PORTS_100G\033[0m (ce0, one per unit -- reported on its own row)"
echo "" echo ""
echo -e "\033[1mSubcommands\033[0m" echo -e "\033[1mSubcommands\033[0m"
echo -e " \033[33minit\033[0m prbs set $BER_POLY + prbs get (checks every lane locked)" echo -e " \033[33minit\033[0m prbs set $BER_POLY + prbs get (checks every lane locked)"
echo -e " \033[33mstart\033[0m prbsstat STArt Interval=$BER_INTERVAL" echo -e " \033[33mstart\033[0m prbsstat STArt Interval=$BER_INTERVAL"
echo -e " \033[33mreport\033[0m prbsstat Ber -> per-lane table + PASS/FAIL + summary" echo -e " \033[33mreport\033[0m prbsstat Ber -> per-lane table + PASS/FAIL + summary"
echo -e " summary splits each unit into 200G ports / 100G p$BER_PORTS_100G / TOTAL"
echo -e " \033[33mstop\033[0m prbsstat STOp" echo -e " \033[33mstop\033[0m prbsstat STOp"
echo -e " \033[33mclear\033[0m prbs clear (disarms PRBS -- 'init' again before measuring)" echo -e " \033[33mclear\033[0m prbs clear (disarms PRBS -- 'init' again before measuring)"
echo -e " \033[33mhelp\033[0m" echo -e " \033[33mhelp\033[0m"
@@ -433,7 +570,7 @@ blanton_ber_help() {
echo -e " \033[33m-u|--unit\033[0m <0|1|all> bcmcmd unit(s); 'all' = 0 1 (default: $BER_UNITS)" echo -e " \033[33m-u|--unit\033[0m <0|1|all> bcmcmd unit(s); 'all' = 0 1 (default: $BER_UNITS)"
echo -e " \033[33m-c|--criteria\033[0m <ber> PASS threshold (default $BER_MAX)" echo -e " \033[33m-c|--criteria\033[0m <ber> PASS threshold (default $BER_MAX)"
echo -e " \033[33m-i|--interval\033[0m <sec> prbsstat Interval (default $BER_INTERVAL)" echo -e " \033[33m-i|--interval\033[0m <sec> prbsstat Interval (default $BER_INTERVAL)"
echo -e " \033[33m-p|--ports\033[0m <list> override the port list" echo -e " \033[33m-p|--ports\033[0m <list> override the whole port list (both groups)"
echo -e " \033[33m-F|--fail-only\033[0m report: list only failing lanes" echo -e " \033[33m-F|--fail-only\033[0m report: list only failing lanes"
echo -e " \033[33m--tsv\033[0m report: tab-separated" echo -e " \033[33m--tsv\033[0m report: tab-separated"
echo -e " \033[33m--raw\033[0m report: also print the raw output" echo -e " \033[33m--raw\033[0m report: also print the raw output"
+10 -1
View File
@@ -34,4 +34,13 @@ BMC_USB_RUNTIME = 172800 ; Unit seconds, 172800 = 2 days
;================================================================ ;================================================================
prompt_login = "sonic login:" prompt_login = "sonic login:"
prompt_sonic = "admin@sonic:~$" prompt_sonic = "admin@sonic:~$"
prompt_sonic_root = "root@sonic:~#" prompt_sonic_root = "root@sonic:~#"
;================================================================
; Reboot or Power Cycle
;================================================================
; 0: Normal Test
; 1: Enable Reboot Cycle
; 2: Enable Power Cycle
EN_Cold_Warm_boot = 0
+90
View File
@@ -0,0 +1,90 @@
cd /usr/sbin
sudo docker pause gbsyncd1
sudo docker pause gbsyncd0
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-get sys 0
//Tuning
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-set sys 0 pre=-10 main=87 post=0
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-set sys 1 pre=-10 main=87 post=0
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-set sys 2 pre=-10 main=87 post=0
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-set sys 3 pre=-10 main=87 post=0
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-set sys 4 pre=-10 main=87 post=0
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-set sys 5 pre=-10 main=87 post=0
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-set sys 6 pre=-10 main=87 post=0
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-set sys 7 pre=-10 main=87 post=0
bcmcmd -n 0 -c " dsh -c 'phydiag 268 tx_taps set lane=0 main=31 pre1=4 post1=13' "
bcmcmd -n 0 -c " dsh -c 'phydiag 268 tx_taps set lane=1 main=31 pre1=4 post1=13' "
bcmcmd -n 0 -c " dsh -c 'phydiag 268 tx_taps set lane=2 main=27 pre1=4 post1=13' "
bcmcmd -n 0 -c " dsh -c 'phydiag 268 tx_taps set lane=3 main=29 pre1=4 post1=13' "
bcmcmd -n 1 -c " dsh -c 'phydiag 268 tx_taps set lane=0 main=29 pre1=4 post1=13' "
bcmcmd -n 1 -c " dsh -c 'phydiag 268 tx_taps set lane=1 main=30 pre1=4 post1=13' "
bcmcmd -n 1 -c " dsh -c 'phydiag 268 tx_taps set lane=2 main=31 pre1=4 post1=13' "
bcmcmd -n 1 -c " dsh -c 'phydiag 268 tx_taps set lane=3 main=30 pre1=4 post1=13' "
以下Test Cycle 目標200次
//Disable Port
bcmcmd -n 0 -c "port 268 en=0"
bcmcmd -n 1 -c "port 268 en=0"
//Wait Link Down
bcmcmd -n 0 -c "ps 268"
bcmcmd -n 1 -c "ps 268"
root@sonic:~# bcmcmd -n 0 -c "ps 268"
ps 268
ena/ speed/ link auto STP lrn max cut loop
port link Lns duplex scan neg? state pause discrd ops medium frame thru FEC back
ce0(268) !ena 4 100G FD SW No Forward Untag FA Backplane 9122 No RS528
drivshell>
root@sonic:~# bcmcmd -n 1 -c "ps 268"
ps 268
ena/ speed/ link auto STP lrn max cut loop
port link Lns duplex scan neg? state pause discrd ops medium frame thru FEC back
ce0(268) !ena 4 100G FD SW No Forward Untag FA Backplane 9122 No RS528
drivshell>
root@sonic:~#
//Enable Port
bcmcmd -n 0 -c "port 268 en=1"
bcmcmd -n 1 -c "port 268 en=1"
//Wait Link Up
bcmcmd -n 0 -c "ps 268"
bcmcmd -n 1 -c "ps 268"
root@sonic:~# bcmcmd -n 0 -c "ps 268"
ps 268
ena/ speed/ link auto STP lrn max cut loop
port link Lns duplex scan neg? state pause discrd ops medium frame thru FEC back
ce0(268) up 4 100G FD SW No Forward Untag FA Backplane 9122 No RS528
drivshell>
root@sonic:~# bcmcmd -n 1 -c "ps 268"
ps 268
ena/ speed/ link auto STP lrn max cut loop
port link Lns duplex scan neg? state pause discrd ops medium frame thru FEC back
ce0(268) up 4 100G FD SW No Forward Untag FA Backplane 9122 No RS528
drivshell>
root@sonic:~#
//Collect DSC Data
bcmcmd -n 0 -c "dsh -c 'phydiag 268 dsc'"
bcmcmd -n 1 -c "dsh -c 'phydiag 268 dsc'"
//讀Retimer TX/RX
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-get sys 0
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-get sys 1
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-get sys 2
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-get sys 3
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-get sys 4
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-get sys 5
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-get sys 6
sudo /usr/sbin/blanton_barchetta_tool --serdes-tx-get sys 7
sudo /usr/sbin/blanton_barchetta_tool --serdes-rx-get sys 0
sudo /usr/sbin/blanton_barchetta_tool --serdes-rx-get sys 1
sudo /usr/sbin/blanton_barchetta_tool --serdes-rx-get sys 2
sudo /usr/sbin/blanton_barchetta_tool --serdes-rx-get sys 3
sudo /usr/sbin/blanton_barchetta_tool --serdes-rx-get sys 4
sudo /usr/sbin/blanton_barchetta_tool --serdes-rx-get sys 5
sudo /usr/sbin/blanton_barchetta_tool --serdes-rx-get sys 6
sudo /usr/sbin/blanton_barchetta_tool --serdes-rx-get sys 7