diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 99ee278..d05867d 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -136,6 +136,7 @@ Blanton_TTL_Script/ │ ├── 100G_PRBS.txt # 100G PRBS 原始指令（port_prbs_monitor.sh 的來源） │ ├── TR518.txt # 內建封包測試 tr 518 → blanton_tr518.sh │ ├── BER_Test.txt # 全機 PRBS BER 指令 → blanton_ber.sh +│ ├── 100G_dsc_cycle.txt # 100G tuning + link 循環 + DSC → bcm_100g_dsc_cycle.sh │ ├── Multiphase_{high3,low3,comboA,comboB,Save}.txt # → blanton_multiphase_margin.sh │ ├── Multiphase_Netname # 38 條軌的 net 名稱（區塊順序對應） │ ├── 100G_Traffic.txt # ce0 VLAN 138 打流 → blanton_traffic_linespeed.sh @@ -201,6 +202,8 @@ Blanton_TTL_Script/ ├── port_prbs_monitor.sh # 100G uplink PRBS 測試（bcmcmd phy diag） │ # ⚠️ V1.1.9 起 A/B/C 內註解掉的呼叫已整段刪除，只剩手動執行 ├── blanton_ber.sh # 全機 SerDes BER 掃描（phydiag prbs / prbsstat） + │ # V1.2.2 起含 port 268（100G ce0），summary 每 unit 分 200G/100G/TOTAL + ├── bcm_100g_dsc_cycle.sh # 100G link down/up 循環 + 每輪 DSC 與 retimer TX/RX 讀回（未驗證） ├── bcm_dsc_scan.sh # 全機 SerDes DSC dump（phydiag dsc，~445 port × 2 unit） │ # 由 EN_TX_FIR 控制，A 與 C 各掃一次；單次約 10 分鐘 ├── bcm_dsc_parse.sh # DSC raw log → _lane.csv / _pm.csv（純 sh + awk） @@ -374,7 +377,7 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc | 模組 | 進入點 | 功能 | |------|--------|------| -| **Script A（Pre-test）** | `1_Blanton_Script_A.ttl` V1.2.1 | root 登入 → `date -s` 對時 → 清 job log → **收掉前一輪殘留**（`hw-test-session finish` → `bgctl reset --yes` → `bgctl stop --all` → `bgctl list` 確認清空）→ `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**（`wait_init.ttl`）→ DUT 清單（boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe）→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → **NFC 基線**（start → 30s → stop → report）→ 100G uplink 設定與狀態 → **BER 基線掃描**（依 `SWB_UNIT`，V1.2.1 新增）→ **一輪流量基線** → **SerDes DSC 基線**（`EN_TX_FIR`）→ `show uptime` | +| **Script A（Pre-test）** | `1_Blanton_Script_A.ttl` V1.2.2 | root 登入 → `date -s` 對時 → 清 job log → **收掉前一輪殘留**（`hw-test-session finish` → `bgctl reset --yes` → `bgctl stop --all` → `bgctl list` 確認清空）→ `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**（`wait_init.ttl`）→ DUT 清單（boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe）→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → **NFC 基線**（start → 30s → stop → report）→ 100G uplink 設定與狀態 → **BER 基線掃描**（依 `SWB_UNIT`，V1.2.1 新增）→ **一輪流量基線** → **SerDes DSC 基線**（`EN_TX_FIR`）→ `show uptime` | | **Script B（Stress）** | `2_Blanton_Script_B.ttl` | **V1.1.8 起 `mlucas-avx2 -cpu 0:15` 由 `systemd-run --unit=mlucas-amm`（`Type=exec`、`Restart=always`）啟動，不再是 shell 的 `&` 背景 job** + `bgctl run` 的 `memtester` / SSD / USB 壓力（v1.1.7 起 SSD/USB 的 `--runtime`/`--passes` 讀 `config.ttl` 的 `SSD_RUNTIME`/`SSD_COUNT`、`USB_RUNTIME`/`USB_COUNT`；BMC memtester 不帶次數 = 跑到被停） + BMC DDR（`bmc-manager run memtester`）→ BMC USB net test（探測 cdc_ncm 介面，`systemd-run` 掛 `BMC_USB_RUNTIME` 秒的 ping，V1.1.8 前寫死 4 小時）→ 啟動 mgmt ping 監控 → **啟動 NFC 輪詢** → `jobs` / `systemctl is-active mlucas-amm` / `bgctl list` → traffic（依 `SWB_UNIT`）→ `while 1` 每輪 PMON + `bgctl list`/`jobs`/`systemctl is-active mlucas-amm` + `sleep 60`（**V1.1.4 起 margin 不在迴圈內**；V1.2.0 起等待走 DUT 端 `sleep` 而非 macro 端 `pause`）| | **Script C（Post-test）** | `3_Blanton_Script_C.ttl` | `kill $(jobs -p)` + **`systemctl stop mlucas-amm`**（systemd unit 不是 shell job，`jobs -p` 抓不到）+ `bgctl stop --all` / `reset` + 停 BMC USB unit、mgmt ping 與 **NFC 輪詢** → PMON + dmesg → 7 組 AER + `ras-mc-ctl` → BMC USB `journalctl` **寫進 `log/bmc_usb_net.log`**（不再印在 console） → `ip -s link show eth0/eth1` → `cat` mlucas log（並 `cp` 進 job 目錄一起帶走）→ **NFC `report`** → traffic `stop` + `report` → **SerDes DSC 收尾**（`EN_TX_FIR`）→ **BER 掃描**（依 `SWB_UNIT`）→ NVMe 健康 → `show reboot-cause` → **margin 掃描**（`EN_Margin`，V1.1.4 由 Script B 移來）→ `uptime` → **job log 複製到 USB（帶時戳）** → `exit` + wait『Script done』→ `hw-test-session finish` | | **Script 4（Thermal / Safety）** | `4_Blanton_Script_thermal_safety.ttl` | **獨立腳本，不接 A→B→C。** Script A 式的環境建立（登入 / session / 風扇 / `wait_init` / DUT 清單 / BMC / PCIe 清除 / `pcie_bus` / PMON / dmesg）→ **滿載**（`mlucas-avx2 -cpu 0:15`，V1.1.8 起與 Script B 同樣掛成 `mlucas-amm` systemd unit + `blanton_tr518.sh start`）→ `while 1`：PMON + `TH6_SWB0/1_power_readback.sh` + `bgctl list` / `jobs` / `systemctl is-active mlucas-amm`，`pause 60`。**沒有收尾段** —— 中斷後 DUT 仍在 `port cd lb=mac` / `l2 learn off` / `test mode nr=yes`，且 CPU 仍滿載，要手動 `blanton_tr518.sh stop` + `systemctl stop mlucas-amm` | @@ -393,7 +396,8 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc | **BMC DDR 壓力** | `bmc_monitor_ddr.sh` | ⚠️ **已停用** —— V1.1.9 起 B/C 內的註解區塊已刪除，改由 `bgctl run bmc-manager run memtester` 取代。檔案保留未刪 | | **SerDes DSC 擷取** | `bcm_dsc_scan.sh [-u 0\|1] [-f ] [-F] [-n] [-C]` + `bcm_dsc_parse.sh -i -o ` | `bcmcmd -n -c "dsh -c 'phydiag dsc'"` 掃全清單（1 個 100G + 445 個 200G port）× 2 unit，raw log 寫進 `/log` 再 `cp` 到 `/host/hw-eval/jobs`。`-f` 支援 `%u` / `%t`；沒帶 `%u` 時自動插 `_u0`/`_u1`。成功與否**不看 `bcmcmd` 的 exit code**，改比對 `SERDES DISPLAY DIAG DATA END` 數量 + 實際迭代筆數，結尾印 `### DSC_SCAN_DONE rc=`。parse 端輸出 `_lane.csv`（每 lane：TXEQ/RXFFE/DFE/EYE/SNR）與 `_pm.csv`（每 pm_id：溫度/PLL/lane 數/統計）。**由 `EN_TX_FIR` 控制，預設關** | | **100G PRBS** | `port_prbs_monitor.sh {start [both\|a\|b]\|stop\|status\|report\|clear}` | 兩個 uplink 各一個 worker，可同時或單獨跑。setup（`lpmode off` → `prbs set` → `prbsstat STArt`）→ 週期 `prbs get`（判定）+ `prbsstat Ber`（只記錄）→ teardown（`STOp` + `prbs clear`）。輸出含 `PRBS OK!` 才算 PASS。**⚠️ V1.1.9 起 A/B/C 內的註解呼叫已刪除，只能手動跑** | -| **SerDes BER 掃描** | `blanton_ber.sh {init\|clear\|start\|stop\|report} [-u 0\|1\|all]` | 全機每個對接 port 的 PRBS BER：`prbs set p=3` + `prbs get`（確認每條 lane 都鎖上）→ `prbsstat STArt Interval=30` →（等一個 interval）→ `prbsstat Ber` → `STOp` → `prbs clear`。`report` 一列一條 lane，**BER < 1e-6 才 PASS**，最後跨 unit 的 summary 帶最差 lane。⚠️ PRBS 期間 link 顯示 DOWN、不過流量。V1.2.1 起 Script A 與 Script C 各跑一次（A 在流量之前、C 在流量報表之後），夾住 soak | +| **SerDes BER 掃描** | `blanton_ber.sh {init\|clear\|start\|stop\|report} [-u 0\|1\|all]` | 全機每個對接 port 的 PRBS BER：`prbs set p=3` + `prbs get`（確認每條 lane 都鎖上）→ `prbsstat STArt Interval=30` →（等一個 interval）→ `prbsstat Ber` → `STOp` → `prbs clear`。`report` 一列一條 lane，**BER < 1e-6 才 PASS**，最後跨 unit 的 summary 帶最差 lane。⚠️ PRBS 期間 link 顯示 DOWN、不過流量。V1.2.1 起 Script A 與 Script C 各跑一次（A 在流量之前、C 在流量報表之後），夾住 soak。**V1.2.2（腳本 v1.1.0）起清單納入 port 268（100G uplink ce0）**，summary 每個 unit 拆成 200G ports / 100G p268 / TOTAL 三列；某組零 lane 算 FAIL | +| **100G link 循環 + DSC** | `bcm_100g_dsc_cycle.sh [-d] [-T] [-P] [-n]` | tuning（`docker pause gbsyncd*` → `serdes-tx-set sys 0..7` → 每 lane `tx_taps set`）一次，之後每輪：`port 268 en=0` → 輪詢 `bcmcmd ps 268` 等 `!ena` → `en=1` → 等 `up` → 兩個 unit 的 `phydiag 268 dsc` + `serdes-tx-get`/`serdes-rx-get sys 0..7`。log 一次執行一份（`~/dsc_log/`），每輪一段 `### TEST CYCLE n/N`，結尾 `### DSC_CYCLE_DONE rc=`。**全 ASCII 輸出**（DUT console 會把 UTF-8 弄成亂碼）。⚠️ 尚未上機驗證 | | **TR518 負載** | `blanton_tr518.sh {start\|stop} [-u 0\|1\|all]` | `port cd lb=mac` → `l2 learn off` → `test mode nr=yes` → `tr 518 ... testphase=1`，然後 settle `TR_SETTLE_SEC`（120s）再讓人量功耗。`stop` 只送 `port cd lb=none`（照 `TR518.txt`），L2 學習與 test mode **不會還原**，`TR_RESTORE_*=1` 才會 | | **PDB / CPB brick** | `pwr_brick.sh show [-r]` | 5 顆 brick 的 Vin / Iin / Vout / Iout / Temp（SWB0 `0x62`/`0x63`、SWB1 `0x60`/`0x61`、CPB `0x26`）。四顆 ADPM12200 **每個量各自一個 PMBus PAGE**（Vin 9、Iin 10、Vout 2、Iout 14、Temp 18），讀之前一定要先寫 PAGE —— `blanton_pwr_data.sh` 完全沒寫，所以整張 PDB 表都是 NA。換算用手冊 DIRECT 公式 `X=(1/m)(Y×10⁻ᴿ−b)`。每個值重讀到通過三道檢查為止（非 `0xFFFF`、非 `0x0000`、落在合理範圍）—— 這條匯流排**只壞高位元組**，`11.98 V` 會變成 `32.00 V`，全一過濾擋不住 | | **SWB 軌功耗** | `TH6_SWB{0,1}_power_readback.sh` | 讀 `show platform voltage/current` **各一次**，19 條軌 V×I 與總功耗。舊版每軌各讀一次（38 次呼叫、每軌不同時刻），總和是不同瞬間的加總 | @@ -804,7 +808,7 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值， - [ ] `CHANGELOG.md`：延續三支腳本頂端的 Version History，集中成一份 - [ ] 三支腳本的 header 版號與 `CHANGELOG.md` 對齊，`config.ttl` 加 `script_version` 並寫進 log 開頭 - [ ] `docs/TTL_Script_Blanton_Status_*.xlsx` 更新為全 OK，或把剩餘項目移進 Future Extensions -- [x] 打 git tag，push `nas` + `gitea`（已至 `V1.2.1`；`publish/` 依 Script A 檔頭版號打包） +- [x] 打 git tag，push `nas` + `gitea`（已至 `V1.2.2`；`publish/` 依 Script A 檔頭版號打包） **驗收條件：** 從 zip 解壓到一台乾淨的測試 NB，照 README 操作，不看原始碼就能完成一輪 A→B→C 並產出 summary；`publish/` 內容與 git tag 一致。 @@ -851,6 +855,10 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值， 開啟後 Script A 與 Script C 各掃一次、整輪多約 20 分鐘；只要單邊資料可以在腳本裡改成 `-u 0`。 掃描本身是唯讀（`phydiag ... dsc` 不改 SerDes 設定），但**它與 `blanton_ber.sh` / `blanton_traffic_linespeed` / `blanton_tr518.sh` 一樣都在動同一批 port，不要同時跑**。 +- **`bcm_100g_dsc_cycle.sh` 尚未上機驗證**：`port en=0/1` 之後 link 多久會動、以及 + `docker pause gbsyncd*` 期間的影響，都要上機才知道。等待是讀 SDK（`bcmcmd ps `）而不是 + `show interfaces status`，所以理論上不受 gbsyncd 暫停影響，但沒實測過。 + 結束時（含 Ctrl-C）會自動 `docker unpause`，`RESUME_GBSYNCD=0` 可關掉。 - **Script 5（Reboot Cycle）尚未上機驗證**：`5_Blanton_Script_Reboot_Cycle.ttl` 隨 V1.1.7 交付， 但還沒在真 DUT 上跑過一輪完整循環。兩個要現場確認的點：`wait "Booting in blind mode"` 是否 真的在每一版開機訊息裡出現（沒出現就會永遠等下去，而且沒有 `timeout` 保護）；以及每輪重跑 @@ -914,6 +922,14 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值， - **margin 在 soak 期間不再取樣**（V1.1.4）：只剩 Script A 的起始值與 Script C 的結束值。 漂移發生在中間就看不到了。若要恢復，比較好的做法是在 B 的迴圈內每 N 輪才掃一次， 而不是每輪都掃。 +- ✅ **`blanton_ber.sh` 不再把 lane 的量測結果當成 shell 錯誤**（v1.1.1，2026-09-15）： + `prbs get` 每條 lane 一行，沒鎖上的會印 `PRBS Error!`，而錯誤樣式是拿 + `grep -i 'ERROR|Invalid|...'` 掃整份輸出的 —— 於是正常的 init 會報「the BCM shell rejected」， + 而且提早 return，**把「哪幾條 lane 沒鎖」這個真正有用的資訊吞掉**。現在 per-lane 行排除在 + 掃描外、改成大小寫敏感、真正的拒絕會把該行印出來。 + ⚠️ **這類「完整性檢查自己看不到失敗」的錯在這個 repo 已經出現三次** + （`bcm_dsc_scan.sh` v1.3.0 數 marker、`blanton_ber.sh` v1.1.1、以及 power cycle 報告的 + ping 解析把 100% 丟包當成「沒資料」）。寫檢查時要問：**失敗長什麼樣，這段程式碼看得到嗎？** - **`blanton_ber.sh` 的 `sleep 30` 與 `BER_INTERVAL` 是兩處要同步改的數字**：Script A（V1.2.1 起） 與 Script C 都寫死 `sendln "sleep 30"`，腳本預設 `BER_INTERVAL=30`。改一邊沒改另一邊，report 會 讀到還沒累積完一個 interval 的數字，而且不會有任何警告。現在有**三處**要一起改。 diff --git a/CLAUDE.md b/CLAUDE.md index 4e9893f..bd9128d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,7 +9,7 @@ DUT 端的取數能力由一組 **bash 工具庫** 提供（CB FPGA `pcimem` → 完整架構見 [ARCHITECTURE.md](ARCHITECTURE.md)。 -**目前狀態（2026-09-08）**：Script A 為 **V1.2.1**（A 集中記錄所有 TTL 的變更）。涵蓋 +**目前狀態（2026-09-15）**：Script A 為 **V1.2.2**（A 集中記錄所有 TTL 的變更）。涵蓋 PCIe AER / DDR EDAC / PMON+TPM / 電壓 margin（9 顆 LTC2980）/ SWB loopback 線速流量 / 10G+1G 管理網路 ping / BMC DDR 與 USB / NVMe 健康，壓力層走平台的 `bgctl`。 switch unit 數量由 `config.ttl` 的 `SWB_UNIT0` / `SWB_UNIT1` 宣告，風扇轉速由 `FAN_SPEED` 指定 @@ -25,6 +25,7 @@ Script B 的 soak 迴圈每輪印 `bgctl list` + `jobs` + `systemctl is-active m `mlucas-amm` transient unit（`Restart=always`），不再是 shell 的 `&` 背景 job —— 掉線也不會停， Script C 用 `systemctl stop mlucas-amm` 收掉並把 log 複製到隨身碟。 V1.1.0 新增全機 SerDes BER 掃描（`blanton_ber.sh`，依 `SWB_UNIT` 分支）；**V1.2.1 起 Script A 也接了一次**， +**V1.2.2 起清單納入 port 268（100G uplink，ce0）**，summary 每個 unit 拆成「200G ports / 100G p268 / TOTAL」三列； 在第一輪流量之前取基線，與 Script C 流量報表之後的收尾掃描一起夾住 soak（同 DSC 的做法）。另有 TR518 負載（`blanton_tr518.sh`，**TTL 尚未接**）、SWB 軌功耗（`TH6_SWB{0,1}_power_readback.sh`）， 線速流量也納入 100G `ce0`（VLAN 138，靠 ingress mirror 維持循環）。 @@ -152,6 +153,8 @@ sleep 30 # 至少等一個 interval 再 rep ./Blanton_Script/blanton_ber.sh report # 每 lane 一列，BER < 1e-6 才 PASS，最後 summary ./Blanton_Script/blanton_ber.sh stop # prbsstat STOp ./Blanton_Script/blanton_ber.sh clear # prbs clear（會解除 PRBS，要重測得再 init） +# summary 每個 unit 三列：200G ports / 100G p268 / TOTAL（兩個 unit 各有自己的 ce0(268)） +# ⚠️ 某一組「一條 lane 都沒回報」算 FAIL 不算 PASS —— 待測的 port 沉默不是通過 # -u 0 / -u 1 單邊；-c 1e-9 換門檻；-F 只列失敗；-f 離線重解 # ⚠️ 不能跟 blanton_traffic_linespeed / blanton_tr518 同時跑（都會接管同一批 port） @@ -171,6 +174,17 @@ sh ~/Blanton_Script/bcm_dsc_parse.sh -i blanton_dsc_ScriptA_u0.log -o u0 # 純 sh + awk（mawk 相容），欄位靠括號群組順序抓，SDK 改排版不會壞； # 群組數 != 9 的列標成 UNPARSED 印到 stderr，不會安靜解錯 +# ── 100G link down/up 循環 + DSC（TTL 尚未接）──────── +./Blanton_Script/bcm_100g_dsc_cycle.sh # 印說明 +./Blanton_Script/bcm_100g_dsc_cycle.sh 200 # 跑 200 輪（-c 200 亦可） +./Blanton_Script/bcm_100g_dsc_cycle.sh 200 -d # 每行指令以灰色 [CMD] 印出來 +./Blanton_Script/bcm_100g_dsc_cycle.sh 20 -T # 跳過 tuning +./Blanton_Script/bcm_100g_dsc_cycle.sh 1 -n # dry-run，不碰硬體也不 sleep +# log: ~/dsc_log/100G_dsc_cycle_test_.log，每輪一段 '### TEST CYCLE n/N' +# 每輪收：兩個 unit 的 phydiag dsc + serdes-tx-get / serdes-rx-get sys 0..7 +# 等 link 是輪詢 'bcmcmd ps 268'（每 -p 秒，上限 -w 秒），不是 show interfaces status +# ⚠️ 尚未上機驗證；與 blanton_ber / blanton_traffic_linespeed / blanton_tr518 互斥 + # ── TR518 負載 + 量功耗（TTL 尚未接）───────────────── ./Blanton_Script/blanton_tr518.sh start # 兩 unit，跑完 settle 120s ./Blanton_Script/TH6_SWB0_power_readback.sh # 19 條軌 V×I + TOTAL POWER @@ -354,6 +368,11 @@ Tera Term 連 COM port (115200-8-N-1) - ⚠️ **兩張管理網卡同網段會 ARP flux**：`mgmt_ping_monitor.sh` 讓 eth0/eth1 同時 up，同一個 /24 下 對端 ARP 可能由任一張卡回應，回包落在沒送封包的那張。`ARP_STRICT=1` 擋掉；判讀時看 `RESULT` 行的 `nic_tx`/`nic_rx`（該卡自己的 packet delta）—— **ping 成功但 `nic_tx` 接近 0 = 封包從另一張卡出去的** +- ⚠️ **`prbs get` 印的每一行是資料，不是 shell 的錯誤訊息**：沒鎖上的 lane 會印 `PRBS Error!`， + 而 `blanton_ber.sh` 從 v1.0.0 就用 `grep -i 'ERROR|Invalid|...'` 掃**整份輸出**判斷「指令被拒絕」—— + 於是**跑得好好的 init 會報 `the BCM shell rejected 'phydiag ... prbs get'`**，而且因為提早 return， + 那段「數 PRBS OK / 總數」的檢查根本沒跑，**你看到一個假錯誤，卻看不到哪幾條 lane 真的沒鎖**。 + v1.1.1 已把 per-lane 行排除在錯誤掃描外、改成大小寫敏感，並把真正的拒絕訊息印出來 - ⚠️ **BER 的 `clear` 才是把 lane 放回正常運作的動作**：V1.2.1 起 Script A 在第一輪流量**之前**跑 BER， 而 `blanton_ber.sh clear` 與緊接著的 `blanton_traffic_linespeed ps` 之間**沒有任何 settle**。 Script C 是流量報表之後才跑 BER，沒有這個順序問題。第一次上機時比對兩邊的 traffic report —— diff --git a/docs/release-notes/v1.2.2.md b/docs/release-notes/v1.2.2.md new file mode 100644 index 0000000..e3ebe3c --- /dev/null +++ b/docs/release-notes/v1.2.2.md @@ -0,0 +1,63 @@ +# v1.2.2 — The 100G uplink joins the BER sweep, and a boot cycle that can pull the power + +## ✨ New features + +**The BER sweep now measures port 268, the 100G uplink** +* The port list ran `256-259` straight to `270-308`, so `ce0(268)` — the 100G uplink — was never in it. It is now its own group alongside the 200G ports. +* The summary gives **each unit three rows**: `200G ports`, `100G p268`, and `TOTAL`. Unit 0 and unit 1 each have their own `ce0(268)`, and a marginal uplink lane averaged into 432 others is invisible; on its own row it is not. +* A group that reports no lane at all now counts as a **failure**, not a pass. Silence on the port under test is the easiest way to ship a green report that measured nothing. + +**Script 5 is now a boot cycle, warm or cold** +* `5_Blanton_Script_Reboot_Cycle.ttl` → `5_Blanton_Script_Boot_Cycle.ttl`, with `EN_Cold_Warm_boot` in `config.ttl` choosing: `1` = `reboot` (warm), `2` = a cold cycle by writing the CB FPGA, `0` = neither. +* The log file and the per-round banner are named after the kind of boot: `Blanton_Reboot_Cycle_.log` / `Reboot Cycle : N`, or `Blanton_Power_Cycle_.log` / `Power Cycle : N`. +* Both stress logs are removed before each run. `qfx5252-stress-*` appends to its `--log` and the macro `cat`s the whole thing, so every cycle was printing back everything the earlier cycles had written — 289 lines at cycle 1 and 36,555 by cycle 199. At roughly 6.2 ms a line on a 115200 console that accounted for **+212 s of the +214 s the cycle grew** over a 199-cycle run. The DUT was never getting slower. + +**`bcm_100g_dsc_cycle.sh` — 100G link down/up cycling with SerDes capture** +* Tuning once, then N cycles of `port 268 en=0` → wait → `en=1` → wait → `phydiag 268 dsc` on both units → retimer `serdes-tx-get` / `serdes-rx-get` for `sys 0..7`. One log per run under `~/dsc_log/`, split by `### TEST CYCLE n/N`. +* Waiting is a **poll of `bcmcmd ps 268`**, not `show interfaces status`. `port en=` changes the SDK, and the SONiC table only follows once gbsyncd has synced it — and this test pauses gbsyncd. +* `-d` echoes every command as a grey `[CMD]` line before running it; `-n` is a dry run that touches nothing; the cycle count can be a bare argument (`./bcm_100g_dsc_cycle.sh 200`). + +## 🐛 Bug fixes + +**`blanton_ber.sh init` reported a hard error on runs that were working** +* `[ERR ] unit 1: the BCM shell rejected 'phydiag ... prbs get'` — except the command was fine. `prbs get` prints one line per lane, a lane that has not locked says `PRBS Error!`, and the rejection check grepped the **whole** output case-insensitively for `ERROR|Invalid|...`. +* Worse than the false alarm: the function returned early, so the lock count that was meant to judge those very lanes never ran. The operator saw an error they could not act on, and did **not** see which lanes had failed to lock. That check has been there since v1.0.0. +* Per-lane result lines are now excluded, the match is case-sensitive, and a genuine rejection prints the offending line. + +## 🔧 Changes + +* **Script A takes a BER baseline before its first traffic run** (v1.2.1), so together with Script C's closing sweep the pair brackets the soak. +* **`EN_TX_FIR`** gates the SerDes DSC scans in Script A and Script C. Default `0`: a full scan is ~10 min and both scripts do one. + +## 📦 Downloads + +| File | Contents | +|---|---| +| `Script_ABC_Blanton_V1.2.2.zip` | The full Tera Term working directory, including `Blanton_Script/` to copy onto the DUT | + +```bash +chmod +x ~/Blanton_Script/*.sh +grep -rlU $'\r' ~/Blanton_Script # expect no output +ls ~/hammer/tools/amd/mlucas-avx2 # must already be there +``` + +## ⚠️ Before you run + +* **Replace `config.ttl` with the macros.** `EN_Cold_Warm_boot` is new, and Tera Term aborts on an undefined variable rather than treating it as 0. +* **Do not run the boot cycle with `EN_Cold_Warm_boot = 0`.** Nothing reboots, the `wait "Booting in blind mode"` underneath has no timeout, and since v1.2.2 there is no `logopen` either — so it hangs *and* leaves no log showing where. `0` is the value for the A → B → C flow, where that macro is not used. +* **The cold path writes a raw FPGA register.** `0x480` is `ICB_CTRL_REG`; the register map carries no bit definitions for it and `0xD0000` is not documented anywhere in this repo. The BDF `0000:05:00.0` is hard-coded while `blanton_fpga_pcimem.sh` auto-detects it because it moves per unit — **check the BDF on every board before running the cold path**. +* **Anything that parses these logs must accept both banners**, `Reboot Cycle : N` and `Power Cycle : N`. A parser that greps only the first finds zero cycles in a power-cycle log and reports nothing without complaining. +* **`bcm_100g_dsc_cycle.sh` has not been run on hardware.** It pauses `gbsyncd0/1` for tuning and unpauses them on exit, Ctrl-C included; `-P` leaves them alone. +* **`sleep 30` and `BER_INTERVAL` now live in three places** — Script A, Script C and `blanton_ber.sh`. +* **BER, the DSC cycle, `blanton_traffic_linespeed` and `blanton_tr518.sh` cannot overlap.** They all take the same ports. +* **Script 4 has no teardown**: finish it with `blanton_tr518.sh stop` **and** `systemctl stop mlucas-amm`. +* **`blanton_multiphase_margin.sh save` is permanent**, and must not be mixed with `margin.sh`. +* **`READ_VIN` and `READ_IIN` are still not in the ADPM12200 coefficient table.** +* **`FAN_SPEED` is 100.** + +## 🔗 Links + +* [ARCHITECTURE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/ARCHITECTURE.md) — test flow, data models, constraints +* [CLAUDE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/CLAUDE.md) — commands and the bench gotchas + +**Full changelog:** [V1.2.1...V1.2.2](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/compare/V1.2.1...V1.2.2) diff --git a/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl b/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl index b576942..3bc07d1 100644 --- a/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl +++ b/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl @@ -1,7 +1,7 @@ ; ============================================================================= ; Script A for Blanton ; Version : V1.2.2 -; Date : 2026-09-09 +; Date : 2026-09-15 ; Author : ETWen ; ============================================================================= ; Version History: @@ -562,7 +562,7 @@ ; concerned: once here and once inside ScriptC. Neither ; can overlap blanton_traffic_linespeed or ; blanton_tr518.sh - they all take over the same ports -; V1.2.2 2026-09-09 +; V1.2.2 2026-09-15 ; 5_.._Boot_Cycle.ttl Renamed from 5_Blanton_Script_Reboot_Cycle.ttl ; It does warm and cold now, so 'Reboot' was too narrow. ; Same macro otherwise - git records it as a rename @@ -634,6 +634,64 @@ ; 0 = normal test (A/B/C), 1 = reboot cycle, 2 = power ; cycle. Default 0 so that opening the shared config cannot ; reboot anyone's DUT by accident +; Blanton_Script/blanton_ber.sh v1.1.0 Add port 268, the 100G uplink +; The port list jumps 256-259 -> 270-308, so 268 (ce0) was +; never measured. It is now its own group: BER_PORTS_200G +; and BER_PORTS_100G, joined for the phydiag command but +; reported apart +; The summary gives each unit three rows - 200G ports / +; 100G p268 / TOTAL - because unit 0 and unit 1 each have +; their own ce0(268). A bad uplink lane averaged into 432 +; others is invisible; on its own row it is not +; TOTAL is not a hand-added sum: every lane is folded into +; its own group and into ALL separately, so the parts +; cannot drift from the whole +; !! a group that reports no lane at all is a FAILURE, not +; a pass. Silence on the port under test is the easiest +; way to ship a green report that measured nothing +; Blanton_Script/blanton_ber.sh v1.1.1 Stop reading lane results as shell errors +; init kept answering "the BCM shell rejected 'phydiag ... +; prbs get'" on runs that were working. The check grepped +; the WHOLE output case-insensitively for +; ERROR|Invalid|..., and 'prbs get' prints one line per +; lane - a lane that had not locked says PRBS Error!, which +; matched +; So a good run reported a hard error, and worse, the +; function returned before the lock-count check that was +; supposed to judge those very lanes. The operator saw a +; failure and did NOT see which lanes were unlocked +; Per-lane lines are now excluded from the check, the match +; is case-sensitive, and a genuine rejection prints the +; offending line so it can be read +; !! this was in the file from v1.0.0. Any earlier init +; that printed that error was probably fine, and any +; earlier init that passed said nothing about lanes that +; had failed to lock +; Blanton_Script/bcm_100g_dsc_cycle.sh Created v1.2.0 - 100G link down/up cycle with DSC capture +; Tuning once (gbsyncd pause, serdes-tx-set sys 0..7, +; tx_taps per lane), then N cycles of: port en=0, wait, +; port en=1, wait, phydiag dsc on both units, serdes-tx-get +; and serdes-rx-get sys 0..7. One log per run under +; ~/dsc_log, split by '### TEST CYCLE n/N' +; Waiting polls 'bcmcmd ps ' every -p seconds up to +; -w, and reads the state from the row whose first field +; ends in (). Asking the SDK matters here: 'port en=' +; changes the SDK, while 'show interfaces status' only +; follows once gbsyncd has synced it - and this test pauses +; gbsyncd +; -d prints every command as [CMD] in grey before running +; it, CMD_DELAY paces them, -n is a dry run that touches +; nothing and sleeps not at all +; !! all output is ASCII English on purpose. The DUT serial +; console mangles UTF-8: a Chinese error message came +; back as "ERROR: -c ? 1", which is worse than no +; message +; !! NOT verified on hardware +; tools/100G_dsc_cycle.txt Created - the hand-typed command list behind that script +; Same role as BER_Test.txt and TR518.txt: the bench +; version of the commands, kept so the .sh can be checked +; against what a person would actually type. Change one and +; change the other ; ============================================================================= include "config.ttl"