diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 48603c2..a9309c5 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -207,6 +207,7 @@ Blanton_TTL_Script/ ├── blanton_ber.sh # 全機 SerDes BER 掃描（phydiag prbs / prbsstat） │ # V1.2.2 起含 port 268（100G ce0），summary 每 unit 分 200G/100G/TOTAL ├── blanton_ber_debug.sh # BER 原始輸出（p=p31、sleep 40、-u 0|1），不判定 + ├── blanton_fdr.sh # FDR 統計（phydiag fdrstat start/clear/Counters/STOp），A/B/C 夾住流量 ├── bcm_100g_dsc_cycle.sh # 100G link down/up 循環 + 每輪 DSC 與 retimer TX/RX 讀回（未驗證） ├── bcm_dsc_scan.sh # 全機 SerDes DSC dump（phydiag dsc，~445 port × 2 unit） │ # 由 EN_TX_FIR 控制，A 與 C 各掃一次；單次約 10 分鐘 @@ -387,7 +388,7 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc | 模組 | 進入點 | 功能 | |------|--------|------| -| **Script A（Pre-test）** | `1_Blanton_Script_A.ttl` V1.2.5 | root 登入 → `date -s` 對時 → 清 job log → **收掉前一輪殘留**（`hw-test-session finish` → `bgctl reset --yes` → `bgctl stop --all` → `bgctl list` 確認清空）→ `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**（`wait_init.ttl`）→ DUT 清單（boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe）→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → **NFC 基線**（start → 30s → stop → report）→ 100G uplink 設定與狀態 → **BER 基線掃描**（依 `SWB_UNIT`，V1.2.1 新增）→ **一輪流量基線** → **SerDes DSC 基線**（`EN_TX_FIR`）→ `show uptime` | +| **Script A（Pre-test）** | `1_Blanton_Script_A.ttl` V1.2.6 | root 登入 → `date -s` 對時 → 清 job log → **收掉前一輪殘留**（`hw-test-session finish` → `bgctl reset --yes` → `bgctl stop --all` → `bgctl list` 確認清空）→ `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**（`wait_init.ttl`）→ DUT 清單（boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe）→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → **NFC 基線**（start → 30s → stop → report）→ 100G uplink 設定與狀態 → **`unused_lanes.sh off`**（V1.2.6）→ **BER 基線掃描**（依 `SWB_UNIT`，V1.2.1 新增）→ **一輪流量基線**（V1.2.6 起兩 unit 時夾 FDR，sleep 40） → **SerDes DSC 基線**（`EN_TX_FIR`）→ `show uptime` | | **Script B（Stress）** | `2_Blanton_Script_B.ttl` | **V1.1.8 起 `mlucas-avx2 -cpu 0:15` 由 `systemd-run --unit=mlucas-amm`（`Type=exec`、`Restart=always`）啟動，不再是 shell 的 `&` 背景 job** + `bgctl run` 的 `memtester` / SSD / USB 壓力（v1.1.7 起 SSD/USB 的 `--runtime`/`--passes` 讀 `config.ttl` 的 `SSD_RUNTIME`/`SSD_COUNT`、`USB_RUNTIME`/`USB_COUNT`；BMC memtester 不帶次數 = 跑到被停） + BMC DDR（`bmc-manager run memtester`）→ BMC USB net test（探測 cdc_ncm 介面，`systemd-run` 掛 `BMC_USB_RUNTIME` 秒的 ping，V1.1.8 前寫死 4 小時）→ 啟動 mgmt ping 監控 → **啟動 NFC 輪詢** → `jobs` / `systemctl is-active mlucas-amm` / `bgctl list` → traffic（依 `SWB_UNIT`）→ `while 1` 每輪 PMON + `bgctl list`/`jobs`/`systemctl is-active mlucas-amm` + `sleep 60`（**V1.1.4 起 margin 不在迴圈內**；V1.2.0 起等待走 DUT 端 `sleep` 而非 macro 端 `pause`）| | **Script C（Post-test）** | `3_Blanton_Script_C.ttl` | `kill $(jobs -p)` + **`systemctl stop mlucas-amm`**（systemd unit 不是 shell job，`jobs -p` 抓不到）+ `bgctl stop --all` / `reset` + 停 BMC USB unit、mgmt ping 與 **NFC 輪詢** → PMON + dmesg → 7 組 AER + `ras-mc-ctl` → BMC USB `journalctl` **寫進 `log/bmc_usb_net.log`**（不再印在 console） → `ip -s link show eth0/eth1` → `cat` mlucas log（並 `cp` 進 job 目錄一起帶走）→ **NFC `report`** → traffic `stop` + `report` → **SerDes DSC 收尾**（`EN_TX_FIR`）→ **BER 掃描**（依 `SWB_UNIT`）→ NVMe 健康 → `show reboot-cause` → **margin 掃描**（`EN_Margin`，V1.1.4 由 Script B 移來）→ `uptime` → **job log 複製到 USB（帶時戳）** → `exit` + wait『Script done』→ `hw-test-session finish` | | **Script 4（Thermal / Safety）** | `4_Blanton_Script_thermal_safety.ttl` | **獨立腳本，不接 A→B→C。** Script A 式的環境建立（登入 / session / 風扇 / `wait_init` / DUT 清單 / BMC / PCIe 清除 / `pcie_bus` / PMON / dmesg）→ **滿載**（`mlucas-avx2 -cpu 0:15`，V1.1.8 起與 Script B 同樣掛成 `mlucas-amm` systemd unit + `blanton_tr518.sh start`）→ `while 1`：PMON + `TH6_SWB0/1_power_readback.sh` + `avt_pmpower.sh`（CPU 封裝功耗，V1.2.3）+ `bgctl list` / `jobs` / `systemctl is-active mlucas-amm`，`pause 60`。**沒有收尾段** —— 中斷後 DUT 仍在 `port cd lb=mac` / `l2 learn off` / `test mode nr=yes`，且 CPU 仍滿載，要手動 `blanton_tr518.sh stop` + `systemctl stop mlucas-amm` | @@ -818,7 +819,7 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值， - [ ] `CHANGELOG.md`：延續三支腳本頂端的 Version History，集中成一份 - [ ] 三支腳本的 header 版號與 `CHANGELOG.md` 對齊，`config.ttl` 加 `script_version` 並寫進 log 開頭 - [ ] `docs/TTL_Script_Blanton_Status_*.xlsx` 更新為全 OK，或把剩餘項目移進 Future Extensions -- [x] 打 git tag，push `nas` + `gitea`（已至 `V1.2.5`；`publish/` 依 Script A 檔頭版號打包） +- [x] 打 git tag，push `nas` + `gitea`（已至 `V1.2.6`；`publish/` 依 Script A 檔頭版號打包） **驗收條件：** 從 zip 解壓到一台乾淨的測試 NB，照 README 操作，不看原始碼就能完成一輪 A→B→C 並產出 summary；`publish/` 內容與 git tag 一致。 diff --git a/CLAUDE.md b/CLAUDE.md index b0e2c19..ec22a58 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,7 +9,7 @@ DUT 端的取數能力由一組 **bash 工具庫** 提供（CB FPGA `pcimem` → 完整架構見 [ARCHITECTURE.md](ARCHITECTURE.md)。 -**目前狀態（2026-10-21）**：Script A 為 **V1.2.5**（A 集中記錄所有 TTL 的變更）。涵蓋 +**目前狀態（2026-10-21）**：Script A 為 **V1.2.6**（A 集中記錄所有 TTL 的變更）。涵蓋 PCIe AER / DDR EDAC / PMON+TPM / 電壓 margin（9 顆 LTC2980）/ SWB loopback 線速流量 / 10G+1G 管理網路 ping / BMC DDR 與 USB / NVMe 健康，壓力層走平台的 `bgctl`。 switch unit 數量由 `config.ttl` 的 `SWB_UNIT0` / `SWB_UNIT1` 宣告，風扇轉速由 `FAN_SPEED` 指定 @@ -43,11 +43,14 @@ NFC tag 讀取由 `nfc_polling.sh` 背景輪詢（A 做基線、B 起、C 停並 ⚠️ Script 4 迴圈原本就自己讀一次，現在**每輪讀兩次**。V1.2.5 另有 `blanton_ber_debug.sh`（BER 原始輸出、`p=p31`、`sleep 40`）、 `retimer_lanes.sh` / `unused_lanes.sh`（把沒接線的 retimer lane / TH6 port 關 TX），以及 `5_1_Blanton_Script_Boot_Cycle_Fast.ttl`（Script 5 拿掉流量那段，只留 `ps`）。 +**V1.2.6 起 FDR 統計（`blanton_fdr.sh`，`phydiag fdrstat`）接進流量**：A 包住流量基線（sleep 15→40）、 +B 在 soak 前 start、C 收尾 rdcnt + stop，**只在兩個 unit 都開時才跑**。Script A 另在 BER 前先 `unused_lanes.sh off`， +⚠️ **沒有任何地方再 `on` 回來**。 以及 **`5_Blanton_Script_Boot_Cycle.ttl`**（重開機循環，**獨立跑不接 A→B→C**：登入 → 風扇 → Check Pt1~Pt13 → `reboot` → `wait "Booting in blind mode"`，`cnt_cycle` 每輪遞增，`while 1` 無止境， 要停就關 Tera Term 的 macro）。**尚未上機驗證**。 流程圖：`docs/Blanton_Test_Flow.drawio`。 -最後一次發布是 **V1.2.5**；SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。 +最後一次發布是 **V1.2.6**；SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。 margin 的 `VNOM` 已於 2026-08-30 改為 Blanton 實測中位數（144 條軌中 86 條上調 0.8~2.5%）。 交付用 `./publish/publish.sh` 打包。 @@ -188,7 +191,14 @@ sleep 30 # 至少等一個 interval 再 rep # 清單同 tools/BER_Test.txt，不含 268；BER_PORTS="...,268" 可加 # ⚠️ 這支是 p=p31（已上機驗證），blanton_ber.sh 仍是 p=3 —— 兩者量的碼型可能不同 -# ── 沒接線的 retimer lane / TH6 port 關 TX（V1.2.5，TTL 未接）── +# ── FDR 統計（V1.2.6，A/B/C 已接，只在 SWB_UNIT0=SWB_UNIT1=1）── +~/Blanton_Script/blanton_fdr.sh start # fdrstat STArt bin_group=both Interval=30，unit 0 → 1 +~/Blanton_Script/blanton_fdr.sh clear # fdrstat clear +~/Blanton_Script/blanton_fdr.sh rdcnt # fdrstat Counters（Interval 結算後才讀，A 用 sleep 40） +~/Blanton_Script/blanton_fdr.sh stop # fdrstat STOp ⚠️ 關鍵字是類推的，未上機驗證 +# -u 0|1 單邊；-i 改 Interval；-n dry-run。ports 預設 '1-259,270-565'（FDR_PORTS 可改） + +# ── 沒接線的 retimer lane / TH6 port 關 TX（V1.2.5；V1.2.6 起 Script A 會 off，沒人 on 回來）── ./Blanton_Script/unused_lanes.sh status # 兩 unit 全部列出 rx_lock / tx_off ./Blanton_Script/unused_lanes.sh off [--asic 0] [--th6] # 預設 TX squelch；--th6 連 retimer 後面的 TH6 port 一起關 ./Blanton_Script/unused_lanes.sh on diff --git a/docs/release-notes/v1.2.6.md b/docs/release-notes/v1.2.6.md new file mode 100644 index 0000000..3de1901 --- /dev/null +++ b/docs/release-notes/v1.2.6.md @@ -0,0 +1,46 @@ +# v1.2.6 — FDR statistics around the traffic run, and unused lanes turned off before the baseline + +## ✨ New features + +**`blanton_fdr.sh` — FDR statistics through `phydiag fdrstat`** +* Four sub-commands, each run on unit 0 then unit 1, over ports `'1-259,270-565'`: + `start` (`fdrstat STArt bin_group=both Interval=30`), `clear` (`fdrstat clear`), `rdcnt` (`fdrstat Counters`), `stop` (`fdrstat STOp`). +* Output is printed as is, under a `### [time] unit N: ...` header — no parsing, no PASS/FAIL. +* `-u 0` / `-u 1` for one switch unit, `-i ` for the interval, `-n` for a dry run. `FDR_PORTS` overrides the port list. + +**FDR runs alongside the traffic test in Script A, B and C** +* **Script A** wraps its traffic baseline: `fdr start` after `init`, `fdr clear` after `clear`, `fdr rdcnt` after the run, `fdr stop` after the traffic stop. The traffic run is now **40 s instead of 15 s**, so the counters are read after the first 30 s interval has been published — the same rule as `prbsstat Ber`. +* **Script B** starts and clears FDR with the traffic setup, before the soak loop — the counters run for the whole soak. +* **Script C** reads them (`rdcnt`) before stopping the traffic and stops FDR after it — the closing read of the counters Script B started. + +**Script A turns off the unused lanes before the BER baseline** +* `unused_lanes.sh off` — TX squelch on the unused retimer lanes and TSC31 (`cd112-cd115`) disabled on the TH6, on both switch boards. + +## 📦 Downloads + +| File | Contents | +|---|---| +| `Script_ABC_Blanton_V1.2.6.zip` | The full Tera Term working directory, including `Blanton_Script/` to copy onto the DUT | + +```bash +chmod +x ~/Blanton_Script/*.sh +grep -rlU $'\r' ~/Blanton_Script # expect no output +``` + +## ⚠️ Before you run + +* **`chmod +x` after deploying.** `blanton_fdr.sh`, `unused_lanes.sh` and `retimer_lanes.sh` are all called by path; `unused_lanes.sh` also refuses to start if `retimer_lanes.sh` is not executable. +* **`blanton_fdr.sh stop` sends `fdrstat STOp` — that keyword is inferred from `prbsstat STOp` and has not been checked on the bench.** `bcmcmd` exits 0 even when the BCM shell rejects a command, so if it is wrong, FDR keeps running and only the log shows it. Check the first run. +* **FDR only runs when both switch units are enabled** (`SWB_UNIT0 = SWB_UNIT1 = 1`). The single-unit branches have no FDR, and Script A's single-unit traffic run is still 15 s. +* **Nothing turns the unused lanes back on.** Script C has no `unused_lanes.sh on`; run it by hand when the DUT is needed in its normal state. Whether a reboot clears it has not been checked. +* **`unused_lanes.sh off` is not gated by `SWB_UNIT0` / `SWB_UNIT1`** — it always acts on both boards. +* **`blanton_ber.sh` (Script A / C) still uses `p=3` and `sleep 30`**; see v1.2.5. +* **Script 4 has no teardown**: finish it with `blanton_tr518.sh stop` **and** `systemctl stop mlucas-amm`. +* **`FAN_SPEED` is 100.** + +## 🔗 Links + +* [ARCHITECTURE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/ARCHITECTURE.md) — test flow, data models, constraints +* [CLAUDE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/CLAUDE.md) — commands and the bench gotchas + +**Full changelog:** [V1.2.5...V1.2.6](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/compare/V1.2.5...V1.2.6) diff --git a/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl b/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl index 5b1701c..55cd8c5 100644 --- a/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl +++ b/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl @@ -1,6 +1,6 @@ ; ============================================================================= ; Script A for Blanton -; Version : V1.2.5 +; Version : V1.2.6 ; Date : 2026-10-21 ; Author : ETWen ; ============================================================================= @@ -848,6 +848,35 @@ ; same as Script 5 ; tools/BER_Test.txt p=3 -> p=p31; add a unit 1 block, a both-units block and ; port en=0 / en=1 for all cabled ports on each unit +; V1.2.6 2026-10-21 Blanton_Script/blanton_fdr.sh Created v1.0.0 - FDR statistics through phydiag fdrstat +; start / clear / rdcnt / stop on ports '1-259,270-565': +; fdrstat STArt bin_group=both Interval=30 / clear / Counters / +; STOp. bcmcmd output printed as is, no parsing or PASS/FAIL +; -u 0 / -u 1 (default both), -i Interval, -n dry run +; !! 'stop' sends 'fdrstat STOp', by analogy with prbsstat STOp +; - NOT verified on the bench. If the BCM shell rejects it, +; FDR keeps running and only the log shows it (bcmcmd exits 0) +; !! called by path - chmod +x ~/Blanton_Script/*.sh +; ScriptA Turn off unused lanes before the BER baseline +; unused_lanes.sh off - both units, TX squelch on the unused +; retimer lanes and TSC31 (cd112-cd115) disabled on the TH6 +; !! nothing turns them back on: no 'unused_lanes.sh on' in +; Script C. They stay off until 'unused_lanes.sh on' is run +; by hand (whether a reboot clears them is not checked) +; !! not gated by SWB_UNIT0 / SWB_UNIT1 - always both units +; !! needs the exec bit on unused_lanes.sh AND retimer_lanes.sh +; ScriptA FDR around the traffic baseline (SWB_UNIT0 = SWB_UNIT1 = 1 only) +; init -> fdr start -> clear -> fdr clear -> ... -> start -> +; sleep 40 -> fdr rdcnt -> traffic stop -> fdr stop -> report +; Traffic sleep 15 -> 40 so rdcnt reads after the first 30 s +; Interval is published (same rule as prbsstat Ber) +; !! the single-unit branches have no FDR and still sleep 15 +; ScriptB fdr start + fdr clear with the traffic setup, before the loop +; (SWB_UNIT0 = SWB_UNIT1 = 1 only) - the counters run for the +; whole soak +; ScriptC fdr rdcnt before traffic stop, fdr stop after it +; (SWB_UNIT0 = SWB_UNIT1 = 1 only) - the closing read of the +; counters Script B started, bracketing the soak ; ============================================================================= include "config.ttl"