diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 7c2a91b..99ee278 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -374,7 +374,7 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc | 模組 | 進入點 | 功能 | |------|--------|------| -| **Script A(Pre-test)** | `1_Blanton_Script_A.ttl` V1.2.0 | root 登入 → `date -s` 對時 → 清 job log → **收掉前一輪殘留**(`hw-test-session finish` → `bgctl reset --yes` → `bgctl stop --all` → `bgctl list` 確認清空)→ `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**(`wait_init.ttl`)→ DUT 清單(boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe)→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → **NFC 基線**(start → 30s → stop → report)→ 100G uplink 設定與狀態 → **一輪流量基線** → **SerDes DSC 基線**(`EN_TX_FIR`)→ `show uptime` | +| **Script A(Pre-test)** | `1_Blanton_Script_A.ttl` V1.2.1 | root 登入 → `date -s` 對時 → 清 job log → **收掉前一輪殘留**(`hw-test-session finish` → `bgctl reset --yes` → `bgctl stop --all` → `bgctl list` 確認清空)→ `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**(`wait_init.ttl`)→ DUT 清單(boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe)→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → **NFC 基線**(start → 30s → stop → report)→ 100G uplink 設定與狀態 → **BER 基線掃描**(依 `SWB_UNIT`,V1.2.1 新增)→ **一輪流量基線** → **SerDes DSC 基線**(`EN_TX_FIR`)→ `show uptime` | | **Script B(Stress)** | `2_Blanton_Script_B.ttl` | **V1.1.8 起 `mlucas-avx2 -cpu 0:15` 由 `systemd-run --unit=mlucas-amm`(`Type=exec`、`Restart=always`)啟動,不再是 shell 的 `&` 背景 job** + `bgctl run` 的 `memtester` / SSD / USB 壓力(v1.1.7 起 SSD/USB 的 `--runtime`/`--passes` 讀 `config.ttl` 的 `SSD_RUNTIME`/`SSD_COUNT`、`USB_RUNTIME`/`USB_COUNT`;BMC memtester 不帶次數 = 跑到被停) + BMC DDR(`bmc-manager run memtester`)→ BMC USB net test(探測 cdc_ncm 介面,`systemd-run` 掛 `BMC_USB_RUNTIME` 秒的 ping,V1.1.8 前寫死 4 小時)→ 啟動 mgmt ping 監控 → **啟動 NFC 輪詢** → `jobs` / `systemctl is-active mlucas-amm` / `bgctl list` → traffic(依 `SWB_UNIT`)→ `while 1` 每輪 PMON + `bgctl list`/`jobs`/`systemctl is-active mlucas-amm` + `sleep 60`(**V1.1.4 起 margin 不在迴圈內**;V1.2.0 起等待走 DUT 端 `sleep` 而非 macro 端 `pause`)| | **Script C(Post-test)** | `3_Blanton_Script_C.ttl` | `kill $(jobs -p)` + **`systemctl stop mlucas-amm`**(systemd unit 不是 shell job,`jobs -p` 抓不到)+ `bgctl stop --all` / `reset` + 停 BMC USB unit、mgmt ping 與 **NFC 輪詢** → PMON + dmesg → 7 組 AER + `ras-mc-ctl` → BMC USB `journalctl` **寫進 `log/bmc_usb_net.log`**(不再印在 console) → `ip -s link show eth0/eth1` → `cat` mlucas log(並 `cp` 進 job 目錄一起帶走)→ **NFC `report`** → traffic `stop` + `report` → **SerDes DSC 收尾**(`EN_TX_FIR`)→ **BER 掃描**(依 `SWB_UNIT`)→ NVMe 健康 → `show reboot-cause` → **margin 掃描**(`EN_Margin`,V1.1.4 由 Script B 移來)→ `uptime` → **job log 複製到 USB(帶時戳)** → `exit` + wait『Script done』→ `hw-test-session finish` | | **Script 4(Thermal / Safety)** | `4_Blanton_Script_thermal_safety.ttl` | **獨立腳本,不接 A→B→C。** Script A 式的環境建立(登入 / session / 風扇 / `wait_init` / DUT 清單 / BMC / PCIe 清除 / `pcie_bus` / PMON / dmesg)→ **滿載**(`mlucas-avx2 -cpu 0:15`,V1.1.8 起與 Script B 同樣掛成 `mlucas-amm` systemd unit + `blanton_tr518.sh start`)→ `while 1`:PMON + `TH6_SWB0/1_power_readback.sh` + `bgctl list` / `jobs` / `systemctl is-active mlucas-amm`,`pause 60`。**沒有收尾段** —— 中斷後 DUT 仍在 `port cd lb=mac` / `l2 learn off` / `test mode nr=yes`,且 CPU 仍滿載,要手動 `blanton_tr518.sh stop` + `systemctl stop mlucas-amm` | @@ -393,7 +393,7 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc | **BMC DDR 壓力** | `bmc_monitor_ddr.sh` | ⚠️ **已停用** —— V1.1.9 起 B/C 內的註解區塊已刪除,改由 `bgctl run bmc-manager run memtester` 取代。檔案保留未刪 | | **SerDes DSC 擷取** | `bcm_dsc_scan.sh [-u 0\|1] [-f ] [-F] [-n] [-C]` + `bcm_dsc_parse.sh -i -o ` | `bcmcmd -n -c "dsh -c 'phydiag dsc'"` 掃全清單(1 個 100G + 445 個 200G port)× 2 unit,raw log 寫進 `/log` 再 `cp` 到 `/host/hw-eval/jobs`。`-f` 支援 `%u` / `%t`;沒帶 `%u` 時自動插 `_u0`/`_u1`。成功與否**不看 `bcmcmd` 的 exit code**,改比對 `SERDES DISPLAY DIAG DATA END` 數量 + 實際迭代筆數,結尾印 `### DSC_SCAN_DONE rc=`。parse 端輸出 `_lane.csv`(每 lane:TXEQ/RXFFE/DFE/EYE/SNR)與 `_pm.csv`(每 pm_id:溫度/PLL/lane 數/統計)。**由 `EN_TX_FIR` 控制,預設關** | | **100G PRBS** | `port_prbs_monitor.sh {start [both\|a\|b]\|stop\|status\|report\|clear}` | 兩個 uplink 各一個 worker,可同時或單獨跑。setup(`lpmode off` → `prbs set` → `prbsstat STArt`)→ 週期 `prbs get`(判定)+ `prbsstat Ber`(只記錄)→ teardown(`STOp` + `prbs clear`)。輸出含 `PRBS OK!` 才算 PASS。**⚠️ V1.1.9 起 A/B/C 內的註解呼叫已刪除,只能手動跑** | -| **SerDes BER 掃描** | `blanton_ber.sh {init\|clear\|start\|stop\|report} [-u 0\|1\|all]` | 全機每個對接 port 的 PRBS BER:`prbs set p=3` + `prbs get`(確認每條 lane 都鎖上)→ `prbsstat STArt Interval=30` →(等一個 interval)→ `prbsstat Ber` → `STOp` → `prbs clear`。`report` 一列一條 lane,**BER < 1e-6 才 PASS**,最後跨 unit 的 summary 帶最差 lane。⚠️ PRBS 期間 link 顯示 DOWN、不過流量 | +| **SerDes BER 掃描** | `blanton_ber.sh {init\|clear\|start\|stop\|report} [-u 0\|1\|all]` | 全機每個對接 port 的 PRBS BER:`prbs set p=3` + `prbs get`(確認每條 lane 都鎖上)→ `prbsstat STArt Interval=30` →(等一個 interval)→ `prbsstat Ber` → `STOp` → `prbs clear`。`report` 一列一條 lane,**BER < 1e-6 才 PASS**,最後跨 unit 的 summary 帶最差 lane。⚠️ PRBS 期間 link 顯示 DOWN、不過流量。V1.2.1 起 Script A 與 Script C 各跑一次(A 在流量之前、C 在流量報表之後),夾住 soak | | **TR518 負載** | `blanton_tr518.sh {start\|stop} [-u 0\|1\|all]` | `port cd lb=mac` → `l2 learn off` → `test mode nr=yes` → `tr 518 ... testphase=1`,然後 settle `TR_SETTLE_SEC`(120s)再讓人量功耗。`stop` 只送 `port cd lb=none`(照 `TR518.txt`),L2 學習與 test mode **不會還原**,`TR_RESTORE_*=1` 才會 | | **PDB / CPB brick** | `pwr_brick.sh show [-r]` | 5 顆 brick 的 Vin / Iin / Vout / Iout / Temp(SWB0 `0x62`/`0x63`、SWB1 `0x60`/`0x61`、CPB `0x26`)。四顆 ADPM12200 **每個量各自一個 PMBus PAGE**(Vin 9、Iin 10、Vout 2、Iout 14、Temp 18),讀之前一定要先寫 PAGE —— `blanton_pwr_data.sh` 完全沒寫,所以整張 PDB 表都是 NA。換算用手冊 DIRECT 公式 `X=(1/m)(Y×10⁻ᴿ−b)`。每個值重讀到通過三道檢查為止(非 `0xFFFF`、非 `0x0000`、落在合理範圍)—— 這條匯流排**只壞高位元組**,`11.98 V` 會變成 `32.00 V`,全一過濾擋不住 | | **SWB 軌功耗** | `TH6_SWB{0,1}_power_readback.sh` | 讀 `show platform voltage/current` **各一次**,19 條軌 V×I 與總功耗。舊版每軌各讀一次(38 次呼叫、每軌不同時刻),總和是不同瞬間的加總 | @@ -804,7 +804,7 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值, - [ ] `CHANGELOG.md`:延續三支腳本頂端的 Version History,集中成一份 - [ ] 三支腳本的 header 版號與 `CHANGELOG.md` 對齊,`config.ttl` 加 `script_version` 並寫進 log 開頭 - [ ] `docs/TTL_Script_Blanton_Status_*.xlsx` 更新為全 OK,或把剩餘項目移進 Future Extensions -- [x] 打 git tag,push `nas` + `gitea`(已至 `V1.2.0`;`publish/` 依 Script A 檔頭版號打包) +- [x] 打 git tag,push `nas` + `gitea`(已至 `V1.2.1`;`publish/` 依 Script A 檔頭版號打包) **驗收條件:** 從 zip 解壓到一台乾淨的測試 NB,照 README 操作,不看原始碼就能完成一輪 A→B→C 並產出 summary;`publish/` 內容與 git tag 一致。 @@ -914,9 +914,14 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值, - **margin 在 soak 期間不再取樣**(V1.1.4):只剩 Script A 的起始值與 Script C 的結束值。 漂移發生在中間就看不到了。若要恢復,比較好的做法是在 B 的迴圈內每 N 輪才掃一次, 而不是每輪都掃。 -- **`blanton_ber.sh` 的 `sleep 30` 與 `BER_INTERVAL` 是兩處要同步改的數字**:Script C 寫死 - `sendln "sleep 30"`,腳本預設 `BER_INTERVAL=30`。改一邊沒改另一邊,report 會讀到還沒累積完 - 一個 interval 的數字,而且不會有任何警告。 +- **`blanton_ber.sh` 的 `sleep 30` 與 `BER_INTERVAL` 是兩處要同步改的數字**:Script A(V1.2.1 起) + 與 Script C 都寫死 `sendln "sleep 30"`,腳本預設 `BER_INTERVAL=30`。改一邊沒改另一邊,report 會 + 讀到還沒累積完一個 interval 的數字,而且不會有任何警告。現在有**三處**要一起改。 +- **Script A 的 BER `clear` 與第一輪流量之間沒有 settle**:V1.2.1 把 BER 掃描放在流量**之前**, + 而 PRBS 是把 lane 拉出正常運作、`clear` 才放回去的。Script A 的 `blanton_ber.sh clear` 之後 + 緊接著就是 `blanton_traffic_linespeed ps` / `init`,中間沒有等待;Script C 的 BER 在流量報表 + 之後,沒有這個順序。上機時比對兩份 traffic report —— Script A 有 NA/FAIL 而 Script C 沒有, + 就是 lane 尚未回穩,在 `clear` 後加一個 `sleep` 即可。**尚未上機驗證。** - **`ce0` 的 ingress mirror 尚未上機驗證**:`mirror port ce0 Mode=Ingress DestPort=ce0` 這條路是 推論出來的(鏡像複本不受 egress filter 的 source-port knockout 限制),語法取自 DUT 的 `help mirror`,但**還沒證明硬體真的接受 destport == srcport**。驗證方式是隔 10 秒看兩次 diff --git a/CLAUDE.md b/CLAUDE.md index 04857f0..4e9893f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,7 +9,7 @@ DUT 端的取數能力由一組 **bash 工具庫** 提供(CB FPGA `pcimem` → 完整架構見 [ARCHITECTURE.md](ARCHITECTURE.md)。 -**目前狀態(2026-09-07)**:Script A 為 **V1.2.0**(A 集中記錄所有 TTL 的變更)。涵蓋 +**目前狀態(2026-09-08)**:Script A 為 **V1.2.1**(A 集中記錄所有 TTL 的變更)。涵蓋 PCIe AER / DDR EDAC / PMON+TPM / 電壓 margin(9 顆 LTC2980)/ SWB loopback 線速流量 / 10G+1G 管理網路 ping / BMC DDR 與 USB / NVMe 健康,壓力層走平台的 `bgctl`。 switch unit 數量由 `config.ttl` 的 `SWB_UNIT0` / `SWB_UNIT1` 宣告,風扇轉速由 `FAN_SPEED` 指定 @@ -24,7 +24,8 @@ Script B 的 soak 迴圈每輪印 `bgctl list` + `jobs` + `systemctl is-active m (V1.0.10 之前完全沒有間隔;V1.2.0 起 A/B/C 的等待一律改成 DUT 端 `sleep` 而非 macro 端 `pause`)。V1.1.8 起 AMD CPU 壓力(`mlucas-avx2`)由 `systemd-run` 掛成 `mlucas-amm` transient unit(`Restart=always`),不再是 shell 的 `&` 背景 job —— 掉線也不會停, Script C 用 `systemctl stop mlucas-amm` 收掉並把 log 複製到隨身碟。 -V1.1.0 新增全機 SerDes BER 掃描(`blanton_ber.sh`,**Script C 已接**,依 `SWB_UNIT` 分支)、 +V1.1.0 新增全機 SerDes BER 掃描(`blanton_ber.sh`,依 `SWB_UNIT` 分支);**V1.2.1 起 Script A 也接了一次**, +在第一輪流量之前取基線,與 Script C 流量報表之後的收尾掃描一起夾住 soak(同 DSC 的做法)。另有 TR518 負載(`blanton_tr518.sh`,**TTL 尚未接**)、SWB 軌功耗(`TH6_SWB{0,1}_power_readback.sh`), 線速流量也納入 100G `ce0`(VLAN 138,靠 ingress mirror 維持循環)。 Script A 開頭會先收掉前一輪殘留(`hw-test-session finish` + `bgctl reset/stop --all`), @@ -353,6 +354,10 @@ Tera Term 連 COM port (115200-8-N-1) - ⚠️ **兩張管理網卡同網段會 ARP flux**:`mgmt_ping_monitor.sh` 讓 eth0/eth1 同時 up,同一個 /24 下 對端 ARP 可能由任一張卡回應,回包落在沒送封包的那張。`ARP_STRICT=1` 擋掉;判讀時看 `RESULT` 行的 `nic_tx`/`nic_rx`(該卡自己的 packet delta)—— **ping 成功但 `nic_tx` 接近 0 = 封包從另一張卡出去的** +- ⚠️ **BER 的 `clear` 才是把 lane 放回正常運作的動作**:V1.2.1 起 Script A 在第一輪流量**之前**跑 BER, + 而 `blanton_ber.sh clear` 與緊接著的 `blanton_traffic_linespeed ps` 之間**沒有任何 settle**。 + Script C 是流量報表之後才跑 BER,沒有這個順序問題。第一次上機時比對兩邊的 traffic report —— + **Script A 有 NA/FAIL 而 Script C 沒有,就是 lane 還沒回穩**,在 `clear` 後補一個 `sleep` 即可 - ⚠️ **PRBS 跑起來時 `show interfaces status` 會顯示 DOWN**,那是 link 離開正常運作模式,不是故障。 `port_prbs_monitor.sh` 的 `start` 因此刻意不印 port status(避免測試員誤判),只在 `prbs clear` 之後的報表裡印 - ⚠️ **`ip -s link` 的欄位順序是 `bytes packets errors ...`**,要 packet 數取的是**第 2 欄**不是第 1 欄 diff --git a/docs/release-notes/v1.2.1.md b/docs/release-notes/v1.2.1.md new file mode 100644 index 0000000..67d10ad --- /dev/null +++ b/docs/release-notes/v1.2.1.md @@ -0,0 +1,49 @@ +# v1.2.1 — A BER baseline, so the SerDes numbers have something to be compared against + +## ✨ New features + +**Script A now takes a SerDes BER sweep before the first traffic run** +* `blanton_ber.sh init → start → sleep 30 → report → stop → clear`, branched on `SWB_UNIT0` / `SWB_UNIT1` exactly like the traffic block beside it. +* Script C already ran the closing sweep after its traffic report. With this one at the front, the pair brackets the soak — the same shape as the DSC scans added in v1.2.0, and the same reason: a BER figure on its own says very little, a *change* in it says a lot. +* The `sleep 30` matches `blanton_ber.sh`'s `BER_INTERVAL`. Reporting before one full interval has accumulated reads a window that has not finished, and nothing warns you about it. + +## 🐛 Bug fixes + +**The new block's comment described the wrong position** +* It was copied out of Script C, which runs BER *after* the traffic report, and the comment came with it and still said "after". In Script A the block runs *before* traffic. The comment now says what the code does and why the ordering is what it is. + +## ⚠️ Before you run + +* **Watch Script A's first traffic report on the first run.** PRBS holds the lanes out of normal operation and `blanton_ber.sh clear` is what releases them — and in Script A nothing settles between that `clear` and the `blanton_traffic_linespeed ps` / `init` that follows. Script C does not have this ordering, so the two reports are a built-in comparison: **if Script A shows NA or FAIL pairs that Script C does not, the lanes had not come back yet**, and a `sleep` after `clear` is the fix. This has not been run on hardware yet. +* **`sleep 30` and `BER_INTERVAL` now live in three places** — Script A, Script C, and `blanton_ber.sh` itself. Changing one without the others silently reports a partial interval. +* **BER, `blanton_traffic_linespeed` and `blanton_tr518.sh` cannot overlap.** They all take over the same ports. Script A now touches those ports twice before the soak even starts. +* **`EN_TX_FIR` is still 0 by default**, so the DSC scans do not run unless you turn them on; a full scan is ~10 min in each of Script A and Script C. +* **Replace `config.ttl` along with the macros.** Tera Term aborts the macro on an undefined variable rather than treating it as 0. +* **`bcm_dsc_scan.sh` needs the execute bit**: `chmod +x ~/Blanton_Script/*.sh` after copying. +* **`--runtime` is the job's overall limit, not a per-pass one.** `--passes` does not extend it. +* **Script 4 has no teardown**: finish it with `blanton_tr518.sh stop` **and** `systemctl stop mlucas-amm`. +* **Script 5 (reboot cycle) has no exit condition.** A 199-cycle / 60-hour run has now been analysed: PCIe AER stayed at 0, all 218 traffic pairs and all 864 lanes passed every cycle, and the BMC never reset. Two things it did surface — `bmc-first-enroll` does not exist on that BMC build, and NFC returned FAIL on every cycle. +* **`blanton_multiphase_margin.sh save` is permanent**, and must not be mixed with `margin.sh`. +* **`READ_VIN` and `READ_IIN` are still not in the ADPM12200 coefficient table.** +* **The BMC USB service log only exists on the USB stick.** +* **`FAN_SPEED` is 100.** +* **Still unverified from v1.1.0**: the `ce0` ingress mirror, the 120 s settle in `blanton_tr518.sh`, and `test mode nr=no` on stop. + +## 📦 Downloads + +| File | Contents | +|---|---| +| `Script_ABC_Blanton_V1.2.1.zip` | The full Tera Term working directory, including `Blanton_Script/` to copy onto the DUT | + +```bash +chmod +x ~/Blanton_Script/*.sh +grep -rlU $'\r' ~/Blanton_Script # expect no output +ls ~/hammer/tools/amd/mlucas-avx2 # must already be there +``` + +## 🔗 Links + +* [ARCHITECTURE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/ARCHITECTURE.md) — test flow, data models, constraints +* [CLAUDE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/CLAUDE.md) — commands and the bench gotchas + +**Full changelog:** [V1.2.0...V1.2.1](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/compare/V1.2.0...V1.2.1)