diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index f442bc7..69634bd 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -161,7 +161,8 @@ Blanton_TTL_Script/ └── Script_ABC_Blanton/ # ← Tera Term 的工作目錄(整包給測試員) │ ├── 4_Blanton_Script_thermal_safety.ttl # 熱 / 安規:獨立跑,不接 A→B→C - ├── 5_Blanton_Script_Reboot_Cycle.ttl # 重開機循環:獨立跑,while 1 無止境(未上機驗證) + ├── 5_Blanton_Script_Boot_Cycle.ttl # 開機循環(warm / cold,EN_Cold_Warm_boot):獨立跑,while 1 無止境 + ├── 6_Blanton_Script_BER_Cycle.ttl # Diag ↔ SONiC 冷開機循環,兩邊各量一次 unit 0 BER:獨立跑,while 1 ├── config.ttl # 全域設定:testcase 名、EN_Margin / EN_TX_FIR / EN_log / EN_timestamp、 │ # 風扇與 SWB 宣告、SSD/USB/BMC_USB 壓力秒數、三個 prompt 字串 ├── hammer/ # 🚫 gitignored — mlucas ELF binary(~99 MB),DUT image 自帶, @@ -219,6 +220,7 @@ Blanton_TTL_Script/ ├── ddr5_temp.sh # DDR5 DIMM 溫度(SPD5118 @ i2c-5 0x50/0x51,reg 0x31) ├── hsc_temp_readback.sh # LTC4287 HSC 溫度(i2c-28 0x40/0x45,內/外部感溫) ├── avt_pmpower.sh # AMD CPU 封裝功耗(AVTCMD pmm get_pmdata;AVT 是 NDA,不隨包) + ├── tray_id_stress.sh # Tray ID I2C stress(bus 34/35/38/39 0x70 reg 0x02,寫 0xA5/0x5A 讀回) ├── usb_target.sh # 偵測 USB 裝置節點/掛載點(排除系統碟) ├── bmc_monitor.sh # BMC 監控:free -m + I2C 寫入/讀回圖樣測試 ├── bmc_monitor_ddr.sh # ⚠️ 已停用:V1.1.9 起 B/C 內的註解區塊已刪除,DDR 壓力走 bgctl + bmc-manager @@ -381,7 +383,7 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc | 模組 | 進入點 | 功能 | |------|--------|------| -| **Script A(Pre-test)** | `1_Blanton_Script_A.ttl` V1.2.3 | root 登入 → `date -s` 對時 → 清 job log → **收掉前一輪殘留**(`hw-test-session finish` → `bgctl reset --yes` → `bgctl stop --all` → `bgctl list` 確認清空)→ `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**(`wait_init.ttl`)→ DUT 清單(boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe)→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → **NFC 基線**(start → 30s → stop → report)→ 100G uplink 設定與狀態 → **BER 基線掃描**(依 `SWB_UNIT`,V1.2.1 新增)→ **一輪流量基線** → **SerDes DSC 基線**(`EN_TX_FIR`)→ `show uptime` | +| **Script A(Pre-test)** | `1_Blanton_Script_A.ttl` V1.2.4 | root 登入 → `date -s` 對時 → 清 job log → **收掉前一輪殘留**(`hw-test-session finish` → `bgctl reset --yes` → `bgctl stop --all` → `bgctl list` 確認清空)→ `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**(`wait_init.ttl`)→ DUT 清單(boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe)→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → **NFC 基線**(start → 30s → stop → report)→ 100G uplink 設定與狀態 → **BER 基線掃描**(依 `SWB_UNIT`,V1.2.1 新增)→ **一輪流量基線** → **SerDes DSC 基線**(`EN_TX_FIR`)→ `show uptime` | | **Script B(Stress)** | `2_Blanton_Script_B.ttl` | **V1.1.8 起 `mlucas-avx2 -cpu 0:15` 由 `systemd-run --unit=mlucas-amm`(`Type=exec`、`Restart=always`)啟動,不再是 shell 的 `&` 背景 job** + `bgctl run` 的 `memtester` / SSD / USB 壓力(v1.1.7 起 SSD/USB 的 `--runtime`/`--passes` 讀 `config.ttl` 的 `SSD_RUNTIME`/`SSD_COUNT`、`USB_RUNTIME`/`USB_COUNT`;BMC memtester 不帶次數 = 跑到被停) + BMC DDR(`bmc-manager run memtester`)→ BMC USB net test(探測 cdc_ncm 介面,`systemd-run` 掛 `BMC_USB_RUNTIME` 秒的 ping,V1.1.8 前寫死 4 小時)→ 啟動 mgmt ping 監控 → **啟動 NFC 輪詢** → `jobs` / `systemctl is-active mlucas-amm` / `bgctl list` → traffic(依 `SWB_UNIT`)→ `while 1` 每輪 PMON + `bgctl list`/`jobs`/`systemctl is-active mlucas-amm` + `sleep 60`(**V1.1.4 起 margin 不在迴圈內**;V1.2.0 起等待走 DUT 端 `sleep` 而非 macro 端 `pause`)| | **Script C(Post-test)** | `3_Blanton_Script_C.ttl` | `kill $(jobs -p)` + **`systemctl stop mlucas-amm`**(systemd unit 不是 shell job,`jobs -p` 抓不到)+ `bgctl stop --all` / `reset` + 停 BMC USB unit、mgmt ping 與 **NFC 輪詢** → PMON + dmesg → 7 組 AER + `ras-mc-ctl` → BMC USB `journalctl` **寫進 `log/bmc_usb_net.log`**(不再印在 console) → `ip -s link show eth0/eth1` → `cat` mlucas log(並 `cp` 進 job 目錄一起帶走)→ **NFC `report`** → traffic `stop` + `report` → **SerDes DSC 收尾**(`EN_TX_FIR`)→ **BER 掃描**(依 `SWB_UNIT`)→ NVMe 健康 → `show reboot-cause` → **margin 掃描**(`EN_Margin`,V1.1.4 由 Script B 移來)→ `uptime` → **job log 複製到 USB(帶時戳)** → `exit` + wait『Script done』→ `hw-test-session finish` | | **Script 4(Thermal / Safety)** | `4_Blanton_Script_thermal_safety.ttl` | **獨立腳本,不接 A→B→C。** Script A 式的環境建立(登入 / session / 風扇 / `wait_init` / DUT 清單 / BMC / PCIe 清除 / `pcie_bus` / PMON / dmesg)→ **滿載**(`mlucas-avx2 -cpu 0:15`,V1.1.8 起與 Script B 同樣掛成 `mlucas-amm` systemd unit + `blanton_tr518.sh start`)→ `while 1`:PMON + `TH6_SWB0/1_power_readback.sh` + `avt_pmpower.sh`(CPU 封裝功耗,V1.2.3)+ `bgctl list` / `jobs` / `systemctl is-active mlucas-amm`,`pause 60`。**沒有收尾段** —— 中斷後 DUT 仍在 `port cd lb=mac` / `l2 learn off` / `test mode nr=yes`,且 CPU 仍滿載,要手動 `blanton_tr518.sh stop` + `systemctl stop mlucas-amm` | @@ -812,7 +814,7 @@ margin status、全部 VRM 軌的 Vin/Vout/Iout/Temp、CB + ICB 溫感讀值, - [ ] `CHANGELOG.md`:延續三支腳本頂端的 Version History,集中成一份 - [ ] 三支腳本的 header 版號與 `CHANGELOG.md` 對齊,`config.ttl` 加 `script_version` 並寫進 log 開頭 - [ ] `docs/TTL_Script_Blanton_Status_*.xlsx` 更新為全 OK,或把剩餘項目移進 Future Extensions -- [x] 打 git tag,push `nas` + `gitea`(已至 `V1.2.3`;`publish/` 依 Script A 檔頭版號打包) +- [x] 打 git tag,push `nas` + `gitea`(已至 `V1.2.4`;`publish/` 依 Script A 檔頭版號打包) **驗收條件:** 從 zip 解壓到一台乾淨的測試 NB,照 README 操作,不看原始碼就能完成一輪 A→B→C 並產出 summary;`publish/` 內容與 git tag 一致。 diff --git a/CLAUDE.md b/CLAUDE.md index 5e18939..6d9a707 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,7 +9,7 @@ DUT 端的取數能力由一組 **bash 工具庫** 提供(CB FPGA `pcimem` → 完整架構見 [ARCHITECTURE.md](ARCHITECTURE.md)。 -**目前狀態(2026-09-23)**:Script A 為 **V1.2.3**(A 集中記錄所有 TTL 的變更)。涵蓋 +**目前狀態(2026-10-01)**:Script A 為 **V1.2.4**(A 集中記錄所有 TTL 的變更)。涵蓋 PCIe AER / DDR EDAC / PMON+TPM / 電壓 margin(9 顆 LTC2980)/ SWB loopback 線速流量 / 10G+1G 管理網路 ping / BMC DDR 與 USB / NVMe 健康,壓力層走平台的 `bgctl`。 switch unit 數量由 `config.ttl` 的 `SWB_UNIT0` / `SWB_UNIT1` 宣告,風扇轉速由 `FAN_SPEED` 指定 @@ -36,11 +36,14 @@ NFC tag 讀取由 `nfc_polling.sh` 背景輪詢(A 做基線、B 起、C 停並 取 PMON + 兩片 SWB 的軌功耗 + **AMD CPU 封裝功耗**(`avt_pmpower.sh`,V1.2.3);**沒有收尾段**,中斷後要手動 `blanton_tr518.sh stop`)。 **V1.2.3 起 PMON 快照(`utils/setup_pmon.ttl`)多了 DDR5 DIMM 溫度(`ddr5_temp.sh`)與 LTC4287 HSC 溫度 (`hsc_temp_readback.sh`)**,A/B/C/4/5 全部吃到;HSC 那支每次約 +4 秒。 +**V1.2.4 新增 Tray ID I2C stress(`tray_id_stress.sh`)**:Script A 跑一次當基線、Script B 每輪 soak 跑一次。 +另新增 **`6_Blanton_Script_BER_Cycle.ttl`**(Diag ↔ SONiC 冷開機循環量 BER,**獨立跑不接 A→B→C**), +操作與分析見 `tmp/BER_Test_Cycle/runbook.md`(tmp 不進 git)。 以及 **`5_Blanton_Script_Boot_Cycle.ttl`**(重開機循環,**獨立跑不接 A→B→C**:登入 → 風扇 → Check Pt1~Pt13 → `reboot` → `wait "Booting in blind mode"`,`cnt_cycle` 每輪遞增,`while 1` 無止境, 要停就關 Tera Term 的 macro)。**尚未上機驗證**。 流程圖:`docs/Blanton_Test_Flow.drawio`。 -最後一次發布是 **V1.2.3**;SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。 +最後一次發布是 **V1.2.4**;SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。 margin 的 `VNOM` 已於 2026-08-30 改為 Blanton 實測中位數(144 條軌中 86 條上調 0.8~2.5%)。 交付用 `./publish/publish.sh` 打包。 @@ -140,6 +143,12 @@ bash ~/Blanton_Script/port_prbs_monitor.sh stop # 收尾會跑 prbsstat # ⚠️ AVT 是 NDA,不在交付包裡:DUT 上要先解壓到 ~/AVT_Linux_NDA_2.8.37/,沒有就每輪 exit 2 # ⚠️ 數值是 SMU telemetry 不是量測,要準就對照 VR 的 PMBus READ_POUT +# ── Tray ID I2C stress(V1.2.4,Script A / B 已接)──── +~/Blanton_Script/tray_id_stress.sh # bus 34/35/38/39,0x70 reg 0x02:寫 0xA5/0x5A 讀回比對 → Summary +~/Blanton_Script/tray_id_stress.sh -n 100 # 重複 100 輪 +~/Blanton_Script/tray_id_stress.sh -R # 跑完不還原(預設會寫回原值並讀回確認) +# exit 0 = 全 PASS、1 = 有 FAIL(讀回不符)或 ERROR(i2cset/i2cget 本身失敗)、2 = 參數錯 + # ── PDB / CPB power brick 讀值 ────────────────────── ./Blanton_Script/pwr_brick.sh show # Vin/Iin/Vout/Iout/Temp × 5 顆 ./Blanton_Script/pwr_brick.sh show -r # 多印 raw word @@ -357,6 +366,11 @@ Tera Term 連 COM port (115200-8-N-1) - ⚠️ **`stress_hhmd.py` / `stress_pcie.py` 已於 2026-08-17 由 KC 移除**(Status 表 ScriptB #5/#7)。 Script B 現在是 **8 個背景 job**(1 mlucas + 5 mem + ssd + usb),不是 10 個。 I2C/PCIe 的持續讀取壓力目前沒有替代品。 +- ⚠️ **`prbsstat Ber` 要等第一個 interval 結算之後才讀:sleep 必須「大於」Interval**(2026-09-30 驗證)。 + `6_Blanton_Script_BER_Cycle.ttl` 用 `Interval=30` + `sleep 40`。拿同一輪 `prbs get` 的錯誤數交叉比對:sleep 10 時 Diag 低 ~60×、 + SONiC 低 ~40×;sleep 30(= Interval)時 Diag 仍低 ~60×(Ber 在 STArt 後 30.15 s 送出,剛好在結算前)。 + ⚠️ **Script A / C 的 BER 也是 `blanton_ber.sh start`(Interval=30)→ TTL `sleep 30` → `report`**,跑在 SONiC 上 —— 正是「sleep = Interval」的邊界情況, + 能不能趕上第一個 interval 結算**未驗證**。建議改成 `sleep 40`(或 `BER_INTERVAL` 調小),並用 `validate_prbsget.py` 的方法驗證一次 - ⚠️ **`pause` 是停 macro,`sleep` 是停 DUT**(V1.2.0 起 A/B/C 一律用後者):`pause N` 期間 macro 不理會 DUT,shell 是閒著的,等於用猜的估上一個指令跑多久;`sendln "sleep N"` + `wait prompt_sonic_root` 把等待放到 DUT 上,prompt 回來才繼續,兩邊不會走散,而且這一段會帶時戳進 log。 diff --git a/docs/release-notes/v1.2.4.md b/docs/release-notes/v1.2.4.md new file mode 100644 index 0000000..58cd5fc --- /dev/null +++ b/docs/release-notes/v1.2.4.md @@ -0,0 +1,54 @@ +# v1.2.4 — Tray ID I2C stress in Script A and B, and a Diag / SONiC BER cycle that reads the BER correctly + +## ✨ New features + +**`tray_id_stress.sh` — Tray ID I2C write / read-back stress** +* Buses `34` / `35` / `38` / `39`, device `0x70`, register `0x02`. Each bus gets `0xA5` written and read back, then `0x5A` written and read back — the two patterns put every bit in both states. +* One line per step (`PASS` / `FAIL` / `ERROR`), then a per-bus **Summary** table and a single `Tray ID stress RESULT: PASS|FAIL` line. Exit `0` = all PASS, `1` = any FAIL or I2C error, `2` = bad arguments. +* `FAIL` (the read worked but returned the wrong value) and `ERROR` (`i2cset` / `i2cget` itself failed) are counted separately, because they point at different faults. +* The original register value is read first and **written back at the end**, verified by a read. `-R` skips that; `-n N` repeats the whole sequence N times. Buses, address, register and patterns are editable at the top of the file. + +**Script A and Script B run it** +* **Script A** runs one pass after `show_dmesg` as the baseline. +* **Script B** runs one pass in every soak round, before `sleep 60` — 12 writes per round, registers restored each time. + +**`6_Blanton_Script_BER_Cycle.ttl` — Diag ↔ SONiC BER cycle (standalone)** +* Each cycle measures PRBS BER on unit 0 in the Diag OS, cold-boots into SONiC, measures again, and cold-boots back to Diag. `while 1`, no end. The boot target is picked in the BIOS Boot Manager (`USB Device` = Diag, `Hard Drive` = SONiC). +* Log: `Logs\Blanton_BER_Cycle_.log`, timestamps always on. + +## 🐛 Bug fixes + +**`prbsstat Ber` was read before the first interval had been published — BER read 40–60× low** +* The BER cycle originally read `prbsstat Ber` 10 s after `prbsstat STArt Interval=30`. Checked against the error counts that `prbs get` reports in the same cycle, that read **Diag ~60× low and SONiC ~40× low**. +* Moving to `sleep 30` (equal to the interval) fixed SONiC but **not Diag**: its `Ber` goes out 30.15 s after `STArt`, just before the first interval is published. SONiC's `bcmcmd` adds ~0.3 s and just makes it. +* The macro now uses **`Interval=30` + `sleep 40`**. Over 73 cycles both sides agree with the `prbs get` estimate to within 0.1 decade. The rule: **the sleep must be longer than the interval.** + +## 📦 Downloads + +| File | Contents | +|---|---| +| `Script_ABC_Blanton_V1.2.4.zip` | The full Tera Term working directory, including `Blanton_Script/` to copy onto the DUT | + +```bash +chmod +x ~/Blanton_Script/*.sh +grep -rlU $'\r' ~/Blanton_Script # expect no output +``` + +## ⚠️ Before you run + +* **`chmod +x` after deploying.** Script A and B call `tray_id_stress.sh` by path; without the exec bit every call is `Permission denied` and the round carries on without the check. +* **Any BER log taken before 2026-09-30 14:13 reads 40–60× low** (BER cycle with `sleep ≤ Interval`). Relative Diag-vs-SONiC trends are usable; absolute values are not. +* **Script A / C's BER is the same edge case**: `blanton_ber.sh start` (Interval=30) → `sleep 30` → `report`, on SONiC. Whether it catches the first interval has **not been verified**. Lengthening that sleep to 40 is the safe change. +* **The BER cycle's SONiC → Diag boot writes a hard-coded BDF** (`pcimem /sys/bus/pci/devices/0000:05:00.0/resource0 0x480`). Check the CB FPGA BDF on every unit before running it. +* **The BER cycle expects the hostname `sonic`.** A unit with another hostname (e.g. `Margin02`) needs `prompt_sonic` / `prompt_sonic_root` edited, or the macro waits forever. +* **The BER cycle sets the fan to 30 % on the SONiC side only** — SONiC runs ~11 °C hotter than Diag, a variable in any Diag-vs-SONiC comparison. +* **Its SONiC password is the `YourPaSsWoRd` placeholder**, and it does not read `config.ttl` or `secret/`. +* **Script 4 has no teardown**: finish it with `blanton_tr518.sh stop` **and** `systemctl stop mlucas-amm`. +* **`FAN_SPEED` is 100.** + +## 🔗 Links + +* [ARCHITECTURE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/ARCHITECTURE.md) — test flow, data models, constraints +* [CLAUDE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/CLAUDE.md) — commands and the bench gotchas + +**Full changelog:** [V1.2.3...V1.2.4](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/compare/V1.2.3...V1.2.4) diff --git a/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl b/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl index d755c0e..7a7a972 100644 --- a/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl +++ b/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl @@ -1,7 +1,7 @@ ; ============================================================================= ; Script A for Blanton -; Version : V1.2.3 -; Date : 2026-09-23 +; Version : V1.2.4 +; Date : 2026-10-01 ; Author : ETWen ; ============================================================================= ; Version History: @@ -740,6 +740,60 @@ ; !! without -q the full PM table is printed every round, and ; Script 4 loops once a minute - the log grows by one full ; table per minute +; V1.2.4 2026-10-01 Blanton_Script/tray_id_stress.sh Created v1.0.0 - Tray ID I2C write / read-back stress +; Buses 34 / 35 / 38 / 39, device 0x70, register 0x02. Each +; bus: write 0xA5, read back, compare; write 0x5A, read back, +; compare - the two patterns put every bit in both states +; One line per step (PASS / FAIL / ERROR), then a per-bus +; Summary table and 'Tray ID stress RESULT: PASS|FAIL' +; Exit 0 = all PASS, 1 = any FAIL or I2C error, 2 = usage +; -n N repeats the sequence N times; buses / addr / reg / +; patterns are editable at the top of the file +; The original value is read first and written back at the +; end (verified by a read). -R leaves the register at 0x5A +; !! FAIL = the read worked but returned the wrong value; +; ERROR = i2cset / i2cget itself failed. They are counted +; apart because they point at different faults +; !! a bus whose original value cannot be read is NOT +; restored (there is nothing to restore to) - the line +; says so +; !! called by path, so it needs the exec bit on the DUT: +; chmod +x ~/Blanton_Script/*.sh +; ScriptA Add Tray ID stress after show_dmesg (one pass, baseline) +; ScriptB Add Tray ID stress to every soak round, before sleep 60 +; Each round now writes 0xA5 / 0x5A to four Tray ID +; registers and restores them - 12 writes per round +; 6_Blanton_Script_BER_Cycle.ttl Added - Diag / SONiC BER cycle, a STANDALONE macro (not A/B/C) +; Each cycle: PRBS BER on unit 0 in the Diag OS, cold boot +; into SONiC, the same PRBS BER again, cold boot back to +; Diag; while 1, no end. Boot selection drives the BIOS +; Boot Manager ('USB Device' = Diag, 'Hard Drive' = SONiC) +; Log: Logs\Blanton_BER_Cycle_.log, timestamps always on +; Report tools (not shipped): tmp/BER_Test_Cycle/ +; parse_multi.py, gen_multi_html.py, validate_prbsget.py +; prbsstat STArt Interval=30, then sleep 40, then Ber +; !! the sleep MUST be longer than the Interval. Checked +; against the error counts that 'prbs get' reports in the +; same cycle: with sleep 10 Diag read ~60x low and SONiC +; ~40x low; with sleep 30 (= Interval) Diag was still +; ~60x low - its Ber is sent 30.15 s after STArt, just +; before the first interval is published. Interval=30 + +; sleep 40 matched on both sides over 73 cycles. Every +; BER log taken before 2026-09-30 14:13 is low by 40-60x +; !! SONiC -> Diag cold boot writes pcimem +; /sys/bus/pci/devices/0000:05:00.0/resource0 0x480 with a +; hard-coded BDF (the Diag side uses fpga_switch.sh, which +; finds the FPGA itself). Check the BDF on every unit +; !! the SONiC password is the YourPaSsWoRd placeholder and +; the macro does not include config.ttl or secret/ +; !! prompt_sonic_root is 'root@sonic:~#'. A unit whose +; hostname is not 'sonic' (M02 = Margin02) needs the +; prompts edited or the macro waits forever +; !! fan-speed-control.sh 30 is set on the SONiC side only; +; SONiC runs ~11 C hotter than Diag, a variable in any +; Diag vs SONiC comparison +; !! measures unit 0 only, port 268 not in the list, yet +; wait_init still waits for BOTH units (SWB_UNIT0/1 = 1) ; ============================================================================= include "config.ttl"