From c3b0d491457fd2898afd9dd9c2d865b8287851f4 Mon Sep 17 00:00:00 2001 From: ETWen Date: Fri, 28 Aug 2026 22:04:31 +0800 Subject: [PATCH] fix(session): Clear the last run's jobs and session before starting After a run that ended badly - Tera Term closed, DUT rebooted mid-soak, macro stopped by hand - the stress jobs kept running and the test session stayed open. The next run then started on a box that was already loaded. Nothing errored; the numbers were just quietly wrong. Script A now opens with hw-test-session finish, bgctl reset --yes, bgctl stop --all, then bgctl list. The list is the point of the sequence: it is the one line in the log that proves the DUT was idle at the start. Script A -> V1.1.1. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM --- ARCHITECTURE.md | 3 +- CLAUDE.md | 6 ++- docs/release-notes/v1.1.1.md | 42 +++++++++++++++++++ src/Script_ABC_Blanton/1_Blanton_Script_A.ttl | 18 +++++++- 4 files changed, 65 insertions(+), 4 deletions(-) create mode 100644 docs/release-notes/v1.1.1.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index f2ef3c3..d7e5898 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -342,7 +342,7 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc | 模組 | 進入點 | 功能 | |------|--------|------| -| **Script A（Pre-test）** | `1_Blanton_Script_A.ttl` V1.0.7 | root 登入 → `date -s` 對時 → 清 job log → `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**（`wait_init.ttl`）→ DUT 清單（boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe）→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → 100G uplink 設定與狀態 → **一輪流量基線** → `show uptime` | +| **Script A（Pre-test）** | `1_Blanton_Script_A.ttl` V1.1.1 | root 登入 → `date -s` 對時 → 清 job log → **收掉前一輪殘留**（`hw-test-session finish` → `bgctl reset --yes` → `bgctl stop --all` → `bgctl list` 確認清空）→ `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**（`wait_init.ttl`）→ DUT 清單（boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe）→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → 100G uplink 設定與狀態 → **一輪流量基線** → `show uptime` | | **Script B（Stress）** | `2_Blanton_Script_B.ttl` | `mlucas-avx2 -cpu 0:15` + `bgctl run` 的 `memtester` / SSD / USB 壓力 + BMC DDR（`bmc-manager run memtester`）→ BMC USB net test（探測 cdc_ncm 介面，`systemd-run` 掛 4 小時 ping）→ 啟動 mgmt ping 監控 → `jobs` / `bgctl list` → traffic（依 `SWB_UNIT`）→ `while 1` 每輪完整 PMON + margin | | **Script C（Post-test）** | `3_Blanton_Script_C.ttl` | `kill $(jobs -p)` + `bgctl stop --all` / `reset` + 停 BMC USB unit 與 mgmt ping → PMON + dmesg → 7 組 AER + `ras-mc-ctl` → BMC USB `journalctl` → `ip -s link show eth0/eth1` → `cat` mlucas log → traffic `stop` + `report` → **BER 掃描**（依 `SWB_UNIT`）→ NVMe 健康 → `show reboot-cause` / `uptime` → **job log 複製到 USB（帶時戳）** → `hw-test-session finish` | | **SWB loopback 流量** | `blanton_traffic_linespeed [-u 0\|1\|all]` | `init`（VLAN 30..137，成對 `cdN`/`cdN+32`）→ `clear` → `start`（`tx 100 length=512`）→ `stop` → `report`（per-pair TX/RX 交叉比對 PASS/FAIL）；另有 `ps`（link/speed）與 `run`（一條龍）。**V0.6.0 起含 100G uplink `ce0`**（VLAN 138）：它沒有 loopback 對象，靠 **ingress mirror 打回自己**維持循環，`report` 最後一列就是它，底下再接一份跨 unit summary | @@ -378,6 +378,7 @@ Script A ── logopen Logs/Blanton_Margin_.log ──┐ (之後所有畫 ├─ 登入 admin → sudo -i → root@sonic:~# │ ├─ date -s "" ← 讓 DUT 時戳可對照 ├─ rm /host/hw-eval/jobs/* ← 清掉上一輪的 job log + ├─ hw-test-session finish / bgctl reset / stop --all / list (收前一輪) ├─ hw-test-session start / log / status ├─ 風扇：max31790 unbind→sleep→bind（18-0020 與 25-0020 各一次） │ → fan-speed-control.sh → 回答 y 確認 │ diff --git a/CLAUDE.md b/CLAUDE.md index 72c0485..aa4547f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,7 +9,7 @@ DUT 端的取數能力由一組 **bash 工具庫** 提供（CB FPGA `pcimem` → 完整架構見 [ARCHITECTURE.md](ARCHITECTURE.md)。 -**目前狀態（2026-08-28）**：Script A 為 **V1.1.0**（A 集中記錄 A/B/C 三支的變更）。涵蓋 +**目前狀態（2026-08-28）**：Script A 為 **V1.1.1**（A 集中記錄 A/B/C 三支的變更）。涵蓋 PCIe AER / DDR EDAC / PMON+TPM / 電壓 margin（9 顆 LTC2980）/ SWB loopback 線速流量 / 10G+1G 管理網路 ping / BMC DDR 與 USB / NVMe 健康，壓力層走平台的 `bgctl`。 switch unit 數量由 `config.ttl` 的 `SWB_UNIT0` / `SWB_UNIT1` 宣告，風扇轉速由 `FAN_SPEED` 指定 @@ -20,7 +20,9 @@ Script B 的 soak 迴圈每輪印 `bgctl list` + `jobs`，並 `pause 60`（V1.0. V1.1.0 新增全機 SerDes BER 掃描（`blanton_ber.sh`，**Script C 已接**，依 `SWB_UNIT` 分支）、 TR518 負載（`blanton_tr518.sh`，**TTL 尚未接**）、SWB 軌功耗（`TH6_SWB{0,1}_power_readback.sh`）， 線速流量也納入 100G `ce0`（VLAN 138，靠 ingress mirror 維持循環）。 -最後一次發布是 **V1.0.11**；SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。 +Script A 開頭會先收掉前一輪殘留（`hw-test-session finish` + `bgctl reset/stop --all`）， +再 `bgctl list` 確認沒有 job 留著佔資源。 +最後一次發布是 **V1.1.0**；SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。 交付用 `./publish/publish.sh` 打包。 ## 技術棧 diff --git a/docs/release-notes/v1.1.1.md b/docs/release-notes/v1.1.1.md new file mode 100644 index 0000000..c2daa9e --- /dev/null +++ b/docs/release-notes/v1.1.1.md @@ -0,0 +1,42 @@ +# v1.1.1 — A run no longer inherits the last one's leftovers + +## 🐛 Bug fixes + +**A previous run's jobs and session survived into the next one** +* Script A now clears the box before it starts: `hw-test-session finish`, `bgctl reset --yes`, `bgctl stop --all`, then `bgctl list` so you can see it came back empty. +* This matters after a run that ended badly — a closed Tera Term window, a DUT reboot mid-soak, a macro stopped by hand. The stress jobs kept running and the session stayed open, so the next run started on a box that was already loaded. Nothing failed; the numbers were just quietly wrong, and the only hint was a `bgctl list` nobody was reading. +* The `bgctl list` at the end is the point of the sequence, not decoration: it is the one line in the log that proves the DUT was idle when the run began. + +## 📦 Downloads + +| File | Contents | +|---|---| +| `Script_ABC_Blanton_V1.1.1.zip` | The full Tera Term working directory, including `Blanton_Script/` to copy onto the DUT | + +**After copying to the DUT:** + +```bash +chmod +x ~/Blanton_Script/*.sh # see below +grep -rlU $'\r' ~/Blanton_Script # expect no output +``` + +`blanton_ber.sh` is invoked by Script C as `./Blanton_Script/blanton_ber.sh`, so it needs the execute bit along with `bmc_monitor.sh` and `usb_target.sh` — and neither git nor a Windows/USB copy carries it reliably. Without the `chmod` the BER section fails with `Permission denied` and the log simply has no BER in it. + +## ⚠️ Before you run + +* **`hw-test-session finish` at the start will complain if no session is open.** That is expected on a fresh DUT and does not stop the run. +* **These three tests cannot share the switch.** `blanton_ber.sh`, `blanton_tr518.sh` and `blanton_traffic_linespeed` all take over port loopback and L2 learning. Script C already orders BER after the traffic report. +* **PRBS takes the links down** while `blanton_ber.sh` is armed. That is the test working, not a fault. +* **`ce0` keeps running until you stop it.** If a run is interrupted between `start` and `stop`, the 100G port stays saturated with nothing in `show interfaces status` to hint at it. +* **Two numbers must stay in step**: Script C's `sleep 30` and `blanton_ber.sh`'s `BER_INTERVAL`. +* **`FAN_SPEED` is 100** — the fans run flat out unless you change `config.ttl`. +* **Declare the switch population** with `SWB_UNIT0` / `SWB_UNIT1`. +* **PRBS on the two 100G uplinks (`port_prbs_monitor.sh`) is still not wired into the macros.** +* **The three items listed as unverified in v1.1.0 are still unverified**: the `ce0` ingress mirror, the 120s settle in `blanton_tr518.sh`, and `test mode nr=no` on stop. See [v1.1.0](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/releases/tag/V1.1.0) for the one-command check on each. + +## 🔗 Links + +* [ARCHITECTURE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/ARCHITECTURE.md) — test flow, data models, constraints +* [CLAUDE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/CLAUDE.md) — commands and the bench gotchas + +**Full changelog:** [V1.1.0...V1.1.1](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/compare/V1.1.0...V1.1.1) diff --git a/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl b/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl index f1f4b81..3efa741 100644 --- a/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl +++ b/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl @@ -1,6 +1,6 @@ ; ============================================================================= ; Script A for Blanton -; Version : V1.1.0 +; Version : V1.1.1 ; Date : 2026-08-28 ; Author : ETWen ; ============================================================================= @@ -132,6 +132,12 @@ ; tools/100G_Traffic.txt ; tools/TR518.txt Corrected on the bench: port cd (not *:port all), ; testphase=1, and an explicit stop +; V1.1.1 2026-08-28 ScriptA Clear a leftover session before starting a new run +; hw-test-session finish -> bgctl reset --yes -> +; bgctl stop --all -> bgctl list (must come back empty) +; A run that ended badly leaves jobs holding CPU and a +; session open; the next run used to inherit both, and +; the stress numbers were quietly wrong because of it ; ============================================================================= include "config.ttl" @@ -171,6 +177,16 @@ wait prompt_sonic_root sendln "rm /host/hw-eval/jobs/*" ; ========== HW Test Session ========== +; Clear session +wait prompt_sonic_root +sendln "hw-test-session finish" +wait prompt_sonic_root +sendln "bgctl reset --yes" +wait prompt_sonic_root +sendln "bgctl stop --all" +wait prompt_sonic_root +sendln "bgctl list" + wait prompt_sonic_root sendln "hw-test-session start" wait prompt_sonic_root