fix(session): Clear the last run's jobs and session before starting
After a run that ended badly - Tera Term closed, DUT rebooted mid-soak, macro stopped by hand - the stress jobs kept running and the test session stayed open. The next run then started on a box that was already loaded. Nothing errored; the numbers were just quietly wrong. Script A now opens with hw-test-session finish, bgctl reset --yes, bgctl stop --all, then bgctl list. The list is the point of the sequence: it is the one line in the log that proves the DUT was idle at the start. Script A -> V1.1.1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
This commit is contained in:
Claude Opus 5
+2
-1
@@ -342,7 +342,7 @@ PROFILE_CHANNELS=( "0:high:8" "1:low:8" ... ) # "channel:operation:change_perc
|
||||
|
||||
| 模組 | 進入點 | 功能 |
|
||||
|------|--------|------|
|
||||
| **Script A（Pre-test）** | `1_Blanton_Script_A.ttl` V1.0.7 | root 登入 → `date -s` 對時 → 清 job log → `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**（`wait_init.ttl`）→ DUT 清單（boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe）→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → 100G uplink 設定與狀態 → **一輪流量基線** → `show uptime` |
|
||||
| **Script A（Pre-test）** | `1_Blanton_Script_A.ttl` V1.1.1 | root 登入 → `date -s` 對時 → 清 job log → **收掉前一輪殘留**（`hw-test-session finish` → `bgctl reset --yes` → `bgctl stop --all` → `bgctl list` 確認清空）→ `hw-test-session start` → **設定風扇轉速** → **等資料面就緒**（`wait_init.ttl`）→ DUT 清單（boot/version/fwutil/syseeprom/ssdhealth/TPM/NVMe）→ BMC enroll + version/status → source bash 工具 → 清/讀 7 組 PCIe AER + `ras-mc-ctl --summary` → `lspci` → margin → PMON → dmesg → **10G/1G 管理網路 ping** → 100G uplink 設定與狀態 → **一輪流量基線** → `show uptime` |
|
||||
| **Script B（Stress）** | `2_Blanton_Script_B.ttl` | `mlucas-avx2 -cpu 0:15` + `bgctl run` 的 `memtester` / SSD / USB 壓力 + BMC DDR（`bmc-manager run memtester`）→ BMC USB net test（探測 cdc_ncm 介面，`systemd-run` 掛 4 小時 ping）→ 啟動 mgmt ping 監控 → `jobs` / `bgctl list` → traffic（依 `SWB_UNIT`）→ `while 1` 每輪完整 PMON + margin |
|
||||
| **Script C（Post-test）** | `3_Blanton_Script_C.ttl` | `kill $(jobs -p)` + `bgctl stop --all` / `reset` + 停 BMC USB unit 與 mgmt ping → PMON + dmesg → 7 組 AER + `ras-mc-ctl` → BMC USB `journalctl` → `ip -s link show eth0/eth1` → `cat` mlucas log → traffic `stop` + `report` → **BER 掃描**（依 `SWB_UNIT`）→ NVMe 健康 → `show reboot-cause` / `uptime` → **job log 複製到 USB（帶時戳）** → `hw-test-session finish` |
|
||||
| **SWB loopback 流量** | `blanton_traffic_linespeed <sub> [-u 0\|1\|all]` | `init`（VLAN 30..137，成對 `cdN`/`cdN+32`）→ `clear` → `start`（`tx 100 length=512`）→ `stop` → `report`（per-pair TX/RX 交叉比對 PASS/FAIL）；另有 `ps`（link/speed）與 `run`（一條龍）。**V0.6.0 起含 100G uplink `ce0`**（VLAN 138）：它沒有 loopback 對象，靠 **ingress mirror 打回自己**維持循環，`report` 最後一列就是它，底下再接一份跨 unit summary |
|
||||
@@ -378,6 +378,7 @@ Script A ── logopen Logs/Blanton_Margin_<ts>.log ──┐ (之後所有畫
|
||||
├─ 登入 admin → sudo -i → root@sonic:~# │
|
||||
├─ date -s "<host 時間>" ← 讓 DUT 時戳可對照
|
||||
├─ rm /host/hw-eval/jobs/* ← 清掉上一輪的 job log
|
||||
├─ hw-test-session finish / bgctl reset / stop --all / list (收前一輪)
|
||||
├─ hw-test-session start / log / status
|
||||
├─ 風扇：max31790 unbind→sleep→bind（18-0020 與 25-0020 各一次）
|
||||
│ → fan-speed-control.sh <FAN_SPEED> → 回答 y 確認 │
|
||||
|
||||
@@ -9,7 +9,7 @@ DUT 端的取數能力由一組 **bash 工具庫** 提供（CB FPGA `pcimem` →
|
||||
|
||||
完整架構見 [ARCHITECTURE.md](ARCHITECTURE.md)。
|
||||
|
||||
**目前狀態（2026-08-28）**：Script A 為 **V1.1.0**（A 集中記錄 A/B/C 三支的變更）。涵蓋
|
||||
**目前狀態（2026-08-28）**：Script A 為 **V1.1.1**（A 集中記錄 A/B/C 三支的變更）。涵蓋
|
||||
PCIe AER / DDR EDAC / PMON+TPM / 電壓 margin（9 顆 LTC2980）/ SWB loopback 線速流量 /
|
||||
10G+1G 管理網路 ping / BMC DDR 與 USB / NVMe 健康，壓力層走平台的 `bgctl`。
|
||||
switch unit 數量由 `config.ttl` 的 `SWB_UNIT0` / `SWB_UNIT1` 宣告，風扇轉速由 `FAN_SPEED` 指定
|
||||
@@ -20,7 +20,9 @@ Script B 的 soak 迴圈每輪印 `bgctl list` + `jobs`，並 `pause 60`（V1.0.
|
||||
V1.1.0 新增全機 SerDes BER 掃描（`blanton_ber.sh`，**Script C 已接**，依 `SWB_UNIT` 分支）、
|
||||
TR518 負載（`blanton_tr518.sh`，**TTL 尚未接**）、SWB 軌功耗（`TH6_SWB{0,1}_power_readback.sh`），
|
||||
線速流量也納入 100G `ce0`（VLAN 138，靠 ingress mirror 維持循環）。
|
||||
最後一次發布是 **V1.0.11**；SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。
|
||||
Script A 開頭會先收掉前一輪殘留（`hw-test-session finish` + `bgctl reset/stop --all`），
|
||||
再 `bgctl list` 確認沒有 job 留著佔資源。
|
||||
最後一次發布是 **V1.1.0**；SWB 的 I2C 通道與位址已於 2026-08-19 上機驗證。
|
||||
交付用 `./publish/publish.sh` 打包。
|
||||
|
||||
## 技術棧
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
# v1.1.1 — A run no longer inherits the last one's leftovers
|
||||
|
||||
## 🐛 Bug fixes
|
||||
|
||||
**A previous run's jobs and session survived into the next one**
|
||||
* Script A now clears the box before it starts: `hw-test-session finish`, `bgctl reset --yes`, `bgctl stop --all`, then `bgctl list` so you can see it came back empty.
|
||||
* This matters after a run that ended badly — a closed Tera Term window, a DUT reboot mid-soak, a macro stopped by hand. The stress jobs kept running and the session stayed open, so the next run started on a box that was already loaded. Nothing failed; the numbers were just quietly wrong, and the only hint was a `bgctl list` nobody was reading.
|
||||
* The `bgctl list` at the end is the point of the sequence, not decoration: it is the one line in the log that proves the DUT was idle when the run began.
|
||||
|
||||
## 📦 Downloads
|
||||
|
||||
| File | Contents |
|
||||
|---|---|
|
||||
| `Script_ABC_Blanton_V1.1.1.zip` | The full Tera Term working directory, including `Blanton_Script/` to copy onto the DUT |
|
||||
|
||||
**After copying to the DUT:**
|
||||
|
||||
```bash
|
||||
chmod +x ~/Blanton_Script/*.sh # see below
|
||||
grep -rlU $'\r' ~/Blanton_Script # expect no output
|
||||
```
|
||||
|
||||
`blanton_ber.sh` is invoked by Script C as `./Blanton_Script/blanton_ber.sh`, so it needs the execute bit along with `bmc_monitor.sh` and `usb_target.sh` — and neither git nor a Windows/USB copy carries it reliably. Without the `chmod` the BER section fails with `Permission denied` and the log simply has no BER in it.
|
||||
|
||||
## ⚠️ Before you run
|
||||
|
||||
* **`hw-test-session finish` at the start will complain if no session is open.** That is expected on a fresh DUT and does not stop the run.
|
||||
* **These three tests cannot share the switch.** `blanton_ber.sh`, `blanton_tr518.sh` and `blanton_traffic_linespeed` all take over port loopback and L2 learning. Script C already orders BER after the traffic report.
|
||||
* **PRBS takes the links down** while `blanton_ber.sh` is armed. That is the test working, not a fault.
|
||||
* **`ce0` keeps running until you stop it.** If a run is interrupted between `start` and `stop`, the 100G port stays saturated with nothing in `show interfaces status` to hint at it.
|
||||
* **Two numbers must stay in step**: Script C's `sleep 30` and `blanton_ber.sh`'s `BER_INTERVAL`.
|
||||
* **`FAN_SPEED` is 100** — the fans run flat out unless you change `config.ttl`.
|
||||
* **Declare the switch population** with `SWB_UNIT0` / `SWB_UNIT1`.
|
||||
* **PRBS on the two 100G uplinks (`port_prbs_monitor.sh`) is still not wired into the macros.**
|
||||
* **The three items listed as unverified in v1.1.0 are still unverified**: the `ce0` ingress mirror, the 120s settle in `blanton_tr518.sh`, and `test mode nr=no` on stop. See [v1.1.0](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/releases/tag/V1.1.0) for the one-command check on each.
|
||||
|
||||
## 🔗 Links
|
||||
|
||||
* [ARCHITECTURE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/ARCHITECTURE.md) — test flow, data models, constraints
|
||||
* [CLAUDE.md](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/src/branch/main/CLAUDE.md) — commands and the bench gotchas
|
||||
|
||||
**Full changelog:** [V1.1.0...V1.1.1](https://etgit.et-wen.com/etwen/Blanton_TTL_Script/compare/V1.1.0...V1.1.1)
|
||||
@@ -1,6 +1,6 @@
|
||||
; =============================================================================
|
||||
; Script A for Blanton
|
||||
; Version : V1.1.0
|
||||
; Version : V1.1.1
|
||||
; Date : 2026-08-28
|
||||
; Author : ETWen
|
||||
; =============================================================================
|
||||
@@ -132,6 +132,12 @@
|
||||
; tools/100G_Traffic.txt
|
||||
; tools/TR518.txt Corrected on the bench: port cd (not *:port all),
|
||||
; testphase=1, and an explicit stop
|
||||
; V1.1.1 2026-08-28 ScriptA Clear a leftover session before starting a new run
|
||||
; hw-test-session finish -> bgctl reset --yes ->
|
||||
; bgctl stop --all -> bgctl list (must come back empty)
|
||||
; A run that ended badly leaves jobs holding CPU and a
|
||||
; session open; the next run used to inherit both, and
|
||||
; the stress numbers were quietly wrong because of it
|
||||
; =============================================================================
|
||||
include "config.ttl"
|
||||
|
||||
@@ -171,6 +177,16 @@ wait prompt_sonic_root
|
||||
sendln "rm /host/hw-eval/jobs/*"
|
||||
|
||||
; ========== HW Test Session ==========
|
||||
; Clear session
|
||||
wait prompt_sonic_root
|
||||
sendln "hw-test-session finish"
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl reset --yes"
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl stop --all"
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl list"
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln "hw-test-session start"
|
||||
wait prompt_sonic_root
|
||||
|
||||
Reference in New Issue
Block a user