etwen/Blanton_TTL_Script
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

fix(stress): Keep the SSD and USB jobs alive for the whole soak

Browse Source 
qfx5252-stress-usb was started with --runtime 60. It finished a minute
into an overnight run and the USB path sat idle for the rest of the
night - nothing failed, nothing said so, and the log looked complete.
SSD had the same shape at --runtime 3600. Both are now 86400, and the
BMC DDR memtester drops its loop count so it runs until stopped.

86400 is 24h, not forever. Past that SSD and USB die quietly and the
only clue is the shorter bgctl list Script B already prints each round.
Said so in the version history, the release notes and ARCHITECTURE.

Adds blanton_multiphase_margin.sh: the multiphase controllers on both
switch boards over the native i2c buses (SWB0 on i2c-13/15, SWB1 on
-14/-16), 19 rails each. high3/low3/comboA/comboB set voltages and print
one line per rail; save commits them and prints Multiphase Save All,
because that list has no per-rail mapping to report against.

Two hazards it carries: save is STORE_USER_ALL and permanent, and these
are NOT the LTC2980s margin.sh drives - the two tools reach the same
rails from opposite sides with no interlock between them.

The emitted commands were diffed against tools/Multiphase_*.txt: argv
identical for all five. The rail-to-netname mapping was cross-checked
against the bus numbers rather than trusted by line position - blocks
1..19 are all bus 13/15 and 20..38 all 14/16, matching the SWB0/SWB1
split of Multiphase_Netname.

blanton_traffic_linespeed v0.7.0 appends PatternRandom=yes to every tx
including ce0, and the defaults become tx 200 length=324. A fixed
pattern can sit on a benign bit sequence for a whole run; a changing one
is what exposes a weak lane. -r no forces fixed, -r "" omits the
argument for an SDK that will not take it. Bytes per burst went 51200 ->
64800, so absolute counter totals do not carry across this build - the
pair cross-check that decides PASS/FAIL does.

Script A -> V1.1.5.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
This commit is contained in:
etwen
2026-09-01 11:04:45 +08:00
co-authored by Claude Opus 5
parent db44f72352
commit 215fed6e95
13 changed files with 2061 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Script_ABC_Blanton/1_Blanton_Script_A.ttl
+34 -2
View File
@@ -1,7 +1,7 @@
; =============================================================================
; Script A for Blanton
; Version : V1.1.4
; Date : 2026-08-31
; Version : V1.1.5
; Date : 2026-09-01
; Author : ETWen
; =============================================================================
; Version History:
@@ -201,6 +201,38 @@
; finish - leaves the session shell first, so finish
; runs in the login shell that opened it
; docs/Blanton_Test_Flow.drawio Add the A/B/C + thermal flow diagram
; V1.1.5 2026-09-01 ScriptB Stress jobs now last the whole soak
; BMC DDR memtester 64M 1 -> 64M (loop count
; dropped = runs until stopped)
; SSD --runtime 3600 -> 86400
; USB --runtime 60 -> 86400
; The USB one finished after 60 SECONDS, so an
; overnight run had USB idle for all but the first
; minute and nothing said so
; !! 86400 is 24h: a soak longer than that leaves SSD
; and USB dead for the remainder - the per-round
; bgctl list (V1.0.10) is where that shows up
; Blanton_Script/blanton_multiphase_margin.sh Add multiphase VRM margin over the native i2c buses
; high3 / low3 / comboA / comboB / save, -u 0|1|all
; 38 rails (19 per SWB), one line each:
; SWB0_0V9_ANLG0 - 0.930V Configure Done
; save has no per-rail map, prints Multiphase Save All
; !! save is STORE_USER_ALL - permanent, survives a
; power cycle
; !! these are NOT the LTC2980s margin.sh drives; the
; two tools reach the same rails from different
; sides - do not run both
; Blanton_Script/blanton_traffic_linespeed.sh v0.7.0 tx gains PatternRandom, defaults change
; tx 100 length=512 -> tx 200 length=324
; every burst now ends PatternRandom=yes, ce0 included
; A fixed pattern can sit on a benign bit sequence for
; a whole run; random is what exposes a weak lane
; -r no forces the fixed pattern, -r "" omits the
; argument entirely (pre-v0.7.0 wire format)
; !! bytes per burst went 51200 -> 64800 (+27%), so
; counter totals are not comparable across this
; build - the PASS/FAIL cross-check is unaffected
; tools/Multiphase_*.txt Bench command references for the new script
; =============================================================================
include "config.ttl"
src/Script_ABC_Blanton/2_Blanton_Script_B.ttl
+4 -3
View File
@@ -26,7 +26,8 @@ sendln "~/hammer/tools/amd/mlucas-avx2 -s l -cpu 0:15 > ~/hammer/tools/amd/mluca
;wait prompt_sonic_root
;sendln "./Blanton_Script/bmc_monitor.sh start"
wait prompt_sonic_root
sendln "bgctl run bmc-manager run '/usr/bin/memtester 64M 1'"
;sendln "bgctl run bmc-manager run '/usr/bin/memtester 64M 1'"
sendln "bgctl run bmc-manager run '/usr/bin/memtester 64M'" ; Continue Execture
; ========== BMC Monitor Test ==========
wait prompt_sonic_root
@@ -40,7 +41,7 @@ sendln "bgctl run /usr/sbin/memtester 1G" ; Continue Execture
; ========== SSD read/write ==========
wait prompt_sonic_root
sendln "bgctl run qfx5252-stress-ssd 1G --runtime 3600 --passes 5 --write --force --log /host/hw-eval/jobs/qfx5252-stress-ssd.log"
sendln "bgctl run qfx5252-stress-ssd 1G --runtime 86400 --passes 5 --write --force --log /host/hw-eval/jobs/qfx5252-stress-ssd.log"
; ========== USB read/write ==========
; ---- Get USB partition node -> usb_dev ----
@@ -51,7 +52,7 @@ waitregex '<<DEV=([^>]*)>>'
usb_dev = groupmatchstr1
wait prompt_sonic_root
sprintf2 cmd "bgctl run qfx5252-stress-usb %s 256M --runtime 60 --passes 1 --write --log /host/hw-eval/jobs/qfx5252-stress-usb.log" usb_dev
sprintf2 cmd "bgctl run qfx5252-stress-usb %s 256M --runtime 86400 --passes 1 --write --log /host/hw-eval/jobs/qfx5252-stress-usb.log" usb_dev
sendln cmd
;wait prompt_sonic_root
src/Script_ABC_Blanton/Blanton_Script/blanton_multiphase_margin.sh
+1047
View File
File diff suppressed because it is too large Load Diff
src/Script_ABC_Blanton/Blanton_Script/blanton_traffic_linespeed.sh
+38 -10
View File
@@ -6,7 +6,7 @@
# V0.1.0 2026-08-17 First version. Bash port of the three Tera Term macros
# utils/traffic_loopback_vlan_setting.ttl (VLAN 30..137,
# port pair cdN <-> cdN+32), utils/traffic_loopback_start.ttl
# (tx 100 length=512 VLantag=<vid>) and
# (tx <count> length=<len> VLantag=<vid>) and
# utils/traffic_loopback_vlan_remove.ttl.
# One DUT = two switch boards -> bcmcmd unit 0 and 1
# (bcmcmd -n <unit>), default both.
@@ -90,6 +90,16 @@
# With only one unit selected the summary reports
# UNDECIDED (exit 3) instead of guessing.
# TL_CE_ENABLE=0 restores the old cd-pairs-only behaviour.
# V0.7.0 2026-09-01 Change: tx bursts carry PatternRandom=yes and the defaults
# become tx 200 length=324 (were 100 / 512).
# Random payload keeps the bit pattern changing instead of
# repeating, which is what makes a marginal lane show up;
# a fixed pattern can sit on a benign sequence all run.
# TL_TX_RANDOM=no forces the fixed pattern, "" leaves the
# argument off entirely (pre-V0.7.0 wire format).
# NOTE: counters scale with count x length, so totals from
# before this build are not comparable - 200x324 moves
# about 27% more bytes per burst than 100x512.
# V0.6.0 2026-08-28 Add: ce0 traffic is now SUSTAINED, not a single burst.
# A cd pair keeps its packets circulating because they
# flood to the partner port; ce0 has no partner and a
@@ -142,6 +152,8 @@
# aliases: -tx | --tx
# -l|--length <n> tx packet length (default $TL_TX_LENGTH)
# aliases: -length | --len
# -r|--random <yes|no> tx PatternRandom (default $TL_TX_RANDOM;
# empty string omits the argument)
# -p|--ports <list|all> ps: port list ('all' = plain 'ps', every port)
# -T|--tolerance <n> report: allowed |delta| in packets (default 0;
# -T 1 absorbs the normal snapshot skew)
@@ -161,7 +173,8 @@
# -h|--help help
#
# Environment overrides:
# BCMCMD_CMD=bcmcmd TL_UNITS="0 1" TL_TX_COUNT=100 TL_TX_LENGTH=512
# BCMCMD_CMD=bcmcmd TL_UNITS="0 1" TL_TX_COUNT=200 TL_TX_LENGTH=324
# TL_TX_RANDOM=yes
# TL_PS_PORTS="1-24,40-96,..." TL_TOLERANCE=0 TL_LOG_DIR=/tmp
# DEBUG_MODE=1
#
@@ -173,8 +186,14 @@
# === Config ==================================================================
BCMCMD_CMD="${BCMCMD_CMD:-bcmcmd}"
TL_UNITS="${TL_UNITS:-0 1}" # one DUT = two switch boards (0 and 1)
TL_TX_COUNT="${TL_TX_COUNT:-100}"
TL_TX_LENGTH="${TL_TX_LENGTH:-512}"
TL_TX_COUNT="${TL_TX_COUNT:-200}"
TL_TX_LENGTH="${TL_TX_LENGTH:-324}"
# PatternRandom=yes|no is appended to every tx burst. Random payload exercises
# the SerDes with changing bit patterns instead of the repeating one a fixed
# pattern produces, which is what makes a marginal lane show up.
# Set to "" to leave the argument off entirely (pre-V0.7.0 behaviour).
TL_TX_RANDOM="${TL_TX_RANDOM:-yes}"
TL_DEFAULT_VLAN="${TL_DEFAULT_VLAN:-1}"
TL_DEFAULT_VLAN_PBM="${TL_DEFAULT_VLAN_PBM:-cd}"
@@ -515,17 +534,23 @@ traffic_linespeed_start() {
_tl_units_check || return 1
_tl_is_uint "$TL_TX_COUNT" || { _tl_err "tx count '$TL_TX_COUNT' is not a positive integer (-tx|-c <n>)"; return 1; }
_tl_is_uint "$TL_TX_LENGTH" || { _tl_err "tx length '$TL_TX_LENGTH' is not a positive integer (-length|-l <n>)"; return 1; }
local rnd=""
case "$TL_TX_RANDOM" in
yes|no) rnd=" PatternRandom=$TL_TX_RANDOM" ;;
"") rnd="" ;;
*) _tl_err "PatternRandom '$TL_TX_RANDOM' must be yes, no or empty (-r|--random)"; return 1 ;;
esac
local unit vid pa pb n fail
for unit in $TL_UNITS; do
n=0; fail=0
_tl_info "unit $unit: tx $TL_TX_COUNT length=$TL_TX_LENGTH per VLAN 30..137"
_tl_info "unit $unit: tx $TL_TX_COUNT length=$TL_TX_LENGTH${rnd} per VLAN 30..137"
while read -r vid pa pb <&3; do
[ -z "$vid" ] && continue
_tl_bcm_q "$unit" "tx $TL_TX_COUNT length=$TL_TX_LENGTH VLantag=$vid" || fail=$((fail+1))
_tl_bcm_q "$unit" "tx $TL_TX_COUNT length=$TL_TX_LENGTH VLantag=$vid${rnd}" || fail=$((fail+1))
n=$((n+1))
done 3<<< "$TL_PAIRS"
if [ "$TL_CE_ENABLE" = "1" ]; then
_tl_bcm_q "$unit" "tx $TL_TX_COUNT length=$TL_TX_LENGTH VLantag=$TL_CE_VLAN" || fail=$((fail+1))
_tl_bcm_q "$unit" "tx $TL_TX_COUNT length=$TL_TX_LENGTH VLantag=$TL_CE_VLAN${rnd}" || fail=$((fail+1))
n=$((n+1))
fi
if [ "$fail" -eq 0 ]; then
@@ -949,7 +974,7 @@ traffic_linespeed_run() {
blanton_traffic_linespeed_help() {
echo -e "\033[1mblanton_traffic_linespeed.sh - SWB loopback line-speed traffic (bcmcmd)\033[0m"
echo ""
echo -e " Units: \033[36m$TL_UNITS\033[0m tx: \033[36m$TL_TX_COUNT pkt x ${TL_TX_LENGTH}B\033[0m"
echo -e " Units: \033[36m$TL_UNITS\033[0m tx: \033[36m$TL_TX_COUNT pkt x ${TL_TX_LENGTH}B\033[0m PatternRandom: \033[36m${TL_TX_RANDOM:-（不加）}\033[0m"
echo -e " VLANs: 30..137 ($(_tl_pair_count) loopback pairs, cdN <-> cdN+32)"
[ "$TL_CE_ENABLE" = "1" ] && \
echo -e " 100G : VLAN \033[36m$TL_CE_VLAN\033[0m port \033[36m$TL_CE_PORT\033[0m mode=\033[36m$TL_CE_MODE\033[0m (cross = unit0.$TL_CE_PORT <-> unit1.$TL_CE_PORT cabled)"
@@ -958,7 +983,7 @@ blanton_traffic_linespeed_help() {
echo -e " \033[33minit\033[0m step 1 vlan remove 1 pbm=cd, vlan create/add each pair"
echo -e " \033[33mclear\033[0m step 2 clear c"
echo -e " \033[33mshow\033[0m step 3/6 show c (run before and after traffic)"
echo -e " \033[33mstart\033[0m step 4 tx <count> length=<len> VLantag=<vid> per VLAN"
echo -e " \033[33mstart\033[0m step 4 tx <count> length=<len> VLantag=<vid> [PatternRandom=<yes|no>] per VLAN"
echo -e " \033[33mstop\033[0m step 5 vlan remove <vid> pbm=<pair> [--destroy also destroys]"
[ "$TL_CE_ENABLE" = "1" ] && [ "$TL_CE_MIRROR" = "1" ] && \
echo -e " \033[33m^ also turns the $TL_CE_PORT mirror off -- without it the 100G loop keeps running\033[0m"
@@ -972,6 +997,7 @@ blanton_traffic_linespeed_help() {
echo -e " \033[33m-u|--unit\033[0m <0|1|all> bcmcmd unit(s); 'all' = 0 1 (default: $TL_UNITS)"
echo -e " \033[33m-c|--count|-tx\033[0m <n> tx packet count (default $TL_TX_COUNT)"
echo -e " \033[33m-l|--length|-length\033[0m <n> tx packet length (default $TL_TX_LENGTH)"
echo -e " \033[33m-r|--random\033[0m <yes|no> tx PatternRandom (default $TL_TX_RANDOM; \"\" 完全不加)"
echo -e " \033[33m-p|--ports\033[0m <list> ps port list, or 'all' for every port"
echo -e " \033[33m-T|--tolerance\033[0m <n> report: allowed |delta| pkts (default $TL_TOLERANCE; -T 1 = snapshot skew)"
echo -e " \033[33m-a|--all\033[0m report: kept for compatibility (all pairs are listed)"
@@ -995,7 +1021,8 @@ blanton_traffic_linespeed_help() {
echo -e " blanton_traffic_linespeed report -u 1 -q | grep -v PASS"
echo -e " blanton_traffic_linespeed report -f /tmp/showc_unit1_*.log -a"
echo -e " blanton_traffic_linespeed show -u 1 | grep -iE 'error|discard|drop'"
echo -e " blanton_traffic_linespeed start -u 0 -tx 100 -length 512"
echo -e " blanton_traffic_linespeed start -u 0 -tx 200 -length 324"
echo -e " blanton_traffic_linespeed start -r no # 固定樣式，不隨機"
echo -e " blanton_traffic_linespeed start -tx 100000 -length 9216 -d"
echo -e " blanton_traffic_linespeed stop --destroy"
echo ""
@@ -1019,6 +1046,7 @@ blanton_traffic_linespeed() {
shift 2 ;;
-c|--count|-tx|--tx) TL_TX_COUNT="$2"; shift 2 ;;
-l|-length|--length|--len) TL_TX_LENGTH="$2"; shift 2 ;;
-r|--random|-PatternRandom|--pattern-random) TL_TX_RANDOM="$2"; shift 2 ;;
-p|--ports) TL_PS_PORTS="$2"; shift 2 ;;
-T|--tolerance) TL_TOLERANCE="$2"; shift 2 ;;
-f|--file) TL_REPORT_FILE="$2"; shift 2 ;;