feat: Add mgmt_ping_monitor.sh and move stress onto bgctl

New Blanton_Script/mgmt_ping_monitor.sh: brings both management NICs up
with iproute2 (`ip link set` / `address replace` / `route replace` with
per-NIC metric) and pings each one's own target -- eth0 -> .30, eth1 ->
.31 -- appending to a rotating log. Same sub-commands as the bmc
monitors, plus `summary` for just the per-leg RESULT lines.

Both NICs stay up, which on a shared subnet lets the target's ARP be
answered by either one, so a reply can land on the NIC that did not
send. ARP_STRICT applies arp_ignore/arp_announce to prevent that, and
each leg logs `ip -s link show` with the interface's own TX/RX packet
delta across the burst as direct evidence of which NIC carried the
traffic. Note `ip -s link` orders columns "bytes packets ...", so the
packet count is the second field.

Script A: hw-test-session start/log/status, a fuller DUT inventory
(version, fwutil, syseeprom, ssdhealth, TPM, nvme smart-log, smartctl),
bmc-first-enroll and bmc-manager version/status, ras-mc-ctl summary, and
both 100G uplinks now brought up rather than 513 being left down.

Script B: stress moves to bgctl (memtester, qfx5252-stress-ssd/-usb) and
the BMC DDR load runs through bmc-manager, replacing the hammer scripts
and the bmc_monitor pair. Adds a BMC USB net test that discovers the
cdc_ncm interface and runs a 4-hour ping under systemd-run.

Script C: stops the bgctl jobs, the BMC USB unit and the mgmt ping, then
collects journalctl, per-NIC counters, ras-mc-ctl and NVMe health, and
closes the session with hw-test-session finish.

utils/show_dmesg.ttl: one combined error regex with `dmesg -T`, an
i2c-filtered view, then `dmesg -C` so Script C's capture shows only what
the soak produced. utils/setup_pmon.ttl: add the TPM FRU/read checks.

Fix: Script B set `timeout = 15` for the cdc_ncm probe and never
restored it, leaving the cap in force for everything after -- including
the traffic init, which walks 108 VLANs per unit and takes far longer
than 15 s. A timed-out wait returns without the prompt, so the macro
would have run ahead of the DUT for the rest of the soak. Reset to 0 at
:skip_ping, where both branches meet.

Script A -> V1.0.6.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
This commit is contained in:
2026-08-24 23:27:37 +08:00
co-authored by Claude Opus 5
parent 9a11c1d5bb
commit e38d24c775
6 changed files with 633 additions and 46 deletions
+67 -11
View File
@@ -1,7 +1,7 @@
; ============================================================================= ; =============================================================================
; Script A for Blanton ; Script A for Blanton
; Version : V1.0.5 ; Version : V1.0.6
; Date : 2026-08-21 ; Date : 2026-08-24
; Author : ETWen ; Author : ETWen
; ============================================================================= ; =============================================================================
; Version History: ; Version History:
@@ -25,6 +25,18 @@
; V1.0.5 2026-08-21 config.ttl Add SWB_UNIT0/SWB_UNIT1 ; V1.0.5 2026-08-21 config.ttl Add SWB_UNIT0/SWB_UNIT1
; utils/wait_init.ttl Follow config.ttl SWB_UNIT0/SWB_UNIT1 ; utils/wait_init.ttl Follow config.ttl SWB_UNIT0/SWB_UNIT1
; ScriptA/B/C Traffic follows SWB_UNIT: -u 0 / -u 1 / skip ; ScriptA/B/C Traffic follows SWB_UNIT: -u 0 / -u 1 / skip
; V1.0.6 2026-08-24 utils/show_dmesg.ttl Edited dmesg grep i2c & clear event
; ScriptA HW Test Session
; ScriptC HW Test Session Finish
; ScriptA bmc-first-enroll, bmc version/status
; ScriptB Stress test DDR, SSD, USB, BMC DDR, BMC USB Stress test
; ScriptC BMC USB Stress test result
; ScriptA DUT Info
; ScriptAC 10G/1G MGMT ping
; ScriptA Edited 100G Port Status
; ScriptA, utils/setup_pmon.ttl Add TPM
; ScriptAC NVME Error Info
; ScriptAC Add PCIe Summary
; ============================================================================= ; =============================================================================
include "config.ttl" include "config.ttl"
@@ -59,21 +71,50 @@ gettime timestr
sprintf2 cmd 'sudo date -s "%s %s"' datestr timestr sprintf2 cmd 'sudo date -s "%s %s"' datestr timestr
sendln cmd sendln cmd
; ========== Show Build Name ========== ; ========== HW Test Session ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "show boot" sendln "hw-test-session start"
; ========== EEPROM Info ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "hpe-eeprom-tlv show --bus 4 --addr 0x50" sendln "hw-test-session log"
wait prompt_sonic_root
sendln "hw-test-session status"
; ========== Wait DUT Init ========== ; ========== Wait DUT Init ==========
wait prompt_sonic_root wait prompt_sonic_root
include "utils/wait_init.ttl" include "utils/wait_init.ttl"
; ========== BMC version ========== ; ========== DUT Info ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "bmc-manager run 'cat /etc/issue'" sendln "show boot"
wait prompt_sonic_root
sendln "show version"
wait prompt_sonic_root
sendln "fwutil show status"
wait prompt_sonic_root
sendln "show system-memory"
wait prompt_sonic_root
sendln "show service"
wait prompt_sonic_root
sendln "show platform syseeprom"
wait prompt_sonic_root
sendln "show platform ssdhealth"
wait prompt_sonic_root
sendln "qfx5252-tpm-version"
wait prompt_sonic_root
sendln "/usr/sbin/nvme smart-log -H /dev/nvme0"
wait prompt_sonic_root
sendln "/usr/sbin/smartctl -x /dev/nvme0"
; ========== BMC version ==========
;wait prompt_sonic_root
;sendln "bmc-manager run 'cat /etc/issue'"
wait prompt_sonic_root
sendln "bmc-first-enroll"
wait prompt_sonic_root
sendln "bmc-manager version"
wait prompt_sonic_root
sendln "bmc-manager status"
; ========== Load Shell Script ========== ; ========== Load Shell Script ==========
wait prompt_sonic_root wait prompt_sonic_root
@@ -98,6 +139,9 @@ include "utils/show_pcie_error_reg_DDR.ttl"
include "utils/show_pcie_error_reg_I210.ttl" include "utils/show_pcie_error_reg_I210.ttl"
include "utils/show_pcie_error_reg_SSD.ttl" include "utils/show_pcie_error_reg_SSD.ttl"
wait prompt_sonic_root
sendln "/usr/sbin/ras-mc-ctl --summary"
; ========== Check PCIE tree + PCIE link status + GET PCIE bandwidth Test ========== ; ========== Check PCIE tree + PCIE link status + GET PCIE bandwidth Test ==========
include "utils/pcie_bus.ttl" include "utils/pcie_bus.ttl"
@@ -115,12 +159,24 @@ include "utils/show_dmesg.ttl"
;wait prompt_sonic_root ;wait prompt_sonic_root
;flushrecv ;flushrecv
; ========== 10G/1G MGMT Ping Test ==========
wait prompt_sonic_root
sendln "./Blanton_Script/mgmt_ping_monitor.sh start"
wait prompt_sonic_root
sendln "~/Blanton_Script/mgmt_ping_monitor.sh stop"
wait prompt_sonic_root
sendln "cat ~/Blanton_Script/log/mgmt_ping.log"
; ========== 100G Port Status ========== ; ========== 100G Port Status ==========
; Lpmode Disable
wait prompt_sonic_root
sendln "sudo sfputil lpmode off Ethernet513"
wait prompt_sonic_root
sendln "sudo sfputil lpmode off Ethernet514"
wait prompt_sonic_root wait prompt_sonic_root
sendln "config interface -n asic0 startup Ethernet513" sendln "config interface -n asic0 startup Ethernet513"
wait prompt_sonic_root wait prompt_sonic_root
sendln "config interface -n asic0 shutdown Ethernet513"
wait prompt_sonic_root
sendln "config interface -n asic1 startup Ethernet514" sendln "config interface -n asic1 startup Ethernet514"
wait prompt_sonic_root wait prompt_sonic_root
sendln "config save -y" sendln "config save -y"
+45 -22
View File
@@ -20,45 +20,68 @@ sendln 'chmod +x ~/hammer/tools/amd/mlucas-avx2'
wait prompt_sonic_root wait prompt_sonic_root
sendln "~/hammer/tools/amd/mlucas-avx2 -s l -cpu 0:15 > ~/hammer/tools/amd/mlucas_amm_log 2>&1 &" sendln "~/hammer/tools/amd/mlucas-avx2 -s l -cpu 0:15 > ~/hammer/tools/amd/mlucas_amm_log 2>&1 &"
; ========== BMC Stress Test ========== ; ========== BMC DDR Stress Test ==========
;wait prompt_sonic_root
;sendln "./Blanton_Script/bmc_monitor_ddr.sh start"
;wait prompt_sonic_root
;sendln "./Blanton_Script/bmc_monitor.sh start"
wait prompt_sonic_root wait prompt_sonic_root
sendln "./Blanton_Script/bmc_monitor_ddr.sh start" sendln "bgctl run bmc-manager run '/usr/bin/memtester 64M 1'"
wait prompt_sonic_root
sendln "./Blanton_Script/bmc_monitor.sh start"
; ========== DDR MEMORY STRESS TEST ========== ; ========== DDR MEMORY STRESS TEST ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "python ~/hammer/tools/stress_mem.py &" sendln "bgctl run /usr/sbin/memtester 1G 100"
wait prompt_sonic_root
sendln "python ~/hammer/tools/stress_mem.py &"
wait prompt_sonic_root
sendln "python ~/hammer/tools/stress_mem.py &"
wait prompt_sonic_root
sendln "python ~/hammer/tools/stress_mem.py &"
wait prompt_sonic_root
sendln "python ~/hammer/tools/stress_mem.py &"
; ========== SSD read/write ========== ; ========== SSD read/write ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "python ~/hammer/tools/stress_ssd.py &" sendln "bgctl run qfx5252-stress-ssd 1G --runtime 3600 --passes 5 --write --force --log /host/hw-eval/current/qfx5252-stress-ssd.log"
; ========== USB read/write ========== ; ========== USB read/write ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "python ~/hammer/tools/stress_usb.py &" sendln "bgctl run qfx5252-stress-usb /dev/sda1 256M --runtime 60 --passes 1 --write --log /host/hw-eval/qfx5252-stress-usb.log"
; ========== SHOW Background job ========== ; ========== SHOW Background job ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "jobs" sendln "jobs"
wait prompt_sonic_root
sendln "bgctl list"
; ========== BMC USB Test ==========
wait prompt_sonic_root
sendln 'echo "NCMIF=$(ls -d /sys/bus/usb/drivers/cdc_ncm/*/net/* 2>/dev/null | head -n 1 | xargs -r basename)"'
timeout = 15
waitregex 'NCMIF=[A-Za-z0-9_-]+'
if result = 0 then
messagebox 'cdc_ncm interface not found. Ping test skipped.' 'ERROR'
goto skip_ping
endif
strlen matchstr
cut_len = result - 6 ; 'NCMIF=' = 6 chars
strcopy matchstr 7 cut_len ncm_if ; -> ncm_if = "eth2" / "eth3"
sprintf2 cmd 'ip -br link show %s' ncm_if
wait prompt_sonic_root
sendln cmd
wait prompt_sonic_root
sendln 'systemctl stop qfx5252-bmc-usb-net-test.service 2>/dev/null || true'
wait prompt_sonic_root
sendln 'systemctl reset-failed qfx5252-bmc-usb-net-test.service 2>/dev/null || true'
sprintf2 cmd 'systemd-run --unit=qfx5252-bmc-usb-net-test --property=Type=exec /usr/bin/ping -I %s -i 1 -W 2 -w 14400 192.168.200.200' ncm_if
wait prompt_sonic_root
sendln cmd
:skip_ping
; restore the default (no cap): the traffic init below walks 108 VLANs per
; unit and takes far longer than the 15 s set for the NCM probe above.
timeout = 0
; ========== 10G/1G MGMT Ping Test ==========
wait prompt_sonic_root
sendln "./Blanton_Script/mgmt_ping_monitor.sh start"
; ========== 100G Port ========== ; ========== 100G Port ==========
; Lpmode Disable
wait prompt_sonic_root
sendln "sudo sfputil lpmode off Ethernet513"
wait prompt_sonic_root
sendln "sudo sfputil lpmode off Ethernet514"
wait prompt_sonic_root wait prompt_sonic_root
sendln "show interfaces status Ethernet513,Ethernet514" sendln "show interfaces status Ethernet513,Ethernet514"
+44 -7
View File
@@ -12,6 +12,21 @@ pause 1
; ========== Kill Process ========== ; ========== Kill Process ==========
include "utils/kill_all_process.ttl" include "utils/kill_all_process.ttl"
; bgctrl all stop
wait prompt_sonic_root
sendln "bgctl stop --all"
wait prompt_sonic_root
sendln "bgctl stop --all"
wait prompt_sonic_root
sendln "bgctl reset --yes"
; BMC USB Test STOP
wait prompt_sonic_root
sendln "systemctl stop qfx5252-bmc-usb-net-test.service"
; 10G/1G MGMT Test STOP
wait prompt_sonic_root
sendln "~/Blanton_Script/mgmt_ping_monitor.sh stop"
; ========== TAKE DATA ========== ; ========== TAKE DATA ==========
include "utils/setup_pmon.ttl" include "utils/setup_pmon.ttl"
include "utils/show_dmesg.ttl" include "utils/show_dmesg.ttl"
@@ -25,22 +40,33 @@ include "utils/show_pcie_error_reg_DDR.ttl"
include "utils/show_pcie_error_reg_I210.ttl" include "utils/show_pcie_error_reg_I210.ttl"
include "utils/show_pcie_error_reg_SSD.ttl" include "utils/show_pcie_error_reg_SSD.ttl"
; ========== BMC TAKE DATA ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "./Blanton_Script/bmc_monitor_ddr.sh stop" sendln "/usr/sbin/ras-mc-ctl --summary"
; ========== BMC DDR TAKE DATA ==========
;wait prompt_sonic_root
;sendln "./Blanton_Script/bmc_monitor_ddr.sh stop"
;wait prompt_sonic_root
;sendln "./Blanton_Script/bmc_monitor.sh stop"
;wait prompt_sonic_root
;sendln "cat ./Blanton_Script/log/bmc_poll.log"
;wait prompt_sonic_root
;sendln "cat ./Blanton_Script/log/bmc_ddr.log"
; ========== BMC USB Test result ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "./Blanton_Script/bmc_monitor.sh stop" sendln "journalctl -u qfx5252-bmc-usb-net-test.service --no-pager"
; ========== 10G/1G MGMT Test result ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "cat ./Blanton_Script/log/bmc_poll.log" sendln "ip -s link show eth0"
wait prompt_sonic_root wait prompt_sonic_root
sendln "cat ./Blanton_Script/log/bmc_ddr.log" sendln "ip -s link show eth1"
; ========== CHECK Stress results ========== ; ========== CHECK Stress results ==========
wait prompt_sonic_root wait prompt_sonic_root
sendln "cat ~/hammer/tools/amd/mlucas_amm_log" sendln "cat ~/hammer/tools/amd/mlucas_amm_log"
wait prompt_sonic_root
sendln "cat ~/hammer/tools/log.stress_ssd"
; ========== CHECK Traffic counters ========== ; ========== CHECK Traffic counters ==========
if SWB_UNIT0 = 1 && SWB_UNIT1 = 1 then if SWB_UNIT0 = 1 && SWB_UNIT1 = 1 then
@@ -62,6 +88,13 @@ else
endif endif
; ========== NVME Err Info ==========
wait prompt_sonic_root
sendln "/usr/sbin/nvme smart-log -H /dev/nvme0"
wait prompt_sonic_root
sendln "/usr/sbin/smartctl -x /dev/nvme0"
wait prompt_sonic_root wait prompt_sonic_root
sendln "show reboot-cause" sendln "show reboot-cause"
@@ -69,4 +102,8 @@ sendln "show reboot-cause"
wait prompt_sonic_root wait prompt_sonic_root
sendln "show uptime" sendln "show uptime"
; ========== HW Test Session ==========
wait prompt_sonic_root
sendln "hw-test-session finish"
messagebox 'GOOD JOB! Test Case DONE' 'teraterm' messagebox 'GOOD JOB! Test Case DONE' 'teraterm'
@@ -0,0 +1,463 @@
#!/bin/bash
###############################################################################
# mgmt_ping_monitor.sh
#
# Version : V1.0.0
# Author : ETWen
# Date : 20260824
#
# Purpose : Exercise both management NICs and ping each one's own target,
# appending everything to a rotating log. One round is
#
# 10G (eth0) : ip link up + address/route replace -> settle
# -> ping N x TARGET_10G -> ip -s link show
# 1G (eth1) : ip link up + address/route replace -> settle
# -> ping N x TARGET_1G -> ip -s link show
#
# Designed to run detached in the background, same sub-commands as
# bmc_monitor.sh: start/stop/status/fg/tail/clear.
#
# Notes : - Both NICs are brought up and LEFT up. Addresses and default
# routes are applied with iproute2 `replace`, which is idempotent:
# re-running a round does not stack duplicate addresses or routes.
# - The default routes carry different metrics (100 / 200), so the
# lower one wins for off-subnet traffic. They do NOT decide which
# NIC a ping leaves by: `ping -I <iface>` binds to the device.
# - !! SAME-SUBNET WARNING !! With both NICs up on one subnet the
# target ARPs for the address and Linux may answer from either
# NIC, so replies can arrive on the wrong one and the result is
# not attributable. ARP_STRICT=1 sets arp_ignore/arp_announce to
# stop that. Leave it on unless the two NICs are on separate
# subnets, as in the iproute2 example this was modelled on.
# - Each leg logs `ip -s link show <iface>` and the TX/RX packet
# delta across the burst, which is the direct evidence of which
# NIC actually carried the traffic.
# - The first pings after bringing a link up are lost to ARP, not to
# the link: a bench capture showed 4x "Destination Host
# Unreachable" on the first burst and 0% loss on the very next
# one. LINK_SETTLE_SEC absorbs that; it is not packet loss.
#
# Version History
# V1.0.0 20260824 Initial Version
# V2.0.0 20260824 Switch from ifconfig to iproute2 (ip link / address
# replace / route replace with per-NIC metric). Both
# NICs now stay up -- no link down between legs. Add
# `ip -s link show` per leg with a TX/RX delta, and
# ARP_STRICT for the same-subnet case.
###############################################################################
set -u
###############################################################################
# User Configurable Section -- normally this is the only part you need to edit
###############################################################################
# --- 10G management port ---
IF_10G="eth0"
IP_10G="192.168.1.99"
TARGET_10G="192.168.1.30" # host this NIC pings
PLEN_10G=24 # prefix length, e.g. 24 = /24
GW_10G="" # default gateway; empty = do not touch routing
METRIC_10G=100 # lower metric wins for off-subnet traffic
# --- 1G management port ---
IF_1G="eth1"
IP_1G="192.168.1.101"
TARGET_1G="192.168.1.31" # host this NIC pings
PLEN_1G=24
GW_1G=""
METRIC_1G=200
# --- ping ---
PING_COUNT=10 # echo requests per leg
MAX_LOSS_PCT=0 # loss above this marks the leg FAIL
# Both NICs stay up. If they share a subnet the target's ARP can be answered
# by either one, so a reply may arrive on the NIC that did not send. Setting
# this applies arp_ignore=1 / arp_announce=2 to both, which keeps each NIC to
# its own address. RAM only -- reverts on reboot. Set 0 for separate subnets.
ARP_STRICT=1
# Prefix for the ip commands; empty because the daemon already runs as root.
SUDO=""
# Seconds to wait after bringing a link up before pinging, so ARP is resolved
# and the PHY has finished autoneg. The bench needed a few seconds on the 1G.
LINK_SETTLE_SEC=5
# Seconds between rounds (0 = start the next round immediately)
INTERVAL_SEC=10
# Per-ping-burst timeout in seconds; stops a wedged ping from freezing the
# loop. 0 disables. Must exceed PING_COUNT (one packet per second).
PING_TIMEOUT_SEC=60
# Stop after this many rounds. 0 = run until stopped manually.
MAX_ROUNDS=0
# Logging
LOG_DIR="" # empty = <script directory>/log
LOG_NAME="mgmt_ping.log"
MAX_LOG_SIZE_MB=100 # rotate above this size; 0 = never rotate
MAX_LOG_KEEP=5 # keep .1 .. .N
# What `start` does with the log left behind by the previous run:
# new = begin with an empty log; previous content and rotated copies are
# discarded (default -- one run, one log)
# archive = rename the previous log to <log>.YYYYmmdd-HHMMSS, then begin empty
# append = keep appending to the existing log
START_LOG_MODE="new"
###############################################################################
# Internal
###############################################################################
SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")"
SCRIPT_PATH="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)/${SCRIPT_NAME}"
[ -n "${LOG_DIR}" ] || LOG_DIR="$(dirname -- "${SCRIPT_PATH}")/log"
LOG_FILE="${LOG_DIR}/${LOG_NAME}"
PID_FILE="${LOG_DIR}/${LOG_NAME%.log}.pid"
LOG_TO_FILE=0
STOP=0
SLEEP_PID=""
ts() { date '+%Y-%m-%d %H:%M:%S'; }
die() { printf '[ERROR] %s\n' "$*" >&2; exit 1; }
on_signal() {
STOP=1
[ -n "${SLEEP_PID}" ] && kill "${SLEEP_PID}" 2>/dev/null
}
interruptible_sleep() {
local sec="$1"
[ "${sec}" -gt 0 ] 2>/dev/null || return 0
sleep "${sec}" &
SLEEP_PID=$!
wait "${SLEEP_PID}" 2>/dev/null
SLEEP_PID=""
}
is_running() {
local pid
[ -f "${PID_FILE}" ] || return 1
pid="$(cat "${PID_FILE}" 2>/dev/null)"
[[ "${pid}" =~ ^[0-9]+$ ]] || return 1
kill -0 "${pid}" 2>/dev/null || return 1
grep -qa "${SCRIPT_NAME}" "/proc/${pid}/cmdline" 2>/dev/null || return 1
printf '%s' "${pid}"
return 0
}
rotate_log_if_needed() {
local max_bytes size i
[ "${LOG_TO_FILE}" -eq 1 ] || return 0
[ "${MAX_LOG_SIZE_MB}" -gt 0 ] || return 0
[ -f "${LOG_FILE}" ] || return 0
max_bytes=$(( MAX_LOG_SIZE_MB * 1024 * 1024 ))
size="$(stat -c %s "${LOG_FILE}" 2>/dev/null || echo 0)"
[ "${size}" -ge "${max_bytes}" ] || return 0
for (( i = MAX_LOG_KEEP - 1; i >= 1; i-- )); do
[ -f "${LOG_FILE}.${i}" ] && mv -f "${LOG_FILE}.${i}" "${LOG_FILE}.$(( i + 1 ))"
done
mv -f "${LOG_FILE}" "${LOG_FILE}.1"
exec >>"${LOG_FILE}" 2>&1
printf '[%s] INFO : log rotated at %s MB\n' "$(ts)" "${MAX_LOG_SIZE_MB}"
}
remove_rotated_logs() {
local i
for (( i = 1; i <= MAX_LOG_KEEP + 1; i++ )); do
rm -f "${LOG_FILE}.${i}"
done
}
prepare_log_on_start() {
local stamp
case "${START_LOG_MODE}" in
append)
return 0
;;
archive)
if [ -s "${LOG_FILE}" ]; then
stamp="$(date '+%Y%m%d-%H%M%S')"
mv -f "${LOG_FILE}" "${LOG_FILE}.${stamp}" \
|| die "cannot archive ${LOG_FILE}"
printf 'previous log archived: %s.%s\n' "${LOG_FILE}" "${stamp}"
fi
remove_rotated_logs
;;
new)
# Truncate rather than unlink: the inode is preserved, so a tail -f
# that is already attached keeps following the new run.
if [ -f "${LOG_FILE}" ]; then
: > "${LOG_FILE}" || die "cannot truncate ${LOG_FILE}"
fi
remove_rotated_logs
;;
*)
die "invalid START_LOG_MODE: ${START_LOG_MODE}"
;;
esac
}
# Cumulative "<rx_packets> <tx_packets>" for an interface, from ip -s link.
# The numbers sit on the line after each RX:/TX: header, and the column order
# is "bytes packets errors ..." -- so packets is $2, not $1.
if_counters() {
${SUDO} ip -s link show "$1" 2>/dev/null | awk '
/RX:/ { getline; rx = $2 }
/TX:/ { getline; tx = $2 }
END { printf "%s %s", (rx == "" ? 0 : rx), (tx == "" ? 0 : tx) }'
}
# Bring the NIC up and (re)apply its address and default route. `replace` is
# idempotent, so repeated rounds do not stack duplicates.
setup_iface() {
local iface="$1" ipaddr="$2" plen="$3" gw="$4" metric="$5"
${SUDO} ip link set "${iface}" up 2>&1 || return 1
${SUDO} ip address replace "${ipaddr}/${plen}" dev "${iface}" 2>&1 || return 1
if [ -n "${gw}" ]; then
${SUDO} ip route replace default via "${gw}" dev "${iface}" \
metric "${metric}" 2>&1 || return 1
fi
return 0
}
# One leg: configure <if>, settle, ping its own target, report. The NIC is
# LEFT UP -- nothing is brought down between legs.
# Usage: ping_leg <label> <if> <ip> <plen> <gw> <metric> <target>
ping_leg() {
local label="$1" iface="$2" ipaddr="$3" plen="$4" gw="$5" metric="$6" target="$7"
local out rc tx rx loss avg verdict
local c0 c1 rx0 tx0 rx1 tx1 drx dtx
printf -- '----- [%s] %s : %s (%s/%s) -> %s -----\n' \
"$(ts)" "${label}" "${iface}" "${ipaddr}" "${plen}" "${target}"
if ! setup_iface "${iface}" "${ipaddr}" "${plen}" "${gw}" "${metric}"; then
printf '[%s] FAIL : cannot configure %s\n\n' "$(ts)" "${iface}"
return 1
fi
interruptible_sleep "${LINK_SETTLE_SEC}"
[ "${STOP}" -eq 0 ] || return 1
c0="$(if_counters "${iface}")"; rx0="${c0% *}"; tx0="${c0#* }"
if [ "${PING_TIMEOUT_SEC}" -gt 0 ]; then
out="$(timeout "${PING_TIMEOUT_SEC}" ping -I "${iface}" -c "${PING_COUNT}" "${target}" 2>&1)"
else
out="$(ping -I "${iface}" -c "${PING_COUNT}" "${target}" 2>&1)"
fi
rc=$?
printf '%s\n' "${out}"
c1="$(if_counters "${iface}")"; rx1="${c1% *}"; tx1="${c1#* }"
drx=$(( rx1 - rx0 )); dtx=$(( tx1 - tx0 ))
printf -- '--- ip -s link show %s ---\n' "${iface}"
${SUDO} ip -s link show "${iface}" 2>&1
# "10 packets transmitted, 10 received, 0% packet loss"
tx="$(printf '%s' "${out}" | sed -n 's/^\([0-9]\+\) packets transmitted.*/\1/p' | tail -1)"
rx="$(printf '%s' "${out}" | sed -n 's/.*[, ]\([0-9]\+\) received.*/\1/p' | tail -1)"
loss="$(printf '%s' "${out}" | sed -n 's/.*[, ]\([0-9]\+\)% packet loss.*/\1/p' | tail -1)"
# "rtt min/avg/max/mdev = 1.044/1.743/2.387/0.437 ms"
avg="$(printf '%s' "${out}" | sed -n 's|.*= [0-9.]*/\([0-9.]*\)/.*|\1|p' | tail -1)"
[ -n "${tx}" ] || tx=0
[ -n "${rx}" ] || rx=0
[ -n "${loss}" ] || loss=100
[ -n "${avg}" ] || avg="-"
if [ "${rc}" -eq 124 ]; then
verdict="FAIL(timeout)"
elif [ "${loss}" -le "${MAX_LOSS_PCT}" ]; then
verdict="PASS"
else
verdict="FAIL"
fi
# one greppable line per leg: grep RESULT <log>
# nic_tx/nic_rx are this interface's own counter delta across the burst --
# if they stay near zero while the ping succeeded, the traffic went out a
# different NIC and the result is not attributable to this one.
printf '[%s] RESULT %-3s %-6s -> %-15s tx=%s rx=%s loss=%s%% rtt_avg=%sms nic_tx=%s nic_rx=%s %s\n\n' \
"$(ts)" "${label}" "${iface}" "${target}" "${tx}" "${rx}" "${loss}" "${avg}" \
"${dtx}" "${drx}" "${verdict}"
[ "${verdict}" = "PASS" ]
}
# arp_ignore=1 : answer ARP only for addresses configured on the incoming NIC
# arp_announce=2: pick the source address from the outgoing NIC's own subnet
# Both are RAM-only sysctls and revert on reboot.
apply_arp_strict() {
local i
[ "${ARP_STRICT}" -eq 1 ] || return 0
for i in "${IF_10G}" "${IF_1G}"; do
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_ignore=1" 2>/dev/null
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_announce=2" 2>/dev/null
done
printf '[%s] INFO : arp_ignore=1 arp_announce=2 applied to %s,%s\n' \
"$(ts)" "${IF_10G}" "${IF_1G}"
}
main_loop() {
local round=0 pass=0 fail=0
command -v ip >/dev/null 2>&1 || die "ip (iproute2) not found"
command -v ping >/dev/null 2>&1 || die "ping not found"
[ "${PING_COUNT}" -gt 0 ] 2>/dev/null || die "PING_COUNT must be > 0"
trap on_signal INT TERM
printf '#############################################################\n'
printf '[%s] START: pid=%d count=%d settle=%ds interval=%ds\n' \
"$(ts)" "$$" "${PING_COUNT}" "${LINK_SETTLE_SEC}" "${INTERVAL_SEC}"
printf '[%s] 10G %s %s/%s -> %s (metric %s)\n' \
"$(ts)" "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}"
printf '[%s] 1G %s %s/%s -> %s (metric %s)\n' \
"$(ts)" "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}"
printf '#############################################################\n'
apply_arp_strict
while [ "${STOP}" -eq 0 ]; do
round=$(( round + 1 ))
printf -- '---------- ROUND %d @ %s ----------\n' "${round}" "$(ts)"
if ping_leg "10G" "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${GW_10G}" \
"${METRIC_10G}" "${TARGET_10G}"; then
pass=$(( pass + 1 )); else fail=$(( fail + 1 ))
fi
[ "${STOP}" -eq 0 ] || break
if ping_leg "1G" "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${GW_1G}" \
"${METRIC_1G}" "${TARGET_1G}"; then
pass=$(( pass + 1 )); else fail=$(( fail + 1 ))
fi
rotate_log_if_needed
if [ "${MAX_ROUNDS}" -gt 0 ] && [ "${round}" -ge "${MAX_ROUNDS}" ]; then
break
fi
[ "${STOP}" -eq 0 ] && interruptible_sleep "${INTERVAL_SEC}"
done
printf '[%s] STOP : pid=%d after %d round(s), legs PASS=%d FAIL=%d\n' \
"$(ts)" "$$" "${round}" "${pass}" "${fail}"
}
###############################################################################
# Sub-commands
###############################################################################
do_start() {
local pid
pid="$(is_running)" && die "already running (pid=${pid})"
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
prepare_log_on_start
if command -v setsid >/dev/null 2>&1; then
setsid "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 &
else
nohup "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 &
disown 2>/dev/null
fi
sleep 1
pid="$(is_running)" || die "start failed, check ${LOG_FILE}"
printf 'started (pid=%s)\nlog: %s\n' "${pid}" "${LOG_FILE}"
}
do_daemon() {
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
LOG_TO_FILE=1
exec >>"${LOG_FILE}" 2>&1
printf '%d\n' "$$" > "${PID_FILE}"
trap 'rm -f "${PID_FILE}"' EXIT
main_loop
}
do_stop() {
local pid i
pid="$(is_running)" || { printf 'not running\n'; rm -f "${PID_FILE}"; return 0; }
kill -TERM -"${pid}" 2>/dev/null || kill -TERM "${pid}" 2>/dev/null
for (( i = 0; i < 20; i++ )); do
kill -0 "${pid}" 2>/dev/null || break
sleep 0.5
done
if kill -0 "${pid}" 2>/dev/null; then
printf 'SIGTERM ignored, sending SIGKILL\n'
kill -KILL -"${pid}" 2>/dev/null || kill -KILL "${pid}" 2>/dev/null
fi
rm -f "${PID_FILE}"
printf 'stopped (pid=%s)\n' "${pid}"
}
do_status() {
local pid
if pid="$(is_running)"; then
printf 'status : running (pid=%s)\n' "${pid}"
else
printf 'status : stopped\n'
fi
printf 'log : %s\n' "${LOG_FILE}"
[ -f "${LOG_FILE}" ] && printf 'size : %s\n' "$(du -h "${LOG_FILE}" | cut -f1)"
printf 'ping : count=%d, interval=%ds, settle=%ds\n' "${PING_COUNT}" "${INTERVAL_SEC}" "${LINK_SETTLE_SEC}"
printf '10G : %s %s/%s -> %s (metric %s)\n' "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}"
printf '1G : %s %s/%s -> %s (metric %s)\n' "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}"
}
usage() {
cat <<EOF
Usage: ${SCRIPT_NAME} {start|stop|status|fg|tail|clear|summary}
start Run detached in the background; the previous log is discarded first
(see START_LOG_MODE)
stop Stop the background instance
status Show pid / log path / configured NICs
fg Run in the foreground, output to the terminal (for a quick check)
tail tail -f the log file (does NOT return -- do not call from a macro)
clear Remove the log file and all rotated copies (must be stopped first)
summary Print just the per-leg RESULT lines from the log
Log file : ${LOG_FILE}
One round = 10G leg then 1G leg, each pinging its own target. Both NICs are
brought up and left up; addresses and routes are applied with
"ip address/route replace", so repeating a round does not stack duplicates.
If both NICs share a subnet, keep ARP_STRICT=1 -- otherwise a reply can come
back on the NIC that did not send it. Check nic_tx/nic_rx on the RESULT line:
they are that interface's own counter delta across the burst.
Edit the User Configurable Section at the top to change interfaces, addresses,
targets, prefix lengths, gateways, metrics or the packet count.
EOF
}
###############################################################################
# Entry point
###############################################################################
case "${1:-}" in
start) do_start ;;
stop) do_stop ;;
status) do_status ;;
fg) LOG_TO_FILE=0; main_loop ;;
tail) tail -n 50 -f "${LOG_FILE}" ;;
summary) grep -a 'RESULT' "${LOG_FILE}" 2>/dev/null || printf 'no results in %s\n' "${LOG_FILE}" ;;
clear)
is_running >/dev/null && die "still running, stop it first"
rm -f "${LOG_FILE}" "${LOG_FILE}".[0-9]*
printf 'log cleared\n'
;;
__daemon) do_daemon ;;
-h|--help|"") usage ;;
*) printf '[ERROR] unknown sub-command: %s\n\n' "$1" >&2; usage; exit 1 ;;
esac
@@ -16,3 +16,8 @@ wait prompt_sonic_root
sendln "show platform leak status" sendln "show platform leak status"
wait prompt_sonic_root wait prompt_sonic_root
sendln "show platform leak channels" sendln "show platform leak channels"
wait prompt_sonic_root
sendln "tpm-dut-test fru"
wait prompt_sonic_root
sendln "tpm-dut-test tpm-read"
+8 -5
View File
@@ -1,8 +1,11 @@
; All Event
wait prompt_sonic_root wait prompt_sonic_root
sendln "date" sendln "dmesg -T | grep -Ei 'error|fail(ed|ure)?|fault|critical|panic|timeout'"
; i2c event
wait prompt_sonic_root wait prompt_sonic_root
sendln "dmesg | grep -i error" sendln "dmesg -T | grep -Ei 'error|fail(ed|ure)?|fault|critical|panic|timeout' | grep -i i2c"
; Clear Event
wait prompt_sonic_root wait prompt_sonic_root
sendln "dmesg | grep -i fail" sendln "dmesg -C"
wait prompt_sonic_root
sendln "dmesg | grep -i warning"