feat: Add mgmt_ping_monitor.sh and move stress onto bgctl
New Blanton_Script/mgmt_ping_monitor.sh: brings both management NICs up with iproute2 (`ip link set` / `address replace` / `route replace` with per-NIC metric) and pings each one's own target -- eth0 -> .30, eth1 -> .31 -- appending to a rotating log. Same sub-commands as the bmc monitors, plus `summary` for just the per-leg RESULT lines. Both NICs stay up, which on a shared subnet lets the target's ARP be answered by either one, so a reply can land on the NIC that did not send. ARP_STRICT applies arp_ignore/arp_announce to prevent that, and each leg logs `ip -s link show` with the interface's own TX/RX packet delta across the burst as direct evidence of which NIC carried the traffic. Note `ip -s link` orders columns "bytes packets ...", so the packet count is the second field. Script A: hw-test-session start/log/status, a fuller DUT inventory (version, fwutil, syseeprom, ssdhealth, TPM, nvme smart-log, smartctl), bmc-first-enroll and bmc-manager version/status, ras-mc-ctl summary, and both 100G uplinks now brought up rather than 513 being left down. Script B: stress moves to bgctl (memtester, qfx5252-stress-ssd/-usb) and the BMC DDR load runs through bmc-manager, replacing the hammer scripts and the bmc_monitor pair. Adds a BMC USB net test that discovers the cdc_ncm interface and runs a 4-hour ping under systemd-run. Script C: stops the bgctl jobs, the BMC USB unit and the mgmt ping, then collects journalctl, per-NIC counters, ras-mc-ctl and NVMe health, and closes the session with hw-test-session finish. utils/show_dmesg.ttl: one combined error regex with `dmesg -T`, an i2c-filtered view, then `dmesg -C` so Script C's capture shows only what the soak produced. utils/setup_pmon.ttl: add the TPM FRU/read checks. Fix: Script B set `timeout = 15` for the cdc_ncm probe and never restored it, leaving the cap in force for everything after -- including the traffic init, which walks 108 VLANs per unit and takes far longer than 15 s. A timed-out wait returns without the prompt, so the macro would have run ahead of the DUT for the rest of the soak. Reset to 0 at :skip_ping, where both branches meet. Script A -> V1.0.6. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
This commit is contained in:
Claude Opus 5
@@ -1,7 +1,7 @@
|
||||
; =============================================================================
|
||||
; Script A for Blanton
|
||||
; Version : V1.0.5
|
||||
; Date : 2026-08-21
|
||||
; Version : V1.0.6
|
||||
; Date : 2026-08-24
|
||||
; Author : ETWen
|
||||
; =============================================================================
|
||||
; Version History:
|
||||
@@ -25,6 +25,18 @@
|
||||
; V1.0.5 2026-08-21 config.ttl Add SWB_UNIT0/SWB_UNIT1
|
||||
; utils/wait_init.ttl Follow config.ttl SWB_UNIT0/SWB_UNIT1
|
||||
; ScriptA/B/C Traffic follows SWB_UNIT: -u 0 / -u 1 / skip
|
||||
; V1.0.6 2026-08-24 utils/show_dmesg.ttl Edited dmesg grep i2c & clear event
|
||||
; ScriptA HW Test Session
|
||||
; ScriptC HW Test Session Finish
|
||||
; ScriptA bmc-first-enroll, bmc version/status
|
||||
; ScriptB Stress test DDR, SSD, USB, BMC DDR, BMC USB Stress test
|
||||
; ScriptC BMC USB Stress test result
|
||||
; ScriptA DUT Info
|
||||
; ScriptAC 10G/1G MGMT ping
|
||||
; ScriptA Edited 100G Port Status
|
||||
; ScriptA, utils/setup_pmon.ttl Add TPM
|
||||
; ScriptAC NVME Error Info
|
||||
; ScriptAC Add PCIe Summary
|
||||
; =============================================================================
|
||||
include "config.ttl"
|
||||
|
||||
@@ -59,21 +71,50 @@ gettime timestr
|
||||
sprintf2 cmd 'sudo date -s "%s %s"' datestr timestr
|
||||
sendln cmd
|
||||
|
||||
; ========== Show Build Name ==========
|
||||
; ========== HW Test Session ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "show boot"
|
||||
|
||||
; ========== EEPROM Info ==========
|
||||
sendln "hw-test-session start"
|
||||
wait prompt_sonic_root
|
||||
sendln "hpe-eeprom-tlv show --bus 4 --addr 0x50"
|
||||
sendln "hw-test-session log"
|
||||
wait prompt_sonic_root
|
||||
sendln "hw-test-session status"
|
||||
|
||||
; ========== Wait DUT Init ==========
|
||||
wait prompt_sonic_root
|
||||
include "utils/wait_init.ttl"
|
||||
|
||||
; ========== BMC version ==========
|
||||
; ========== DUT Info ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "bmc-manager run 'cat /etc/issue'"
|
||||
sendln "show boot"
|
||||
wait prompt_sonic_root
|
||||
sendln "show version"
|
||||
wait prompt_sonic_root
|
||||
sendln "fwutil show status"
|
||||
wait prompt_sonic_root
|
||||
sendln "show system-memory"
|
||||
wait prompt_sonic_root
|
||||
sendln "show service"
|
||||
wait prompt_sonic_root
|
||||
sendln "show platform syseeprom"
|
||||
wait prompt_sonic_root
|
||||
sendln "show platform ssdhealth"
|
||||
wait prompt_sonic_root
|
||||
sendln "qfx5252-tpm-version"
|
||||
wait prompt_sonic_root
|
||||
sendln "/usr/sbin/nvme smart-log -H /dev/nvme0"
|
||||
wait prompt_sonic_root
|
||||
sendln "/usr/sbin/smartctl -x /dev/nvme0"
|
||||
|
||||
; ========== BMC version ==========
|
||||
;wait prompt_sonic_root
|
||||
;sendln "bmc-manager run 'cat /etc/issue'"
|
||||
wait prompt_sonic_root
|
||||
sendln "bmc-first-enroll"
|
||||
wait prompt_sonic_root
|
||||
sendln "bmc-manager version"
|
||||
wait prompt_sonic_root
|
||||
sendln "bmc-manager status"
|
||||
|
||||
|
||||
; ========== Load Shell Script ==========
|
||||
wait prompt_sonic_root
|
||||
@@ -98,6 +139,9 @@ include "utils/show_pcie_error_reg_DDR.ttl"
|
||||
include "utils/show_pcie_error_reg_I210.ttl"
|
||||
include "utils/show_pcie_error_reg_SSD.ttl"
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln "/usr/sbin/ras-mc-ctl --summary"
|
||||
|
||||
|
||||
; ========== Check PCIE tree + PCIE link status + GET PCIE bandwidth Test ==========
|
||||
include "utils/pcie_bus.ttl"
|
||||
@@ -115,12 +159,24 @@ include "utils/show_dmesg.ttl"
|
||||
;wait prompt_sonic_root
|
||||
;flushrecv
|
||||
|
||||
; ========== 10G/1G MGMT Ping Test ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "./Blanton_Script/mgmt_ping_monitor.sh start"
|
||||
wait prompt_sonic_root
|
||||
sendln "~/Blanton_Script/mgmt_ping_monitor.sh stop"
|
||||
wait prompt_sonic_root
|
||||
sendln "cat ~/Blanton_Script/log/mgmt_ping.log"
|
||||
|
||||
|
||||
; ========== 100G Port Status ==========
|
||||
; Lpmode Disable
|
||||
wait prompt_sonic_root
|
||||
sendln "sudo sfputil lpmode off Ethernet513"
|
||||
wait prompt_sonic_root
|
||||
sendln "sudo sfputil lpmode off Ethernet514"
|
||||
wait prompt_sonic_root
|
||||
sendln "config interface -n asic0 startup Ethernet513"
|
||||
wait prompt_sonic_root
|
||||
sendln "config interface -n asic0 shutdown Ethernet513"
|
||||
wait prompt_sonic_root
|
||||
sendln "config interface -n asic1 startup Ethernet514"
|
||||
wait prompt_sonic_root
|
||||
sendln "config save -y"
|
||||
|
||||
@@ -20,45 +20,68 @@ sendln 'chmod +x ~/hammer/tools/amd/mlucas-avx2'
|
||||
wait prompt_sonic_root
|
||||
sendln "~/hammer/tools/amd/mlucas-avx2 -s l -cpu 0:15 > ~/hammer/tools/amd/mlucas_amm_log 2>&1 &"
|
||||
|
||||
; ========== BMC Stress Test ==========
|
||||
; ========== BMC DDR Stress Test ==========
|
||||
;wait prompt_sonic_root
|
||||
;sendln "./Blanton_Script/bmc_monitor_ddr.sh start"
|
||||
;wait prompt_sonic_root
|
||||
;sendln "./Blanton_Script/bmc_monitor.sh start"
|
||||
wait prompt_sonic_root
|
||||
sendln "./Blanton_Script/bmc_monitor_ddr.sh start"
|
||||
wait prompt_sonic_root
|
||||
sendln "./Blanton_Script/bmc_monitor.sh start"
|
||||
|
||||
sendln "bgctl run bmc-manager run '/usr/bin/memtester 64M 1'"
|
||||
|
||||
; ========== DDR MEMORY STRESS TEST ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_mem.py &"
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_mem.py &"
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_mem.py &"
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_mem.py &"
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_mem.py &"
|
||||
sendln "bgctl run /usr/sbin/memtester 1G 100"
|
||||
|
||||
; ========== SSD read/write ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_ssd.py &"
|
||||
sendln "bgctl run qfx5252-stress-ssd 1G --runtime 3600 --passes 5 --write --force --log /host/hw-eval/current/qfx5252-stress-ssd.log"
|
||||
|
||||
; ========== USB read/write ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "python ~/hammer/tools/stress_usb.py &"
|
||||
sendln "bgctl run qfx5252-stress-usb /dev/sda1 256M --runtime 60 --passes 1 --write --log /host/hw-eval/qfx5252-stress-usb.log"
|
||||
|
||||
; ========== SHOW Background job ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "jobs"
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl list"
|
||||
|
||||
; ========== BMC USB Test ==========
|
||||
wait prompt_sonic_root
|
||||
sendln 'echo "NCMIF=$(ls -d /sys/bus/usb/drivers/cdc_ncm/*/net/* 2>/dev/null | head -n 1 | xargs -r basename)"'
|
||||
timeout = 15
|
||||
waitregex 'NCMIF=[A-Za-z0-9_-]+'
|
||||
if result = 0 then
|
||||
messagebox 'cdc_ncm interface not found. Ping test skipped.' 'ERROR'
|
||||
goto skip_ping
|
||||
endif
|
||||
strlen matchstr
|
||||
cut_len = result - 6 ; 'NCMIF=' = 6 chars
|
||||
strcopy matchstr 7 cut_len ncm_if ; -> ncm_if = "eth2" / "eth3"
|
||||
|
||||
sprintf2 cmd 'ip -br link show %s' ncm_if
|
||||
wait prompt_sonic_root
|
||||
sendln cmd
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln 'systemctl stop qfx5252-bmc-usb-net-test.service 2>/dev/null || true'
|
||||
wait prompt_sonic_root
|
||||
sendln 'systemctl reset-failed qfx5252-bmc-usb-net-test.service 2>/dev/null || true'
|
||||
sprintf2 cmd 'systemd-run --unit=qfx5252-bmc-usb-net-test --property=Type=exec /usr/bin/ping -I %s -i 1 -W 2 -w 14400 192.168.200.200' ncm_if
|
||||
wait prompt_sonic_root
|
||||
sendln cmd
|
||||
|
||||
:skip_ping
|
||||
; restore the default (no cap): the traffic init below walks 108 VLANs per
|
||||
; unit and takes far longer than the 15 s set for the NCM probe above.
|
||||
timeout = 0
|
||||
|
||||
; ========== 10G/1G MGMT Ping Test ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "./Blanton_Script/mgmt_ping_monitor.sh start"
|
||||
|
||||
|
||||
; ========== 100G Port ==========
|
||||
; Lpmode Disable
|
||||
wait prompt_sonic_root
|
||||
sendln "sudo sfputil lpmode off Ethernet513"
|
||||
wait prompt_sonic_root
|
||||
sendln "sudo sfputil lpmode off Ethernet514"
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln "show interfaces status Ethernet513,Ethernet514"
|
||||
|
||||
|
||||
@@ -12,6 +12,21 @@ pause 1
|
||||
|
||||
; ========== Kill Process ==========
|
||||
include "utils/kill_all_process.ttl"
|
||||
; bgctrl all stop
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl stop --all"
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl stop --all"
|
||||
wait prompt_sonic_root
|
||||
sendln "bgctl reset --yes"
|
||||
; BMC USB Test STOP
|
||||
wait prompt_sonic_root
|
||||
sendln "systemctl stop qfx5252-bmc-usb-net-test.service"
|
||||
; 10G/1G MGMT Test STOP
|
||||
wait prompt_sonic_root
|
||||
sendln "~/Blanton_Script/mgmt_ping_monitor.sh stop"
|
||||
|
||||
|
||||
; ========== TAKE DATA ==========
|
||||
include "utils/setup_pmon.ttl"
|
||||
include "utils/show_dmesg.ttl"
|
||||
@@ -25,22 +40,33 @@ include "utils/show_pcie_error_reg_DDR.ttl"
|
||||
include "utils/show_pcie_error_reg_I210.ttl"
|
||||
include "utils/show_pcie_error_reg_SSD.ttl"
|
||||
|
||||
; ========== BMC TAKE DATA ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "./Blanton_Script/bmc_monitor_ddr.sh stop"
|
||||
sendln "/usr/sbin/ras-mc-ctl --summary"
|
||||
|
||||
; ========== BMC DDR TAKE DATA ==========
|
||||
;wait prompt_sonic_root
|
||||
;sendln "./Blanton_Script/bmc_monitor_ddr.sh stop"
|
||||
;wait prompt_sonic_root
|
||||
;sendln "./Blanton_Script/bmc_monitor.sh stop"
|
||||
;wait prompt_sonic_root
|
||||
;sendln "cat ./Blanton_Script/log/bmc_poll.log"
|
||||
;wait prompt_sonic_root
|
||||
;sendln "cat ./Blanton_Script/log/bmc_ddr.log"
|
||||
|
||||
; ========== BMC USB Test result ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "./Blanton_Script/bmc_monitor.sh stop"
|
||||
sendln "journalctl -u qfx5252-bmc-usb-net-test.service --no-pager"
|
||||
|
||||
; ========== 10G/1G MGMT Test result ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "cat ./Blanton_Script/log/bmc_poll.log"
|
||||
sendln "ip -s link show eth0"
|
||||
wait prompt_sonic_root
|
||||
sendln "cat ./Blanton_Script/log/bmc_ddr.log"
|
||||
sendln "ip -s link show eth1"
|
||||
|
||||
|
||||
; ========== CHECK Stress results ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "cat ~/hammer/tools/amd/mlucas_amm_log"
|
||||
wait prompt_sonic_root
|
||||
sendln "cat ~/hammer/tools/log.stress_ssd"
|
||||
|
||||
; ========== CHECK Traffic counters ==========
|
||||
if SWB_UNIT0 = 1 && SWB_UNIT1 = 1 then
|
||||
@@ -62,6 +88,13 @@ else
|
||||
|
||||
endif
|
||||
|
||||
; ========== NVME Err Info ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "/usr/sbin/nvme smart-log -H /dev/nvme0"
|
||||
wait prompt_sonic_root
|
||||
sendln "/usr/sbin/smartctl -x /dev/nvme0"
|
||||
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln "show reboot-cause"
|
||||
|
||||
@@ -69,4 +102,8 @@ sendln "show reboot-cause"
|
||||
wait prompt_sonic_root
|
||||
sendln "show uptime"
|
||||
|
||||
; ========== HW Test Session ==========
|
||||
wait prompt_sonic_root
|
||||
sendln "hw-test-session finish"
|
||||
|
||||
messagebox 'GOOD JOB! Test Case DONE' 'teraterm'
|
||||
@@ -0,0 +1,463 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# mgmt_ping_monitor.sh
|
||||
#
|
||||
# Version : V1.0.0
|
||||
# Author : ETWen
|
||||
# Date : 20260824
|
||||
#
|
||||
# Purpose : Exercise both management NICs and ping each one's own target,
|
||||
# appending everything to a rotating log. One round is
|
||||
#
|
||||
# 10G (eth0) : ip link up + address/route replace -> settle
|
||||
# -> ping N x TARGET_10G -> ip -s link show
|
||||
# 1G (eth1) : ip link up + address/route replace -> settle
|
||||
# -> ping N x TARGET_1G -> ip -s link show
|
||||
#
|
||||
# Designed to run detached in the background, same sub-commands as
|
||||
# bmc_monitor.sh: start/stop/status/fg/tail/clear.
|
||||
#
|
||||
# Notes : - Both NICs are brought up and LEFT up. Addresses and default
|
||||
# routes are applied with iproute2 `replace`, which is idempotent:
|
||||
# re-running a round does not stack duplicate addresses or routes.
|
||||
# - The default routes carry different metrics (100 / 200), so the
|
||||
# lower one wins for off-subnet traffic. They do NOT decide which
|
||||
# NIC a ping leaves by: `ping -I <iface>` binds to the device.
|
||||
# - !! SAME-SUBNET WARNING !! With both NICs up on one subnet the
|
||||
# target ARPs for the address and Linux may answer from either
|
||||
# NIC, so replies can arrive on the wrong one and the result is
|
||||
# not attributable. ARP_STRICT=1 sets arp_ignore/arp_announce to
|
||||
# stop that. Leave it on unless the two NICs are on separate
|
||||
# subnets, as in the iproute2 example this was modelled on.
|
||||
# - Each leg logs `ip -s link show <iface>` and the TX/RX packet
|
||||
# delta across the burst, which is the direct evidence of which
|
||||
# NIC actually carried the traffic.
|
||||
# - The first pings after bringing a link up are lost to ARP, not to
|
||||
# the link: a bench capture showed 4x "Destination Host
|
||||
# Unreachable" on the first burst and 0% loss on the very next
|
||||
# one. LINK_SETTLE_SEC absorbs that; it is not packet loss.
|
||||
#
|
||||
# Version History
|
||||
# V1.0.0 20260824 Initial Version
|
||||
# V2.0.0 20260824 Switch from ifconfig to iproute2 (ip link / address
|
||||
# replace / route replace with per-NIC metric). Both
|
||||
# NICs now stay up -- no link down between legs. Add
|
||||
# `ip -s link show` per leg with a TX/RX delta, and
|
||||
# ARP_STRICT for the same-subnet case.
|
||||
###############################################################################
|
||||
|
||||
set -u
|
||||
|
||||
###############################################################################
|
||||
# User Configurable Section -- normally this is the only part you need to edit
|
||||
###############################################################################
|
||||
|
||||
# --- 10G management port ---
|
||||
IF_10G="eth0"
|
||||
IP_10G="192.168.1.99"
|
||||
TARGET_10G="192.168.1.30" # host this NIC pings
|
||||
PLEN_10G=24 # prefix length, e.g. 24 = /24
|
||||
GW_10G="" # default gateway; empty = do not touch routing
|
||||
METRIC_10G=100 # lower metric wins for off-subnet traffic
|
||||
|
||||
# --- 1G management port ---
|
||||
IF_1G="eth1"
|
||||
IP_1G="192.168.1.101"
|
||||
TARGET_1G="192.168.1.31" # host this NIC pings
|
||||
PLEN_1G=24
|
||||
GW_1G=""
|
||||
METRIC_1G=200
|
||||
|
||||
# --- ping ---
|
||||
PING_COUNT=10 # echo requests per leg
|
||||
MAX_LOSS_PCT=0 # loss above this marks the leg FAIL
|
||||
|
||||
# Both NICs stay up. If they share a subnet the target's ARP can be answered
|
||||
# by either one, so a reply may arrive on the NIC that did not send. Setting
|
||||
# this applies arp_ignore=1 / arp_announce=2 to both, which keeps each NIC to
|
||||
# its own address. RAM only -- reverts on reboot. Set 0 for separate subnets.
|
||||
ARP_STRICT=1
|
||||
|
||||
# Prefix for the ip commands; empty because the daemon already runs as root.
|
||||
SUDO=""
|
||||
|
||||
# Seconds to wait after bringing a link up before pinging, so ARP is resolved
|
||||
# and the PHY has finished autoneg. The bench needed a few seconds on the 1G.
|
||||
LINK_SETTLE_SEC=5
|
||||
|
||||
# Seconds between rounds (0 = start the next round immediately)
|
||||
INTERVAL_SEC=10
|
||||
|
||||
# Per-ping-burst timeout in seconds; stops a wedged ping from freezing the
|
||||
# loop. 0 disables. Must exceed PING_COUNT (one packet per second).
|
||||
PING_TIMEOUT_SEC=60
|
||||
|
||||
# Stop after this many rounds. 0 = run until stopped manually.
|
||||
MAX_ROUNDS=0
|
||||
|
||||
# Logging
|
||||
LOG_DIR="" # empty = <script directory>/log
|
||||
LOG_NAME="mgmt_ping.log"
|
||||
MAX_LOG_SIZE_MB=100 # rotate above this size; 0 = never rotate
|
||||
MAX_LOG_KEEP=5 # keep .1 .. .N
|
||||
|
||||
# What `start` does with the log left behind by the previous run:
|
||||
# new = begin with an empty log; previous content and rotated copies are
|
||||
# discarded (default -- one run, one log)
|
||||
# archive = rename the previous log to <log>.YYYYmmdd-HHMMSS, then begin empty
|
||||
# append = keep appending to the existing log
|
||||
START_LOG_MODE="new"
|
||||
|
||||
###############################################################################
|
||||
# Internal
|
||||
###############################################################################
|
||||
SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")"
|
||||
SCRIPT_PATH="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)/${SCRIPT_NAME}"
|
||||
[ -n "${LOG_DIR}" ] || LOG_DIR="$(dirname -- "${SCRIPT_PATH}")/log"
|
||||
LOG_FILE="${LOG_DIR}/${LOG_NAME}"
|
||||
PID_FILE="${LOG_DIR}/${LOG_NAME%.log}.pid"
|
||||
|
||||
LOG_TO_FILE=0
|
||||
STOP=0
|
||||
SLEEP_PID=""
|
||||
|
||||
ts() { date '+%Y-%m-%d %H:%M:%S'; }
|
||||
die() { printf '[ERROR] %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
on_signal() {
|
||||
STOP=1
|
||||
[ -n "${SLEEP_PID}" ] && kill "${SLEEP_PID}" 2>/dev/null
|
||||
}
|
||||
|
||||
interruptible_sleep() {
|
||||
local sec="$1"
|
||||
[ "${sec}" -gt 0 ] 2>/dev/null || return 0
|
||||
sleep "${sec}" &
|
||||
SLEEP_PID=$!
|
||||
wait "${SLEEP_PID}" 2>/dev/null
|
||||
SLEEP_PID=""
|
||||
}
|
||||
|
||||
is_running() {
|
||||
local pid
|
||||
[ -f "${PID_FILE}" ] || return 1
|
||||
pid="$(cat "${PID_FILE}" 2>/dev/null)"
|
||||
[[ "${pid}" =~ ^[0-9]+$ ]] || return 1
|
||||
kill -0 "${pid}" 2>/dev/null || return 1
|
||||
grep -qa "${SCRIPT_NAME}" "/proc/${pid}/cmdline" 2>/dev/null || return 1
|
||||
printf '%s' "${pid}"
|
||||
return 0
|
||||
}
|
||||
|
||||
rotate_log_if_needed() {
|
||||
local max_bytes size i
|
||||
[ "${LOG_TO_FILE}" -eq 1 ] || return 0
|
||||
[ "${MAX_LOG_SIZE_MB}" -gt 0 ] || return 0
|
||||
[ -f "${LOG_FILE}" ] || return 0
|
||||
|
||||
max_bytes=$(( MAX_LOG_SIZE_MB * 1024 * 1024 ))
|
||||
size="$(stat -c %s "${LOG_FILE}" 2>/dev/null || echo 0)"
|
||||
[ "${size}" -ge "${max_bytes}" ] || return 0
|
||||
|
||||
for (( i = MAX_LOG_KEEP - 1; i >= 1; i-- )); do
|
||||
[ -f "${LOG_FILE}.${i}" ] && mv -f "${LOG_FILE}.${i}" "${LOG_FILE}.$(( i + 1 ))"
|
||||
done
|
||||
mv -f "${LOG_FILE}" "${LOG_FILE}.1"
|
||||
exec >>"${LOG_FILE}" 2>&1
|
||||
printf '[%s] INFO : log rotated at %s MB\n' "$(ts)" "${MAX_LOG_SIZE_MB}"
|
||||
}
|
||||
|
||||
remove_rotated_logs() {
|
||||
local i
|
||||
for (( i = 1; i <= MAX_LOG_KEEP + 1; i++ )); do
|
||||
rm -f "${LOG_FILE}.${i}"
|
||||
done
|
||||
}
|
||||
|
||||
prepare_log_on_start() {
|
||||
local stamp
|
||||
case "${START_LOG_MODE}" in
|
||||
append)
|
||||
return 0
|
||||
;;
|
||||
archive)
|
||||
if [ -s "${LOG_FILE}" ]; then
|
||||
stamp="$(date '+%Y%m%d-%H%M%S')"
|
||||
mv -f "${LOG_FILE}" "${LOG_FILE}.${stamp}" \
|
||||
|| die "cannot archive ${LOG_FILE}"
|
||||
printf 'previous log archived: %s.%s\n' "${LOG_FILE}" "${stamp}"
|
||||
fi
|
||||
remove_rotated_logs
|
||||
;;
|
||||
new)
|
||||
# Truncate rather than unlink: the inode is preserved, so a tail -f
|
||||
# that is already attached keeps following the new run.
|
||||
if [ -f "${LOG_FILE}" ]; then
|
||||
: > "${LOG_FILE}" || die "cannot truncate ${LOG_FILE}"
|
||||
fi
|
||||
remove_rotated_logs
|
||||
;;
|
||||
*)
|
||||
die "invalid START_LOG_MODE: ${START_LOG_MODE}"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Cumulative "<rx_packets> <tx_packets>" for an interface, from ip -s link.
|
||||
# The numbers sit on the line after each RX:/TX: header, and the column order
|
||||
# is "bytes packets errors ..." -- so packets is $2, not $1.
|
||||
if_counters() {
|
||||
${SUDO} ip -s link show "$1" 2>/dev/null | awk '
|
||||
/RX:/ { getline; rx = $2 }
|
||||
/TX:/ { getline; tx = $2 }
|
||||
END { printf "%s %s", (rx == "" ? 0 : rx), (tx == "" ? 0 : tx) }'
|
||||
}
|
||||
|
||||
# Bring the NIC up and (re)apply its address and default route. `replace` is
|
||||
# idempotent, so repeated rounds do not stack duplicates.
|
||||
setup_iface() {
|
||||
local iface="$1" ipaddr="$2" plen="$3" gw="$4" metric="$5"
|
||||
|
||||
${SUDO} ip link set "${iface}" up 2>&1 || return 1
|
||||
${SUDO} ip address replace "${ipaddr}/${plen}" dev "${iface}" 2>&1 || return 1
|
||||
if [ -n "${gw}" ]; then
|
||||
${SUDO} ip route replace default via "${gw}" dev "${iface}" \
|
||||
metric "${metric}" 2>&1 || return 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# One leg: configure <if>, settle, ping its own target, report. The NIC is
|
||||
# LEFT UP -- nothing is brought down between legs.
|
||||
# Usage: ping_leg <label> <if> <ip> <plen> <gw> <metric> <target>
|
||||
ping_leg() {
|
||||
local label="$1" iface="$2" ipaddr="$3" plen="$4" gw="$5" metric="$6" target="$7"
|
||||
local out rc tx rx loss avg verdict
|
||||
local c0 c1 rx0 tx0 rx1 tx1 drx dtx
|
||||
|
||||
printf -- '----- [%s] %s : %s (%s/%s) -> %s -----\n' \
|
||||
"$(ts)" "${label}" "${iface}" "${ipaddr}" "${plen}" "${target}"
|
||||
|
||||
if ! setup_iface "${iface}" "${ipaddr}" "${plen}" "${gw}" "${metric}"; then
|
||||
printf '[%s] FAIL : cannot configure %s\n\n' "$(ts)" "${iface}"
|
||||
return 1
|
||||
fi
|
||||
interruptible_sleep "${LINK_SETTLE_SEC}"
|
||||
[ "${STOP}" -eq 0 ] || return 1
|
||||
|
||||
c0="$(if_counters "${iface}")"; rx0="${c0% *}"; tx0="${c0#* }"
|
||||
|
||||
if [ "${PING_TIMEOUT_SEC}" -gt 0 ]; then
|
||||
out="$(timeout "${PING_TIMEOUT_SEC}" ping -I "${iface}" -c "${PING_COUNT}" "${target}" 2>&1)"
|
||||
else
|
||||
out="$(ping -I "${iface}" -c "${PING_COUNT}" "${target}" 2>&1)"
|
||||
fi
|
||||
rc=$?
|
||||
printf '%s\n' "${out}"
|
||||
|
||||
c1="$(if_counters "${iface}")"; rx1="${c1% *}"; tx1="${c1#* }"
|
||||
drx=$(( rx1 - rx0 )); dtx=$(( tx1 - tx0 ))
|
||||
|
||||
printf -- '--- ip -s link show %s ---\n' "${iface}"
|
||||
${SUDO} ip -s link show "${iface}" 2>&1
|
||||
|
||||
# "10 packets transmitted, 10 received, 0% packet loss"
|
||||
tx="$(printf '%s' "${out}" | sed -n 's/^\([0-9]\+\) packets transmitted.*/\1/p' | tail -1)"
|
||||
rx="$(printf '%s' "${out}" | sed -n 's/.*[, ]\([0-9]\+\) received.*/\1/p' | tail -1)"
|
||||
loss="$(printf '%s' "${out}" | sed -n 's/.*[, ]\([0-9]\+\)% packet loss.*/\1/p' | tail -1)"
|
||||
# "rtt min/avg/max/mdev = 1.044/1.743/2.387/0.437 ms"
|
||||
avg="$(printf '%s' "${out}" | sed -n 's|.*= [0-9.]*/\([0-9.]*\)/.*|\1|p' | tail -1)"
|
||||
|
||||
[ -n "${tx}" ] || tx=0
|
||||
[ -n "${rx}" ] || rx=0
|
||||
[ -n "${loss}" ] || loss=100
|
||||
[ -n "${avg}" ] || avg="-"
|
||||
|
||||
if [ "${rc}" -eq 124 ]; then
|
||||
verdict="FAIL(timeout)"
|
||||
elif [ "${loss}" -le "${MAX_LOSS_PCT}" ]; then
|
||||
verdict="PASS"
|
||||
else
|
||||
verdict="FAIL"
|
||||
fi
|
||||
|
||||
# one greppable line per leg: grep RESULT <log>
|
||||
# nic_tx/nic_rx are this interface's own counter delta across the burst --
|
||||
# if they stay near zero while the ping succeeded, the traffic went out a
|
||||
# different NIC and the result is not attributable to this one.
|
||||
printf '[%s] RESULT %-3s %-6s -> %-15s tx=%s rx=%s loss=%s%% rtt_avg=%sms nic_tx=%s nic_rx=%s %s\n\n' \
|
||||
"$(ts)" "${label}" "${iface}" "${target}" "${tx}" "${rx}" "${loss}" "${avg}" \
|
||||
"${dtx}" "${drx}" "${verdict}"
|
||||
|
||||
[ "${verdict}" = "PASS" ]
|
||||
}
|
||||
|
||||
# arp_ignore=1 : answer ARP only for addresses configured on the incoming NIC
|
||||
# arp_announce=2: pick the source address from the outgoing NIC's own subnet
|
||||
# Both are RAM-only sysctls and revert on reboot.
|
||||
apply_arp_strict() {
|
||||
local i
|
||||
[ "${ARP_STRICT}" -eq 1 ] || return 0
|
||||
for i in "${IF_10G}" "${IF_1G}"; do
|
||||
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_ignore=1" 2>/dev/null
|
||||
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_announce=2" 2>/dev/null
|
||||
done
|
||||
printf '[%s] INFO : arp_ignore=1 arp_announce=2 applied to %s,%s\n' \
|
||||
"$(ts)" "${IF_10G}" "${IF_1G}"
|
||||
}
|
||||
|
||||
main_loop() {
|
||||
local round=0 pass=0 fail=0
|
||||
|
||||
command -v ip >/dev/null 2>&1 || die "ip (iproute2) not found"
|
||||
command -v ping >/dev/null 2>&1 || die "ping not found"
|
||||
[ "${PING_COUNT}" -gt 0 ] 2>/dev/null || die "PING_COUNT must be > 0"
|
||||
|
||||
trap on_signal INT TERM
|
||||
|
||||
printf '#############################################################\n'
|
||||
printf '[%s] START: pid=%d count=%d settle=%ds interval=%ds\n' \
|
||||
"$(ts)" "$$" "${PING_COUNT}" "${LINK_SETTLE_SEC}" "${INTERVAL_SEC}"
|
||||
printf '[%s] 10G %s %s/%s -> %s (metric %s)\n' \
|
||||
"$(ts)" "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}"
|
||||
printf '[%s] 1G %s %s/%s -> %s (metric %s)\n' \
|
||||
"$(ts)" "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}"
|
||||
printf '#############################################################\n'
|
||||
|
||||
apply_arp_strict
|
||||
|
||||
while [ "${STOP}" -eq 0 ]; do
|
||||
round=$(( round + 1 ))
|
||||
printf -- '---------- ROUND %d @ %s ----------\n' "${round}" "$(ts)"
|
||||
|
||||
if ping_leg "10G" "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${GW_10G}" \
|
||||
"${METRIC_10G}" "${TARGET_10G}"; then
|
||||
pass=$(( pass + 1 )); else fail=$(( fail + 1 ))
|
||||
fi
|
||||
[ "${STOP}" -eq 0 ] || break
|
||||
|
||||
if ping_leg "1G" "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${GW_1G}" \
|
||||
"${METRIC_1G}" "${TARGET_1G}"; then
|
||||
pass=$(( pass + 1 )); else fail=$(( fail + 1 ))
|
||||
fi
|
||||
|
||||
rotate_log_if_needed
|
||||
|
||||
if [ "${MAX_ROUNDS}" -gt 0 ] && [ "${round}" -ge "${MAX_ROUNDS}" ]; then
|
||||
break
|
||||
fi
|
||||
[ "${STOP}" -eq 0 ] && interruptible_sleep "${INTERVAL_SEC}"
|
||||
done
|
||||
|
||||
printf '[%s] STOP : pid=%d after %d round(s), legs PASS=%d FAIL=%d\n' \
|
||||
"$(ts)" "$$" "${round}" "${pass}" "${fail}"
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Sub-commands
|
||||
###############################################################################
|
||||
do_start() {
|
||||
local pid
|
||||
pid="$(is_running)" && die "already running (pid=${pid})"
|
||||
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
|
||||
prepare_log_on_start
|
||||
|
||||
if command -v setsid >/dev/null 2>&1; then
|
||||
setsid "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 &
|
||||
else
|
||||
nohup "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 &
|
||||
disown 2>/dev/null
|
||||
fi
|
||||
|
||||
sleep 1
|
||||
pid="$(is_running)" || die "start failed, check ${LOG_FILE}"
|
||||
printf 'started (pid=%s)\nlog: %s\n' "${pid}" "${LOG_FILE}"
|
||||
}
|
||||
|
||||
do_daemon() {
|
||||
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
|
||||
LOG_TO_FILE=1
|
||||
exec >>"${LOG_FILE}" 2>&1
|
||||
printf '%d\n' "$$" > "${PID_FILE}"
|
||||
trap 'rm -f "${PID_FILE}"' EXIT
|
||||
main_loop
|
||||
}
|
||||
|
||||
do_stop() {
|
||||
local pid i
|
||||
pid="$(is_running)" || { printf 'not running\n'; rm -f "${PID_FILE}"; return 0; }
|
||||
|
||||
kill -TERM -"${pid}" 2>/dev/null || kill -TERM "${pid}" 2>/dev/null
|
||||
for (( i = 0; i < 20; i++ )); do
|
||||
kill -0 "${pid}" 2>/dev/null || break
|
||||
sleep 0.5
|
||||
done
|
||||
if kill -0 "${pid}" 2>/dev/null; then
|
||||
printf 'SIGTERM ignored, sending SIGKILL\n'
|
||||
kill -KILL -"${pid}" 2>/dev/null || kill -KILL "${pid}" 2>/dev/null
|
||||
fi
|
||||
rm -f "${PID_FILE}"
|
||||
printf 'stopped (pid=%s)\n' "${pid}"
|
||||
}
|
||||
|
||||
do_status() {
|
||||
local pid
|
||||
if pid="$(is_running)"; then
|
||||
printf 'status : running (pid=%s)\n' "${pid}"
|
||||
else
|
||||
printf 'status : stopped\n'
|
||||
fi
|
||||
printf 'log : %s\n' "${LOG_FILE}"
|
||||
[ -f "${LOG_FILE}" ] && printf 'size : %s\n' "$(du -h "${LOG_FILE}" | cut -f1)"
|
||||
printf 'ping : count=%d, interval=%ds, settle=%ds\n' "${PING_COUNT}" "${INTERVAL_SEC}" "${LINK_SETTLE_SEC}"
|
||||
printf '10G : %s %s/%s -> %s (metric %s)\n' "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}"
|
||||
printf '1G : %s %s/%s -> %s (metric %s)\n' "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}"
|
||||
}
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: ${SCRIPT_NAME} {start|stop|status|fg|tail|clear|summary}
|
||||
|
||||
start Run detached in the background; the previous log is discarded first
|
||||
(see START_LOG_MODE)
|
||||
stop Stop the background instance
|
||||
status Show pid / log path / configured NICs
|
||||
fg Run in the foreground, output to the terminal (for a quick check)
|
||||
tail tail -f the log file (does NOT return -- do not call from a macro)
|
||||
clear Remove the log file and all rotated copies (must be stopped first)
|
||||
summary Print just the per-leg RESULT lines from the log
|
||||
|
||||
Log file : ${LOG_FILE}
|
||||
|
||||
One round = 10G leg then 1G leg, each pinging its own target. Both NICs are
|
||||
brought up and left up; addresses and routes are applied with
|
||||
"ip address/route replace", so repeating a round does not stack duplicates.
|
||||
|
||||
If both NICs share a subnet, keep ARP_STRICT=1 -- otherwise a reply can come
|
||||
back on the NIC that did not send it. Check nic_tx/nic_rx on the RESULT line:
|
||||
they are that interface's own counter delta across the burst.
|
||||
|
||||
Edit the User Configurable Section at the top to change interfaces, addresses,
|
||||
targets, prefix lengths, gateways, metrics or the packet count.
|
||||
EOF
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Entry point
|
||||
###############################################################################
|
||||
case "${1:-}" in
|
||||
start) do_start ;;
|
||||
stop) do_stop ;;
|
||||
status) do_status ;;
|
||||
fg) LOG_TO_FILE=0; main_loop ;;
|
||||
tail) tail -n 50 -f "${LOG_FILE}" ;;
|
||||
summary) grep -a 'RESULT' "${LOG_FILE}" 2>/dev/null || printf 'no results in %s\n' "${LOG_FILE}" ;;
|
||||
clear)
|
||||
is_running >/dev/null && die "still running, stop it first"
|
||||
rm -f "${LOG_FILE}" "${LOG_FILE}".[0-9]*
|
||||
printf 'log cleared\n'
|
||||
;;
|
||||
__daemon) do_daemon ;;
|
||||
-h|--help|"") usage ;;
|
||||
*) printf '[ERROR] unknown sub-command: %s\n\n' "$1" >&2; usage; exit 1 ;;
|
||||
esac
|
||||
@@ -16,3 +16,8 @@ wait prompt_sonic_root
|
||||
sendln "show platform leak status"
|
||||
wait prompt_sonic_root
|
||||
sendln "show platform leak channels"
|
||||
|
||||
wait prompt_sonic_root
|
||||
sendln "tpm-dut-test fru"
|
||||
wait prompt_sonic_root
|
||||
sendln "tpm-dut-test tpm-read"
|
||||
@@ -1,8 +1,11 @@
|
||||
; All Event
|
||||
wait prompt_sonic_root
|
||||
sendln "date"
|
||||
sendln "dmesg -T | grep -Ei 'error|fail(ed|ure)?|fault|critical|panic|timeout'"
|
||||
|
||||
; i2c event
|
||||
wait prompt_sonic_root
|
||||
sendln "dmesg | grep -i error"
|
||||
sendln "dmesg -T | grep -Ei 'error|fail(ed|ure)?|fault|critical|panic|timeout' | grep -i i2c"
|
||||
|
||||
; Clear Event
|
||||
wait prompt_sonic_root
|
||||
sendln "dmesg | grep -i fail"
|
||||
wait prompt_sonic_root
|
||||
sendln "dmesg | grep -i warning"
|
||||
sendln "dmesg -C"
|
||||
|
||||
Reference in New Issue
Block a user