etwen/Blanton_TTL_Script
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

feat(mgmt): Ping both NICs at once, and stop counting ARP as loss

Browse Source 
The bench run reported FAIL=3 with no NIC errors and no drops. All three
failures were on eth1 and all three were missing exactly icmp_seq=1,
with the other nine replies present and sub-millisecond. That is
neighbour resolution: once the ARP entry for the target expires, the
first echo request is spent resolving it and ping counts it as loss.
Rounds sat about 24 s apart, right on the edge of the default reachable
time, so it happened on some rounds and not others.

mgmt_ping_monitor.sh V3.0.0:

- One discarded ping per NIC before the measured run resolves ARP, so
  MAX_LOSS_PCT can stay at 0 and a FAIL means a real drop. Raising the
  threshold instead would have hidden genuine single-packet loss, and a
  longer settle would not help -- nothing triggers ARP until the ping.
- Both NICs now ping at the same time and keep accumulating until
  stopped, rather than alternating fixed bursts. Each writes its own raw
  capture.
- `stop` sends SIGINT rather than SIGTERM, because ping prints its
  statistics block on interrupt and that block is what the report
  parses, then renders into the log: the last TAIL_LINES entries per
  NIC, each one's statistics and verdict, and `ip -s link show` for both
  interfaces.
- `-D -O` are on by default so every line carries a timestamp and a
  request that got no reply is visible in the tail instead of merely
  absent. Clear PING_EXTRA_OPTS if the platform's ping rejects them.

Since the script no longer re-execs itself to daemonise, `bash
mgmt_ping_monitor.sh start` works without the execute bit.

.gitignore: the new *.raw / *.pid / *.state runtime files were outside
the *.log rule and would have been committed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
This commit is contained in:
etwen
2026-08-26 09:53:20 +08:00
co-authored by Claude Opus 5
parent 1b8d5a8e91
commit a31c186142
3 changed files with 276 additions and 327 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+5
View File
@@ -12,6 +12,11 @@ For_AI/
!**/logs/.gitkeep !**/logs/.gitkeep
!**/Logs/.gitkeep !**/Logs/.gitkeep
# monitor 腳本的執行期產物（原始擷取、pid、狀態）
*.raw
*.pid
*.state
# 打包輸出：產出不進 git，打包工具本身要進 # 打包輸出：產出不進 git，打包工具本身要進
publish/* publish/*
!publish/.gitkeep !publish/.gitkeep
src/Script_ABC_Blanton/1_Blanton_Script_A.ttl
+4
View File
@@ -51,6 +51,10 @@
; ScriptB USB stress uses the detected node, not a hard-coded /dev/sda1 ; ScriptB USB stress uses the detected node, not a hard-coded /dev/sda1
; ScriptB DDR memtester runs continuously (drop the 100-pass limit) ; ScriptB DDR memtester runs continuously (drop the 100-pass limit)
; ScriptC Move bgctl reset --yes after the USB archive ; ScriptC Move bgctl reset --yes after the USB archive
; Blanton_Script/mgmt_ping_monitor.sh V3.0.0 - both NICs ping simultaneously, accumulating
; + ARP warm-up so a stale neighbour entry is not counted as loss
; + stop renders last-N per NIC, stats, and both ip -s link show
; .gitignore Ignore monitor *.raw / *.pid / *.state
; ============================================================================= ; =============================================================================
include "config.ttl" include "config.ttl"
src/Script_ABC_Blanton/Blanton_Script/mgmt_ping_monitor.sh
+267 -327
View File
@@ -2,48 +2,41 @@
############################################################################### ###############################################################################
# mgmt_ping_monitor.sh # mgmt_ping_monitor.sh
# #
# Version : V1.0.0 # Version : V3.0.0
# Author : ETWen # Author : ETWen
# Date : 20260824 # Date : 20260826
# #
# Purpose : Exercise both management NICs and ping each one's own target, # Purpose : Ping from BOTH management NICs at the same time, continuously, and
# appending everything to a rotating log. One round is # keep accumulating until stopped:
# #
# 10G (eth0) : ip link up + address/route replace -> settle # 10G (eth0) -> TARGET_10G
# -> ping N x TARGET_10G -> ip -s link show # 1G (eth1) -> TARGET_1G
# 1G (eth1) : ip link up + address/route replace -> settle
# -> ping N x TARGET_1G -> ip -s link show
# #
# Designed to run detached in the background, same sub-commands as # `stop` renders a report into the log: the last TAIL_LINES entries
# bmc_monitor.sh: start/stop/status/fg/tail/clear. # per NIC, each one's ping statistics and verdict, then
# `ip -s link show` for both interfaces.
# #
# Notes : - Both NICs are brought up and LEFT up. Addresses and default # Notes : - Both NICs are configured with iproute2 `replace`, which is
# routes are applied with iproute2 `replace`, which is idempotent: # idempotent, and both are left up. On a shared subnet the target's
# re-running a round does not stack duplicate addresses or routes. # ARP can be answered by either NIC, so ARP_STRICT keeps each one
# - The default routes carry different metrics (100 / 200), so the # to its own address.
# lower one wins for off-subnet traffic. They do NOT decide which # - ARP WARM-UP: the first packet after a neighbour entry expires is
# NIC a ping leaves by: `ping -I <iface>` binds to the device. # spent resolving ARP and is counted as loss. On the bench this
# - !! SAME-SUBNET WARNING !! With both NICs up on one subnet the # produced three FAILs whose only missing packet was icmp_seq=1,
# target ARPs for the address and Linux may answer from either # every time, with zero NIC errors or drops. One discarded ping
# NIC, so replies can arrive on the wrong one and the result is # before the measured run removes that artefact, which is what lets
# not attributable. ARP_STRICT=1 sets arp_ignore/arp_announce to # MAX_LOSS_PCT stay at 0 and still mean something.
# stop that. Leave it on unless the two NICs are on separate # - MANAGEMENT CONNECTIVITY IS IN USE while this runs. Drive it from
# subnets, as in the iproute2 example this was modelled on. # the serial console.
# - Each leg logs `ip -s link show <iface>` and the TX/RX packet
# delta across the burst, which is the direct evidence of which
# NIC actually carried the traffic.
# - The first pings after bringing a link up are lost to ARP, not to
# the link: a bench capture showed 4x "Destination Host
# Unreachable" on the first burst and 0% loss on the very next
# one. LINK_SETTLE_SEC absorbs that; it is not packet loss.
# #
# Version History # Version History
# V1.0.0 20260824 Initial Version # V1.0.0 20260824 Initial Version (ifconfig, one NIC at a time)
# V2.0.0 20260824 Switch from ifconfig to iproute2 (ip link / address # V2.0.0 20260824 iproute2; both NICs stay up; per-NIC target;
# replace / route replace with per-NIC metric). Both # ip -s link show + TX/RX delta per leg
# NICs now stay up -- no link down between legs. Add # V3.0.0 20260826 Both NICs ping SIMULTANEOUSLY and continuously instead
# `ip -s link show` per leg with a TX/RX delta, and # of alternating fixed bursts. Add the ARP warm-up.
# ARP_STRICT for the same-subnet case. # `stop` renders last-N per NIC + statistics + both
# ip -s link outputs into the log.
############################################################################### ###############################################################################
set -u set -u
@@ -63,149 +56,67 @@ METRIC_10G=100 # lower metric wins for off-subnet traffic
# --- 1G management port --- # --- 1G management port ---
IF_1G="eth1" IF_1G="eth1"
IP_1G="192.168.1.101" IP_1G="192.168.1.101"
TARGET_1G="192.168.1.31" # host this NIC pings TARGET_1G="192.168.1.31"
PLEN_1G=24 PLEN_1G=24
GW_1G="" GW_1G=""
METRIC_1G=200 METRIC_1G=200
# --- ping --- # --- ping ---
PING_COUNT=10 # echo requests per leg PING_INTERVAL_SEC=1 # seconds between echo requests (ping -i)
MAX_LOSS_PCT=0 # loss above this marks the leg FAIL MAX_LOSS_PCT=0 # loss above this marks the NIC FAIL
TAIL_LINES=20 # how many recent entries per NIC the report shows
# Both NICs stay up. If they share a subnet the target's ARP can be answered # Extra ping flags. -D timestamps every line, -O prints a marker for a request
# by either one, so a reply may arrive on the NIC that did not send. Setting # that got no reply, so a drop is visible in the tail instead of just missing.
# this applies arp_ignore=1 / arp_announce=2 to both, which keeps each NIC to # Clear this if the platform's ping does not accept them.
# its own address. RAM only -- reverts on reboot. Set 0 for separate subnets. PING_EXTRA_OPTS="-D -O"
# One discarded ping per NIC before the measured run, to resolve ARP. Without
# it the first packet of the run is lost to neighbour resolution and looks
# exactly like a link fault. 0 disables.
ARP_WARMUP=1
ARP_WARMUP_TIMEOUT=2 # seconds to wait for the warm-up reply
# Both NICs stay up. If they share a subnet the target's ARP can be answered by
# either one, so a reply may arrive on the NIC that did not send. This applies
# arp_ignore=1 / arp_announce=2 to both. RAM only -- reverts on reboot.
ARP_STRICT=1 ARP_STRICT=1
# Prefix for the ip commands; empty because the daemon already runs as root. # Prefix for the ip commands; empty because this already runs as root.
SUDO="" SUDO=""
# Seconds to wait after bringing a link up before pinging, so ARP is resolved # Seconds to wait after bringing a link up before pinging.
# and the PHY has finished autoneg. The bench needed a few seconds on the 1G.
LINK_SETTLE_SEC=5 LINK_SETTLE_SEC=5
# Seconds between rounds (0 = start the next round immediately)
INTERVAL_SEC=10
# Per-ping-burst timeout in seconds; stops a wedged ping from freezing the
# loop. 0 disables. Must exceed PING_COUNT (one packet per second).
PING_TIMEOUT_SEC=60
# Stop after this many rounds. 0 = run until stopped manually.
MAX_ROUNDS=0
# Logging # Logging
LOG_DIR="" # empty = <script directory>/log LOG_DIR="" # empty = <script directory>/log
LOG_NAME="mgmt_ping.log" LOG_NAME="mgmt_ping.log"
MAX_LOG_SIZE_MB=100 # rotate above this size; 0 = never rotate
MAX_LOG_KEEP=5 # keep .1 .. .N
# What `start` does with the log left behind by the previous run: # What `start` does with the log left behind by the previous run:
# new = begin with an empty log; previous content and rotated copies are # new = discard it (default -- one run, one log)
# discarded (default -- one run, one log) # archive = rename it to <log>.YYYYmmdd-HHMMSS first
# archive = rename the previous log to <log>.YYYYmmdd-HHMMSS, then begin empty # append = keep it
# append = keep appending to the existing log
START_LOG_MODE="new" START_LOG_MODE="new"
############################################################################### ###############################################################################
# Internal # Internal
############################################################################### ###############################################################################
SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")" SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")"
SCRIPT_PATH="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)/${SCRIPT_NAME}" SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
[ -n "${LOG_DIR}" ] || LOG_DIR="$(dirname -- "${SCRIPT_PATH}")/log" [ -n "${LOG_DIR}" ] || LOG_DIR="${SCRIPT_DIR}/log"
LOG_FILE="${LOG_DIR}/${LOG_NAME}"
PID_FILE="${LOG_DIR}/${LOG_NAME%.log}.pid"
LOG_TO_FILE=0 LOG_FILE="${LOG_DIR}/${LOG_NAME}"
STOP=0 BASE="${LOG_DIR}/${LOG_NAME%.log}"
SLEEP_PID="" PID_FILE="${BASE}.pid"
STATE_FILE="${BASE}.state"
RAW_10G="${BASE}_${IF_10G}.raw"
RAW_1G="${BASE}_${IF_1G}.raw"
ts() { date '+%Y-%m-%d %H:%M:%S'; } ts() { date '+%Y-%m-%d %H:%M:%S'; }
die() { printf '[ERROR] %s\n' "$*" >&2; exit 1; } die() { printf '[ERROR] %s\n' "$*" >&2; exit 1; }
on_signal() { # "<rx_packets> <tx_packets>" for an interface. ip -s link orders the columns
STOP=1 # "bytes packets errors ...", so packets is $2, not $1.
[ -n "${SLEEP_PID}" ] && kill "${SLEEP_PID}" 2>/dev/null
}
interruptible_sleep() {
local sec="$1"
[ "${sec}" -gt 0 ] 2>/dev/null || return 0
sleep "${sec}" &
SLEEP_PID=$!
wait "${SLEEP_PID}" 2>/dev/null
SLEEP_PID=""
}
is_running() {
local pid
[ -f "${PID_FILE}" ] || return 1
pid="$(cat "${PID_FILE}" 2>/dev/null)"
[[ "${pid}" =~ ^[0-9]+$ ]] || return 1
kill -0 "${pid}" 2>/dev/null || return 1
grep -qa "${SCRIPT_NAME}" "/proc/${pid}/cmdline" 2>/dev/null || return 1
printf '%s' "${pid}"
return 0
}
rotate_log_if_needed() {
local max_bytes size i
[ "${LOG_TO_FILE}" -eq 1 ] || return 0
[ "${MAX_LOG_SIZE_MB}" -gt 0 ] || return 0
[ -f "${LOG_FILE}" ] || return 0
max_bytes=$(( MAX_LOG_SIZE_MB * 1024 * 1024 ))
size="$(stat -c %s "${LOG_FILE}" 2>/dev/null || echo 0)"
[ "${size}" -ge "${max_bytes}" ] || return 0
for (( i = MAX_LOG_KEEP - 1; i >= 1; i-- )); do
[ -f "${LOG_FILE}.${i}" ] && mv -f "${LOG_FILE}.${i}" "${LOG_FILE}.$(( i + 1 ))"
done
mv -f "${LOG_FILE}" "${LOG_FILE}.1"
exec >>"${LOG_FILE}" 2>&1
printf '[%s] INFO : log rotated at %s MB\n' "$(ts)" "${MAX_LOG_SIZE_MB}"
}
remove_rotated_logs() {
local i
for (( i = 1; i <= MAX_LOG_KEEP + 1; i++ )); do
rm -f "${LOG_FILE}.${i}"
done
}
prepare_log_on_start() {
local stamp
case "${START_LOG_MODE}" in
append)
return 0
;;
archive)
if [ -s "${LOG_FILE}" ]; then
stamp="$(date '+%Y%m%d-%H%M%S')"
mv -f "${LOG_FILE}" "${LOG_FILE}.${stamp}" \
|| die "cannot archive ${LOG_FILE}"
printf 'previous log archived: %s.%s\n' "${LOG_FILE}" "${stamp}"
fi
remove_rotated_logs
;;
new)
# Truncate rather than unlink: the inode is preserved, so a tail -f
# that is already attached keeps following the new run.
if [ -f "${LOG_FILE}" ]; then
: > "${LOG_FILE}" || die "cannot truncate ${LOG_FILE}"
fi
remove_rotated_logs
;;
*)
die "invalid START_LOG_MODE: ${START_LOG_MODE}"
;;
esac
}
# Cumulative "<rx_packets> <tx_packets>" for an interface, from ip -s link.
# The numbers sit on the line after each RX:/TX: header, and the column order
# is "bytes packets errors ..." -- so packets is $2, not $1.
if_counters() { if_counters() {
${SUDO} ip -s link show "$1" 2>/dev/null | awk ' ${SUDO} ip -s link show "$1" 2>/dev/null | awk '
/RX:/ { getline; rx = $2 } /RX:/ { getline; rx = $2 }
@@ -213,88 +124,18 @@ if_counters() {
END { printf "%s %s", (rx == "" ? 0 : rx), (tx == "" ? 0 : tx) }' END { printf "%s %s", (rx == "" ? 0 : rx), (tx == "" ? 0 : tx) }'
} }
# Bring the NIC up and (re)apply its address and default route. `replace` is # Bring the NIC up and (re)apply its address and default route.
# idempotent, so repeated rounds do not stack duplicates.
setup_iface() { setup_iface() {
local iface="$1" ipaddr="$2" plen="$3" gw="$4" metric="$5" local iface="$1" ipaddr="$2" plen="$3" gw="$4" metric="$5"
${SUDO} ip link set "${iface}" up 2>&1 || return 1
${SUDO} ip link set "${iface}" up 2>&1 || return 1 ${SUDO} ip address replace "${ipaddr}/${plen}" dev "${iface}" 2>&1 || return 1
${SUDO} ip address replace "${ipaddr}/${plen}" dev "${iface}" 2>&1 || return 1
if [ -n "${gw}" ]; then if [ -n "${gw}" ]; then
${SUDO} ip route replace default via "${gw}" dev "${iface}" \ ${SUDO} ip route replace default via "${gw}" dev "${iface}" \
metric "${metric}" 2>&1 || return 1 metric "${metric}" 2>&1 || return 1
fi fi
return 0 return 0
} }
# One leg: configure <if>, settle, ping its own target, report. The NIC is
# LEFT UP -- nothing is brought down between legs.
# Usage: ping_leg <label> <if> <ip> <plen> <gw> <metric> <target>
ping_leg() {
local label="$1" iface="$2" ipaddr="$3" plen="$4" gw="$5" metric="$6" target="$7"
local out rc tx rx loss avg verdict
local c0 c1 rx0 tx0 rx1 tx1 drx dtx
printf -- '----- [%s] %s : %s (%s/%s) -> %s -----\n' \
"$(ts)" "${label}" "${iface}" "${ipaddr}" "${plen}" "${target}"
if ! setup_iface "${iface}" "${ipaddr}" "${plen}" "${gw}" "${metric}"; then
printf '[%s] FAIL : cannot configure %s\n\n' "$(ts)" "${iface}"
return 1
fi
interruptible_sleep "${LINK_SETTLE_SEC}"
[ "${STOP}" -eq 0 ] || return 1
c0="$(if_counters "${iface}")"; rx0="${c0% *}"; tx0="${c0#* }"
if [ "${PING_TIMEOUT_SEC}" -gt 0 ]; then
out="$(timeout "${PING_TIMEOUT_SEC}" ping -I "${iface}" -c "${PING_COUNT}" "${target}" 2>&1)"
else
out="$(ping -I "${iface}" -c "${PING_COUNT}" "${target}" 2>&1)"
fi
rc=$?
printf '%s\n' "${out}"
c1="$(if_counters "${iface}")"; rx1="${c1% *}"; tx1="${c1#* }"
drx=$(( rx1 - rx0 )); dtx=$(( tx1 - tx0 ))
printf -- '--- ip -s link show %s ---\n' "${iface}"
${SUDO} ip -s link show "${iface}" 2>&1
# "10 packets transmitted, 10 received, 0% packet loss"
tx="$(printf '%s' "${out}" | sed -n 's/^\([0-9]\+\) packets transmitted.*/\1/p' | tail -1)"
rx="$(printf '%s' "${out}" | sed -n 's/.*[, ]\([0-9]\+\) received.*/\1/p' | tail -1)"
loss="$(printf '%s' "${out}" | sed -n 's/.*[, ]\([0-9]\+\)% packet loss.*/\1/p' | tail -1)"
# "rtt min/avg/max/mdev = 1.044/1.743/2.387/0.437 ms"
avg="$(printf '%s' "${out}" | sed -n 's|.*= [0-9.]*/\([0-9.]*\)/.*|\1|p' | tail -1)"
[ -n "${tx}" ] || tx=0
[ -n "${rx}" ] || rx=0
[ -n "${loss}" ] || loss=100
[ -n "${avg}" ] || avg="-"
if [ "${rc}" -eq 124 ]; then
verdict="FAIL(timeout)"
elif [ "${loss}" -le "${MAX_LOSS_PCT}" ]; then
verdict="PASS"
else
verdict="FAIL"
fi
# one greppable line per leg: grep RESULT <log>
# nic_tx/nic_rx are this interface's own counter delta across the burst --
# if they stay near zero while the ping succeeded, the traffic went out a
# different NIC and the result is not attributable to this one.
printf '[%s] RESULT %-3s %-6s -> %-15s tx=%s rx=%s loss=%s%% rtt_avg=%sms nic_tx=%s nic_rx=%s %s\n\n' \
"$(ts)" "${label}" "${iface}" "${target}" "${tx}" "${rx}" "${loss}" "${avg}" \
"${dtx}" "${drx}" "${verdict}"
[ "${verdict}" = "PASS" ]
}
# arp_ignore=1 : answer ARP only for addresses configured on the incoming NIC
# arp_announce=2: pick the source address from the outgoing NIC's own subnet
# Both are RAM-only sysctls and revert on reboot.
apply_arp_strict() { apply_arp_strict() {
local i local i
[ "${ARP_STRICT}" -eq 1 ] || return 0 [ "${ARP_STRICT}" -eq 1 ] || return 0
@@ -302,143 +143,245 @@ apply_arp_strict() {
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_ignore=1" 2>/dev/null ${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_ignore=1" 2>/dev/null
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_announce=2" 2>/dev/null ${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_announce=2" 2>/dev/null
done done
printf '[%s] INFO : arp_ignore=1 arp_announce=2 applied to %s,%s\n' \
"$(ts)" "${IF_10G}" "${IF_1G}"
} }
main_loop() { # Resolve the neighbour so the measured run does not spend its first packet on
local round=0 pass=0 fail=0 # ARP. Result is deliberately discarded.
arp_warmup() {
local iface="$1" target="$2"
[ "${ARP_WARMUP}" -eq 1 ] || return 0
ping -I "${iface}" -c 1 -W "${ARP_WARMUP_TIMEOUT}" "${target}" >/dev/null 2>&1
return 0
}
command -v ip >/dev/null 2>&1 || die "ip (iproute2) not found" # echo "pid_10g pid_1g" and return 0 when both are alive
command -v ping >/dev/null 2>&1 || die "ping not found" is_running() {
[ "${PING_COUNT}" -gt 0 ] 2>/dev/null || die "PING_COUNT must be > 0" local p0 p1
[ -f "${PID_FILE}" ] || return 1
p0="$(sed -n '1p' "${PID_FILE}" 2>/dev/null)"
p1="$(sed -n '2p' "${PID_FILE}" 2>/dev/null)"
[[ "${p0}" =~ ^[0-9]+$ ]] || return 1
[[ "${p1}" =~ ^[0-9]+$ ]] || return 1
kill -0 "${p0}" 2>/dev/null || kill -0 "${p1}" 2>/dev/null || return 1
printf '%s %s' "${p0}" "${p1}"
return 0
}
trap on_signal INT TERM ###############################################################################
# Report
###############################################################################
printf '#############################################################\n' # The recent entries for one NIC: replies and, thanks to -O, the requests that
printf '[%s] START: pid=%d count=%d settle=%ds interval=%ds\n' \ # got none.
"$(ts)" "$$" "${PING_COUNT}" "${LINK_SETTLE_SEC}" "${INTERVAL_SEC}" tail_entries() {
printf '[%s] 10G %s %s/%s -> %s (metric %s)\n' \ grep -aE 'bytes from|no answer|Unreachable|Time to live' "$1" 2>/dev/null \
"$(ts)" "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}" | tail -n "${TAIL_LINES}"
printf '[%s] 1G %s %s/%s -> %s (metric %s)\n' \ }
"$(ts)" "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}"
printf '#############################################################\n'
apply_arp_strict # The trailing "--- x ping statistics ---" block.
stats_block() {
sed -n '/ping statistics ---/,$p' "$1" 2>/dev/null
}
while [ "${STOP}" -eq 0 ]; do # render_leg <label> <iface> <target> <raw> <rx0> <tx0>
round=$(( round + 1 )) render_leg() {
printf -- '---------- ROUND %d @ %s ----------\n' "${round}" "$(ts)" local label="$1" iface="$2" target="$3" raw="$4" rx0="$5" tx0="$6"
local st tx rx loss avg verdict c1 rx1 tx1 drx dtx
if ping_leg "10G" "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${GW_10G}" \ printf -- '----- %s : %s -> %s : last %s entries -----\n' \
"${METRIC_10G}" "${TARGET_10G}"; then "${label}" "${iface}" "${target}" "${TAIL_LINES}"
pass=$(( pass + 1 )); else fail=$(( fail + 1 )) tail_entries "${raw}"
fi printf '\n'
[ "${STOP}" -eq 0 ] || break st="$(stats_block "${raw}")"
printf '%s\n' "${st}"
if ping_leg "1G" "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${GW_1G}" \ tx="$(printf '%s' "${st}" | sed -n 's/^\([0-9]\+\) packets transmitted.*/\1/p' | tail -1)"
"${METRIC_1G}" "${TARGET_1G}"; then rx="$(printf '%s' "${st}" | sed -n 's/.*[, ]\([0-9]\+\) received.*/\1/p' | tail -1)"
pass=$(( pass + 1 )); else fail=$(( fail + 1 )) loss="$(printf '%s' "${st}" | sed -n 's/.*[, ]\([0-9]\+\)% packet loss.*/\1/p' | tail -1)"
fi avg="$(printf '%s' "${st}" | sed -n 's|.*= [0-9.]*/\([0-9.]*\)/.*|\1|p' | tail -1)"
[ -n "${tx}" ] || tx=0
[ -n "${rx}" ] || rx=0
[ -n "${loss}" ] || loss=100
[ -n "${avg}" ] || avg="-"
rotate_log_if_needed c1="$(if_counters "${iface}")"; rx1="${c1% *}"; tx1="${c1#* }"
drx=$(( rx1 - rx0 )); dtx=$(( tx1 - tx0 ))
if [ "${MAX_ROUNDS}" -gt 0 ] && [ "${round}" -ge "${MAX_ROUNDS}" ]; then if [ "${loss}" -le "${MAX_LOSS_PCT}" ] && [ "${tx}" -gt 0 ]; then
break verdict="PASS"
fi else
[ "${STOP}" -eq 0 ] && interruptible_sleep "${INTERVAL_SEC}" verdict="FAIL"
done fi
printf '[%s] STOP : pid=%d after %d round(s), legs PASS=%d FAIL=%d\n' \ # nic_tx/nic_rx are this interface's own counter delta over the whole run.
"$(ts)" "$$" "${round}" "${pass}" "${fail}" # A pass with nic_tx near zero means the traffic left on the other NIC.
printf '[%s] RESULT %-3s %-6s -> %-15s tx=%s rx=%s loss=%s%% rtt_avg=%sms nic_tx=%s nic_rx=%s %s\n\n' \
"$(ts)" "${label}" "${iface}" "${target}" "${tx}" "${rx}" "${loss}" "${avg}" \
"${dtx}" "${drx}" "${verdict}"
}
write_report() {
local rx0_10g tx0_10g rx0_1g tx0_1g started
started="$(sed -n '1p' "${STATE_FILE}" 2>/dev/null)"
rx0_10g="$(sed -n '2p' "${STATE_FILE}" 2>/dev/null)"; rx0_10g="${rx0_10g:-0}"
tx0_10g="$(sed -n '3p' "${STATE_FILE}" 2>/dev/null)"; tx0_10g="${tx0_10g:-0}"
rx0_1g="$(sed -n '4p' "${STATE_FILE}" 2>/dev/null)"; rx0_1g="${rx0_1g:-0}"
tx0_1g="$(sed -n '5p' "${STATE_FILE}" 2>/dev/null)"; tx0_1g="${tx0_1g:-0}"
{
printf '#############################################################\n'
printf '[%s] mgmt ping report\n' "$(ts)"
printf ' started : %s\n' "${started:-unknown}"
printf ' stopped : %s\n' "$(ts)"
printf ' 10G : %s %s/%s -> %s (metric %s)\n' \
"${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}"
printf ' 1G : %s %s/%s -> %s (metric %s)\n' \
"${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}"
printf ' ping : -i %s %s, max loss %s%%\n' \
"${PING_INTERVAL_SEC}" "${PING_EXTRA_OPTS}" "${MAX_LOSS_PCT}"
printf '#############################################################\n\n'
render_leg "10G" "${IF_10G}" "${TARGET_10G}" "${RAW_10G}" "${rx0_10g}" "${tx0_10g}"
render_leg "1G" "${IF_1G}" "${TARGET_1G}" "${RAW_1G}" "${rx0_1g}" "${tx0_1g}"
printf -- '--- ip -s link show %s ---\n' "${IF_10G}"
${SUDO} ip -s link show "${IF_10G}" 2>&1
printf '\n'
printf -- '--- ip -s link show %s ---\n' "${IF_1G}"
${SUDO} ip -s link show "${IF_1G}" 2>&1
} >> "${LOG_FILE}"
} }
############################################################################### ###############################################################################
# Sub-commands # Sub-commands
############################################################################### ###############################################################################
do_start() { prepare_log_on_start() {
local pid local stamp
pid="$(is_running)" && die "already running (pid=${pid})" case "${START_LOG_MODE}" in
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}" append) return 0 ;;
prepare_log_on_start archive)
if [ -s "${LOG_FILE}" ]; then
if command -v setsid >/dev/null 2>&1; then stamp="$(date '+%Y%m%d-%H%M%S')"
setsid "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 & mv -f "${LOG_FILE}" "${LOG_FILE}.${stamp}" || die "cannot archive ${LOG_FILE}"
else printf 'previous log archived: %s.%s\n' "${LOG_FILE}" "${stamp}"
nohup "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 & fi
disown 2>/dev/null ;;
fi new)
# Truncate rather than unlink, so a tail -f already attached keeps
sleep 1 # following the new run.
pid="$(is_running)" || die "start failed, check ${LOG_FILE}" [ -f "${LOG_FILE}" ] && { : > "${LOG_FILE}" || die "cannot truncate ${LOG_FILE}"; }
printf 'started (pid=%s)\nlog: %s\n' "${pid}" "${LOG_FILE}" ;;
*) die "invalid START_LOG_MODE: ${START_LOG_MODE}" ;;
esac
} }
do_daemon() { do_start() {
local pids p0 p1 c
pids="$(is_running)" && die "already running (pids=${pids})"
command -v ip >/dev/null 2>&1 || die "ip (iproute2) not found"
command -v ping >/dev/null 2>&1 || die "ping not found"
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}" mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
LOG_TO_FILE=1 prepare_log_on_start
exec >>"${LOG_FILE}" 2>&1 : > "${RAW_10G}"
printf '%d\n' "$$" > "${PID_FILE}" : > "${RAW_1G}"
trap 'rm -f "${PID_FILE}"' EXIT
main_loop setup_iface "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${GW_10G}" "${METRIC_10G}" \
|| die "cannot configure ${IF_10G}"
setup_iface "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${GW_1G}" "${METRIC_1G}" \
|| die "cannot configure ${IF_1G}"
apply_arp_strict
sleep "${LINK_SETTLE_SEC}"
# Spend the ARP resolution here, not on the first measured packet.
arp_warmup "${IF_10G}" "${TARGET_10G}"
arp_warmup "${IF_1G}" "${TARGET_1G}"
# Counter baseline, taken after the warm-up so its packets are excluded.
{ c="$(if_counters "${IF_10G}")"
printf '%s\n%s\n%s\n' "$(ts)" "${c% *}" "${c#* }"
c="$(if_counters "${IF_1G}")"
printf '%s\n%s\n' "${c% *}" "${c#* }"
} > "${STATE_FILE}"
# Both NICs ping at the same time and keep accumulating until stopped.
nohup ping -I "${IF_10G}" -i "${PING_INTERVAL_SEC}" ${PING_EXTRA_OPTS} \
"${TARGET_10G}" >> "${RAW_10G}" 2>&1 &
p0=$!
nohup ping -I "${IF_1G}" -i "${PING_INTERVAL_SEC}" ${PING_EXTRA_OPTS} \
"${TARGET_1G}" >> "${RAW_1G}" 2>&1 &
p1=$!
disown "${p0}" 2>/dev/null
disown "${p1}" 2>/dev/null
printf '%s\n%s\n' "${p0}" "${p1}" > "${PID_FILE}"
sleep 1
kill -0 "${p0}" 2>/dev/null || die "10G ping failed to start, see ${RAW_10G}"
kill -0 "${p1}" 2>/dev/null || die "1G ping failed to start, see ${RAW_1G}"
printf 'started (10G pid=%s, 1G pid=%s)\nlog: %s\n' "${p0}" "${p1}" "${LOG_FILE}"
} }
do_stop() { do_stop() {
local pid i local pids p0 p1 i
pid="$(is_running)" || { printf 'not running\n'; rm -f "${PID_FILE}"; return 0; }
kill -TERM -"${pid}" 2>/dev/null || kill -TERM "${pid}" 2>/dev/null pids="$(is_running)" || {
printf 'not running\n'
rm -f "${PID_FILE}"
return 0
}
p0="${pids% *}"; p1="${pids#* }"
# SIGINT, not SIGTERM: ping prints its statistics block on interrupt, and
# that block is what the report parses.
kill -INT "${p0}" 2>/dev/null
kill -INT "${p1}" 2>/dev/null
for (( i = 0; i < 20; i++ )); do for (( i = 0; i < 20; i++ )); do
kill -0 "${pid}" 2>/dev/null || break kill -0 "${p0}" 2>/dev/null || kill -0 "${p1}" 2>/dev/null || break
sleep 0.5 sleep 0.5
done done
if kill -0 "${pid}" 2>/dev/null; then kill -KILL "${p0}" 2>/dev/null
printf 'SIGTERM ignored, sending SIGKILL\n' kill -KILL "${p1}" 2>/dev/null
kill -KILL -"${pid}" 2>/dev/null || kill -KILL "${pid}" 2>/dev/null
fi write_report
rm -f "${PID_FILE}" rm -f "${PID_FILE}"
printf 'stopped (pid=%s)\n' "${pid}" printf 'stopped (10G pid=%s, 1G pid=%s)\nreport: %s\n' "${p0}" "${p1}" "${LOG_FILE}"
} }
do_status() { do_status() {
local pid local pids
if pid="$(is_running)"; then if pids="$(is_running)"; then
printf 'status : running (pid=%s)\n' "${pid}" printf 'status : running (10G pid=%s, 1G pid=%s)\n' "${pids% *}" "${pids#* }"
else else
printf 'status : stopped\n' printf 'status : stopped\n'
fi fi
printf 'log : %s\n' "${LOG_FILE}" printf 'log : %s\n' "${LOG_FILE}"
[ -f "${LOG_FILE}" ] && printf 'size : %s\n' "$(du -h "${LOG_FILE}" | cut -f1)" printf '10G : %s %s/%s -> %s\n' "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}"
printf 'ping : count=%d, interval=%ds, settle=%ds\n' "${PING_COUNT}" "${INTERVAL_SEC}" "${LINK_SETTLE_SEC}" printf '1G : %s %s/%s -> %s\n' "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}"
printf '10G : %s %s/%s -> %s (metric %s)\n' "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}" [ -f "${RAW_10G}" ] && printf '10G recv: %s\n' "$(grep -ac 'bytes from' "${RAW_10G}")"
printf '1G : %s %s/%s -> %s (metric %s)\n' "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}" [ -f "${RAW_1G}" ] && printf '1G recv: %s\n' "$(grep -ac 'bytes from' "${RAW_1G}")"
} }
usage() { usage() {
cat <<EOF cat <<EOF
Usage: ${SCRIPT_NAME} {start|stop|status|fg|tail|clear|summary} Usage: ${SCRIPT_NAME} {start|stop|status|summary|clear}
start Run detached in the background; the previous log is discarded first start Configure both NICs, warm up ARP, then ping from BOTH at the same
(see START_LOG_MODE) time and keep accumulating. The previous log is discarded first.
stop Stop the background instance stop Stop both pings and render the report into the log
status Show pid / log path / configured NICs status Show pids, configured NICs and replies received so far
fg Run in the foreground, output to the terminal (for a quick check) summary Print just the RESULT lines from the log
tail tail -f the log file (does NOT return -- do not call from a macro) clear Remove the log and the raw captures (must be stopped first)
clear Remove the log file and all rotated copies (must be stopped first)
summary Print just the per-leg RESULT lines from the log
Log file : ${LOG_FILE} Log file : ${LOG_FILE}
One round = 10G leg then 1G leg, each pinging its own target. Both NICs are The report holds, per NIC, the last ${TAIL_LINES} entries and the ping
brought up and left up; addresses and routes are applied with statistics, then "ip -s link show" for both interfaces.
"ip address/route replace", so repeating a round does not stack duplicates.
If both NICs share a subnet, keep ARP_STRICT=1 -- otherwise a reply can come Both NICs are used at once, so management connectivity is in play -- drive
back on the NIC that did not send it. Check nic_tx/nic_rx on the RESULT line: this from the serial console. Edit the User Configurable Section at the top
they are that interface's own counter delta across the burst. to change interfaces, addresses, targets or the ping options.
Edit the User Configurable Section at the top to change interfaces, addresses,
targets, prefix lengths, gateways, metrics or the packet count.
EOF EOF
} }
@@ -446,18 +389,15 @@ EOF
# Entry point # Entry point
############################################################################### ###############################################################################
case "${1:-}" in case "${1:-}" in
start) do_start ;; start) do_start ;;
stop) do_stop ;; stop) do_stop ;;
status) do_status ;; status) do_status ;;
fg) LOG_TO_FILE=0; main_loop ;; summary) grep -a 'RESULT' "${LOG_FILE}" 2>/dev/null || printf 'no results in %s\n' "${LOG_FILE}" ;;
tail) tail -n 50 -f "${LOG_FILE}" ;;
summary) grep -a 'RESULT' "${LOG_FILE}" 2>/dev/null || printf 'no results in %s\n' "${LOG_FILE}" ;;
clear) clear)
is_running >/dev/null && die "still running, stop it first" is_running >/dev/null && die "still running, stop it first"
rm -f "${LOG_FILE}" "${LOG_FILE}".[0-9]* rm -f "${LOG_FILE}" "${LOG_FILE}".[0-9]* "${RAW_10G}" "${RAW_1G}" "${STATE_FILE}"
printf 'log cleared\n' printf 'log cleared\n'
;; ;;
__daemon) do_daemon ;;
-h|--help|"") usage ;; -h|--help|"") usage ;;
*) printf '[ERROR] unknown sub-command: %s\n\n' "$1" >&2; usage; exit 1 ;; *) printf '[ERROR] unknown sub-command: %s\n\n' "$1" >&2; usage; exit 1 ;;
esac esac