etwen/Blanton_TTL_Script
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

feat(mgmt): Ping both NICs at once, and stop counting ARP as loss

Browse Source 
The bench run reported FAIL=3 with no NIC errors and no drops. All three
failures were on eth1 and all three were missing exactly icmp_seq=1,
with the other nine replies present and sub-millisecond. That is
neighbour resolution: once the ARP entry for the target expires, the
first echo request is spent resolving it and ping counts it as loss.
Rounds sat about 24 s apart, right on the edge of the default reachable
time, so it happened on some rounds and not others.

mgmt_ping_monitor.sh V3.0.0:

- One discarded ping per NIC before the measured run resolves ARP, so
  MAX_LOSS_PCT can stay at 0 and a FAIL means a real drop. Raising the
  threshold instead would have hidden genuine single-packet loss, and a
  longer settle would not help -- nothing triggers ARP until the ping.
- Both NICs now ping at the same time and keep accumulating until
  stopped, rather than alternating fixed bursts. Each writes its own raw
  capture.
- `stop` sends SIGINT rather than SIGTERM, because ping prints its
  statistics block on interrupt and that block is what the report
  parses, then renders into the log: the last TAIL_LINES entries per
  NIC, each one's statistics and verdict, and `ip -s link show` for both
  interfaces.
- `-D -O` are on by default so every line carries a timestamp and a
  request that got no reply is visible in the tail instead of merely
  absent. Clear PING_EXTRA_OPTS if the platform's ping rejects them.

Since the script no longer re-execs itself to daemonise, `bash
mgmt_ping_monitor.sh start` works without the execute bit.

.gitignore: the new *.raw / *.pid / *.state runtime files were outside
the *.log rule and would have been committed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
This commit is contained in:
etwen
2026-08-26 09:53:20 +08:00
co-authored by Claude Opus 5
parent 1b8d5a8e91
commit a31c186142
3 changed files with 276 additions and 327 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+5
View File
@@ -12,6 +12,11 @@ For_AI/
!**/logs/.gitkeep
!**/Logs/.gitkeep
# monitor 腳本的執行期產物（原始擷取、pid、狀態）
*.raw
*.pid
*.state
# 打包輸出：產出不進 git，打包工具本身要進
publish/*
!publish/.gitkeep
src/Script_ABC_Blanton/1_Blanton_Script_A.ttl
+4
View File
@@ -51,6 +51,10 @@
; ScriptB USB stress uses the detected node, not a hard-coded /dev/sda1
; ScriptB DDR memtester runs continuously (drop the 100-pass limit)
; ScriptC Move bgctl reset --yes after the USB archive
; Blanton_Script/mgmt_ping_monitor.sh V3.0.0 - both NICs ping simultaneously, accumulating
; + ARP warm-up so a stale neighbour entry is not counted as loss
; + stop renders last-N per NIC, stats, and both ip -s link show
; .gitignore Ignore monitor *.raw / *.pid / *.state
; =============================================================================
include "config.ttl"
src/Script_ABC_Blanton/Blanton_Script/mgmt_ping_monitor.sh
+261 -321
View File
@@ -2,48 +2,41 @@
###############################################################################
# mgmt_ping_monitor.sh
#
# Version : V1.0.0
# Version : V3.0.0
# Author : ETWen
# Date : 20260824
# Date : 20260826
#
# Purpose : Exercise both management NICs and ping each one's own target,
# appending everything to a rotating log. One round is
# Purpose : Ping from BOTH management NICs at the same time, continuously, and
# keep accumulating until stopped:
#
# 10G (eth0) : ip link up + address/route replace -> settle
# -> ping N x TARGET_10G -> ip -s link show
# 1G (eth1) : ip link up + address/route replace -> settle
# -> ping N x TARGET_1G -> ip -s link show
# 10G (eth0) -> TARGET_10G
# 1G (eth1) -> TARGET_1G
#
# Designed to run detached in the background, same sub-commands as
# bmc_monitor.sh: start/stop/status/fg/tail/clear.
# `stop` renders a report into the log: the last TAIL_LINES entries
# per NIC, each one's ping statistics and verdict, then
# `ip -s link show` for both interfaces.
#
# Notes : - Both NICs are brought up and LEFT up. Addresses and default
# routes are applied with iproute2 `replace`, which is idempotent:
# re-running a round does not stack duplicate addresses or routes.
# - The default routes carry different metrics (100 / 200), so the
# lower one wins for off-subnet traffic. They do NOT decide which
# NIC a ping leaves by: `ping -I <iface>` binds to the device.
# - !! SAME-SUBNET WARNING !! With both NICs up on one subnet the
# target ARPs for the address and Linux may answer from either
# NIC, so replies can arrive on the wrong one and the result is
# not attributable. ARP_STRICT=1 sets arp_ignore/arp_announce to
# stop that. Leave it on unless the two NICs are on separate
# subnets, as in the iproute2 example this was modelled on.
# - Each leg logs `ip -s link show <iface>` and the TX/RX packet
# delta across the burst, which is the direct evidence of which
# NIC actually carried the traffic.
# - The first pings after bringing a link up are lost to ARP, not to
# the link: a bench capture showed 4x "Destination Host
# Unreachable" on the first burst and 0% loss on the very next
# one. LINK_SETTLE_SEC absorbs that; it is not packet loss.
# Notes : - Both NICs are configured with iproute2 `replace`, which is
# idempotent, and both are left up. On a shared subnet the target's
# ARP can be answered by either NIC, so ARP_STRICT keeps each one
# to its own address.
# - ARP WARM-UP: the first packet after a neighbour entry expires is
# spent resolving ARP and is counted as loss. On the bench this
# produced three FAILs whose only missing packet was icmp_seq=1,
# every time, with zero NIC errors or drops. One discarded ping
# before the measured run removes that artefact, which is what lets
# MAX_LOSS_PCT stay at 0 and still mean something.
# - MANAGEMENT CONNECTIVITY IS IN USE while this runs. Drive it from
# the serial console.
#
# Version History
# V1.0.0 20260824 Initial Version
# V2.0.0 20260824 Switch from ifconfig to iproute2 (ip link / address
# replace / route replace with per-NIC metric). Both
# NICs now stay up -- no link down between legs. Add
# `ip -s link show` per leg with a TX/RX delta, and
# ARP_STRICT for the same-subnet case.
# V1.0.0 20260824 Initial Version (ifconfig, one NIC at a time)
# V2.0.0 20260824 iproute2; both NICs stay up; per-NIC target;
# ip -s link show + TX/RX delta per leg
# V3.0.0 20260826 Both NICs ping SIMULTANEOUSLY and continuously instead
# of alternating fixed bursts. Add the ARP warm-up.
# `stop` renders last-N per NIC + statistics + both
# ip -s link outputs into the log.
###############################################################################
set -u
@@ -63,149 +56,67 @@ METRIC_10G=100 # lower metric wins for off-subnet traffic
# --- 1G management port ---
IF_1G="eth1"
IP_1G="192.168.1.101"
TARGET_1G="192.168.1.31" # host this NIC pings
TARGET_1G="192.168.1.31"
PLEN_1G=24
GW_1G=""
METRIC_1G=200
# --- ping ---
PING_COUNT=10 # echo requests per leg
MAX_LOSS_PCT=0 # loss above this marks the leg FAIL
PING_INTERVAL_SEC=1 # seconds between echo requests (ping -i)
MAX_LOSS_PCT=0 # loss above this marks the NIC FAIL
TAIL_LINES=20 # how many recent entries per NIC the report shows
# Both NICs stay up. If they share a subnet the target's ARP can be answered
# by either one, so a reply may arrive on the NIC that did not send. Setting
# this applies arp_ignore=1 / arp_announce=2 to both, which keeps each NIC to
# its own address. RAM only -- reverts on reboot. Set 0 for separate subnets.
# Extra ping flags. -D timestamps every line, -O prints a marker for a request
# that got no reply, so a drop is visible in the tail instead of just missing.
# Clear this if the platform's ping does not accept them.
PING_EXTRA_OPTS="-D -O"
# One discarded ping per NIC before the measured run, to resolve ARP. Without
# it the first packet of the run is lost to neighbour resolution and looks
# exactly like a link fault. 0 disables.
ARP_WARMUP=1
ARP_WARMUP_TIMEOUT=2 # seconds to wait for the warm-up reply
# Both NICs stay up. If they share a subnet the target's ARP can be answered by
# either one, so a reply may arrive on the NIC that did not send. This applies
# arp_ignore=1 / arp_announce=2 to both. RAM only -- reverts on reboot.
ARP_STRICT=1
# Prefix for the ip commands; empty because the daemon already runs as root.
# Prefix for the ip commands; empty because this already runs as root.
SUDO=""
# Seconds to wait after bringing a link up before pinging, so ARP is resolved
# and the PHY has finished autoneg. The bench needed a few seconds on the 1G.
# Seconds to wait after bringing a link up before pinging.
LINK_SETTLE_SEC=5
# Seconds between rounds (0 = start the next round immediately)
INTERVAL_SEC=10
# Per-ping-burst timeout in seconds; stops a wedged ping from freezing the
# loop. 0 disables. Must exceed PING_COUNT (one packet per second).
PING_TIMEOUT_SEC=60
# Stop after this many rounds. 0 = run until stopped manually.
MAX_ROUNDS=0
# Logging
LOG_DIR="" # empty = <script directory>/log
LOG_NAME="mgmt_ping.log"
MAX_LOG_SIZE_MB=100 # rotate above this size; 0 = never rotate
MAX_LOG_KEEP=5 # keep .1 .. .N
# What `start` does with the log left behind by the previous run:
# new = begin with an empty log; previous content and rotated copies are
# discarded (default -- one run, one log)
# archive = rename the previous log to <log>.YYYYmmdd-HHMMSS, then begin empty
# append = keep appending to the existing log
# new = discard it (default -- one run, one log)
# archive = rename it to <log>.YYYYmmdd-HHMMSS first
# append = keep it
START_LOG_MODE="new"
###############################################################################
# Internal
###############################################################################
SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")"
SCRIPT_PATH="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)/${SCRIPT_NAME}"
[ -n "${LOG_DIR}" ] || LOG_DIR="$(dirname -- "${SCRIPT_PATH}")/log"
LOG_FILE="${LOG_DIR}/${LOG_NAME}"
PID_FILE="${LOG_DIR}/${LOG_NAME%.log}.pid"
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
[ -n "${LOG_DIR}" ] || LOG_DIR="${SCRIPT_DIR}/log"
LOG_TO_FILE=0
STOP=0
SLEEP_PID=""
LOG_FILE="${LOG_DIR}/${LOG_NAME}"
BASE="${LOG_DIR}/${LOG_NAME%.log}"
PID_FILE="${BASE}.pid"
STATE_FILE="${BASE}.state"
RAW_10G="${BASE}_${IF_10G}.raw"
RAW_1G="${BASE}_${IF_1G}.raw"
ts() { date '+%Y-%m-%d %H:%M:%S'; }
die() { printf '[ERROR] %s\n' "$*" >&2; exit 1; }
on_signal() {
STOP=1
[ -n "${SLEEP_PID}" ] && kill "${SLEEP_PID}" 2>/dev/null
}
interruptible_sleep() {
local sec="$1"
[ "${sec}" -gt 0 ] 2>/dev/null || return 0
sleep "${sec}" &
SLEEP_PID=$!
wait "${SLEEP_PID}" 2>/dev/null
SLEEP_PID=""
}
is_running() {
local pid
[ -f "${PID_FILE}" ] || return 1
pid="$(cat "${PID_FILE}" 2>/dev/null)"
[[ "${pid}" =~ ^[0-9]+$ ]] || return 1
kill -0 "${pid}" 2>/dev/null || return 1
grep -qa "${SCRIPT_NAME}" "/proc/${pid}/cmdline" 2>/dev/null || return 1
printf '%s' "${pid}"
return 0
}
rotate_log_if_needed() {
local max_bytes size i
[ "${LOG_TO_FILE}" -eq 1 ] || return 0
[ "${MAX_LOG_SIZE_MB}" -gt 0 ] || return 0
[ -f "${LOG_FILE}" ] || return 0
max_bytes=$(( MAX_LOG_SIZE_MB * 1024 * 1024 ))
size="$(stat -c %s "${LOG_FILE}" 2>/dev/null || echo 0)"
[ "${size}" -ge "${max_bytes}" ] || return 0
for (( i = MAX_LOG_KEEP - 1; i >= 1; i-- )); do
[ -f "${LOG_FILE}.${i}" ] && mv -f "${LOG_FILE}.${i}" "${LOG_FILE}.$(( i + 1 ))"
done
mv -f "${LOG_FILE}" "${LOG_FILE}.1"
exec >>"${LOG_FILE}" 2>&1
printf '[%s] INFO : log rotated at %s MB\n' "$(ts)" "${MAX_LOG_SIZE_MB}"
}
remove_rotated_logs() {
local i
for (( i = 1; i <= MAX_LOG_KEEP + 1; i++ )); do
rm -f "${LOG_FILE}.${i}"
done
}
prepare_log_on_start() {
local stamp
case "${START_LOG_MODE}" in
append)
return 0
;;
archive)
if [ -s "${LOG_FILE}" ]; then
stamp="$(date '+%Y%m%d-%H%M%S')"
mv -f "${LOG_FILE}" "${LOG_FILE}.${stamp}" \
|| die "cannot archive ${LOG_FILE}"
printf 'previous log archived: %s.%s\n' "${LOG_FILE}" "${stamp}"
fi
remove_rotated_logs
;;
new)
# Truncate rather than unlink: the inode is preserved, so a tail -f
# that is already attached keeps following the new run.
if [ -f "${LOG_FILE}" ]; then
: > "${LOG_FILE}" || die "cannot truncate ${LOG_FILE}"
fi
remove_rotated_logs
;;
*)
die "invalid START_LOG_MODE: ${START_LOG_MODE}"
;;
esac
}
# Cumulative "<rx_packets> <tx_packets>" for an interface, from ip -s link.
# The numbers sit on the line after each RX:/TX: header, and the column order
# is "bytes packets errors ..." -- so packets is $2, not $1.
# "<rx_packets> <tx_packets>" for an interface. ip -s link orders the columns
# "bytes packets errors ...", so packets is $2, not $1.
if_counters() {
${SUDO} ip -s link show "$1" 2>/dev/null | awk '
/RX:/ { getline; rx = $2 }
@@ -213,11 +124,9 @@ if_counters() {
END { printf "%s %s", (rx == "" ? 0 : rx), (tx == "" ? 0 : tx) }'
}
# Bring the NIC up and (re)apply its address and default route. `replace` is
# idempotent, so repeated rounds do not stack duplicates.
# Bring the NIC up and (re)apply its address and default route.
setup_iface() {
local iface="$1" ipaddr="$2" plen="$3" gw="$4" metric="$5"
${SUDO} ip link set "${iface}" up 2>&1 || return 1
${SUDO} ip address replace "${ipaddr}/${plen}" dev "${iface}" 2>&1 || return 1
if [ -n "${gw}" ]; then
@@ -227,74 +136,6 @@ setup_iface() {
return 0
}
# One leg: configure <if>, settle, ping its own target, report. The NIC is
# LEFT UP -- nothing is brought down between legs.
# Usage: ping_leg <label> <if> <ip> <plen> <gw> <metric> <target>
ping_leg() {
local label="$1" iface="$2" ipaddr="$3" plen="$4" gw="$5" metric="$6" target="$7"
local out rc tx rx loss avg verdict
local c0 c1 rx0 tx0 rx1 tx1 drx dtx
printf -- '----- [%s] %s : %s (%s/%s) -> %s -----\n' \
"$(ts)" "${label}" "${iface}" "${ipaddr}" "${plen}" "${target}"
if ! setup_iface "${iface}" "${ipaddr}" "${plen}" "${gw}" "${metric}"; then
printf '[%s] FAIL : cannot configure %s\n\n' "$(ts)" "${iface}"
return 1
fi
interruptible_sleep "${LINK_SETTLE_SEC}"
[ "${STOP}" -eq 0 ] || return 1
c0="$(if_counters "${iface}")"; rx0="${c0% *}"; tx0="${c0#* }"
if [ "${PING_TIMEOUT_SEC}" -gt 0 ]; then
out="$(timeout "${PING_TIMEOUT_SEC}" ping -I "${iface}" -c "${PING_COUNT}" "${target}" 2>&1)"
else
out="$(ping -I "${iface}" -c "${PING_COUNT}" "${target}" 2>&1)"
fi
rc=$?
printf '%s\n' "${out}"
c1="$(if_counters "${iface}")"; rx1="${c1% *}"; tx1="${c1#* }"
drx=$(( rx1 - rx0 )); dtx=$(( tx1 - tx0 ))
printf -- '--- ip -s link show %s ---\n' "${iface}"
${SUDO} ip -s link show "${iface}" 2>&1
# "10 packets transmitted, 10 received, 0% packet loss"
tx="$(printf '%s' "${out}" | sed -n 's/^\([0-9]\+\) packets transmitted.*/\1/p' | tail -1)"
rx="$(printf '%s' "${out}" | sed -n 's/.*[, ]\([0-9]\+\) received.*/\1/p' | tail -1)"
loss="$(printf '%s' "${out}" | sed -n 's/.*[, ]\([0-9]\+\)% packet loss.*/\1/p' | tail -1)"
# "rtt min/avg/max/mdev = 1.044/1.743/2.387/0.437 ms"
avg="$(printf '%s' "${out}" | sed -n 's|.*= [0-9.]*/\([0-9.]*\)/.*|\1|p' | tail -1)"
[ -n "${tx}" ] || tx=0
[ -n "${rx}" ] || rx=0
[ -n "${loss}" ] || loss=100
[ -n "${avg}" ] || avg="-"
if [ "${rc}" -eq 124 ]; then
verdict="FAIL(timeout)"
elif [ "${loss}" -le "${MAX_LOSS_PCT}" ]; then
verdict="PASS"
else
verdict="FAIL"
fi
# one greppable line per leg: grep RESULT <log>
# nic_tx/nic_rx are this interface's own counter delta across the burst --
# if they stay near zero while the ping succeeded, the traffic went out a
# different NIC and the result is not attributable to this one.
printf '[%s] RESULT %-3s %-6s -> %-15s tx=%s rx=%s loss=%s%% rtt_avg=%sms nic_tx=%s nic_rx=%s %s\n\n' \
"$(ts)" "${label}" "${iface}" "${target}" "${tx}" "${rx}" "${loss}" "${avg}" \
"${dtx}" "${drx}" "${verdict}"
[ "${verdict}" = "PASS" ]
}
# arp_ignore=1 : answer ARP only for addresses configured on the incoming NIC
# arp_announce=2: pick the source address from the outgoing NIC's own subnet
# Both are RAM-only sysctls and revert on reboot.
apply_arp_strict() {
local i
[ "${ARP_STRICT}" -eq 1 ] || return 0
@@ -302,143 +143,245 @@ apply_arp_strict() {
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_ignore=1" 2>/dev/null
${SUDO} sysctl -qw "net.ipv4.conf.${i}.arp_announce=2" 2>/dev/null
done
printf '[%s] INFO : arp_ignore=1 arp_announce=2 applied to %s,%s\n' \
"$(ts)" "${IF_10G}" "${IF_1G}"
}
main_loop() {
local round=0 pass=0 fail=0
# Resolve the neighbour so the measured run does not spend its first packet on
# ARP. Result is deliberately discarded.
arp_warmup() {
local iface="$1" target="$2"
[ "${ARP_WARMUP}" -eq 1 ] || return 0
ping -I "${iface}" -c 1 -W "${ARP_WARMUP_TIMEOUT}" "${target}" >/dev/null 2>&1
return 0
}
command -v ip >/dev/null 2>&1 || die "ip (iproute2) not found"
command -v ping >/dev/null 2>&1 || die "ping not found"
[ "${PING_COUNT}" -gt 0 ] 2>/dev/null || die "PING_COUNT must be > 0"
# echo "pid_10g pid_1g" and return 0 when both are alive
is_running() {
local p0 p1
[ -f "${PID_FILE}" ] || return 1
p0="$(sed -n '1p' "${PID_FILE}" 2>/dev/null)"
p1="$(sed -n '2p' "${PID_FILE}" 2>/dev/null)"
[[ "${p0}" =~ ^[0-9]+$ ]] || return 1
[[ "${p1}" =~ ^[0-9]+$ ]] || return 1
kill -0 "${p0}" 2>/dev/null || kill -0 "${p1}" 2>/dev/null || return 1
printf '%s %s' "${p0}" "${p1}"
return 0
}
trap on_signal INT TERM
###############################################################################
# Report
###############################################################################
# The recent entries for one NIC: replies and, thanks to -O, the requests that
# got none.
tail_entries() {
grep -aE 'bytes from|no answer|Unreachable|Time to live' "$1" 2>/dev/null \
| tail -n "${TAIL_LINES}"
}
# The trailing "--- x ping statistics ---" block.
stats_block() {
sed -n '/ping statistics ---/,$p' "$1" 2>/dev/null
}
# render_leg <label> <iface> <target> <raw> <rx0> <tx0>
render_leg() {
local label="$1" iface="$2" target="$3" raw="$4" rx0="$5" tx0="$6"
local st tx rx loss avg verdict c1 rx1 tx1 drx dtx
printf -- '----- %s : %s -> %s : last %s entries -----\n' \
"${label}" "${iface}" "${target}" "${TAIL_LINES}"
tail_entries "${raw}"
printf '\n'
st="$(stats_block "${raw}")"
printf '%s\n' "${st}"
tx="$(printf '%s' "${st}" | sed -n 's/^\([0-9]\+\) packets transmitted.*/\1/p' | tail -1)"
rx="$(printf '%s' "${st}" | sed -n 's/.*[, ]\([0-9]\+\) received.*/\1/p' | tail -1)"
loss="$(printf '%s' "${st}" | sed -n 's/.*[, ]\([0-9]\+\)% packet loss.*/\1/p' | tail -1)"
avg="$(printf '%s' "${st}" | sed -n 's|.*= [0-9.]*/\([0-9.]*\)/.*|\1|p' | tail -1)"
[ -n "${tx}" ] || tx=0
[ -n "${rx}" ] || rx=0
[ -n "${loss}" ] || loss=100
[ -n "${avg}" ] || avg="-"
c1="$(if_counters "${iface}")"; rx1="${c1% *}"; tx1="${c1#* }"
drx=$(( rx1 - rx0 )); dtx=$(( tx1 - tx0 ))
if [ "${loss}" -le "${MAX_LOSS_PCT}" ] && [ "${tx}" -gt 0 ]; then
verdict="PASS"
else
verdict="FAIL"
fi
# nic_tx/nic_rx are this interface's own counter delta over the whole run.
# A pass with nic_tx near zero means the traffic left on the other NIC.
printf '[%s] RESULT %-3s %-6s -> %-15s tx=%s rx=%s loss=%s%% rtt_avg=%sms nic_tx=%s nic_rx=%s %s\n\n' \
"$(ts)" "${label}" "${iface}" "${target}" "${tx}" "${rx}" "${loss}" "${avg}" \
"${dtx}" "${drx}" "${verdict}"
}
write_report() {
local rx0_10g tx0_10g rx0_1g tx0_1g started
started="$(sed -n '1p' "${STATE_FILE}" 2>/dev/null)"
rx0_10g="$(sed -n '2p' "${STATE_FILE}" 2>/dev/null)"; rx0_10g="${rx0_10g:-0}"
tx0_10g="$(sed -n '3p' "${STATE_FILE}" 2>/dev/null)"; tx0_10g="${tx0_10g:-0}"
rx0_1g="$(sed -n '4p' "${STATE_FILE}" 2>/dev/null)"; rx0_1g="${rx0_1g:-0}"
tx0_1g="$(sed -n '5p' "${STATE_FILE}" 2>/dev/null)"; tx0_1g="${tx0_1g:-0}"
{
printf '#############################################################\n'
printf '[%s] START: pid=%d count=%d settle=%ds interval=%ds\n' \
"$(ts)" "$$" "${PING_COUNT}" "${LINK_SETTLE_SEC}" "${INTERVAL_SEC}"
printf '[%s] 10G %s %s/%s -> %s (metric %s)\n' \
"$(ts)" "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}"
printf '[%s] 1G %s %s/%s -> %s (metric %s)\n' \
"$(ts)" "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}"
printf '#############################################################\n'
printf '[%s] mgmt ping report\n' "$(ts)"
printf ' started : %s\n' "${started:-unknown}"
printf ' stopped : %s\n' "$(ts)"
printf ' 10G : %s %s/%s -> %s (metric %s)\n' \
"${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}"
printf ' 1G : %s %s/%s -> %s (metric %s)\n' \
"${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}"
printf ' ping : -i %s %s, max loss %s%%\n' \
"${PING_INTERVAL_SEC}" "${PING_EXTRA_OPTS}" "${MAX_LOSS_PCT}"
printf '#############################################################\n\n'
apply_arp_strict
render_leg "10G" "${IF_10G}" "${TARGET_10G}" "${RAW_10G}" "${rx0_10g}" "${tx0_10g}"
render_leg "1G" "${IF_1G}" "${TARGET_1G}" "${RAW_1G}" "${rx0_1g}" "${tx0_1g}"
while [ "${STOP}" -eq 0 ]; do
round=$(( round + 1 ))
printf -- '---------- ROUND %d @ %s ----------\n' "${round}" "$(ts)"
if ping_leg "10G" "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${GW_10G}" \
"${METRIC_10G}" "${TARGET_10G}"; then
pass=$(( pass + 1 )); else fail=$(( fail + 1 ))
fi
[ "${STOP}" -eq 0 ] || break
if ping_leg "1G" "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${GW_1G}" \
"${METRIC_1G}" "${TARGET_1G}"; then
pass=$(( pass + 1 )); else fail=$(( fail + 1 ))
fi
rotate_log_if_needed
if [ "${MAX_ROUNDS}" -gt 0 ] && [ "${round}" -ge "${MAX_ROUNDS}" ]; then
break
fi
[ "${STOP}" -eq 0 ] && interruptible_sleep "${INTERVAL_SEC}"
done
printf '[%s] STOP : pid=%d after %d round(s), legs PASS=%d FAIL=%d\n' \
"$(ts)" "$$" "${round}" "${pass}" "${fail}"
printf -- '--- ip -s link show %s ---\n' "${IF_10G}"
${SUDO} ip -s link show "${IF_10G}" 2>&1
printf '\n'
printf -- '--- ip -s link show %s ---\n' "${IF_1G}"
${SUDO} ip -s link show "${IF_1G}" 2>&1
} >> "${LOG_FILE}"
}
###############################################################################
# Sub-commands
###############################################################################
do_start() {
local pid
pid="$(is_running)" && die "already running (pid=${pid})"
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
prepare_log_on_start
if command -v setsid >/dev/null 2>&1; then
setsid "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 &
else
nohup "${SCRIPT_PATH}" __daemon >/dev/null 2>&1 &
disown 2>/dev/null
prepare_log_on_start() {
local stamp
case "${START_LOG_MODE}" in
append) return 0 ;;
archive)
if [ -s "${LOG_FILE}" ]; then
stamp="$(date '+%Y%m%d-%H%M%S')"
mv -f "${LOG_FILE}" "${LOG_FILE}.${stamp}" || die "cannot archive ${LOG_FILE}"
printf 'previous log archived: %s.%s\n' "${LOG_FILE}" "${stamp}"
fi
sleep 1
pid="$(is_running)" || die "start failed, check ${LOG_FILE}"
printf 'started (pid=%s)\nlog: %s\n' "${pid}" "${LOG_FILE}"
;;
new)
# Truncate rather than unlink, so a tail -f already attached keeps
# following the new run.
[ -f "${LOG_FILE}" ] && { : > "${LOG_FILE}" || die "cannot truncate ${LOG_FILE}"; }
;;
*) die "invalid START_LOG_MODE: ${START_LOG_MODE}" ;;
esac
}
do_daemon() {
do_start() {
local pids p0 p1 c
pids="$(is_running)" && die "already running (pids=${pids})"
command -v ip >/dev/null 2>&1 || die "ip (iproute2) not found"
command -v ping >/dev/null 2>&1 || die "ping not found"
mkdir -p "${LOG_DIR}" || die "cannot create ${LOG_DIR}"
LOG_TO_FILE=1
exec >>"${LOG_FILE}" 2>&1
printf '%d\n' "$$" > "${PID_FILE}"
trap 'rm -f "${PID_FILE}"' EXIT
main_loop
prepare_log_on_start
: > "${RAW_10G}"
: > "${RAW_1G}"
setup_iface "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${GW_10G}" "${METRIC_10G}" \
|| die "cannot configure ${IF_10G}"
setup_iface "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${GW_1G}" "${METRIC_1G}" \
|| die "cannot configure ${IF_1G}"
apply_arp_strict
sleep "${LINK_SETTLE_SEC}"
# Spend the ARP resolution here, not on the first measured packet.
arp_warmup "${IF_10G}" "${TARGET_10G}"
arp_warmup "${IF_1G}" "${TARGET_1G}"
# Counter baseline, taken after the warm-up so its packets are excluded.
{ c="$(if_counters "${IF_10G}")"
printf '%s\n%s\n%s\n' "$(ts)" "${c% *}" "${c#* }"
c="$(if_counters "${IF_1G}")"
printf '%s\n%s\n' "${c% *}" "${c#* }"
} > "${STATE_FILE}"
# Both NICs ping at the same time and keep accumulating until stopped.
nohup ping -I "${IF_10G}" -i "${PING_INTERVAL_SEC}" ${PING_EXTRA_OPTS} \
"${TARGET_10G}" >> "${RAW_10G}" 2>&1 &
p0=$!
nohup ping -I "${IF_1G}" -i "${PING_INTERVAL_SEC}" ${PING_EXTRA_OPTS} \
"${TARGET_1G}" >> "${RAW_1G}" 2>&1 &
p1=$!
disown "${p0}" 2>/dev/null
disown "${p1}" 2>/dev/null
printf '%s\n%s\n' "${p0}" "${p1}" > "${PID_FILE}"
sleep 1
kill -0 "${p0}" 2>/dev/null || die "10G ping failed to start, see ${RAW_10G}"
kill -0 "${p1}" 2>/dev/null || die "1G ping failed to start, see ${RAW_1G}"
printf 'started (10G pid=%s, 1G pid=%s)\nlog: %s\n' "${p0}" "${p1}" "${LOG_FILE}"
}
do_stop() {
local pid i
pid="$(is_running)" || { printf 'not running\n'; rm -f "${PID_FILE}"; return 0; }
local pids p0 p1 i
kill -TERM -"${pid}" 2>/dev/null || kill -TERM "${pid}" 2>/dev/null
pids="$(is_running)" || {
printf 'not running\n'
rm -f "${PID_FILE}"
return 0
}
p0="${pids% *}"; p1="${pids#* }"
# SIGINT, not SIGTERM: ping prints its statistics block on interrupt, and
# that block is what the report parses.
kill -INT "${p0}" 2>/dev/null
kill -INT "${p1}" 2>/dev/null
for (( i = 0; i < 20; i++ )); do
kill -0 "${pid}" 2>/dev/null || break
kill -0 "${p0}" 2>/dev/null || kill -0 "${p1}" 2>/dev/null || break
sleep 0.5
done
if kill -0 "${pid}" 2>/dev/null; then
printf 'SIGTERM ignored, sending SIGKILL\n'
kill -KILL -"${pid}" 2>/dev/null || kill -KILL "${pid}" 2>/dev/null
fi
kill -KILL "${p0}" 2>/dev/null
kill -KILL "${p1}" 2>/dev/null
write_report
rm -f "${PID_FILE}"
printf 'stopped (pid=%s)\n' "${pid}"
printf 'stopped (10G pid=%s, 1G pid=%s)\nreport: %s\n' "${p0}" "${p1}" "${LOG_FILE}"
}
do_status() {
local pid
if pid="$(is_running)"; then
printf 'status : running (pid=%s)\n' "${pid}"
local pids
if pids="$(is_running)"; then
printf 'status : running (10G pid=%s, 1G pid=%s)\n' "${pids% *}" "${pids#* }"
else
printf 'status : stopped\n'
fi
printf 'log : %s\n' "${LOG_FILE}"
[ -f "${LOG_FILE}" ] && printf 'size : %s\n' "$(du -h "${LOG_FILE}" | cut -f1)"
printf 'ping : count=%d, interval=%ds, settle=%ds\n' "${PING_COUNT}" "${INTERVAL_SEC}" "${LINK_SETTLE_SEC}"
printf '10G : %s %s/%s -> %s (metric %s)\n' "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}" "${METRIC_10G}"
printf '1G : %s %s/%s -> %s (metric %s)\n' "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}" "${METRIC_1G}"
printf '10G : %s %s/%s -> %s\n' "${IF_10G}" "${IP_10G}" "${PLEN_10G}" "${TARGET_10G}"
printf '1G : %s %s/%s -> %s\n' "${IF_1G}" "${IP_1G}" "${PLEN_1G}" "${TARGET_1G}"
[ -f "${RAW_10G}" ] && printf '10G recv: %s\n' "$(grep -ac 'bytes from' "${RAW_10G}")"
[ -f "${RAW_1G}" ] && printf '1G recv: %s\n' "$(grep -ac 'bytes from' "${RAW_1G}")"
}
usage() {
cat <<EOF
Usage: ${SCRIPT_NAME} {start|stop|status|fg|tail|clear|summary}
Usage: ${SCRIPT_NAME} {start|stop|status|summary|clear}
start Run detached in the background; the previous log is discarded first
(see START_LOG_MODE)
stop Stop the background instance
status Show pid / log path / configured NICs
fg Run in the foreground, output to the terminal (for a quick check)
tail tail -f the log file (does NOT return -- do not call from a macro)
clear Remove the log file and all rotated copies (must be stopped first)
summary Print just the per-leg RESULT lines from the log
start Configure both NICs, warm up ARP, then ping from BOTH at the same
time and keep accumulating. The previous log is discarded first.
stop Stop both pings and render the report into the log
status Show pids, configured NICs and replies received so far
summary Print just the RESULT lines from the log
clear Remove the log and the raw captures (must be stopped first)
Log file : ${LOG_FILE}
One round = 10G leg then 1G leg, each pinging its own target. Both NICs are
brought up and left up; addresses and routes are applied with
"ip address/route replace", so repeating a round does not stack duplicates.
The report holds, per NIC, the last ${TAIL_LINES} entries and the ping
statistics, then "ip -s link show" for both interfaces.
If both NICs share a subnet, keep ARP_STRICT=1 -- otherwise a reply can come
back on the NIC that did not send it. Check nic_tx/nic_rx on the RESULT line:
they are that interface's own counter delta across the burst.
Edit the User Configurable Section at the top to change interfaces, addresses,
targets, prefix lengths, gateways, metrics or the packet count.
Both NICs are used at once, so management connectivity is in play -- drive
this from the serial console. Edit the User Configurable Section at the top
to change interfaces, addresses, targets or the ping options.
EOF
}
@@ -449,15 +392,12 @@ case "${1:-}" in
start) do_start ;;
stop) do_stop ;;
status) do_status ;;
fg) LOG_TO_FILE=0; main_loop ;;
tail) tail -n 50 -f "${LOG_FILE}" ;;
summary) grep -a 'RESULT' "${LOG_FILE}" 2>/dev/null || printf 'no results in %s\n' "${LOG_FILE}" ;;
clear)
is_running >/dev/null && die "still running, stop it first"
rm -f "${LOG_FILE}" "${LOG_FILE}".[0-9]*
rm -f "${LOG_FILE}" "${LOG_FILE}".[0-9]* "${RAW_10G}" "${RAW_1G}" "${STATE_FILE}"
printf 'log cleared\n'
;;
__daemon) do_daemon ;;
-h|--help|"") usage ;;
*) printf '[ERROR] unknown sub-command: %s\n\n' "$1" >&2; usage; exit 1 ;;
esac