From a31c186142b97b43c1b0c2fdcc5e1f950cc40680 Mon Sep 17 00:00:00 2001 From: ETWen Date: Wed, 26 Aug 2026 09:53:20 +0800 Subject: [PATCH] feat(mgmt): Ping both NICs at once, and stop counting ARP as loss The bench run reported FAIL=3 with no NIC errors and no drops. All three failures were on eth1 and all three were missing exactly icmp_seq=1, with the other nine replies present and sub-millisecond. That is neighbour resolution: once the ARP entry for the target expires, the first echo request is spent resolving it and ping counts it as loss. Rounds sat about 24 s apart, right on the edge of the default reachable time, so it happened on some rounds and not others. mgmt_ping_monitor.sh V3.0.0: - One discarded ping per NIC before the measured run resolves ARP, so MAX_LOSS_PCT can stay at 0 and a FAIL means a real drop. Raising the threshold instead would have hidden genuine single-packet loss, and a longer settle would not help -- nothing triggers ARP until the ping. - Both NICs now ping at the same time and keep accumulating until stopped, rather than alternating fixed bursts. Each writes its own raw capture. - `stop` sends SIGINT rather than SIGTERM, because ping prints its statistics block on interrupt and that block is what the report parses, then renders into the log: the last TAIL_LINES entries per NIC, each one's statistics and verdict, and `ip -s link show` for both interfaces. - `-D -O` are on by default so every line carries a timestamp and a request that got no reply is visible in the tail instead of merely absent. Clear PING_EXTRA_OPTS if the platform's ping rejects them. Since the script no longer re-execs itself to daemonise, `bash mgmt_ping_monitor.sh start` works without the execute bit. .gitignore: the new *.raw / *.pid / *.state runtime files were outside the *.log rule and would have been committed. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM --- .gitignore | 5 + src/Script_ABC_Blanton/1_Blanton_Script_A.ttl | 4 + .../Blanton_Script/mgmt_ping_monitor.sh | 594 ++++++++---------- 3 files changed, 276 insertions(+), 327 deletions(-) diff --git a/.gitignore b/.gitignore index 8a1fadb..03cda2a 100644 --- a/.gitignore +++ b/.gitignore @@ -12,6 +12,11 @@ For_AI/ !**/logs/.gitkeep !**/Logs/.gitkeep +# monitor 腳本的執行期產物(原始擷取、pid、狀態) +*.raw +*.pid +*.state + # 打包輸出:產出不進 git,打包工具本身要進 publish/* !publish/.gitkeep diff --git a/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl b/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl index 4e604d7..3392895 100644 --- a/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl +++ b/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl @@ -51,6 +51,10 @@ ; ScriptB USB stress uses the detected node, not a hard-coded /dev/sda1 ; ScriptB DDR memtester runs continuously (drop the 100-pass limit) ; ScriptC Move bgctl reset --yes after the USB archive +; Blanton_Script/mgmt_ping_monitor.sh V3.0.0 - both NICs ping simultaneously, accumulating +; + ARP warm-up so a stale neighbour entry is not counted as loss +; + stop renders last-N per NIC, stats, and both ip -s link show +; .gitignore Ignore monitor *.raw / *.pid / *.state ; ============================================================================= include "config.ttl" diff --git a/src/Script_ABC_Blanton/Blanton_Script/mgmt_ping_monitor.sh b/src/Script_ABC_Blanton/Blanton_Script/mgmt_ping_monitor.sh index 7ed96e3..c6acc99 100644 --- a/src/Script_ABC_Blanton/Blanton_Script/mgmt_ping_monitor.sh +++ b/src/Script_ABC_Blanton/Blanton_Script/mgmt_ping_monitor.sh @@ -2,48 +2,41 @@ ############################################################################### # mgmt_ping_monitor.sh # -# Version : V1.0.0 +# Version : V3.0.0 # Author : ETWen -# Date : 20260824 +# Date : 20260826 # -# Purpose : Exercise both management NICs and ping each one's own target, -# appending everything to a rotating log. One round is +# Purpose : Ping from BOTH management NICs at the same time, continuously, and +# keep accumulating until stopped: # -# 10G (eth0) : ip link up + address/route replace -> settle -# -> ping N x TARGET_10G -> ip -s link show -# 1G (eth1) : ip link up + address/route replace -> settle -# -> ping N x TARGET_1G -> ip -s link show +# 10G (eth0) -> TARGET_10G +# 1G (eth1) -> TARGET_1G # -# Designed to run detached in the background, same sub-commands as -# bmc_monitor.sh: start/stop/status/fg/tail/clear. +# `stop` renders a report into the log: the last TAIL_LINES entries +# per NIC, each one's ping statistics and verdict, then +# `ip -s link show` for both interfaces. # -# Notes : - Both NICs are brought up and LEFT up. Addresses and default -# routes are applied with iproute2 `replace`, which is idempotent: -# re-running a round does not stack duplicate addresses or routes. -# - The default routes carry different metrics (100 / 200), so the -# lower one wins for off-subnet traffic. They do NOT decide which -# NIC a ping leaves by: `ping -I ` binds to the device. -# - !! SAME-SUBNET WARNING !! With both NICs up on one subnet the -# target ARPs for the address and Linux may answer from either -# NIC, so replies can arrive on the wrong one and the result is -# not attributable. ARP_STRICT=1 sets arp_ignore/arp_announce to -# stop that. Leave it on unless the two NICs are on separate -# subnets, as in the iproute2 example this was modelled on. -# - Each leg logs `ip -s link show ` and the TX/RX packet -# delta across the burst, which is the direct evidence of which -# NIC actually carried the traffic. -# - The first pings after bringing a link up are lost to ARP, not to -# the link: a bench capture showed 4x "Destination Host -# Unreachable" on the first burst and 0% loss on the very next -# one. LINK_SETTLE_SEC absorbs that; it is not packet loss. +# Notes : - Both NICs are configured with iproute2 `replace`, which is +# idempotent, and both are left up. On a shared subnet the target's +# ARP can be answered by either NIC, so ARP_STRICT keeps each one +# to its own address. +# - ARP WARM-UP: the first packet after a neighbour entry expires is +# spent resolving ARP and is counted as loss. On the bench this +# produced three FAILs whose only missing packet was icmp_seq=1, +# every time, with zero NIC errors or drops. One discarded ping +# before the measured run removes that artefact, which is what lets +# MAX_LOSS_PCT stay at 0 and still mean something. +# - MANAGEMENT CONNECTIVITY IS IN USE while this runs. Drive it from +# the serial console. # # Version History -# V1.0.0 20260824 Initial Version -# V2.0.0 20260824 Switch from ifconfig to iproute2 (ip link / address -# replace / route replace with per-NIC metric). Both -# NICs now stay up -- no link down between legs. Add -# `ip -s link show` per leg with a TX/RX delta, and -# ARP_STRICT for the same-subnet case. +# V1.0.0 20260824 Initial Version (ifconfig, one NIC at a time) +# V2.0.0 20260824 iproute2; both NICs stay up; per-NIC target; +# ip -s link show + TX/RX delta per leg +# V3.0.0 20260826 Both NICs ping SIMULTANEOUSLY and continuously instead +# of alternating fixed bursts. Add the ARP warm-up. +# `stop` renders last-N per NIC + statistics + both +# ip -s link outputs into the log. ############################################################################### set -u @@ -63,149 +56,67 @@ METRIC_10G=100 # lower metric wins for off-subnet traffic # --- 1G management port --- IF_1G="eth1" IP_1G="192.168.1.101" -TARGET_1G="192.168.1.31" # host this NIC pings +TARGET_1G="192.168.1.31" PLEN_1G=24 GW_1G="" METRIC_1G=200 # --- ping --- -PING_COUNT=10 # echo requests per leg -MAX_LOSS_PCT=0 # loss above this marks the leg FAIL +PING_INTERVAL_SEC=1 # seconds between echo requests (ping -i) +MAX_LOSS_PCT=0 # loss above this marks the NIC FAIL +TAIL_LINES=20 # how many recent entries per NIC the report shows -# Both NICs stay up. If they share a subnet the target's ARP can be answered -# by either one, so a reply may arrive on the NIC that did not send. Setting -# this applies arp_ignore=1 / arp_announce=2 to both, which keeps each NIC to -# its own address. RAM only -- reverts on reboot. Set 0 for separate subnets. +# Extra ping flags. -D timestamps every line, -O prints a marker for a request +# that got no reply, so a drop is visible in the tail instead of just missing. +# Clear this if the platform's ping does not accept them. +PING_EXTRA_OPTS="-D -O" + +# One discarded ping per NIC before the measured run, to resolve ARP. Without +# it the first packet of the run is lost to neighbour resolution and looks +# exactly like a link fault. 0 disables. +ARP_WARMUP=1 +ARP_WARMUP_TIMEOUT=2 # seconds to wait for the warm-up reply + +# Both NICs stay up. If they share a subnet the target's ARP can be answered by +# either one, so a reply may arrive on the NIC that did not send. This applies +# arp_ignore=1 / arp_announce=2 to both. RAM only -- reverts on reboot. ARP_STRICT=1 -# Prefix for the ip commands; empty because the daemon already runs as root. +# Prefix for the ip commands; empty because this already runs as root. SUDO="" -# Seconds to wait after bringing a link up before pinging, so ARP is resolved -# and the PHY has finished autoneg. The bench needed a few seconds on the 1G. +# Seconds to wait after bringing a link up before pinging. LINK_SETTLE_SEC=5 -# Seconds between rounds (0 = start the next round immediately) -INTERVAL_SEC=10 - -# Per-ping-burst timeout in seconds; stops a wedged ping from freezing the -# loop. 0 disables. Must exceed PING_COUNT (one packet per second). -PING_TIMEOUT_SEC=60 - -# Stop after this many rounds. 0 = run until stopped manually. -MAX_ROUNDS=0 - # Logging -LOG_DIR="" # empty =