etwen/Blanton_TTL_Script
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 24 Wiki Activity

24 Releases 24 Tags

RSS Feed
  • V1.0.8 5e1e4a6a8f
    Compare

    V1.0.8 — Both management links tested at once, and ARP stops looking like packet loss Stable

    etwen released this 2026-08-26 10:13:34 +08:00 | 50 commits to main since this release

    ✨ New features

    Both management NICs are pinged simultaneously and continuously

    • mgmt_ping_monitor.sh no longer alternates fixed bursts. start configures both NICs, then pings from each at the same time and keeps accumulating until stopped — so a soak-long run is one continuous measurement rather than a series of snapshots.
    • stop renders a report into the log: the last 20 entries per NIC, each one's ping statistics and PASS/FAIL, then ip -s link show for both interfaces.
    • Every line is timestamped, and a request that got no reply prints a marker instead of merely being absent — a drop is visible in the tail, not inferred from a gap in the sequence numbers.
    • status shows both pids and how many replies each NIC has received so far, which is the quick way to see one side is dead without waiting for the report.

    The USB stress target is detected, not assumed

    • New usb_target.sh reports an inserted USB device's node, mount point or by-id path. Script B asks it for the node and builds the stress command from the answer, so a stick that enumerates as sdb no longer sends a write test at whatever /dev/sda1 happens to be.
    • It excludes whatever disk backs / and /host — this platform can boot from a USB DOM, which also reports as USB — refuses to guess when several USB disks are present, and when it mounts, verifies the result is actually writable rather than trusting that mount succeeded.

    BMC I2C integrity is exercised through the soak

    • bmc_monitor.sh is back in the run and now writes two complementary patterns to a BMC scratch register and reads each back. One pattern alone cannot catch a bit stuck the same way it was written; repeating the pair through the soak turns an intermittent I2C fault into something the log records rather than something the tester has to witness.

    Test artefacts leave the DUT with the run

    • Script C copies bmc_poll.log and mgmt_ping.log into the job directory before it is archived to USB, so the monitors' output travels with the bgctl job logs.
    • DDR stress now runs continuously instead of stopping after 100 passes, matching the other soak loads.

    🐛 Bug fixes

    Three FAILs that were not link faults

    • A bench run reported FAIL=3 with zero NIC errors, zero drops and sub-millisecond replies. Every failure was missing exactly icmp_seq=1 and nothing else: once the neighbour entry for the target expires, the first echo request is spent resolving ARP and ping counts it as loss. A discarded warm-up ping per NIC now absorbs that, which is what lets the loss threshold stay at zero and still mean something. Raising the threshold instead would have hidden genuine single-packet loss.

    Job logs were being cleared before they were collected

    • Script C ran bgctl reset --yes while killing processes — before the job directory is copied to USB. Moved to after the archive, so a run's own logs are collected before anything clears them.

    The fan controller was bound twice

    • One of the two MAX31790 controllers received a second bind immediately after the first, which could only fail because the driver was already attached.

    📦 Downloads

    File Contents
    Script_ABC_Blanton_V1.0.8.zip The full Tera Term working directory, including Blanton_Script/ to copy onto the DUT

    After copying to the DUT:

    chmod +x ~/Blanton_Script/*.sh      # see below
grep -rlU $'\r' ~/Blanton_Script    # expect no output

    mgmt_ping_monitor.sh no longer re-execs itself, so bash mgmt_ping_monitor.sh start works without the execute bit. bmc_monitor.sh and usb_target.sh still need it — and neither git (mode 100644) nor a Windows/USB copy carries it. Without the chmod, start fails with Permission denied and the later cat finds nothing: the section ends up empty and nothing reports an error.

    ⚠️ Before you run

    • Declare the switch population. SWB_UNIT0 / SWB_UNIT1 in config.ttl say which switch units this DUT has. Traffic and the readiness gate both follow them; with neither set the traffic stage is skipped rather than failing against hardware that is not there.
    • Set the fan speed you want. FAN_SPEED in config.ttl is applied at the start of every run.
    • Management connectivity is in use, not preserved. Both NICs are up and pinging — drive the run from the serial console.
    • If the two management NICs share a subnet, keep ARP_STRICT=1. Otherwise the target's ARP can be answered by either NIC and a reply may arrive on the one that did not send. Check nic_tx / nic_rx on the RESULT line: they are that interface's own counter delta.
    • Two settings persist to config_db.json and survive a reboot: LLDP is left disabled and the 100G uplinks are left configured. Restore them before the DUT moves on.

    🔗 Links

    Full changelog: V1.0.7...V1.0.8

    Downloads
  • V1.0.7 ef7cbdffd4
    Compare

    V1.0.7 — Management-link ping test, and the fans start where you set them Stable

    etwen released this 2026-08-25 14:22:22 +08:00 | 56 commits to main since this release

    ✨ New features

    10G / 1G management port ping test

    • New mgmt_ping_monitor.sh exercises both management NICs in one loop: it brings each up with iproute2, waits for the link to settle, then pings that NIC's own target — eth0 → 192.168.1.30, eth1 → 192.168.1.31.
    • Runs detached like the other monitors: start / stop / status / fg / summary. start clears the previous log, so one run means one log.
    • Every leg writes a single greppable line — RESULT 10G eth0 -> 192.168.1.30 tx=10 rx=10 loss=0% rtt_avg=1.743ms nic_tx=10 nic_rx=10 PASS — so a whole soak can be read with summary instead of scrolling.
    • nic_tx / nic_rx are that interface's own counter delta across the burst. If a ping passes but they stay near zero, the traffic left on the other NIC and the number does not belong to this one.
    • Script A runs one round as a baseline; Script B leaves it running through the soak.

    Fan speed is set at the start of every run

    • FAN_SPEED in config.ttl (default 30) is applied by Script A, so a run no longer inherits whatever the previous test left the fans doing.
    • Both MAX31790 controllers are rebound first, which clears a controller left in an odd state.

    Stress load moved onto the platform's own job runner

    • CPU, DDR, SSD and USB stress now go through bgctl, replacing the ~/hammer/tools/stress_*.py scripts. bgctl list shows what is running; Script C stops everything with bgctl stop --all.
    • BMC DDR stress runs through bmc-manager, and a new BMC USB test discovers the cdc_ncm interface and pings across it for four hours under systemd-run.

    More of the DUT on record

    • Script A now captures the boot image, show version, firmware status, system EEPROM, SSD health, TPM version and NVMe SMART data before the test starts.
    • PCIe AER now sits next to ras-mc-ctl --summary, and Script C closes with NVMe health so a disk that degraded during the soak is visible.
    • dmesg capture uses human-readable timestamps and one combined error pattern, then clears the ring buffer — so Script C's dmesg shows only what the soak produced, not everything since boot.
    • Each run opens with hw-test-session start and closes with finish; Script C copies the job logs to /mnt/usb/jobs-<date>_<time> so they leave the DUT with the run they belong to.

    🐛 Bug fixes

    Script A no longer hangs waiting for ports that are already up

    • The readiness gate required more than 216 ports up, but a fully cabled unit reports exactly 216 — 108 loopback pairs times two. Script A sat in the poll loop forever on the very state it was waiting for. It now proceeds at 216.
    • The comments in wait_init.ttl still described the old "more than" rule; they now match the code, so the off-by-one cannot be reintroduced by reading the wrong line.

    The macro no longer runs ahead of the DUT during the soak

    • Script B set a 15-second wait cap for the cdc_ncm probe and never restored it. In Tera Term that cap is global, so it stayed in force for everything after — including the traffic setup, which walks 108 VLANs per switch unit and takes far longer. A timed-out wait returns without the prompt, leaving every later command issued a step early.

    Fan controller is no longer bound twice

    • One of the two MAX31790 controllers received a second bind immediately after the first, which could only fail because the driver was already attached.

    📦 Downloads

    File Contents
    Script_ABC_Blanton_V1.0.7.zip The full Tera Term working directory, including Blanton_Script/ to copy onto the DUT

    After copying to the DUT:

    chmod +x ~/Blanton_Script/*.sh      # required -- see below
grep -rlU $'\r' ~/Blanton_Script    # expect no output

    The chmod +x is not optional. mgmt_ping_monitor.sh is executed rather than sourced and re-execs its own path to daemonise, and neither git (mode 100644) nor a Windows/USB copy carries the execute bit. Without it start fails with Permission denied and the later cat finds nothing — the section ends up empty and nothing reports an error.

    ⚠️ Before you run

    • Declare the switch population. SWB_UNIT0 / SWB_UNIT1 in config.ttl say which switch units this DUT has. Traffic and the readiness gate both follow them: a single unit gets -u 0 or -u 1, and with neither set the traffic stage is skipped instead of failing against hardware that is not there.
    • Both management NICs stay up. If they share a subnet, keep ARP_STRICT=1 — otherwise the target's ARP can be answered by either NIC and a reply may arrive on the one that did not send.
    • Management connectivity is exercised, not preserved. Drive the run from the serial console.
    • Two settings persist to config_db.json and survive a reboot: LLDP is left disabled, and the 100G uplinks are left configured. Restore them before the DUT moves on.

    🔗 Links

    Full changelog: V1.0.5...V1.0.7

    Downloads
  • V1.0.5 9a11c1d5bb
    Compare

    V1.0.5 — Partly populated DUTs and readiness gating Stable

    etwen released this 2026-08-21 16:06:54 +08:00 | 60 commits to main since this release

    Blanton TTL Script V1.0.5

    One test run is Script A (baseline) -> B (soak) -> C (verdict), all captured in a single Tera Term log.

    ✨ New features

    Partly populated DUTs

    • config.ttl declares the switch population once:

      SWB_UNIT0 = 1   ; this DUT has switch unit 0
SWB_UNIT1 = 1   ; this DUT has switch unit 1

    • The traffic blocks in A, B and C branch on it: no -u when both units are fitted (the tool's own TL_UNITS="0 1" applies), -u 0 or -u 1 for a single one, and the whole stage is skipped when neither is set

    • utils/wait_init.ttl reads the same two flags, so the wait and the traffic cannot disagree about what is installed

    Readiness gate

    • Script A now waits on the data plane rather than the sensors: it polls bcmcmd -n <u> -c ps | grep -w up | wc -l and proceeds once every fitted unit reports at least WT_MIN (216) ports up — 216 being 108 loopback pairs x 2, i.e. every cabled port

    BMC

    • bmc_monitor.sh (samples free -m) and bmc_monitor_ddr.sh (memtester) run through the soak via bmc-manager run, started by Script B and collected by Script C

    Inventory and link state

    • Script A: boot image, board EEPROM (hpe-eeprom-tlv), BMC version, 100G uplink configuration and status, uptime
    • Script C: show reboot-cause and uptime, so an unplanned reset during the soak shows up in the closing snapshot
    • LLDP disabled and transceiver low-power mode cleared before traffic, so the loopback counters only reflect the injected burst

    🐛 Bug fixes

    • Script A never got past the port wait. The threshold was strictly greater than 216, but a fully cabled unit reports exactly 216, so the loop spun forever on the very state it was waiting for. Now "at least 216"
    • Single-unit traffic paths were unreachable. All nine branch conditions compared SWB_UNIT0 against itself, so a DUT with only unit 0 ran the both-unit commands against an absent unit 1, and a DUT with only unit 1 skipped traffic altogether
    • config save -y was sent without waiting for the prompt, leaving the macro one step ahead of the DUT from that point on

    ⚠️ Deployment notes

    scp -r src/Script_ABC_Blanton/Blanton_Script admin@<DUT>:~/
chmod +x ~/Blanton_Script/bmc_monitor*.sh      # required -- see below
grep -rlU $'\r' ~/Blanton_Script               # expect no output

    The chmod +x is still not optional. bmc_monitor*.sh are executed rather than sourced, do_start re-execs its own path, and neither git (mode 100644) nor a Windows/USB copy carries the execute bit. Without it Script B's start fails with Permission denied and Script C's cat finds nothing — the BMC section ends up empty and nothing reports an error.

    Two settings in this build persist to config_db.json and survive a reboot: LLDP is left disabled, and Ethernet513 is left shut down. Restore them before the DUT moves on to other work.

    📦 Downloads

    File Contents
    Script_ABC_Blanton_V1.0.5.zip The full Tera Term working directory, including Blanton_Script/ to scp onto the DUT
    Downloads
  • V1.0.3 419b680980
    Compare

    V1.0.3 — Traffic, BMC monitoring and SWB margin fixes Stable

    etwen released this 2026-08-21 09:33:06 +08:00 | 67 commits to main since this release

    Blanton TTL Script V1.0.3

    One test run is Script A (baseline) -> B (soak) -> C (verdict), all captured in a single Tera Term log.

    ✨ New features

    Traffic

    • blanton_traffic_linespeed.sh drives SWB loopback line-speed traffic over bcmcmd, VLAN 30..137 pairing cdN with cdN+32
    • Per-pair TX/RX cross-check (cdA.TX == cdB.RX) yields PASS/FAIL; awk on the DUT, tools/bcm_mibpair_report_V1.1.0.py off-box
    • All three scripts now cover both switch units

    BMC

    • bmc_monitor.sh samples free -m; bmc_monitor_ddr.sh runs memtester as a stress load. Both reach the BMC through bmc-manager run from the host, so no interactive SSH to the BMC is needed
    • Script B starts them, Script C stops them and folds their logs into the master log

    Data collection

    • Script A records show boot, the BMC version, 100G port status and show uptime, and waits for pmon to come up before taking a baseline (utils/wait_init.ttl)
    • Script C adds show reboot-cause and show uptime, so an unplanned reset during the soak is visible in the closing snapshot
    • PMON snapshot extended with leak status and leak channels

    Packaging

    • publish/publish.sh builds publish/Script_ABC_Blanton_<Ver>/ (-z for a zip). The version is read from Script A's header, so the folder name cannot disagree with the macro

    🐛 Bug fixes

    • Every counter in the traffic report printed as 2147483647 — the DUT's awk casts to a 32-bit int for %d, and a real line-speed run is around 1e12. Now formatted with %.0f. Verdicts were never affected: the cross-check compares doubles and only the display path was wrong
    • SWB I2C channels and LTC2977 addresses — corrected against the bench. CONN13/15 and CONN14/16 were paired to the wrong channels, and addresses step by 2, not 1. This also clears the collision where two CONNs both landed on 0x5E on Ch6
    • CRLF made margin.sh unrunnable on SONiC ($'\r': command not found). .gitattributes now pins *.sh and *.conf to LF. settings/*.conf was the quieter half: they are sourced, so CB_I2C_CH=6 became "6\r"
    • margin.sh v2.6.0 — the SWB path never found the cb_i2c backend, so TRANSPORT=swb always failed to load. It now searches the parent Blanton_Script/ directory, and the duplicated backend copies were removed
    • Script C hung — it called a subcommand that runs tail -f and never returns, so the run never reached the stress results or the closing messagebox. Replaced with a direct cat of the logs

    ⚠️ Deployment notes

    scp -r src/Script_ABC_Blanton/Blanton_Script admin@<DUT>:~/
chmod +x ~/Blanton_Script/bmc_monitor*.sh      # required -- see below
grep -rlU $'\r' ~/Blanton_Script               # expect no output

    The chmod +x is not optional. bmc_monitor*.sh are the first scripts here meant to be executed rather than sourced, do_start re-execs its own path, and neither git (mode 100644) nor a Windows/USB copy carries the execute bit. Without it Script B's start fails with Permission denied and Script C's cat finds nothing — the BMC section ends up empty and nothing reports an error.

    📦 Downloads

    File Contents
    Script_ABC_Blanton_V1.0.3.zip The full Tera Term working directory, including Blanton_Script/ to scp onto the DUT
    Downloads