Every PDB reading came back NA, and the cause was a base error. The four
ADPM12200 bricks put each measurement on its own PMBus PAGE, written
before the read. The bring-up spreadsheet listed those page numbers in
decimal but with an 0x prefix:
Vin PAGE 9 -> it said 0x00 (0x09)
Iin PAGE 10 -> it said 0x10 (0x0A)
Vout PAGE 2 -> it said 0x02 correct by luck
Iout PAGE 14 -> it said 0x14 (0x0E)
Temp PAGE 18 -> it said 0x18 (0x12)
Only Vout worked, because 2 reads the same in either base. Hence Iin and
Iout answering 0xFFFF, Temp answering 0x0000, and a Vin scale factor
having to be invented to make 50 V appear out of a page-0 register.
pwr_brick.sh reads all five bricks with the right pages and the
datasheet's DIRECT equation, X = (1/m)(Y x 10^-R - b): voltage Y x 8 mV,
current Y x 0.04 A, temperature Y x 0.01 C. Cross-checked rather than
assumed - READ_VOUT 0x05D4 decodes to 11936 mV, exactly what
'show platform voltage' reports for that rail through a separate sensor
path. The datasheet is committed alongside so the numbers are checkable.
Left flagged: Table 3 does not list READ_VIN or READ_IIN. They borrow the
voltage and current coefficients here, which is consistent with a ~50 V
input but is not something the datasheet states.
Each value is re-read until it passes three checks: not 0xFFFF, not
0x0000, and inside a plausibility window. The window is the one that
matters - this bus corrupts the HIGH BYTE only, which turns 11.98 V into
32.00 V. Positive, plausible in magnitude, and invisible to any all-ones
filter. 0x0000 is rejected for the mirror-image reason: on the
temperature register it decodes to a believable 0 C.
Adds comboA/comboB margin profiles per board. comboB is comboA with every
direction flipped, so the pair covers each rail high and low while its
neighbours sit the other way. NC channels stay nominal in both - there is
nothing to margin on an unused rail.
Script C re-enables the BMC USB journalctl dump, which had been commented
out, so that service log now reaches the master log.
Script A -> V1.1.6.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
qfx5252-stress-usb was started with --runtime 60. It finished a minute
into an overnight run and the USB path sat idle for the rest of the
night - nothing failed, nothing said so, and the log looked complete.
SSD had the same shape at --runtime 3600. Both are now 86400, and the
BMC DDR memtester drops its loop count so it runs until stopped.
86400 is 24h, not forever. Past that SSD and USB die quietly and the
only clue is the shorter bgctl list Script B already prints each round.
Said so in the version history, the release notes and ARCHITECTURE.
Adds blanton_multiphase_margin.sh: the multiphase controllers on both
switch boards over the native i2c buses (SWB0 on i2c-13/15, SWB1 on
-14/-16), 19 rails each. high3/low3/comboA/comboB set voltages and print
one line per rail; save commits them and prints Multiphase Save All,
because that list has no per-rail mapping to report against.
Two hazards it carries: save is STORE_USER_ALL and permanent, and these
are NOT the LTC2980s margin.sh drives - the two tools reach the same
rails from opposite sides with no interlock between them.
The emitted commands were diffed against tools/Multiphase_*.txt: argv
identical for all five. The rail-to-netname mapping was cross-checked
against the bus numbers rather than trusted by line position - blocks
1..19 are all bus 13/15 and 20..38 all 14/16, matching the SWB0/SWB1
split of Multiphase_Netname.
blanton_traffic_linespeed v0.7.0 appends PatternRandom=yes to every tx
including ce0, and the defaults become tx 200 length=324. A fixed
pattern can sit on a benign bit sequence for a whole run; a changing one
is what exposes a weak lane. -r no forces fixed, -r "" omits the
argument for an SDK that will not take it. Bytes per burst went 51200 ->
64800, so absolute counter totals do not carry across this build - the
pair cross-check that decides PASS/FAIL does.
Script A -> V1.1.5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
4_Blanton_Script_thermal_safety.ttl brings the DUT up the way Script A
does, then loads it - mlucas-avx2 -cpu 0:15 in the background plus
blanton_tr518.sh start - and samples platform data and both switch
boards' rail power every minute. Power per rail under sustained full
load, over hours, is the whole point of it.
It is standalone: nothing starts it, it starts nothing, and it must not
share a DUT with Script B because both take over the cd ports.
It also has no teardown. When you interrupt it the unit is still in
port cd lb=mac / l2 learn off / test mode nr=yes and mlucas is still
running, and show interfaces status does not reveal any of that. Called
out in the version history, the release notes, CLAUDE.md gotchas and the
ARCHITECTURE future-work list, because whoever uses the box next will not
find out on their own.
Script B's per-round loop drops the margin scan. The loop is meant to be
a quick minute-by-minute sample and margin_status_all.sh walks nine
LTC2980s over I2C, which dominated the round. Script C takes it instead,
once, before show uptime. The trade-off is real: margin drift DURING the
soak is no longer visible, only a start and an end reading. Documented
rather than glossed over, with a suggestion (scan every N rounds) if it
needs to come back.
Script C also sends exit and waits for "Script done" before
hw-test-session finish, so finish runs in the login shell rather than
inside the session it is closing.
Adds docs/Blanton_Test_Flow.drawio: all four scripts side by side, with
the config flags that change each step marked, Script B's one real goto
drawn, and the A->B->C handoffs shown as the manual steps they are.
The new script arrived LF-only while every other .ttl is CRLF and
.gitattributes marks *.ttl as -text, so what is committed is exactly what
Tera Term reads. Converted.
Script A -> V1.1.4.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
86 of the 144 margin channels moved up +0.8% .. +2.5% to the measured
median for this platform (3.3 -> 3.38, 0.9 -> 0.915, 0.8 -> 0.82, ...).
That is not a display change. VNOM is the denominator of margin_status'
deviation %, and it is also the basis margin_set derives high/low/OV/UV
from - so both the number you read and the voltage you apply move with
it. A rail steady at 0.756 V used to read +0.80% against a nominal 0.75
and now reads 0.00%. Margin logs from before and after cannot be
compared: the reference moved, the hardware did not.
comboA/comboB are replaced by combo_SWB_CONN13..16_{high,low}. The
per-channel percentages are positional, so they follow the rails of one
specific board - applying a CONN13 profile to a CONN16 margins the wrong
rails and nothing detects it. SWB0 and SWB1 share a profile per CONN
because their rails are identical.
Fixes two things the deletion broke:
- margin_apply_profile_all.sh still named comboA for all nine boards,
which would now stop at "Profile not found" nine times over. It names
the matching per-CONN profile instead, with CB on the generic
combo_high3, and a note that the low pass is _high -> _low.
- docs/LTC2980_channel_map.csv is generated from settings/*.conf, so
all 86 changed rails were stale in it. Regenerated.
The eight new profiles also still carried comboA's header comment
("alternating high/low"), which describes neither what they contain nor
what they are for.
Script A -> V1.1.3.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
nfc_polling.sh wraps blantons_nfc_validate.py in a background loop and
keeps score: start/stop/status/tail/report/clear/fg. Script A takes a
30s baseline, Script B starts the poll for the soak, Script C stops it
with the other monitors and reports.
report only reads the log, so it can be run mid-soak without disturbing
anything - and it says RUNNING so a snapshot is not read as the final
answer. Each round appends a machine-readable RESULT line, which keeps
the tally independent of the tool's wording.
ERROR is counted apart from FAIL. A round where the tool printed neither
[PASS] nor [FAIL] - crashed, missing, hung past the timeout - measured
nothing, and that is a different fault from a tag that would not read.
Rolling them together hides whichever one you are not looking for.
Two habits from earlier bugs are baked in: tail does not follow (tail -f
never returns and a macro waiting on the prompt hangs there), and the .py
is called through python3 rather than ./ (it arrives mode 100644 and a
USB copy carries no execute bit).
Also fixes Script C copying "nfc_polling.log" to the job dir when the
file is "nfc_poll.log". The cp failed, nothing checked it, and the run
finished looking fine - you would find out at the desk, with a job
archive that had no NFC log in it.
Script A -> V1.1.2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
After a run that ended badly - Tera Term closed, DUT rebooted mid-soak,
macro stopped by hand - the stress jobs kept running and the test session
stayed open. The next run then started on a box that was already loaded.
Nothing errored; the numbers were just quietly wrong.
Script A now opens with hw-test-session finish, bgctl reset --yes,
bgctl stop --all, then bgctl list. The list is the point of the sequence:
it is the one line in the log that proves the DUT was idle at the start.
Script A -> V1.1.1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
Three new tools and one fix to an old one.
blanton_ber.sh - PRBS BER across every cabled port, init/start/report/
stop/clear, -u 0|1|all. A lane passes only below 1e-6 (equal to the
threshold fails). report ends with a per-unit summary carrying the worst
lane, so an all-PASS run still shows how much margin there was. init
checks every lane locked, because an unlocked lane still reports a BER
and it is a meaningless one. Script C runs it after the traffic report,
following SWB_UNIT0/SWB_UNIT1 like the traffic block does.
blanton_tr518.sh - the built-in packet test, used to load the rails
before measuring power. start settles 120s so the reading afterwards is
loaded rather than mid-ramp. stop sends port cd lb=none and nothing else,
exactly what the bench procedure does - l2 learning and test mode are
left as start set them, and stop says so instead of quietly leaving the
box in a state nobody asked about.
TH6_SWB{0,1}_power_readback.sh - 19 rails, V x I and total. These read
the two sensor tables ONCE. The previous version read them per rail: 38
invocations of a slow CLI at 38 different instants, so the "total" was a
sum of readings seconds apart - and under load the currents move (TH6_CORE
186 A -> 636 A). A rail missing from the table now warns on stderr instead
of silently contributing 0 W.
blanton_traffic_linespeed.sh - ce0 (VLAN 138) joins the test. It has no
loopback partner and a switch never sends a frame back out its ingress
port, so a tx burst was one round trip and then silence. init now sets an
ingress mirror of ce0 back to itself; the mirrored copy is not
egress-filtered, so packets keep lapping until stop turns the mirror off.
stop is what ends it - removing the VLAN membership does not, because the
copy never consulted the VLAN. TL_CE_MODE also drops to 'self': ps ce
shows one ce port per unit and a unit-0-only burst returned on unit 0.
Read as cross-unit, both links dying would still report PASS (0 == 0).
Fixes a parser bug worth remembering: bcmcmd returns CRLF, and the BER is
the last token on its line, so the CR stayed glued to the number. Every
lane read as NA, and printing the CR sent the cursor to column 1 so the
RESULT column overwrote the start of the row. One stray \r, two symptoms.
Not yet verified on hardware, all called out in the release notes: the
ce0 mirror (destport == srcport may be refused), the 120s settle (tr 518
may be time-limited), and 'test mode nr=no' on stop.
Script A -> V1.1.0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
FAN_SPEED 30 -> 100. A soak is a thermal test of everything except the
cooling, so the cooling should not be one of the variables in it. At 30%
a long run can throttle partway through, and throttled results read like
a different fault entirely.
Folded into V1.0.11 rather than bumping the version: the tag has not
moved on and this is the same delivery.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
margin_save discarded the result of every write, so a NACK printed
"Change Saved" exactly like a successful store. For the one permanent
operation in this tool that is the worst place to stay quiet. Each store
is now checked, named on failure, and reflected in the exit code.
STORE_USER_ALL also went out as 0x15 followed by a dummy 0x00 - an SMBus
write-byte where the command is a send-byte. The LTC2977 tolerated it,
but tolerance is not correctness. Now i2cset ... 0x15 c on the native
path and a no-data cb_pmbus_write on the FPGA path.
Two new subcommands:
margin_save all - all nine boards in settings/, then re-init
whichever board was loaded beforehand so a
following margin_status still reports the board
you were looking at. Does not stop on the first
failure; a half-saved set is harder to reason
about than a fully-attempted one.
margin_save blanton - this platform exposes the CB FPGA F3 I2C
channels as native Linux i2c buses (Ch6->13,
Ch7->14, Ch8->15, Ch9->16; CB stays on bus 4),
so all 18 LTC2977 can be stored with plain
i2cset: no margin_init, no FPGA channel setup,
no pcimem. 18 commands instead of 42.
The 18 raw commands were verified on the DUT 2026-08-27; the wrapper was
tested against stubbed hardware only. Release notes say so.
Board list is MARGIN_BLANTON_STORE at the top of margin.sh - comment out
what this DUT does not have rather than letting it talk to absent boards.
Missing bus and unresponsive chip are reported as distinct failures.
MARGIN_STORE_SETTLE (0.5s) sits between stores because nothing here polls
the part's busy bit, and blanton fires 18 in a row.
Script A -> V1.0.11.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
Script B's monitoring loop had no pause at all - each round started the
instant the last one ended, so an overnight soak filled the log with
near-identical samples taken seconds apart. The section was even labelled
"Get data every 10mins" while running with no delay whatsoever.
The loop now ends each round with pause 60 and prints both job lists:
bgctl list for the platform job runner, jobs for anything the macro
backgrounded in that shell. A stress job that dies mid-soak shows up as a
shorter list on the next round instead of going unnoticed until the end.
60s is still hard-coded and the Status sheet asks for 10 minutes, so the
Phase 3 item stays open - this makes the loop sane, it does not close it.
Also adds docs/LTC2980_channel_map.csv: the nine settings/*.conf flattened
into one Board,CONN,Ch,NetName,Vnom sheet, generated by
tools/gen_channel_map.sh. The CSV is output, not source. NC channels are
kept as NC/- because the channel number is also the PMBus page - dropping
the gaps would shift every channel after them.
Two things the merged table makes visible: SWB0 and SWB1 carry identical
net names and voltages (only the bus and CB_I2C_CH differ), so those are
two copies that can drift apart silently; and there are exactly 8 NC
channels, all on CONN14/CONN15.
tools/100G_PRBS.txt, TR518.txt and fan_ctrl.txt join the existing
traffic_loopback_*.txt references. TR518 is not scripted yet and cannot
share the hardware with blanton_traffic_linespeed - both drive port
loopback and L2 learning.
Script A -> V1.0.10.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
port_prbs_monitor.sh drives PRBS on both 100G uplinks: lpmode off, arm
the pattern, then poll `phy diag <phy> prbs get` in the background until
stopped, counting a poll as PASS only when the output says "PRBS OK!".
prbsstat Ber is captured beside every poll for the record but does not
decide the verdict. `stop` runs prbsstat STOp and prbs clear before
rendering the report, so the test does not leave PRBS armed.
`start a` / `start b` runs one uplink alone, which is how a genuine port
fault is separated from the DUT not coping with two PRBS streams at
once. A port that was not run reports SKIP rather than FAIL -- deciding
on poll count alone would have made a deliberate single-port run look
like half the hardware was broken.
The port status is deliberately not printed at `start`: with PRBS armed
the link reports DOWN, which reads as a failure to anyone glancing at
the console. It appears in the report instead, after prbs clear and
labelled as the recovered state.
The calls in Script A, B and C are committed but commented out -- PRBS
is still under bring-up.
Fix: the job directory was written as /host/hw-eval/current/jobs in some
places and /host/hw-eval/jobs in others, so what Script B wrote was not
what Script C collected. Unified on /host/hw-eval/jobs.
Fix: Script C copied to /mnt/usb assuming it was mounted. It now finds
the device with usb_target.sh and mounts it first, so the logs actually
leave the DUT.
Script C also reorders its results -- stress logs, then the MGMT ping
report, then 100G status -- and cats the bgctl ssd/usb logs.
Script A -> V1.0.9. Docs and docs/release-notes/v1.0.9.md updated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
bmc_monitor.sh comes back into the run with a second job beyond the
`free -m` sampling: an I2C pattern test against bus 1, address 0x41,
offset 0x00C0 -- write 55AA55AA and read it back, then write AA55AA55
and read that back. Two complementary patterns catch a stuck bit either
way round, and repeating it through the soak turns an intermittent I2C
fault into something the log shows rather than something the tester has
to catch live.
Script B starts it alongside the other stress; Script C stops it and
cats log/bmc_poll.log.
bmc_monitor_ddr.sh stays disabled -- BMC DDR stress runs through
bgctl + bmc-manager.
Docs: the "disabled" notes I added last time now applied to only half
the pair and read as wrong for bmc_monitor.sh. Split so the monitor is
documented as live with its I2C test, and only the _ddr variant is
marked disabled.
Script A -> V1.0.8.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
config.ttl gains FAN_SPEED (30). Script A rebinds the two max31790
controllers (18-0020, 25-0020) and applies the speed with
fan-speed-control.sh, answering its confirmation prompt, so a run starts
from a known thermal state rather than whatever the last test left.
Script A also clears /host/hw-eval/current/jobs before the run, and
Script C copies that directory to /mnt/usb/jobs-<date>_<time> so the
bgctl job logs leave the DUT with the run they belong to.
Fix: 25-0020 was bound twice in a row -- the second bind can only fail,
since the driver is already attached. Removed; the two controllers now
have a matching unbind/sleep/bind each.
Script A -> V1.0.7, history covering the config.ttl and Script C changes
as well.
Docs: fan control and the job-log archive added to Tech Stack, Key
Features and Data Flow; the config.ttl example refreshed with FAN_SPEED
and the SWB_UNIT flags. Three risks recorded in Future Extensions --
`rm` without -r cannot clear the job subdirectories that Script C copies
with -r, the fan confirmation wait has no timeout and would hang
silently if the prompt ever changes, and /mnt/usb is never checked for
being mounted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
Script A: wait 45 s between starting and stopping mgmt_ping_monitor.
`start` returns after one second, but a round is roughly 30 s -- without
the pause the log held only the START banner and no RESULT line.
Docs brought up to date with the eight commits since 419b680:
- Tech Stack gains bgctl, systemd-run, iproute2 and hw-test-session;
the stress row moves off the hammer scripts
- Structure and Key Features cover mgmt_ping_monitor.sh, and mark
bmc_monitor{,_ddr}.sh as disabled -- their call sites in B and C are
commented out, though the files remain
- Data Flow rewritten for all three scripts
- Two constraints added: TTL's `timeout` is global and must be restored,
and two management NICs on one subnet cause ARP flux (hence
ARP_STRICT, and nic_tx/nic_rx as the evidence of which NIC sent)
- CLAUDE gotchas add the `ip -s link` column order, that a monitor's
`start` returns in one second rather than after a round, and that
show_dmesg now ends with `dmesg -C`
- Phase 3 ticks the move to bgctl; Phase 5 gains the mgmt-ping, NVMe and
EDAC parsing targets. Future Extensions notes that Script C never cats
mgmt_ping.log, so the soak's ping results stay on the DUT
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
publish/publish.sh packages src/Script_ABC_Blanton into
publish/Script_ABC_Blanton_<Ver>/, with -z for a zip, -f to overwrite,
-n for a dry run.
The version is read from Script A's "; Version :" header rather than
passed in, so the folder name cannot disagree with what the tester sees
on opening the macro. The header is CRLF, so the carriage return is
stripped -- left in, it becomes part of the directory name. Captured
logs are excluded (they carry DUT serials) but the empty logs/ is kept
because logopen writes there. .gitignore now admits the tool while
still ignoring its output.
config.ttl: testcase ENV, margin scan off for this run.
Docs brought up to date with the 14 commits since 8aa5c90 -- SWB
channel/address map, the traffic and BMC tooling, line-ending policy,
and the exec-bit and tail -f traps.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
Bring both docs in line with the five commits pulled from Gitea.
- BDF: pcimem V1.5.0 auto-detects the CB FPGA BDF, so the "edit the four
lines per DUT" guidance was inverted. Documented the detection order
and fpga_rescan, and flagged that utils/show_pcie_error_reg_*.ttl
still hard-codes its 7 BDFs -- the footgun is only half gone
- Traffic: documented blanton_traffic_linespeed.sh (bcmcmd SWB loopback,
per-pair TX/RX cross-check) and tools/bcm_mibpair_report + the three
loopback command tables; noted the route taken differs from the
SONiC-CLI approach Phase 4 originally planned
- Stress: stress_hhmd and stress_pcie were removed on 08-17, so the job
count is 8 not 10; soak loop now pulls full setup_pmon (9 items)
- Constraints: added the bcmcmd exit-0 and stdin-eating gotchas
- Phases: ticked 10 completed items, corrected Phase 3/4 acceptance
criteria, and moved BDF auto-detect out of Future Extensions
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
Bring the Blanton Tera Term TTL test suite under version control and
document how the pieces fit together.
- ARCHITECTURE.md: A/B/C script roles, host-to-DUT-to-FPGA data path,
config/settings/profile data models, PCIe AER + EDAC topology table,
10 key constraints, and 6 development phases derived from the
Status_20260814 spreadsheet's On-Going items
- CLAUDE.md: stack, smoke-test commands, conventions, and the hardware
footguns (per-unit BDF, VSPI vs VI2C timing, repeated START,
STORE_USER_ALL, ARB_LOST false positives)
- secret/: gitignored credential store with README + .example template,
so the DUT login stops living in 1_Blanton_Script_A.ttl
- .gitignore: secret/*, For_AI/, *.log, publish/, *.DSN
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM
etwenandClaude Opus 5