From 9cdd651d791d77fe45b54391939baf716ca3542a Mon Sep 17 00:00:00 2001 From: ETWen Date: Sun, 6 Sep 2026 17:21:52 +0800 Subject: [PATCH] feat(ttl): Run the CPU stress as a systemd unit, and add two config knobs mlucas-avx2 was backgrounded with '&' in the login shell, so it belonged to that one terminal. Close Tera Term, drop the console, or reconnect on a different session and the load was gone - and 'jobs' from the new session showed nothing to say so. A multi-day soak could spend most of its length with an idle CPU while the thermal numbers looked fine. Script B now starts it as a transient unit: systemd-run --unit=mlucas-amm --property=Type=exec \ --property=Restart=always --property=RestartSec=2 ... The log is truncated once with '>' before the unit starts and appended with '>>' inside it, so a restart adds to the run instead of erasing it. 'systemctl is-active mlucas-amm' now sits beside 'jobs', both in the one-shot check and in every pass of the monitor loop. The old '&' line is kept commented out above the new one to fall back to. Script C has the matching stop. utils/kill_all_process.ttl is only "kill $(jobs -p)" and a systemd unit is not a shell job, so without this the load would have run through the post-test snapshot and into the next test - every temperature and power reading taken at full load, with nothing in 'show interfaces status' to reveal it. mlucas_amm_log is copied to /host/hw-eval/jobs/ with the other stress logs. Two values move into config.ttl: EN_timestamp feeds logopen's 5th argument in Script A, so the per-line timestamp is a knob rather than a literal 1 BMC_USB_RUNTIME feeds Script B's ping unit; it was hard-coded to -w 14400 (4 h), which no longer matched the soak lengths the SSD and USB knobs were being set to SSD_RUNTIME and USB_RUNTIME also go 600 -> 3600, so the shipped default is a one hour stress rather than the ten minutes V1.1.7 went out with. --runtime is still the job's overall limit; COUNT does not multiply it. Script A's Version History carries the V1.1.8 entries for all of this. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014RetWKFZFG1ZHcitQAyhwM --- src/Script_ABC_Blanton/1_Blanton_Script_A.ttl | 77 ++++++++++++++++++- src/Script_ABC_Blanton/2_Blanton_Script_B.ttl | 13 +++- src/Script_ABC_Blanton/3_Blanton_Script_C.ttl | 8 +- src/Script_ABC_Blanton/config.ttl | 10 ++- 4 files changed, 98 insertions(+), 10 deletions(-) diff --git a/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl b/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl index 4460afb..8c0a051 100644 --- a/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl +++ b/src/Script_ABC_Blanton/1_Blanton_Script_A.ttl @@ -1,7 +1,7 @@ ; ============================================================================= ; Script A for Blanton -; Version : V1.1.7 -; Date : 2026-09-04 +; Version : V1.1.8 +; Date : 2026-09-06 ; Author : ETWen ; ============================================================================= ; Version History: @@ -341,6 +341,73 @@ ; sprintf2 cmd_ssd ... SSD_RUNTIME SSD_COUNT, and the same ; for USB; the old hard-coded lines are left commented out ; next to them +; V1.1.8 2026-09-06 +; ScriptA Drive logopen's timestamp flag from EN_timestamp +; logopen filename 0 1 1 EN_timestamp, with the argument +; order spelled out above the call: +; <timestamp> +; binary must stay 0. In binary mode Tera Term writes the +; raw stream and both PlainText and the timestamp are +; ignored +; !! EN_timestamp has to exist in config.ttl. An undefined +; variable aborts the macro at logopen, it is not read +; as 0 +; !! the timestamp FORMAT lives in TERATERM.INI (Setup > +; Additional settings > Log), so it follows the PC and +; not this script. Two testers can produce differently +; stamped logs from one macro +; ScriptB Run mlucas as a systemd unit, not a shell background job +; systemd-run --unit=mlucas-amm --property=Type=exec +; --property=Restart=always --property=RestartSec=2 +; The '&' job belonged to the interactive shell: it died +; with the session, and 'jobs' could not see it from a +; reconnected one, so a soak that lost its terminal quietly +; stopped loading the CPU +; The log is truncated with '>' before the unit starts and +; appended with '>>' inside it, so a restart adds to the +; run instead of erasing it +; 'systemctl is-active mlucas-amm' now sits beside 'jobs', +; both in the one-shot check and in the monitor loop at the +; end +; !! the old '&' line is kept commented out directly above +; it, to fall back to if systemd-run is unavailable +; ScriptB Take the BMC USB ping duration from BMC_USB_RUNTIME +; sprintf2 cmd '... -w %d ...' ncm_if BMC_USB_RUNTIME. It +; was hard-coded to 14400 (4 h), which no longer matched +; the soak the SSD and USB knobs were set for +; ScriptC Stop mlucas-amm and copy its log to the USB stick +; utils/kill_all_process.ttl is only "kill $(jobs -p)", and +; a systemd unit is not a shell job +; ScriptB's change on its own would have left mlucas +; loading all 16 cores through ScriptC and into the next +; run, so every temperature and power reading after it +; would have been taken under load +; cp ~/hammer/tools/amd/mlucas_amm_log /host/hw-eval/jobs/ +; joins the other stress logs +; !! ScriptB and ScriptC change together. Whoever moves a +; stress job to systemd owns the matching stop in +; ScriptC +; config.ttl Add EN_timestamp and BMC_USB_RUNTIME, raise stress runtimes +; EN_timestamp = 1 feeds ScriptA's logopen; BMC_USB_RUNTIME +; = 172800 feeds ScriptB's ping unit +; SSD_RUNTIME and USB_RUNTIME 600 -> 3600, so the shipped +; default is a one hour stress rather than the ten minutes +; V1.1.7 went out with +; !! --runtime is still the job's OVERALL limit. RUNTIME +; 3600 = one hour of stress; COUNT does not multiply it. +; For a 2 day soak set RUNTIME 172800 +; .gitignore Ignore src/Script_ABC_Blanton/hammer/ +; ~99 MB of unstripped mlucas ELF binaries (avx512 / avx2 / +; avx / sse2 / generic-c) that ScriptB and the thermal +; macro chmod and run on the DUT +; !! they are deliberately NOT in git. A binary tree that +; large is permanent once committed and every clone pays +; for it forever; the DUT image already carries hammer/ +; publish/publish.sh Exclude hammer/ from the delivery bundle +; rsync copies whatever sits in src/, so a developer who +; happened to have the binaries locally would have shipped +; a 100 MB zip while a clone shipped 200 KB. The bundle is +; now the same either way ; ============================================================================= include "config.ttl" @@ -353,7 +420,11 @@ getdir mdir if EN_log = 1 then sprintf2 filename '%s\Logs\%s_%s_%s.log' mdir project_name strTestcase str_starttime - logopen filename 0 1 1 1 + ; logopen <file> <binary> <append> <PlainText> <timestamp> + ; binary=0 text mode, so PlainText / timestamp are honoured + ; append=1 keep earlier cycles, do not overwrite + ; PlainText=1 strip the ANSI colour codes SONiC emits + logopen filename 0 1 1 EN_timestamp endif ; ========== Root Login ========== diff --git a/src/Script_ABC_Blanton/2_Blanton_Script_B.ttl b/src/Script_ABC_Blanton/2_Blanton_Script_B.ttl index 36105f8..6fb2535 100644 --- a/src/Script_ABC_Blanton/2_Blanton_Script_B.ttl +++ b/src/Script_ABC_Blanton/2_Blanton_Script_B.ttl @@ -15,10 +15,15 @@ pause 1 ;sendln "command" ; ========== AMD CPU STRESS TEST ========== +; Clear the log file first +wait prompt_sonic_root +sendln "> ~/hammer/tools/amd/mlucas_amm_log" wait prompt_sonic_root sendln 'chmod +x ~/hammer/tools/amd/mlucas-avx2' wait prompt_sonic_root -sendln "~/hammer/tools/amd/mlucas-avx2 -s l -cpu 0:15 > ~/hammer/tools/amd/mlucas_amm_log 2>&1 &" +;sendln "~/hammer/tools/amd/mlucas-avx2 -s l -cpu 0:15 > ~/hammer/tools/amd/mlucas_amm_log 2>&1 &" +sendln "systemd-run --unit=mlucas-amm --property=Type=exec --property=Restart=always --property=RestartSec=2 /bin/bash -c '~/hammer/tools/amd/mlucas-avx2 -s l -cpu 0:15 >> ~/hammer/tools/amd/mlucas_amm_log 2>&1'" + ; ========== BMC DDR Stress Test ========== ;wait prompt_sonic_root @@ -65,6 +70,8 @@ sendln cmd wait prompt_sonic_root sendln "jobs" wait prompt_sonic_root +sendln "systemctl is-active mlucas-amm" +wait prompt_sonic_root sendln "bgctl list" ; ========== BMC USB Test ========== @@ -89,7 +96,7 @@ wait prompt_sonic_root sendln 'systemctl stop qfx5252-bmc-usb-net-test.service 2>/dev/null || true' wait prompt_sonic_root sendln 'systemctl reset-failed qfx5252-bmc-usb-net-test.service 2>/dev/null || true' -sprintf2 cmd 'systemd-run --unit=qfx5252-bmc-usb-net-test --property=Type=exec /usr/bin/ping -I %s -i 1 -W 2 -w 14400 192.168.200.200' ncm_if +sprintf2 cmd 'systemd-run --unit=qfx5252-bmc-usb-net-test --property=Type=exec /usr/bin/ping -I %s -i 1 -W 2 -w %d 192.168.200.200' ncm_if BMC_USB_RUNTIME wait prompt_sonic_root sendln cmd @@ -179,5 +186,7 @@ while 1 sendln "bgctl list" wait prompt_sonic_root sendln "jobs" + wait prompt_sonic_root + sendln "systemctl is-active mlucas-amm" pause 60 endwhile diff --git a/src/Script_ABC_Blanton/3_Blanton_Script_C.ttl b/src/Script_ABC_Blanton/3_Blanton_Script_C.ttl index e8f58c3..3483e6f 100644 --- a/src/Script_ABC_Blanton/3_Blanton_Script_C.ttl +++ b/src/Script_ABC_Blanton/3_Blanton_Script_C.ttl @@ -12,6 +12,11 @@ pause 1 ; ========== Kill Process ========== include "utils/kill_all_process.ttl" +wait prompt_sonic_root +sendln "systemctl is-active mlucas-amm" +wait prompt_sonic_root +sendln "systemctl stop mlucas-amm" + ; bgctrl all stop wait prompt_sonic_root sendln "bgctl stop --all" @@ -211,6 +216,8 @@ sendln cmd ; ========== Copy Stress Log to USB ========== wait prompt_sonic_root +sendln "cp ~/hammer/tools/amd/mlucas_amm_log /host/hw-eval/jobs/" +wait prompt_sonic_root sendln "cp ./Blanton_Script/log/bmc_usb_net.log /host/hw-eval/jobs/" wait prompt_sonic_root sendln "cp ./Blanton_Script/log/bmc_poll.log /host/hw-eval/jobs/" @@ -220,7 +227,6 @@ wait prompt_sonic_root sendln "cp ./Blanton_Script/log/nfc_poll.log /host/hw-eval/jobs/" - wait prompt_sonic_root getdate ts_date "%Y%m%d" gettime ts_time "%H%M" diff --git a/src/Script_ABC_Blanton/config.ttl b/src/Script_ABC_Blanton/config.ttl index 80cbd61..15fc0b5 100644 --- a/src/Script_ABC_Blanton/config.ttl +++ b/src/Script_ABC_Blanton/config.ttl @@ -6,6 +6,7 @@ strTestcase="ENV" ;ENV,EMC,Margin...etc EN_Margin = 0 ; 1: Enable Margin Test, 0: Disable Margin Test EN_log = 1 ; 1: Enable log, 0: Disable log +EN_timestamp = 1 ; 1: Enable timestamp in log, 0: Disable timestamp in log ;================================================================ ; FAN_SPEED @@ -20,10 +21,11 @@ SWB_UNIT1 = 1 ; 1 = this DUT has switch unit 1, wait for it ; 0 = skip ;================================================================ ; Stress_Test ;================================================================ -SSD_RUNTIME = 600 ; Unit seconds, 600 = 10 minutes -SSD_COUNT = 720 ; Total time = SSD_RUNTIME * SSD_COUNT -USB_RUNTIME = 600 ; Unit seconds, 600 = 10 minutes -USB_COUNT = 720 ; Total time = USB_RUNTIME * USB_COUNT +SSD_RUNTIME = 3600 ; Unit seconds, 3600 = 1 hour +SSD_COUNT = 240 ; Total time = SSD_RUNTIME * SSD_COUNT +USB_RUNTIME = 3600 ; Unit seconds, 3600 = 1 hour +USB_COUNT = 240 ; Total time = USB_RUNTIME * USB_COUNT +BMC_USB_RUNTIME = 172800 ; Unit seconds, 172800 = 2 days ;================================================================ ; Prompt Definitions