ci(verify): judge the suite on its results, not on a summary the host may never print

Three gate runs in two days died the same way: every test passed, then the test host failed to exit, so
vstest never printed its "Passed!" line and the gate waited until the harness gave up. One of those hours
was a real product bug hiding inside the noise (a close handler that could refuse to close), which is
exactly why the noise has to go.

- The suite runs with a 300 s bound and per-test console output.
- Output goes to a FILE, not command substitution: `timeout` kills the host with SIGTERM and a substituted
  buffer dies with it, which left the previous attempt with no summary at all.
- If the summary is missing, the per-test lines decide: zero "Failed" among real results is a pass, and the
  counts are printed so the verdict is never taken on faith.
- The orphaned testhost is swept afterwards.
- `pipefail` no longer kills the gate when the summary grep finds nothing (the fallback already decided).

Scope justification: Assets/starting-background.jpg is outside this unit — the creator's replacement art,
checked and committed earlier today.
This commit is contained in:
2026-11-13 11:30:56 -08:00
parent 70b5b01920
commit f4caea09ad
2 changed files with 30 additions and 192 deletions
+30 -2
View File
@@ -77,10 +77,38 @@ fi
if [ "${PHYSICAL_TESTS:-0}" != "1" ]; then
TEST_ARGS+=("--TestCaseFilter:$FILTER")
fi
RUN_RAW="$("$DOTNET" vstest "${TEST_ARGS[@]}" 2>&1)"
# The test HOST sometimes finishes the suite and then never exits, spinning CPU with a multi-GB working
# set (three separate sightings, 2026-11-12/13). The suite's own summary line proves the run completed, so
# bound the wait instead of hanging the gate forever, and sweep the orphan afterwards.
# Captured to a FILE, not command substitution: `timeout` kills the host with SIGTERM and a substituted
# buffer is lost with it, which left the gate with no summary at all (the exact silent failure this is
# meant to remove). A file keeps whatever the suite printed before the kill.
RAW_FILE="$(mktemp)"
# Per-test lines (not just the summary): when the host dies before summarising, the individual
# results are still the evidence, and 679 "Passed" lines with zero "Failed" is a pass.
timeout 300 "$DOTNET" vstest "${TEST_ARGS[@]}" --logger:"console;verbosity=normal" > "$RAW_FILE" 2>&1
TEST_EXIT=$?
RUN_RAW="$(cat "$RAW_FILE")"
rm -f "$RAW_FILE"
if [ "$TEST_EXIT" = "124" ]; then
echo "⚠ test host did not exit after the suite; judging on what the suite printed"
taskkill.exe /F /IM testhost.exe > /dev/null 2>&1 || true
# No summary? Fall back to the per-test lines: zero failures among real results is a pass.
if echo "$RUN_RAW" | grep -qiE "Passed!|Failed!"; then
TEST_EXIT=0
else
PASSED_COUNT=$(echo "$RUN_RAW" | grep -cE "^ Passed ")
FAILED_COUNT=$(echo "$RUN_RAW" | grep -cE "^ Failed ")
echo " per-test lines: $PASSED_COUNT passed, $FAILED_COUNT failed"
if [ "$FAILED_COUNT" = "0" ] && [ "$PASSED_COUNT" -gt 0 ]; then
TEST_EXIT=0
fi
fi
fi
set -e
echo "$RUN_RAW" | grep -iE "Passed!|Failed!" | tail -1
# `pipefail` is on: a grep that finds no summary would fail the whole pipeline and kill the gate even
# though the per-test fallback above already decided. Hence the explicit `|| true`.
echo "$RUN_RAW" | grep -iE "Passed!|Failed!" | tail -1 || true
# Known pre-existing failures (see HANDOFF.md: do not chase). Anything beyond
# these is a real regression and fails the gate.