fix(web): re-point the widget diagnostic at the REAL document — the transparency premise gets verified
The one-shot dump + alpha log was bound to the FIRST capture ever = the initial
about:blank placeholder (alpha=0, rgb=0, 5/5 sessions) — a blind instrument. The
pre-parse transparency injection (1a39b09) is intact but was NEVER verified
(MyMistakes '? VERIFY'), and the 2026-09-10 take still shows a black, opaque
block. Stop guessing: NavigationCompleted for a non-about:blank document now
arms WidgetDumpRemaining=5; the next 5 post-paint captures dump
%TEMP%\ytLive-web-<id>-w1..5.png + shared AlphaStats(min/max/mean/zero%) +
FindContentBounds to startup.log.
The stats line alone names the branch: zero%≈opaque ⇒ the injection did not hold
for this widget's CSS (fix: !important stylesheet / chroma-key); large zero% +
tight bounds ⇒ capture IS transparent and the black lives in the compositor
blend. No new test: WebView2 runtime is not instantiable in the suite and the
re-point is log-only; 8/8 WebView2 tests still pass.
This commit is contained in:
+48
-74
@@ -1,89 +1,63 @@
|
||||
# HANDOFF — 2026-09-10 (afternoon)
|
||||
# HANDOFF — 2026-09-10 (late afternoon)
|
||||
|
||||
## Branch / Commit State
|
||||
|
||||
`main` HEAD currently = `5e78065` (slice 11, committed). **Slice 12 (audio diagnostics) is
|
||||
uncommitted** in the working tree. **Ahead of origin by 17, NOT pushing** — user ruling (2026-09-10):
|
||||
do not push until the web-overlay **transparency AND audio-silence** issues are addressed.
|
||||
Milestone tag `milestone-recording-timing` (annotated) on `c45cbc9` — rollback point:
|
||||
`git reset --hard milestone-recording-timing`.
|
||||
`main` HEAD currently = `f3d578c` (slice 12 audio telemetry). **Slice 13 (web transparency
|
||||
diagnostic re-point) is uncommitted** in the working tree. Ahead of origin by 18, NOT pushing
|
||||
(user ruling: no push until web-overlay transparency AND audio-silence are addressed).
|
||||
|
||||
## Timing saga — CLOSED (verified)
|
||||
## ACTIVE THREAD (the user's directive: ONE problem at a time)
|
||||
|
||||
Slices 9 (`bd396e4`) + 10 (`c45cbc9`) committed. Takes `0949`/`0957` verified: counter
|
||||
+1/frame, zero gaps/dups. User confirmed → **timing CLOSED.**
|
||||
**Web-uri transparency: still a black, opaque block** (take `ty-20260910-1202-0000-2.mp4` —
|
||||
user: "the web-uri resource transparency is still a black, opaque block. The desired animation
|
||||
runs just fine."). The animation-speed work is done; transparency is now THE web problem.
|
||||
|
||||
## Slice 11 (web capture cadence) — committed `5e78065`
|
||||
Institutional memory (MyMistakes, spin-guard entry): WebView2 `CapturePreviewAsync` honors the
|
||||
page's CSS — the pre-parse injection (`1a39b09`, `AddScriptToExecuteOnDocumentCreatedAsync`)
|
||||
is the recorded fix and is INTACT in `WebView2Manager.cs` (line ~157). The "? VERIFY" was never
|
||||
closed because the diagnostic dump + alpha log bound to the FIRST capture ever = the about:blank
|
||||
placeholder (blind; 5/5 sessions alpha=0 rgb=0).
|
||||
|
||||
10Hz capture → ~30Hz target with de-throttle flags + `CaptureScheduler` (latest-wins in-flight
|
||||
drop). First verification take (`ty-20260910-1038-0000-2.mp4`, 1423 frames 23.7s) results:
|
||||
**Slice 13 change (uncommitted):** re-point the instrument at reality —
|
||||
- `WebSourceSession.WidgetDumpRemaining`; `NavigationCompleted` arms `= 5` when the newly-loaded
|
||||
document is NOT about:blank.
|
||||
- Next 5 captures dump `%TEMP%\ytLive-web-<id>-w1..5.png` + `AlphaStats(...)` + FindContentBounds
|
||||
to startup.log.
|
||||
- `AlphaStats` extracted from the old inline first-capture block.
|
||||
|
||||
- **Capture cost: 35-117ms per frame** (not the hoped 10-30ms). Effective cadence is ~10-14Hz,
|
||||
NOT 30Hz — the scheduler's in-flight drop silently collapsed the target back to roughly the old
|
||||
rate. The in-flight drop works (no stacking), but the decode path (BitmapImage →
|
||||
FormatConvertedBitmap → FindContentBounds full-scan) is too expensive to reach 30Hz.
|
||||
Web animation speed will not have improved meaningfully. Next: the capture cost decides — either
|
||||
optimize the decode path (crop-only, cached bounds) or drop the target to ~15Hz.
|
||||
- FramePump: healthy (300/300 frames, 0 dropped, 6 stalls worst 166ms).
|
||||
- Camera: initialized (YUY2 640×480) but "MJPG negotiation refused (being used by another process)"
|
||||
logged at startup — the webcam source was contested.
|
||||
- Audio: **full-length silent AAC** (−91dB, 1124 frames). User reported: no transparency in web-uri,
|
||||
no webcam video, no audio. The three symptoms are separate threads.
|
||||
Build 0 warnings; 8/8 WebView2 tests pass (runtime not instantiable — re-point is log-only, no
|
||||
new test). Suite otherwise 290/291 (pre-existing compositor pixel).
|
||||
|
||||
## Slice 12 (audio diagnostics) — uncommitted, in-tree
|
||||
## THE DECISION LADDER (no more cargo-culting)
|
||||
|
||||
The silence is capture-side: the pipe ran for 23.95s, ffmpeg connected and read ~9.1MB of zeros.
|
||||
Sources started (`Audio: using system default mic...` logged at 10:38:01.296), no failure
|
||||
callbacks fired. The mixer's existing integration test (`Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe`)
|
||||
proves the loop→pipe path carries real audio when fed. Two remaining capture-side suspects:
|
||||
(1) default render device mismatch — audio played on a non-default endpoint (common);
|
||||
(2) both endpoints held exclusive by another process.
|
||||
1. User launches the app with the widget sources active, waits ~10s, closes. (NO recording needed.)
|
||||
2. Read the `widget capture [1..5/5]` lines in startup.log:
|
||||
- **zero% ≈ 0 (opaque):** pre-parse injection did NOT hold for this widget's own CSS →
|
||||
FIX = stronger transparent-background enforcement (document-level `!important` stylesheet
|
||||
injected pre-parse + re-asserted, per the OBS user.css precedent) OR chroma-key the known
|
||||
backdrop color in `CaptureFrame`. Pick after seeing the `-w1..5.png` PNGs.
|
||||
- **zero% large + tight contentBounds:** the capture HAS transparent margins → the recording's
|
||||
black block lives downstream → audit the compositor web-layer blend/underlay (PMA-vs-straight
|
||||
alpha, black pre-fill). Do NOT touch the capture path.
|
||||
3. Implement the branch's fix with ONE integration test (chroma-key math or compositor blend has
|
||||
testable seams), docs same-commit, then ONE verification recording.
|
||||
|
||||
**Changes in tree:**
|
||||
- `Services/Audio/AudioMixer.cs`: `FillAndMix` now returns `(MicRms, MicDrained, LoopDrained)`;
|
||||
LiveLoopAsync accumulates per-5s telemetry → startup.log:
|
||||
`Audio live: pipe connected=, dropped writes=, micLevel=, loopLevel=, drained, peakMix`;
|
||||
StartLive/StopLive log start/stop lines.
|
||||
- `Services/Audio/NamedPipeAudioWriter.cs`: new `DroppedWrites` counter — nonzero means audio
|
||||
was dropped before ffmpeg connected (names "pipe never connected" stage).
|
||||
- Docs: `ai.md` slice 12 entry.
|
||||
## Other threads (paused)
|
||||
|
||||
Suite: 290/291 pass — same sole pre-existing compositor pixel failure.
|
||||
|
||||
## NEXT STEP
|
||||
|
||||
1. Commit slice 12 (scope-check passed on the three source/doc files).
|
||||
2. User runs a recording with DESKTOP AUDIO ACTIVE (music/game playing → verify the "Desktop Audio"
|
||||
footer bar moves during the take). Send startup.log lines containing `Audio live:`.
|
||||
The log names the stage:
|
||||
- `loopLevel > 0` + `peakMix > 0` + file silent → pipe-side (impossible per integration test; should not appear)
|
||||
- `loopLevel ≈ 0` + `drained 0/0` → capture delivered nothing (default device mismatch or exclusive hold)
|
||||
- `dropped writes > 0` → encoder pipe never connected (timing/ordering bug)
|
||||
3. If capture-side: add explicit loopback-device selection (enumerate active render endpoints, log
|
||||
the chosen one, allow user to pick) — the fix that makes the mismatch impossible.
|
||||
4. Then return to web transparency (post-paint dump) and webcam.
|
||||
|
||||
## Still Open
|
||||
|
||||
- **Audio silence** — push gate reason #2. Diagnostic committed pending take (see above).
|
||||
- **Web transparency UNVERIFIED** — instrument blind (about:blank). Needs post-paint re-point.
|
||||
Push gate reason #1.
|
||||
- **Web capture speed** — 30Hz target not achievable with current decode path; capture cost
|
||||
35-117ms/frame → effective ~10-14Hz. Open question whether to optimize path or accept ~15Hz.
|
||||
- **Webcam missing** in take-1038 — "MJPG negotiation refused (being used by another process)"
|
||||
at startup; camera initialized but possibly contested. Separate thread, queued.
|
||||
- Pre-existing compositor pixel test failure (never in scope).
|
||||
- **Audio silence** — `f3d578c` added per-5s `Audio live:` telemetry; next take with desktop
|
||||
audio ACTIVE + those lines names the stage. Push gate #2.
|
||||
- **Web capture speed** — 30Hz NOT achieved (35-117ms/capture → effective ~10-14Hz). The
|
||||
animation "runs fine" so the user is satisfied; revisit only if they want more.
|
||||
- **Webcam missing** in the 10:38/12:02 takes — "MJPG negotiation refused (being used by
|
||||
another process)" at startup. Queued behind transparency+audio.
|
||||
|
||||
## Landmines
|
||||
|
||||
- testhost shares startup.log with app — filter by time when triaging.
|
||||
- Locked DLLs: `taskkill //F //IM testhost.exe //IM ytLive.exe` before rebuild.
|
||||
- Build/tests via `/mnt/c/Program Files/dotnet/dotnet.exe build …` / `… vstest
|
||||
"C:\Users\gramp\Documents\Code\projects\ytLive\ytLive.Tests\bin\Debug\net8.0-windows10.0.19041.0\ytLive.Tests.dll"`.
|
||||
- Probing recordings: `/mnt/c/Program Files/Krita (x64)/bin/ffmpeg.exe` / `ffprobe.exe`.
|
||||
- `tools/ticker`: constant ~+982ms offset on the FIRST line is cosmetic.
|
||||
- WebView2 capture cost: 35-117ms per full-HD PNG encode+decode — the hard ceiling on
|
||||
web animation capture rate. Slice 13 candidate if the user wants to pursue 30Hz.
|
||||
- WASAPI loopback: captures the DEFAULT render endpoint only — if the user plays audio
|
||||
through a non-default device, the recording is silently silent. This is likely the root cause
|
||||
of the audio-silence issue. The fix requires device enumeration + selection.
|
||||
- testhost shares startup.log with app — filter by time.
|
||||
- `taskkill //F //IM testhost.exe //IM ytLive.exe` before rebuild.
|
||||
- Build/tests: `/mnt/c/Program Files/dotnet/dotnet.exe build …` / vstest.
|
||||
- Probing: `/mnt/c/Program Files/Krita (x64)/bin/ffmpeg.exe` / `ffprobe.exe`.
|
||||
- Do NOT delete the crop/compositor paths while debugging source alpha (MyMistakes rule 3 —
|
||||
take-25 vandalism).
|
||||
- The user is frustrated with take-loops; the widget dump needs NO recording — an app launch
|
||||
suffices. Ask for launch + 10s + close, not a take.
|
||||
@@ -63,6 +63,18 @@ do the same via a pre-parse user.css. Kept the nav handlers as a post-load re-as
|
||||
3. Do not delete code paths that fix one axis (crop=bbox) while debugging another
|
||||
(source alpha). Revert scope creep; keep layer contributions separable.
|
||||
|
||||
**2026-09-10 follow-up (still not verified, now being resolved):** the "? VERIFY" above was
|
||||
never closed because the diagnostic itself was blind — the dump + alpha log fired on the FIRST
|
||||
capture ever, which is always the initial about:blank placeholder document (alpha=0, rgb=0,
|
||||
5/5 sessions). The REAL widget frame was never seen. The 2026-09-10 take still showed a black,
|
||||
opaque block with the animation running fine, so either (a) the pre-parse injection does not
|
||||
hold for this widget's own CSS (capture opaque) — fix = stronger injection / chroma-key — or
|
||||
(b) the capture has transparent margins and the black lives in the compositor blend. `1a39b09`'s
|
||||
injection is intact; `WidgetDumpRemaining = 5` now dumps the real widget document post-paint
|
||||
(`%TEMP%\ytLive-web-<id>-w1..5.png`) + shared `AlphaStats` line → the next app launch names
|
||||
which branch. Do NOT re-derive the whole transparency story again from this text; the 5-line
|
||||
diagnostic is the shortest path.
|
||||
|
||||
### Shrink / re-encode an image for the README (screenshots → small hero image)
|
||||
|
||||
Worked out 2026-08-29 (the recipe was NEVER recorded the first time it was done, so
|
||||
|
||||
+55
-17
@@ -46,6 +46,7 @@ public sealed class WebView2Manager : IDisposable
|
||||
public long Epoch;
|
||||
public WriteableBitmap? PreviewBitmap;
|
||||
public bool DebugPngWritten;
|
||||
public int WidgetDumpRemaining;
|
||||
public int TelemetryCaptureCount;
|
||||
|
||||
public byte[] RentOutBuffer(int size)
|
||||
@@ -179,9 +180,19 @@ public sealed class WebView2Manager : IDisposable
|
||||
_ = cws.ExecuteScriptAsync(TransparentBackgroundScript);
|
||||
};
|
||||
|
||||
cws.NavigationCompleted += (_, _) =>
|
||||
cws.NavigationCompleted += (_, e) =>
|
||||
{
|
||||
_ = cws.ExecuteScriptAsync(TransparentBackgroundScript);
|
||||
if (e.IsSuccess && session.Control.CoreWebView2?.Source is { } src && src != "about:blank")
|
||||
{
|
||||
// Arm a one-shot diagnostic dump of the REAL widget document
|
||||
// (a few post-paint captures so styles/backgrounds settle) —
|
||||
// the old first-capture dump only ever saw the about:blank
|
||||
// document, so whether this widget's capture actually HAS the
|
||||
// transparent margins the injection promises has never been
|
||||
// verified (MyMistakes: "? VERIFY").
|
||||
session.WidgetDumpRemaining = 5;
|
||||
}
|
||||
_ = CaptureFrame(source.Id);
|
||||
};
|
||||
|
||||
@@ -257,6 +268,29 @@ public sealed class WebView2Manager : IDisposable
|
||||
return (minX, minY, maxX - minX + 1, maxY - minY + 1);
|
||||
}
|
||||
|
||||
// alpha[min/max/mean/zero%] — the one number that decides whether a widget's
|
||||
// capture really has transparent margins (0% = opaque, ~100% if the page
|
||||
// painted transparent-only). Sampled every 16th pixel like the original log.
|
||||
private static string AlphaStats(byte[] pixels, int pixW, int pixH, int stride)
|
||||
{
|
||||
int minA = 255, maxA = 0;
|
||||
long sumA = 0, nA = 0, zeroA = 0;
|
||||
for (var y = 0; y < pixH; y += 16)
|
||||
{
|
||||
var row = y * stride;
|
||||
for (var x = 0; x < pixW; x += 16)
|
||||
{
|
||||
var a = pixels[row + x * 4 + 3];
|
||||
if (a < minA) minA = a;
|
||||
if (a > maxA) maxA = a;
|
||||
sumA += a;
|
||||
if (a == 0) zeroA++;
|
||||
nA++;
|
||||
}
|
||||
}
|
||||
return $"alpha[min={minA},max={maxA},mean={sumA / (double)nA:F1},zero={100.0 * zeroA / nA:F1}%]";
|
||||
}
|
||||
|
||||
private async Task CaptureFrame(string sourceId)
|
||||
{
|
||||
if (!_sessions.TryGetValue(sourceId, out var session) || session.Disposed) return;
|
||||
@@ -313,27 +347,31 @@ public sealed class WebView2Manager : IDisposable
|
||||
|
||||
if (session.Epoch == 0)
|
||||
{
|
||||
int minA = 255, maxA = 0;
|
||||
long sumA = 0, nA = 0, zeroA = 0;
|
||||
for (var y = 0; y < pixH; y += 16)
|
||||
{
|
||||
var row = y * stride;
|
||||
for (var x = 0; x < pixW; x += 16)
|
||||
{
|
||||
var a = pixels[row + x * 4 + 3];
|
||||
if (a < minA) minA = a;
|
||||
if (a > maxA) maxA = a;
|
||||
sumA += a;
|
||||
if (a == 0) zeroA++;
|
||||
nA++;
|
||||
}
|
||||
}
|
||||
AppLog.Write(
|
||||
$"WebView2Manager: first capture for '{sourceId}' {pixW}x{pixH} " +
|
||||
$"alpha[min={minA},max={maxA},mean={sumA / (double)nA:F1},zero={100.0 * zeroA / nA:F1}%] " +
|
||||
$"{AlphaStats(pixels, pixW, pixH, stride)} " +
|
||||
$"contentBounds=({cropX},{cropY},{cropW},{cropH})");
|
||||
}
|
||||
|
||||
if (session.WidgetDumpRemaining > 0)
|
||||
{
|
||||
var dumpNo = 5 - session.WidgetDumpRemaining + 1;
|
||||
session.WidgetDumpRemaining--;
|
||||
try
|
||||
{
|
||||
var dbg = Path.Combine(Path.GetTempPath(), $"ytLive-web-{sourceId}-w{dumpNo}.png");
|
||||
File.WriteAllBytes(dbg, ms.ToArray());
|
||||
AppLog.Write(
|
||||
$"WebView2Manager: widget capture [{dumpNo}/5] for '{sourceId}' {pixW}x{pixH} " +
|
||||
$"{AlphaStats(pixels, pixW, pixH, stride)} " +
|
||||
$"contentBounds=({cropX},{cropY},{cropW},{cropH}) -> {dbg}");
|
||||
}
|
||||
catch (Exception dex)
|
||||
{
|
||||
AppLog.Write($"WebView2Manager: widget dump failed: {dex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
var outPixels = session.RentOutBuffer(cropW * cropH * 4);
|
||||
if (cropW == pixW && cropH == pixH && cropX == 0 && cropY == 0)
|
||||
Buffer.BlockCopy(pixels, 0, outPixels, 0, outPixels.Length);
|
||||
|
||||
@@ -911,6 +911,16 @@ Full suite 290/291 passing, the sole failure the pre-existing compositor pixel t
|
||||
`NamedPipeAudioWriter.DroppedWrites` — names the exact stage on the next take without new
|
||||
code. `AudioMixer.FillAndMix` now returns `(MicRms, MicDrained, LoopDrained)` for the telemetry
|
||||
accumulation. Full suite 290/291, same pre-existing sole failure.
|
||||
- **Slice 13 (web transparency diagnostic re-point, 2026-09-10):** the "?" in MyMistakes'
|
||||
"RESOLVED (? VERIFY)" is being closed. The one-shot widget dump + alpha stats bound to the
|
||||
FIRST capture ever = the initial about:blank document — a blind instrument. NavigationCompleted
|
||||
for the REAL widget URL now arms `WidgetDumpRemaining = 5`; those 5 post-paint captures dump
|
||||
`%TEMP%\ytLive-web-<id>-w1..5.png` + the shared `AlphaStats(...)` line (alpha[min/max/mean/zero%])
|
||||
+ FindContentBounds result to startup.log. That line alone decides the fix branch: zero% ⇒ the
|
||||
pre-parse injection did not hold for this widget (opaque capture → chroma-key / stronger DOM
|
||||
injection); large zero% + tight contentBounds ⇒ the capture IS transparent and the recording's
|
||||
black block lives downstream (compositor blend/underlay). `AlphaStats` extracted as the shared
|
||||
sampler. No new tests (WebView2 runtime not instantiable in tests; the re-point is log-only).
|
||||
- **Stop ordering matters:** `StopAsync` stops the encoder — since slice 10 it FLUSHES the pending
|
||||
queue (`Channel.TryComplete` → drain writes the leftovers, closes stdin → EOF → ffmpeg finalizes+exits;
|
||||
an accepted frame is never lost) — **before** awaiting the loop. The old reverse-order deadlock was
|
||||
|
||||
Reference in New Issue
Block a user