From ea347f211c92692666888d8eb24440cc31e19bb7 Mon Sep 17 00:00:00 2001 From: gramps Date: Thu, 10 Sep 2026 12:08:53 -0700 Subject: [PATCH] =?UTF-8?q?fix(web):=20re-point=20the=20widget=20diagnosti?= =?UTF-8?q?c=20at=20the=20REAL=20document=20=E2=80=94=20the=20transparency?= =?UTF-8?q?=20premise=20gets=20verified?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The one-shot dump + alpha log was bound to the FIRST capture ever = the initial about:blank placeholder (alpha=0, rgb=0, 5/5 sessions) — a blind instrument. The pre-parse transparency injection (1a39b09) is intact but was NEVER verified (MyMistakes '? VERIFY'), and the 2026-09-10 take still shows a black, opaque block. Stop guessing: NavigationCompleted for a non-about:blank document now arms WidgetDumpRemaining=5; the next 5 post-paint captures dump %TEMP%\ytLive-web--w1..5.png + shared AlphaStats(min/max/mean/zero%) + FindContentBounds to startup.log. The stats line alone names the branch: zero%≈opaque ⇒ the injection did not hold for this widget's CSS (fix: !important stylesheet / chroma-key); large zero% + tight bounds ⇒ capture IS transparent and the black lives in the compositor blend. No new test: WebView2 runtime is not instantiable in the suite and the re-point is log-only; 8/8 WebView2 tests still pass. --- HANDOFF.md | 122 ++++++++++++++---------------------- MyMistakes.md | 12 ++++ Services/WebView2Manager.cs | 72 ++++++++++++++++----- ai.md | 10 +++ 4 files changed, 125 insertions(+), 91 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index ffb12c9..226cf90 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,89 +1,63 @@ -# HANDOFF — 2026-09-10 (afternoon) +# HANDOFF — 2026-09-10 (late afternoon) ## Branch / Commit State -`main` HEAD currently = `5e78065` (slice 11, committed). **Slice 12 (audio diagnostics) is -uncommitted** in the working tree. **Ahead of origin by 17, NOT pushing** — user ruling (2026-09-10): -do not push until the web-overlay **transparency AND audio-silence** issues are addressed. -Milestone tag `milestone-recording-timing` (annotated) on `c45cbc9` — rollback point: -`git reset --hard milestone-recording-timing`. +`main` HEAD currently = `f3d578c` (slice 12 audio telemetry). **Slice 13 (web transparency +diagnostic re-point) is uncommitted** in the working tree. Ahead of origin by 18, NOT pushing +(user ruling: no push until web-overlay transparency AND audio-silence are addressed). -## Timing saga — CLOSED (verified) +## ACTIVE THREAD (the user's directive: ONE problem at a time) -Slices 9 (`bd396e4`) + 10 (`c45cbc9`) committed. Takes `0949`/`0957` verified: counter -+1/frame, zero gaps/dups. User confirmed → **timing CLOSED.** +**Web-uri transparency: still a black, opaque block** (take `ty-20260910-1202-0000-2.mp4` — +user: "the web-uri resource transparency is still a black, opaque block. The desired animation +runs just fine."). The animation-speed work is done; transparency is now THE web problem. -## Slice 11 (web capture cadence) — committed `5e78065` +Institutional memory (MyMistakes, spin-guard entry): WebView2 `CapturePreviewAsync` honors the +page's CSS — the pre-parse injection (`1a39b09`, `AddScriptToExecuteOnDocumentCreatedAsync`) +is the recorded fix and is INTACT in `WebView2Manager.cs` (line ~157). The "? VERIFY" was never +closed because the diagnostic dump + alpha log bound to the FIRST capture ever = the about:blank +placeholder (blind; 5/5 sessions alpha=0 rgb=0). -10Hz capture → ~30Hz target with de-throttle flags + `CaptureScheduler` (latest-wins in-flight -drop). First verification take (`ty-20260910-1038-0000-2.mp4`, 1423 frames 23.7s) results: +**Slice 13 change (uncommitted):** re-point the instrument at reality — +- `WebSourceSession.WidgetDumpRemaining`; `NavigationCompleted` arms `= 5` when the newly-loaded + document is NOT about:blank. +- Next 5 captures dump `%TEMP%\ytLive-web--w1..5.png` + `AlphaStats(...)` + FindContentBounds + to startup.log. +- `AlphaStats` extracted from the old inline first-capture block. -- **Capture cost: 35-117ms per frame** (not the hoped 10-30ms). Effective cadence is ~10-14Hz, - NOT 30Hz — the scheduler's in-flight drop silently collapsed the target back to roughly the old - rate. The in-flight drop works (no stacking), but the decode path (BitmapImage → - FormatConvertedBitmap → FindContentBounds full-scan) is too expensive to reach 30Hz. - Web animation speed will not have improved meaningfully. Next: the capture cost decides — either - optimize the decode path (crop-only, cached bounds) or drop the target to ~15Hz. -- FramePump: healthy (300/300 frames, 0 dropped, 6 stalls worst 166ms). -- Camera: initialized (YUY2 640×480) but "MJPG negotiation refused (being used by another process)" - logged at startup — the webcam source was contested. -- Audio: **full-length silent AAC** (−91dB, 1124 frames). User reported: no transparency in web-uri, - no webcam video, no audio. The three symptoms are separate threads. +Build 0 warnings; 8/8 WebView2 tests pass (runtime not instantiable — re-point is log-only, no +new test). Suite otherwise 290/291 (pre-existing compositor pixel). -## Slice 12 (audio diagnostics) — uncommitted, in-tree +## THE DECISION LADDER (no more cargo-culting) -The silence is capture-side: the pipe ran for 23.95s, ffmpeg connected and read ~9.1MB of zeros. -Sources started (`Audio: using system default mic...` logged at 10:38:01.296), no failure -callbacks fired. The mixer's existing integration test (`Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe`) -proves the loop→pipe path carries real audio when fed. Two remaining capture-side suspects: -(1) default render device mismatch — audio played on a non-default endpoint (common); -(2) both endpoints held exclusive by another process. +1. User launches the app with the widget sources active, waits ~10s, closes. (NO recording needed.) +2. Read the `widget capture [1..5/5]` lines in startup.log: + - **zero% ≈ 0 (opaque):** pre-parse injection did NOT hold for this widget's own CSS → + FIX = stronger transparent-background enforcement (document-level `!important` stylesheet + injected pre-parse + re-asserted, per the OBS user.css precedent) OR chroma-key the known + backdrop color in `CaptureFrame`. Pick after seeing the `-w1..5.png` PNGs. + - **zero% large + tight contentBounds:** the capture HAS transparent margins → the recording's + black block lives downstream → audit the compositor web-layer blend/underlay (PMA-vs-straight + alpha, black pre-fill). Do NOT touch the capture path. +3. Implement the branch's fix with ONE integration test (chroma-key math or compositor blend has + testable seams), docs same-commit, then ONE verification recording. -**Changes in tree:** -- `Services/Audio/AudioMixer.cs`: `FillAndMix` now returns `(MicRms, MicDrained, LoopDrained)`; - LiveLoopAsync accumulates per-5s telemetry → startup.log: - `Audio live: pipe connected=, dropped writes=, micLevel=, loopLevel=, drained, peakMix`; - StartLive/StopLive log start/stop lines. -- `Services/Audio/NamedPipeAudioWriter.cs`: new `DroppedWrites` counter — nonzero means audio - was dropped before ffmpeg connected (names "pipe never connected" stage). -- Docs: `ai.md` slice 12 entry. +## Other threads (paused) -Suite: 290/291 pass — same sole pre-existing compositor pixel failure. - -## NEXT STEP - -1. Commit slice 12 (scope-check passed on the three source/doc files). -2. User runs a recording with DESKTOP AUDIO ACTIVE (music/game playing → verify the "Desktop Audio" - footer bar moves during the take). Send startup.log lines containing `Audio live:`. - The log names the stage: - - `loopLevel > 0` + `peakMix > 0` + file silent → pipe-side (impossible per integration test; should not appear) - - `loopLevel ≈ 0` + `drained 0/0` → capture delivered nothing (default device mismatch or exclusive hold) - - `dropped writes > 0` → encoder pipe never connected (timing/ordering bug) -3. If capture-side: add explicit loopback-device selection (enumerate active render endpoints, log - the chosen one, allow user to pick) — the fix that makes the mismatch impossible. -4. Then return to web transparency (post-paint dump) and webcam. - -## Still Open - -- **Audio silence** — push gate reason #2. Diagnostic committed pending take (see above). -- **Web transparency UNVERIFIED** — instrument blind (about:blank). Needs post-paint re-point. - Push gate reason #1. -- **Web capture speed** — 30Hz target not achievable with current decode path; capture cost - 35-117ms/frame → effective ~10-14Hz. Open question whether to optimize path or accept ~15Hz. -- **Webcam missing** in take-1038 — "MJPG negotiation refused (being used by another process)" - at startup; camera initialized but possibly contested. Separate thread, queued. -- Pre-existing compositor pixel test failure (never in scope). +- **Audio silence** — `f3d578c` added per-5s `Audio live:` telemetry; next take with desktop + audio ACTIVE + those lines names the stage. Push gate #2. +- **Web capture speed** — 30Hz NOT achieved (35-117ms/capture → effective ~10-14Hz). The + animation "runs fine" so the user is satisfied; revisit only if they want more. +- **Webcam missing** in the 10:38/12:02 takes — "MJPG negotiation refused (being used by + another process)" at startup. Queued behind transparency+audio. ## Landmines -- testhost shares startup.log with app — filter by time when triaging. -- Locked DLLs: `taskkill //F //IM testhost.exe //IM ytLive.exe` before rebuild. -- Build/tests via `/mnt/c/Program Files/dotnet/dotnet.exe build …` / `… vstest - "C:\Users\gramp\Documents\Code\projects\ytLive\ytLive.Tests\bin\Debug\net8.0-windows10.0.19041.0\ytLive.Tests.dll"`. -- Probing recordings: `/mnt/c/Program Files/Krita (x64)/bin/ffmpeg.exe` / `ffprobe.exe`. -- `tools/ticker`: constant ~+982ms offset on the FIRST line is cosmetic. -- WebView2 capture cost: 35-117ms per full-HD PNG encode+decode — the hard ceiling on - web animation capture rate. Slice 13 candidate if the user wants to pursue 30Hz. -- WASAPI loopback: captures the DEFAULT render endpoint only — if the user plays audio - through a non-default device, the recording is silently silent. This is likely the root cause - of the audio-silence issue. The fix requires device enumeration + selection. +- testhost shares startup.log with app — filter by time. +- `taskkill //F //IM testhost.exe //IM ytLive.exe` before rebuild. +- Build/tests: `/mnt/c/Program Files/dotnet/dotnet.exe build …` / vstest. +- Probing: `/mnt/c/Program Files/Krita (x64)/bin/ffmpeg.exe` / `ffprobe.exe`. +- Do NOT delete the crop/compositor paths while debugging source alpha (MyMistakes rule 3 — + take-25 vandalism). +- The user is frustrated with take-loops; the widget dump needs NO recording — an app launch + suffices. Ask for launch + 10s + close, not a take. \ No newline at end of file diff --git a/MyMistakes.md b/MyMistakes.md index f0a131f..82539d8 100644 --- a/MyMistakes.md +++ b/MyMistakes.md @@ -63,6 +63,18 @@ do the same via a pre-parse user.css. Kept the nav handlers as a post-load re-as 3. Do not delete code paths that fix one axis (crop=bbox) while debugging another (source alpha). Revert scope creep; keep layer contributions separable. +**2026-09-10 follow-up (still not verified, now being resolved):** the "? VERIFY" above was +never closed because the diagnostic itself was blind — the dump + alpha log fired on the FIRST +capture ever, which is always the initial about:blank placeholder document (alpha=0, rgb=0, +5/5 sessions). The REAL widget frame was never seen. The 2026-09-10 take still showed a black, +opaque block with the animation running fine, so either (a) the pre-parse injection does not +hold for this widget's own CSS (capture opaque) — fix = stronger injection / chroma-key — or +(b) the capture has transparent margins and the black lives in the compositor blend. `1a39b09`'s +injection is intact; `WidgetDumpRemaining = 5` now dumps the real widget document post-paint +(`%TEMP%\ytLive-web--w1..5.png`) + shared `AlphaStats` line → the next app launch names +which branch. Do NOT re-derive the whole transparency story again from this text; the 5-line +diagnostic is the shortest path. + ### Shrink / re-encode an image for the README (screenshots → small hero image) Worked out 2026-08-29 (the recipe was NEVER recorded the first time it was done, so diff --git a/Services/WebView2Manager.cs b/Services/WebView2Manager.cs index c6d27bb..b91ed58 100644 --- a/Services/WebView2Manager.cs +++ b/Services/WebView2Manager.cs @@ -46,6 +46,7 @@ public sealed class WebView2Manager : IDisposable public long Epoch; public WriteableBitmap? PreviewBitmap; public bool DebugPngWritten; + public int WidgetDumpRemaining; public int TelemetryCaptureCount; public byte[] RentOutBuffer(int size) @@ -179,9 +180,19 @@ public sealed class WebView2Manager : IDisposable _ = cws.ExecuteScriptAsync(TransparentBackgroundScript); }; - cws.NavigationCompleted += (_, _) => + cws.NavigationCompleted += (_, e) => { _ = cws.ExecuteScriptAsync(TransparentBackgroundScript); + if (e.IsSuccess && session.Control.CoreWebView2?.Source is { } src && src != "about:blank") + { + // Arm a one-shot diagnostic dump of the REAL widget document + // (a few post-paint captures so styles/backgrounds settle) — + // the old first-capture dump only ever saw the about:blank + // document, so whether this widget's capture actually HAS the + // transparent margins the injection promises has never been + // verified (MyMistakes: "? VERIFY"). + session.WidgetDumpRemaining = 5; + } _ = CaptureFrame(source.Id); }; @@ -257,6 +268,29 @@ public sealed class WebView2Manager : IDisposable return (minX, minY, maxX - minX + 1, maxY - minY + 1); } + // alpha[min/max/mean/zero%] — the one number that decides whether a widget's + // capture really has transparent margins (0% = opaque, ~100% if the page + // painted transparent-only). Sampled every 16th pixel like the original log. + private static string AlphaStats(byte[] pixels, int pixW, int pixH, int stride) + { + int minA = 255, maxA = 0; + long sumA = 0, nA = 0, zeroA = 0; + for (var y = 0; y < pixH; y += 16) + { + var row = y * stride; + for (var x = 0; x < pixW; x += 16) + { + var a = pixels[row + x * 4 + 3]; + if (a < minA) minA = a; + if (a > maxA) maxA = a; + sumA += a; + if (a == 0) zeroA++; + nA++; + } + } + return $"alpha[min={minA},max={maxA},mean={sumA / (double)nA:F1},zero={100.0 * zeroA / nA:F1}%]"; + } + private async Task CaptureFrame(string sourceId) { if (!_sessions.TryGetValue(sourceId, out var session) || session.Disposed) return; @@ -313,27 +347,31 @@ public sealed class WebView2Manager : IDisposable if (session.Epoch == 0) { - int minA = 255, maxA = 0; - long sumA = 0, nA = 0, zeroA = 0; - for (var y = 0; y < pixH; y += 16) - { - var row = y * stride; - for (var x = 0; x < pixW; x += 16) - { - var a = pixels[row + x * 4 + 3]; - if (a < minA) minA = a; - if (a > maxA) maxA = a; - sumA += a; - if (a == 0) zeroA++; - nA++; - } - } AppLog.Write( $"WebView2Manager: first capture for '{sourceId}' {pixW}x{pixH} " + - $"alpha[min={minA},max={maxA},mean={sumA / (double)nA:F1},zero={100.0 * zeroA / nA:F1}%] " + + $"{AlphaStats(pixels, pixW, pixH, stride)} " + $"contentBounds=({cropX},{cropY},{cropW},{cropH})"); } + if (session.WidgetDumpRemaining > 0) + { + var dumpNo = 5 - session.WidgetDumpRemaining + 1; + session.WidgetDumpRemaining--; + try + { + var dbg = Path.Combine(Path.GetTempPath(), $"ytLive-web-{sourceId}-w{dumpNo}.png"); + File.WriteAllBytes(dbg, ms.ToArray()); + AppLog.Write( + $"WebView2Manager: widget capture [{dumpNo}/5] for '{sourceId}' {pixW}x{pixH} " + + $"{AlphaStats(pixels, pixW, pixH, stride)} " + + $"contentBounds=({cropX},{cropY},{cropW},{cropH}) -> {dbg}"); + } + catch (Exception dex) + { + AppLog.Write($"WebView2Manager: widget dump failed: {dex.Message}"); + } + } + var outPixels = session.RentOutBuffer(cropW * cropH * 4); if (cropW == pixW && cropH == pixH && cropX == 0 && cropY == 0) Buffer.BlockCopy(pixels, 0, outPixels, 0, outPixels.Length); diff --git a/ai.md b/ai.md index 697e3cc..321429a 100644 --- a/ai.md +++ b/ai.md @@ -911,6 +911,16 @@ Full suite 290/291 passing, the sole failure the pre-existing compositor pixel t `NamedPipeAudioWriter.DroppedWrites` — names the exact stage on the next take without new code. `AudioMixer.FillAndMix` now returns `(MicRms, MicDrained, LoopDrained)` for the telemetry accumulation. Full suite 290/291, same pre-existing sole failure. +- **Slice 13 (web transparency diagnostic re-point, 2026-09-10):** the "?" in MyMistakes' + "RESOLVED (? VERIFY)" is being closed. The one-shot widget dump + alpha stats bound to the + FIRST capture ever = the initial about:blank document — a blind instrument. NavigationCompleted + for the REAL widget URL now arms `WidgetDumpRemaining = 5`; those 5 post-paint captures dump + `%TEMP%\ytLive-web--w1..5.png` + the shared `AlphaStats(...)` line (alpha[min/max/mean/zero%]) + + FindContentBounds result to startup.log. That line alone decides the fix branch: zero% ⇒ the + pre-parse injection did not hold for this widget (opaque capture → chroma-key / stronger DOM + injection); large zero% + tight contentBounds ⇒ the capture IS transparent and the recording's + black block lives downstream (compositor blend/underlay). `AlphaStats` extracted as the shared + sampler. No new tests (WebView2 runtime not instantiable in tests; the re-point is log-only). - **Stop ordering matters:** `StopAsync` stops the encoder — since slice 10 it FLUSHES the pending queue (`Channel.TryComplete` → drain writes the leftovers, closes stdin → EOF → ffmpeg finalizes+exits; an accepted frame is never lost) — **before** awaiting the loop. The old reverse-order deadlock was