feat: web sources frame-captured via composition (CoreWebView2CompositionController → Windows.Graphics.Capture); PNG poll + CaptureScheduler deleted

The ~30Hz CapturePreviewAsync PNG poll capped real cadence at ~20Hz
(35–165ms full-HD encode+decode), so a 60fps widget still juddered at ~1/6
speed. Replaced polling with frame-driven capture of the composition
controller's root visual — the mechanism WebView2CompositionControl and
Flutter's webview_windows use (graphics_context.cc captures the root
surface_ visual via CreateGraphicsCaptureItemFromVisual; reference:
github.com/microsoft/Windows.UI.Composition.WinUI / flutter-internal
webview_windows). Frames now arrive at the renderer's own pace; capture
memory is epoch'd ring reuse + one crop-sized shared WriteableBitmap.

New Services/WebCaptureFrameSource.cs owns GraphicsCaptureItem + free-
threaded Direct3D11CaptureFramePool + session (Straight alpha readback,
per-frame FindContentBounds → CropBounds). WebView2Manager reworked around
per-session composition controllers + one UI-thread Compositor created via
the CoreMessaging CreateDispatcherQueueController P/Invoke (the 19041
projection lacks CreateOnCurrentThread); internal seam ctor
(Dispatcher, Func<string,IScreenCaptureSource>?) for hermetic tests.
CaptureScheduler.cs deleted; the three SetCaptureInterval cadence hooks
removed; InitWebView2() moved from MainWindow ctor to Loaded (a parent
HWND must exist for the composition controller); the hidden WebViewHostPanel
overlay deleted. TransparentBackgroundScript unchanged.

Tests: WebView2ManagerTests reworked — 4 control-size + scheduler tests
dropped, FindContentBounds tests moved to WebCaptureFrameSource, ONE
integration test (Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds)
drives the seam with a FakeWebSource + background-STA DispatcherPump.
Suite 293/293, 0 warnings.

NOTE: composition path NOT yet verified on a device — the take is the
next step. Web work committed locally only (no push per standing rule).

Docs same-commit: ai.md Slice 14 + supersede marker on Slice 11, HANDOFF,
MyMistakes (CoreMessaging DQ + namespace-landmine recipe), TASK 17,
Controls/ViewModels/Services indexes.
This commit is contained in:
2026-09-14 16:14:00 -07:00
parent 40c453805d
commit c01206fb8a
16 changed files with 890 additions and 581 deletions
+61 -5
View File
@@ -893,6 +893,8 @@ seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<Compos
(2026-09-10, the "widget animates too slow" take):** the recording is 60fps but the WebView2
capture loop was a blind 100ms `DispatcherTimer` = 10Hz — a 60fps-designed widget was sampled 6×
under its native rate (repeated-footage slow-mo). Two stacked throttles:
> Superseded by **Slice 14** (below): `CaptureScheduler` and the cadence hooks were the polling
> design; frame-driven composition capture replaced them — nothing here is current architecture.
(1) **the 10Hz cap** (unconditional, code-proven) and
(2) **Chromium hidden-page throttling** (`requestAnimationFrame` parked, JS timers clamped to ~1s
per WebView2Feedback#1172/#3070 + Chrome-88 timer throttling) whenever the app window is unfocused
@@ -935,11 +937,65 @@ Full suite 290/291 passing, the sole failure the pre-existing compositor pixel t
FIRST capture ever = the initial about:blank document — a blind instrument. NavigationCompleted
for the REAL widget URL now arms `WidgetDumpRemaining = 5`; those 5 post-paint captures dump
`%TEMP%\ytLive-web-<id>-w1..5.png` + the shared `AlphaStats(...)` line (alpha[min/max/mean/zero%])
+ FindContentBounds result to startup.log. That line alone decides the fix branch: zero% ⇒ the
pre-parse injection did not hold for this widget (opaque capture → chroma-key / stronger DOM
injection); large zero% + tight contentBounds ⇒ the capture IS transparent and the recording's
black block lives downstream (compositor blend/underlay). `AlphaStats` extracted as the shared
sampler. No new tests (WebView2 runtime not instantiable in tests; the re-point is log-only).
+ FindContentBounds result to startup.log. That line alone decides the fix branch: zero% ⇒ the
pre-parse injection did not hold for this widget (opaque capture → chroma-key / stronger DOM
injection); large zero% + tight contentBounds ⇒ the capture IS transparent and the recording's
black block lives downstream (compositor blend/underlay). `AlphaStats` extracted as the shared
sampler. No new tests (WebView2 runtime not instantiable in tests; the re-point is log-only).
- **Slice 14 — web frames are now COMPOSITION-CAPTURED; the PNG poll + `CaptureScheduler` are gone
(2026-09-14, the "still not 60fps" take):** slice 11's ~30Hz was still a `CapturePreviewAsync` PNG
poll — every 1920×1080 full-HD encode+decode cost 35–165ms, so the session budget capped real
cadence at ~20Hz for a 60fps-designed widget. Root fix = frame-driven capture, not faster polling:
the WebView2 renderer now feeds `Windows.Graphics.Capture` directly through a
`CoreWebView2CompositionController` (the mechanism `WebView2CompositionControl` and Flutter's
`webview_windows` use — see `graphics_context.cc` `CreateGraphicsCaptureItemFromVisual`, which
captures the root `surface_` visual).
- **Pipeline (per session):** `CreateCoreWebView2CompositionControllerAsync(WindowHandle)` — the
parent HWND must exist, so `MainWindow.InitWebView2()` moved from the ctor to **Loaded** (was
`InitWebView2(WebViewHostPanel)`; the hidden XAML `WebViewHostPanel` overlay is deleted).
`controller.RootVisualTarget` = a child `ContainerVisual` (`RelativeSizeAdjustment = 1,1`) under a
root `ContainerVisual` (1920×1080, `IsVisible=true`) built on ONE `Windows.UI.Composition.Compositor`
(created once on the UI thread); `Bounds = 0,0,1920,1080`, `BoundsMode=UseRawPixels`,
`ShouldDetectMonitorScaleChanges=false`, `RasterizationScale=1.0`, `IsVisible=true`,
`DefaultBackgroundColor=Transparent`; then `GraphicsCaptureItem.CreateFromVisual(root)`.
Frames pull from a free-threaded `Direct3D11CaptureFramePool` (2 buffers) →
`SoftwareBitmap.CreateCopyFromSurfaceAsync` (BGRA, `BitmapAlphaMode.Straight` — the compositor
blends STRAIGHT alpha, so premultiplied readback would wreck corner anti-aliasing) → per-frame
`FindContentBounds` alpha-bbox on the capture worker → `VideoFrame.CropBounds` stamped → ring up to
2 fresh + Epoch (reuse; GC lesson slice 8) → dispatcher-coalesced copy (Render priority) into ONE
crop-sized shared `WriteableBitmap`; `PreviewBitmapChanged` raised only on bitmap (re)creation.
Also unpumped: the OBS-side de-throttle flags (slice 11) stay — they fix the hidden-page JS
throttling; the capture path itself no longer polls.
- **Implemented in:** new `Services/WebCaptureFrameSource.cs` (owns item+pool+session — mirrors
`ScreenCaptureFrameSource`, which servers as the copy template) + reworked `Services/WebView2Manager.cs`
(owns controllers, the visual tree, nav/script wiring, preview; internal seam ctor
`(Dispatcher, Func<string, IScreenCaptureSource>?)` so the tests instantiate zero WinRT; public
ctor `(Dispatcher)`).
- **CoreMessaging DQ recipe (record-once):** the 19041 CsWinRT projection has NO
`DispatcherQueueController.CreateOnCurrentThread()` (CS0117 — only `CreateOnDedicatedThread` +
`FromAbi(IntPtr)`). P/Invoke `coreMessaging.dll!CreateDispatcherQueueController` with struct
`DispatcherQueueOptions { DwSize, ThreadType = 2 (DQTYPE_THREAD_CURRENT), ApartmentType = 2 (DQTAT_COM_STA) }`,
wrap via `DispatcherQueueController.FromAbi(ptr)` (mirror of `CaptureInterop`), only then
`new Compositor()` — all once on the WPF UI thread (the app dispatches Render there).
- **Compile notes (each cost a build cycle):** `Compositor` collides with the repo's OWN
`ytLive.Services.Compositor` namespace → fully-qualify `Windows.UI.Composition.Compositor`;
`CoreWebView2CompositionController` exposes **`Close()`**, not `Dispose()`; `Color` is ambiguous
(`System.Drawing` vs `System.Windows.Media`) → `System.Drawing.Color.Transparent`.
- **Dropped:** `Services/CaptureScheduler.cs` (DELETED — no cadence to schedule), all three
`SetCaptureInterval` hooks in `MainViewModel.Streaming.Operations.cs`, the XAML `WebViewHostPanel`.
`TransparentBackgroundScript` const is unchanged (a test pins it).
- **Test model (Good Dog):** reworked `ytLive.Tests/WebView2ManagerTests.cs` — dropped the 4
control-size tests + the `CaptureScheduler_Drops…` test; `FindContentBounds` tests moved to
`WebCaptureFrameSource.FindContentBounds`; the ONE integration test
`Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds` drives the internal seam with
`FakeWebSource : IScreenCaptureSource` + a real background-STA `DispatcherPump` (copied from
`ScreenCaptureManagerTests`): register → one crop-sized preview bitmap published → `CropBounds` +
`IsOpaque=false` on `GetLatestFrame` → back-to-back pumps coalesce to latest. Byte assertion compares
the DENSE 2×2 crop (`CropBytes` helper — slices source rows with stride gaps, a contiguous range
spans rows wrongly).
- Full suite **293/293 green, 0 warnings**. Audio untouched. **NOT YET VERIFIED ON DEVICE** — the
composition path needs a real 60fps-widget run (open item; see HANDOFF). Web work commits stay
LOCAL (no push) until the user greenlights.
- **Stop ordering matters:** `StopAsync` stops the encoder — since slice 10 it FLUSHES the pending
queue (`Channel.TryComplete` → drain writes the leftovers, closes stdin → EOF → ffmpeg finalizes+exits;
an accepted frame is never lost) — **before** awaiting the loop. The old reverse-order deadlock was