feat: web sources frame-captured via composition (CoreWebView2CompositionController → Windows.Graphics.Capture); PNG poll + CaptureScheduler deleted
The ~30Hz CapturePreviewAsync PNG poll capped real cadence at ~20Hz (35–165ms full-HD encode+decode), so a 60fps widget still juddered at ~1/6 speed. Replaced polling with frame-driven capture of the composition controller's root visual — the mechanism WebView2CompositionControl and Flutter's webview_windows use (graphics_context.cc captures the root surface_ visual via CreateGraphicsCaptureItemFromVisual; reference: github.com/microsoft/Windows.UI.Composition.WinUI / flutter-internal webview_windows). Frames now arrive at the renderer's own pace; capture memory is epoch'd ring reuse + one crop-sized shared WriteableBitmap. New Services/WebCaptureFrameSource.cs owns GraphicsCaptureItem + free- threaded Direct3D11CaptureFramePool + session (Straight alpha readback, per-frame FindContentBounds → CropBounds). WebView2Manager reworked around per-session composition controllers + one UI-thread Compositor created via the CoreMessaging CreateDispatcherQueueController P/Invoke (the 19041 projection lacks CreateOnCurrentThread); internal seam ctor (Dispatcher, Func<string,IScreenCaptureSource>?) for hermetic tests. CaptureScheduler.cs deleted; the three SetCaptureInterval cadence hooks removed; InitWebView2() moved from MainWindow ctor to Loaded (a parent HWND must exist for the composition controller); the hidden WebViewHostPanel overlay deleted. TransparentBackgroundScript unchanged. Tests: WebView2ManagerTests reworked — 4 control-size + scheduler tests dropped, FindContentBounds tests moved to WebCaptureFrameSource, ONE integration test (Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds) drives the seam with a FakeWebSource + background-STA DispatcherPump. Suite 293/293, 0 warnings. NOTE: composition path NOT yet verified on a device — the take is the next step. Web work committed locally only (no push per standing rule). Docs same-commit: ai.md Slice 14 + supersede marker on Slice 11, HANDOFF, MyMistakes (CoreMessaging DQ + namespace-landmine recipe), TASK 17, Controls/ViewModels/Services indexes.
This commit is contained in:
@@ -893,6 +893,8 @@ seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<Compos
|
||||
(2026-09-10, the "widget animates too slow" take):** the recording is 60fps but the WebView2
|
||||
capture loop was a blind 100ms `DispatcherTimer` = 10Hz — a 60fps-designed widget was sampled 6×
|
||||
under its native rate (repeated-footage slow-mo). Two stacked throttles:
|
||||
> Superseded by **Slice 14** (below): `CaptureScheduler` and the cadence hooks were the polling
|
||||
> design; frame-driven composition capture replaced them — nothing here is current architecture.
|
||||
(1) **the 10Hz cap** (unconditional, code-proven) and
|
||||
(2) **Chromium hidden-page throttling** (`requestAnimationFrame` parked, JS timers clamped to ~1s
|
||||
per WebView2Feedback#1172/#3070 + Chrome-88 timer throttling) whenever the app window is unfocused
|
||||
@@ -935,11 +937,65 @@ Full suite 290/291 passing, the sole failure the pre-existing compositor pixel t
|
||||
FIRST capture ever = the initial about:blank document — a blind instrument. NavigationCompleted
|
||||
for the REAL widget URL now arms `WidgetDumpRemaining = 5`; those 5 post-paint captures dump
|
||||
`%TEMP%\ytLive-web-<id>-w1..5.png` + the shared `AlphaStats(...)` line (alpha[min/max/mean/zero%])
|
||||
+ FindContentBounds result to startup.log. That line alone decides the fix branch: zero% ⇒ the
|
||||
pre-parse injection did not hold for this widget (opaque capture → chroma-key / stronger DOM
|
||||
injection); large zero% + tight contentBounds ⇒ the capture IS transparent and the recording's
|
||||
black block lives downstream (compositor blend/underlay). `AlphaStats` extracted as the shared
|
||||
sampler. No new tests (WebView2 runtime not instantiable in tests; the re-point is log-only).
|
||||
+ FindContentBounds result to startup.log. That line alone decides the fix branch: zero% ⇒ the
|
||||
pre-parse injection did not hold for this widget (opaque capture → chroma-key / stronger DOM
|
||||
injection); large zero% + tight contentBounds ⇒ the capture IS transparent and the recording's
|
||||
black block lives downstream (compositor blend/underlay). `AlphaStats` extracted as the shared
|
||||
sampler. No new tests (WebView2 runtime not instantiable in tests; the re-point is log-only).
|
||||
- **Slice 14 — web frames are now COMPOSITION-CAPTURED; the PNG poll + `CaptureScheduler` are gone
|
||||
(2026-09-14, the "still not 60fps" take):** slice 11's ~30Hz was still a `CapturePreviewAsync` PNG
|
||||
poll — every 1920×1080 full-HD encode+decode cost 35–165ms, so the session budget capped real
|
||||
cadence at ~20Hz for a 60fps-designed widget. Root fix = frame-driven capture, not faster polling:
|
||||
the WebView2 renderer now feeds `Windows.Graphics.Capture` directly through a
|
||||
`CoreWebView2CompositionController` (the mechanism `WebView2CompositionControl` and Flutter's
|
||||
`webview_windows` use — see `graphics_context.cc` `CreateGraphicsCaptureItemFromVisual`, which
|
||||
captures the root `surface_` visual).
|
||||
- **Pipeline (per session):** `CreateCoreWebView2CompositionControllerAsync(WindowHandle)` — the
|
||||
parent HWND must exist, so `MainWindow.InitWebView2()` moved from the ctor to **Loaded** (was
|
||||
`InitWebView2(WebViewHostPanel)`; the hidden XAML `WebViewHostPanel` overlay is deleted).
|
||||
`controller.RootVisualTarget` = a child `ContainerVisual` (`RelativeSizeAdjustment = 1,1`) under a
|
||||
root `ContainerVisual` (1920×1080, `IsVisible=true`) built on ONE `Windows.UI.Composition.Compositor`
|
||||
(created once on the UI thread); `Bounds = 0,0,1920,1080`, `BoundsMode=UseRawPixels`,
|
||||
`ShouldDetectMonitorScaleChanges=false`, `RasterizationScale=1.0`, `IsVisible=true`,
|
||||
`DefaultBackgroundColor=Transparent`; then `GraphicsCaptureItem.CreateFromVisual(root)`.
|
||||
Frames pull from a free-threaded `Direct3D11CaptureFramePool` (2 buffers) →
|
||||
`SoftwareBitmap.CreateCopyFromSurfaceAsync` (BGRA, `BitmapAlphaMode.Straight` — the compositor
|
||||
blends STRAIGHT alpha, so premultiplied readback would wreck corner anti-aliasing) → per-frame
|
||||
`FindContentBounds` alpha-bbox on the capture worker → `VideoFrame.CropBounds` stamped → ring up to
|
||||
2 fresh + Epoch (reuse; GC lesson slice 8) → dispatcher-coalesced copy (Render priority) into ONE
|
||||
crop-sized shared `WriteableBitmap`; `PreviewBitmapChanged` raised only on bitmap (re)creation.
|
||||
Also unpumped: the OBS-side de-throttle flags (slice 11) stay — they fix the hidden-page JS
|
||||
throttling; the capture path itself no longer polls.
|
||||
- **Implemented in:** new `Services/WebCaptureFrameSource.cs` (owns item+pool+session — mirrors
|
||||
`ScreenCaptureFrameSource`, which servers as the copy template) + reworked `Services/WebView2Manager.cs`
|
||||
(owns controllers, the visual tree, nav/script wiring, preview; internal seam ctor
|
||||
`(Dispatcher, Func<string, IScreenCaptureSource>?)` so the tests instantiate zero WinRT; public
|
||||
ctor `(Dispatcher)`).
|
||||
- **CoreMessaging DQ recipe (record-once):** the 19041 CsWinRT projection has NO
|
||||
`DispatcherQueueController.CreateOnCurrentThread()` (CS0117 — only `CreateOnDedicatedThread` +
|
||||
`FromAbi(IntPtr)`). P/Invoke `coreMessaging.dll!CreateDispatcherQueueController` with struct
|
||||
`DispatcherQueueOptions { DwSize, ThreadType = 2 (DQTYPE_THREAD_CURRENT), ApartmentType = 2 (DQTAT_COM_STA) }`,
|
||||
wrap via `DispatcherQueueController.FromAbi(ptr)` (mirror of `CaptureInterop`), only then
|
||||
`new Compositor()` — all once on the WPF UI thread (the app dispatches Render there).
|
||||
- **Compile notes (each cost a build cycle):** `Compositor` collides with the repo's OWN
|
||||
`ytLive.Services.Compositor` namespace → fully-qualify `Windows.UI.Composition.Compositor`;
|
||||
`CoreWebView2CompositionController` exposes **`Close()`**, not `Dispose()`; `Color` is ambiguous
|
||||
(`System.Drawing` vs `System.Windows.Media`) → `System.Drawing.Color.Transparent`.
|
||||
- **Dropped:** `Services/CaptureScheduler.cs` (DELETED — no cadence to schedule), all three
|
||||
`SetCaptureInterval` hooks in `MainViewModel.Streaming.Operations.cs`, the XAML `WebViewHostPanel`.
|
||||
`TransparentBackgroundScript` const is unchanged (a test pins it).
|
||||
- **Test model (Good Dog):** reworked `ytLive.Tests/WebView2ManagerTests.cs` — dropped the 4
|
||||
control-size tests + the `CaptureScheduler_Drops…` test; `FindContentBounds` tests moved to
|
||||
`WebCaptureFrameSource.FindContentBounds`; the ONE integration test
|
||||
`Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds` drives the internal seam with
|
||||
`FakeWebSource : IScreenCaptureSource` + a real background-STA `DispatcherPump` (copied from
|
||||
`ScreenCaptureManagerTests`): register → one crop-sized preview bitmap published → `CropBounds` +
|
||||
`IsOpaque=false` on `GetLatestFrame` → back-to-back pumps coalesce to latest. Byte assertion compares
|
||||
the DENSE 2×2 crop (`CropBytes` helper — slices source rows with stride gaps, a contiguous range
|
||||
spans rows wrongly).
|
||||
- Full suite **293/293 green, 0 warnings**. Audio untouched. **NOT YET VERIFIED ON DEVICE** — the
|
||||
composition path needs a real 60fps-widget run (open item; see HANDOFF). Web work commits stay
|
||||
LOCAL (no push) until the user greenlights.
|
||||
- **Stop ordering matters:** `StopAsync` stops the encoder — since slice 10 it FLUSHES the pending
|
||||
queue (`Channel.TryComplete` → drain writes the leftovers, closes stdin → EOF → ffmpeg finalizes+exits;
|
||||
an accepted frame is never lost) — **before** awaiting the loop. The old reverse-order deadlock was
|
||||
|
||||
Reference in New Issue
Block a user