feat: web sources frame-captured via composition (CoreWebView2CompositionController → Windows.Graphics.Capture); PNG poll + CaptureScheduler deleted
The ~30Hz CapturePreviewAsync PNG poll capped real cadence at ~20Hz (35–165ms full-HD encode+decode), so a 60fps widget still juddered at ~1/6 speed. Replaced polling with frame-driven capture of the composition controller's root visual — the mechanism WebView2CompositionControl and Flutter's webview_windows use (graphics_context.cc captures the root surface_ visual via CreateGraphicsCaptureItemFromVisual; reference: github.com/microsoft/Windows.UI.Composition.WinUI / flutter-internal webview_windows). Frames now arrive at the renderer's own pace; capture memory is epoch'd ring reuse + one crop-sized shared WriteableBitmap. New Services/WebCaptureFrameSource.cs owns GraphicsCaptureItem + free- threaded Direct3D11CaptureFramePool + session (Straight alpha readback, per-frame FindContentBounds → CropBounds). WebView2Manager reworked around per-session composition controllers + one UI-thread Compositor created via the CoreMessaging CreateDispatcherQueueController P/Invoke (the 19041 projection lacks CreateOnCurrentThread); internal seam ctor (Dispatcher, Func<string,IScreenCaptureSource>?) for hermetic tests. CaptureScheduler.cs deleted; the three SetCaptureInterval cadence hooks removed; InitWebView2() moved from MainWindow ctor to Loaded (a parent HWND must exist for the composition controller); the hidden WebViewHostPanel overlay deleted. TransparentBackgroundScript unchanged. Tests: WebView2ManagerTests reworked — 4 control-size + scheduler tests dropped, FindContentBounds tests moved to WebCaptureFrameSource, ONE integration test (Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds) drives the seam with a FakeWebSource + background-STA DispatcherPump. Suite 293/293, 0 warnings. NOTE: composition path NOT yet verified on a device — the take is the next step. Web work committed locally only (no push per standing rule). Docs same-commit: ai.md Slice 14 + supersede marker on Slice 11, HANDOFF, MyMistakes (CoreMessaging DQ + namespace-landmine recipe), TASK 17, Controls/ViewModels/Services indexes.
This commit is contained in:
+38
-1
@@ -418,7 +418,8 @@ CLR→real-ffmpeg run to the native Windows suite.
|
||||
"not a regression" for weeks without ever recording WHY.)
|
||||
|
||||
**The trap:** `VisualTreeHelper.HitTest(window, pt)` returned the window's
|
||||
`WebViewHostPanel` overlay (`IsHitTestVisible="False"`, `Opacity=0`, ZERO children) for
|
||||
`WebViewHostPanel` overlay (`IsHitTestVisible="False"`, `Opacity=0`, ZERO children — since the
|
||||
2026-09-14 composition-capture redesign the overlay is DELETED, but the API lesson stands) for
|
||||
EVERY point in the window — so a "corner is grabbable" assertion could never pass, and
|
||||
it looked like a real interaction bug. The actual input pipeline (`UIElement.InputHitTest`,
|
||||
what Mouse routing uses) correctly returned the element's Grid at elem-center/corner-in/
|
||||
@@ -508,6 +509,42 @@ while live (`IsEditMode`) since a mid-stream advance flip is meaningless. Test r
|
||||
`StartLive_NegativeOffset_AdvancesAudio_ByDroppingTheStreamHead` (emit 6×0.9 into an 8-tick budget,
|
||||
long 0.2 bed, wire must show only 0.2).
|
||||
|
||||
## Composition video capture of WebView2: the CoreMessaging DQ recipe + the build gotchas
|
||||
|
||||
(2026-09-14, the ~20→60fps web-capture redesign — record-once so the next GPU-path work never
|
||||
re-derives it. Everything that follows was already settled by OBS/Flutter `webview_windows`; the
|
||||
cost this session was the 19041 projection's gaps.)
|
||||
|
||||
1. **No `DispatcherQueueController.CreateOnCurrentThread()` on the 10.0.19041 projection**
|
||||
(CS0117 — only `CreateOnDedicatedThread` + `FromAbi(IntPtr)` exist). P/Invoke
|
||||
`coreMessaging.dll!CreateDispatcherQueueController` with a sequential
|
||||
`DispatcherQueueOptions { DwSize, ThreadType=2 (DQTYPE_THREAD_CURRENT), ApartmentType=2 (DQTAT_COM_STA) }`,
|
||||
wrap via `DispatcherQueueController.FromAbi(ptr)` (mirror of `CaptureInterop`), THEN `new Compositor()`.
|
||||
One controller + one compositor per UI thread, created once.
|
||||
2. **`CoreWebView2CompositionController` needs a real parent HWND** — there is no window in the
|
||||
MainWindow ctor, so `InitWebView2()` moved from the ctor to `MainWindow_Loaded` (source
|
||||
registration is guarded by `ContainsKey`, so the layout-load path is safe).
|
||||
3. **Capture the ROOT visual, not the control.** `GraphicsItem` for a composition controller comes
|
||||
from `GraphicsCaptureItem.CreateFromVisual(root)` where `root` is your own 1920×1080
|
||||
`ContainerVisual` (`IsVisible=true`) holding the controller's `RootVisualTarget` as a
|
||||
`RelativeSizeAdjustment=1,1` child — exactly what `webview_windows`'s `graphics_context.cc`
|
||||
does (`CreateGraphicsCaptureItemFromVisual` on the root `surface_` visual). Hide nothing,
|
||||
move nothing, poll nothing — the capture is frame-driven at the renderer's pace.
|
||||
4. **Straight alpha, never premultiplied.** Read back with `BitmapAlphaMode.Straight`; the
|
||||
compositor's blend is straight-alpha, so `Premultiplied` readback wrecks corner anti-aliasing.
|
||||
5. **Namespace landmines that each cost a build cycle in `Services/`:** `Compositor` resolves to the
|
||||
repo's OWN `ytLive.Services.Compositor` namespace — fully-qualify `Windows.UI.Composition.Compositor`;
|
||||
`CoreWebView2CompositionController.Close()` is the disposal call (no `Dispose()`); `Color` is
|
||||
ambiguous (`System.Drawing` vs `System.Windows.Media`) — qualify `System.Drawing.Color.Transparent`.
|
||||
6. **Testing a compositor-built bitmap without a runtime:** the internal seam ctor
|
||||
`(Dispatcher, Func<string, IScreenCaptureSource>?)` + a `FakeWebSource` + a real background-STA
|
||||
`DispatcherPump` (borrowed from ScreenCaptureManagerTests) drives the whole session path hermetic.
|
||||
When byte-comparing a crop, slice the source with stride gaps (helper `CropBytes`) — a contiguous
|
||||
range silently spans rows.
|
||||
|
||||
Recipe verified on green suite + 0-warning build; the composition path itself still needs a device
|
||||
take (open item, HANDOFF).
|
||||
|
||||
## A pipe-read test that samples one tick is a timing flake by construction
|
||||
|
||||
(2026-09-14, re-discovered) `Mix_HonorsProviderGains_AndGameMute_KillsTheLoopback` fails
|
||||
|
||||
Reference in New Issue
Block a user