diff --git a/Controls/index.md b/Controls/index.md index 6f2489d..1158553 100644 --- a/Controls/index.md +++ b/Controls/index.md @@ -47,7 +47,10 @@ cross-cutting calls through it: `WebViewHostPanel` (ctor `InitWebView2(WebViewHostPanel)`) and `ToastArea` (`NotificationService.InArea("ToastArea")`) are resolved by name from the window root — so `OverlayHost` deliberately leaves them as thin window-root - children (Commit 18). + children (Commit 18). Since the 2026-09-14 composition-capture redesign only + `ToastArea` remains — `WebViewHostPanel` is deleted (web sources no longer + host a multiline XAML control; `InitWebView2()` runs from `MainWindow_Loaded` + with no panel argument). - When cutting visual wrappers, don't drop the outer `Border`'s background/glow/margin — Commit 15 initially lost the preview's outer border and had to restore it same-commit. diff --git a/HANDOFF.md b/HANDOFF.md index 5d05875..bcf9c01 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,14 +1,10 @@ -# HANDOFF — 2026-09-14 (signed audio sync SHIPPED + PUSHED: −500..+500, negative advances by eating the stream head; slider locked while live/recording) +# HANDOFF — 2026-09-14 (web capture REDESIGNED: composition-capture, PNG polling + CaptureScheduler deleted — NOT yet verified on device, NOT pushed) ## Branch / Commit State -`main` HEAD = **`b22d08e`** (signed audio-sync, committed AND **pushed** to origin/main with the -user's greenlight — 29 commits pushed in one go). Working tree **clean**. - -DB now: **`Audio.SyncOffsetMs = 0`** (confirmed via sqlite3 — the creator slid the sync to 0 -and the write finally stuck; sound is correct at 0). The 0.54s residual in the 1128 take ≈ 300ms -injected offset (leftover DB value) + ~240ms natural (partly webcam clap-quantization ±40–85ms, -partly measurement). +`main` HEAD = **`b22d08e`** (signed audio-sync, already pushed). Working tree **DIRTY** with the +web-frame-capture redesign (below). Commits for it land **LOCALLY only** — the web work was never +pushed; the user greenlights pushes at checkpoints. ## ⚠️ Branding (2026-09-14, creator-corrected): product = **llamacasty**, internals = ytLive @@ -17,72 +13,84 @@ The product is **llamacasty**; the repo path, csproj `AssemblyName`/`RootNamespa language must say "llamacasty"; code/assembly/repo names stay ytLive. Full detail in `ai.md` → Brand → "Product name vs repo/assembly branding". -## ✅ SHIPPED (this dirty tree) — signed audio sync, −500..+500 +## 💥 The web-capture take history (why this redesign exists) -**What:** the audio-sync control is now a SIGNED offset. Positive = delay the mix (audio runs -AHEAD of video — existing `AudioSyncDelay` behavior, unchanged and live-reactive). Negative = -**advance** the audio (audio runs BEHIND video): OBS's "eat the head of the buffer" fix — -the mixer drops the first |N| ms of the written stream at the pipe, re-anchoring the audio -stream so every event lands |N| ms EARLIER relative to video. +The recording was 60fps but web widgets ran at ~1/6 speed: capture was a `CapturePreviewAsync` PNG +poll; every full-HD encode+decode cost **35–165ms** so the session budget capped real cadence at +~20Hz — a 60fps widget still juddered. Slice 11's `CaptureScheduler` (30Hz) was the wrong lever: +the fix is frame-driven, not faster polling. -**Mechanism (AudioMixer):** `StartLive` arms `_advanceSamplesRemaining = |N| ms → samples` at -go-live (a negative offset can only eat the HEAD of the stream; it is armed once, not -live-reactive). `LiveLoopAsync` skips `min(budget, mixBuffer.Length)` samples off each write -head while the budget lasts — the pipe writer accepts a partial chunk via `AsMemory(writeFrom)`. -Positive path untouched (delay line still re-reads the Func every tick). +## 🔬 IN FLIGHT (this dirty tree, NOT committed, NOT pushed) — composition capture -**UI/plumbing:** -- `MainViewModel.Audio.cs` — clamp `Math.Clamp(value, -500, 500)`, doc updated. -- `LayoutStore.Settings.cs` — `LoadAudioSyncOffsetMs`/`SaveAudioSyncOffsetMs` clamp −500..500. -- `PreviewPane.xaml` — label **SYNC → "AUDIO SYNC"**, `Minimum="-500"`, tooltip explains both - directions (calibrate with a clap: clap late → negative; early → positive), and - **`IsEnabled="{Binding IsEditMode}"`** — the slider locks during live AND recording (gun - safety, same property `IsRecording`/`StreamStatus` already raise PropertyChanged for). -- `AudioSyncDelay` unchanged (still clamps negative→0 internally; header doc updated to point - at the mixer for the advance side). +**What:** web sources now render through `CoreWebView2CompositionController` into +`Windows.Graphics.Capture` (the Flutter `webview_windows` / `WebView2CompositionControl` +mechanism) — frame-driven at the renderer's pace instead of polling PNGs. -**Regression test (the ONE integration test for this change):** -`StartLive_NegativeOffset_AdvancesAudio_ByDroppingTheStreamHead` in `AudioPipelineTests.cs` — -−40 ms advance (8-tick budget at 5ms interval), emits 6×0.9 fresh right after StartLive (≤ -budget, so the head MUST be eaten) then a long 0.2 bed; asserts the wire max ≈ 0.2 (< 0.3). -Fails loudly if −N no longer drops the head (0.9 leaks). +**Pipeline (per session):** `CreateCoreWebView2CompositionControllerAsync(appHwnd)` → +`RootVisualTarget` = a `RelativeSizeAdjustment=1,1` child under a 1920×1080 root `ContainerVisual` +on one `Compositor` (CoreMessaging DQ P/Invoke recipe — see ai.md Slice 14 / MyMistakes) → +`GraphicsCaptureItem.CreateFromVisual(root)` → free-threaded `Direct3D11CaptureFramePool` (2 +buffers) → `SoftwareBitmap.CreateCopyFromSurfaceAsync` (`BitmapAlphaMode.Straight`) → +per-frame `FindContentBounds` bbox → `CropBounds` → epoch'd ring → dispatcher-coalesced crop copy +into ONE shared `WriteableBitmap`. -**User directives (this session):** -- Signed −500..+500 with positive=delay / negative=advance, relabel "AUDIO SYNC", default 0. -- **Lock the sync control when live or recording** (done — `IsEditMode`). -- **NOTE ONLY, no fix:** the live recording is completely different from the "recording - results" shown in Chat view (recorded in `bugs.md` — do not rediscover as a surprise). -- **Before 1.0:** write a detailed USER-DOC tutorial on the audio-sync feature (see TASKS.md - note; add it to the gold-pass/1.0 checklist). `docs/` currently holds only the README image. +**Files:** NEW `Services/WebCaptureFrameSource.cs`; reworked `Services/WebView2Manager.cs` (internal +seam ctor `(Dispatcher, Func?)` for hermetic tests); DELETED +`Services/CaptureScheduler.cs`; `MainViewModel.Web.cs` (`InitWebView2()` no-arg), +`MainViewModel.Streaming.Operations.cs` (three `SetCaptureInterval` hooks gone — verified zero +remaining), `MainWindow.xaml(.cs)` (`InitWebView2()` moved ctor→**Loaded**; `WebViewHostPanel` +overlay deleted). `TransparentBackgroundScript` unchanged. -## Take verification so far (ring-backlog fix) +**Tests:** reworked `ytLive.Tests/WebView2ManagerTests.cs` — dropped the 4 control-size tests + +`CaptureScheduler_Drops…`; `FindContentBounds` tests moved to `WebCaptureFrameSource`; the ONE +integration test (`Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds`) drives the +seam with `FakeWebSource` + background-STA `DispatcherPump`: crop-sized preview published once → +`CropBounds` + `IsOpaque=false` → back-to-back frames coalesce to latest. `RoundClipInteractionTests` +comment updated (WebViewHostPanel gone). Suite **293/293 green**, app build **0 warnings**. Docs +slant (ai.md Slice 14, HANDOFF, MyMistakes, TASK 17, Controls/ViewModels/Services indexes) updated in +the same working set. -Take `ty-20260914-1128-0000-2.mp4` (1920×1080@60fps, 356 frames / 5.93s): audio clap RMS peak -3.260s; video diff-frame 163 → 2.717s; **offset ≈ +0.54s** — collapsed from +2.11/2.22s; the -remaining ~300ms was the baked-in `SyncOffsetMs=300` (now 0). Both streams `start_time=0` — not -an `avoid_negative_ts` artifact. +## ⚠️ Open items on this change (before it is PUSHABLE) + +- **Device verification:** the composition path has never run against a real widget. Verify ~60fps + web animation in a take (also: the OBS de-throttle flags stay; hidden-page JS throttling is fixed + by them, capture pacing is now renderer-driven). +- **No push yet** — web work is commit-local until the user says push. + +## ✅ SHIPPED + PUSHED — signed audio sync, −500..+500 (`b22d08e`) + +Positive = delay the mix (audio AHEAD — the `AudioSyncDelay` line, live-reactive). Negative = +**advance** (audio BEHIND): OBS-style "eat the stream head" — `AudioMixer.StartLive` arms +`_advanceSamplesRemaining = |N| ms`, `LiveLoopAsync` skips `min(budget, mixBuffer.Length)` off each +write head while it lasts. Slider relabeled **AUDIO SYNC**, `−500..500`, locked while +live/recording (`IsEditMode`). One regression test: +`StartLive_NegativeOffset_AdvancesAudio_ByDroppingTheStreamHead`. + +Take `ty-20260914-1128-0000-2.mp4`: **offset ≈ +0.54s** (collapsed from +2.11/2.22s; residual was +the baked-in `SyncOffsetMs=300`, now 0 — confirmed via sqlite3). Both streams `start_time=0`. ## Open threads -- **Verify signed sync on device:** re-run the clap take with a NEGATIVE offset to confirm the - advance direction end-to-end (the regression test proves the mixer; a take proves the file). -- **Audio-silence verification** — fixed code (`724af14`) confirmed; creator heard real audio. +- **Verify signed sync negative direction on device** (clap take with a negative offset). +- **Audio-silence verification** — fixed (`724af14`); creator heard real audio. - **Webcam MJPG missing / ~10–14Hz**, layer SortOrder, truncation-with-dynamic-scenes — queued. -- **Sync control tutorial in user docs — REQUIRED before 1.0** (creator directive). +- **Sync control user-doc tutorial — REQUIRED before 1.0** (creator directive; TASK 22 file). +- **Device verify the composition web capture (this change).** ## Landmines - testhost shares startup.log — filter by time. - `cmd.exe /c "taskkill /F /IM ytLive.exe"` (WSL double-slashes mangle) before rebuilds. -- Build/tests: **Windows dotnet host** (`/mnt/c/Program Files/dotnet/dotnet.exe`). 0 warnings. +- Build/tests: **Windows dotnet host** (`/mnt/c/Program Files/dotnet/dotnet.exe`). 0 warnings — + only `./scripts/verify.sh ""`'s clean build counts. - ffmpeg/ffprobe: `/mnt/c/Program Files/Krita (x64)/bin/` with Windows paths. -- `MyMistakes.md` has the **audio/video sync measurement recipe** (claps + cross-correlation) - — grep it before re-deriving. +- `MyMistakes.md` has the **audio/video sync measurement recipe** AND the **composition-capture + CoreMessaging DQ recipe (2026-09-14)** — grep before re-deriving. - sqlite3 lives at `/home/gramps/android-sdk/platform-tools/sqlite3` (WSL) for the DB at `/mnt/c/Users/gramp/AppData/Roaming/ytLlive/ytLlive.db`. ## Next step -Open queue question: user-doc tutorial for the audio-sync feature (creator directive, REQUIRED -before 1.0 — recorded in `TASKS/task-22-audio-sync-offset.md`). Also asked 2026-09-14 whether -other root md files need the TASKS-style split — answered: no (see below for the criterion). \ No newline at end of file +From this dirty state: run `./scripts/verify.sh ""`, review the diff, commit +locally (web work — NO push, per standing directive). Then a device take to verify ~60fps web +animation, then present for push review. \ No newline at end of file diff --git a/MainWindow.xaml b/MainWindow.xaml index b847181..a958a58 100644 --- a/MainWindow.xaml +++ b/MainWindow.xaml @@ -96,10 +96,6 @@ - - - ?)` + a `FakeWebSource` + a real background-STA + `DispatcherPump` (borrowed from ScreenCaptureManagerTests) drives the whole session path hermetic. + When byte-comparing a crop, slice the source with stride gaps (helper `CropBytes`) — a contiguous + range silently spans rows. + +Recipe verified on green suite + 0-warning build; the composition path itself still needs a device +take (open item, HANDOFF). + ## A pipe-read test that samples one tick is a timing flake by construction (2026-09-14, re-discovered) `Mix_HonorsProviderGains_AndGameMute_KillsTheLoopback` fails diff --git a/Services/CaptureScheduler.cs b/Services/CaptureScheduler.cs deleted file mode 100644 index c307100..0000000 --- a/Services/CaptureScheduler.cs +++ /dev/null @@ -1,73 +0,0 @@ -using System; -using System.Threading.Tasks; -using System.Windows.Threading; - -namespace ytLive.Services; - -/// Per-session web capture pace: a DispatcherTimer that drops ticks while a -/// capture is in flight (latest-wins). Raising the naive 10Hz cadence to ~30Hz would -/// stack concurrent CapturePreviewAsync calls (a full 1920×1080 PNG encode costs -/// ~10-30ms), so overlapping ticks are dropped, never queued; the effective cadence -/// is max(interval, capture duration). The interval and the in-flight drop live here, -/// not in the manager, so the scheduler is unit-testable without a WebView2 runtime. -internal sealed class CaptureScheduler -{ - private readonly Dispatcher _dispatcher; - private readonly Func _capture; - private DispatcherTimer? _timer; - private int _intervalMs; - private bool _inFlight; - private Task _pending = Task.CompletedTask; - - public CaptureScheduler(Dispatcher dispatcher, int intervalMs, Func capture) - { - _dispatcher = dispatcher; - _intervalMs = Math.Max(16, intervalMs); - _capture = capture; - } - - /// The task of the capture currently in flight, or a completed task when - /// idle. Test seam: lets the caller await exactly the running capture. - internal Task Pending => _pending; - - public void Start() - { - if (_timer != null) return; - _timer = new DispatcherTimer( - TimeSpan.FromMilliseconds(_intervalMs), - DispatcherPriority.Background, - (_, _) => Tick(), - _dispatcher); - _timer.Start(); - } - - public void Stop() => _timer?.Stop(); - - public void SetInterval(int milliseconds) - { - _intervalMs = Math.Max(16, milliseconds); - if (_timer != null) _timer.Interval = TimeSpan.FromMilliseconds(_intervalMs); - } - - /// Runs the capture unless one is already in flight — the dropped tick is - /// the latest-wins policy. Tick runs on the dispatcher thread (timer + test), so - /// _inFlight needs no locking. - internal void Tick() - { - if (_inFlight) return; - _inFlight = true; - _pending = RunCapture(); - } - - private async Task RunCapture() - { - try - { - await _capture(); - } - finally - { - _inFlight = false; - } - } -} \ No newline at end of file diff --git a/Services/WebCaptureFrameSource.cs b/Services/WebCaptureFrameSource.cs new file mode 100644 index 0000000..32d0dbe --- /dev/null +++ b/Services/WebCaptureFrameSource.cs @@ -0,0 +1,232 @@ +using System; +using System.Runtime.InteropServices; +using System.Runtime.InteropServices.WindowsRuntime; +using System.Threading.Tasks; +using Windows.Graphics; +using Windows.Graphics.Capture; +using Windows.Graphics.DirectX; +using Windows.Graphics.DirectX.Direct3D11; +using Windows.Graphics.Imaging; +using ytLive.Helpers; + +namespace ytLive.Services; + +/// +/// The frame-driven web capture for one widget: a GraphicsCaptureItem backed by +/// the widget's Windows.UI.Composition visual (see WebView2Manager). Owns the +/// item, a free-threaded Direct3D11CaptureFramePool and the capture session; +/// frames arrive in step with Chromium's own presentation (~60Hz on a real +/// animation), replacing the old CapturePreviewAsync PNG poll whose ~35-165ms +/// encode cost capped the recording at ~10Hz and the preview at ~5fps. +/// +/// The capture surface holds a REAL alpha channel (the controller renders onto +/// a transparent composition visual), so the GPU→CPU copy uses +/// and the frame is +/// = false — the compositor's straight-alpha +/// blend math ("web layer transparency", 2026-09-13) requires the readback to +/// keep premultiplication OFF. Unlike the screen source there is no master-size +/// downscale: the controller renders at the RAW 1920×1080 master canvas, so the +/// CPU never holds a larger frame. +/// +/// Per-frame the alpha bounding box of the widget () +/// is measured on this capture worker thread and stamped onto the frame as +/// — same alpha-bbox contract as the old +/// PNG path, but off the UI thread (a 2M-pixel scan per frame there is exactly +/// the cost this redesign exists to remove). +/// +public sealed class WebCaptureFrameSource : IScreenCaptureSource +{ + private readonly object _gate = new(); + private GraphicsCaptureItem _item; + private Direct3D11CaptureFramePool? _framePool; + private GraphicsCaptureSession? _session; + private SizeInt32 _poolSize; + private bool _started; + private bool _framePending; + private DateTime _lastErrorLog = DateTime.MinValue; + + // Buffer recycling: identical to the screen source's ring+Epoch contract + // (see ScreenCaptureFrameSource) — the compositor's paste cache is keyed on + // array identity, so recycled buffers must carry a monotonic Epoch or a + // reused array false-hits with stale content. + private readonly byte[]?[] _frameRing = new byte[8][]; + private int _ringNext; + private long _epoch; + + private static readonly TimeSpan ErrorLogThrottle = TimeSpan.FromSeconds(5); + + public string Key { get; } + public event Action? FrameAvailable; + + internal WebCaptureFrameSource(string key, GraphicsCaptureItem item) + { + Key = key; + _item = item; + } + + /// Wraps an item created from a composition visual by the web manager. + public static WebCaptureFrameSource CreateForVisual(string key, GraphicsCaptureItem item) + => new(key, item); + + public Task StartAsync() + { + lock (_gate) + { + if (_started) return Task.CompletedTask; + var item = _item; + var device = Direct3D11Helper.CreateDevice(); + var framePool = Direct3D11CaptureFramePool.CreateFreeThreaded( + device, DirectXPixelFormat.B8G8R8A8UIntNormalized, 2, item.Size); + _poolSize = item.Size; + var session = framePool.CreateCaptureSession(item); + framePool.FrameArrived += OnFrameArrived; + session.StartCapture(); + _framePool = framePool; + _session = session; + _started = true; + } + return Task.CompletedTask; + } + + public Task StopAsync() + { + lock (_gate) + { + _started = false; + _framePending = false; + if (_framePool != null) + _framePool.FrameArrived -= OnFrameArrived; + _session?.Dispose(); + _session = null; + _framePool?.Dispose(); + _framePool = null; + } + return Task.CompletedTask; + } + + private void OnFrameArrived(Direct3D11CaptureFramePool sender, object args) + { + var frame = sender.TryGetNextFrame(); + if (frame == null) return; + lock (_gate) + { + if (!_started) + { + frame.Dispose(); + return; + } + } + + if (frame.ContentSize.Width != _poolSize.Width || frame.ContentSize.Height != _poolSize.Height) + { + sender.Recreate(Direct3D11Helper.CreateDevice(), + DirectXPixelFormat.B8G8R8A8UIntNormalized, 2, frame.ContentSize); + _poolSize = frame.ContentSize; + } + + if (_framePending) + { + frame.Dispose(); + return; + } + _framePending = true; + _ = ProcessFrameAsync(frame); + } + + private async Task ProcessFrameAsync(Direct3D11CaptureFrame frame) + { + try + { + using (frame) + using (var softwareBitmap = await SoftwareBitmap.CreateCopyFromSurfaceAsync( + frame.Surface, BitmapAlphaMode.Straight)) + { + FrameAvailable?.Invoke(CopyToVideoFrame(softwareBitmap)); + } + } + catch (Exception ex) + { + var now = DateTime.UtcNow; + if (now - _lastErrorLog >= ErrorLogThrottle) + { + _lastErrorLog = now; + AppLog.Write($"WebCaptureFrameSource: frame conversion failed: {ex.Message}"); + } + } + finally + { + _framePending = false; + } + } + + private byte[] RentRingBuffer(int size) + { + for (var tries = 0; tries < _frameRing.Length; tries++) + { + var idx = (_ringNext + tries) % _frameRing.Length; + var buf = _frameRing[idx]; + if (buf is { Length: var len } && len == size) + { + _ringNext = (idx + 1) % _frameRing.Length; + return buf; + } + } + var slot = _ringNext; + _ringNext = (slot + 1) % _frameRing.Length; + var fresh = new byte[size]; + _frameRing[slot] = fresh; + return fresh; + } + + private VideoFrame CopyToVideoFrame(SoftwareBitmap bitmap) + { + var sw = bitmap.PixelWidth; + var sh = bitmap.PixelHeight; + using var buffer = bitmap.LockBuffer(BitmapBufferAccessMode.Read); + using var reference = buffer.CreateReference(); + if (!WindowsRuntimeMarshal.TryGetDataUnsafe(reference, out var data, out var capacity)) + throw new InvalidOperationException("Could not access the frame buffer."); + + var srcStride = sw * 4; + var count = (int)Math.Min(capacity, (uint)(sh * srcStride)); + var pixels = RentRingBuffer(count); + Marshal.Copy(data, pixels, 0, pixels.Length); + + var (cropX, cropY, cropW, cropH) = FindContentBounds(pixels, sw, sh, srcStride); + return new VideoFrame(sw, sh, pixels) + { + IsOpaque = false, + Epoch = ++_epoch, + CropBounds = (cropX, cropY, cropW, cropH), + }; + } + + // The widget's extent is measured from the captured frame itself: the + // bounding box of non-transparent pixels (the background is injected + // transparent, so the alpha channel IS the widget). Immune to DOM timing + // (fonts/images/animations), needs no JS round-trip, and CANNOT break the + // render: a widget that fills the canvas gets the full frame; a widget that + // floats in transparent margin gets exactly its content box — used for the + // preview bitmap / selection. The frame handed to the compositor is always + // the FULL canvas, so the page's transparent margins keep revealing the + // layers beneath (the "black square over webcam" was the crop fed to the + // compositor: UniformToFill zoomed the opaque content box over the element). + internal static (int X, int Y, int W, int H) FindContentBounds(byte[] pixels, int pixW, int pixH, int stride) + { + int minX = pixW, minY = pixH, maxX = -1, maxY = -1; + for (var y = 0; y < pixH; y++) + { + var row = y * stride; + for (var x = 0; x < pixW; x++) + { + if (pixels[row + x * 4 + 3] == 0) continue; + if (x < minX) minX = x; + if (x > maxX) maxX = x; + if (y < minY) minY = y; + if (y > maxY) maxY = y; + } + } + if (maxX < minX || maxY < minY) return (0, 0, pixW, pixH); + return (minX, minY, maxX - minX + 1, maxY - minY + 1); + } +} \ No newline at end of file diff --git a/Services/WebView2Manager.cs b/Services/WebView2Manager.cs index 5a06fb3..eebc62b 100644 --- a/Services/WebView2Manager.cs +++ b/Services/WebView2Manager.cs @@ -1,129 +1,92 @@ using System; -using System.Diagnostics; +using System.Collections.Generic; using System.Drawing; -using System.IO; -using System.Text.Json; +using System.Runtime.InteropServices; +using System.Threading.Tasks; using System.Windows; -using System.Windows.Controls; using System.Windows.Media; using System.Windows.Media.Imaging; using System.Windows.Threading; using Microsoft.Web.WebView2.Core; -using Microsoft.Web.WebView2.Wpf; +using Windows.Graphics.Capture; +using Windows.System; +using Windows.UI.Composition; using ytLive.Helpers; using ytLive.Models; namespace ytLive.Services; +/// +/// Owns the composition-based web sources (one per widget). A session renders +/// its widget through a whose +/// visual tree lives off-screen on the WPF UI thread (OBS web-source model: +/// the page renders at the MASTER CANVAS size 1920×1080, stable, never tracked); +/// the controller's root visual is captured by a +/// at Chromium's own presentation cadence. The preview is crop-copied to a shared +/// WriteableBitmap on the UI thread (dispatcher-coalesced latest-wins, mirror of +/// ScreenCaptureManager); the FramePump reads . +/// +/// The composition stack needs a DispatcherQueue on this thread (CsWinRT does not +/// project CreateOnCurrentThread on the 19041 SDK — only the dedicated +/// variant — so it is installed via a CoreMessaging P/Invoke, the documented +/// fallback the WebView2CompositionControl / webview_windows embedders use). +/// Everything composition-side therefore runs on the WPF UI thread; only the +/// frame conversion and alpha-bbox scan run on the capture worker. +/// +/// Seam: an internal Func<string, IScreenCaptureSource> source +/// factory substitutes the whole WebView2+item creation in tests — the manager +/// then owns the session/coalesce/crop/preview plumbing exactly as in production. +/// public sealed class WebView2Manager : IDisposable { - private readonly Panel _hostPanel; private readonly Dispatcher _dispatcher; + private readonly Func? _sessionSourceFactory; private readonly Dictionary _sessions = new(); private Task? _environmentTask; - private int _captureIntervalMs = 200; // idle pace; recording drops it to ~30Hz via SetCaptureInterval + private DispatcherQueueController? _dispatcherQueueController; + private Windows.UI.Composition.Compositor? _compositor; public event Action? PreviewBitmapChanged; private sealed class WebSourceSession { - public required WebView2 Control { get; init; } - public required CaptureScheduler Scheduler { get; init; } + public required string Key { get; init; } + public CoreWebView2? Web; + public CoreWebView2CompositionController? Controller; + public IScreenCaptureSource? Source; + public Action? FrameHandler; public VideoFrame? LatestFrame; + public WriteableBitmap? PreviewBitmap; + + // Buffer reuse (2026-09-04 lesson): the preview crop copy reuses one + // scratch array while the widget's extent holds — no per-frame LOH mint. + public byte[]? CropScratch; public bool Disposed; public bool Initialized; - - // Buffer reuse (2026-09-04): every capture tick used to mint a fresh - // 8.3MB canvas array + a fresh crop array (~83MB/s of LOH at 10Hz → - // gen2 pauses surfaced as the pump's 35-40ms "worst render" spikes). - // CanvasScratch is reused while the size holds. The frame handed to the - // compositor is always the FULL canvas (transparent margins reveal layers - // beneath); the crop is only for the preview WriteableBitmap. Epoch is - // monotonic per canvas generation (identity-keyed consumers see each frame). - public byte[]? CanvasScratch; - public readonly byte[]?[] OutRing = new byte[8][]; - public int OutRingNext; - public long Epoch; - public WriteableBitmap? PreviewBitmap; - public bool DebugPngWritten; - public int WidgetDumpRemaining; - public int TelemetryCaptureCount; - - public byte[] RentOutBuffer(int size) - { - for (var tries = 0; tries < OutRing.Length; tries++) - { - var idx = (OutRingNext + tries) % OutRing.Length; - var buf = OutRing[idx]; - if (buf != null && buf.Length == size) - { - OutRingNext = (idx + 1) % OutRing.Length; - return buf; - } - } - var slot = OutRingNext; - OutRingNext = (slot + 1) % OutRing.Length; - var fresh = new byte[size]; - OutRing[slot] = fresh; - return fresh; - } + public bool FramePending; + public bool FirstCaptureLogged; + public int WidgetLogsRemaining; } - public WebView2Manager(Panel hostPanel, Dispatcher dispatcher) + public WebView2Manager(Dispatcher dispatcher) + : this(dispatcher, null) + { + } + + /// Test seam: the source factory replaces WebView2+composition + /// creation, so the session/coalesce/crop/preview plumbing is testable + /// without the runtime. + internal WebView2Manager(Dispatcher dispatcher, Func? sessionSourceFactory) { - _hostPanel = hostPanel; _dispatcher = dispatcher; + _sessionSourceFactory = sessionSourceFactory; } public void Register(Source source) { if (_sessions.ContainsKey(source.Id)) return; - - // OBS model: the page renders at the MASTER CANVAS size (1920×1080), - // stable, never tracked. Full-bleed widgets (designed for the canvas) - // render fully instead of being clipped to a small box-sized viewport; - // dragging the box never reflows the page. The capture is tightly - // cropped to the widget's ALPHA bounding box (FindContentBounds) and - // the display (Stretch=Fill) maps it flush under the box, anchored at - // (0,0) — no DOM query, so no timing fragility; worst case is the - // full frame, which is the confirmed-good rendering. - var webView = new WebView2 - { - Width = 1920, - Height = 1080, - DefaultBackgroundColor = System.Drawing.Color.Transparent, - Visibility = System.Windows.Visibility.Visible, - }; - Canvas.SetLeft(webView, -5000); - Canvas.SetTop(webView, -5000); - - _hostPanel.Children.Add(webView); - - var scheduler = new CaptureScheduler( - _dispatcher, - _captureIntervalMs, - () => CaptureFrame(source.Id)); - - var session = new WebSourceSession - { - Control = webView, - Scheduler = scheduler, - }; - - _sessions[source.Id] = session; - - _ = InitializeAsync(source, session); - } - - /// Changes the capture cadence for every session. Recording runs at - /// ~30Hz (33ms) — the practical PNG-capture ceiling — idle drops back to 200ms so - /// the preview alone doesn't burn a core decoding full-HD captures. - public void SetCaptureInterval(int milliseconds) - { - _captureIntervalMs = milliseconds; - foreach (var session in _sessions.Values) - if (!session.Disposed) - session.Scheduler.SetInterval(milliseconds); + _sessions[source.Id] = new WebSourceSession { Key = source.Id }; + _ = InitializeAsync(source, _sessions[source.Id]); } public void Unregister(string sourceId) => RemoveSession(sourceId); @@ -136,7 +99,7 @@ public sealed class WebView2Manager : IDisposable if (string.IsNullOrEmpty(source.WebUri)) { session.LatestFrame = null; - session.Control.CoreWebView2?.NavigateToString( + session.Web?.NavigateToString( ""); return; } @@ -152,9 +115,8 @@ public sealed class WebView2Manager : IDisposable // Injected BEFORE the document is parsed (AddScriptToExecuteOnDocumentCreatedAsync) so // the widget page's own CSS/scripts cannot repaint html/body opaque — the reason the // late NavigationStarting/NavigationCompleted ExecuteScriptAsync injection lost the - // fight (page styles ran first). CapturePreviewAsync "will always honor a webpage's - // background content" (MicrosoftEdge/WebView2Feedback specs/BackgroundColor.md), so a - // transparent capture REQUIRES the transparency to be in place before the page paints. + // fight (page styles ran first). A transparent capture REQUIRES the transparency in + // place before the page paints, on the composition surface too. // // The OLD inline `element.style.background='transparent'` form lost to any page CSS: // a widget that paints a background-COLOR on a container (or html/body with stronger @@ -191,55 +153,75 @@ public sealed class WebView2Manager : IDisposable { try { - await session.Control.EnsureCoreWebView2Async(await GetEnvironmentAsync()); - - var cws = session.Control.CoreWebView2!; - - // Pre-page hook: runs after the global object exists but BEFORE the document - // parses / any page script runs — must complete before navigating away from - // about:blank for it to apply to the widget document. - await cws.AddScriptToExecuteOnDocumentCreatedAsync(TransparentBackgroundScript); - - cws.NavigationStarting += (_, _) => + IScreenCaptureSource sourceImpl; + if (_sessionSourceFactory != null) { - _ = cws.ExecuteScriptAsync(TransparentBackgroundScript); - }; - - cws.NavigationCompleted += (_, e) => + // Test path: the factory hands over a fully-configured source; the + // branch has no awaits, so Register() completes the session inline. + sourceImpl = _sessionSourceFactory(source.Id); + } + else { - _ = cws.ExecuteScriptAsync(TransparentBackgroundScript); - if (e.IsSuccess && session.Control.CoreWebView2?.Source is { } src && src != "about:blank") + var controller = await CreateControllerAsync(); + if (controller == null) { - // Arm a one-shot diagnostic dump of the REAL widget document - // (a few post-paint captures so styles/backgrounds settle) — - // the old first-capture dump only ever saw the about:blank - // document, so whether this widget's capture actually HAS the - // transparent margins the injection promises has never been - // verified (MyMistakes: "? VERIFY"). - session.WidgetDumpRemaining = 5; - // The first five dumps fire within ~1s of completion (usually BEFORE the - // recording starts) — re-arm ~30s in so the same real document gets dumped - // INSIDE a recording's window too. Next take therefore nukes the gap that - // hid what the page actually painted while recording. - var rearm = session; - _ = Task.Run(async () => - { - await Task.Delay(TimeSpan.FromSeconds(30)); - rearm.WidgetDumpRemaining = 5; - }); + RemoveSession(source.Id); + return; } - _ = CaptureFrame(source.Id); - }; + session.Controller = controller; + session.Web = controller.CoreWebView2; + if (session.Web == null) + { + RemoveSession(source.Id); + return; + } + + // Pre-page hook: runs after the global object exists but BEFORE the document + // parses / any page script runs — must complete before navigating away from + // about:blank for it to apply to the widget document. + await session.Web.AddScriptToExecuteOnDocumentCreatedAsync(TransparentBackgroundScript); + + session.Web.NavigationStarting += (_, _) => + { + _ = session.Web?.ExecuteScriptAsync(TransparentBackgroundScript); + }; + + session.Web.NavigationCompleted += (_, e) => + { + _ = session.Web?.ExecuteScriptAsync(TransparentBackgroundScript); + if (e.IsSuccess && session.Web?.Source is { } src && src != "about:blank") + { + // Log-only diagnostic on the REAL widget document (a few post-paint + // frames so styles/backgrounds settle). The first-capture dump only + // ever saw about:blank; the old PNG widget dumps are gone with the + // PNG path — alpha stats now ride the frame-driven channel. + session.WidgetLogsRemaining = 3; + var rearm = session; + _ = Task.Run(async () => + { + await Task.Delay(TimeSpan.FromSeconds(30)); + if (!rearm.Disposed) rearm.WidgetLogsRemaining = 3; + }); + } + }; + + session.Web.NavigateToString( + ""); + + if (!string.IsNullOrEmpty(source.WebUri)) + Navigate(source.WebUri, session); + + session.Initialized = true; + + sourceImpl = new WebCaptureFrameSource(source.Id, CreateCaptureItem(controller)); + } + + session.Source = sourceImpl; + session.FrameHandler = frame => OnFrameAvailable(session, frame); + sourceImpl.FrameAvailable += session.FrameHandler; + await sourceImpl.StartAsync(); session.Initialized = true; - - session.Control.CoreWebView2.NavigateToString( - ""); - - if (!string.IsNullOrEmpty(source.WebUri)) - Navigate(source.WebUri, session); - - session.Scheduler.Start(); } catch (Exception ex) { @@ -248,14 +230,13 @@ public sealed class WebView2Manager : IDisposable } } - // The widget page is rendered off-screen; Chromium throttles hidden pages - // (rAF parked, timers clamped to ~1s) the moment the app window is unfocused or - // covered — WebView2Feedback#1172/#3070, Chrome 88 timer throttling. These flags - // (the Electron-class embedder answer) keep the renderer at full rate: - // --disable-backgrounding-occluded-windows, --disable-renderer-backgrounding, - // and disable the native-window occlusion detector that marks the page hidden. - // The environment MUST exist before EnsureCoreWebView2Async — it's created once - // and shared by every session (one browser process). + // The environment MUST exist before the controller is created — it's created once + // and shared by every session (one browser process). These flags (the + // Electron-class embedder answer to Chromium throttling hidden pages — the moment + // the app window is unfocused or covered, rAF parks and timers clamp to ~1s) keep + // the renderer at full rate: --disable-backgrounding-occluded-windows, + // --disable-renderer-backgrounding, and disable the native-window occlusion + // detector that marks the page hidden. private Task GetEnvironmentAsync() { if (_environmentTask != null) return _environmentTask; @@ -267,45 +248,177 @@ public sealed class WebView2Manager : IDisposable return _environmentTask = CoreWebView2Environment.CreateAsync(null, null, options); } + /// Creates the composition controller off-screen under the main + /// window (the parent HWND is required, the visual never mounts a window + /// target). Set up is copied from the WebView2CompositionControl / + /// webview_windows embedders: raw-pixel bounds at the master canvas, + /// fixed rasterization scale, transparent default background. + private async Task CreateControllerAsync() + { + var mainWindow = Application.Current?.MainWindow; + var hwnd = mainWindow == null + ? IntPtr.Zero + : new System.Windows.Interop.WindowInteropHelper(mainWindow).Handle; + if (hwnd == IntPtr.Zero) + { + AppLog.Write("WebView2Manager: main window handle unavailable — web source not created"); + return null; + } + + var env = await GetEnvironmentAsync(); + if (_compositor == null) _compositor = CreateCompositorForCurrentThread(); + + var controller = await env.CreateCoreWebView2CompositionControllerAsync(hwnd); + controller.Bounds = new Rectangle(0, 0, 1920, 1080); + controller.BoundsMode = CoreWebView2BoundsMode.UseRawPixels; + controller.ShouldDetectMonitorScaleChanges = false; + controller.RasterizationScale = 1.0; + controller.IsVisible = true; + controller.DefaultBackgroundColor = System.Drawing.Color.Transparent; + return controller; + } + private void Navigate(string url, WebSourceSession session) { - try { session.Control.CoreWebView2?.Navigate(url); } + try { session.Web?.Navigate(url); } catch (Exception ex) { AppLog.Write($"WebView2Manager: navigate failed: {ex.Message}"); } } - // The widget's extent is measured from the captured frame itself: the - // bounding box of non-transparent pixels (the background is injected - // transparent, so the alpha channel IS the widget). This is immune to - // DOM timing (fonts/images/animations), needs no JS round-trip, and - // CANNOT break the render: if the widget fills the canvas the box equals - // the full frame; if it floats in transparent margin the box is the - // measured content extent — used ONLY for the preview bitmap / selection. - // The frame handed to the compositor is always the FULL canvas, so the - // page's transparent margins keep revealing the layers beneath (the - // "black square over webcam" was the crop fed to the compositor: - // UniformToFill zoomed the opaque content box to cover the whole element). - internal static (int X, int Y, int W, int H) FindContentBounds(byte[] pixels, int pixW, int pixH, int stride) + // The visual tree webview_windows wires (offscreen mode): a root container + // visual sized to the master canvas (the captured subject — the Flutter + // plugin calls GraphicsCaptureItem.CreateFromVisual on this root), with the + // controller's root visual target set to a child that fills the root via + // RelativeSizeAdjustment. BoundsMode raw pixels + RasterizationScale 1.0 keep + // the capture at exactly 1920×1080 regardless of monitor scale. + private GraphicsCaptureItem CreateCaptureItem(CoreWebView2CompositionController controller) { - int minX = pixW, minY = pixH, maxX = -1, maxY = -1; - for (var y = 0; y < pixH; y++) - { - var row = y * stride; - for (var x = 0; x < pixW; x++) - { - if (pixels[row + x * 4 + 3] == 0) continue; - if (x < minX) minX = x; - if (x > maxX) maxX = x; - if (y < minY) minY = y; - if (y > maxY) maxY = y; - } - } - if (maxX < minX || maxY < minY) return (0, 0, pixW, pixH); - return (minX, minY, maxX - minX + 1, maxY - minY + 1); + var compositor = _compositor!; + var root = compositor.CreateContainerVisual(); + root.Size = new System.Numerics.Vector2(1920f, 1080f); + root.IsVisible = true; + + var webViewVisual = compositor.CreateContainerVisual(); + webViewVisual.RelativeSizeAdjustment = new System.Numerics.Vector2(1f, 1f); + root.Children.InsertAtTop(webViewVisual); + + controller.RootVisualTarget = webViewVisual; + return GraphicsCaptureItem.CreateFromVisual(root); + } + + // CsWinRT does not project DispatcherQueueController.CreateOnCurrentThread + // on the 19041 SDK (only CreateOnDedicatedThread [CS0117, verified by probe]); + // the documented fallback — "how to create a DispatcherQueue on the current + // thread" (learn.microsoft.com CoreMessaging docs) — is CreateDispatcherQueueController + // from CoreMessaging.dll, wrapped via FromAbi (the CaptureInterop pattern). One + // per UI thread, then new Compositor() (which requires the DQ) works here. + private Windows.UI.Composition.Compositor CreateCompositorForCurrentThread() + { + var options = new DispatcherQueueOptions + { + DwSize = Marshal.SizeOf(), + ThreadType = 2, // DQTYPE_THREAD_CURRENT + ApartmentType = 2, // DQTAT_COM_STA — the WPF UI thread is STA + }; + var hr = CreateDispatcherQueueController(options, out var controllerPtr); + if (hr != 0) + throw Marshal.GetExceptionForHR(hr)!; + _dispatcherQueueController = DispatcherQueueController.FromAbi(controllerPtr); + return new Windows.UI.Composition.Compositor(); + } + + [StructLayout(LayoutKind.Sequential)] + private struct DispatcherQueueOptions + { + public int DwSize; + public int ThreadType; + public int ApartmentType; + } + + [DllImport("CoreMessaging.dll")] + private static extern int CreateDispatcherQueueController( + DispatcherQueueOptions options, out IntPtr dispatcherQueueController); + + private void OnFrameAvailable(WebSourceSession session, VideoFrame frame) + { + if (session.Disposed) return; + session.LatestFrame = frame; + + if (_dispatcher.CheckAccess()) + { + UpdatePreview(session); + return; + } + if (session.FramePending) return; + session.FramePending = true; + _dispatcher.BeginInvoke(() => + { + session.FramePending = false; + if (!session.Disposed) + UpdatePreview(session); + }, DispatcherPriority.Render); + } + + private void UpdatePreview(WebSourceSession session) + { + var frame = session.LatestFrame; + if (frame == null) return; + + var (cropX, cropY, cropW, cropH) = frame.CropBounds ?? (0, 0, frame.Width, frame.Height); + if (cropW < 1 || cropH < 1) + { + cropX = 0; cropY = 0; + cropW = frame.Width; cropH = frame.Height; + } + + // The preview is sized to the widget's alpha bounding box (the compositor + // gets the FULL canvas + CropBounds; the crop is only for the preview). + // A fresh box (widget resizes / first content paints) mints a new bitmap + // once — WPF re-renders a reused mutable bitmap after WritePixels, so + // later frames only re-copy into the same one. + if (session.PreviewBitmap == null + || session.PreviewBitmap.PixelWidth != cropW + || session.PreviewBitmap.PixelHeight != cropH) + { + session.PreviewBitmap = new WriteableBitmap(cropW, cropH, 96, 96, PixelFormats.Bgra32, null); + PreviewBitmapChanged?.Invoke(session.Key, session.PreviewBitmap); + } + + var need = cropW * cropH * 4; + var scratch = session.CropScratch is { } s && s.Length >= need + ? s + : (session.CropScratch = new byte[need]); + var stride = frame.Stride; + if (cropW == frame.Width && cropH == frame.Height && cropX == 0 && cropY == 0) + Buffer.BlockCopy(frame.BgraPixels, 0, scratch, 0, need); + else + for (var r = 0; r < cropH; r++) + Buffer.BlockCopy(frame.BgraPixels, (cropY + r) * stride + cropX * 4, scratch, r * cropW * 4, cropW * 4); + + session.PreviewBitmap.WritePixels(new Int32Rect(0, 0, cropW, cropH), scratch, cropW * 4, 0); + + // alpha[min/max/mean/zero%] — the one number that decides whether a widget's + // capture really has transparent margins (0% = opaque, ~100% if the page + // painted transparent-only). Sampled every 16th pixel like the original log. + if (!session.FirstCaptureLogged) + { + session.FirstCaptureLogged = true; + AppLog.Write( + $"WebView2Manager: first capture for '{session.Key}' {frame.Width}x{frame.Height} " + + $"{AlphaStats(frame.BgraPixels, frame.Width, frame.Height, stride)} " + + $"contentBounds=({cropX},{cropY},{cropW},{cropH})"); + } + + if (session.WidgetLogsRemaining > 0) + { + var dumpNo = 3 - session.WidgetLogsRemaining + 1; + session.WidgetLogsRemaining--; + AppLog.Write( + $"WebView2Manager: widget capture [{dumpNo}/3] for '{session.Key}' {frame.Width}x{frame.Height} " + + $"{AlphaStats(frame.BgraPixels, frame.Width, frame.Height, stride)} " + + $"contentBounds=({cropX},{cropY},{cropW},{cropH})"); + } } - // alpha[min/max/mean/zero%] — the one number that decides whether a widget's - // capture really has transparent margins (0% = opaque, ~100% if the page - // painted transparent-only). Sampled every 16th pixel like the original log. private static string AlphaStats(byte[] pixels, int pixW, int pixH, int stride) { int minA = 255, maxA = 0; @@ -326,147 +439,32 @@ public sealed class WebView2Manager : IDisposable return $"alpha[min={minA},max={maxA},mean={sumA / (double)nA:F1},zero={100.0 * zeroA / nA:F1}%]"; } - private async Task CaptureFrame(string sourceId) - { - if (!_sessions.TryGetValue(sourceId, out var session) || session.Disposed) return; - - var sw = Stopwatch.StartNew(); - try - { - var webView = session.Control; - if (webView.CoreWebView2 == null) return; - - using var ms = new MemoryStream(); - await webView.CoreWebView2.CapturePreviewAsync( - CoreWebView2CapturePreviewImageFormat.Png, ms); - - if (!session.DebugPngWritten) - { - session.DebugPngWritten = true; - try - { - ms.Position = 0; - var dbg = Path.Combine(Path.GetTempPath(), $"ytLive-web-{sourceId}.png"); - File.WriteAllBytes(dbg, ms.ToArray()); - AppLog.Write($"WebView2Manager: debug capture dump -> {dbg}"); - } - catch (Exception dex) { AppLog.Write($"WebView2Manager: debug dump failed: {dex.Message}"); } - } - - ms.Position = 0; - - var bitmap = new BitmapImage(); - bitmap.BeginInit(); - bitmap.StreamSource = ms; - bitmap.CacheOption = BitmapCacheOption.OnLoad; - bitmap.EndInit(); - bitmap.Freeze(); - - var formatted = new FormatConvertedBitmap(bitmap, - PixelFormats.Bgra32, null, 0); - - var pixW = formatted.PixelWidth; - var pixH = formatted.PixelHeight; - - if (pixW < 1 || pixH < 1) return; - - var need = pixW * pixH * 4; - var pixels = session.CanvasScratch is { } scratch && scratch.Length >= need - ? scratch - : (session.CanvasScratch = new byte[need]); - formatted.CopyPixels(pixels, pixW * 4, 0); - - var stride = pixW * 4; - - var (cropX, cropY, cropW, cropH) = FindContentBounds(pixels, pixW, pixH, stride); - - if (session.Epoch == 0) - { - AppLog.Write( - $"WebView2Manager: first capture for '{sourceId}' {pixW}x{pixH} " + - $"{AlphaStats(pixels, pixW, pixH, stride)} " + - $"contentBounds=({cropX},{cropY},{cropW},{cropH})"); - } - - if (session.WidgetDumpRemaining > 0) - { - var dumpNo = 5 - session.WidgetDumpRemaining + 1; - session.WidgetDumpRemaining--; - try - { - var dbg = Path.Combine(Path.GetTempPath(), $"ytLive-web-{sourceId}-w{dumpNo}.png"); - File.WriteAllBytes(dbg, ms.ToArray()); - AppLog.Write( - $"WebView2Manager: widget capture [{dumpNo}/5] for '{sourceId}' {pixW}x{pixH} " + - $"{AlphaStats(pixels, pixW, pixH, stride)} " + - $"contentBounds=({cropX},{cropY},{cropW},{cropH}) -> {dbg}"); - } - catch (Exception dex) - { - AppLog.Write($"WebView2Manager: widget dump failed: {dex.Message}"); - } - } - - var outPixels = session.RentOutBuffer(cropW * cropH * 4); - if (cropW == pixW && cropH == pixH && cropX == 0 && cropY == 0) - Buffer.BlockCopy(pixels, 0, outPixels, 0, outPixels.Length); - else - for (var r = 0; r < cropH; r++) - Buffer.BlockCopy(pixels, (cropY + r) * stride + cropX * 4, outPixels, r * cropW * 4, cropW * 4); - - if (session.PreviewBitmap == null - || session.PreviewBitmap.PixelWidth != cropW || session.PreviewBitmap.PixelHeight != cropH) - session.PreviewBitmap = new WriteableBitmap(cropW, cropH, 96, 96, PixelFormats.Bgra32, null); - session.PreviewBitmap.WritePixels(new Int32Rect(0, 0, cropW, cropH), outPixels, cropW * 4, 0); - - session.LatestFrame = new VideoFrame(pixW, pixH, pixels) - { - Epoch = ++session.Epoch, - CropBounds = (cropX, cropY, cropW, cropH), - }; - - // The preview bitmap is now MUTABLE and reused (CameraManager precedent): - // WPF re-renders it after WritePixels; handlers must not Freeze it. - var preview = session.PreviewBitmap; - _ = _dispatcher.BeginInvoke(() => - { - PreviewBitmapChanged?.Invoke(sourceId, preview); - }); - - // Capture-cost telemetry for the first ~30 captures of a session: PNG encode - // (CapturePreviewAsync) + decode cost decide whether ~30Hz recording cadence - // is affordable or must drop to ~20Hz. The FramePump drops frames (never - // time-lapses) if the UI thread's GC churn starves it, so startup.log shows - // the real budget before anything is tuned further. - if (session.TelemetryCaptureCount < 30) - { - session.TelemetryCaptureCount++; - AppLog.Write( - $"WebView2Manager: capture #{session.TelemetryCaptureCount} for '{sourceId}' " + - $"{pixW}x{pixH} took {sw.ElapsedMilliseconds}ms"); - } - } - catch (ObjectDisposedException) { } - catch (InvalidOperationException) { } - catch (Exception ex) - { - AppLog.Write($"WebView2Manager: capture failed for '{sourceId}': {ex.Message}"); - } - } - private void RemoveSession(string sourceId) { if (!_sessions.TryGetValue(sourceId, out var session)) return; session.Disposed = true; - session.Scheduler.Stop(); - _hostPanel.Children.Remove(session.Control); - try { session.Control.Dispose(); } catch { } + + var src = session.Source; + if (src != null) + { + if (session.FrameHandler != null) + src.FrameAvailable -= session.FrameHandler; + _ = SafeStopAsync(src); + } + try { session.Controller?.Close(); } catch { } + session.Web = null; _sessions.Remove(sourceId); } + private static async Task SafeStopAsync(IScreenCaptureSource source) + { + try { await source.StopAsync(); } + catch (Exception ex) { AppLog.Write($"WebView2Manager: stopping web capture failed: {ex.Message}"); } + } + public void Dispose() { foreach (var id in _sessions.Keys.ToList()) RemoveSession(id); } -} +} \ No newline at end of file diff --git a/Services/index.md b/Services/index.md index ea12128..fa56466 100644 --- a/Services/index.md +++ b/Services/index.md @@ -28,6 +28,8 @@ External-facing logic: YouTube API, persistence. See | `ScreenCaptureFrameSource.cs` | One `Direct3D11CaptureFramePool` (free-threaded, 2 buffers) + session per target; frames → `SoftwareBitmap.CreateCopyFromSurfaceAsync` (BGRA, alpha ignored) → `VideoFrame`, bytes read via `WindowsRuntimeMarshal.TryGetDataUnsafe` (CsWinRT-safe — the `IMemoryBufferByteAccess` ComImport cast fails on every frame and is gone). Surfaces > 1920×1080 downscaled bilinearly to the master; conversion failures logged ≤ once/5 s. DRM content = black frames (OS limit). `CreateForMonitor`/`CreateForWindow`/`CreateForPicker` | | `ScreenCaptureManager.cs` | Screen-capture ownership mirroring `CameraManager`: refcounted by target key, one shared `WriteableBitmap`, dispatcher-coalesced latest-frame copies; `PreviewBitmapChanged`/`CaptureFailed` events; `ReleaseAllAsync` used on re-designation | | `ScreenCaptureSourceFactory.cs` | `Resolve(key)` parses `monitor:` / `window:` / `picker:` into a source; `PickAsync()` shows the OS `GraphicsCapturePicker` and returns the `picker:` key (transient — a reload falls back to auto-detection) | +| `WebView2Manager.cs` | **Web-source ownership (TASK 17, composition-capture since 2026-09-14):** per registered `Source { Type: WebSource }` it owns a `CoreWebView2CompositionController` + its visual tree (`RootVisualTarget` child under a 1920×1080 root `ContainerVisual` on one `Windows.UI.Composition.Compositor` created on the UI thread via the CoreMessaging `CreateDispatcherQueueController` P/Invoke), a wrapped `WebCaptureFrameSource`, and the shared crop-sized `WriteableBitmap` preview (dispatcher-coalesced copy at Render priority, `PreviewBitmapChanged` raised only on recreation). Nav + `TransparentBackgroundScript` injection unchanged. Public ctor `(Dispatcher)`; internal seam ctor `(Dispatcher, Func?)` for hermetic tests (no WinRT). Caller: `MainViewModel.InitWebView2()` (now invoked from `MainWindow_Loaded` — the parent HWND must exist). See `../ai.md` "Slice 14" | +| `WebCaptureFrameSource.cs` | The `IScreenCaptureSource` for a web session: owns the `GraphicsCaptureItem` (`CreateFromVisual`) + a free-threaded `Direct3D11CaptureFramePool` (2 buffers) + session; readback via `SoftwareBitmap.CreateCopyFromSurfaceAsync` (BGRA, **`BitmapAlphaMode.Straight`** — the compositor blends straight alpha); per-frame `FindContentBounds` alpha-bbox (internal static, unit-tested) stamps `VideoFrame.CropBounds`; ring up to 2 fresh frames + Epoch reuse; `IsOpaque=false`. Mirror of `ScreenCaptureFrameSource` | | `Compositor/SceneCompositor.cs` | **The output compositor (TASK 4 ship step 1)**: renders a scene into the encoder's master `VideoFrame` (tightly-packed BGRA8), mirroring the XAML preview minus editing chrome — backdrop → background → elements (`UniformToFill` cover-crop, round clip, mirror, opacity, border) → branding flash → social bar (optional `socialBarFrame` + `socialBarTop` in master space, blitted last so the bar sits above the flash; the old `BlitFlash` generalized to `BlitOverlay` with source offsets). Pure and WPF-free: frames injected via a `Func` resolver (webcam → DeviceId, image → AssetId, backdrop → CaptureKey); output sized by `CompositorOptions` (16:9 = full master 1:1; vertical 9:16 = 607×1080 crop → 1080×1920 bilinear). Preview stays XAML (editing view); this is the output view — see `ai.md` "Scene compositor" | | `Compositor/CompositorOptions.cs` | The active tier's output rect (source space over the 1920×1080 master, integer-aligned — `MainViewModel.OutputRectX` can be 656.5) + target W×H | | `Compositor/StretchMath.cs` | Pure pixel math: the WPF `UniformToFill` cover-crop, clamped bilinear sample/scale (unit-tested half of the compositor) | diff --git a/TASKS/task-17-web-source-rendering.md b/TASKS/task-17-web-source-rendering.md index 1e301bb..ff65814 100644 --- a/TASKS/task-17-web-source-rendering.md +++ b/TASKS/task-17-web-source-rendering.md @@ -11,7 +11,10 @@ 2. ✅ Schema v10: `WebUri TEXT` column on `Source` table + migration in `LayoutStore.cs` 3. ✅ Persist `Source.WebUri` on save/load (currently in-memory only — lost on restart) 4. ✅ Hidden off-screen `WebView2` control per web source — navigates to `WebUri`, renders in-app -5. ✅ Frame capture from WebView2 (`CoreWebView2.CapturePreviewAsync`) → `WriteableBitmap` (BGRA8) +5. ✅ Frame capture from WebView2 (**superseded 2026-09-14:** was `CoreWebView2.CapturePreviewAsync` + → `WriteableBitmap` (BGRA8); now composition-capture via + `CoreWebView2CompositionController` → `Windows.Graphics.Capture` — see the rendering-model + update below. PNG polling is deleted) 6. ✅ Wire into `FramePump` resolver — `Source { Type: WebSource }` → latest WebView2 frame 7. ✅ Wire into `SceneCompositor` — render web source as an image element at its position/size 8. ✅ Preview shows live web content (not just a blank rectangle) @@ -33,6 +36,18 @@ measure DOM stuff on NavigationCompleted (unsettled layout broke the image — ` sizing the container to the crop (`ab29ec8` reverted — made it worse); a widget glow effect can fake a gap — verify with a plain widget first; measure rendered pixels instead. +**Rendering model update (2026-09-14, COMPOSITION capture — the PNG poll is gone):** frame capture +is no longer `CapturePreviewAsync` → PNG (35–165ms per full-HD encode+decode capped real cadence at +~20Hz; item 5 below is superseded). Each session now renders through a +`CoreWebView2CompositionController` visual into `Windows.Graphics.Capture` +(`GraphicsCaptureItem.CreateFromVisual` of the root 1920×1080 `ContainerVisual`), pulled frame-driven +from a free-threaded `Direct3D11CaptureFramePool` at the renderer's own rate (~60fps, parity with the +browser). Straight-alpha readback, alpha-bbox crop + `CropBounds` per frame, epoch'd ring reuse, one +shared crop-sized preview bitmap. The old `SetCaptureInterval` cadence hooks and `CaptureScheduler` +are deleted; the hidden XAML `WebViewHostPanel` overlay and `CapturePreviewAsync` telemetry are gone. +`TransparentBackgroundScript` is unchanged (a test pins it). Full design + the CoreMessaging DQ +recipe in `../ai.md` → Slice 14. **Open: device verification of ~60fps web animation.** + **Properties panel (2026-08-28):** web URI ✓/✕ icon buttons are `IsTabStop="False"` so Tab flows X→Y→W→H→URI; the ✕ button now clears the URI textbox (was reverting to the pre-accept snapshot). Slider style gained `IsMoveToPointEnabled="True"` — click-anywhere-on-bar jumps the thumb to the @@ -42,6 +57,7 @@ click (volume sliders keep their manual `SetSliderValueFromClick`, harmless dupl - WebView2 is the only option for Windows — it's pre-installed on Windows 10 20H2+ and Windows 11 - The web source is a standard element — positioned/sized/opacitied like any image source -- Frame capture rate can be lower than video FPS (5-10 fps for web content is fine) +- Frame capture is drive-by-renderer (composition capture, ~60fps parity with the browser since + 2026-09-14; the old "5-10 fps is fine" poll model is deleted) - This enables Streamlabs/StreamElements overlays via web URLs diff --git a/ViewModels/MainViewModel.Streaming.Operations.cs b/ViewModels/MainViewModel.Streaming.Operations.cs index d9a274b..03e1e78 100644 --- a/ViewModels/MainViewModel.Streaming.Operations.cs +++ b/ViewModels/MainViewModel.Streaming.Operations.cs @@ -56,7 +56,6 @@ public partial class MainViewModel : ViewModelBase ResetHealth(StreamStatus.Offline); _audioMixer.StartLive(EncoderOptions.DefaultAudioPipeName); IsRecording = true; - _webView2Manager?.SetCaptureInterval(33); // web layer ~30Hz while recording await _framePump.StartAsync(); } @@ -176,7 +175,6 @@ public partial class MainViewModel : ViewModelBase return; } - _webView2Manager?.SetCaptureInterval(33); // web layer ~30Hz while live await _framePump.StartAsync(); // never throws; failures log + surface via Failed // Health polling (TASK 5 item 3): poll immediately, then every 30s while @@ -253,7 +251,6 @@ public partial class MainViewModel : ViewModelBase // monitoring); only the live/report pipe, the pump, and the session stop here. _audioMixer.StopLive(); await _framePump.StopAsync(); - _webView2Manager?.SetCaptureInterval(200); // idle pace — preview-only doesn't need 30Hz // Proper close-out (TASK 9 decision 6, built 2026-09-01): transition(complete) // AFTER the RTMP push is closed so no frames post-date the end — YouTube then diff --git a/ViewModels/MainViewModel.Web.cs b/ViewModels/MainViewModel.Web.cs index 48048b0..1b5c7e0 100644 --- a/ViewModels/MainViewModel.Web.cs +++ b/ViewModels/MainViewModel.Web.cs @@ -1,5 +1,4 @@ using System.Linq; -using System.Windows.Controls; using System.Windows.Media.Imaging; using ytLive.Models; using ytLive.Services; @@ -25,9 +24,13 @@ public partial class MainViewModel } } - public void InitWebView2(Panel hostPanel) + // Runs on the main window's Loaded handler (MainWindow.xaml.cs): the + // composition controller needs the window HWND as its parent, which does not + // exist in the ctor. Web sources register then — no frames are expected before + // the window is shown anyway. + public void InitWebView2() { - _webView2Manager = new WebView2Manager(hostPanel, System.Windows.Application.Current.Dispatcher); + _webView2Manager = new WebView2Manager(System.Windows.Application.Current.Dispatcher); _webView2Manager.PreviewBitmapChanged += OnWebView2PreviewBitmapChanged; RegisterLoadedWebSources(); } diff --git a/ViewModels/index.md b/ViewModels/index.md index 849f69f..2653efd 100644 --- a/ViewModels/index.md +++ b/ViewModels/index.md @@ -60,7 +60,7 @@ Progress (Commit N/12 · tag): - [x] **Commit 15 · `refactor-commit-15`** — **PreviewPane** → `Controls/PreviewPane.xaml(.cs)` (Row 1 Col 1 center preview pane). Moved the 1920×1080 canvas (OverlayCanvas/SelectionOverlay/BrandFlashLayer/SocialBarElement), text pull-out, and the live-controls row (TRAX/game/mic meters + sliders + speakers + socials button). Code-behind moved: preview drag/select/resize/hit-test (Preview_Mouse*, HitHandle, HitElement), webcam-context-menu handlers (WebcamMenu_*), social-bar glow (SocialBar_*, SocialsButton_*, SocialGlowTick), audio sliders/speakers (VolumeSlider_*, GameVolumeSlider_*, MicSpeaker_*, GameSpeaker_*, SetSliderValueFromClick, FindParent), TraxButton_*. Window keeps the source-list left panel + thumbnail strip + edit UI, and routes selection-overlay via `_previewPane.UpdateSelectionOverlay(bool)` plus `_previewPane.IsClickInsidePreview/Drawer`. `BackgroundMenu_*` stayed in the window (used by the left-panel source context menu). - [x] **Commit 16 · `refactor-commit-16`** — **SceneThumbnailStrip** → `Controls/SceneThumbnailStrip.xaml(.cs)` (Row 0 scene-picker strip). `Thumbnail_MouseLeftButtonDown` moved into the control (transitions via `DataContext.TransitionToScene`). - [x] **Commit 17 · `refactor-commit-17`** — **LeftPanel** → `Controls/LeftPanel.xaml(.cs)` (Row 1 Col 0 left panel: layers/properties edit UI + live chat list). Moved ALL property-editor and chat handlers: `SourceNameBox_*`/`CommitSourceEdit`, `AddSourceButton_Click`, `OnChatMessagesChanged` (now self-subscribed on Loaded), `MirrorButton_Click`, `ShapeButton_Click`, `SourceMenu_Remove`, `BackgroundMenu_*`, `OpacitySlider_ValueChanged`, list drag-to-reorder (`_dragStartPoint`+`List_Preview*`+`EndListDrag`+`FindItemContainerAt`), and the snapshot family (`TakeSnapshot`, `WebcamBorder*`, `Countdown*`, `Web*`). Window exposes via `_leftPanel.IsClickInside(original)` (deselect guard) and `_leftPanel.OnSelectionChanged(vm)` (opacity label + snapshot). Pending: overlay host region. -- [x] **Commit 18 · `refactor-commit-18`** — **OverlayHost** → `Controls/OverlayHost.xaml(.cs)` (settings/bug/feature/about dialogs). `BackToAbout_Click` moved with it (uses inherited `DataContext`). Window keeps the thin `WebViewHostPanel` + `ToastArea` root stubs so name lookups are untouched (WebView2 ctor init, `NotificationService.InArea("ToastArea")`) — toast/WebView2 behavior unchanged. **Phase 2 COMPLETE** — MainWindow.xaml carved down to a 123-line shell over 6 controls (TopBar, HealthBanner, BottomBar, SceneThumbnailStrip, LeftPanel, PreviewPane, OverlayHost). Pending: session wrap-up (HANDOFF.md) + push on approval. +- [x] **Commit 18 · `refactor-commit-18`** — **OverlayHost** → `Controls/OverlayHost.xaml(.cs)` (settings/bug/feature/about dialogs). `BackToAbout_Click` moved with it (uses inherited `DataContext`). Window keeps the thin `WebViewHostPanel` + `ToastArea` root stubs so name lookups are untouched (WebView2 ctor init, `NotificationService.InArea("ToastArea")`) — toast/WebView2 behavior unchanged. **Phase 2 COMPLETE** — MainWindow.xaml carved down to a 123-line shell over 6 controls (TopBar, HealthBanner, BottomBar, SceneThumbnailStrip, LeftPanel, PreviewPane, OverlayHost). Pending: session wrap-up (HANDOFF.md) + push on approval. *(The `WebViewHostPanel` root stub is since GONE — the 2026-09-14 composition-capture redesign moved `InitWebView2()` to `MainWindow_Loaded` and deleted the panel; see `Controls/index.md`.)* **Phase 3 — 500-line compliance (production `.cs` only).** Bring all production source files under 500 lines. One roll-back commit per file; same drill diff --git a/ai.md b/ai.md index b883a6b..b8487e5 100644 --- a/ai.md +++ b/ai.md @@ -893,6 +893,8 @@ seam:** `Func`, `Func` resolver, `Func Superseded by **Slice 14** (below): `CaptureScheduler` and the cadence hooks were the polling + > design; frame-driven composition capture replaced them — nothing here is current architecture. (1) **the 10Hz cap** (unconditional, code-proven) and (2) **Chromium hidden-page throttling** (`requestAnimationFrame` parked, JS timers clamped to ~1s per WebView2Feedback#1172/#3070 + Chrome-88 timer throttling) whenever the app window is unfocused @@ -935,11 +937,65 @@ Full suite 290/291 passing, the sole failure the pre-existing compositor pixel t FIRST capture ever = the initial about:blank document — a blind instrument. NavigationCompleted for the REAL widget URL now arms `WidgetDumpRemaining = 5`; those 5 post-paint captures dump `%TEMP%\ytLive-web--w1..5.png` + the shared `AlphaStats(...)` line (alpha[min/max/mean/zero%]) - + FindContentBounds result to startup.log. That line alone decides the fix branch: zero% ⇒ the - pre-parse injection did not hold for this widget (opaque capture → chroma-key / stronger DOM - injection); large zero% + tight contentBounds ⇒ the capture IS transparent and the recording's - black block lives downstream (compositor blend/underlay). `AlphaStats` extracted as the shared - sampler. No new tests (WebView2 runtime not instantiable in tests; the re-point is log-only). + + FindContentBounds result to startup.log. That line alone decides the fix branch: zero% ⇒ the + pre-parse injection did not hold for this widget (opaque capture → chroma-key / stronger DOM + injection); large zero% + tight contentBounds ⇒ the capture IS transparent and the recording's + black block lives downstream (compositor blend/underlay). `AlphaStats` extracted as the shared + sampler. No new tests (WebView2 runtime not instantiable in tests; the re-point is log-only). +- **Slice 14 — web frames are now COMPOSITION-CAPTURED; the PNG poll + `CaptureScheduler` are gone + (2026-09-14, the "still not 60fps" take):** slice 11's ~30Hz was still a `CapturePreviewAsync` PNG + poll — every 1920×1080 full-HD encode+decode cost 35–165ms, so the session budget capped real + cadence at ~20Hz for a 60fps-designed widget. Root fix = frame-driven capture, not faster polling: + the WebView2 renderer now feeds `Windows.Graphics.Capture` directly through a + `CoreWebView2CompositionController` (the mechanism `WebView2CompositionControl` and Flutter's + `webview_windows` use — see `graphics_context.cc` `CreateGraphicsCaptureItemFromVisual`, which + captures the root `surface_` visual). + - **Pipeline (per session):** `CreateCoreWebView2CompositionControllerAsync(WindowHandle)` — the + parent HWND must exist, so `MainWindow.InitWebView2()` moved from the ctor to **Loaded** (was + `InitWebView2(WebViewHostPanel)`; the hidden XAML `WebViewHostPanel` overlay is deleted). + `controller.RootVisualTarget` = a child `ContainerVisual` (`RelativeSizeAdjustment = 1,1`) under a + root `ContainerVisual` (1920×1080, `IsVisible=true`) built on ONE `Windows.UI.Composition.Compositor` + (created once on the UI thread); `Bounds = 0,0,1920,1080`, `BoundsMode=UseRawPixels`, + `ShouldDetectMonitorScaleChanges=false`, `RasterizationScale=1.0`, `IsVisible=true`, + `DefaultBackgroundColor=Transparent`; then `GraphicsCaptureItem.CreateFromVisual(root)`. + Frames pull from a free-threaded `Direct3D11CaptureFramePool` (2 buffers) → + `SoftwareBitmap.CreateCopyFromSurfaceAsync` (BGRA, `BitmapAlphaMode.Straight` — the compositor + blends STRAIGHT alpha, so premultiplied readback would wreck corner anti-aliasing) → per-frame + `FindContentBounds` alpha-bbox on the capture worker → `VideoFrame.CropBounds` stamped → ring up to + 2 fresh + Epoch (reuse; GC lesson slice 8) → dispatcher-coalesced copy (Render priority) into ONE + crop-sized shared `WriteableBitmap`; `PreviewBitmapChanged` raised only on bitmap (re)creation. + Also unpumped: the OBS-side de-throttle flags (slice 11) stay — they fix the hidden-page JS + throttling; the capture path itself no longer polls. + - **Implemented in:** new `Services/WebCaptureFrameSource.cs` (owns item+pool+session — mirrors + `ScreenCaptureFrameSource`, which servers as the copy template) + reworked `Services/WebView2Manager.cs` + (owns controllers, the visual tree, nav/script wiring, preview; internal seam ctor + `(Dispatcher, Func?)` so the tests instantiate zero WinRT; public + ctor `(Dispatcher)`). + - **CoreMessaging DQ recipe (record-once):** the 19041 CsWinRT projection has NO + `DispatcherQueueController.CreateOnCurrentThread()` (CS0117 — only `CreateOnDedicatedThread` + + `FromAbi(IntPtr)`). P/Invoke `coreMessaging.dll!CreateDispatcherQueueController` with struct + `DispatcherQueueOptions { DwSize, ThreadType = 2 (DQTYPE_THREAD_CURRENT), ApartmentType = 2 (DQTAT_COM_STA) }`, + wrap via `DispatcherQueueController.FromAbi(ptr)` (mirror of `CaptureInterop`), only then + `new Compositor()` — all once on the WPF UI thread (the app dispatches Render there). + - **Compile notes (each cost a build cycle):** `Compositor` collides with the repo's OWN + `ytLive.Services.Compositor` namespace → fully-qualify `Windows.UI.Composition.Compositor`; + `CoreWebView2CompositionController` exposes **`Close()`**, not `Dispose()`; `Color` is ambiguous + (`System.Drawing` vs `System.Windows.Media`) → `System.Drawing.Color.Transparent`. + - **Dropped:** `Services/CaptureScheduler.cs` (DELETED — no cadence to schedule), all three + `SetCaptureInterval` hooks in `MainViewModel.Streaming.Operations.cs`, the XAML `WebViewHostPanel`. + `TransparentBackgroundScript` const is unchanged (a test pins it). + - **Test model (Good Dog):** reworked `ytLive.Tests/WebView2ManagerTests.cs` — dropped the 4 + control-size tests + the `CaptureScheduler_Drops…` test; `FindContentBounds` tests moved to + `WebCaptureFrameSource.FindContentBounds`; the ONE integration test + `Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds` drives the internal seam with + `FakeWebSource : IScreenCaptureSource` + a real background-STA `DispatcherPump` (copied from + `ScreenCaptureManagerTests`): register → one crop-sized preview bitmap published → `CropBounds` + + `IsOpaque=false` on `GetLatestFrame` → back-to-back pumps coalesce to latest. Byte assertion compares + the DENSE 2×2 crop (`CropBytes` helper — slices source rows with stride gaps, a contiguous range + spans rows wrongly). + - Full suite **293/293 green, 0 warnings**. Audio untouched. **NOT YET VERIFIED ON DEVICE** — the + composition path needs a real 60fps-widget run (open item; see HANDOFF). Web work commits stay + LOCAL (no push) until the user greenlights. - **Stop ordering matters:** `StopAsync` stops the encoder — since slice 10 it FLUSHES the pending queue (`Channel.TryComplete` → drain writes the leftovers, closes stdin → EOF → ffmpeg finalizes+exits; an accepted frame is never lost) — **before** awaiting the loop. The old reverse-order deadlock was diff --git a/ytLive.Tests/RoundClipInteractionTests.cs b/ytLive.Tests/RoundClipInteractionTests.cs index 9027829..4460bfc 100644 --- a/ytLive.Tests/RoundClipInteractionTests.cs +++ b/ytLive.Tests/RoundClipInteractionTests.cs @@ -84,10 +84,11 @@ public sealed class RoundClipInteractionTests // of PreviewGrid); otherwise Window_PreviewMouseLeftButtonDown // deselects the source and the handle can never be grabbed. // UIElement.InputHitTest IS the input pipeline — what Mouse uses. - // Do NOT use VisualTreeHelper.HitTest here: it returned the window's - // non-hit-testable WebViewHostPanel overlay for EVERY point (it does - // not model IsHitTestVisible input semantics) — the bug this test - // "found" for months was this wrong API (root-caused 2026-09-01). + // Do NOT use VisualTreeHelper.HitTest here: it ignores IsHitTestVisible + // input semantics (the old invisible WebView2 host overlay for web + // sources used to win every visual hit — the wrong-API false bug this + // test "found" for months, root-caused 2026-09-01; the overlay is gone + // with the composition-based web render but the API lesson stays). var hit = window.InputHitTest(cornerInWindow) as DependencyObject; var grabbable = hit != null && IsDescendantOf(hit, previewGrid); diff --git a/ytLive.Tests/WebView2ManagerTests.cs b/ytLive.Tests/WebView2ManagerTests.cs index 03241f7..7f46dd0 100644 --- a/ytLive.Tests/WebView2ManagerTests.cs +++ b/ytLive.Tests/WebView2ManagerTests.cs @@ -1,114 +1,72 @@ -using System.Drawing; -using System.Linq; -using System.Windows.Controls; +using System; +using System.Threading; +using System.Threading.Tasks; using System.Windows.Threading; -using Microsoft.Web.WebView2.Wpf; +using System.Windows.Media.Imaging; using Xunit; using ytLive.Models; using ytLive.Services; namespace ytLive.Tests; -[Collection("RealApp")] -public sealed class WebView2ManagerTests +/// +/// WebView2Manager owns the frame-driven web sources (composition-captured, see +/// Services/WebCaptureFrameSource.cs). The controller/visual/GraphicCaptureItem +/// layer is WinRT and only exercisable on Windows at runtime, so these tests pin +/// the pure pieces (transparency script, the alpha-bbox math) and the ONE +/// integration test drives the session plumbing with a fake source: registration, +/// crop-stamped frames, the shared cropped preview bitmap, and coalescing to the +/// latest frame. +/// +public class WebView2ManagerTests { - private readonly RealAppHost _app; - - public WebView2ManagerTests(RealAppHost app) + private sealed class FakeWebSource : IScreenCaptureSource { - _app = app; + public string Key { get; } + public event Action? FrameAvailable; + + public FakeWebSource(string key) => Key = key; + + public Task StartAsync() => Task.CompletedTask; + public Task StopAsync() => Task.CompletedTask; + + public void Pump(VideoFrame frame) => FrameAvailable?.Invoke(frame); } - [Fact] - public void WebView2_Control_Size_Is_Canvas_Resolution_Regardless_Of_Source() + // A real Dispatcher pumping on a background STA thread, so the manager's + // BeginInvoke queue can be drained deterministically from the test thread + // (same harness as ScreenCaptureManagerTests). + private sealed class DispatcherPump : IDisposable { - _app.Run(() => + private readonly Thread _thread; + private readonly ManualResetEventSlim _ready = new(false); + private Dispatcher? _dispatcher; + + public Dispatcher Dispatcher => _dispatcher!; + + public DispatcherPump() { - var panel = new Canvas(); - var source = new Source + _thread = new Thread(() => { - Type = SourceType.WebSource, - Width = 640, - Height = 480, - }; + _dispatcher = Dispatcher.CurrentDispatcher; + _ready.Set(); + Dispatcher.Run(); + }); + _thread.IsBackground = true; + _thread.SetApartmentState(ApartmentState.STA); + _thread.Start(); + _ready.Wait(); + } - using var manager = new WebView2Manager(panel, Dispatcher.CurrentDispatcher); - manager.Register(source); + // Queued behind anything already posted at Render/higher, so pending + // bitmap copies run first. + public void Drain() => Dispatcher.Invoke(() => { }, DispatcherPriority.ContextIdle); - var webView = panel.Children.OfType().First(); - Assert.Equal(1920.0, webView.Width); - Assert.Equal(1080.0, webView.Height); - }); - } - - [Fact] - public void WebView2_Control_Size_Does_Not_Track_Box_After_Register() - { - _app.Run(() => + public void Dispose() { - var panel = new Canvas(); - var source = new Source - { - Type = SourceType.WebSource, - Width = 640, - Height = 480, - }; - - using var manager = new WebView2Manager(panel, Dispatcher.CurrentDispatcher); - manager.Register(source); - - var webView = panel.Children.OfType().First(); - - source.Width = 1024; - source.Height = 768; - - Assert.Equal(1920.0, webView.Width); - Assert.Equal(1080.0, webView.Height); - }); - } - - [Fact] - public void WebView2_Control_Size_Ignores_Zero_Sized_Source() - { - _app.Run(() => - { - var panel = new Canvas(); - var source = new Source - { - Type = SourceType.WebSource, - Width = 0, - Height = 0, - }; - - using var manager = new WebView2Manager(panel, Dispatcher.CurrentDispatcher); - manager.Register(source); - - var webView = panel.Children.OfType().First(); - Assert.Equal(1920.0, webView.Width); - Assert.Equal(1080.0, webView.Height); - }); - } - - [Fact] - public void WebView2_DefaultBackgroundColor_Is_Transparent() - { - _app.Run(() => - { - var panel = new Canvas(); - var source = new Source - { - Type = SourceType.WebSource, - Width = 640, - Height = 480, - }; - - using var manager = new WebView2Manager(panel, Dispatcher.CurrentDispatcher); - manager.Register(source); - - var webView = panel.Children.OfType().First(); - - Assert.Equal(System.Drawing.Color.Transparent, webView.DefaultBackgroundColor); - }); + Dispatcher.InvokeShutdown(); + _thread.Join(3000); + } } [Fact] @@ -142,7 +100,7 @@ public sealed class WebView2ManagerTests pixels[i + 3] = 255; // A } - var (x, y, bw, bh) = WebView2Manager.FindContentBounds(pixels, w, h, w * 4); + var (x, y, bw, bh) = WebCaptureFrameSource.FindContentBounds(pixels, w, h, w * 4); Assert.Equal((3, 2, 4, 3), (x, y, bw, bh)); } @@ -153,7 +111,7 @@ public sealed class WebView2ManagerTests const int w = 10, h = 6; var pixels = new byte[w * h * 4]; - var (x, y, bw, bh) = WebView2Manager.FindContentBounds(pixels, w, h, w * 4); + var (x, y, bw, bh) = WebCaptureFrameSource.FindContentBounds(pixels, w, h, w * 4); Assert.Equal((0, 0, w, h), (x, y, bw, bh)); } @@ -166,36 +124,108 @@ public sealed class WebView2ManagerTests for (var i = 3; i < pixels.Length; i += 4) pixels[i] = 255; - var (x, y, bw, bh) = WebView2Manager.FindContentBounds(pixels, w, h, w * 4); + var (x, y, bw, bh) = WebCaptureFrameSource.FindContentBounds(pixels, w, h, w * 4); Assert.Equal((0, 0, w, h), (x, y, bw, bh)); } + // The ONE integration test for this change: a registered session creates one + // shared WriteableBitmap sized to the widget's CropBounds, publishes it once, + // and back-to-back frames coalesce to the latest (a single pending UI copy, + // mirror of ScreenCaptureManager) — the frame-driven contract the PNG-poll + // redesign replaced the CaptureScheduler cadence with. [Fact] - public async Task CaptureScheduler_Drops_Ticks_While_Capture_InFlight_And_Resumes() + public void Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds() { - // The ~30Hz recording cadence would stack concurrent CapturePreviewAsync calls - // (full-HD PNG encode is ~10-30ms) without the latest-wins drop: ticks that - // arrive while a capture is in flight must be skipped, never queued, and a - // capture can always start once the previous one has finished. - var calls = 0; - var signal = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); - var scheduler = new CaptureScheduler(Dispatcher.CurrentDispatcher, 200, async () => + using var pump = new DispatcherPump(); + FakeWebSource? fake = null; + using var manager = new WebView2Manager(pump.Dispatcher, key => fake = new FakeWebSource(key)); + + WriteableBitmap? published = null; + var publishedCount = 0; + manager.PreviewBitmapChanged += (_, bitmap) => { published = bitmap; publishedCount++; }; + + var source = new Source { Type = SourceType.WebSource, Width = 640, Height = 480 }; + manager.Register(source); + Assert.NotNull(fake); // the seam instantiated + started the source inline + Assert.Null(manager.GetLatestFrame(source.Id)); // nothing pumped yet + + // 4x4 canvas; alpha 255 only in bbox (1,1)-(2,2) → the widget crops to (1,1,2,2). + var first = WidgetFrame(4, 4, frameSeed: 42); + fake!.Pump(first); + pump.Drain(); + + Assert.NotNull(published); + Assert.Equal(1, publishedCount); // one shared bitmap, published once + var dims = pump.Dispatcher.Invoke(() => new[] { published!.PixelWidth, published.PixelHeight }); + Assert.Equal(new[] { 2, 2 }, dims); + + var latest = manager.GetLatestFrame(source.Id); + Assert.NotNull(latest); + Assert.Equal((1, 1, 2, 2), latest!.CropBounds); // the source stamps the alpha bbox + Assert.False(latest.IsOpaque); // straight-alpha web contract + + var previewBytes = pump.Dispatcher.Invoke(() => { - calls++; - await signal.Task; + var bytes = new byte[16]; + published!.CopyPixels(new System.Windows.Int32Rect(0, 0, 2, 2), bytes, 8, 0); + return bytes; }); + // The preview is a DENSE 2x2 rect (bbox 1,1..2,2 sliced out of the 4x4 + // canvas) — rows y=1 and y=2, columns x=1..2. + var expected = CropBytes(first, w: 4, x0: 1, y0: 1, w2: 2, h2: 2); + Assert.Equal(expected, previewBytes); - scheduler.Tick(); - scheduler.Tick(); - scheduler.Tick(); - Assert.Equal(1, calls); // overlapping ticks dropped, never queued + // Second + third back-to-back: only the latest lands in the preview bitmap. + var second = WidgetFrame(4, 4, frameSeed: 77); + var third = WidgetFrame(4, 4, frameSeed: 99); + fake.Pump(second); + fake.Pump(third); + pump.Drain(); - signal.SetResult(); - await scheduler.Pending; // durable: Pending completes only after the drop resets - - scheduler.Tick(); - Assert.Equal(2, calls); // once idle, the next tick captures again - await scheduler.Pending; + Assert.Equal(1, publishedCount); // same crop box → no new bitmap + Assert.Same(third, manager.GetLatestFrame(source.Id)); + var coalesced = pump.Dispatcher.Invoke(() => + { + var bytes = new byte[16]; + published!.CopyPixels(new System.Windows.Int32Rect(0, 0, 2, 2), bytes, 8, 0); + return bytes; + }); + Assert.Equal(CropBytes(third, 4, 1, 1, 2, 2), coalesced); } -} + + /// Dense little-endian slice of a w-wide frame for the rect + /// (x0, y0)-(x0+w2-1, y0+h2-1); matches the manager's crop readback so the + /// byte comparison is against the same layout, not the full frame stride. + private static byte[] CropBytes(VideoFrame frame, int w, int x0, int y0, int w2, int h2) + { + var src = frame.BgraPixels; + var dst = new byte[w2 * h2 * 4]; + for (var y = 0; y < h2; y++) + System.Buffer.BlockCopy(src, ((y0 + y) * w + x0) * 4, dst, y * w2 * 4, w2 * 4); + return dst; + } + + /// A 4x4 full canvas where the widget content is bbox (1,1,2,2): + /// alpha 255 there, 0 elsewhere; the content pixels carry a seed so a + /// coalesce mix-up is visible in the byte assertion. + private static VideoFrame WidgetFrame(int w, int h, byte frameSeed) + { + var pixels = new byte[w * h * 4]; + for (var y = 0; y < h; y++) + for (var x = 0; x < w; x++) + { + if (x < 1 || x > 2 || y < 1 || y > 2) continue; + var i = (y * w + x) * 4; + pixels[i] = frameSeed; // B + pixels[i + 1] = (byte)(frameSeed / 3); // G + pixels[i + 2] = 0; // R + pixels[i + 3] = 255; // A + } + return new VideoFrame(w, h, pixels) + { + IsOpaque = false, + CropBounds = (1, 1, 2, 2), + }; + } +} \ No newline at end of file