feat: web sources frame-captured via composition (CoreWebView2CompositionController → Windows.Graphics.Capture); PNG poll + CaptureScheduler deleted
The ~30Hz CapturePreviewAsync PNG poll capped real cadence at ~20Hz (35–165ms full-HD encode+decode), so a 60fps widget still juddered at ~1/6 speed. Replaced polling with frame-driven capture of the composition controller's root visual — the mechanism WebView2CompositionControl and Flutter's webview_windows use (graphics_context.cc captures the root surface_ visual via CreateGraphicsCaptureItemFromVisual; reference: github.com/microsoft/Windows.UI.Composition.WinUI / flutter-internal webview_windows). Frames now arrive at the renderer's own pace; capture memory is epoch'd ring reuse + one crop-sized shared WriteableBitmap. New Services/WebCaptureFrameSource.cs owns GraphicsCaptureItem + free- threaded Direct3D11CaptureFramePool + session (Straight alpha readback, per-frame FindContentBounds → CropBounds). WebView2Manager reworked around per-session composition controllers + one UI-thread Compositor created via the CoreMessaging CreateDispatcherQueueController P/Invoke (the 19041 projection lacks CreateOnCurrentThread); internal seam ctor (Dispatcher, Func<string,IScreenCaptureSource>?) for hermetic tests. CaptureScheduler.cs deleted; the three SetCaptureInterval cadence hooks removed; InitWebView2() moved from MainWindow ctor to Loaded (a parent HWND must exist for the composition controller); the hidden WebViewHostPanel overlay deleted. TransparentBackgroundScript unchanged. Tests: WebView2ManagerTests reworked — 4 control-size + scheduler tests dropped, FindContentBounds tests moved to WebCaptureFrameSource, ONE integration test (Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds) drives the seam with a FakeWebSource + background-STA DispatcherPump. Suite 293/293, 0 warnings. NOTE: composition path NOT yet verified on a device — the take is the next step. Web work committed locally only (no push per standing rule). Docs same-commit: ai.md Slice 14 + supersede marker on Slice 11, HANDOFF, MyMistakes (CoreMessaging DQ + namespace-landmine recipe), TASK 17, Controls/ViewModels/Services indexes.
This commit is contained in:
+4
-1
@@ -47,7 +47,10 @@ cross-cutting calls through it:
|
||||
`WebViewHostPanel` (ctor `InitWebView2(WebViewHostPanel)`) and `ToastArea`
|
||||
(`NotificationService.InArea("ToastArea")`) are resolved by name from the
|
||||
window root — so `OverlayHost` deliberately leaves them as thin window-root
|
||||
children (Commit 18).
|
||||
children (Commit 18). Since the 2026-09-14 composition-capture redesign only
|
||||
`ToastArea` remains — `WebViewHostPanel` is deleted (web sources no longer
|
||||
host a multiline XAML control; `InitWebView2()` runs from `MainWindow_Loaded`
|
||||
with no panel argument).
|
||||
- When cutting visual wrappers, don't drop the outer `Border`'s
|
||||
background/glow/margin — Commit 15 initially lost the preview's outer border
|
||||
and had to restore it same-commit.
|
||||
|
||||
+63
-55
@@ -1,14 +1,10 @@
|
||||
# HANDOFF — 2026-09-14 (signed audio sync SHIPPED + PUSHED: −500..+500, negative advances by eating the stream head; slider locked while live/recording)
|
||||
# HANDOFF — 2026-09-14 (web capture REDESIGNED: composition-capture, PNG polling + CaptureScheduler deleted — NOT yet verified on device, NOT pushed)
|
||||
|
||||
## Branch / Commit State
|
||||
|
||||
`main` HEAD = **`b22d08e`** (signed audio-sync, committed AND **pushed** to origin/main with the
|
||||
user's greenlight — 29 commits pushed in one go). Working tree **clean**.
|
||||
|
||||
DB now: **`Audio.SyncOffsetMs = 0`** (confirmed via sqlite3 — the creator slid the sync to 0
|
||||
and the write finally stuck; sound is correct at 0). The 0.54s residual in the 1128 take ≈ 300ms
|
||||
injected offset (leftover DB value) + ~240ms natural (partly webcam clap-quantization ±40–85ms,
|
||||
partly measurement).
|
||||
`main` HEAD = **`b22d08e`** (signed audio-sync, already pushed). Working tree **DIRTY** with the
|
||||
web-frame-capture redesign (below). Commits for it land **LOCALLY only** — the web work was never
|
||||
pushed; the user greenlights pushes at checkpoints.
|
||||
|
||||
## ⚠️ Branding (2026-09-14, creator-corrected): product = **llamacasty**, internals = ytLive
|
||||
|
||||
@@ -17,72 +13,84 @@ The product is **llamacasty**; the repo path, csproj `AssemblyName`/`RootNamespa
|
||||
language must say "llamacasty"; code/assembly/repo names stay ytLive. Full detail in
|
||||
`ai.md` → Brand → "Product name vs repo/assembly branding".
|
||||
|
||||
## ✅ SHIPPED (this dirty tree) — signed audio sync, −500..+500
|
||||
## 💥 The web-capture take history (why this redesign exists)
|
||||
|
||||
**What:** the audio-sync control is now a SIGNED offset. Positive = delay the mix (audio runs
|
||||
AHEAD of video — existing `AudioSyncDelay` behavior, unchanged and live-reactive). Negative =
|
||||
**advance** the audio (audio runs BEHIND video): OBS's "eat the head of the buffer" fix —
|
||||
the mixer drops the first |N| ms of the written stream at the pipe, re-anchoring the audio
|
||||
stream so every event lands |N| ms EARLIER relative to video.
|
||||
The recording was 60fps but web widgets ran at ~1/6 speed: capture was a `CapturePreviewAsync` PNG
|
||||
poll; every full-HD encode+decode cost **35–165ms** so the session budget capped real cadence at
|
||||
~20Hz — a 60fps widget still juddered. Slice 11's `CaptureScheduler` (30Hz) was the wrong lever:
|
||||
the fix is frame-driven, not faster polling.
|
||||
|
||||
**Mechanism (AudioMixer):** `StartLive` arms `_advanceSamplesRemaining = |N| ms → samples` at
|
||||
go-live (a negative offset can only eat the HEAD of the stream; it is armed once, not
|
||||
live-reactive). `LiveLoopAsync` skips `min(budget, mixBuffer.Length)` samples off each write
|
||||
head while the budget lasts — the pipe writer accepts a partial chunk via `AsMemory(writeFrom)`.
|
||||
Positive path untouched (delay line still re-reads the Func every tick).
|
||||
## 🔬 IN FLIGHT (this dirty tree, NOT committed, NOT pushed) — composition capture
|
||||
|
||||
**UI/plumbing:**
|
||||
- `MainViewModel.Audio.cs` — clamp `Math.Clamp(value, -500, 500)`, doc updated.
|
||||
- `LayoutStore.Settings.cs` — `LoadAudioSyncOffsetMs`/`SaveAudioSyncOffsetMs` clamp −500..500.
|
||||
- `PreviewPane.xaml` — label **SYNC → "AUDIO SYNC"**, `Minimum="-500"`, tooltip explains both
|
||||
directions (calibrate with a clap: clap late → negative; early → positive), and
|
||||
**`IsEnabled="{Binding IsEditMode}"`** — the slider locks during live AND recording (gun
|
||||
safety, same property `IsRecording`/`StreamStatus` already raise PropertyChanged for).
|
||||
- `AudioSyncDelay` unchanged (still clamps negative→0 internally; header doc updated to point
|
||||
at the mixer for the advance side).
|
||||
**What:** web sources now render through `CoreWebView2CompositionController` into
|
||||
`Windows.Graphics.Capture` (the Flutter `webview_windows` / `WebView2CompositionControl`
|
||||
mechanism) — frame-driven at the renderer's pace instead of polling PNGs.
|
||||
|
||||
**Regression test (the ONE integration test for this change):**
|
||||
`StartLive_NegativeOffset_AdvancesAudio_ByDroppingTheStreamHead` in `AudioPipelineTests.cs` —
|
||||
−40 ms advance (8-tick budget at 5ms interval), emits 6×0.9 fresh right after StartLive (≤
|
||||
budget, so the head MUST be eaten) then a long 0.2 bed; asserts the wire max ≈ 0.2 (< 0.3).
|
||||
Fails loudly if −N no longer drops the head (0.9 leaks).
|
||||
**Pipeline (per session):** `CreateCoreWebView2CompositionControllerAsync(appHwnd)` →
|
||||
`RootVisualTarget` = a `RelativeSizeAdjustment=1,1` child under a 1920×1080 root `ContainerVisual`
|
||||
on one `Compositor` (CoreMessaging DQ P/Invoke recipe — see ai.md Slice 14 / MyMistakes) →
|
||||
`GraphicsCaptureItem.CreateFromVisual(root)` → free-threaded `Direct3D11CaptureFramePool` (2
|
||||
buffers) → `SoftwareBitmap.CreateCopyFromSurfaceAsync` (`BitmapAlphaMode.Straight`) →
|
||||
per-frame `FindContentBounds` bbox → `CropBounds` → epoch'd ring → dispatcher-coalesced crop copy
|
||||
into ONE shared `WriteableBitmap`.
|
||||
|
||||
**User directives (this session):**
|
||||
- Signed −500..+500 with positive=delay / negative=advance, relabel "AUDIO SYNC", default 0.
|
||||
- **Lock the sync control when live or recording** (done — `IsEditMode`).
|
||||
- **NOTE ONLY, no fix:** the live recording is completely different from the "recording
|
||||
results" shown in Chat view (recorded in `bugs.md` — do not rediscover as a surprise).
|
||||
- **Before 1.0:** write a detailed USER-DOC tutorial on the audio-sync feature (see TASKS.md
|
||||
note; add it to the gold-pass/1.0 checklist). `docs/` currently holds only the README image.
|
||||
**Files:** NEW `Services/WebCaptureFrameSource.cs`; reworked `Services/WebView2Manager.cs` (internal
|
||||
seam ctor `(Dispatcher, Func<string, IScreenCaptureSource>?)` for hermetic tests); DELETED
|
||||
`Services/CaptureScheduler.cs`; `MainViewModel.Web.cs` (`InitWebView2()` no-arg),
|
||||
`MainViewModel.Streaming.Operations.cs` (three `SetCaptureInterval` hooks gone — verified zero
|
||||
remaining), `MainWindow.xaml(.cs)` (`InitWebView2()` moved ctor→**Loaded**; `WebViewHostPanel`
|
||||
overlay deleted). `TransparentBackgroundScript` unchanged.
|
||||
|
||||
## Take verification so far (ring-backlog fix)
|
||||
**Tests:** reworked `ytLive.Tests/WebView2ManagerTests.cs` — dropped the 4 control-size tests +
|
||||
`CaptureScheduler_Drops…`; `FindContentBounds` tests moved to `WebCaptureFrameSource`; the ONE
|
||||
integration test (`Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds`) drives the
|
||||
seam with `FakeWebSource` + background-STA `DispatcherPump`: crop-sized preview published once →
|
||||
`CropBounds` + `IsOpaque=false` → back-to-back frames coalesce to latest. `RoundClipInteractionTests`
|
||||
comment updated (WebViewHostPanel gone). Suite **293/293 green**, app build **0 warnings**. Docs
|
||||
slant (ai.md Slice 14, HANDOFF, MyMistakes, TASK 17, Controls/ViewModels/Services indexes) updated in
|
||||
the same working set.
|
||||
|
||||
Take `ty-20260914-1128-0000-2.mp4` (1920×1080@60fps, 356 frames / 5.93s): audio clap RMS peak
|
||||
3.260s; video diff-frame 163 → 2.717s; **offset ≈ +0.54s** — collapsed from +2.11/2.22s; the
|
||||
remaining ~300ms was the baked-in `SyncOffsetMs=300` (now 0). Both streams `start_time=0` — not
|
||||
an `avoid_negative_ts` artifact.
|
||||
## ⚠️ Open items on this change (before it is PUSHABLE)
|
||||
|
||||
- **Device verification:** the composition path has never run against a real widget. Verify ~60fps
|
||||
web animation in a take (also: the OBS de-throttle flags stay; hidden-page JS throttling is fixed
|
||||
by them, capture pacing is now renderer-driven).
|
||||
- **No push yet** — web work is commit-local until the user says push.
|
||||
|
||||
## ✅ SHIPPED + PUSHED — signed audio sync, −500..+500 (`b22d08e`)
|
||||
|
||||
Positive = delay the mix (audio AHEAD — the `AudioSyncDelay` line, live-reactive). Negative =
|
||||
**advance** (audio BEHIND): OBS-style "eat the stream head" — `AudioMixer.StartLive` arms
|
||||
`_advanceSamplesRemaining = |N| ms`, `LiveLoopAsync` skips `min(budget, mixBuffer.Length)` off each
|
||||
write head while it lasts. Slider relabeled **AUDIO SYNC**, `−500..500`, locked while
|
||||
live/recording (`IsEditMode`). One regression test:
|
||||
`StartLive_NegativeOffset_AdvancesAudio_ByDroppingTheStreamHead`.
|
||||
|
||||
Take `ty-20260914-1128-0000-2.mp4`: **offset ≈ +0.54s** (collapsed from +2.11/2.22s; residual was
|
||||
the baked-in `SyncOffsetMs=300`, now 0 — confirmed via sqlite3). Both streams `start_time=0`.
|
||||
|
||||
## Open threads
|
||||
|
||||
- **Verify signed sync on device:** re-run the clap take with a NEGATIVE offset to confirm the
|
||||
advance direction end-to-end (the regression test proves the mixer; a take proves the file).
|
||||
- **Audio-silence verification** — fixed code (`724af14`) confirmed; creator heard real audio.
|
||||
- **Verify signed sync negative direction on device** (clap take with a negative offset).
|
||||
- **Audio-silence verification** — fixed (`724af14`); creator heard real audio.
|
||||
- **Webcam MJPG missing / ~10–14Hz**, layer SortOrder, truncation-with-dynamic-scenes — queued.
|
||||
- **Sync control tutorial in user docs — REQUIRED before 1.0** (creator directive).
|
||||
- **Sync control user-doc tutorial — REQUIRED before 1.0** (creator directive; TASK 22 file).
|
||||
- **Device verify the composition web capture (this change).**
|
||||
|
||||
## Landmines
|
||||
|
||||
- testhost shares startup.log — filter by time.
|
||||
- `cmd.exe /c "taskkill /F /IM ytLive.exe"` (WSL double-slashes mangle) before rebuilds.
|
||||
- Build/tests: **Windows dotnet host** (`/mnt/c/Program Files/dotnet/dotnet.exe`). 0 warnings.
|
||||
- Build/tests: **Windows dotnet host** (`/mnt/c/Program Files/dotnet/dotnet.exe`). 0 warnings —
|
||||
only `./scripts/verify.sh "<files>"`'s clean build counts.
|
||||
- ffmpeg/ffprobe: `/mnt/c/Program Files/Krita (x64)/bin/` with Windows paths.
|
||||
- `MyMistakes.md` has the **audio/video sync measurement recipe** (claps + cross-correlation)
|
||||
— grep it before re-deriving.
|
||||
- `MyMistakes.md` has the **audio/video sync measurement recipe** AND the **composition-capture
|
||||
CoreMessaging DQ recipe (2026-09-14)** — grep before re-deriving.
|
||||
- sqlite3 lives at `/home/gramps/android-sdk/platform-tools/sqlite3` (WSL) for the DB at
|
||||
`/mnt/c/Users/gramp/AppData/Roaming/ytLlive/ytLlive.db`.
|
||||
|
||||
## Next step
|
||||
|
||||
Open queue question: user-doc tutorial for the audio-sync feature (creator directive, REQUIRED
|
||||
before 1.0 — recorded in `TASKS/task-22-audio-sync-offset.md`). Also asked 2026-09-14 whether
|
||||
other root md files need the TASKS-style split — answered: no (see below for the criterion).
|
||||
From this dirty state: run `./scripts/verify.sh "<all changed files>"`, review the diff, commit
|
||||
locally (web work — NO push, per standing directive). Then a device take to verify ~60fps web
|
||||
animation, then present for push review.
|
||||
@@ -96,10 +96,6 @@
|
||||
<!-- ═══ OVERLAY: Settings / Bug / Feature / About ═══ -->
|
||||
<controls:OverlayHost Grid.RowSpan="3"/>
|
||||
|
||||
<!-- ═══ WEB SOURCE HOST: hidden panel for WebView2 controls ═══ -->
|
||||
<Canvas x:Name="WebViewHostPanel" Grid.RowSpan="4"
|
||||
Opacity="0" IsHitTestVisible="False" Background="Transparent"/>
|
||||
|
||||
<!-- ═══ TOASTS: non-blocking notifications (bottom-right, above footer) ═══ -->
|
||||
<notifications:NotificationArea x:Name="ToastArea"
|
||||
Grid.RowSpan="4"
|
||||
|
||||
+4
-1
@@ -22,7 +22,6 @@ public partial class MainWindow : Window
|
||||
InitializeComponent();
|
||||
AppLog.Write("MainWindow ctor: after InitializeComponent");
|
||||
_viewModel = (MainViewModel)DataContext;
|
||||
_viewModel.InitWebView2(WebViewHostPanel);
|
||||
_viewModel.PropertyChanged += OnViewModelPropertyChanged;
|
||||
_topBar = (Controls.TopBar)FindName("TopBar")!;
|
||||
_previewPane = (Controls.PreviewPane)FindName("PreviewPane")!;
|
||||
@@ -37,6 +36,10 @@ public partial class MainWindow : Window
|
||||
private void MainWindow_Loaded(object sender, RoutedEventArgs e)
|
||||
{
|
||||
AppLog.Write("MainWindow loaded");
|
||||
// Web sources need the window HWND as the composition controller's parent,
|
||||
// which does not exist in the ctor — InitWebView2 runs here (Loaded), after
|
||||
// SourceInitialized. Nothing renders from web frames before this.
|
||||
_viewModel.InitWebView2();
|
||||
}
|
||||
|
||||
// ─── Window geometry persistence (launch size/position) ───
|
||||
|
||||
+38
-1
@@ -418,7 +418,8 @@ CLR→real-ffmpeg run to the native Windows suite.
|
||||
"not a regression" for weeks without ever recording WHY.)
|
||||
|
||||
**The trap:** `VisualTreeHelper.HitTest(window, pt)` returned the window's
|
||||
`WebViewHostPanel` overlay (`IsHitTestVisible="False"`, `Opacity=0`, ZERO children) for
|
||||
`WebViewHostPanel` overlay (`IsHitTestVisible="False"`, `Opacity=0`, ZERO children — since the
|
||||
2026-09-14 composition-capture redesign the overlay is DELETED, but the API lesson stands) for
|
||||
EVERY point in the window — so a "corner is grabbable" assertion could never pass, and
|
||||
it looked like a real interaction bug. The actual input pipeline (`UIElement.InputHitTest`,
|
||||
what Mouse routing uses) correctly returned the element's Grid at elem-center/corner-in/
|
||||
@@ -508,6 +509,42 @@ while live (`IsEditMode`) since a mid-stream advance flip is meaningless. Test r
|
||||
`StartLive_NegativeOffset_AdvancesAudio_ByDroppingTheStreamHead` (emit 6×0.9 into an 8-tick budget,
|
||||
long 0.2 bed, wire must show only 0.2).
|
||||
|
||||
## Composition video capture of WebView2: the CoreMessaging DQ recipe + the build gotchas
|
||||
|
||||
(2026-09-14, the ~20→60fps web-capture redesign — record-once so the next GPU-path work never
|
||||
re-derives it. Everything that follows was already settled by OBS/Flutter `webview_windows`; the
|
||||
cost this session was the 19041 projection's gaps.)
|
||||
|
||||
1. **No `DispatcherQueueController.CreateOnCurrentThread()` on the 10.0.19041 projection**
|
||||
(CS0117 — only `CreateOnDedicatedThread` + `FromAbi(IntPtr)` exist). P/Invoke
|
||||
`coreMessaging.dll!CreateDispatcherQueueController` with a sequential
|
||||
`DispatcherQueueOptions { DwSize, ThreadType=2 (DQTYPE_THREAD_CURRENT), ApartmentType=2 (DQTAT_COM_STA) }`,
|
||||
wrap via `DispatcherQueueController.FromAbi(ptr)` (mirror of `CaptureInterop`), THEN `new Compositor()`.
|
||||
One controller + one compositor per UI thread, created once.
|
||||
2. **`CoreWebView2CompositionController` needs a real parent HWND** — there is no window in the
|
||||
MainWindow ctor, so `InitWebView2()` moved from the ctor to `MainWindow_Loaded` (source
|
||||
registration is guarded by `ContainsKey`, so the layout-load path is safe).
|
||||
3. **Capture the ROOT visual, not the control.** `GraphicsItem` for a composition controller comes
|
||||
from `GraphicsCaptureItem.CreateFromVisual(root)` where `root` is your own 1920×1080
|
||||
`ContainerVisual` (`IsVisible=true`) holding the controller's `RootVisualTarget` as a
|
||||
`RelativeSizeAdjustment=1,1` child — exactly what `webview_windows`'s `graphics_context.cc`
|
||||
does (`CreateGraphicsCaptureItemFromVisual` on the root `surface_` visual). Hide nothing,
|
||||
move nothing, poll nothing — the capture is frame-driven at the renderer's pace.
|
||||
4. **Straight alpha, never premultiplied.** Read back with `BitmapAlphaMode.Straight`; the
|
||||
compositor's blend is straight-alpha, so `Premultiplied` readback wrecks corner anti-aliasing.
|
||||
5. **Namespace landmines that each cost a build cycle in `Services/`:** `Compositor` resolves to the
|
||||
repo's OWN `ytLive.Services.Compositor` namespace — fully-qualify `Windows.UI.Composition.Compositor`;
|
||||
`CoreWebView2CompositionController.Close()` is the disposal call (no `Dispose()`); `Color` is
|
||||
ambiguous (`System.Drawing` vs `System.Windows.Media`) — qualify `System.Drawing.Color.Transparent`.
|
||||
6. **Testing a compositor-built bitmap without a runtime:** the internal seam ctor
|
||||
`(Dispatcher, Func<string, IScreenCaptureSource>?)` + a `FakeWebSource` + a real background-STA
|
||||
`DispatcherPump` (borrowed from ScreenCaptureManagerTests) drives the whole session path hermetic.
|
||||
When byte-comparing a crop, slice the source with stride gaps (helper `CropBytes`) — a contiguous
|
||||
range silently spans rows.
|
||||
|
||||
Recipe verified on green suite + 0-warning build; the composition path itself still needs a device
|
||||
take (open item, HANDOFF).
|
||||
|
||||
## A pipe-read test that samples one tick is a timing flake by construction
|
||||
|
||||
(2026-09-14, re-discovered) `Mix_HonorsProviderGains_AndGameMute_KillsTheLoopback` fails
|
||||
|
||||
@@ -1,73 +0,0 @@
|
||||
using System;
|
||||
using System.Threading.Tasks;
|
||||
using System.Windows.Threading;
|
||||
|
||||
namespace ytLive.Services;
|
||||
|
||||
/// <summary>Per-session web capture pace: a DispatcherTimer that drops ticks while a
|
||||
/// capture is in flight (latest-wins). Raising the naive 10Hz cadence to ~30Hz would
|
||||
/// stack concurrent CapturePreviewAsync calls (a full 1920×1080 PNG encode costs
|
||||
/// ~10-30ms), so overlapping ticks are dropped, never queued; the effective cadence
|
||||
/// is max(interval, capture duration). The interval and the in-flight drop live here,
|
||||
/// not in the manager, so the scheduler is unit-testable without a WebView2 runtime.</summary>
|
||||
internal sealed class CaptureScheduler
|
||||
{
|
||||
private readonly Dispatcher _dispatcher;
|
||||
private readonly Func<Task> _capture;
|
||||
private DispatcherTimer? _timer;
|
||||
private int _intervalMs;
|
||||
private bool _inFlight;
|
||||
private Task _pending = Task.CompletedTask;
|
||||
|
||||
public CaptureScheduler(Dispatcher dispatcher, int intervalMs, Func<Task> capture)
|
||||
{
|
||||
_dispatcher = dispatcher;
|
||||
_intervalMs = Math.Max(16, intervalMs);
|
||||
_capture = capture;
|
||||
}
|
||||
|
||||
/// <summary>The task of the capture currently in flight, or a completed task when
|
||||
/// idle. Test seam: lets the caller await exactly the running capture.</summary>
|
||||
internal Task Pending => _pending;
|
||||
|
||||
public void Start()
|
||||
{
|
||||
if (_timer != null) return;
|
||||
_timer = new DispatcherTimer(
|
||||
TimeSpan.FromMilliseconds(_intervalMs),
|
||||
DispatcherPriority.Background,
|
||||
(_, _) => Tick(),
|
||||
_dispatcher);
|
||||
_timer.Start();
|
||||
}
|
||||
|
||||
public void Stop() => _timer?.Stop();
|
||||
|
||||
public void SetInterval(int milliseconds)
|
||||
{
|
||||
_intervalMs = Math.Max(16, milliseconds);
|
||||
if (_timer != null) _timer.Interval = TimeSpan.FromMilliseconds(_intervalMs);
|
||||
}
|
||||
|
||||
/// <summary>Runs the capture unless one is already in flight — the dropped tick is
|
||||
/// the latest-wins policy. Tick runs on the dispatcher thread (timer + test), so
|
||||
/// _inFlight needs no locking.</summary>
|
||||
internal void Tick()
|
||||
{
|
||||
if (_inFlight) return;
|
||||
_inFlight = true;
|
||||
_pending = RunCapture();
|
||||
}
|
||||
|
||||
private async Task RunCapture()
|
||||
{
|
||||
try
|
||||
{
|
||||
await _capture();
|
||||
}
|
||||
finally
|
||||
{
|
||||
_inFlight = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,232 @@
|
||||
using System;
|
||||
using System.Runtime.InteropServices;
|
||||
using System.Runtime.InteropServices.WindowsRuntime;
|
||||
using System.Threading.Tasks;
|
||||
using Windows.Graphics;
|
||||
using Windows.Graphics.Capture;
|
||||
using Windows.Graphics.DirectX;
|
||||
using Windows.Graphics.DirectX.Direct3D11;
|
||||
using Windows.Graphics.Imaging;
|
||||
using ytLive.Helpers;
|
||||
|
||||
namespace ytLive.Services;
|
||||
|
||||
/// <summary>
|
||||
/// The frame-driven web capture for one widget: a GraphicsCaptureItem backed by
|
||||
/// the widget's Windows.UI.Composition visual (see WebView2Manager). Owns the
|
||||
/// item, a free-threaded Direct3D11CaptureFramePool and the capture session;
|
||||
/// frames arrive in step with Chromium's own presentation (~60Hz on a real
|
||||
/// animation), replacing the old CapturePreviewAsync PNG poll whose ~35-165ms
|
||||
/// encode cost capped the recording at ~10Hz and the preview at ~5fps.
|
||||
///
|
||||
/// The capture surface holds a REAL alpha channel (the controller renders onto
|
||||
/// a transparent composition visual), so the GPU→CPU copy uses
|
||||
/// <see cref="BitmapAlphaMode.Straight"/> and the frame is
|
||||
/// <see cref="VideoFrame.IsOpaque"/> = false — the compositor's straight-alpha
|
||||
/// blend math ("web layer transparency", 2026-09-13) requires the readback to
|
||||
/// keep premultiplication OFF. Unlike the screen source there is no master-size
|
||||
/// downscale: the controller renders at the RAW 1920×1080 master canvas, so the
|
||||
/// CPU never holds a larger frame.
|
||||
///
|
||||
/// Per-frame the alpha bounding box of the widget (<see cref="FindContentBounds"/>)
|
||||
/// is measured on this capture worker thread and stamped onto the frame as
|
||||
/// <see cref="VideoFrame.CropBounds"/> — same alpha-bbox contract as the old
|
||||
/// PNG path, but off the UI thread (a 2M-pixel scan per frame there is exactly
|
||||
/// the cost this redesign exists to remove).
|
||||
/// </summary>
|
||||
public sealed class WebCaptureFrameSource : IScreenCaptureSource
|
||||
{
|
||||
private readonly object _gate = new();
|
||||
private GraphicsCaptureItem _item;
|
||||
private Direct3D11CaptureFramePool? _framePool;
|
||||
private GraphicsCaptureSession? _session;
|
||||
private SizeInt32 _poolSize;
|
||||
private bool _started;
|
||||
private bool _framePending;
|
||||
private DateTime _lastErrorLog = DateTime.MinValue;
|
||||
|
||||
// Buffer recycling: identical to the screen source's ring+Epoch contract
|
||||
// (see ScreenCaptureFrameSource) — the compositor's paste cache is keyed on
|
||||
// array identity, so recycled buffers must carry a monotonic Epoch or a
|
||||
// reused array false-hits with stale content.
|
||||
private readonly byte[]?[] _frameRing = new byte[8][];
|
||||
private int _ringNext;
|
||||
private long _epoch;
|
||||
|
||||
private static readonly TimeSpan ErrorLogThrottle = TimeSpan.FromSeconds(5);
|
||||
|
||||
public string Key { get; }
|
||||
public event Action<VideoFrame>? FrameAvailable;
|
||||
|
||||
internal WebCaptureFrameSource(string key, GraphicsCaptureItem item)
|
||||
{
|
||||
Key = key;
|
||||
_item = item;
|
||||
}
|
||||
|
||||
/// <summary>Wraps an item created from a composition visual by the web manager.</summary>
|
||||
public static WebCaptureFrameSource CreateForVisual(string key, GraphicsCaptureItem item)
|
||||
=> new(key, item);
|
||||
|
||||
public Task StartAsync()
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
if (_started) return Task.CompletedTask;
|
||||
var item = _item;
|
||||
var device = Direct3D11Helper.CreateDevice();
|
||||
var framePool = Direct3D11CaptureFramePool.CreateFreeThreaded(
|
||||
device, DirectXPixelFormat.B8G8R8A8UIntNormalized, 2, item.Size);
|
||||
_poolSize = item.Size;
|
||||
var session = framePool.CreateCaptureSession(item);
|
||||
framePool.FrameArrived += OnFrameArrived;
|
||||
session.StartCapture();
|
||||
_framePool = framePool;
|
||||
_session = session;
|
||||
_started = true;
|
||||
}
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public Task StopAsync()
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
_started = false;
|
||||
_framePending = false;
|
||||
if (_framePool != null)
|
||||
_framePool.FrameArrived -= OnFrameArrived;
|
||||
_session?.Dispose();
|
||||
_session = null;
|
||||
_framePool?.Dispose();
|
||||
_framePool = null;
|
||||
}
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
private void OnFrameArrived(Direct3D11CaptureFramePool sender, object args)
|
||||
{
|
||||
var frame = sender.TryGetNextFrame();
|
||||
if (frame == null) return;
|
||||
lock (_gate)
|
||||
{
|
||||
if (!_started)
|
||||
{
|
||||
frame.Dispose();
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
if (frame.ContentSize.Width != _poolSize.Width || frame.ContentSize.Height != _poolSize.Height)
|
||||
{
|
||||
sender.Recreate(Direct3D11Helper.CreateDevice(),
|
||||
DirectXPixelFormat.B8G8R8A8UIntNormalized, 2, frame.ContentSize);
|
||||
_poolSize = frame.ContentSize;
|
||||
}
|
||||
|
||||
if (_framePending)
|
||||
{
|
||||
frame.Dispose();
|
||||
return;
|
||||
}
|
||||
_framePending = true;
|
||||
_ = ProcessFrameAsync(frame);
|
||||
}
|
||||
|
||||
private async Task ProcessFrameAsync(Direct3D11CaptureFrame frame)
|
||||
{
|
||||
try
|
||||
{
|
||||
using (frame)
|
||||
using (var softwareBitmap = await SoftwareBitmap.CreateCopyFromSurfaceAsync(
|
||||
frame.Surface, BitmapAlphaMode.Straight))
|
||||
{
|
||||
FrameAvailable?.Invoke(CopyToVideoFrame(softwareBitmap));
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
var now = DateTime.UtcNow;
|
||||
if (now - _lastErrorLog >= ErrorLogThrottle)
|
||||
{
|
||||
_lastErrorLog = now;
|
||||
AppLog.Write($"WebCaptureFrameSource: frame conversion failed: {ex.Message}");
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
_framePending = false;
|
||||
}
|
||||
}
|
||||
|
||||
private byte[] RentRingBuffer(int size)
|
||||
{
|
||||
for (var tries = 0; tries < _frameRing.Length; tries++)
|
||||
{
|
||||
var idx = (_ringNext + tries) % _frameRing.Length;
|
||||
var buf = _frameRing[idx];
|
||||
if (buf is { Length: var len } && len == size)
|
||||
{
|
||||
_ringNext = (idx + 1) % _frameRing.Length;
|
||||
return buf;
|
||||
}
|
||||
}
|
||||
var slot = _ringNext;
|
||||
_ringNext = (slot + 1) % _frameRing.Length;
|
||||
var fresh = new byte[size];
|
||||
_frameRing[slot] = fresh;
|
||||
return fresh;
|
||||
}
|
||||
|
||||
private VideoFrame CopyToVideoFrame(SoftwareBitmap bitmap)
|
||||
{
|
||||
var sw = bitmap.PixelWidth;
|
||||
var sh = bitmap.PixelHeight;
|
||||
using var buffer = bitmap.LockBuffer(BitmapBufferAccessMode.Read);
|
||||
using var reference = buffer.CreateReference();
|
||||
if (!WindowsRuntimeMarshal.TryGetDataUnsafe(reference, out var data, out var capacity))
|
||||
throw new InvalidOperationException("Could not access the frame buffer.");
|
||||
|
||||
var srcStride = sw * 4;
|
||||
var count = (int)Math.Min(capacity, (uint)(sh * srcStride));
|
||||
var pixels = RentRingBuffer(count);
|
||||
Marshal.Copy(data, pixels, 0, pixels.Length);
|
||||
|
||||
var (cropX, cropY, cropW, cropH) = FindContentBounds(pixels, sw, sh, srcStride);
|
||||
return new VideoFrame(sw, sh, pixels)
|
||||
{
|
||||
IsOpaque = false,
|
||||
Epoch = ++_epoch,
|
||||
CropBounds = (cropX, cropY, cropW, cropH),
|
||||
};
|
||||
}
|
||||
|
||||
// The widget's extent is measured from the captured frame itself: the
|
||||
// bounding box of non-transparent pixels (the background is injected
|
||||
// transparent, so the alpha channel IS the widget). Immune to DOM timing
|
||||
// (fonts/images/animations), needs no JS round-trip, and CANNOT break the
|
||||
// render: a widget that fills the canvas gets the full frame; a widget that
|
||||
// floats in transparent margin gets exactly its content box — used for the
|
||||
// preview bitmap / selection. The frame handed to the compositor is always
|
||||
// the FULL canvas, so the page's transparent margins keep revealing the
|
||||
// layers beneath (the "black square over webcam" was the crop fed to the
|
||||
// compositor: UniformToFill zoomed the opaque content box over the element).
|
||||
internal static (int X, int Y, int W, int H) FindContentBounds(byte[] pixels, int pixW, int pixH, int stride)
|
||||
{
|
||||
int minX = pixW, minY = pixH, maxX = -1, maxY = -1;
|
||||
for (var y = 0; y < pixH; y++)
|
||||
{
|
||||
var row = y * stride;
|
||||
for (var x = 0; x < pixW; x++)
|
||||
{
|
||||
if (pixels[row + x * 4 + 3] == 0) continue;
|
||||
if (x < minX) minX = x;
|
||||
if (x > maxX) maxX = x;
|
||||
if (y < minY) minY = y;
|
||||
if (y > maxY) maxY = y;
|
||||
}
|
||||
}
|
||||
if (maxX < minX || maxY < minY) return (0, 0, pixW, pixH);
|
||||
return (minX, minY, maxX - minX + 1, maxY - minY + 1);
|
||||
}
|
||||
}
|
||||
+308
-310
@@ -1,129 +1,92 @@
|
||||
using System;
|
||||
using System.Diagnostics;
|
||||
using System.Collections.Generic;
|
||||
using System.Drawing;
|
||||
using System.IO;
|
||||
using System.Text.Json;
|
||||
using System.Runtime.InteropServices;
|
||||
using System.Threading.Tasks;
|
||||
using System.Windows;
|
||||
using System.Windows.Controls;
|
||||
using System.Windows.Media;
|
||||
using System.Windows.Media.Imaging;
|
||||
using System.Windows.Threading;
|
||||
using Microsoft.Web.WebView2.Core;
|
||||
using Microsoft.Web.WebView2.Wpf;
|
||||
using Windows.Graphics.Capture;
|
||||
using Windows.System;
|
||||
using Windows.UI.Composition;
|
||||
using ytLive.Helpers;
|
||||
using ytLive.Models;
|
||||
|
||||
namespace ytLive.Services;
|
||||
|
||||
/// <summary>
|
||||
/// Owns the composition-based web sources (one per widget). A session renders
|
||||
/// its widget through a <see cref="CoreWebView2CompositionController"/> whose
|
||||
/// visual tree lives off-screen on the WPF UI thread (OBS web-source model:
|
||||
/// the page renders at the MASTER CANVAS size 1920×1080, stable, never tracked);
|
||||
/// the controller's root visual is captured by a <see cref="WebCaptureFrameSource"/>
|
||||
/// at Chromium's own presentation cadence. The preview is crop-copied to a shared
|
||||
/// WriteableBitmap on the UI thread (dispatcher-coalesced latest-wins, mirror of
|
||||
/// ScreenCaptureManager); the FramePump reads <see cref="GetLatestFrame"/>.
|
||||
///
|
||||
/// The composition stack needs a DispatcherQueue on this thread (CsWinRT does not
|
||||
/// project <c>CreateOnCurrentThread</c> on the 19041 SDK — only the dedicated
|
||||
/// variant — so it is installed via a CoreMessaging P/Invoke, the documented
|
||||
/// fallback the WebView2CompositionControl / webview_windows embedders use).
|
||||
/// Everything composition-side therefore runs on the WPF UI thread; only the
|
||||
/// frame conversion and alpha-bbox scan run on the capture worker.
|
||||
///
|
||||
/// Seam: an internal <c>Func<string, IScreenCaptureSource></c> source
|
||||
/// factory substitutes the whole WebView2+item creation in tests — the manager
|
||||
/// then owns the session/coalesce/crop/preview plumbing exactly as in production.
|
||||
/// </summary>
|
||||
public sealed class WebView2Manager : IDisposable
|
||||
{
|
||||
private readonly Panel _hostPanel;
|
||||
private readonly Dispatcher _dispatcher;
|
||||
private readonly Func<string, IScreenCaptureSource>? _sessionSourceFactory;
|
||||
private readonly Dictionary<string, WebSourceSession> _sessions = new();
|
||||
private Task<CoreWebView2Environment>? _environmentTask;
|
||||
private int _captureIntervalMs = 200; // idle pace; recording drops it to ~30Hz via SetCaptureInterval
|
||||
private DispatcherQueueController? _dispatcherQueueController;
|
||||
private Windows.UI.Composition.Compositor? _compositor;
|
||||
|
||||
public event Action<string, WriteableBitmap>? PreviewBitmapChanged;
|
||||
|
||||
private sealed class WebSourceSession
|
||||
{
|
||||
public required WebView2 Control { get; init; }
|
||||
public required CaptureScheduler Scheduler { get; init; }
|
||||
public required string Key { get; init; }
|
||||
public CoreWebView2? Web;
|
||||
public CoreWebView2CompositionController? Controller;
|
||||
public IScreenCaptureSource? Source;
|
||||
public Action<VideoFrame>? FrameHandler;
|
||||
public VideoFrame? LatestFrame;
|
||||
public WriteableBitmap? PreviewBitmap;
|
||||
|
||||
// Buffer reuse (2026-09-04 lesson): the preview crop copy reuses one
|
||||
// scratch array while the widget's extent holds — no per-frame LOH mint.
|
||||
public byte[]? CropScratch;
|
||||
public bool Disposed;
|
||||
public bool Initialized;
|
||||
|
||||
// Buffer reuse (2026-09-04): every capture tick used to mint a fresh
|
||||
// 8.3MB canvas array + a fresh crop array (~83MB/s of LOH at 10Hz →
|
||||
// gen2 pauses surfaced as the pump's 35-40ms "worst render" spikes).
|
||||
// CanvasScratch is reused while the size holds. The frame handed to the
|
||||
// compositor is always the FULL canvas (transparent margins reveal layers
|
||||
// beneath); the crop is only for the preview WriteableBitmap. Epoch is
|
||||
// monotonic per canvas generation (identity-keyed consumers see each frame).
|
||||
public byte[]? CanvasScratch;
|
||||
public readonly byte[]?[] OutRing = new byte[8][];
|
||||
public int OutRingNext;
|
||||
public long Epoch;
|
||||
public WriteableBitmap? PreviewBitmap;
|
||||
public bool DebugPngWritten;
|
||||
public int WidgetDumpRemaining;
|
||||
public int TelemetryCaptureCount;
|
||||
|
||||
public byte[] RentOutBuffer(int size)
|
||||
{
|
||||
for (var tries = 0; tries < OutRing.Length; tries++)
|
||||
{
|
||||
var idx = (OutRingNext + tries) % OutRing.Length;
|
||||
var buf = OutRing[idx];
|
||||
if (buf != null && buf.Length == size)
|
||||
{
|
||||
OutRingNext = (idx + 1) % OutRing.Length;
|
||||
return buf;
|
||||
}
|
||||
}
|
||||
var slot = OutRingNext;
|
||||
OutRingNext = (slot + 1) % OutRing.Length;
|
||||
var fresh = new byte[size];
|
||||
OutRing[slot] = fresh;
|
||||
return fresh;
|
||||
}
|
||||
public bool FramePending;
|
||||
public bool FirstCaptureLogged;
|
||||
public int WidgetLogsRemaining;
|
||||
}
|
||||
|
||||
public WebView2Manager(Panel hostPanel, Dispatcher dispatcher)
|
||||
public WebView2Manager(Dispatcher dispatcher)
|
||||
: this(dispatcher, null)
|
||||
{
|
||||
}
|
||||
|
||||
/// <summary>Test seam: the source factory replaces WebView2+composition
|
||||
/// creation, so the session/coalesce/crop/preview plumbing is testable
|
||||
/// without the runtime.</summary>
|
||||
internal WebView2Manager(Dispatcher dispatcher, Func<string, IScreenCaptureSource>? sessionSourceFactory)
|
||||
{
|
||||
_hostPanel = hostPanel;
|
||||
_dispatcher = dispatcher;
|
||||
_sessionSourceFactory = sessionSourceFactory;
|
||||
}
|
||||
|
||||
public void Register(Source source)
|
||||
{
|
||||
if (_sessions.ContainsKey(source.Id)) return;
|
||||
|
||||
// OBS model: the page renders at the MASTER CANVAS size (1920×1080),
|
||||
// stable, never tracked. Full-bleed widgets (designed for the canvas)
|
||||
// render fully instead of being clipped to a small box-sized viewport;
|
||||
// dragging the box never reflows the page. The capture is tightly
|
||||
// cropped to the widget's ALPHA bounding box (FindContentBounds) and
|
||||
// the display (Stretch=Fill) maps it flush under the box, anchored at
|
||||
// (0,0) — no DOM query, so no timing fragility; worst case is the
|
||||
// full frame, which is the confirmed-good rendering.
|
||||
var webView = new WebView2
|
||||
{
|
||||
Width = 1920,
|
||||
Height = 1080,
|
||||
DefaultBackgroundColor = System.Drawing.Color.Transparent,
|
||||
Visibility = System.Windows.Visibility.Visible,
|
||||
};
|
||||
Canvas.SetLeft(webView, -5000);
|
||||
Canvas.SetTop(webView, -5000);
|
||||
|
||||
_hostPanel.Children.Add(webView);
|
||||
|
||||
var scheduler = new CaptureScheduler(
|
||||
_dispatcher,
|
||||
_captureIntervalMs,
|
||||
() => CaptureFrame(source.Id));
|
||||
|
||||
var session = new WebSourceSession
|
||||
{
|
||||
Control = webView,
|
||||
Scheduler = scheduler,
|
||||
};
|
||||
|
||||
_sessions[source.Id] = session;
|
||||
|
||||
_ = InitializeAsync(source, session);
|
||||
}
|
||||
|
||||
/// <summary>Changes the capture cadence for every session. Recording runs at
|
||||
/// ~30Hz (33ms) — the practical PNG-capture ceiling — idle drops back to 200ms so
|
||||
/// the preview alone doesn't burn a core decoding full-HD captures.</summary>
|
||||
public void SetCaptureInterval(int milliseconds)
|
||||
{
|
||||
_captureIntervalMs = milliseconds;
|
||||
foreach (var session in _sessions.Values)
|
||||
if (!session.Disposed)
|
||||
session.Scheduler.SetInterval(milliseconds);
|
||||
_sessions[source.Id] = new WebSourceSession { Key = source.Id };
|
||||
_ = InitializeAsync(source, _sessions[source.Id]);
|
||||
}
|
||||
|
||||
public void Unregister(string sourceId) => RemoveSession(sourceId);
|
||||
@@ -136,7 +99,7 @@ public sealed class WebView2Manager : IDisposable
|
||||
if (string.IsNullOrEmpty(source.WebUri))
|
||||
{
|
||||
session.LatestFrame = null;
|
||||
session.Control.CoreWebView2?.NavigateToString(
|
||||
session.Web?.NavigateToString(
|
||||
"<html><body style='background:transparent;margin:0'></body></html>");
|
||||
return;
|
||||
}
|
||||
@@ -152,9 +115,8 @@ public sealed class WebView2Manager : IDisposable
|
||||
// Injected BEFORE the document is parsed (AddScriptToExecuteOnDocumentCreatedAsync) so
|
||||
// the widget page's own CSS/scripts cannot repaint html/body opaque — the reason the
|
||||
// late NavigationStarting/NavigationCompleted ExecuteScriptAsync injection lost the
|
||||
// fight (page styles ran first). CapturePreviewAsync "will always honor a webpage's
|
||||
// background content" (MicrosoftEdge/WebView2Feedback specs/BackgroundColor.md), so a
|
||||
// transparent capture REQUIRES the transparency to be in place before the page paints.
|
||||
// fight (page styles ran first). A transparent capture REQUIRES the transparency in
|
||||
// place before the page paints, on the composition surface too.
|
||||
//
|
||||
// The OLD inline `element.style.background='transparent'` form lost to any page CSS:
|
||||
// a widget that paints a background-COLOR on a container (or html/body with stronger
|
||||
@@ -191,55 +153,75 @@ public sealed class WebView2Manager : IDisposable
|
||||
{
|
||||
try
|
||||
{
|
||||
await session.Control.EnsureCoreWebView2Async(await GetEnvironmentAsync());
|
||||
|
||||
var cws = session.Control.CoreWebView2!;
|
||||
|
||||
// Pre-page hook: runs after the global object exists but BEFORE the document
|
||||
// parses / any page script runs — must complete before navigating away from
|
||||
// about:blank for it to apply to the widget document.
|
||||
await cws.AddScriptToExecuteOnDocumentCreatedAsync(TransparentBackgroundScript);
|
||||
|
||||
cws.NavigationStarting += (_, _) =>
|
||||
IScreenCaptureSource sourceImpl;
|
||||
if (_sessionSourceFactory != null)
|
||||
{
|
||||
_ = cws.ExecuteScriptAsync(TransparentBackgroundScript);
|
||||
};
|
||||
|
||||
cws.NavigationCompleted += (_, e) =>
|
||||
// Test path: the factory hands over a fully-configured source; the
|
||||
// branch has no awaits, so Register() completes the session inline.
|
||||
sourceImpl = _sessionSourceFactory(source.Id);
|
||||
}
|
||||
else
|
||||
{
|
||||
_ = cws.ExecuteScriptAsync(TransparentBackgroundScript);
|
||||
if (e.IsSuccess && session.Control.CoreWebView2?.Source is { } src && src != "about:blank")
|
||||
var controller = await CreateControllerAsync();
|
||||
if (controller == null)
|
||||
{
|
||||
// Arm a one-shot diagnostic dump of the REAL widget document
|
||||
// (a few post-paint captures so styles/backgrounds settle) —
|
||||
// the old first-capture dump only ever saw the about:blank
|
||||
// document, so whether this widget's capture actually HAS the
|
||||
// transparent margins the injection promises has never been
|
||||
// verified (MyMistakes: "? VERIFY").
|
||||
session.WidgetDumpRemaining = 5;
|
||||
// The first five dumps fire within ~1s of completion (usually BEFORE the
|
||||
// recording starts) — re-arm ~30s in so the same real document gets dumped
|
||||
// INSIDE a recording's window too. Next take therefore nukes the gap that
|
||||
// hid what the page actually painted while recording.
|
||||
var rearm = session;
|
||||
_ = Task.Run(async () =>
|
||||
{
|
||||
await Task.Delay(TimeSpan.FromSeconds(30));
|
||||
rearm.WidgetDumpRemaining = 5;
|
||||
});
|
||||
RemoveSession(source.Id);
|
||||
return;
|
||||
}
|
||||
_ = CaptureFrame(source.Id);
|
||||
};
|
||||
|
||||
session.Controller = controller;
|
||||
session.Web = controller.CoreWebView2;
|
||||
if (session.Web == null)
|
||||
{
|
||||
RemoveSession(source.Id);
|
||||
return;
|
||||
}
|
||||
|
||||
// Pre-page hook: runs after the global object exists but BEFORE the document
|
||||
// parses / any page script runs — must complete before navigating away from
|
||||
// about:blank for it to apply to the widget document.
|
||||
await session.Web.AddScriptToExecuteOnDocumentCreatedAsync(TransparentBackgroundScript);
|
||||
|
||||
session.Web.NavigationStarting += (_, _) =>
|
||||
{
|
||||
_ = session.Web?.ExecuteScriptAsync(TransparentBackgroundScript);
|
||||
};
|
||||
|
||||
session.Web.NavigationCompleted += (_, e) =>
|
||||
{
|
||||
_ = session.Web?.ExecuteScriptAsync(TransparentBackgroundScript);
|
||||
if (e.IsSuccess && session.Web?.Source is { } src && src != "about:blank")
|
||||
{
|
||||
// Log-only diagnostic on the REAL widget document (a few post-paint
|
||||
// frames so styles/backgrounds settle). The first-capture dump only
|
||||
// ever saw about:blank; the old PNG widget dumps are gone with the
|
||||
// PNG path — alpha stats now ride the frame-driven channel.
|
||||
session.WidgetLogsRemaining = 3;
|
||||
var rearm = session;
|
||||
_ = Task.Run(async () =>
|
||||
{
|
||||
await Task.Delay(TimeSpan.FromSeconds(30));
|
||||
if (!rearm.Disposed) rearm.WidgetLogsRemaining = 3;
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
session.Web.NavigateToString(
|
||||
"<html><body style='background:transparent;margin:0'></body></html>");
|
||||
|
||||
if (!string.IsNullOrEmpty(source.WebUri))
|
||||
Navigate(source.WebUri, session);
|
||||
|
||||
session.Initialized = true;
|
||||
|
||||
sourceImpl = new WebCaptureFrameSource(source.Id, CreateCaptureItem(controller));
|
||||
}
|
||||
|
||||
session.Source = sourceImpl;
|
||||
session.FrameHandler = frame => OnFrameAvailable(session, frame);
|
||||
sourceImpl.FrameAvailable += session.FrameHandler;
|
||||
await sourceImpl.StartAsync();
|
||||
session.Initialized = true;
|
||||
|
||||
session.Control.CoreWebView2.NavigateToString(
|
||||
"<html><body style='background:transparent;margin:0'></body></html>");
|
||||
|
||||
if (!string.IsNullOrEmpty(source.WebUri))
|
||||
Navigate(source.WebUri, session);
|
||||
|
||||
session.Scheduler.Start();
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -248,14 +230,13 @@ public sealed class WebView2Manager : IDisposable
|
||||
}
|
||||
}
|
||||
|
||||
// The widget page is rendered off-screen; Chromium throttles hidden pages
|
||||
// (rAF parked, timers clamped to ~1s) the moment the app window is unfocused or
|
||||
// covered — WebView2Feedback#1172/#3070, Chrome 88 timer throttling. These flags
|
||||
// (the Electron-class embedder answer) keep the renderer at full rate:
|
||||
// --disable-backgrounding-occluded-windows, --disable-renderer-backgrounding,
|
||||
// and disable the native-window occlusion detector that marks the page hidden.
|
||||
// The environment MUST exist before EnsureCoreWebView2Async — it's created once
|
||||
// and shared by every session (one browser process).
|
||||
// The environment MUST exist before the controller is created — it's created once
|
||||
// and shared by every session (one browser process). These flags (the
|
||||
// Electron-class embedder answer to Chromium throttling hidden pages — the moment
|
||||
// the app window is unfocused or covered, rAF parks and timers clamp to ~1s) keep
|
||||
// the renderer at full rate: --disable-backgrounding-occluded-windows,
|
||||
// --disable-renderer-backgrounding, and disable the native-window occlusion
|
||||
// detector that marks the page hidden.
|
||||
private Task<CoreWebView2Environment> GetEnvironmentAsync()
|
||||
{
|
||||
if (_environmentTask != null) return _environmentTask;
|
||||
@@ -267,45 +248,177 @@ public sealed class WebView2Manager : IDisposable
|
||||
return _environmentTask = CoreWebView2Environment.CreateAsync(null, null, options);
|
||||
}
|
||||
|
||||
/// <summary>Creates the composition controller off-screen under the main
|
||||
/// window (the parent HWND is required, the visual never mounts a window
|
||||
/// target). Set up is copied from the WebView2CompositionControl /
|
||||
/// webview_windows embedders: raw-pixel bounds at the master canvas,
|
||||
/// fixed rasterization scale, transparent default background.</summary>
|
||||
private async Task<CoreWebView2CompositionController?> CreateControllerAsync()
|
||||
{
|
||||
var mainWindow = Application.Current?.MainWindow;
|
||||
var hwnd = mainWindow == null
|
||||
? IntPtr.Zero
|
||||
: new System.Windows.Interop.WindowInteropHelper(mainWindow).Handle;
|
||||
if (hwnd == IntPtr.Zero)
|
||||
{
|
||||
AppLog.Write("WebView2Manager: main window handle unavailable — web source not created");
|
||||
return null;
|
||||
}
|
||||
|
||||
var env = await GetEnvironmentAsync();
|
||||
if (_compositor == null) _compositor = CreateCompositorForCurrentThread();
|
||||
|
||||
var controller = await env.CreateCoreWebView2CompositionControllerAsync(hwnd);
|
||||
controller.Bounds = new Rectangle(0, 0, 1920, 1080);
|
||||
controller.BoundsMode = CoreWebView2BoundsMode.UseRawPixels;
|
||||
controller.ShouldDetectMonitorScaleChanges = false;
|
||||
controller.RasterizationScale = 1.0;
|
||||
controller.IsVisible = true;
|
||||
controller.DefaultBackgroundColor = System.Drawing.Color.Transparent;
|
||||
return controller;
|
||||
}
|
||||
|
||||
private void Navigate(string url, WebSourceSession session)
|
||||
{
|
||||
try { session.Control.CoreWebView2?.Navigate(url); }
|
||||
try { session.Web?.Navigate(url); }
|
||||
catch (Exception ex) { AppLog.Write($"WebView2Manager: navigate failed: {ex.Message}"); }
|
||||
}
|
||||
|
||||
// The widget's extent is measured from the captured frame itself: the
|
||||
// bounding box of non-transparent pixels (the background is injected
|
||||
// transparent, so the alpha channel IS the widget). This is immune to
|
||||
// DOM timing (fonts/images/animations), needs no JS round-trip, and
|
||||
// CANNOT break the render: if the widget fills the canvas the box equals
|
||||
// the full frame; if it floats in transparent margin the box is the
|
||||
// measured content extent — used ONLY for the preview bitmap / selection.
|
||||
// The frame handed to the compositor is always the FULL canvas, so the
|
||||
// page's transparent margins keep revealing the layers beneath (the
|
||||
// "black square over webcam" was the crop fed to the compositor:
|
||||
// UniformToFill zoomed the opaque content box to cover the whole element).
|
||||
internal static (int X, int Y, int W, int H) FindContentBounds(byte[] pixels, int pixW, int pixH, int stride)
|
||||
// The visual tree webview_windows wires (offscreen mode): a root container
|
||||
// visual sized to the master canvas (the captured subject — the Flutter
|
||||
// plugin calls GraphicsCaptureItem.CreateFromVisual on this root), with the
|
||||
// controller's root visual target set to a child that fills the root via
|
||||
// RelativeSizeAdjustment. BoundsMode raw pixels + RasterizationScale 1.0 keep
|
||||
// the capture at exactly 1920×1080 regardless of monitor scale.
|
||||
private GraphicsCaptureItem CreateCaptureItem(CoreWebView2CompositionController controller)
|
||||
{
|
||||
int minX = pixW, minY = pixH, maxX = -1, maxY = -1;
|
||||
for (var y = 0; y < pixH; y++)
|
||||
{
|
||||
var row = y * stride;
|
||||
for (var x = 0; x < pixW; x++)
|
||||
{
|
||||
if (pixels[row + x * 4 + 3] == 0) continue;
|
||||
if (x < minX) minX = x;
|
||||
if (x > maxX) maxX = x;
|
||||
if (y < minY) minY = y;
|
||||
if (y > maxY) maxY = y;
|
||||
}
|
||||
}
|
||||
if (maxX < minX || maxY < minY) return (0, 0, pixW, pixH);
|
||||
return (minX, minY, maxX - minX + 1, maxY - minY + 1);
|
||||
var compositor = _compositor!;
|
||||
var root = compositor.CreateContainerVisual();
|
||||
root.Size = new System.Numerics.Vector2(1920f, 1080f);
|
||||
root.IsVisible = true;
|
||||
|
||||
var webViewVisual = compositor.CreateContainerVisual();
|
||||
webViewVisual.RelativeSizeAdjustment = new System.Numerics.Vector2(1f, 1f);
|
||||
root.Children.InsertAtTop(webViewVisual);
|
||||
|
||||
controller.RootVisualTarget = webViewVisual;
|
||||
return GraphicsCaptureItem.CreateFromVisual(root);
|
||||
}
|
||||
|
||||
// CsWinRT does not project DispatcherQueueController.CreateOnCurrentThread
|
||||
// on the 19041 SDK (only CreateOnDedicatedThread [CS0117, verified by probe]);
|
||||
// the documented fallback — "how to create a DispatcherQueue on the current
|
||||
// thread" (learn.microsoft.com CoreMessaging docs) — is CreateDispatcherQueueController
|
||||
// from CoreMessaging.dll, wrapped via FromAbi (the CaptureInterop pattern). One
|
||||
// per UI thread, then new Compositor() (which requires the DQ) works here.
|
||||
private Windows.UI.Composition.Compositor CreateCompositorForCurrentThread()
|
||||
{
|
||||
var options = new DispatcherQueueOptions
|
||||
{
|
||||
DwSize = Marshal.SizeOf<DispatcherQueueOptions>(),
|
||||
ThreadType = 2, // DQTYPE_THREAD_CURRENT
|
||||
ApartmentType = 2, // DQTAT_COM_STA — the WPF UI thread is STA
|
||||
};
|
||||
var hr = CreateDispatcherQueueController(options, out var controllerPtr);
|
||||
if (hr != 0)
|
||||
throw Marshal.GetExceptionForHR(hr)!;
|
||||
_dispatcherQueueController = DispatcherQueueController.FromAbi(controllerPtr);
|
||||
return new Windows.UI.Composition.Compositor();
|
||||
}
|
||||
|
||||
[StructLayout(LayoutKind.Sequential)]
|
||||
private struct DispatcherQueueOptions
|
||||
{
|
||||
public int DwSize;
|
||||
public int ThreadType;
|
||||
public int ApartmentType;
|
||||
}
|
||||
|
||||
[DllImport("CoreMessaging.dll")]
|
||||
private static extern int CreateDispatcherQueueController(
|
||||
DispatcherQueueOptions options, out IntPtr dispatcherQueueController);
|
||||
|
||||
private void OnFrameAvailable(WebSourceSession session, VideoFrame frame)
|
||||
{
|
||||
if (session.Disposed) return;
|
||||
session.LatestFrame = frame;
|
||||
|
||||
if (_dispatcher.CheckAccess())
|
||||
{
|
||||
UpdatePreview(session);
|
||||
return;
|
||||
}
|
||||
if (session.FramePending) return;
|
||||
session.FramePending = true;
|
||||
_dispatcher.BeginInvoke(() =>
|
||||
{
|
||||
session.FramePending = false;
|
||||
if (!session.Disposed)
|
||||
UpdatePreview(session);
|
||||
}, DispatcherPriority.Render);
|
||||
}
|
||||
|
||||
private void UpdatePreview(WebSourceSession session)
|
||||
{
|
||||
var frame = session.LatestFrame;
|
||||
if (frame == null) return;
|
||||
|
||||
var (cropX, cropY, cropW, cropH) = frame.CropBounds ?? (0, 0, frame.Width, frame.Height);
|
||||
if (cropW < 1 || cropH < 1)
|
||||
{
|
||||
cropX = 0; cropY = 0;
|
||||
cropW = frame.Width; cropH = frame.Height;
|
||||
}
|
||||
|
||||
// The preview is sized to the widget's alpha bounding box (the compositor
|
||||
// gets the FULL canvas + CropBounds; the crop is only for the preview).
|
||||
// A fresh box (widget resizes / first content paints) mints a new bitmap
|
||||
// once — WPF re-renders a reused mutable bitmap after WritePixels, so
|
||||
// later frames only re-copy into the same one.
|
||||
if (session.PreviewBitmap == null
|
||||
|| session.PreviewBitmap.PixelWidth != cropW
|
||||
|| session.PreviewBitmap.PixelHeight != cropH)
|
||||
{
|
||||
session.PreviewBitmap = new WriteableBitmap(cropW, cropH, 96, 96, PixelFormats.Bgra32, null);
|
||||
PreviewBitmapChanged?.Invoke(session.Key, session.PreviewBitmap);
|
||||
}
|
||||
|
||||
var need = cropW * cropH * 4;
|
||||
var scratch = session.CropScratch is { } s && s.Length >= need
|
||||
? s
|
||||
: (session.CropScratch = new byte[need]);
|
||||
var stride = frame.Stride;
|
||||
if (cropW == frame.Width && cropH == frame.Height && cropX == 0 && cropY == 0)
|
||||
Buffer.BlockCopy(frame.BgraPixels, 0, scratch, 0, need);
|
||||
else
|
||||
for (var r = 0; r < cropH; r++)
|
||||
Buffer.BlockCopy(frame.BgraPixels, (cropY + r) * stride + cropX * 4, scratch, r * cropW * 4, cropW * 4);
|
||||
|
||||
session.PreviewBitmap.WritePixels(new Int32Rect(0, 0, cropW, cropH), scratch, cropW * 4, 0);
|
||||
|
||||
// alpha[min/max/mean/zero%] — the one number that decides whether a widget's
|
||||
// capture really has transparent margins (0% = opaque, ~100% if the page
|
||||
// painted transparent-only). Sampled every 16th pixel like the original log.
|
||||
if (!session.FirstCaptureLogged)
|
||||
{
|
||||
session.FirstCaptureLogged = true;
|
||||
AppLog.Write(
|
||||
$"WebView2Manager: first capture for '{session.Key}' {frame.Width}x{frame.Height} " +
|
||||
$"{AlphaStats(frame.BgraPixels, frame.Width, frame.Height, stride)} " +
|
||||
$"contentBounds=({cropX},{cropY},{cropW},{cropH})");
|
||||
}
|
||||
|
||||
if (session.WidgetLogsRemaining > 0)
|
||||
{
|
||||
var dumpNo = 3 - session.WidgetLogsRemaining + 1;
|
||||
session.WidgetLogsRemaining--;
|
||||
AppLog.Write(
|
||||
$"WebView2Manager: widget capture [{dumpNo}/3] for '{session.Key}' {frame.Width}x{frame.Height} " +
|
||||
$"{AlphaStats(frame.BgraPixels, frame.Width, frame.Height, stride)} " +
|
||||
$"contentBounds=({cropX},{cropY},{cropW},{cropH})");
|
||||
}
|
||||
}
|
||||
|
||||
// alpha[min/max/mean/zero%] — the one number that decides whether a widget's
|
||||
// capture really has transparent margins (0% = opaque, ~100% if the page
|
||||
// painted transparent-only). Sampled every 16th pixel like the original log.
|
||||
private static string AlphaStats(byte[] pixels, int pixW, int pixH, int stride)
|
||||
{
|
||||
int minA = 255, maxA = 0;
|
||||
@@ -326,147 +439,32 @@ public sealed class WebView2Manager : IDisposable
|
||||
return $"alpha[min={minA},max={maxA},mean={sumA / (double)nA:F1},zero={100.0 * zeroA / nA:F1}%]";
|
||||
}
|
||||
|
||||
private async Task CaptureFrame(string sourceId)
|
||||
{
|
||||
if (!_sessions.TryGetValue(sourceId, out var session) || session.Disposed) return;
|
||||
|
||||
var sw = Stopwatch.StartNew();
|
||||
try
|
||||
{
|
||||
var webView = session.Control;
|
||||
if (webView.CoreWebView2 == null) return;
|
||||
|
||||
using var ms = new MemoryStream();
|
||||
await webView.CoreWebView2.CapturePreviewAsync(
|
||||
CoreWebView2CapturePreviewImageFormat.Png, ms);
|
||||
|
||||
if (!session.DebugPngWritten)
|
||||
{
|
||||
session.DebugPngWritten = true;
|
||||
try
|
||||
{
|
||||
ms.Position = 0;
|
||||
var dbg = Path.Combine(Path.GetTempPath(), $"ytLive-web-{sourceId}.png");
|
||||
File.WriteAllBytes(dbg, ms.ToArray());
|
||||
AppLog.Write($"WebView2Manager: debug capture dump -> {dbg}");
|
||||
}
|
||||
catch (Exception dex) { AppLog.Write($"WebView2Manager: debug dump failed: {dex.Message}"); }
|
||||
}
|
||||
|
||||
ms.Position = 0;
|
||||
|
||||
var bitmap = new BitmapImage();
|
||||
bitmap.BeginInit();
|
||||
bitmap.StreamSource = ms;
|
||||
bitmap.CacheOption = BitmapCacheOption.OnLoad;
|
||||
bitmap.EndInit();
|
||||
bitmap.Freeze();
|
||||
|
||||
var formatted = new FormatConvertedBitmap(bitmap,
|
||||
PixelFormats.Bgra32, null, 0);
|
||||
|
||||
var pixW = formatted.PixelWidth;
|
||||
var pixH = formatted.PixelHeight;
|
||||
|
||||
if (pixW < 1 || pixH < 1) return;
|
||||
|
||||
var need = pixW * pixH * 4;
|
||||
var pixels = session.CanvasScratch is { } scratch && scratch.Length >= need
|
||||
? scratch
|
||||
: (session.CanvasScratch = new byte[need]);
|
||||
formatted.CopyPixels(pixels, pixW * 4, 0);
|
||||
|
||||
var stride = pixW * 4;
|
||||
|
||||
var (cropX, cropY, cropW, cropH) = FindContentBounds(pixels, pixW, pixH, stride);
|
||||
|
||||
if (session.Epoch == 0)
|
||||
{
|
||||
AppLog.Write(
|
||||
$"WebView2Manager: first capture for '{sourceId}' {pixW}x{pixH} " +
|
||||
$"{AlphaStats(pixels, pixW, pixH, stride)} " +
|
||||
$"contentBounds=({cropX},{cropY},{cropW},{cropH})");
|
||||
}
|
||||
|
||||
if (session.WidgetDumpRemaining > 0)
|
||||
{
|
||||
var dumpNo = 5 - session.WidgetDumpRemaining + 1;
|
||||
session.WidgetDumpRemaining--;
|
||||
try
|
||||
{
|
||||
var dbg = Path.Combine(Path.GetTempPath(), $"ytLive-web-{sourceId}-w{dumpNo}.png");
|
||||
File.WriteAllBytes(dbg, ms.ToArray());
|
||||
AppLog.Write(
|
||||
$"WebView2Manager: widget capture [{dumpNo}/5] for '{sourceId}' {pixW}x{pixH} " +
|
||||
$"{AlphaStats(pixels, pixW, pixH, stride)} " +
|
||||
$"contentBounds=({cropX},{cropY},{cropW},{cropH}) -> {dbg}");
|
||||
}
|
||||
catch (Exception dex)
|
||||
{
|
||||
AppLog.Write($"WebView2Manager: widget dump failed: {dex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
var outPixels = session.RentOutBuffer(cropW * cropH * 4);
|
||||
if (cropW == pixW && cropH == pixH && cropX == 0 && cropY == 0)
|
||||
Buffer.BlockCopy(pixels, 0, outPixels, 0, outPixels.Length);
|
||||
else
|
||||
for (var r = 0; r < cropH; r++)
|
||||
Buffer.BlockCopy(pixels, (cropY + r) * stride + cropX * 4, outPixels, r * cropW * 4, cropW * 4);
|
||||
|
||||
if (session.PreviewBitmap == null
|
||||
|| session.PreviewBitmap.PixelWidth != cropW || session.PreviewBitmap.PixelHeight != cropH)
|
||||
session.PreviewBitmap = new WriteableBitmap(cropW, cropH, 96, 96, PixelFormats.Bgra32, null);
|
||||
session.PreviewBitmap.WritePixels(new Int32Rect(0, 0, cropW, cropH), outPixels, cropW * 4, 0);
|
||||
|
||||
session.LatestFrame = new VideoFrame(pixW, pixH, pixels)
|
||||
{
|
||||
Epoch = ++session.Epoch,
|
||||
CropBounds = (cropX, cropY, cropW, cropH),
|
||||
};
|
||||
|
||||
// The preview bitmap is now MUTABLE and reused (CameraManager precedent):
|
||||
// WPF re-renders it after WritePixels; handlers must not Freeze it.
|
||||
var preview = session.PreviewBitmap;
|
||||
_ = _dispatcher.BeginInvoke(() =>
|
||||
{
|
||||
PreviewBitmapChanged?.Invoke(sourceId, preview);
|
||||
});
|
||||
|
||||
// Capture-cost telemetry for the first ~30 captures of a session: PNG encode
|
||||
// (CapturePreviewAsync) + decode cost decide whether ~30Hz recording cadence
|
||||
// is affordable or must drop to ~20Hz. The FramePump drops frames (never
|
||||
// time-lapses) if the UI thread's GC churn starves it, so startup.log shows
|
||||
// the real budget before anything is tuned further.
|
||||
if (session.TelemetryCaptureCount < 30)
|
||||
{
|
||||
session.TelemetryCaptureCount++;
|
||||
AppLog.Write(
|
||||
$"WebView2Manager: capture #{session.TelemetryCaptureCount} for '{sourceId}' " +
|
||||
$"{pixW}x{pixH} took {sw.ElapsedMilliseconds}ms");
|
||||
}
|
||||
}
|
||||
catch (ObjectDisposedException) { }
|
||||
catch (InvalidOperationException) { }
|
||||
catch (Exception ex)
|
||||
{
|
||||
AppLog.Write($"WebView2Manager: capture failed for '{sourceId}': {ex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
private void RemoveSession(string sourceId)
|
||||
{
|
||||
if (!_sessions.TryGetValue(sourceId, out var session)) return;
|
||||
session.Disposed = true;
|
||||
session.Scheduler.Stop();
|
||||
_hostPanel.Children.Remove(session.Control);
|
||||
try { session.Control.Dispose(); } catch { }
|
||||
|
||||
var src = session.Source;
|
||||
if (src != null)
|
||||
{
|
||||
if (session.FrameHandler != null)
|
||||
src.FrameAvailable -= session.FrameHandler;
|
||||
_ = SafeStopAsync(src);
|
||||
}
|
||||
try { session.Controller?.Close(); } catch { }
|
||||
session.Web = null;
|
||||
_sessions.Remove(sourceId);
|
||||
}
|
||||
|
||||
private static async Task SafeStopAsync(IScreenCaptureSource source)
|
||||
{
|
||||
try { await source.StopAsync(); }
|
||||
catch (Exception ex) { AppLog.Write($"WebView2Manager: stopping web capture failed: {ex.Message}"); }
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
foreach (var id in _sessions.Keys.ToList())
|
||||
RemoveSession(id);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -28,6 +28,8 @@ External-facing logic: YouTube API, persistence. See
|
||||
| `ScreenCaptureFrameSource.cs` | One `Direct3D11CaptureFramePool` (free-threaded, 2 buffers) + session per target; frames → `SoftwareBitmap.CreateCopyFromSurfaceAsync` (BGRA, alpha ignored) → `VideoFrame`, bytes read via `WindowsRuntimeMarshal.TryGetDataUnsafe` (CsWinRT-safe — the `IMemoryBufferByteAccess` ComImport cast fails on every frame and is gone). Surfaces > 1920×1080 downscaled bilinearly to the master; conversion failures logged ≤ once/5 s. DRM content = black frames (OS limit). `CreateForMonitor`/`CreateForWindow`/`CreateForPicker` |
|
||||
| `ScreenCaptureManager.cs` | Screen-capture ownership mirroring `CameraManager`: refcounted by target key, one shared `WriteableBitmap`, dispatcher-coalesced latest-frame copies; `PreviewBitmapChanged`/`CaptureFailed` events; `ReleaseAllAsync` used on re-designation |
|
||||
| `ScreenCaptureSourceFactory.cs` | `Resolve(key)` parses `monitor:<n>` / `window:<hwnd>` / `picker:<name>` into a source; `PickAsync()` shows the OS `GraphicsCapturePicker` and returns the `picker:` key (transient — a reload falls back to auto-detection) |
|
||||
| `WebView2Manager.cs` | **Web-source ownership (TASK 17, composition-capture since 2026-09-14):** per registered `Source { Type: WebSource }` it owns a `CoreWebView2CompositionController` + its visual tree (`RootVisualTarget` child under a 1920×1080 root `ContainerVisual` on one `Windows.UI.Composition.Compositor` created on the UI thread via the CoreMessaging `CreateDispatcherQueueController` P/Invoke), a wrapped `WebCaptureFrameSource`, and the shared crop-sized `WriteableBitmap` preview (dispatcher-coalesced copy at Render priority, `PreviewBitmapChanged` raised only on recreation). Nav + `TransparentBackgroundScript` injection unchanged. Public ctor `(Dispatcher)`; internal seam ctor `(Dispatcher, Func<string, IScreenCaptureSource>?)` for hermetic tests (no WinRT). Caller: `MainViewModel.InitWebView2()` (now invoked from `MainWindow_Loaded` — the parent HWND must exist). See `../ai.md` "Slice 14" |
|
||||
| `WebCaptureFrameSource.cs` | The `IScreenCaptureSource` for a web session: owns the `GraphicsCaptureItem` (`CreateFromVisual`) + a free-threaded `Direct3D11CaptureFramePool` (2 buffers) + session; readback via `SoftwareBitmap.CreateCopyFromSurfaceAsync` (BGRA, **`BitmapAlphaMode.Straight`** — the compositor blends straight alpha); per-frame `FindContentBounds` alpha-bbox (internal static, unit-tested) stamps `VideoFrame.CropBounds`; ring up to 2 fresh frames + Epoch reuse; `IsOpaque=false`. Mirror of `ScreenCaptureFrameSource` |
|
||||
| `Compositor/SceneCompositor.cs` | **The output compositor (TASK 4 ship step 1)**: renders a scene into the encoder's master `VideoFrame` (tightly-packed BGRA8), mirroring the XAML preview minus editing chrome — backdrop → background → elements (`UniformToFill` cover-crop, round clip, mirror, opacity, border) → branding flash → social bar (optional `socialBarFrame` + `socialBarTop` in master space, blitted last so the bar sits above the flash; the old `BlitFlash` generalized to `BlitOverlay` with source offsets). Pure and WPF-free: frames injected via a `Func<SceneElement, VideoFrame?>` resolver (webcam → DeviceId, image → AssetId, backdrop → CaptureKey); output sized by `CompositorOptions` (16:9 = full master 1:1; vertical 9:16 = 607×1080 crop → 1080×1920 bilinear). Preview stays XAML (editing view); this is the output view — see `ai.md` "Scene compositor" |
|
||||
| `Compositor/CompositorOptions.cs` | The active tier's output rect (source space over the 1920×1080 master, integer-aligned — `MainViewModel.OutputRectX` can be 656.5) + target W×H |
|
||||
| `Compositor/StretchMath.cs` | Pure pixel math: the WPF `UniformToFill` cover-crop, clamped bilinear sample/scale (unit-tested half of the compositor) |
|
||||
|
||||
@@ -11,7 +11,10 @@
|
||||
2. ✅ Schema v10: `WebUri TEXT` column on `Source` table + migration in `LayoutStore.cs`
|
||||
3. ✅ Persist `Source.WebUri` on save/load (currently in-memory only — lost on restart)
|
||||
4. ✅ Hidden off-screen `WebView2` control per web source — navigates to `WebUri`, renders in-app
|
||||
5. ✅ Frame capture from WebView2 (`CoreWebView2.CapturePreviewAsync`) → `WriteableBitmap` (BGRA8)
|
||||
5. ✅ Frame capture from WebView2 (**superseded 2026-09-14:** was `CoreWebView2.CapturePreviewAsync`
|
||||
→ `WriteableBitmap` (BGRA8); now composition-capture via
|
||||
`CoreWebView2CompositionController` → `Windows.Graphics.Capture` — see the rendering-model
|
||||
update below. PNG polling is deleted)
|
||||
6. ✅ Wire into `FramePump` resolver — `Source { Type: WebSource }` → latest WebView2 frame
|
||||
7. ✅ Wire into `SceneCompositor` — render web source as an image element at its position/size
|
||||
8. ✅ Preview shows live web content (not just a blank rectangle)
|
||||
@@ -33,6 +36,18 @@ measure DOM stuff on NavigationCompleted (unsettled layout broke the image — `
|
||||
sizing the container to the crop (`ab29ec8` reverted — made it worse); a widget glow effect can
|
||||
fake a gap — verify with a plain widget first; measure rendered pixels instead.
|
||||
|
||||
**Rendering model update (2026-09-14, COMPOSITION capture — the PNG poll is gone):** frame capture
|
||||
is no longer `CapturePreviewAsync` → PNG (35–165ms per full-HD encode+decode capped real cadence at
|
||||
~20Hz; item 5 below is superseded). Each session now renders through a
|
||||
`CoreWebView2CompositionController` visual into `Windows.Graphics.Capture`
|
||||
(`GraphicsCaptureItem.CreateFromVisual` of the root 1920×1080 `ContainerVisual`), pulled frame-driven
|
||||
from a free-threaded `Direct3D11CaptureFramePool` at the renderer's own rate (~60fps, parity with the
|
||||
browser). Straight-alpha readback, alpha-bbox crop + `CropBounds` per frame, epoch'd ring reuse, one
|
||||
shared crop-sized preview bitmap. The old `SetCaptureInterval` cadence hooks and `CaptureScheduler`
|
||||
are deleted; the hidden XAML `WebViewHostPanel` overlay and `CapturePreviewAsync` telemetry are gone.
|
||||
`TransparentBackgroundScript` is unchanged (a test pins it). Full design + the CoreMessaging DQ
|
||||
recipe in `../ai.md` → Slice 14. **Open: device verification of ~60fps web animation.**
|
||||
|
||||
**Properties panel (2026-08-28):** web URI ✓/✕ icon buttons are `IsTabStop="False"` so Tab flows
|
||||
X→Y→W→H→URI; the ✕ button now clears the URI textbox (was reverting to the pre-accept snapshot).
|
||||
Slider style gained `IsMoveToPointEnabled="True"` — click-anywhere-on-bar jumps the thumb to the
|
||||
@@ -42,6 +57,7 @@ click (volume sliders keep their manual `SetSliderValueFromClick`, harmless dupl
|
||||
|
||||
- WebView2 is the only option for Windows — it's pre-installed on Windows 10 20H2+ and Windows 11
|
||||
- The web source is a standard element — positioned/sized/opacitied like any image source
|
||||
- Frame capture rate can be lower than video FPS (5-10 fps for web content is fine)
|
||||
- Frame capture is drive-by-renderer (composition capture, ~60fps parity with the browser since
|
||||
2026-09-14; the old "5-10 fps is fine" poll model is deleted)
|
||||
- This enables Streamlabs/StreamElements overlays via web URLs
|
||||
|
||||
|
||||
@@ -56,7 +56,6 @@ public partial class MainViewModel : ViewModelBase
|
||||
ResetHealth(StreamStatus.Offline);
|
||||
_audioMixer.StartLive(EncoderOptions.DefaultAudioPipeName);
|
||||
IsRecording = true;
|
||||
_webView2Manager?.SetCaptureInterval(33); // web layer ~30Hz while recording
|
||||
await _framePump.StartAsync();
|
||||
}
|
||||
|
||||
@@ -176,7 +175,6 @@ public partial class MainViewModel : ViewModelBase
|
||||
return;
|
||||
}
|
||||
|
||||
_webView2Manager?.SetCaptureInterval(33); // web layer ~30Hz while live
|
||||
await _framePump.StartAsync(); // never throws; failures log + surface via Failed
|
||||
|
||||
// Health polling (TASK 5 item 3): poll immediately, then every 30s while
|
||||
@@ -253,7 +251,6 @@ public partial class MainViewModel : ViewModelBase
|
||||
// monitoring); only the live/report pipe, the pump, and the session stop here.
|
||||
_audioMixer.StopLive();
|
||||
await _framePump.StopAsync();
|
||||
_webView2Manager?.SetCaptureInterval(200); // idle pace — preview-only doesn't need 30Hz
|
||||
|
||||
// Proper close-out (TASK 9 decision 6, built 2026-09-01): transition(complete)
|
||||
// AFTER the RTMP push is closed so no frames post-date the end — YouTube then
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
using System.Linq;
|
||||
using System.Windows.Controls;
|
||||
using System.Windows.Media.Imaging;
|
||||
using ytLive.Models;
|
||||
using ytLive.Services;
|
||||
@@ -25,9 +24,13 @@ public partial class MainViewModel
|
||||
}
|
||||
}
|
||||
|
||||
public void InitWebView2(Panel hostPanel)
|
||||
// Runs on the main window's Loaded handler (MainWindow.xaml.cs): the
|
||||
// composition controller needs the window HWND as its parent, which does not
|
||||
// exist in the ctor. Web sources register then — no frames are expected before
|
||||
// the window is shown anyway.
|
||||
public void InitWebView2()
|
||||
{
|
||||
_webView2Manager = new WebView2Manager(hostPanel, System.Windows.Application.Current.Dispatcher);
|
||||
_webView2Manager = new WebView2Manager(System.Windows.Application.Current.Dispatcher);
|
||||
_webView2Manager.PreviewBitmapChanged += OnWebView2PreviewBitmapChanged;
|
||||
RegisterLoadedWebSources();
|
||||
}
|
||||
|
||||
+1
-1
@@ -60,7 +60,7 @@ Progress (Commit N/12 · tag):
|
||||
- [x] **Commit 15 · `refactor-commit-15`** — **PreviewPane** → `Controls/PreviewPane.xaml(.cs)` (Row 1 Col 1 center preview pane). Moved the 1920×1080 canvas (OverlayCanvas/SelectionOverlay/BrandFlashLayer/SocialBarElement), text pull-out, and the live-controls row (TRAX/game/mic meters + sliders + speakers + socials button). Code-behind moved: preview drag/select/resize/hit-test (Preview_Mouse*, HitHandle, HitElement), webcam-context-menu handlers (WebcamMenu_*), social-bar glow (SocialBar_*, SocialsButton_*, SocialGlowTick), audio sliders/speakers (VolumeSlider_*, GameVolumeSlider_*, MicSpeaker_*, GameSpeaker_*, SetSliderValueFromClick, FindParent), TraxButton_*. Window keeps the source-list left panel + thumbnail strip + edit UI, and routes selection-overlay via `_previewPane.UpdateSelectionOverlay(bool)` plus `_previewPane.IsClickInsidePreview/Drawer`. `BackgroundMenu_*` stayed in the window (used by the left-panel source context menu).
|
||||
- [x] **Commit 16 · `refactor-commit-16`** — **SceneThumbnailStrip** → `Controls/SceneThumbnailStrip.xaml(.cs)` (Row 0 scene-picker strip). `Thumbnail_MouseLeftButtonDown` moved into the control (transitions via `DataContext.TransitionToScene`).
|
||||
- [x] **Commit 17 · `refactor-commit-17`** — **LeftPanel** → `Controls/LeftPanel.xaml(.cs)` (Row 1 Col 0 left panel: layers/properties edit UI + live chat list). Moved ALL property-editor and chat handlers: `SourceNameBox_*`/`CommitSourceEdit`, `AddSourceButton_Click`, `OnChatMessagesChanged` (now self-subscribed on Loaded), `MirrorButton_Click`, `ShapeButton_Click`, `SourceMenu_Remove`, `BackgroundMenu_*`, `OpacitySlider_ValueChanged`, list drag-to-reorder (`_dragStartPoint`+`List_Preview*`+`EndListDrag`+`FindItemContainerAt`), and the snapshot family (`TakeSnapshot`, `WebcamBorder*`, `Countdown*`, `Web*`). Window exposes via `_leftPanel.IsClickInside(original)` (deselect guard) and `_leftPanel.OnSelectionChanged(vm)` (opacity label + snapshot). Pending: overlay host region.
|
||||
- [x] **Commit 18 · `refactor-commit-18`** — **OverlayHost** → `Controls/OverlayHost.xaml(.cs)` (settings/bug/feature/about dialogs). `BackToAbout_Click` moved with it (uses inherited `DataContext`). Window keeps the thin `WebViewHostPanel` + `ToastArea` root stubs so name lookups are untouched (WebView2 ctor init, `NotificationService.InArea("ToastArea")`) — toast/WebView2 behavior unchanged. **Phase 2 COMPLETE** — MainWindow.xaml carved down to a 123-line shell over 6 controls (TopBar, HealthBanner, BottomBar, SceneThumbnailStrip, LeftPanel, PreviewPane, OverlayHost). Pending: session wrap-up (HANDOFF.md) + push on approval.
|
||||
- [x] **Commit 18 · `refactor-commit-18`** — **OverlayHost** → `Controls/OverlayHost.xaml(.cs)` (settings/bug/feature/about dialogs). `BackToAbout_Click` moved with it (uses inherited `DataContext`). Window keeps the thin `WebViewHostPanel` + `ToastArea` root stubs so name lookups are untouched (WebView2 ctor init, `NotificationService.InArea("ToastArea")`) — toast/WebView2 behavior unchanged. **Phase 2 COMPLETE** — MainWindow.xaml carved down to a 123-line shell over 6 controls (TopBar, HealthBanner, BottomBar, SceneThumbnailStrip, LeftPanel, PreviewPane, OverlayHost). Pending: session wrap-up (HANDOFF.md) + push on approval. *(The `WebViewHostPanel` root stub is since GONE — the 2026-09-14 composition-capture redesign moved `InitWebView2()` to `MainWindow_Loaded` and deleted the panel; see `Controls/index.md`.)*
|
||||
|
||||
**Phase 3 — 500-line compliance (production `.cs` only).** Bring all production
|
||||
source files under 500 lines. One roll-back commit per file; same drill
|
||||
|
||||
@@ -893,6 +893,8 @@ seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<Compos
|
||||
(2026-09-10, the "widget animates too slow" take):** the recording is 60fps but the WebView2
|
||||
capture loop was a blind 100ms `DispatcherTimer` = 10Hz — a 60fps-designed widget was sampled 6×
|
||||
under its native rate (repeated-footage slow-mo). Two stacked throttles:
|
||||
> Superseded by **Slice 14** (below): `CaptureScheduler` and the cadence hooks were the polling
|
||||
> design; frame-driven composition capture replaced them — nothing here is current architecture.
|
||||
(1) **the 10Hz cap** (unconditional, code-proven) and
|
||||
(2) **Chromium hidden-page throttling** (`requestAnimationFrame` parked, JS timers clamped to ~1s
|
||||
per WebView2Feedback#1172/#3070 + Chrome-88 timer throttling) whenever the app window is unfocused
|
||||
@@ -935,11 +937,65 @@ Full suite 290/291 passing, the sole failure the pre-existing compositor pixel t
|
||||
FIRST capture ever = the initial about:blank document — a blind instrument. NavigationCompleted
|
||||
for the REAL widget URL now arms `WidgetDumpRemaining = 5`; those 5 post-paint captures dump
|
||||
`%TEMP%\ytLive-web-<id>-w1..5.png` + the shared `AlphaStats(...)` line (alpha[min/max/mean/zero%])
|
||||
+ FindContentBounds result to startup.log. That line alone decides the fix branch: zero% ⇒ the
|
||||
pre-parse injection did not hold for this widget (opaque capture → chroma-key / stronger DOM
|
||||
injection); large zero% + tight contentBounds ⇒ the capture IS transparent and the recording's
|
||||
black block lives downstream (compositor blend/underlay). `AlphaStats` extracted as the shared
|
||||
sampler. No new tests (WebView2 runtime not instantiable in tests; the re-point is log-only).
|
||||
+ FindContentBounds result to startup.log. That line alone decides the fix branch: zero% ⇒ the
|
||||
pre-parse injection did not hold for this widget (opaque capture → chroma-key / stronger DOM
|
||||
injection); large zero% + tight contentBounds ⇒ the capture IS transparent and the recording's
|
||||
black block lives downstream (compositor blend/underlay). `AlphaStats` extracted as the shared
|
||||
sampler. No new tests (WebView2 runtime not instantiable in tests; the re-point is log-only).
|
||||
- **Slice 14 — web frames are now COMPOSITION-CAPTURED; the PNG poll + `CaptureScheduler` are gone
|
||||
(2026-09-14, the "still not 60fps" take):** slice 11's ~30Hz was still a `CapturePreviewAsync` PNG
|
||||
poll — every 1920×1080 full-HD encode+decode cost 35–165ms, so the session budget capped real
|
||||
cadence at ~20Hz for a 60fps-designed widget. Root fix = frame-driven capture, not faster polling:
|
||||
the WebView2 renderer now feeds `Windows.Graphics.Capture` directly through a
|
||||
`CoreWebView2CompositionController` (the mechanism `WebView2CompositionControl` and Flutter's
|
||||
`webview_windows` use — see `graphics_context.cc` `CreateGraphicsCaptureItemFromVisual`, which
|
||||
captures the root `surface_` visual).
|
||||
- **Pipeline (per session):** `CreateCoreWebView2CompositionControllerAsync(WindowHandle)` — the
|
||||
parent HWND must exist, so `MainWindow.InitWebView2()` moved from the ctor to **Loaded** (was
|
||||
`InitWebView2(WebViewHostPanel)`; the hidden XAML `WebViewHostPanel` overlay is deleted).
|
||||
`controller.RootVisualTarget` = a child `ContainerVisual` (`RelativeSizeAdjustment = 1,1`) under a
|
||||
root `ContainerVisual` (1920×1080, `IsVisible=true`) built on ONE `Windows.UI.Composition.Compositor`
|
||||
(created once on the UI thread); `Bounds = 0,0,1920,1080`, `BoundsMode=UseRawPixels`,
|
||||
`ShouldDetectMonitorScaleChanges=false`, `RasterizationScale=1.0`, `IsVisible=true`,
|
||||
`DefaultBackgroundColor=Transparent`; then `GraphicsCaptureItem.CreateFromVisual(root)`.
|
||||
Frames pull from a free-threaded `Direct3D11CaptureFramePool` (2 buffers) →
|
||||
`SoftwareBitmap.CreateCopyFromSurfaceAsync` (BGRA, `BitmapAlphaMode.Straight` — the compositor
|
||||
blends STRAIGHT alpha, so premultiplied readback would wreck corner anti-aliasing) → per-frame
|
||||
`FindContentBounds` alpha-bbox on the capture worker → `VideoFrame.CropBounds` stamped → ring up to
|
||||
2 fresh + Epoch (reuse; GC lesson slice 8) → dispatcher-coalesced copy (Render priority) into ONE
|
||||
crop-sized shared `WriteableBitmap`; `PreviewBitmapChanged` raised only on bitmap (re)creation.
|
||||
Also unpumped: the OBS-side de-throttle flags (slice 11) stay — they fix the hidden-page JS
|
||||
throttling; the capture path itself no longer polls.
|
||||
- **Implemented in:** new `Services/WebCaptureFrameSource.cs` (owns item+pool+session — mirrors
|
||||
`ScreenCaptureFrameSource`, which servers as the copy template) + reworked `Services/WebView2Manager.cs`
|
||||
(owns controllers, the visual tree, nav/script wiring, preview; internal seam ctor
|
||||
`(Dispatcher, Func<string, IScreenCaptureSource>?)` so the tests instantiate zero WinRT; public
|
||||
ctor `(Dispatcher)`).
|
||||
- **CoreMessaging DQ recipe (record-once):** the 19041 CsWinRT projection has NO
|
||||
`DispatcherQueueController.CreateOnCurrentThread()` (CS0117 — only `CreateOnDedicatedThread` +
|
||||
`FromAbi(IntPtr)`). P/Invoke `coreMessaging.dll!CreateDispatcherQueueController` with struct
|
||||
`DispatcherQueueOptions { DwSize, ThreadType = 2 (DQTYPE_THREAD_CURRENT), ApartmentType = 2 (DQTAT_COM_STA) }`,
|
||||
wrap via `DispatcherQueueController.FromAbi(ptr)` (mirror of `CaptureInterop`), only then
|
||||
`new Compositor()` — all once on the WPF UI thread (the app dispatches Render there).
|
||||
- **Compile notes (each cost a build cycle):** `Compositor` collides with the repo's OWN
|
||||
`ytLive.Services.Compositor` namespace → fully-qualify `Windows.UI.Composition.Compositor`;
|
||||
`CoreWebView2CompositionController` exposes **`Close()`**, not `Dispose()`; `Color` is ambiguous
|
||||
(`System.Drawing` vs `System.Windows.Media`) → `System.Drawing.Color.Transparent`.
|
||||
- **Dropped:** `Services/CaptureScheduler.cs` (DELETED — no cadence to schedule), all three
|
||||
`SetCaptureInterval` hooks in `MainViewModel.Streaming.Operations.cs`, the XAML `WebViewHostPanel`.
|
||||
`TransparentBackgroundScript` const is unchanged (a test pins it).
|
||||
- **Test model (Good Dog):** reworked `ytLive.Tests/WebView2ManagerTests.cs` — dropped the 4
|
||||
control-size tests + the `CaptureScheduler_Drops…` test; `FindContentBounds` tests moved to
|
||||
`WebCaptureFrameSource.FindContentBounds`; the ONE integration test
|
||||
`Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds` drives the internal seam with
|
||||
`FakeWebSource : IScreenCaptureSource` + a real background-STA `DispatcherPump` (copied from
|
||||
`ScreenCaptureManagerTests`): register → one crop-sized preview bitmap published → `CropBounds` +
|
||||
`IsOpaque=false` on `GetLatestFrame` → back-to-back pumps coalesce to latest. Byte assertion compares
|
||||
the DENSE 2×2 crop (`CropBytes` helper — slices source rows with stride gaps, a contiguous range
|
||||
spans rows wrongly).
|
||||
- Full suite **293/293 green, 0 warnings**. Audio untouched. **NOT YET VERIFIED ON DEVICE** — the
|
||||
composition path needs a real 60fps-widget run (open item; see HANDOFF). Web work commits stay
|
||||
LOCAL (no push) until the user greenlights.
|
||||
- **Stop ordering matters:** `StopAsync` stops the encoder — since slice 10 it FLUSHES the pending
|
||||
queue (`Channel.TryComplete` → drain writes the leftovers, closes stdin → EOF → ffmpeg finalizes+exits;
|
||||
an accepted frame is never lost) — **before** awaiting the loop. The old reverse-order deadlock was
|
||||
|
||||
@@ -84,10 +84,11 @@ public sealed class RoundClipInteractionTests
|
||||
// of PreviewGrid); otherwise Window_PreviewMouseLeftButtonDown
|
||||
// deselects the source and the handle can never be grabbed.
|
||||
// UIElement.InputHitTest IS the input pipeline — what Mouse uses.
|
||||
// Do NOT use VisualTreeHelper.HitTest here: it returned the window's
|
||||
// non-hit-testable WebViewHostPanel overlay for EVERY point (it does
|
||||
// not model IsHitTestVisible input semantics) — the bug this test
|
||||
// "found" for months was this wrong API (root-caused 2026-09-01).
|
||||
// Do NOT use VisualTreeHelper.HitTest here: it ignores IsHitTestVisible
|
||||
// input semantics (the old invisible WebView2 host overlay for web
|
||||
// sources used to win every visual hit — the wrong-API false bug this
|
||||
// test "found" for months, root-caused 2026-09-01; the overlay is gone
|
||||
// with the composition-based web render but the API lesson stays).
|
||||
var hit = window.InputHitTest(cornerInWindow) as DependencyObject;
|
||||
var grabbable = hit != null && IsDescendantOf(hit, previewGrid);
|
||||
|
||||
|
||||
@@ -1,114 +1,72 @@
|
||||
using System.Drawing;
|
||||
using System.Linq;
|
||||
using System.Windows.Controls;
|
||||
using System;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using System.Windows.Threading;
|
||||
using Microsoft.Web.WebView2.Wpf;
|
||||
using System.Windows.Media.Imaging;
|
||||
using Xunit;
|
||||
using ytLive.Models;
|
||||
using ytLive.Services;
|
||||
|
||||
namespace ytLive.Tests;
|
||||
|
||||
[Collection("RealApp")]
|
||||
public sealed class WebView2ManagerTests
|
||||
/// <summary>
|
||||
/// WebView2Manager owns the frame-driven web sources (composition-captured, see
|
||||
/// Services/WebCaptureFrameSource.cs). The controller/visual/GraphicCaptureItem
|
||||
/// layer is WinRT and only exercisable on Windows at runtime, so these tests pin
|
||||
/// the pure pieces (transparency script, the alpha-bbox math) and the ONE
|
||||
/// integration test drives the session plumbing with a fake source: registration,
|
||||
/// crop-stamped frames, the shared cropped preview bitmap, and coalescing to the
|
||||
/// latest frame.
|
||||
/// </summary>
|
||||
public class WebView2ManagerTests
|
||||
{
|
||||
private readonly RealAppHost _app;
|
||||
|
||||
public WebView2ManagerTests(RealAppHost app)
|
||||
private sealed class FakeWebSource : IScreenCaptureSource
|
||||
{
|
||||
_app = app;
|
||||
public string Key { get; }
|
||||
public event Action<VideoFrame>? FrameAvailable;
|
||||
|
||||
public FakeWebSource(string key) => Key = key;
|
||||
|
||||
public Task StartAsync() => Task.CompletedTask;
|
||||
public Task StopAsync() => Task.CompletedTask;
|
||||
|
||||
public void Pump(VideoFrame frame) => FrameAvailable?.Invoke(frame);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void WebView2_Control_Size_Is_Canvas_Resolution_Regardless_Of_Source()
|
||||
// A real Dispatcher pumping on a background STA thread, so the manager's
|
||||
// BeginInvoke queue can be drained deterministically from the test thread
|
||||
// (same harness as ScreenCaptureManagerTests).
|
||||
private sealed class DispatcherPump : IDisposable
|
||||
{
|
||||
_app.Run(() =>
|
||||
private readonly Thread _thread;
|
||||
private readonly ManualResetEventSlim _ready = new(false);
|
||||
private Dispatcher? _dispatcher;
|
||||
|
||||
public Dispatcher Dispatcher => _dispatcher!;
|
||||
|
||||
public DispatcherPump()
|
||||
{
|
||||
var panel = new Canvas();
|
||||
var source = new Source
|
||||
_thread = new Thread(() =>
|
||||
{
|
||||
Type = SourceType.WebSource,
|
||||
Width = 640,
|
||||
Height = 480,
|
||||
};
|
||||
_dispatcher = Dispatcher.CurrentDispatcher;
|
||||
_ready.Set();
|
||||
Dispatcher.Run();
|
||||
});
|
||||
_thread.IsBackground = true;
|
||||
_thread.SetApartmentState(ApartmentState.STA);
|
||||
_thread.Start();
|
||||
_ready.Wait();
|
||||
}
|
||||
|
||||
using var manager = new WebView2Manager(panel, Dispatcher.CurrentDispatcher);
|
||||
manager.Register(source);
|
||||
// Queued behind anything already posted at Render/higher, so pending
|
||||
// bitmap copies run first.
|
||||
public void Drain() => Dispatcher.Invoke(() => { }, DispatcherPriority.ContextIdle);
|
||||
|
||||
var webView = panel.Children.OfType<WebView2>().First();
|
||||
Assert.Equal(1920.0, webView.Width);
|
||||
Assert.Equal(1080.0, webView.Height);
|
||||
});
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void WebView2_Control_Size_Does_Not_Track_Box_After_Register()
|
||||
{
|
||||
_app.Run(() =>
|
||||
public void Dispose()
|
||||
{
|
||||
var panel = new Canvas();
|
||||
var source = new Source
|
||||
{
|
||||
Type = SourceType.WebSource,
|
||||
Width = 640,
|
||||
Height = 480,
|
||||
};
|
||||
|
||||
using var manager = new WebView2Manager(panel, Dispatcher.CurrentDispatcher);
|
||||
manager.Register(source);
|
||||
|
||||
var webView = panel.Children.OfType<WebView2>().First();
|
||||
|
||||
source.Width = 1024;
|
||||
source.Height = 768;
|
||||
|
||||
Assert.Equal(1920.0, webView.Width);
|
||||
Assert.Equal(1080.0, webView.Height);
|
||||
});
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void WebView2_Control_Size_Ignores_Zero_Sized_Source()
|
||||
{
|
||||
_app.Run(() =>
|
||||
{
|
||||
var panel = new Canvas();
|
||||
var source = new Source
|
||||
{
|
||||
Type = SourceType.WebSource,
|
||||
Width = 0,
|
||||
Height = 0,
|
||||
};
|
||||
|
||||
using var manager = new WebView2Manager(panel, Dispatcher.CurrentDispatcher);
|
||||
manager.Register(source);
|
||||
|
||||
var webView = panel.Children.OfType<WebView2>().First();
|
||||
Assert.Equal(1920.0, webView.Width);
|
||||
Assert.Equal(1080.0, webView.Height);
|
||||
});
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void WebView2_DefaultBackgroundColor_Is_Transparent()
|
||||
{
|
||||
_app.Run(() =>
|
||||
{
|
||||
var panel = new Canvas();
|
||||
var source = new Source
|
||||
{
|
||||
Type = SourceType.WebSource,
|
||||
Width = 640,
|
||||
Height = 480,
|
||||
};
|
||||
|
||||
using var manager = new WebView2Manager(panel, Dispatcher.CurrentDispatcher);
|
||||
manager.Register(source);
|
||||
|
||||
var webView = panel.Children.OfType<WebView2>().First();
|
||||
|
||||
Assert.Equal(System.Drawing.Color.Transparent, webView.DefaultBackgroundColor);
|
||||
});
|
||||
Dispatcher.InvokeShutdown();
|
||||
_thread.Join(3000);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -142,7 +100,7 @@ public sealed class WebView2ManagerTests
|
||||
pixels[i + 3] = 255; // A
|
||||
}
|
||||
|
||||
var (x, y, bw, bh) = WebView2Manager.FindContentBounds(pixels, w, h, w * 4);
|
||||
var (x, y, bw, bh) = WebCaptureFrameSource.FindContentBounds(pixels, w, h, w * 4);
|
||||
|
||||
Assert.Equal((3, 2, 4, 3), (x, y, bw, bh));
|
||||
}
|
||||
@@ -153,7 +111,7 @@ public sealed class WebView2ManagerTests
|
||||
const int w = 10, h = 6;
|
||||
var pixels = new byte[w * h * 4];
|
||||
|
||||
var (x, y, bw, bh) = WebView2Manager.FindContentBounds(pixels, w, h, w * 4);
|
||||
var (x, y, bw, bh) = WebCaptureFrameSource.FindContentBounds(pixels, w, h, w * 4);
|
||||
|
||||
Assert.Equal((0, 0, w, h), (x, y, bw, bh));
|
||||
}
|
||||
@@ -166,36 +124,108 @@ public sealed class WebView2ManagerTests
|
||||
for (var i = 3; i < pixels.Length; i += 4)
|
||||
pixels[i] = 255;
|
||||
|
||||
var (x, y, bw, bh) = WebView2Manager.FindContentBounds(pixels, w, h, w * 4);
|
||||
var (x, y, bw, bh) = WebCaptureFrameSource.FindContentBounds(pixels, w, h, w * 4);
|
||||
|
||||
Assert.Equal((0, 0, w, h), (x, y, bw, bh));
|
||||
}
|
||||
|
||||
// The ONE integration test for this change: a registered session creates one
|
||||
// shared WriteableBitmap sized to the widget's CropBounds, publishes it once,
|
||||
// and back-to-back frames coalesce to the latest (a single pending UI copy,
|
||||
// mirror of ScreenCaptureManager) — the frame-driven contract the PNG-poll
|
||||
// redesign replaced the CaptureScheduler cadence with.
|
||||
[Fact]
|
||||
public async Task CaptureScheduler_Drops_Ticks_While_Capture_InFlight_And_Resumes()
|
||||
public void Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds()
|
||||
{
|
||||
// The ~30Hz recording cadence would stack concurrent CapturePreviewAsync calls
|
||||
// (full-HD PNG encode is ~10-30ms) without the latest-wins drop: ticks that
|
||||
// arrive while a capture is in flight must be skipped, never queued, and a
|
||||
// capture can always start once the previous one has finished.
|
||||
var calls = 0;
|
||||
var signal = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
var scheduler = new CaptureScheduler(Dispatcher.CurrentDispatcher, 200, async () =>
|
||||
using var pump = new DispatcherPump();
|
||||
FakeWebSource? fake = null;
|
||||
using var manager = new WebView2Manager(pump.Dispatcher, key => fake = new FakeWebSource(key));
|
||||
|
||||
WriteableBitmap? published = null;
|
||||
var publishedCount = 0;
|
||||
manager.PreviewBitmapChanged += (_, bitmap) => { published = bitmap; publishedCount++; };
|
||||
|
||||
var source = new Source { Type = SourceType.WebSource, Width = 640, Height = 480 };
|
||||
manager.Register(source);
|
||||
Assert.NotNull(fake); // the seam instantiated + started the source inline
|
||||
Assert.Null(manager.GetLatestFrame(source.Id)); // nothing pumped yet
|
||||
|
||||
// 4x4 canvas; alpha 255 only in bbox (1,1)-(2,2) → the widget crops to (1,1,2,2).
|
||||
var first = WidgetFrame(4, 4, frameSeed: 42);
|
||||
fake!.Pump(first);
|
||||
pump.Drain();
|
||||
|
||||
Assert.NotNull(published);
|
||||
Assert.Equal(1, publishedCount); // one shared bitmap, published once
|
||||
var dims = pump.Dispatcher.Invoke(() => new[] { published!.PixelWidth, published.PixelHeight });
|
||||
Assert.Equal(new[] { 2, 2 }, dims);
|
||||
|
||||
var latest = manager.GetLatestFrame(source.Id);
|
||||
Assert.NotNull(latest);
|
||||
Assert.Equal((1, 1, 2, 2), latest!.CropBounds); // the source stamps the alpha bbox
|
||||
Assert.False(latest.IsOpaque); // straight-alpha web contract
|
||||
|
||||
var previewBytes = pump.Dispatcher.Invoke(() =>
|
||||
{
|
||||
calls++;
|
||||
await signal.Task;
|
||||
var bytes = new byte[16];
|
||||
published!.CopyPixels(new System.Windows.Int32Rect(0, 0, 2, 2), bytes, 8, 0);
|
||||
return bytes;
|
||||
});
|
||||
// The preview is a DENSE 2x2 rect (bbox 1,1..2,2 sliced out of the 4x4
|
||||
// canvas) — rows y=1 and y=2, columns x=1..2.
|
||||
var expected = CropBytes(first, w: 4, x0: 1, y0: 1, w2: 2, h2: 2);
|
||||
Assert.Equal(expected, previewBytes);
|
||||
|
||||
scheduler.Tick();
|
||||
scheduler.Tick();
|
||||
scheduler.Tick();
|
||||
Assert.Equal(1, calls); // overlapping ticks dropped, never queued
|
||||
// Second + third back-to-back: only the latest lands in the preview bitmap.
|
||||
var second = WidgetFrame(4, 4, frameSeed: 77);
|
||||
var third = WidgetFrame(4, 4, frameSeed: 99);
|
||||
fake.Pump(second);
|
||||
fake.Pump(third);
|
||||
pump.Drain();
|
||||
|
||||
signal.SetResult();
|
||||
await scheduler.Pending; // durable: Pending completes only after the drop resets
|
||||
|
||||
scheduler.Tick();
|
||||
Assert.Equal(2, calls); // once idle, the next tick captures again
|
||||
await scheduler.Pending;
|
||||
Assert.Equal(1, publishedCount); // same crop box → no new bitmap
|
||||
Assert.Same(third, manager.GetLatestFrame(source.Id));
|
||||
var coalesced = pump.Dispatcher.Invoke(() =>
|
||||
{
|
||||
var bytes = new byte[16];
|
||||
published!.CopyPixels(new System.Windows.Int32Rect(0, 0, 2, 2), bytes, 8, 0);
|
||||
return bytes;
|
||||
});
|
||||
Assert.Equal(CropBytes(third, 4, 1, 1, 2, 2), coalesced);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Dense little-endian slice of a w-wide frame for the rect
|
||||
/// (x0, y0)-(x0+w2-1, y0+h2-1); matches the manager's crop readback so the
|
||||
/// byte comparison is against the same layout, not the full frame stride.</summary>
|
||||
private static byte[] CropBytes(VideoFrame frame, int w, int x0, int y0, int w2, int h2)
|
||||
{
|
||||
var src = frame.BgraPixels;
|
||||
var dst = new byte[w2 * h2 * 4];
|
||||
for (var y = 0; y < h2; y++)
|
||||
System.Buffer.BlockCopy(src, ((y0 + y) * w + x0) * 4, dst, y * w2 * 4, w2 * 4);
|
||||
return dst;
|
||||
}
|
||||
|
||||
/// <summary>A 4x4 full canvas where the widget content is bbox (1,1,2,2):
|
||||
/// alpha 255 there, 0 elsewhere; the content pixels carry a seed so a
|
||||
/// coalesce mix-up is visible in the byte assertion.</summary>
|
||||
private static VideoFrame WidgetFrame(int w, int h, byte frameSeed)
|
||||
{
|
||||
var pixels = new byte[w * h * 4];
|
||||
for (var y = 0; y < h; y++)
|
||||
for (var x = 0; x < w; x++)
|
||||
{
|
||||
if (x < 1 || x > 2 || y < 1 || y > 2) continue;
|
||||
var i = (y * w + x) * 4;
|
||||
pixels[i] = frameSeed; // B
|
||||
pixels[i + 1] = (byte)(frameSeed / 3); // G
|
||||
pixels[i + 2] = 0; // R
|
||||
pixels[i + 3] = 255; // A
|
||||
}
|
||||
return new VideoFrame(w, h, pixels)
|
||||
{
|
||||
IsOpaque = false,
|
||||
CropBounds = (1, 1, 2, 2),
|
||||
};
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user