Screen backdrop (schema v5/v6) + five-scene catalog + webcam polish: live desktop/game capture as a permanent non-deletable bottom layer (Source.IsBackdrop), auto full-screen game detection (Win32FullScreenDetector) else primary display, GraphicsCapturePicker re-designation, refcounted shared ScreenCaptureManager, Live-only backdrop by policy (HasBackdrop + v5-v6 backfill + EnforceBackdropPolicy, checkbox gone), SceneCatalog (Starting/Live/BRB/Chat/Ending) with + button re-adding missing scenes, webcam mid-session transparent-container fix (GetPreviewBitmap propagation), Chat half-screen-area cap, 'Add Webcam' always opens the picker (SwapWebcamIdentityAsync, no silent resurrect), WindowsRuntimeMarshal frame-read + 5s-throttled errors, docs updated, tests (65 passing)

This commit is contained in:
2026-08-07 14:15:50 -07:00
parent e037ba027b
commit ad9b3e1e48
27 changed files with 2232 additions and 116 deletions
+109 -16
View File
@@ -53,8 +53,14 @@ dotnet.exe vstest "C:\...\ytLive.Tests\bin\Debug\net8.0-windows10.0.19041.0\ytLi
Good Dog Rule: ONE integration test per branch, ONE test per PR. Current: TokenStore DPAPI
roundtrip/corrupt/missing/clear, OAuth exchange/refresh/ClearSession, CameraManager refcount +
frame pump + failure handling (fakes for the WinRT seams), real-`MainWindow` round-clip
interaction test, LayoutStore delete roundtrip, LayoutStore pre-round-rect-dims roundtrip
25 passing.
interaction test, LayoutStore delete roundtrip, LayoutStore pre-round-rect-dims roundtrip,
LayoutStore backdrop roundtrip, LayoutStore HasBackdrop roundtrip + v5→v6 non-Live backfill,
ScreenCaptureManager refcount + shared-bitmap + coalescing
(fake `IScreenCaptureSource` + a real background-STA `Dispatcher`), BackdropTests (EnsureBackdrop
insert/idempotent/heal + HasBackdrop gate, IsLiveCapture, DisplaySource, INPC), SceneCatalogTests
(the five canonical scenes, Live-only backdrop policy, EnforceBackdropPolicy), WebcamSafeguardTests
(the per-scene size clamp incl. the Chat half-screen-area cap) —
65 passing.
### Real-MainWindow tests MUST be hermetic (DB pollution bug)
@@ -81,7 +87,7 @@ C# / WPF (.NET 8) following MVVM:
|------|------|
| `Models/` | Plain data types — Scene, Source (incl. `ClipShape`, `IsMirrored`, `VideoImageSource`), QualityOption, StreamConfig, StreamHealth, YouTubeChannel, ChatMessage |
| `ViewModels/` | MainViewModel — exposes collections + commands for the UI; GoLiveViewModel, ReuseImageViewModel, CameraPickerViewModel |
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`** |
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)** |
| `Helpers/` | ViewModelBase (INotifyPropertyChanged), RelayCommand, ImageCache, AppLog (file logger), FocusPreservingListBox, OAuthCredentials, **TokenStore (DPAPI session persistence)**, visibility converters |
| `Themes/` | `Controls.xaml` — the single dark-theme source, merged once in `App.xaml` (see `Themes/index.md`) |
| `MainWindow.xaml` | Dark theme; layout: top bar (controls), center (preview), left (scenes/sources), right (chat), bottom (health) |
@@ -93,6 +99,7 @@ C# / WPF (.NET 8) following MVVM:
- ViewModels are constructed in XAML (`<vm:MainViewModel/>` as DataContext)
- Services are currently instantiated in MainViewModel's constructor — no DI container yet
- Layout persists to SQLite (`Microsoft.Data.Sqlite`); scenes/sources/asset bytes stored in the DB, asset identity is a SHA-256 content hash (1:M reuse, no file paths — assets are always available)
- **Five-scene catalog (`Models/SceneCatalog.cs`):** the product is exactly Starting/Live/BRB/Chat/Ending — work with less, never more (the escape hatch for "more" is OBS). Scenes are matched **by name** (`SceneCatalog.Is`, case-insensitive trim). Empty DBs seed all five; the scenes-header "+" (`ShowAddScene`/`MissingScenes` on `MainViewModel`) only appears while ≥1 canonical scene is missing and its menu lists only the missing ones, re-adding them by name (`AddSceneCommand`). Renaming a canonical scene makes it missing again; `AddScene` rejects non-canonical names.
- Theming: all custom styles live in `Themes/Controls.xaml`, merged in `App.xaml` — never duplicate styles per-window (dialog duplicates were consolidated into this dictionary)
- Resolution tiers (bottom bar): 1080p60@8 (default) → 1080p30@8 → 720p60@6 → 720p30@6**Vertical 1080p60@8 (9:16, 1080×1920)**. The composition master frame is **always 1920×1080** — a tier is an output rect + target resolution over that master, so source geometry is never rewritten (no rounding drift). 16:9 tiers use the full frame; the vertical tier uses a centered **607×1080** window and the preview dims the cropped side strips at 55% black with an accent outline (semi-crop — the cut area stays visible). A resolution badge in the preview corner shows the active tier; the bottom bar shows bitrate/FPS. A **tooltip** explains finding upload bandwidth — an in-app speed test was deliberately dropped (unreliable). The future encoder crops the master to the rect and scales to the tier's Width×Height
- Crash diagnosis: `AppLog` writes startup checkpoints to `%APPDATA%\ytLlive\startup.log`; `App.xaml.cs` logs `DispatcherUnhandledException`/`AppDomain.UnhandledException`. When WPF won't run from WSL, this log is how you find the failure (it caught the `MenuItemRole.Separator` XAML crash and the ComboBox SelectionBoxItem bug)
@@ -100,11 +107,80 @@ C# / WPF (.NET 8) following MVVM:
### Current limitations / TODOs
- `Helpers/OAuthCredentials.cs` contains the real ClientId/ClientSecret. Auth is complete and the session **persists via Windows DPAPI** (`Helpers/TokenStore.cs``%APPDATA%\ytLlive\ytLlive.auth`, CurrentUser scope), reloaded best-effort at startup with a proactive refresh of a near-expiry access token. Sign-in/Change Account lives **inside the Start Stream dialog** (two-state flow — no separate Connect button). A **graceful End Livestream signs out**: `StopStream()` clears the session + token, so the next go-live needs a fresh sign-in; a crash never runs End, so the token survives and the creator stays signed in. `YouTubeAuthService` takes an optional `HttpClient` + `sessionChanged` callback (test seam + save hook; services are still constructed in `MainViewModel`)
- Scene/source/asset layout persists (SQLite, schema v4); the OAuth session persists (DPAPI); the paid-unlock state does not (yet — itch.io key verification pending)
- Scene/source/asset layout persists (SQLite, schema v6); the OAuth session persists (DPAPI); the paid-unlock state does not (yet — itch.io key verification pending)
- `YouTubeStreamService` uses hardcoded `1080p`/`60fps` and per-broadcast streams — must switch to the v3 `variable` reusable stream
- Webcam capture is shipped (milestone 1); **screen capture, scene compositing/encoding, RTMP are next**
- Webcam capture is shipped (milestone 1); the live desktop/game backdrop is shipped (ship task #1); **window capture (non-backdrop), scene compositing/encoding, RTMP are next**
- `StreamConfig` defaults (`TargetBitrate=6000`, `Resolution="1920x1080"`) are stale — the live dropdown drives `StreamHealth.CurrentBitrate`/`FPS` instead
### Screen backdrop capture (TASK 3 ship task #1)
The backdrop is the **live desktop/game capture as a permanent, non-deletable bottom layer**
rendered in every scene — the "Screen" source from the minimal set, done as content-swap
instead of a normal draggable source.
- **Model (schema v6):** `Source.IsBackdrop` (persisted) marks the one backdrop per scene; `Source.CaptureKey`
(persisted) names the target — `monitor:<n>`, `window:<hwnd>`, or `picker:<displayname>`. The backdrop is
a real `Source` of `Type DisplayCapture`, inserted **first** (`MainViewModel.EnsureBackdrop(scene)`,
internal static — runs on layout load + every `AddScene`, healing any scene missing one), fixed at
X=0/Y=0/1920×1080, and excluded from drag/hit-test/remove/reorder (remove is guarded in `RemoveElement`;
`IsDraggableElement` never matches live types; the element template sets `IsHitTestVisible=false` for
backdrops; the row's remove button and "Remove Source" menu item are hidden). **`Scene.HasBackdrop`
(persisted, default off) is the Live-only policy flag** — the backdrop belongs to the canonical Live
scene alone (see `SceneCatalog`). `EnsureBackdrop` returns null for a flag-less scene, so
Starting/BRB/Chat/Ending compose their own layers. The one-time v5→v6 backfill turns those four scenes
off and drops their backdrop sources, and **`EnforceBackdropPolicy` (internal static, runs after every
load)** re-normalizes the flag by scene name and strips any backdrop that lingers in a non-Live scene —
the flag is owned by policy, never the user. There is no scene-list "Backdrop" checkbox anymore
(the old `ToggleSceneBackdropCommand` is gone); "Change Capture…"/"Refresh Capture"/"Capture Display"
only show in the Live scene's preview menu (`CanChangeBackdrop`). The static Background, if any,
renders **above** the backdrop.
- **Detection (launch + focus only, no live session listener):** `Win32FullScreenDetector` =
`GetForegroundWindow` + `DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)` + `MonitorFromWindow` +
`GetMonitorInfo`; a window is full-screen when its frame covers all four monitor edges; own process is
excluded; monitor **index = `EnumDisplayMonitors` enumeration order** (the same order
`ScreenCaptureSourceFactory` uses to map index→HMONITOR via `Win32FullScreenDetector.GetMonitorHandle`).
`IFullScreenDetector` also exposes `GetDisplays()` (`DisplayInfo`: index/name/resolution/bounds/`IsPrimary`,
friendly name via `EnumDisplayDevices`) + `PrimaryMonitorIndex()` for the in-app "Capture Display"
submenu. At launch `ReacquireScreenCaptures` keys the backdrop to the full-screen game's monitor (or the
**primary** display — never assumed monitor 0).
On **Deactivated**, `NoteBackgroundWindow` samples the foreground ~250ms later (so an alt-tab to a game
lands before the app re-activates); on the next **Activated**, `RefreshBackdropAutoCapture` re-runs
detection against that sample and re-designates. Null detection (our app, the desktop, a normal window)
leaves the current capture alone.
- **Capture (WinRT GraphicsCapture):** `ScreenCaptureFrameSource` creates a free-threaded
`Direct3D11CaptureFramePool` (2 buffers, `B8G8R8A8UIntNormalized`) + `GraphicsCaptureSession`; frames →
`SoftwareBitmap.CreateCopyFromSurfaceAsync` (alpha ignored) → `VideoFrame` (BGRA8), bytes read via
`WindowsRuntimeMarshal.TryGetDataUnsafe` (the same CsWinRT-safe read the webcam path uses) — the
`IMemoryBufferByteAccess` ComImport cast threw `Invalid cast` on **every frame** under CsWinRT, which
flooded `startup.log` (~5 MB in a session) and burned CPU, so it is gone. Surfaces larger than the
1920×1080 master are downscaled bilinearly to the master (`DownscaleBgra`) before the copy, and
per-frame conversion failures are logged at most once per 5 s (`ErrorLogThrottle`). DRM-protected
content delivers black frames (OS limitation, documented). Frame pool
pauses while the app is minimized — capture keeps running, the pool just stops delivering.
- **Ownership:** `ScreenCaptureManager` mirrors `CameraManager` — refcounted by target key, one shared
`WriteableBitmap` per key, dispatcher-coalesced latest-frame copies, `PreviewBitmapChanged`/`CaptureFailed`
events, `ReleaseAllAsync` on re-designation. `ScreenCaptureSourceFactory.Resolve(key)` parses the key into
a source; `PickAsync()` shows the OS `GraphicsCapturePicker` ("Change Capture…", owner window set via the
`IInitializeWithWindow` ComImport) and returns a **transient** `picker:` key — a reload falls back to
auto-detection.
- **CsWinRT projection gaps hand-rolled:** `Windows.Graphics.Direct3D11.Direct3D11Helper` is not projected,
so `Direct3D11Helper` P/Invokes `d3d11.dll!D3D11CreateDevice` (hardware, BGRA_SUPPORT, explicit 11.1-first
feature array) → QI `IDXGIDevice` → the WinRT interop export
`CreateDirect3D11DeviceFromDXGIDevice``MarshalInterface<IDirect3DDevice>.FromAbi` (one shared device per
process). Do **not** switch back to the QI-for-`IDirect3DDxgiInterfaceAccess` trick: the raw D3D11 device
no longer exposes that interface on newer Windows (verified E_NOINTERFACE on build 26200, hardware and
WARP alike) while `CreateDirect3D11DeviceFromDXGIDevice` keeps working.
`IInitializeWithWindow` are ComImports in `CaptureInterop.cs`. All WinRT projections were verified by
reflection against the built `Microsoft.Windows.SDK.NET.dll` before writing the interop.
- **Known v1 limits:** full-desktop captures are CPU-copied at native resolution (GPU downscale = encoder
task); window capture (`window:<hwnd>`) and multi-monitor live re-targeting beyond the auto-detected
game are behind the picker; picker-based captures don't survive reload.
- **GPU posture:** same as webcam — CPU frames, WPF hardware-presents; D3DImage GPU compositing deferred
to the encoder task.
- **Preview watermark:** the "Preview" placeholder hides while a backdrop renders —
`ShowPreviewPlaceholder` now also checks `BackdropImage` (raised on backdrop change), so a scene with
live capture shows the feed instead of the "nothing here" label.
### Webcam capture (TASK 3 milestone 1)
- **Seam-first:** everything above the WinRT layer speaks only `VideoFrame` (normalized tightly-packed
@@ -131,11 +207,16 @@ C# / WPF (.NET 8) following MVVM:
border/`IsVisible`). `Scene.Elements` holds images (`Source`) and, at most once, the webcam
(`WebcamSceneConfig`); `Scene.WebcamConfig` is the accessor. `CameraManager` refcounts capture
sessions by `DeviceId` (a session starts at `RefCount = 1`; repeat acquire bumps it; the last
release stops + disposes). The Add Webcam menu greys out when the **active** scene already has a
config (`CanAddWebcamToActiveScene`); showing a hidden webcam reuses the existing config
(`CanShowWebcamInActiveScene` / empty-canvas right-click "Show Webcam"). **Removing the last webcam
config anywhere clears the identity** (`_webcam = null`), so re-adding opens the picker again
instead of resurrecting the old camera.
release stops + disposes). **"Add Webcam" ALWAYS opens the Windows camera picker** — the creator is
never silently handed the previous camera (which used to happen after deleting one scene's webcam
while another scene still used it; that identity survived, so re-adding bypassed the choice).
Picking a different camera than the current app-wide one swaps it everywhere via
`SwapWebcamIdentityAsync` (the same path "Change Webcam…" uses), keeping the singleton honest;
picking the same one just places the config. The Add Webcam menu greys out when the **active**
scene already has a config (`CanAddWebcamToActiveScene`); showing a hidden webcam reuses the
existing config (`CanShowWebcamInActiveScene` / empty-canvas right-click "Show Webcam" — which, on
a config-less canvas, delegates to Add Webcam and so also picks). **Removing the last webcam
config anywhere clears the identity** (`_webcam = null`), which also drops "Change Webcam…".
- **Round→rect restores the aspect (persisted, schema v4):** `SceneElement.ToggleClipShape()`
snapshots the rectangular Width/Height into public `RectWidth`/`RectHeight` before going Round and
restores them when switching back — otherwise the Round resize lock (square) would leave a square
@@ -152,6 +233,12 @@ C# / WPF (.NET 8) following MVVM:
`PreviewBitmapChanged`. Frames arrive on a worker thread; `CameraManager` coalesces onto the dispatcher
(at most one pending copy per session, at `Render` priority, always copying the latest frame) so a 60fps
device never drowns the render thread.
- **Webcam added mid-session must get the live frames:** `PreviewBitmapChanged` fires **once** (the first
frame creates the shared bitmap); later frames only mutate that bitmap in place, so a config that didn't
exist at first-frame time would never receive it — the empty/transparent container you'd see adding a
webcam to Chat while Live already had the camera. `CameraManager.GetPreviewBitmap(deviceId)` exposes the
current shared bitmap; `AddWebcamToActiveSceneAsync` assigns it to the new config right before
`AcquireAsync`, and `ReacquireWebcam` re-propagates it to every config after a reload.
- **Clip/mirror/border:** per-element `ClipShape` (Traditional rectangle / Round ellipse) + `IsMirrored`
(`ScaleX = -1`) + the OSB-standard static border (`BorderColor` `#RRGGBB` or `""`=none, `BorderOpacity`
01, `BorderWidth` 020, `BorderAnimation` `None|Pulse|Chase|Rainbow|Shimmer|MarchingAnts|Glow|
@@ -160,13 +247,19 @@ C# / WPF (.NET 8) following MVVM:
rendering stays static until the animation tier ships — Border Color, Opacity/Thickness sliders, Hide in
this scene, Remove); persisted in the layout DB. The Add menu shows when no webcam exists; the empty
preview canvas has its own Show Webcam entry.
- The Round `Ellipse` is wrapped in a `Viewbox Stretch="Uniform"` holding a `1x1` Grid, so it renders
as a true circle (diameter = the shorter element dimension) instead of an oval stretched to the
element rect — and the traditional `Image` keeps `UniformToFill` over the full rect. The Round
border is a centered `Ellipse` at `Width/Height = RoundBorderSize`.
- The Round webcam is an `Image Stretch="UniformToFill"` with an `EllipseGeometry` clip
(`Center=0.5,0.5` `RadiusX/Y=0.5`), inside a `Viewbox Stretch="Uniform"` holding a `1x1` Grid, so it
renders as a true circle (diameter = the shorter element dimension) instead of an oval stretched to
the element rect — the traditional `Image` keeps `UniformToFill` over the full rect. The clip is
geometry, **not an `ImageBrush`**: a brush re-rasterizes the frequently-updated `WriteableBitmap` per
frame on the render thread, which is what made the live webcam crawl while round. The Round border is
a centered `Ellipse` at `Width/Height = RoundBorderSize`.
- Resizing locks to a square (`_resizeAspect = 1`) while `ClipShape == Round`.
- **Webcam size clamp:** `ClampWebcamToBounds` (internal — test seam) enforces 50% of the 1920×1080
master per dimension (960×540 max) and no less than 10% (192×108) at resize + load;
- **Webcam size clamp:** `ClampWebcamToBounds(config, sceneName)` (internal — test seam) enforces the
max per dimension at resize + load and no less than 10% of the master (192×108). The cap is picked
**by canonical scene name**: 50% per dimension (960×540) everywhere except the **Chat scene**, which
may reach half the screen **area** (~1358×764 @16:9) so the viewer sees the creator better
(`MaxWebcamWidthFor`/`MaxWebcamHeightFor`; a renamed Chat loses the bigger cap).
`RoundBorderSize` follows the clamped height. `WebcamSafeguardTests` guards the clamp.
- **Hit-testing:** a `Grid` without `Background` only hit-tests where its children draw, so clicks in
the empty corners of a round clip fell through to `Window_PreviewMouseLeftButtonDown` and deselected