From 804b1d2f5f05090983eba8a312b4b3e9f8cdc2ea Mon Sep 17 00:00:00 2001 From: gramps Date: Mon, 26 Oct 2026 10:25:15 -0700 Subject: [PATCH] docs(memory): split the frame pipeline out of ai.md into docs/ai ai.md is the mandatory read every session; at 2,702 lines / ~64k tokens it costs the whole subsystem history even when the task never touches it. This is a verbatim move, not a compression - nothing is deleted and each stub is written from the moved text. - docs/ai/frame-pipeline.md: 'Frame cadence', 'Per-element composite timing + scene dump', 'What the first instrumented take said', 'Render budget', 'Scene compositor' and 'Live frame pipeline', byte-identical, plus a short 'Related in ai.md' footer and two stale 'below' pointers corrected. - ai.md keeps two stubs carrying the invariants that must not be missed (overlay on every render path, VideoFrame.Epoch, StripPool threads, Release-only render numbers; arrival-stamped rawvideo, non-blocking submit, stop order, burned _outputIndex judge) and links the new file. - docs/ai/index.md + docs/index.md route the split; schema.md inventory lists the new tier. - 4 index-row references and 2 TASKS.md anchors repointed; ai.md's Services row too. Measured: ai.md 63.6k -> 47.1k tokens, 2,702 -> 1,975 lines. Progressive disclosure per schema.md convention 5. No source touched; no push. --- Helpers/index.md | 4 +- Services/index.md | 4 +- TASKS.md | 4 +- ai.md | 810 ++------------------------------------ docs/ai/frame-pipeline.md | 803 +++++++++++++++++++++++++++++++++++++ docs/ai/index.md | 18 + docs/index.md | 11 + schema.md | 1 + 8 files changed, 881 insertions(+), 774 deletions(-) create mode 100644 docs/ai/frame-pipeline.md create mode 100644 docs/ai/index.md create mode 100644 docs/index.md diff --git a/Helpers/index.md b/Helpers/index.md index bf77170..9fa3204 100644 --- a/Helpers/index.md +++ b/Helpers/index.md @@ -15,8 +15,8 @@ Cross-cutting utilities. See [`schema.md`](../schema.md) for the memory-map conv | `InverseBoolToVisibilityConverter.cs` / `NotNullToVisibilityConverter.cs` | XAML value converters for visibility bindings | | `AllTrueToVisibilityConverter.cs` | Multi-value AND-gate → Visible; gates the layer-row capture menu items on row-IsBackground + staged-scene-is-Live (TASK 25) | | `CameraConflictProbe.cs` | Best-effort diagnostic: when a camera won't start, enumerates running processes known to hold cameras (OBS, Zoom, Teams, NVIDIA Broadcast, browsers, etc.) — Windows doesn't expose "which process has this device" via any public API, so this is a suspect list, not a verdict | -| `StripPool.cs` | Four DEDICATED background threads at `BelowNormal` priority that split one composite into horizontal row strips (the caller runs a strip too); started lazily, runs inline below 64 rows. UNIT 5 — do NOT demote the ThreadPool from `Parallel.For`, it leaks into unrelated work (see ai.md "Render budget") | -| `FrameCadence.cs` | Pure arrival-cadence counter (`FrameCadence` + `FrameCadenceSummary.Describe`): frames, gaps, hitches, longest/avg gap for a window of a timestamped stream. Fed by `MediaCaptureFrameSource` (camera stamps vs our arrivals) and `FramePump` (distinct webcam Epochs that reached the output) — UNIT 0, see ai.md "Frame cadence" | +| `StripPool.cs` | Four DEDICATED background threads at `BelowNormal` priority that split one composite into horizontal row strips (the caller runs a strip too); started lazily, runs inline below 64 rows. UNIT 5 — do NOT demote the ThreadPool from `Parallel.For`, it leaks into unrelated work (see [`docs/ai/frame-pipeline.md`](../docs/ai/frame-pipeline.md) → "Render budget") | +| `FrameCadence.cs` | Pure arrival-cadence counter (`FrameCadence` + `FrameCadenceSummary.Describe`): frames, gaps, hitches, longest/avg gap for a window of a timestamped stream. Fed by `MediaCaptureFrameSource` (camera stamps vs our arrivals) and `FramePump` (distinct webcam Epochs that reached the output) — UNIT 0, see [`docs/ai/frame-pipeline.md`](../docs/ai/frame-pipeline.md) → "Frame cadence" | | `HexToBrushConverter.cs` | A "#RRGGBB" hex string → `SolidColorBrush` (transparent on bad input), so a template can paint a colour swatch straight from the palette. Registered in the theme as `HexToBrush` — the theme paints the colour, never the UI code (TASK 63) | | `ClosedBoxTemplateSelector.cs` | An `ItemTemplateSelector` that gives a ComboBox's CLOSED box a different template from its drop-down rows: WPF calls it with the `ContentPresenter`'s templated parent — the `ComboBox` for the closed box, the `ComboBoxItem` for a row. The only way to tell them apart (`SelectionBoxItemTemplate` is read-only and mirrors `ItemTemplate`) — TASK 63 | diff --git a/Services/index.md b/Services/index.md index b2d8fcc..bd887bb 100644 --- a/Services/index.md +++ b/Services/index.md @@ -42,7 +42,7 @@ External-facing logic: YouTube API, persistence. See | `WebView2Manager.cs` | **Web-source ownership (TASK 17, composition-capture since 2026-09-14):** per registered `Source { Type: WebSource }` it owns a `CoreWebView2CompositionController` + its visual tree (`RootVisualTarget` child under a 1920×1080 root `ContainerVisual` on one `Windows.UI.Composition.Compositor` created on the UI thread via the CoreMessaging `CreateDispatcherQueueController` P/Invoke), a wrapped `WebCaptureFrameSource`, and the shared crop-sized `WriteableBitmap` preview (dispatcher-coalesced copy at Render priority, `PreviewBitmapChanged` raised only on recreation). Nav + `TransparentBackgroundScript` injection unchanged. Public ctor `(Dispatcher)`; internal seam ctor `(Dispatcher, Func?)` for hermetic tests (no WinRT). Caller: `MainViewModel.InitWebView2()` (now invoked from `MainWindow_Loaded` — the parent HWND must exist). See `../ai.md` "Slice 14" | | `WebCaptureFrameSource.cs` | The `IScreenCaptureSource` for a web session: owns the `GraphicsCaptureItem` (`CreateFromVisual`) + a free-threaded `Direct3D11CaptureFramePool` (2 buffers) + session; readback via `SoftwareBitmap.CreateCopyFromSurfaceAsync` (BGRA, **`BitmapAlphaMode.Straight`** — the compositor blends straight alpha); per-frame `FindContentBounds` alpha-bbox (internal static, unit-tested) stamps `VideoFrame.CropBounds`; ring up to 2 fresh frames + Epoch reuse; `IsOpaque=false`. Mirror of `ScreenCaptureFrameSource` | | `Compositor/SceneCompositor.cs` | **The output compositor (TASK 4 ship step 1)**: renders a scene into the encoder's master `VideoFrame` (tightly-packed BGRA8), mirroring the XAML preview minus editing chrome — backdrop → background → elements (`UniformToFill` cover-crop, round clip, mirror, opacity, border) → branding flash → social bar (optional `socialBarFrame` + `socialBarTop` in master space, blitted last so the bar sits above the flash; the old `BlitFlash` generalized to `BlitOverlay` with source offsets). **TASK 47:** `Render` + `CompositeLayers` take a trailing `tickerFrame` blitted at (0,0) after the bar — the alert ticker is a message-scrolling dynamic overlay, never baked. Pure and WPF-free: frames injected via a `Func` resolver (webcam → DeviceId, image → AssetId, backdrop → CaptureKey); output sized by `CompositorOptions` (16:9 = full master 1:1; vertical 9:16 = 607×1080 crop → 1080×1920 bilinear). Preview stays XAML (editing view); this is the output view — see `ai.md` "Scene compositor". **2026-10-16 (measurement only):** `Render`/`CompositeLayers`/`BakeStaticBase` take a trailing optional `CompositeTimings?` — every stage and every element timed (`# `, nested `raster`) and reported by the pump's parts line; zero cost for callers that pass nothing | -| `Compositor/CompositeTimings.cs` | **Per-element composite timing (2026-10-16, measurement only)**: the collector the pump owns for a stats window — `Add(stage, ticks)` accumulates, `Describe(renders, minMs = 0.1)` prints the averaged breakdown slowest-first. Stage names: `backdrop`/`base`/`flash`/`social`/`ticker`/`zoom`/`scale`/`overlay`, `bake backdrop`/`bake overlays`/`bake #i name`, and **one entry per element `# `** with the nested `raster` sub-stage (paste-cache rasterize miss) — the line that names WHICH of the six elements eats the render budget. See `ai.md` "Per-element composite timing + scene dump" | +| `Compositor/CompositeTimings.cs` | **Per-element composite timing (2026-10-16, measurement only)**: the collector the pump owns for a stats window — `Add(stage, ticks)` accumulates, `Describe(renders, minMs = 0.1)` prints the averaged breakdown slowest-first. Stage names: `backdrop`/`base`/`flash`/`social`/`ticker`/`zoom`/`scale`/`overlay`, `bake backdrop`/`bake overlays`/`bake #i name`, and **one entry per element `# `** with the nested `raster` sub-stage (paste-cache rasterize miss) — the line that names WHICH of the six elements eats the render budget. See [`docs/ai/frame-pipeline.md`](../docs/ai/frame-pipeline.md) → "Per-element composite timing + scene dump" | | `Compositor/CompositorOptions.cs` | The active tier's output rect (source space over the 1920×1080 master, integer-aligned — `MainViewModel.OutputRectX` can be 656.5) + target W×H | | `Compositor/StretchMath.cs` | Pure pixel math: the WPF `UniformToFill` cover-crop, clamped bilinear sample/scale (unit-tested half of the compositor) | | `Compositor/StaticPixelCache.cs` | Asset bytes → cached BGRA8 `VideoFrame`, decoded once per content hash — the output path's raw-pixel source (the preview uses ImageCache's `BitmapImage`) | @@ -58,7 +58,7 @@ External-facing logic: YouTube API, persistence. See | `Encoder/FfmpegArgs.cs` | Pure FFmpeg command-line builder: rawvideo `pipe:0` input, `anullsrc` silent audio (WASAPI replaces it), H.264+AAC, closed GOP (`-g fps×4 -keyint_min -sc_threshold 0 -bf 0`, `yuv420p`), FLV → RTMP | | `Encoder/FfmpegProgressParser.cs` | Pure parser for `frame=/fps=/size=/time=/bitrate=` stats lines → `FfmpegProgress` | | `Encoder/FfmpegEncoderPicker.cs` | Pure H.264 encoder picker from `-encoders` output: NVENC → QSV → AMF → OpenH264; **never returns libx264** (GPL) | -| `Encoder/FramePump.cs` | **TASK 4 ship step 5**: the live frame producer — while live it snapshots the active scene each tick, resolves every element to its latest frame (`Func` resolver), composites it into the tier's output frame, and paces frames into the encoder at the tier's FPS. All collaborators constructor-injected seams; free of WPF and the capture managers. `StartAsync` never throws (failures log + `Failed`); no RTMP URL = encoder skipped; `StopAsync` stops the encoder before awaiting the loop (backpressure deadlock); `ProcessFailed` self-stops. Since the bar bug-fix branch it takes an optional `socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam — a pre-rendered bar strip composited last (above the flash) at the top edge or `SourceRectHeight − bar height`. **TASK 47:** adds an `alertTicker: Func?` ctor seam (parallel to `socialBar`), read per tick and threaded through `RenderScene`/`RenderFull` + **mixed into `BuildFullRenderSignature`** so the ticker scroll changes the cache key. See `ai.md` "Live frame pipeline". **2026-10-16:** owns a `CompositeTimings` collector, passes it through `RenderScene`/`RenderFull`/`GetBakedBase`/`CompositeLayers`, logs `FramePump parts (N renders/5s): …` (per-stage breakdown, slowest first) beside the stats line each 5s window, and dumps the scene shape — `FramePump scene dump:` header + one line per element (z-index, name, kind, type, visibility, rect, opacity, clip) — at pump start and on every scene change | +| `Encoder/FramePump.cs` | **TASK 4 ship step 5**: the live frame producer — while live it snapshots the active scene each tick, resolves every element to its latest frame (`Func` resolver), composites it into the tier's output frame, and paces frames into the encoder at the tier's FPS. All collaborators constructor-injected seams; free of WPF and the capture managers. `StartAsync` never throws (failures log + `Failed`); no RTMP URL = encoder skipped; `StopAsync` stops the encoder before awaiting the loop (backpressure deadlock); `ProcessFailed` self-stops. Since the bar bug-fix branch it takes an optional `socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam — a pre-rendered bar strip composited last (above the flash) at the top edge or `SourceRectHeight − bar height`. **TASK 47:** adds an `alertTicker: Func?` ctor seam (parallel to `socialBar`), read per tick and threaded through `RenderScene`/`RenderFull` + **mixed into `BuildFullRenderSignature`** so the ticker scroll changes the cache key. See [`docs/ai/frame-pipeline.md`](../docs/ai/frame-pipeline.md) → "Live frame pipeline". **2026-10-16:** owns a `CompositeTimings` collector, passes it through `RenderScene`/`RenderFull`/`GetBakedBase`/`CompositeLayers`, logs `FramePump parts (N renders/5s): …` (per-stage breakdown, slowest first) beside the stats line each 5s window, and dumps the scene shape — `FramePump scene dump:` header + one line per element (z-index, name, kind, type, visibility, rect, opacity, clip) — at pump start and on every scene change | | `Encoder/FfmpegLocator.cs` | The default locator: PATH probe wins, then the cache, then pull+extract (`ffmpeg.exe` **+ the `libav*.dll` family** via a staging dir, failures via `AppLog`). Pinned to BtbN `autobuild-2026-08-09-13-03` **lgpl-shared** (dynamic linking = LGPL compliance without static-relink §6 material; no GPL libx264; NVENC/QSV/AMF + libopenh264 + native AAC — see `ai.md` → Licensing). Search dirs / tools dir / downloader constructor-injected for hermetic tests. Wired into the encoder since ship step 5 | | `INotificationService.cs` | **Toast seam (TASK 24)**: `Show(title, message, AppNotificationSeverity)` + `AppNotificationSeverity { Info, Success, Warning, Error }` + `Info/Success/Warning/Error` extension methods — non-blocking user-facing messages; VMs/services raise through this so failures surface without MessageBox blocking the UI thread | | `NotificationService.cs` | The default seam impl (Notification.Wpf 11.0.0, MIT): wraps a library `NotificationManager`; routes to the in-window `NotificationArea` named `ToastArea` (MainWindow bottom-right above the footer); lifetimes via the pure `BuildRequest` mapping + `Cards` tint table — Info/Success ~4s, Warning ~8s auto-dismiss, Error sticky (`NeverExpires()` = `TimeSpan.MaxValue`, never null); dark card tints matching the health-banner language; all Show calls marshal to the captured Dispatcher. See [`../ai.md`](../ai.md) "Toast notifications" | diff --git a/TASKS.md b/TASKS.md index bd11744..ab787bb 100644 --- a/TASKS.md +++ b/TASKS.md @@ -182,7 +182,7 @@ second encoder path needing a "redirect". Record+simulcast is one ffmpeg with tw (user-queued): static (+) catalog rows (Background, YouTubeEvent, quoted labels) with reason-greying. - **🔶 Webcam cadence → resolution — IN PROGRESS, creator-queued 2026-10-05** (DHH/lipreading: the webcam judders, ~1 hitch in 5, invisible in the output file). **UNIT 0 shipped** (counters + - `Helpers/FrameCadence.cs`; see `ai.md` → "Frame cadence"). **Needs a creator recording before the + `Helpers/FrameCadence.cs`; see [`docs/ai/frame-pipeline.md`](docs/ai/frame-pipeline.md) → "Frame cadence"). **Needs a creator recording before the next unit.** Remaining units, all agreed 2026-10-05, one Good Dog test each: - **⛔ UNIT 1 IS NOT NEXT — re-scoped 2026-10-05.** The first real reading exonerated the camera: it delivers 30.0/s with zero hitches; the compositor manages **6.8 composites/s at ~141ms each**, @@ -191,7 +191,7 @@ second encoder path needing a "redirect". Record+simulcast is one ffmpeg with tw and schedule it AFTER the compositor work. - **✅ SHIPPED 2026-10-16 (measurement only): per-element composite timing + a scene dump.** `CompositeTimings` + the pump's `FramePump parts (N renders/5s): …` line and - `FramePump scene dump:` block (see `ai.md` → "Per-element composite timing + scene dump"). + `FramePump scene dump:` block (see [`docs/ai/frame-pipeline.md`](docs/ai/frame-pipeline.md) → "Per-element composite timing + scene dump"). One integration test: `ytLive.Tests/CompositeTimingTests.cs`. **NOW WAITING ON A CREATOR TAKE (Release):** the parts line must name which of the 6 elements eats the ~140ms and the dump must show the shape of the slow take vs the fast one (10:03 → 12:01, 29–34ms → 143–155ms, same shape). diff --git a/ai.md b/ai.md index 897d5ee..3d4d31c 100644 --- a/ai.md +++ b/ai.md @@ -5,6 +5,11 @@ > palette and launch strategy in [`MARCOM.md`](MARCOM.md) (gitignored — read > it before any branding or marketing work); directory maps in each folder's > `index.md`. Reading order: this file → `TASKS.md` → `MARCOM.md` → `/index.md` → source. +> +> **Subsystem detail is split out:** large blocks of this file now live under +> [`docs/ai/`](docs/ai/index.md) and this file keeps a short invariant stub + link where each was +> removed. Open [`docs/ai/index.md`](docs/ai/index.md) to find the right one, or ignore it entirely +> until your task touches a moved subsystem. ## Response style @@ -259,7 +264,7 @@ C# / WPF (.NET 8) following MVVM: |------|------| | `Models/` | Plain data types — Scene, Source (incl. `ClipShape`, `IsMirrored`, `VideoImageSource`), QualityOption, StreamConfig, StreamHealth, YouTubeChannel, ChatMessage, **Socials (`SocialService` enum + `SocialEntry`/`SocialsConfig` + `SocialServiceIcons`) — the social bar** | | `ViewModels/` | MainViewModel — `public partial class`, one file per functional area (Scenes, Background, Webcam, Audio, Trax, Socials, Streaming, Chat, Overlays, Account, License, Recording — split complete, see `ViewModels/index.md`); **Chat.cs is a thin delegating facade over `Services/ChatOverlayLayer.cs` (Commit G, first true decomposition), and now also over `Services/AlertOverlayLayer.cs` (TASK 43 — the native alert box, same true-component pattern)**; GoLiveViewModel, ReuseImageViewModel, CameraPickerViewModel, **SocialsDialogViewModel** | -| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **media source: `IMediaFrameSource` + `MediaVideoSource` (spawns ffmpeg `rawvideo` BGRA decode) + `MediaVideoSourceManager` (refcount-by-path session owner) + pure `RawVideoFrameReader` + `IDecodeProcess`/`FfmpegDecodeProcess` process seam (binary-stdout mirror of `IEncoderProcess`; see "Media source")**, **compositor: `SceneCompositor` + `CompositorOptions` + pure `StretchMath` + `StaticPixelCache` (see "Scene compositor")**, **alerts: `ChatOverlayLayer` + `AlertOverlayLayer`/`AlertRenderer` (the native celebration-zone overlay, TASK 43 — see "Native events & alerts")**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat`/`VoiceFilterChain`/`LowShelfFilter`/`HighShelfFilter`/`NoiseGate`/`Compressor`/`AutoDucker`/`AudioRingBuffer`/`TinyResampler`/`AudioSyncDelay` + `MusicPlayer` + `ITraxPlayer` (TRAX toggle seam — FakePlayer in tests; the dot is green/red/gray and the error scrolls terse in the desktop-audio bar; MusicVolume fixed 0.5) + `IAudioPipeWriter`/`NamedPipeAudioWriter` (see "Live audio capture")**, **encoder: `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/`FfmpegEncoderProcess` + `IFfmpegLocator`/`FfmpegLocator` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` + the `FramePump` frame producer (see "Live encoder" + "Live frame pipeline")**, **notifications: `INotificationService` seam (`AppNotificationSeverity` Info/Success/Warning/Error) + `NotificationService` (Notification.Wpf toasts, see "Toast notifications")** | +| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **media source: `IMediaFrameSource` + `MediaVideoSource` (spawns ffmpeg `rawvideo` BGRA decode) + `MediaVideoSourceManager` (refcount-by-path session owner) + pure `RawVideoFrameReader` + `IDecodeProcess`/`FfmpegDecodeProcess` process seam (binary-stdout mirror of `IEncoderProcess`; see "Media source")**, **compositor: `SceneCompositor` + `CompositorOptions` + pure `StretchMath` + `StaticPixelCache` (see [`docs/ai/frame-pipeline.md`](docs/ai/frame-pipeline.md) → "Scene compositor")**, **alerts: `ChatOverlayLayer` + `AlertOverlayLayer`/`AlertRenderer` (the native celebration-zone overlay, TASK 43 — see "Native events & alerts")**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat`/`VoiceFilterChain`/`LowShelfFilter`/`HighShelfFilter`/`NoiseGate`/`Compressor`/`AutoDucker`/`AudioRingBuffer`/`TinyResampler`/`AudioSyncDelay` + `MusicPlayer` + `ITraxPlayer` (TRAX toggle seam — FakePlayer in tests; the dot is green/red/gray and the error scrolls terse in the desktop-audio bar; MusicVolume fixed 0.5) + `IAudioPipeWriter`/`NamedPipeAudioWriter` (see "Live audio capture")**, **encoder: `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/`FfmpegEncoderProcess` + `IFfmpegLocator`/`FfmpegLocator` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` + the `FramePump` frame producer (see "Live encoder" + [`docs/ai/frame-pipeline.md`](docs/ai/frame-pipeline.md) → "Live frame pipeline")**, **notifications: `INotificationService` seam (`AppNotificationSeverity` Info/Success/Warning/Error) + `NotificationService` (Notification.Wpf toasts, see "Toast notifications")** | | `Helpers/` | ViewModelBase (INotifyPropertyChanged), RelayCommand, ImageCache, AppLog (file logger), FocusPreservingListBox, OAuthCredentials, **TokenStore (DPAPI session persistence)**, **BuildStamp (wordmark release counter `#N`, +1 per commit, generated by `GenerateBuildStamp` in ytLive.csproj via git rev-list; the per-build GUID now logs to startup.log only — 2026-09-04)**, visibility converters | | `Themes/` | `Controls.xaml` — the single dark-theme source, merged once in `App.xaml` (see `Themes/index.md`) | | `MainWindow.xaml` | Dark theme; layout: top bar (controls), center (preview + live controls below), left (scenes/sources), right (chat), bottom (gear + stream stats + resolution) | @@ -845,292 +850,27 @@ to 16:9 the desktop layout is restored exactly and the circle option is offered broke `WebcamBackgroundPillTests` — that test finds the webcam's panel by **binding text**, so a new panel must use a different binding (`IsWebcamSelected`) or the selector matches twice. -### Frame cadence — how a dropped frame became measurable (UNIT 0, 2026-10-05) +### Rendering budget, measurement & the compositor — moved -**The problem this exists to solve.** The recorded file cannot distinguish a dropped camera frame -from a padded repeat. `-f rawvideo` carries no per-frame timestamps — ffmpeg assigns pts from frame -order at the DECLARED fps — so a webcam delivering 25 of its 30 frames still muxes a full-length, -correctly-timed 60fps file. Measured on take `ty-20261005-1001-821-0001.mp4` (85.6s, 1080p60): the -lower-right webcam region held 25.6 distinct images/s against the camera's 30fps, with 1,153 normal -33ms intervals, 281 gaps of 50–100ms, and 96 bunched 17ms gaps — roughly one hitch in five, which is -what makes a face judder while the file looks flawless. (The "webcam renders at ~18fps" figure that -used to sit in this map was WRONG: it came from a different session and counted every newly rendered -layer, not webcam arrivals. Corrected 2026-10-05.) +Frame cadence, per-element composite timing, the scene dump, the render budget and the +`SceneCompositor` design live in **[`docs/ai/frame-pipeline.md`](docs/ai/frame-pipeline.md)** — +read it before changing the render loop, the compositor, or anything that moves per-frame cost. -**`Helpers/FrameCadence.cs`** — one pure counter, three feeds. `Add(timestamp)` on any shared -monotonic clock; `Summarize()` → frames, rate (gaps ÷ span, NOT frames ÷ span — N frames span N-1 -periods), gaps, hitches (default threshold 50ms ≈ 1.5× a 30fps period), longest and average gap; -`Describe(label)` gives every producer the same one-line shape so two lines can be read against each -other. An out-of-order stamp counts as a frame but produces no gap AND does not become the reference -for the next interval — otherwise one bad stamp invents a multi-second "hitch" twice. +Invariants that hold even if you never open that file: -**Three measurement points, and what each separates:** - -| Line | Fed by | Answers | -|---|---|---| -| `Camera cadence WxH: arrivals … \| camera …` | `MediaCaptureFrameSource` | device/USB/driver cadence (`MediaFrame.SystemRelativeTime`, QPC domain, nullable) vs what the reader handed us. `TryAcquireLatestFrame` keeps only the LATEST frame, so falling behind is silent by design — this is where that shows | -| `webcam …` on the pump's stats line | `FramePump.TimedResolver` | distinct webcam `VideoFrame.Epoch` values that reached a composite — what actually got into the output | -| `frames per 5s from R renders` | `FramePump` | submits vs REAL composites — the duplicate-on-lag ratio, the only honest reading of how much of the stream is a repeat | - -**Why the stats-line denominators changed (the actual bug).** Every average was divided by -`statFrames`, the number of frames SUBMITTED — which deadline padding always fills to the full 60fps -target. A pump fed one webcam frame per 200ms logged "300/300 frames per 5s, avg render 12.4ms" -while producing 60 distinct composites: the render average was diluted ~5× and the loss was -invisible. Render/resolve/wait are now divided by `statRenders`; submit stays per-frame (it really -happens per submit). Regression test: -`FramePumpTests.Stats_ReportRealRendersAndDistinctWebcamFrames_NotJustPaddedSubmits`. - -**Reading the three together:** camera 30/s steady + arrivals 25/s = WE dropped it (reader or -compositor). Ragged in both = the device, the USB link, or the driver. Pump `webcam` below -`arrivals` = the compositor missed frames the reader had already delivered. - -**✅ WHAT THE FIRST REAL READING SAID (2026-10-05) — the camera was never the problem.** -`ty-20261005-1201-556-0001.mp4` + its log: the C920 delivers **30.0/s with ZERO gaps ≥50ms**, and the -device's own stamps are identical, so the camera, USB link, and driver are all clean. The pump did -**34 composites per 5s (6.8/s) at ~141ms each** (worst 294ms), and the file holds **6.8 distinct -webcam images/s at 88.7% duplicates with no arrival hitches at all**. The judder is entirely us -holding one face for 140ms and padding the rest of a 60fps container. `resolve` is 0.1ms, so it is -pure pixel work in the composite, not frame lookup. Two consequences, both load-bearing: - -- **720p is a sharpness win, NOT a smoothness win** — it triples the bytes stretched per render on a - path already 4.4× over budget. Schedule the exclusive lock AFTER the compositor work or it makes - the judder measurably worse. -- **The 10:01 take hit 25.6 images/s and the 12:01 take 6.8, same scene shape - (`full-render split=0 elements=6 dynamic=5`), same 640×480 camera.** Composites went 29–34ms → - 143–155ms between sessions with nothing logged explaining it. That comparison was made by EYE on - two takes because the log carried no per-element breakdown and no scene dump — both now exist - (see "Per-element composite timing + scene dump" below); naming the culprit is a reading, not a - guess, and guessing is explicitly out of bounds. **Superseded 2026-10-16:** "same scene shape" was - an eye comparison of takes whose geometry was never logged, and the next bad take still on disk is - the *vertical* tier (below). A Release 1920×1080 take now measures 1.0ms with no element over - 0.4ms — see "Per-element composite timing + scene dump" → what the first instrumented take said. - -**Counter subtlety, so it is not re-derived:** the pump's `webcam` line read 13/s while the file held -6.8/s. Not a bug — the counter stamps a frame when the resolver FIRST sees it, and one composite takes -141ms, so ~4 camera frames arrive during it (signature pass stamps one epoch, the actual draw uses a -later one). The file is the truth; the pump counter is a pipeline-health signal and overcounts by -design. - -**Still unmeasured:** the camera is stuck at the C920 default 640×480 YUY2 @30 because all 185 -`SetMediaStreamPropertiesAsync` attempts are refused under `SharedReadOnly`, so 720p MJPG is available -— but see above, it is a quality win only. - -### Per-element composite timing + scene dump (UNIT 0 follow-up, 2026-10-16 — MEASUREMENT ONLY) - -The gap the cadence work left open is now closed by instrumentation, and closed by nothing else: the -141ms take was never compared to the 30ms take because **no line in the log said which element cost -what, and no line said what the scene WAS.** Both now exist. No fix has been attempted, and none is -allowed until a real reading names the culprit. - -- **`Services/Compositor/CompositeTimings.cs`** — a collector owned by the pump, threaded through - `SceneCompositor.Render` / `CompositeLayers` / `BakeStaticBase` / `BlitBakedElement` / - `BlitCachedLayer` (and `SceneGraph.GetBakedBase` → bake path) as an optional last parameter, so - every other caller (thumbnails, tests, transition preview) is untouched and costs nothing. - `Add(stage, ticks)` accumulates; `Describe(renders, minMs = 0.1)` renders the averaged breakdown - slowest-first, dropping sub-0.1ms noise. -- **Stage names** (stable, greppable): `backdrop` (prefill + live capture + background art + their - resolves), `base` (the cached-bake `BlockCopy`), `flash`, `social`, `ticker`, `zoom`, `scale`, - `overlay` (the fully-static path's per-frame overlay copies), `bake backdrop`, `bake overlays`, and - **one entry per element as `# `** — the index IS the z-order, so the log names the - layer by both position and name. Bake-pass elements are prefixed `bake #…` so a one-off rebake - cannot masquerade as a per-composite cost. -- **`# raster` nests inside its element's total** and is the paste-cache rasterize miss - (`BlitCachedLayer`'s `raster == null` branch) — the direct answer to "is one element re-sampling - every tick, or is the paste itself slow". A stage's time includes that stage's frame resolve; - `resolve` is separately totalled in the existing stats line. -- **`FramePump parts (N renders/5s): …`** — logged once per 5s window (same window as - `FramePump stats`), averaged per render so the numbers compare directly with `avg render`, then the - collector is reset. **No parts line, or a short one, is a reading too:** the fully-static shortcut - and C4 cache hits composite no elements (`cache 92R/130H` beside a missing parts line is expected). -- **`FramePump scene dump: '' elements=N split=S`** followed by one line per element - (`[i] name kind=… type=… vis=… rect=x,y WxH op=… clip=…`), logged when the pump first runs and - whenever the active scene changes. This is the record the 10:01/12:01 comparison never had. -- Gate test: `ytLive.Tests/CompositeTimingTests.cs` (one `[Fact]`) — a REAL composite of a full-canvas - element plus a 64×64 one must rank the full-canvas element first, carry the `raster` sub-stage and - the fixed stages, and the dump must name both elements with z-index/type/rect and echo the split. -- **`scripts/webcam-cadence.py`** — the other half of a reading: what the FILE holds (distinct webcam - images/s + hold histogram), beside what the pump says it did. Recipe and its three traps (H.264 - re-encoding, the round-clip crop, scene switches): `MyMistakes.md` → "Measuring webcam CAMERA cadence - in a recording". - -**Read takes in Release.** The creator launches `dotnet run -c Release`; Debug pixel paths are -several× slower, so a Debug number is not a reading of this pipeline (see Run). - -**Derivative check (AGENTS.md):** this is not invented — OBS's own answer to "which source is slow" is -the **Source Profiler**, which reports the *average execution time of each source's tick and render -functions* ([docs.obsproject.com → Source Profiler](https://docs.obsproject.com/reference-libobs-util-source-profiler), -per-source timers via `util/profiler.h`). Same shape here: one timer per element per composite, reported -as an average, plus the stage split. The `# raster` sub-stage is the extra this case needs — -it separates "re-sampled every tick" from "slow paste" without another session. - -### ✅ What the first instrumented take said (2026-10-16, Release) — the judder DID NOT reproduce - -Take `ty-20261016-1841-559-0002.mp4` (174.5s, 1920×1080@60, Release build `e95154d9` — a **recording**, -not preview; scene `Live`, switched to `Chat` at 28s, back to `Live` at 44s) and its log window: - -| Reading | This take | -|---|---| -| composites | **301 per 5s = 60/s**, every slot; 0 dropped, 0 stalls after warm-up | -| avg render | **1.0–1.1ms**, worst 2.5ms — ~7% of the 16.67ms budget | -| `parts` | `#5 HD Pro Webcam C920 0.4ms, #5 … raster 0.2ms, backdrop 0.2ms` — the other four elements are under the 0.1ms print floor | -| C4 full-composite cache | **151R/150H** per window — half the ticks were a whole-frame `BlockCopy` | -| webcam at the resolver | **30.0/s**, longest gap **50.0ms** (one extra 60Hz frame = sampling phase, not a drop) | -| the file itself | **29.6 distinct webcam images/s, 82% of images held exactly 2 frames**, longest hold 267ms (`scripts/webcam-cadence.py`) | - -Nothing in this take is over budget and **no element is expensive — there is no culprit here to fix.** -The two stalls in the take are not per-composite cost: 124ms on the very first render (warm-up) and -53ms on the switch into `Chat` (a rebake). - -**Scale of the disagreement:** the historic slow readings are 141ms (Oct 5 12:01) and 45ms (Oct 9 -19:48) — 40–140× the ~1ms this take needed per composite. Three facts about those takes, all read off -the same log: - -- **Oct 9 19:48 (`ty-20261009-1948-121-0000-2.mp4`) was the VERTICAL tier**, not a 16:9 one: - `FramePump started (1080×1920 @ 60 fps) scene='Live' comp=608x1080` — the 607×1080 centre crop - bilinear-upscaled to 1080×1920 (2M px), 92 renders/5s, avg render 45.0ms, worst 278ms, 92 stalls, - `cache 92R/0H`. The file on disk is 11.6s and every frame is identical (a static desktop), so it - cannot be read for cadence — only its cost. -- **Oct 5 12:01 (`…1201-556-0001.mp4`, no longer on disk) logged `FramePump started (1920×1080 @ 60 - fps)` with NO `scene=/comp=` suffix** — that suffix did not exist yet, so its source rect was never - recorded and the map's "21:9 → 1920×803" explanation stays a hypothesis. -- **A take can latch mid-take.** Oct 5 12:29 goes 1.5ms (avg render) → 3.0ms → **45.2ms within 15 - seconds and stays there** (`cache 10R/0H`, then `98R/0H`). So the slow state is not a fixed property - of a scene geometry. The consistent signature of every slow window is `cache xR/**0H**`: the C4 - whole-frame cache never reused once, so every tick composited. The healthy take shows the other - stable state (composite 1ms → every second tick is byte-identical → whole-frame reuse). **The latch - is the thing to test on a reproducing take** — it is a hypothesis, not a reading; the `parts` line - is what will decide it. - -**What this does NOT say.** This take is the 1:1 master tier and its scene logs -`full-render split=0 elements=6 dynamic=4` (the bad-period takes logged `dynamic=5`), so it never -exercises the failing geometry. "The healthy path is healthy" is not "the judder is gone". Reproducing -it needs a Release take in the failing configuration (vertical tier first, then the 21:9 display -config) — that take's `parts` line names the stage, and only then is a fix unit allowed. - -**Reading the `parts` line's denominators:** like `avg render`, it is divided by `statRenders`, which -counts C4 REUSES as well as real composites (the 301 above = 151 real + 150 whole-frame copies). So in -a healthy 30fps-camera take both averages are ~2× optimistic; the `cache NR/NH` pair on the same line -says how much of the window was real composite work. `CompositeTimings.Describe` drops stages under -0.1ms by default, and **a missing or short parts line is itself a reading** — a fully-static window -(every tick a C4 hit) composites no elements at all. - -### Render budget — getting a composite inside 16.67ms (UNITs 4-5, 2026-10-05) - -The pump emits one encoder frame per 16.67ms slot and **pads with a duplicate when a composite -overruns**, so the whole problem is "render + submit + untimed work ≤ 16.67ms, ideally ≤12ms to -leave headroom". `FramePump` now reports per-stage timings (`social`/`ticker`/`flash`/`acquireScratch`) -so a slow stage names itself, and logs the scene name + compositor source rect at pump start. - -**Why a 21:9 monitor cost 141ms and a 16:9 one 33ms — the fast path is EXACTLY 1920×1080.** A -21:9 display shrinks the source rect to 1920×803 ("fit", not "fill"), so the opaque-backdrop 1:1 row -walk is skipped, the base is prefilled with opaque black (~2M byte writes), and the backdrop is -resampled per-pixel (~1.54M pixels with an alpha branch). Every recorded A/B lines up: -21:9 → 141ms, 16:9 → 30-33ms, 2560×1440 (fits to 1920×1080 exactly) → fast path. -**Corrected 2026-10-16 — the "never Debug-vs-Release" half of that is withdrawn.** A Release -1920×1080 take measures **1.0–1.1ms** where an Oct 9 take of the same tier measured 10.4ms, and -`Build ` carries no build configuration, so the config of every pre-2026-10-16 take is -UNKNOWN: no historic render number may be compared against a Release take. The 21:9 explanation -itself stays a hypothesis — the 141ms take's start line predates the `scene=/comp=` suffix, so its -source rect was never recorded. A take can also latch mid-take (see the instrumentation section). - -**Unit 4 — cheap wins, all pixel-identical:** -- Opaque 1:1 fast paths expanded to the whole-canvas opaque case (a `BlockCopy`, not a per-pixel - alpha branch). -- **Prefill only the uncovered rows.** When the backdrop does not cover the canvas, the letterbox - bands are filled instead of the whole buffer; with no backdrop at all it still fills everything. -- **Per-column LUT** for the scaled blit (`BlitContentRaw`): the source column, its neighbour index - and the 8.8 weights are computed once per column instead of a divide + clamp per pixel. - The table comes from `ArrayPool.Shared` and is returned in a `finally` — a single static - buffer is a **data race** the moment strips run in parallel (see below). - -**Unit 5 — strip-parallel composite: `Helpers/StripPool.cs`.** One composite splits into four -horizontal row strips; the calling thread takes a strip too, so four workers never means five-way -oversubscription. Below 64 rows the fan-out costs more than it saves and runs inline. Each strip must -write only its own rows — the blits already partition by row, which is why the output stays -byte-identical. - -**The demotion must come from DEDICATED threads, not the shared pool.** The obvious -`Thread.CurrentThread.Priority = BelowNormal` inside `Parallel.For` is wrong: `Parallel.For` runs on -shared ThreadPool threads, so the demotion leaks into every unrelated work item that thread later -picks up. Research (https://stackoverflow.com/questions/524483/net-backgroundworker-objects-thread-priority -— "you have next to no control of which thread pool thread will actually run your task"; -https://stackoverflow.com/questions/1578599/c-sharp-lower-thread-priority-in-thread-pool — "You -shouldn't mess with the priority of the regular ThreadPool, since you aren't the only consumer ... -write your own producer/consumer queue, using your own Thread(s) as the consumer(s) - setting the -thread priority yourself") says exactly that, and **a real failed attempt confirmed it**: a full test -run after demoting the pool threads broke an unrelated audio-mix timing test. `StripPool` IS that -producer/consumer queue — four `Thread`s with `Priority = BelowNormal`, `IsBackground = true`, -started lazily on first use, each fed a `Queue` + `SemaphoreSlim`. Four is the measured sweet -spot: fewer leaves render over budget, more steals cycles from the game and the encoder and made -render **worse**. - -Gate test: `SceneCompositorTests.StripParallel_RepeatedRenders_AreByteIdentical` — the same scene -(scaled round webcam + mirrored image + master-sized flash over a short backdrop, 240 rows so all -three fanned-out paths run) must come out byte-for-byte identical across 8 renders, and must differ -from a backdrop-only render so it cannot pass vacuously. It earned its keep immediately: it caught a -loop-variable capture (`new Thread(() => Worker(i))` where `i` was `Workers` by the time the thread -ran) that crashed the test host on first run. - -### Scene compositor (TASK 4 ship step 1 — shipped 2026-08-10, plan in TASKS.md) - -The encoder needs the master 1920×1080 frame **without** the preview's editing chrome (SelectionOverlay, -DimRects, output-rect outline, badge, placeholder). WPF's `RenderTargetBitmap` is software-rendered and -captures the visual tree *including* chrome, so the preview can't be captured — the output is a **second, -parallel software compositor** over the `VideoFrame` (BGRA8) seam, and the XAML preview -(`MainWindow.xaml` CanvasGrid + element DataTemplate) is the rendering contract it replicates. Two -renderers must agree: geometry, `UniformToFill` cover-crop, round clip, mirror, border, z-order. Preview -stays XAML (editing view); the compositor is the output view. - -- **The `DimRects` shading is itself preview chrome, so it renders as a SIBLING of `CanvasGrid` - (above it, below the `ZoomBox`), never a descendant** (2026-10-08, creator ruling): the zoom box's - `VisualBrush` samples all of `CanvasGrid`, and if the dims were inside it the box would show the - dim shading — especially after the box is dragged away and the fixed capture slides under the dims. - -- **Render the active tier's output rect directly** (`CompositorOptions {SourceRectX/Y/W/H, - OutputWidth, OutputHeight}`, fed from `MainViewModel.OutputRect*`): 16:9 = full 1920×1080 1:1; the - vertical 9:16 tier = composite the centered 607×1080 crop then bilinear-upscale to 1080×1920. -- **CPU posture:** with FFmpeg as a subprocess the master crosses a CPU readback to the pipe every frame - anyway, so GPU compositing buys little at this layer count (2-3 live layers; static layers - pre-composite once into a cached base). GPU effort belongs to **NVENC** (the encoder), not composition; - if composition grows (wipes, filters, many layers), a D3D11 compositor can replace this one **behind - the same seam** — the CPU master buffer stays the contract. -- **Every global overlay must be composited on EVERY render path (bug found 2026-09-26).** The - frame pump has three exits: `RenderFull` (no graph / no static base), `CompositeLayers` (static base - + dynamic tail), and the **fully-static shortcut**, which stretches the cached bake and returned it - directly. That third path silently **discarded every per-frame overlay** — the social bar, the alert - ticker and now the brand flash. A creator whose scene was one static background saw none of them, - and the flash looked "preview-only forever" for a reason unrelated to the tier. `SceneCompositor.Overlay` - is therefore `internal static` (it copies before blitting, so the cached bake is never mutated) and - the shortcut now composites overlays onto a copy before stretching. **When adding an overlay, grep - for all three paths** — a fourth exit that forgets it fails the same silent way. -- **`VideoFrame` buffers are recycled by some producers, and `Epoch` is not optional.** A producer that - hands out the same array across frames MUST bump `Epoch`, or `FramePump`'s buffer-identity paste cache - false-hits and freezes stale content (the take-11 class of bug). The `C4` cache signature mixes - `Epoch`, so a per-frame overlay naturally invalidates it. `BrandFlashPresenter` reuses one 8MB - master buffer and stamps `Epoch` per read for exactly this reason — allocating a fresh 8MB frame per - tick is not an option at 1080p30. -- **Dev-only: two instances side by side (`Helpers/InstanceProfile.cs`).** Pre-1.0 the creator runs one - instance to stream and another to screen-capture it. Nothing prevented a second instance (no mutex, no - port); what broke it was shared state — the whole-scene read/write layout DB (clobber), Chromium's - **exclusive** lock on the WebView2 user data folder (second instance won't start), the auth token store - (a test instance overwrites the real YouTube sign-in), and `startup.log`. Set - `YTLIVE_INSTANCE=` and that process gets a private root at `%APPDATA%\ytLlive\instances\\` - for the DB, the auth file, the log and the WebView2 folder. Recording folder and the ffmpeg `tools\` - cache stay shared on purpose; global hotkeys stay un-namespaced (Windows refusing the second - `RegisterHotKey` is the correct answer). The **entire implementation is inside `#if DEBUG`** — a - Release build compiles to `DataRoot => DefaultRoot` and `WebViewDataFolder => null`, and the call - sites are unconditional so Release cannot drift. Proof + harness: `ytLive.Tests/InstanceIsolationTests.cs`. - ⚠️ Grepping the binary for `YTLIVE_INSTANCE` proves nothing — a `const` is inlined and appears in - neither build; check the `InstanceVariable` **field** in metadata instead. -- **Branding flash is composited by the output path too** (it's on the live output, per Monetization), - passed in as a pre-rendered `VideoFrame?` — the compositor core stays pure byte-math, no WPF. Likely a - bundled asset rather than runtime text rendering (deterministic, no font/layout risk). -- **Frame sources are injected** via a `Func` resolver - (`SceneCompositor.Render(scene, frameFor, flashFrame, options)`) — the caller maps each element to - its frame (webcam → `DeviceId`, image → `AssetId` via `StaticPixelCache`, backdrop → `CaptureKey`), - so the compositor is pure, WPF-free, and hermetic to test. The capture managers wire into that - resolver in the encoder step, not the compositor step. The master buffer (the compositor's return - value) is the seam a future D3D11 compositor would honor identically. +- **Every global overlay must be composited on EVERY render path** (pump, preview/thumbnails, scene + dump — `SceneCompositor.Overlay` is shared for all three). A feature wired into one path is silently + absent from the output; that bug (flash + social bar "preview-only forever", 2026-09-26) is why the + rule exists. +- **`VideoFrame.Epoch` is not optional.** A producer that recycles buffers MUST bump `Epoch` per + frame, or `FramePump`'s buffer-identity paste cache and the full-render signature serve stale + pixels. `BrandFlashPresenter` reuses one 8MB master buffer and stamps `Epoch` per read for exactly + this reason. +- **`StripPool` owns four dedicated `BelowNormal` background threads** for strip-parallel + compositing; strips below 64 rows run inline. Never demote threads inside `Parallel.For` — it runs + on thread-pool threads and starves the scheduler (that mistake once broke audio-mix timing). +- **Read render numbers in Release only.** Debug pixel paths are several× slower; a Release 1080p + composite measures ~1.0–1.1ms against the 16.67ms budget, and only Release takes are evidence. ### FFmpeg locator (TASK 4 ship step 2 — shipped 2026-08-10; **rewritten to BUNDLED 2026-09-28**, plan in TASKS.md) @@ -1431,492 +1171,26 @@ devices, no timers). test `Mix_HonorsProviderGains_AndGameMute_KillsTheLoopback` proving the provider gains reach the pipe and mute silences the loopback). -### Live frame pipeline (TASK 4 ship step 5 — shipped 2026-08-12, health stats 2026-08-13, plan in TASKS.md) +### The live frame pipeline — moved -The **`FramePump`** (`Services/Encoder/`) is the live frame producer: while live it snapshots the active -scene each tick, resolves every element to its latest frame, composites it into the tier's output frame, -and paces frames into the encoder at the tier's FPS. **rawvideo is stamped by ARRIVAL:** ffmpeg assigns -pts from frame order at the declared fps — supply rate = output speed. A starved producer (pump < fps) -ships a time-lapse, truncated file with NO error (take-2 lesson, 2026-09-01); the pump therefore logs -`FramePump stats: n/target frames per 5s from R renders, avg render Xms (resolve R2), avg submit Yms` so a slow stage names itself (the `frames … from R renders` pair and the render-count denominators arrived with UNIT 0 — see "Frame cadence" below; before that every average was divided by the SUBMITTED count, which deadline padding always fills to the full target, so a pump fed one webcam frame per 200ms still logged "300/300 frames" and its render average was diluted ~an order of magnitude) — and since 2026-09-04 so does the BUILD: every build gets a GUID stamped -into `Helpers/BuildStamp` (csproj `GenerateBuildStamp` target, fresh per compile — no lying incremental build), logged at startup -("Build xxxxxxxx (compiled ...)"). **Take 6 (stamp-less, ambiguous): render 35-41ms with slice 3 built or not unknown — attribution is why the stamp exists; takes 7+ are -readable.** If the breakdown says resolve dominates, the suspects are pre-cached chat/config thrash or the web frame; if blit dominates, the general-path element sizes. **Pattern — everything is a constructor-injected -seam:** `Func`, `Func` resolver, `Func`, -`Func`, `Func`, `Action` log, and an injectable pacing delay -(default `Task.Delay`; tests inject `Task.Yield`). The pump is free of WPF and of the capture managers. +`FramePump` and the whole path from frame producer to encoder (pacing, pool/paste caches, slice +history, teardown, health stats) live in +**[`docs/ai/frame-pipeline.md`](docs/ai/frame-pipeline.md)**. -- **`StartAsync` never throws** — the VM fires-and-forgets it from the sync command handler; failures log - + surface via the `Failed` event. `BuildEncoderOptions` now always returns options (TASK 18) with - `StreamEnabled`/`RecordEnabled`/`RecordPath` baked from the pills + `_activeRecordPath`; the pump skips the - encoder only when `options == null`, i.e. "no output configured". `MainViewModel._rtmpUrlProvider` is that - seam — a `Func` that now yields the reusable stream's ingest URL (TASK 9, shipped 2026-08-16): - loaded from the `LayoutStore` cache at startup and set fresh by `PrepareAndStartLiveAsync` before the pump - starts. **The pump reads the URL once at startup**, which is why go-live ensures the stream BEFORE - `StartAsync`. -- **Deadline pacing + row-blit render (take-3 starvation fix, 2026-09-03):** take 3 recorded 30s into a - 2.1s file and the 5s stats line named the cause in one number — `17/300 frames per 5s, avg render - 258.1ms, avg submit 1.5ms`. Two defects, both fixed: (1) the pump slept the FULL interval after each - render, so period = render + interval — OBS's `video_thread` (libobs/media-io/video-io.c) pattern - replaces it: absolute `nextTick += intervalTicks` deadline, sleep only the remainder, and on overrun - skip the wait. (NOTE, slice 9: the original "skip the missed ticks (rebase)" half of this fix was - wrong — see the slice 9 correction below.) (2) the compositor did - per-pixel float sampling + `Math.Round` blending over all 2.07M master pixels (backdrop) and scanned - the whole destination per overlay (a 64px social-bar strip cost 2M iterations). `SceneCompositor` now - follows the libyuv pattern (BSD-3, chromium.googlesource.com/libyuv/libyuv — cited per the - derivative-work rule): 1:1 aligned blits take a row-walk fast path (bilinear at scale 1 + integer - offset is the identity) with per-pixel alpha branch and integer fixed-point blend; `BlitOverlay` - clips to the intersection rect; a full-cover live backdrop skips the opaque-black pre-fill. The - general per-pixel path handled scaled/round/mirrored elements. Test: - `Pump_Paces_To_The_Deadline_Compensating_Render_Cost` records the requested wait - via the pacing seam (a seam fake must genuinely await — a synchronous completed task runs the whole - pump loop on `StartAsync`'s continuation and hangs the run; MyMistakes). -- **Slice 2 of the render fix (2026-09-04, take 4):** the pacing held (file no longer truncated at - loop level) but render stayed at 58.9ms — the per-pixel row walk was still 2M managed iterations and - every tick allocated a fresh 8.3MB master buffer. Three changes: (1) **`VideoFrame.IsOpaque`** — a - producer-contract flag (only the screen-capture and webcam paths set it; DWM/MediaCapture fill alpha - 255 by contract); a full-canvas, aligned, opacity-1 blit of an opaque frame is now ONE - `Buffer.BlockCopy` (~1.5ms) instead of the loop, and the black pre-fill is skipped when the backdrop - covers. (2) **integer fixed-point bilinear** in the general `BlitContent` path (row-hoisted invariants, - no divisions, no `Math.Round`) — within ±1 of the float reference, inside the ±2 test tolerance. - (3) **scratch pool in the pump** — `AcquireScratch`/`ReleaseScratch` recycle the master buffer - (max 4, keyed by length, owned-by-reference so bake-cache/social-bar/static-art arrays can never be - captured); release happens strictly AFTER `SubmitFrameAsync` returns (the write to stdin copies), - and the free-list Contains guard makes the transition Cut path safe (`BlendFrame` returns the - to-frame itself, aliasing the scratch). The per-tick `fromScene` render in the transition branch was - **dead weight** (BlendFrame uses `TransitionService.FromFrame` captured at `Start`, never the pump's) — - removed, and the pump's `fromSceneProvider` seam + `MainViewModel` call site went with it. Tests: - `Pump_Pools_ScratchBuffers_Across_Frames_Without_Stale_Pixels` (the ONE: alternating backdrop colors - pin every frame's content, repeated backing-array identity proves the pool recycles), - `Composite_OpaqueFullCover_Backdrop_CopiesEveryPixel_Into_Scratch` (memcpy branch + scratch - sentinel). `FakeEncoder.SubmitFrameAsync` now snapshots bytes like the real stdin write — holding the - reference would race legitimate recycling. Take 5 must show `avg render ≤ ~10ms, ≈300/300`; the - vertical tier's final 1080×1920 `BilinearScale` still allocates fresh per frame (same GC lesson when - someone streams vertical — recorded as a follow-up, not silently "done"). -- **Slice 3 — chat raster-on-change (2026-09-04, take 5):** take 5 measured `138/300, avg render - 25.5ms` — the per-tick blits were cheap now, but `ResolveOutputFrame` → `RenderChatBox` ran a FULL - WPF raster (`ChatBoxRenderer`: FormattedText + `RenderTargetBitmap` + CopyPixels + channel swap) - EVERY tick whenever the message buffer was non-empty — and the buffer survives between sessions, - so even a signed-out record-only take paid it. Established answer, same as OBS text sources: - **raster on change, blit the cache every tick.** `ChatOverlayLayer` now keeps a content version - (`Messages.CollectionChanged` → `_contentVersion++`) plus a config key (box size + all Chat* - props); `RenderFrame` returns the cached `VideoFrame` by identity until either changes (the - compositor only ever reads a cached frame). Test: `ChatOverlayLayerCacheTests` (RealApp, real - renderer — `Same()` for unchanged inputs, `NotSame()` on message/config change, null on empty). - Accepted cost pending take 6: one slow tick (~15-25ms) per arriving message; if chat-burst frame - loss shows up, the next slice moves the re-render off-tick (debounced, dispatcher-side). -- **Slice 5 — paste cache for non-opaque layers (2026-09-04, take 7/8 data):** the render/resolve - split (added in the stamp commit) finally named the last thing honestly: `resolve ≈ 0` but - `render 26-27ms` (124-135/300, ≈2.2x) — the chat fix HAD worked; the remaining cost was the - compositor re-rasterizing EVERY layer per tick even when its pixels never changed (this creator's - Live scene: chat 159k + web widget 271k + image 95k + webcam 156k px ≈ 680k samples @ ~38ns each). - OBS's actual shape: sources cache their surface, the compositor pastes. `SceneCompositor` now - routes non-opaque layers through `BlitCachedLayer`: a layer rasterizes ONCE into an element-space, - TRANSPARENT-based frame keyed by (source-array identity, source W×H, ceil'd dst rect, round, - mirror), then every later tick PASTES it (integer position, row alpha-blend, opacity applied at - paste). **Correction (2026-09-12, the web-widget "black box"):** the raster builds on a - TRANSPARENT base, but the sampler's PARTIAL-alpha branch used the opaque-dst blend — it - premultiplied the color into RGB and forced `alpha=255`. A translucent widget pixel then pasted - as opaque darkened ink (the box in recordings) while the raw-bitmap preview stayed correct. - `BlitContentRaw` now takes `transparentDst` (the raster call passes `true` and writes straight - color + straight alpha; the paste rows do the source-over). Master paths are bit-identical. - Guard: `PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk`. Producers hand out fresh immutable arrays, so array-identity keys can never serve stale - content; dict bounded at 48, cleared wholesale on overflow. Only changing content (webcam device - frames, web capture ticks, chat messages) resamples; the opaque backdrop keeps its memcpy path. - Position/opacity drags are now near-free (no resample — paste params, not cache keys). - ONE integration test: `PasteCache_RepeatRender_IsByteIdentical_And_ContentChangePropagates` - (raster-vs-paste byte equality incl. round-clip margins + new-array propagation); the whole - existing pixel suite guards sampler semantics. Take 9 must show `≈300/300, avg render ≤ ~8ms`; - if it lands there, the recording saga closes. Follow-ups unchanged: vertical-tier scale alloc, - debounced chat re-render under message bursts. -- **Slice 6 — the sleep quantum WAS the ceiling (take 9, 2026-09-04):** period measured ~37ms while - work (render+submit) was ~25 — the missing ~12ms is `Task.Delay` rounding EVERY request up to the - Windows clock tick (documented ~15.6ms default; learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.task.delay). - A frame finishing 3ms early requests a 3ms wait and sleeps a full 15.6 — capping the producer at - ~27fps NO MATTER how fast the compositor ran. This is why takes 7→9 showed zero playback change - despite real render wins: the sleep floor dominated everything above it. Established fix (game-loop - canon — stackoverflow.com/questions/5441464; timeBeginPeriod — learn.microsoft.com/en-us/windows/win32/api/timeapi/nf-timeapi-timebeginperiod): - `timeBeginPeriod(1)` for the pump's lifetime (paired `timeEndPeriod` in the finally), sleep only the - BULK of the remainder (request minus 2ms), `Thread.SpinWait` the last ~2ms across the deadline; - blown deadlines still rebase. (SUPERSEDED in part: slice 9 deleted the rebase — deadlines only - advance; slice 10 replaced the burst re-write with one fresh composite per iteration.) Stats gained - `avg wait Xms` so render+submit+wait must ≈ period — - the accounting is closed, no stage can hide again. Same slice: the webcam's `IsOpaque` paste-cache - bypass was removed (it re-sampled ~156k px every tick even between identical device frames; the - cached paste beats the sampler on hits and costs the same on misses). Take 10 verdict: `≈300/300` - honest fps — if short, the wait/render split names the remaining term with no ambiguity left. -- **Slice 7 — the loop was ON the UI thread the whole time (take-8 numbers, 2026-09-04):** with - the wait finally measured, take 8 (59a02a5b) showed the impossible pair — `render 22ms, wait 10ms` - against a 16.7ms deadline: a blown deadline rebases with NO wait. The wait was the producer - **queued behind the live preview on the dispatcher**: `PumpAsync` starts from a UI command handler - and its `await` continuations inherit the UI `SynchronizationContext`, so the "WPF-free" compositor - actually rendered on the UI thread and ran only when WPF let it. OBS runs its media loops on - dedicated threads for exactly this reason. Fix: `_pumpTask = Task.Run(() => PumpAsync(...))` - (no sync context inside → continuations stay on the pool). Side effects handled, not ignored: - `StaticPixelCache.Get` now locks (pool miss-decodes race UI callers); `ChatOverlayLayer.RenderFrame` - checks its cache off-thread but MARSHALS the rare raster miss to the dispatcher (DrawingVisual/ - RenderTargetBitmap are UI-thread objects) and re-validates there; pump events already marshalled. - Also: `GCLatencyMode.SustainedLowLatency` for the pump's life, stats gained `worst render Xms` - (spike visibility — bimodal averages hid them), webcam now routes through the paste cache (bypass - re-sampled 156k px even between identical device frames). Tests: `Pump_Produces_OffTheStartingContext` - (the ONE — inline-pumping SyncContext proves continuations never return to the starting thread) + - 70/70. Take 9 verdict: wait ≈ true remainder (period → 16.7, n → ~300); `worst render` names any - remaining raster-miss spikes; render+submit+wait still == period — the accounting holds. -- **Slice 8 — capture buffer ring + paste-cache Epoch (take 11, 2026-09-04):** take 11 validated - the architecture — typical frames now land `work ~10ms + wait ~6.8ms = 16.7`, exactly the deadline - (212/300). The whole remaining gap is PERIODIC 35-65ms worst-render spikes that worsened across - the take (189→147 frames/5s) — the signature of gen2 GC pauses, now provable via `gen2 +N` in - every stats window. Biggest churn was structural: `ScreenCaptureFrameSource` minted a fresh -~8.3MB `byte[]` per DWM frame (~500MB/s LOH). It now rotates a shared-frame ring (4-deep at slice 8, - deepened to 8-deep after the "flash" finding — consumer holds must never outlive depth × source - period), size-matched per slot, with reused downscale row scratch. Recycled - arrays would poison the paste cache (it keys on array IDENTITY), so `VideoFrame.Epoch` — - monotonic per producer frame, 0 for fresh-array producers — joins the key. Regression test - `PasteCache_RecycledArrayWithNewEpoch_ReRasterizes_NotStaleHits` fails on the old key by - construction. The camera producer carried the same fresh-array churn (110-220MB/s at 30-60fps): - it now rotates its own 8-deep ring + Epoch (same identity rule), and the WebView2 capture - reuses a canvas scratch + 8-deep output ring + a reused WriteableBitmap instead of minting two - fresh arrays + a new bitmap per 10Hz tick. Next suspect if gen2 stays hot: the WPF preview load - itself driving gen2 — recorded as follow-up, untouched this slice. -- **Slice 9 — the "rebase" WAS the recording time-lapse (2026-09-10, the recording playback-timing - take):** slice 6's "blown deadlines still rebase" was the recording-timing bug all along. In a real - take the pump emitted `215-219/300 per 5s` (~43fps) while `render+wait == period` still looked closed — - the rebase ERASED every missed slot (deadline = wall-now again), so the missed ticks never showed in - the stats. rawvideo has no per-frame timestamps: ffmpeg muxes by frame count at `-framerate 60`, so a - 43fps reality was authored into a 60fps container and **every recording played ~1.4x fast** (takes - confirmed with a WSL ticker visible in the recording: file duration 11.44s vs ~15.5s wall). Two - defects, both fixed the OBS way (`libobs/media-io/video-io.c` — the video thread NEVER resets its - deadline; every interval tick outputs ONE frame, and a late render means repeated content — judder — - never a skipped timestamp): - (1) **count-based CFR emission:** `while (GetTimestamp() >= nextTick) { SubmitFrame(frame); nextTick += intervalTicks; statFrames++; }` - submits exactly one frame per crossed slot, re-writing the current composite when the render overruns - (duplicated footage = correct duration, not a time-lapse), and `nextTick` is NEVER reset to wall-now; - (2) **`-re` removed from `FfmpegArgs`** — it was a second, fighting pacer on the rawvideo demux - (its "Resumed reading … after a lag" grew 0.79s→4.82s across the take, leaving the pump behind its own - honest-stats count). One fix covers the game background AND the webcam — both flow through this one - pump. Muxed duration is now frame-count ÷ fps == wall time by construction. Test: the pump suite stays - green unchanged — the pacing test's `` (cap 120 ≈ - 2s at 60fps) owned by a dedicated drain task with the stdin write + flush; the caller NEVER blocks on - the pipe. Pixels are copied into an `ArrayPool` buffer before enqueue (the write now happens later on - the drain thread, so the pump's scratch reusable the moment submit returns — same contract as before). - (2) **drop-on-overflow (creator's choice: freshness over coverage):** when the queue is full the NEWEST - frame is dropped and counted (`IFfmpegEncoder.DroppedFrames`, `Interlocked`); the session never freezes - or smears — it drops. Stop flushes the whole queue then EOF (`Channel.TryComplete` → drain writes - leftovers → closes stdin → ffmpeg finalizes+exits), so no accepted frame is ever lost at stop. - (3) **pump loops ONE submit per iteration** (replacing slice 9's burst): every due slot gets a FRESH - composite — no catch-up slot ever repeats frozen content; missed slots vanish as a count-based gap. - The deadline counter is still never reset to wall-now (slice 9 ruling). - (4) **burned-in frame counter (the new judge):** `_outputIndex` is burned into a 6-digit dot-matrix - strip (white box + black 5×7 glyphs, bottom-right corner) of EVERY composite before submit. The WSL - ticker is demoted because its own timers smear under Windows host load; decoding the recording and - reading the strip is clock-independent: the number advances +1 per frame and jumps by exactly the - counted drops (queue overflow OR skipped catch-up slots). Stats gained `worst submit Xms`, `dropped N`, - `stalls K`; a stall logger names any iteration > 2× interval with its render/submit split — with the - queue, submit is ~1ms, so a stall means render/resolve. The strip sits above the social bar (bar is - composited, then burned over) — a debug judge, tiny at 1080p. Test: - `Backpressure_QueueOverflow_DropsFrames_AndNeverBlocks` (the ONE: a 40ms-per-frame sink fake makes - every enqueue fill the queue; submit must return instantly, drops are counted, stop flushes exactly - submitted − dropped bytes). Full suite 289/290 (the pre-existing compositor pixel failure unchanged). - Audio untouched (follow-up). WSL ticker-under-load reliability check still to run (informational — the - burned counter is the judge). -- **Slice 11 — web-layer animation ran at ~1/6 speed; capture cadence 10Hz→~30Hz + de-throttle - (2026-09-10, the "widget animates too slow" take):** the recording is 60fps but the WebView2 - capture loop was a blind 100ms `DispatcherTimer` = 10Hz — a 60fps-designed widget was sampled 6× - under its native rate (repeated-footage slow-mo). Two stacked throttles: - > Superseded by **Slice 14** (below): `CaptureScheduler` and the cadence hooks were the polling - > design; frame-driven composition capture replaced them — nothing here is current architecture. - (1) **the 10Hz cap** (unconditional, code-proven) and - (2) **Chromium hidden-page throttling** (`requestAnimationFrame` parked, JS timers clamped to ~1s - per WebView2Feedback#1172/#3070 + Chrome-88 timer throttling) whenever the app window is unfocused - or covered — the page is off-screen at (-5000,-5000), so it is a hidden page the moment the host - window loses occlusion. - Fixed: - (1) **`CaptureScheduler` (new, `Services/CaptureScheduler.cs`)** replaces the per-session - `DispatcherTimer`: a dispatcher timer that DROPS ticks while a capture is in flight (latest-wins, - never queues) — the in-flight drop is what made raising the cadence safe (concurrent full-HD PNG - `CapturePreviewAsync` calls would stack ~10-30ms encodes and publish stale-after-fresh). Interval is - owner-configurable: recording 33ms (~30Hz), idle 200ms. Unit-tested without a WebView2 runtime - (the ONE test: `CaptureScheduler_Drops_Ticks_While_Capture_InFlight_And_Resumes` — overlapping - ticks dropped, capture resumes when idle, driven by a TCS so it is deterministic). - (2) **de-throttle browser args**: the shared `CoreWebView2Environment` is created with - `--disable-backgrounding-occluded-windows --disable-renderer-backgrounding - --disable-features=CalculateNativeWinOcclusion` (the Electron/Streamlabs-class embedder answer for - occluded-window animation throttling) BEFORE `EnsureCoreWebView2Async`. - (3) **cadence hook**: `MainViewModel` sets `SetCaptureInterval(33)` on record/stream start and - `(200)` on stop (`MainViewModel.Streaming.Operations.cs`). - (4) **capture-cost telemetry**: first 30 captures per session log elapsed ms to startup.log — - PNG-encode+decode cost decides whether ~30Hz stays or drops to ~20Hz; the FramePump drops frames - (never time-lapses, slice 10) if UI-thread GC churn starves it, so the cost is observable. -Full suite 290/291 passing, the sole failure the pre-existing compositor pixel test. The web-overlay - transparency verification (the first-capture diagnostic bound to about:blank) and the audio-silence - item remain open push-gate items — both untouched by this slice. -- **Slice 12 (audio diagnostics, 2026-09-10):** the recorded audio is full-length silent AAC - (−91dB, 1124 frames / 23.95s in the latest take) — the named-pipe delivered ~9.1MB of zeros to - ffmpeg for the entire recording, so the loop ran and connected, but both WASAPI capture sources - delivered nothing. Sources started fine (`Audio: using system default mic...` logged), no failure - callbacks fired, and the mixer's existing integration test (`Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe`) - proves the loop→pipe math is sound. The fault is capture-side: either the default render endpoint - carried nothing (audio played on a non-default device — common), or both endpoints were held - exclusive, or genuinely nothing played. Added permanent per-5s live-loop telemetry to startup.log - (`Audio live: pipe connected=, dropped writes=, micLevel=, loopLevel=, drained, peakMix`) and - `NamedPipeAudioWriter.DroppedWrites` — names the exact stage on the next take without new - code. `AudioMixer.FillAndMix` now returns `(MicRms, MicDrained, LoopDrained)` for the telemetry - accumulation. Full suite 290/291, same pre-existing sole failure. -- **Slice 13 (web transparency diagnostic re-point, 2026-09-10):** the "?" in MyMistakes' - "RESOLVED (? VERIFY)" is being closed. The one-shot widget dump + alpha stats bound to the - FIRST capture ever = the initial about:blank document — a blind instrument. NavigationCompleted - for the REAL widget URL now arms `WidgetDumpRemaining = 5`; those 5 post-paint captures dump - `%TEMP%\ytLive-web--w1..5.png` + the shared `AlphaStats(...)` line (alpha[min/max/mean/zero%]) - + FindContentBounds result to startup.log. That line alone decides the fix branch: zero% ⇒ the - pre-parse injection did not hold for this widget (opaque capture → chroma-key / stronger DOM - injection); large zero% + tight contentBounds ⇒ the capture IS transparent and the recording's - black block lives downstream (compositor blend/underlay). `AlphaStats` extracted as the shared - sampler. No new tests (WebView2 runtime not instantiable in tests; the re-point is log-only). -- **Slice 14 — web frames are now COMPOSITION-CAPTURED; the PNG poll + `CaptureScheduler` are gone - (2026-09-14, the "still not 60fps" take):** slice 11's ~30Hz was still a `CapturePreviewAsync` PNG - poll — every 1920×1080 full-HD encode+decode cost 35–165ms, so the session budget capped real - cadence at ~20Hz for a 60fps-designed widget. Root fix = frame-driven capture, not faster polling: - the WebView2 renderer now feeds `Windows.Graphics.Capture` directly through a - `CoreWebView2CompositionController` (the mechanism `WebView2CompositionControl` and Flutter's - `webview_windows` use — see `graphics_context.cc` `CreateGraphicsCaptureItemFromVisual`, which - captures the root `surface_` visual). - - **Pipeline (per session):** `CreateCoreWebView2CompositionControllerAsync(WindowHandle)` — the - parent HWND must exist, so `MainWindow.InitWebView2()` moved from the ctor to **Loaded** (was - `InitWebView2(WebViewHostPanel)`; the hidden XAML `WebViewHostPanel` overlay is deleted). - `controller.RootVisualTarget` = a child `ContainerVisual` (`RelativeSizeAdjustment = 1,1`) under a - root `ContainerVisual` (1920×1080, `IsVisible=true`) built on ONE `Windows.UI.Composition.Compositor` - (created once on the UI thread); `Bounds = 0,0,1920,1080`, `BoundsMode=UseRawPixels`, - `ShouldDetectMonitorScaleChanges=false`, `RasterizationScale=1.0`, `IsVisible=true`, - `DefaultBackgroundColor=Transparent`; then `GraphicsCaptureItem.CreateFromVisual(root)`. - Frames pull from a free-threaded `Direct3D11CaptureFramePool` (2 buffers) → - `SoftwareBitmap.CreateCopyFromSurfaceAsync` (BGRA, `BitmapAlphaMode.Straight` — the compositor - blends STRAIGHT alpha, so premultiplied readback would wreck corner anti-aliasing) → per-frame - `FindContentBounds` alpha-bbox on the capture worker → `VideoFrame.CropBounds` stamped → ring up to - 2 fresh + Epoch (reuse; GC lesson slice 8) → dispatcher-coalesced copy (Render priority) into ONE - crop-sized shared `WriteableBitmap`; `PreviewBitmapChanged` raised only on bitmap (re)creation. - Also unpumped: the OBS-side de-throttle flags (slice 11) stay — they fix the hidden-page JS - throttling; the capture path itself no longer polls. - - **Implemented in:** new `Services/WebCaptureFrameSource.cs` (owns item+pool+session — mirrors - `ScreenCaptureFrameSource`, which servers as the copy template) + reworked `Services/WebView2Manager.cs` - (owns controllers, the visual tree, nav/script wiring, preview; internal seam ctor - `(Dispatcher, Func?)` so the tests instantiate zero WinRT; public - ctor `(Dispatcher)`). - - **CoreMessaging DQ recipe (record-once):** the 19041 CsWinRT projection has NO - `DispatcherQueueController.CreateOnCurrentThread()` (CS0117 — only `CreateOnDedicatedThread` + - `FromAbi(IntPtr)`). P/Invoke `coreMessaging.dll!CreateDispatcherQueueController` with struct - `DispatcherQueueOptions { DwSize, ThreadType = 2 (DQTYPE_THREAD_CURRENT), ApartmentType = 2 (DQTAT_COM_STA) }`, - wrap via `DispatcherQueueController.FromAbi(ptr)` (mirror of `CaptureInterop`), only then - `new Compositor()` — all once on the WPF UI thread (the app dispatches Render there). - - **Compile notes (each cost a build cycle):** `Compositor` collides with the repo's OWN - `ytLive.Services.Compositor` namespace → fully-qualify `Windows.UI.Composition.Compositor`; - `CoreWebView2CompositionController` exposes **`Close()`**, not `Dispose()`; `Color` is ambiguous - (`System.Drawing` vs `System.Windows.Media`) → `System.Drawing.Color.Transparent`. - - **Dropped:** `Services/CaptureScheduler.cs` (DELETED — no cadence to schedule), all three - `SetCaptureInterval` hooks in `MainViewModel.Streaming.Operations.cs`, the XAML `WebViewHostPanel`. - `TransparentBackgroundScript` const is unchanged (a test pins it). - - **Test model (Good Dog):** reworked `ytLive.Tests/WebView2ManagerTests.cs` — dropped the 4 - control-size tests + the `CaptureScheduler_Drops…` test; `FindContentBounds` tests moved to - `WebCaptureFrameSource.FindContentBounds`; the ONE integration test - `Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds` drives the internal seam with - `FakeWebSource : IScreenCaptureSource` + a real background-STA `DispatcherPump` (copied from - `ScreenCaptureManagerTests`): register → one crop-sized preview bitmap published → `CropBounds` + - `IsOpaque=false` on `GetLatestFrame` → back-to-back pumps coalesce to latest. Byte assertion compares - the DENSE 2×2 crop (`CropBytes` helper — slices source rows with stride gaps, a contiguous range - spans rows wrongly). - - Full suite **293/293 green, 0 warnings**. Audio untouched. **NOT YET VERIFIED ON DEVICE** — the - composition path needs a real 60fps-widget run (open item; see HANDOFF). Web work commits stay - LOCAL (no push) until the user greenlights. -- **Slice 15 — one frame per deadline SLOT: duplicate-on-lag, never skip (2026-09-14, device takes - ty-1723/1726):** slice 10's freshness choice — a render overrun VANISHES the missed slots from the - file — authored ACCELERATED playback: ~35ms render cost vs the 16.6ms slot, one fresh frame per - 35ms (log wall-1726: `FramePump stall: iteration 33ms (> 2× the 17ms interval): worst render - 33-46ms`), and a 60fps container muxed on arrival → 1723: 697 frames = 11.62s video vs 11.84s - audio; 1726: 163 frames = 2.72s vs 2.93s. The video also ended 0.21–0.24s before the audio ("audio - speeds up, cuts off at the end"). OBS's answer is duplicate-on-lag: `libobs/media-io/video-io.c` - emits one frame per tick and a lagger DUPLICATES the newest frame, counted as "lagged frames due - to rendering lag/stalls" (obs-output.c) — a wall-time hole never exists (docs.obsproject.com/ - backend-design: "If the video frame queue is full, it will duplicate the last frame"). The pump's - submit is now a bounded catch-up: `while (now >= nextTick) { submit; nextTick += intervalTicks; }` - over a `deadlineNow` captured once per iteration — the LATEST composite, fresh on the first missed - slot, repeated (OBS's duplication) for the rest. Duration == wall under any render load, at the - cost of a short judder during a stall. The burst is safe because `Channel.TryWrite` - never blocks (slice 10's queue — the take-9 smear was the BLOCKING pipe-write re-copying a stale buffer - during a long freeze; here each emit is nanoseconds). The burned `_outputIndex` (slice 10 judge) - moved INSIDE the submit loop so every emitted slot carries its own +1 (and the old unconditional - pre-gate bump no longer gaps the sequence on non-submitting fast-render iterations). **Good Dog - test** `Pump_Overrun_Renders_EmitsEverySlot_NotSkipped`: 60fps, resolver sleeps 35ms, asserts ≥0.65 - of the wall slots are emitted (the skip-pump wrote ~1/35ms ≈ 200 in 7s; the slot pump ~415). Full - suite **294/294 green, 0 warnings**. The +0.6s audio-late clap reading on 1726 was confounded by - the 1.1x acceleration — re-measure on device; if a real residual remains it is the audio pipeline. - No push (web/A/V work stays local). -- **Slice 16 — the desktop capture conversion was the bottleneck: fast downscale + - cadence throttle + reuse-distance ring (2026-09-14, device take ty-1742):** slice 15 - fixed pacing but the 1742 desktop layer was still jerky/frozen with a horizontal - tear. Decoded to raw frames and audited: the **desktop band was frozen 21s of 23.35s - (90%)** — ~6.1 content updates/s, freeze runs up to 2.28-2.78s, dup-run max 90 - frames (1.5s). The render stat (`worst render 33-36ms`) was real but MOOT: the - capture CONVERSION was the wall. The monitor delivers at the **240Hz DWM cadence** - (~4.2ms); with one-in-flight conversions and each 2560×1440→1920×1080 - `DownscaleBgra` at ~150ms under load, `LatestFrame` updated a handful of times/s — - the desktop feed inside a 60fps file read ~90% frozen. (Webcam + audio were their - own paths — fine, matching the report.) Three changes, all in - `Services/ScreenCaptureFrameSource.cs` (+ `Services/FrameRingBuffer.cs`): - (1) `DownscaleBgra` is now **integer 8.8 fixed-point, "shift only at the end"** — - the exact two-stage math of `SceneCompositor.Bilinear` (MyMistakes take-4 rule), - dropping double-per-pixel to a row-walk of integer ops (the capture downscale had - stayed the naive float twin of the 258ms disaster); (2) a **10ms conversion floor** - (`MinConvertInterval`) so the 240Hz tail stops queuing ~150ms of serialized - conversion per second — the open edge sits just above the ~60/s the pump can use; - (3) the ring is a **reuse-distance pool** (`FrameRingBuffer`, depth 8, redLine 4): - a buffer is only rewritten ≥4 rents after its last hand-out else a fresh allocation, - replacing the blind round-robin — since `ScreenCaptureManager` keeps - `session.LatestFrame` across conversions and the dispatcher preview copy lags, - "who released the buffer" needs a consumer API that doesn't exist; a reuse - DISTANCE needs no cooperation (the 1742 new-top/old-bottom tear is that - read-under-write, structural now). Telemetry added: a startup.log line every 2s - (`frames/s, conv avg/max ms, skip busy/cadence, ring allocs`) so the device take - can be judged numerically. **Good Dog test** - `Ring_NoLap_ReusesOnlyAfterRedLineRents` (depth 3/redLine 4 exercises the red-line - skip; the 8/4 config settles at 8 buffers and never grows). Full suite **295/295 - green, 0 warnings**. C4 (compositor Epoch-cached downscale / blit-on-change) was - DEFERRED: the slice-15 approved plan assumed a native-res relocation; the - measurement recast it — relocating a ~30ms float downscale to the render thread - just moves the same cost into the slot budget. Re-measure on device; if render - still >16.6ms slots after capture feeds ≤60 real updates/s, add C4. No push. -- **Slice 17 — the OS readback was the real wall: overlapping conversions + monotonic - publish + deeper pool (2026-09-14, device take ty-1824):** slice 16's downscale fix - landed but the desktop was still ~90% frozen on the 1824 take (band 6.8/s updates, max - freeze 4.85s). The new 2s telemetry was decisive: `conv avg 46-50ms max ~61ms` with - `skip busy 37-83` — the **GPU→CPU readback (`CreateCopyFromSurfaceAsync`), not - `DownscaleBgra`**, is the ~47ms wall (240Hz HDR compositing + encoder + 2 capture - devices share the GPU); at one-in-flight that caps the desktop feed at ~17-20 - updates/s, which is the file's whole-frame ~7 content-moments/s. Two facts reshaped - the fix: **(a)** shrinking the pool size does NOT scale the desktop (Microsoft docs: - "If content is larger than the frame, the contents are **clipped**") — readback stays - at native 2560×1440; **(b)** delivery is healthy (60-100 arrivals/s), so the lever is - conversion throughput, not the pool size. Changes in `Services/ScreenCaptureFrameSource.cs`: - conversions overlap up to **MaxConcurrentConversions = 3** (pool deepened to 5 buffers - so in-flight frames fit), each completion publishes ONLY if its Epoch is strictly - newer than the last published (`MonotonicGate` — a slow older completion must never - overwrite a newer LatestFrame), Epoch increments via `Interlocked`, and the - DownscaleBgra row-scratch became per-conversion locals (concurrent callers). The - render side (33-42ms → ~30 unique composites/s) is the NEXT cap after capture speeds - up — that's the C4 slice, queued right after this re-measure. **Good Dog test** - `PublishGate_TryPublish_OnlyStrictlyNewerWins`. Full suite **296/296 green, 0 - warnings**. NOT YET DEVICE-VERIFIED; target: telemetry frames/s jumps ≥ ~30-40 and the - band audit drops below ~50% frozen. No push. -- **Slice 18 — C4: the FramePump blit-on-change composite cache (2026-09-15, take ty-1841 - verdict):** the 1841 take proved capture fixed (band ~20 fresh updates/s, no tears, pacing - clean) but logged a FramePump **stall on EVERY iteration** (`totalMs 21-44`, - `render=full-render split=0 elements=6 dynamic=4`, worst render 166ms startup spike) → - ~22-28 composites/s hard-caps the desktop inside the 60fps file. The SceneGraph split - CANNOT fix this wired: `GetSplitPoint` returns **0** because the live-capture backdrop is - element 0 and must stay dynamic — a baked capture goes stale — so every tick runs the full - re-composite even when no input changed (the compositor's per-layer paste cache already makes - identical frames identical per-element; the waste is re-COMPOSITING the whole 8.3MB frame). - Same shape as OBS (sources cache their surface, the compositor renders on update only — - the pattern this file has cited since the 2026-09-04 paste-cache slice) at the FRAME level. - `FramePump.RenderFull` now wraps both full-render call sites (no-SceneGraph + split=0 - fallback): it computes `BuildFullRenderSignature` (a deterministic hash of the crop/output - dims, the social bar identity + top, and — per element, MIRRORING the compositor's own - resolution — the element ref, layout/visual bits, and the frame each element would resolve - through the SAME resolver seam, using buffer-array identity + Epoch + CropBounds; a changed - frame forces a re-composite, an unchanged one never serves stale bytes) and, when it matches - the last composite, reuses the cache with ONE `Buffer.BlockCopy` (~3ms) instead. The cache - buffer is a **separate long-lived array** — never the scratch pool — because the caller burns - the frame counter and recycles the scratch AFTER `RenderScene` returns; the cache is written - from the rendered scratch BEFORE returning (pre-burn, pre-recycle). Engagement is gated on - the 1:1 config (`SourceRect == Output`, the only deployed tier); the vertical tier's cached - fresh-scale-buffer follow-up stays exactly where it was. Telemetry: internal `CacheHits`/ - `CacheRenders` counters surface as `cache {renders}R/{hits}H` on the 5s stats line, and the - ticks' key is an internal `OutputIndex` (burned counter, stable across one tick's resolver - passes). **Good Dog test** - `FullRenderCache_StaticInputs_RenderOnce_Then_Reuse_UntilInputChanges`: static scene → - exactly ONE composite (renders==1, hits>0, byte-identical content above the burn strip), - then a new frame (new array + monotonic Epoch) invalidates and propagates. Existing pool test - now passes a STABLE scene (production hands `StagedScene`; a fresh scene per tick churns the - element refs inside the signature and hides the cache — the tests that care about cache - behavior must mirror production) and keys its red/blue alternation on `OutputIndex`/frame - count so the tick's two resolver passes (signature + render) don't double-advance a call-count - flip. Full suite **297/297 green, 0 warnings**; verify.sh scope-locked to FramePump.cs + - FramePumpTests.cs + docs. Target next take: `cache` hits dominate on TV holds, `avg render` - drops toward the BlockCopy, stalls vanish, then the band audit + clap re-measure decide push. - No push. -- **Stop ordering matters:** `StopAsync` stops the encoder — since slice 10 it FLUSHES the pending - queue (`Channel.TryComplete` → drain writes the leftovers, closes stdin → EOF → ffmpeg finalizes+exits; - an accepted frame is never lost) — **before** awaiting the loop. The old reverse-order deadlock was - a submit stuck on pipe backpressure; the drain thread owns that wait now, and stopping the encoder - first keeps the pump's own (non-blocking) submits from racing the completed queue as a spurious drop. - `ProcessFailed` self-stops the pump. `Failed` while live flips - `StreamStatus.Error` (minimal). -- **Health stats (ship step 6, shipped 2026-08-13):** `HealthUpdated` is bound to the bottom bar — - `MainViewModel.OnFramePumpHealthUpdated` marshals to the UI thread (the encoder's stderr loop raises on - a background thread) and copies into `CurrentHealth` (the bottom bar's existing `CurrentHealth.*` - bindings). `ResetHealth(status)` zeroes dropped/duration on go-live and on End so stats never linger - from a previous session; bitrate/FPS stay on the tier's targets (`ApplyStreamQuality`). The bar shows - real encoder values since TASK 9 (2026-08-16) wired the reusable stream URL into `_rtmpUrlProvider`. -- **`MainViewModel` owns the resolver** (`ResolveOutputFrame`): `WebcamSceneConfig` → - `CameraManager.GetLatestFrame(DeviceKeyForWebcam(WebcamId))` — **device-keyed** (configs carry the - identity GUID; sessions key on the Windows device id — the old direct-GUID lookup returned null - forever and the webcam was structurally absent from output; take-2 fix 2026-09-01, - `WebcamOutputKeyTests`), `Source { IsLiveCapture, CaptureKey }` → - `ScreenCaptureManager.GetLatestFrame(CaptureKey)` (the new accessor mirroring `CameraManager`), image/ - background → `StaticPixelCache.Get(AssetId)`. `BuildCompositorOptions` rounds the VM's `OutputRect*` - doubles to ints — the vertical 607.5 half-pixel crop rounds to a perfectly-centered **608** (`Math.Round`, - ToEven); `BuildEncoderOptions` fills W×H/FPS/bitrate from the tier once the URL provider yields one. -- **Media source decoder (TASK 21, Increment B — 2026-08-31):** `MediaVideoSource` (`Services/MediaVideoSource.cs`) - decodes a local file to `VideoFrame`s by spawning ffmpeg with - `-f rawvideo -pix_fmt bgra -vf scale=WxH -an` and draining the raw BGRA stdout pipe through the pure - `RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raising `FrameAvailable` per frame. Implements - `IMediaFrameSource` (`Services/IMediaFrameSource.cs`: `Key`, `FrameAvailable`, `Completed`, `StartAsync`/`StopAsync`). - The subprocess sits behind `IDecodeProcess`/`FfmpegDecodeProcess` (binary-stdout mirror of the encoder's - `IEncoderProcess`), and the path comes from `IFfmpegLocator` — codec-agnostic, uses the ffmpeg already shipped, - testable with a fake process. Decode sessions are owned app-wide by `MediaVideoSourceManager` - (`Services/MediaVideoSourceManager.cs`): refcounted by `MediaPath`, `Func` factory - seam (mirror of `ScreenCaptureManager`), `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`, - coalescing each file's frames onto the UI dispatcher onto one shared `WriteableBitmap`. Frames emit as fast as - the pipe produces them. Wired into the live pipeline (slice 1 step 4): `ResolveOutputFrame` reads - `_mediaManager.GetLatestFrame(MediaPath)` for a `MediaSource`, and `Source.DisplaySource` - (`Models/Source.cs`) routes `VideoImageSource` for `MediaSource` so the preview/canvas show decoded video. - Slice 2a shipped the probe seam: `FfmpegFrameRateParser` (pure, `avg_frame_rate=` then - `r_frame_rate=` rational N/N/M, unknown→null) + `IFrameRateProbe`/`FfmpegFrameRateProbe` (derives sibling `ffprobe.exe` - from the located ffmpeg dir, reuses the `IDecodeProcess` seam for the ffprobe subprocess text; null if - ffprobe absent) + `FfmpegLocator.ProbeFileName` now also extracts `ffprobe.exe` from the pinned archive - (conditional — old caches without it just get no pacing). Slice 2b wired pacing: `MediaVideoSource` - takes optional `IFrameRateProbe?` + `Func? delay` (default `Task.Delay`) - seams, probes FPS once in `RunAsync`, and delays by 1/fps after each emitted frame; no probe/unknown → no - pacing (ffmpeg's own pipe backpressure already throttles the decode). Slice 3 added loop control: - `IMediaFrameSource.Looping` (bool); `MediaVideoSource` takes a `Func` process factory (a single - `Process` can't be re-`Start()`ed, so each loop pass creates a fresh decoder) and wraps the decode in a - `do…while (Looping)` — restart on natural EOF instead of raising `Completed`. Production wiring (MainViewModel - media factory): passes `() => new FfmpegDecodeProcess()` as the factory, `FfmpegFrameRateProbe(new FfmpegLocator(), - () => new FfmpegDecodeProcess())` as the probe. - Still open: session acquisition on add/remove (UI picker) + wiring `Source.MediaIsLooping` into - `IMediaFrameSource.Looping` (needs a manager-level per-path loop provider, comes with the picker slice). -- **Social bar on the output (bar bug-fix branch):** the `FramePump` takes an optional - `socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam, re-read **every frame** (so a - mid-stream position flip applies immediately). The strip is pre-rasterized by `Compositor/SocialBarRenderer.cs` - (WPF glue — WPF glue here is fine because the strip is rendered once per config change on the UI thread, - the resulting immutable frame is then composited pure-CPU by `SceneCompositor`), and `SceneCompositor.Render` - blits it **last — above the branding flash** at `socialBarTop` (0 = top, `SourceRectHeight − barHeight` = - bottom) in master space. `MainViewModel` owns the frame (`_socialBarFrame`, rebuilt by `RenderSocialBarFrame` - on load/save/`NotifySocialsChanged`). -- Known consideration: the pump reads the active scene on a background thread while the UI can still edit - it; a concurrent-mutation exception is contained (logged + `Failed` + the pump stops) rather than - crashing. The background thread + video pipeline is the new reality since this step. +Invariants that hold even if you never open that file: + +- **rawvideo is stamped by ARRIVAL, never by a timestamp.** ffmpeg assigns pts from frame order at + the declared fps, so supply rate = output speed. The pump duplicates the previous frame on a missed + deadline (correct duration) and **never resets `nextTick` to wall-now** — rebasing turns a hitch + into a silent time-lapse (take-2 lesson, 2026-09-01). +- **Submit never blocks.** The encoder queue is bounded and a drain task owns the stdin write + + flush; on overflow the **newest frame is dropped and counted** (`DroppedFrames`) — overload costs + content, never a frozen or smeared session. +- **Stop order:** `StopAsync` stops the encoder first — flushing the pending queue to EOF so the file + finalizes with no accepted frame lost — **before** awaiting the pump loop; the reverse order + deadlocked on pipe backpressure. +- **Judge output in Release, by the burned `_outputIndex`** — the 6-digit counter burned into each + frame is what lets a recorded file be audited for drops after the fact; stats alone are not proof. ### Social bar (TASK 6 — shipped 2026-08-12, plan in TASKS.md; bar bug-fix branch 2026-08-13) diff --git a/docs/ai/frame-pipeline.md b/docs/ai/frame-pipeline.md new file mode 100644 index 0000000..f81c157 --- /dev/null +++ b/docs/ai/frame-pipeline.md @@ -0,0 +1,803 @@ +# Frame pipeline — compositor, render budget, measurement, and the live pump + +> Split out of [`ai.md`](../../ai.md) (2026-10-26) so a session reads only what its task touches; +> `ai.md` keeps a short invariant stub for this file. Directory map: [`index.md`](index.md). + +This is the subsystem that turns scenes into the encoder's frames: the CPU compositor that renders +the master, the budget it must fit inside, the instrumentation that measures it, and the +`FramePump` that paces frames into ffmpeg. Everything here is current architecture plus the +measured history that produced it; superseded designs are marked inline. + +--- + +## Budget, measurement & the compositor + +### Frame cadence — how a dropped frame became measurable (UNIT 0, 2026-10-05) + +**The problem this exists to solve.** The recorded file cannot distinguish a dropped camera frame +from a padded repeat. `-f rawvideo` carries no per-frame timestamps — ffmpeg assigns pts from frame +order at the DECLARED fps — so a webcam delivering 25 of its 30 frames still muxes a full-length, +correctly-timed 60fps file. Measured on take `ty-20261005-1001-821-0001.mp4` (85.6s, 1080p60): the +lower-right webcam region held 25.6 distinct images/s against the camera's 30fps, with 1,153 normal +33ms intervals, 281 gaps of 50–100ms, and 96 bunched 17ms gaps — roughly one hitch in five, which is +what makes a face judder while the file looks flawless. (The "webcam renders at ~18fps" figure that +used to sit in this map was WRONG: it came from a different session and counted every newly rendered +layer, not webcam arrivals. Corrected 2026-10-05.) + +**`Helpers/FrameCadence.cs`** — one pure counter, three feeds. `Add(timestamp)` on any shared +monotonic clock; `Summarize()` → frames, rate (gaps ÷ span, NOT frames ÷ span — N frames span N-1 +periods), gaps, hitches (default threshold 50ms ≈ 1.5× a 30fps period), longest and average gap; +`Describe(label)` gives every producer the same one-line shape so two lines can be read against each +other. An out-of-order stamp counts as a frame but produces no gap AND does not become the reference +for the next interval — otherwise one bad stamp invents a multi-second "hitch" twice. + +**Three measurement points, and what each separates:** + +| Line | Fed by | Answers | +|---|---|---| +| `Camera cadence WxH: arrivals … \| camera …` | `MediaCaptureFrameSource` | device/USB/driver cadence (`MediaFrame.SystemRelativeTime`, QPC domain, nullable) vs what the reader handed us. `TryAcquireLatestFrame` keeps only the LATEST frame, so falling behind is silent by design — this is where that shows | +| `webcam …` on the pump's stats line | `FramePump.TimedResolver` | distinct webcam `VideoFrame.Epoch` values that reached a composite — what actually got into the output | +| `frames per 5s from R renders` | `FramePump` | submits vs REAL composites — the duplicate-on-lag ratio, the only honest reading of how much of the stream is a repeat | + +**Why the stats-line denominators changed (the actual bug).** Every average was divided by +`statFrames`, the number of frames SUBMITTED — which deadline padding always fills to the full 60fps +target. A pump fed one webcam frame per 200ms logged "300/300 frames per 5s, avg render 12.4ms" +while producing 60 distinct composites: the render average was diluted ~5× and the loss was +invisible. Render/resolve/wait are now divided by `statRenders`; submit stays per-frame (it really +happens per submit). Regression test: +`FramePumpTests.Stats_ReportRealRendersAndDistinctWebcamFrames_NotJustPaddedSubmits`. + +**Reading the three together:** camera 30/s steady + arrivals 25/s = WE dropped it (reader or +compositor). Ragged in both = the device, the USB link, or the driver. Pump `webcam` below +`arrivals` = the compositor missed frames the reader had already delivered. + +**✅ WHAT THE FIRST REAL READING SAID (2026-10-05) — the camera was never the problem.** +`ty-20261005-1201-556-0001.mp4` + its log: the C920 delivers **30.0/s with ZERO gaps ≥50ms**, and the +device's own stamps are identical, so the camera, USB link, and driver are all clean. The pump did +**34 composites per 5s (6.8/s) at ~141ms each** (worst 294ms), and the file holds **6.8 distinct +webcam images/s at 88.7% duplicates with no arrival hitches at all**. The judder is entirely us +holding one face for 140ms and padding the rest of a 60fps container. `resolve` is 0.1ms, so it is +pure pixel work in the composite, not frame lookup. Two consequences, both load-bearing: + +- **720p is a sharpness win, NOT a smoothness win** — it triples the bytes stretched per render on a + path already 4.4× over budget. Schedule the exclusive lock AFTER the compositor work or it makes + the judder measurably worse. +- **The 10:01 take hit 25.6 images/s and the 12:01 take 6.8, same scene shape + (`full-render split=0 elements=6 dynamic=5`), same 640×480 camera.** Composites went 29–34ms → + 143–155ms between sessions with nothing logged explaining it. That comparison was made by EYE on + two takes because the log carried no per-element breakdown and no scene dump — both now exist + (see "Per-element composite timing + scene dump" below); naming the culprit is a reading, not a + guess, and guessing is explicitly out of bounds. **Superseded 2026-10-16:** "same scene shape" was + an eye comparison of takes whose geometry was never logged, and the next bad take still on disk is + the *vertical* tier (below). A Release 1920×1080 take now measures 1.0ms with no element over + 0.4ms — see "Per-element composite timing + scene dump" → what the first instrumented take said. + +**Counter subtlety, so it is not re-derived:** the pump's `webcam` line read 13/s while the file held +6.8/s. Not a bug — the counter stamps a frame when the resolver FIRST sees it, and one composite takes +141ms, so ~4 camera frames arrive during it (signature pass stamps one epoch, the actual draw uses a +later one). The file is the truth; the pump counter is a pipeline-health signal and overcounts by +design. + +**Still unmeasured:** the camera is stuck at the C920 default 640×480 YUY2 @30 because all 185 +`SetMediaStreamPropertiesAsync` attempts are refused under `SharedReadOnly`, so 720p MJPG is available +— but see above, it is a quality win only. + +### Per-element composite timing + scene dump (UNIT 0 follow-up, 2026-10-16 — MEASUREMENT ONLY) + +The gap the cadence work left open is now closed by instrumentation, and closed by nothing else: the +141ms take was never compared to the 30ms take because **no line in the log said which element cost +what, and no line said what the scene WAS.** Both now exist. No fix has been attempted, and none is +allowed until a real reading names the culprit. + +- **`Services/Compositor/CompositeTimings.cs`** — a collector owned by the pump, threaded through + `SceneCompositor.Render` / `CompositeLayers` / `BakeStaticBase` / `BlitBakedElement` / + `BlitCachedLayer` (and `SceneGraph.GetBakedBase` → bake path) as an optional last parameter, so + every other caller (thumbnails, tests, transition preview) is untouched and costs nothing. + `Add(stage, ticks)` accumulates; `Describe(renders, minMs = 0.1)` renders the averaged breakdown + slowest-first, dropping sub-0.1ms noise. +- **Stage names** (stable, greppable): `backdrop` (prefill + live capture + background art + their + resolves), `base` (the cached-bake `BlockCopy`), `flash`, `social`, `ticker`, `zoom`, `scale`, + `overlay` (the fully-static path's per-frame overlay copies), `bake backdrop`, `bake overlays`, and + **one entry per element as `# `** — the index IS the z-order, so the log names the + layer by both position and name. Bake-pass elements are prefixed `bake #…` so a one-off rebake + cannot masquerade as a per-composite cost. +- **`# raster` nests inside its element's total** and is the paste-cache rasterize miss + (`BlitCachedLayer`'s `raster == null` branch) — the direct answer to "is one element re-sampling + every tick, or is the paste itself slow". A stage's time includes that stage's frame resolve; + `resolve` is separately totalled in the existing stats line. +- **`FramePump parts (N renders/5s): …`** — logged once per 5s window (same window as + `FramePump stats`), averaged per render so the numbers compare directly with `avg render`, then the + collector is reset. **No parts line, or a short one, is a reading too:** the fully-static shortcut + and C4 cache hits composite no elements (`cache 92R/130H` beside a missing parts line is expected). +- **`FramePump scene dump: '' elements=N split=S`** followed by one line per element + (`[i] name kind=… type=… vis=… rect=x,y WxH op=… clip=…`), logged when the pump first runs and + whenever the active scene changes. This is the record the 10:01/12:01 comparison never had. +- Gate test: `ytLive.Tests/CompositeTimingTests.cs` (one `[Fact]`) — a REAL composite of a full-canvas + element plus a 64×64 one must rank the full-canvas element first, carry the `raster` sub-stage and + the fixed stages, and the dump must name both elements with z-index/type/rect and echo the split. +- **`scripts/webcam-cadence.py`** — the other half of a reading: what the FILE holds (distinct webcam + images/s + hold histogram), beside what the pump says it did. Recipe and its three traps (H.264 + re-encoding, the round-clip crop, scene switches): `MyMistakes.md` → "Measuring webcam CAMERA cadence + in a recording". + +**Read takes in Release.** The creator launches `dotnet run -c Release`; Debug pixel paths are +several× slower, so a Debug number is not a reading of this pipeline (see +[`ai.md` → Run](../../ai.md#run)). + +**Derivative check (AGENTS.md):** this is not invented — OBS's own answer to "which source is slow" is +the **Source Profiler**, which reports the *average execution time of each source's tick and render +functions* ([docs.obsproject.com → Source Profiler](https://docs.obsproject.com/reference-libobs-util-source-profiler), +per-source timers via `util/profiler.h`). Same shape here: one timer per element per composite, reported +as an average, plus the stage split. The `# raster` sub-stage is the extra this case needs — +it separates "re-sampled every tick" from "slow paste" without another session. + +### ✅ What the first instrumented take said (2026-10-16, Release) — the judder DID NOT reproduce + +Take `ty-20261016-1841-559-0002.mp4` (174.5s, 1920×1080@60, Release build `e95154d9` — a **recording**, +not preview; scene `Live`, switched to `Chat` at 28s, back to `Live` at 44s) and its log window: + +| Reading | This take | +|---|---| +| composites | **301 per 5s = 60/s**, every slot; 0 dropped, 0 stalls after warm-up | +| avg render | **1.0–1.1ms**, worst 2.5ms — ~7% of the 16.67ms budget | +| `parts` | `#5 HD Pro Webcam C920 0.4ms, #5 … raster 0.2ms, backdrop 0.2ms` — the other four elements are under the 0.1ms print floor | +| C4 full-composite cache | **151R/150H** per window — half the ticks were a whole-frame `BlockCopy` | +| webcam at the resolver | **30.0/s**, longest gap **50.0ms** (one extra 60Hz frame = sampling phase, not a drop) | +| the file itself | **29.6 distinct webcam images/s, 82% of images held exactly 2 frames**, longest hold 267ms (`scripts/webcam-cadence.py`) | + +Nothing in this take is over budget and **no element is expensive — there is no culprit here to fix.** +The two stalls in the take are not per-composite cost: 124ms on the very first render (warm-up) and +53ms on the switch into `Chat` (a rebake). + +**Scale of the disagreement:** the historic slow readings are 141ms (Oct 5 12:01) and 45ms (Oct 9 +19:48) — 40–140× the ~1ms this take needed per composite. Three facts about those takes, all read off +the same log: + +- **Oct 9 19:48 (`ty-20261009-1948-121-0000-2.mp4`) was the VERTICAL tier**, not a 16:9 one: + `FramePump started (1080×1920 @ 60 fps) scene='Live' comp=608x1080` — the 607×1080 centre crop + bilinear-upscaled to 1080×1920 (2M px), 92 renders/5s, avg render 45.0ms, worst 278ms, 92 stalls, + `cache 92R/0H`. The file on disk is 11.6s and every frame is identical (a static desktop), so it + cannot be read for cadence — only its cost. +- **Oct 5 12:01 (`…1201-556-0001.mp4`, no longer on disk) logged `FramePump started (1920×1080 @ 60 + fps)` with NO `scene=/comp=` suffix** — that suffix did not exist yet, so its source rect was never + recorded and the map's "21:9 → 1920×803" explanation stays a hypothesis. +- **A take can latch mid-take.** Oct 5 12:29 goes 1.5ms (avg render) → 3.0ms → **45.2ms within 15 + seconds and stays there** (`cache 10R/0H`, then `98R/0H`). So the slow state is not a fixed property + of a scene geometry. The consistent signature of every slow window is `cache xR/**0H**`: the C4 + whole-frame cache never reused once, so every tick composited. The healthy take shows the other + stable state (composite 1ms → every second tick is byte-identical → whole-frame reuse). **The latch + is the thing to test on a reproducing take** — it is a hypothesis, not a reading; the `parts` line + is what will decide it. + +**What this does NOT say.** This take is the 1:1 master tier and its scene logs +`full-render split=0 elements=6 dynamic=4` (the bad-period takes logged `dynamic=5`), so it never +exercises the failing geometry. "The healthy path is healthy" is not "the judder is gone". Reproducing +it needs a Release take in the failing configuration (vertical tier first, then the 21:9 display +config) — that take's `parts` line names the stage, and only then is a fix unit allowed. + +**Reading the `parts` line's denominators:** like `avg render`, it is divided by `statRenders`, which +counts C4 REUSES as well as real composites (the 301 above = 151 real + 150 whole-frame copies). So in +a healthy 30fps-camera take both averages are ~2× optimistic; the `cache NR/NH` pair on the same line +says how much of the window was real composite work. `CompositeTimings.Describe` drops stages under +0.1ms by default, and **a missing or short parts line is itself a reading** — a fully-static window +(every tick a C4 hit) composites no elements at all. + +### Render budget — getting a composite inside 16.67ms (UNITs 4-5, 2026-10-05) + +The pump emits one encoder frame per 16.67ms slot and **pads with a duplicate when a composite +overruns**, so the whole problem is "render + submit + untimed work ≤ 16.67ms, ideally ≤12ms to +leave headroom". `FramePump` now reports per-stage timings (`social`/`ticker`/`flash`/`acquireScratch`) +so a slow stage names itself, and logs the scene name + compositor source rect at pump start. + +**Why a 21:9 monitor cost 141ms and a 16:9 one 33ms — the fast path is EXACTLY 1920×1080.** A +21:9 display shrinks the source rect to 1920×803 ("fit", not "fill"), so the opaque-backdrop 1:1 row +walk is skipped, the base is prefilled with opaque black (~2M byte writes), and the backdrop is +resampled per-pixel (~1.54M pixels with an alpha branch). Every recorded A/B lines up: +21:9 → 141ms, 16:9 → 30-33ms, 2560×1440 (fits to 1920×1080 exactly) → fast path. +**Corrected 2026-10-16 — the "never Debug-vs-Release" half of that is withdrawn.** A Release +1920×1080 take measures **1.0–1.1ms** where an Oct 9 take of the same tier measured 10.4ms, and +`Build ` carries no build configuration, so the config of every pre-2026-10-16 take is +UNKNOWN: no historic render number may be compared against a Release take. The 21:9 explanation +itself stays a hypothesis — the 141ms take's start line predates the `scene=/comp=` suffix, so its +source rect was never recorded. A take can also latch mid-take (see the instrumentation section). + +**Unit 4 — cheap wins, all pixel-identical:** +- Opaque 1:1 fast paths expanded to the whole-canvas opaque case (a `BlockCopy`, not a per-pixel + alpha branch). +- **Prefill only the uncovered rows.** When the backdrop does not cover the canvas, the letterbox + bands are filled instead of the whole buffer; with no backdrop at all it still fills everything. +- **Per-column LUT** for the scaled blit (`BlitContentRaw`): the source column, its neighbour index + and the 8.8 weights are computed once per column instead of a divide + clamp per pixel. + The table comes from `ArrayPool.Shared` and is returned in a `finally` — a single static + buffer is a **data race** the moment strips run in parallel (see below). + +**Unit 5 — strip-parallel composite: `Helpers/StripPool.cs`.** One composite splits into four +horizontal row strips; the calling thread takes a strip too, so four workers never means five-way +oversubscription. Below 64 rows the fan-out costs more than it saves and runs inline. Each strip must +write only its own rows — the blits already partition by row, which is why the output stays +byte-identical. + +**The demotion must come from DEDICATED threads, not the shared pool.** The obvious +`Thread.CurrentThread.Priority = BelowNormal` inside `Parallel.For` is wrong: `Parallel.For` runs on +shared ThreadPool threads, so the demotion leaks into every unrelated work item that thread later +picks up. Research (https://stackoverflow.com/questions/524483/net-backgroundworker-objects-thread-priority +— "you have next to no control of which thread pool thread will actually run your task"; +https://stackoverflow.com/questions/1578599/c-sharp-lower-thread-priority-in-thread-pool — "You +shouldn't mess with the priority of the regular ThreadPool, since you aren't the only consumer ... +write your own producer/consumer queue, using your own Thread(s) as the consumer(s) - setting the +thread priority yourself") says exactly that, and **a real failed attempt confirmed it**: a full test +run after demoting the pool threads broke an unrelated audio-mix timing test. `StripPool` IS that +producer/consumer queue — four `Thread`s with `Priority = BelowNormal`, `IsBackground = true`, +started lazily on first use, each fed a `Queue` + `SemaphoreSlim`. Four is the measured sweet +spot: fewer leaves render over budget, more steals cycles from the game and the encoder and made +render **worse**. + +Gate test: `SceneCompositorTests.StripParallel_RepeatedRenders_AreByteIdentical` — the same scene +(scaled round webcam + mirrored image + master-sized flash over a short backdrop, 240 rows so all +three fanned-out paths run) must come out byte-for-byte identical across 8 renders, and must differ +from a backdrop-only render so it cannot pass vacuously. It earned its keep immediately: it caught a +loop-variable capture (`new Thread(() => Worker(i))` where `i` was `Workers` by the time the thread +ran) that crashed the test host on first run. + +### Scene compositor (TASK 4 ship step 1 — shipped 2026-08-10, plan in TASKS.md) + +The encoder needs the master 1920×1080 frame **without** the preview's editing chrome (SelectionOverlay, +DimRects, output-rect outline, badge, placeholder). WPF's `RenderTargetBitmap` is software-rendered and +captures the visual tree *including* chrome, so the preview can't be captured — the output is a **second, +parallel software compositor** over the `VideoFrame` (BGRA8) seam, and the XAML preview +(`MainWindow.xaml` CanvasGrid + element DataTemplate) is the rendering contract it replicates. Two +renderers must agree: geometry, `UniformToFill` cover-crop, round clip, mirror, border, z-order. Preview +stays XAML (editing view); the compositor is the output view. + +- **The `DimRects` shading is itself preview chrome, so it renders as a SIBLING of `CanvasGrid` + (above it, below the `ZoomBox`), never a descendant** (2026-10-08, creator ruling): the zoom box's + `VisualBrush` samples all of `CanvasGrid`, and if the dims were inside it the box would show the + dim shading — especially after the box is dragged away and the fixed capture slides under the dims. + +- **Render the active tier's output rect directly** (`CompositorOptions {SourceRectX/Y/W/H, + OutputWidth, OutputHeight}`, fed from `MainViewModel.OutputRect*`): 16:9 = full 1920×1080 1:1; the + vertical 9:16 tier = composite the centered 607×1080 crop then bilinear-upscale to 1080×1920. +- **CPU posture:** with FFmpeg as a subprocess the master crosses a CPU readback to the pipe every frame + anyway, so GPU compositing buys little at this layer count (2-3 live layers; static layers + pre-composite once into a cached base). GPU effort belongs to **NVENC** (the encoder), not composition; + if composition grows (wipes, filters, many layers), a D3D11 compositor can replace this one **behind + the same seam** — the CPU master buffer stays the contract. +- **Every global overlay must be composited on EVERY render path (bug found 2026-09-26).** The + frame pump has three exits: `RenderFull` (no graph / no static base), `CompositeLayers` (static base + + dynamic tail), and the **fully-static shortcut**, which stretches the cached bake and returned it + directly. That third path silently **discarded every per-frame overlay** — the social bar, the alert + ticker and now the brand flash. A creator whose scene was one static background saw none of them, + and the flash looked "preview-only forever" for a reason unrelated to the tier. `SceneCompositor.Overlay` + is therefore `internal static` (it copies before blitting, so the cached bake is never mutated) and + the shortcut now composites overlays onto a copy before stretching. **When adding an overlay, grep + for all three paths** — a fourth exit that forgets it fails the same silent way. +- **`VideoFrame` buffers are recycled by some producers, and `Epoch` is not optional.** A producer that + hands out the same array across frames MUST bump `Epoch`, or `FramePump`'s buffer-identity paste cache + false-hits and freezes stale content (the take-11 class of bug). The `C4` cache signature mixes + `Epoch`, so a per-frame overlay naturally invalidates it. `BrandFlashPresenter` reuses one 8MB + master buffer and stamps `Epoch` per read for exactly this reason — allocating a fresh 8MB frame per + tick is not an option at 1080p30. +- **Dev-only: two instances side by side (`Helpers/InstanceProfile.cs`).** Pre-1.0 the creator runs one + instance to stream and another to screen-capture it. Nothing prevented a second instance (no mutex, no + port); what broke it was shared state — the whole-scene read/write layout DB (clobber), Chromium's + **exclusive** lock on the WebView2 user data folder (second instance won't start), the auth token store + (a test instance overwrites the real YouTube sign-in), and `startup.log`. Set + `YTLIVE_INSTANCE=` and that process gets a private root at `%APPDATA%\ytLlive\instances\\` + for the DB, the auth file, the log and the WebView2 folder. Recording folder and the ffmpeg `tools\` + cache stay shared on purpose; global hotkeys stay un-namespaced (Windows refusing the second + `RegisterHotKey` is the correct answer). The **entire implementation is inside `#if DEBUG`** — a + Release build compiles to `DataRoot => DefaultRoot` and `WebViewDataFolder => null`, and the call + sites are unconditional so Release cannot drift. Proof + harness: `ytLive.Tests/InstanceIsolationTests.cs`. + ⚠️ Grepping the binary for `YTLIVE_INSTANCE` proves nothing — a `const` is inlined and appears in + neither build; check the `InstanceVariable` **field** in metadata instead. +- **Branding flash is composited by the output path too** (it's on the live output, per Monetization), + passed in as a pre-rendered `VideoFrame?` — the compositor core stays pure byte-math, no WPF. Likely a + bundled asset rather than runtime text rendering (deterministic, no font/layout risk). +- **Frame sources are injected** via a `Func` resolver + (`SceneCompositor.Render(scene, frameFor, flashFrame, options)`) — the caller maps each element to + its frame (webcam → `DeviceId`, image → `AssetId` via `StaticPixelCache`, backdrop → `CaptureKey`), + so the compositor is pure, WPF-free, and hermetic to test. The capture managers wire into that + resolver in the encoder step, not the compositor step. The master buffer (the compositor's return + value) is the seam a future D3D11 compositor would honor identically. + +--- + +## The live pump — frame producer to encoder + +### Live frame pipeline (TASK 4 ship step 5 — shipped 2026-08-12, health stats 2026-08-13, plan in TASKS.md) + +The **`FramePump`** (`Services/Encoder/`) is the live frame producer: while live it snapshots the active +scene each tick, resolves every element to its latest frame, composites it into the tier's output frame, +and paces frames into the encoder at the tier's FPS. **rawvideo is stamped by ARRIVAL:** ffmpeg assigns +pts from frame order at the declared fps — supply rate = output speed. A starved producer (pump < fps) +ships a time-lapse, truncated file with NO error (take-2 lesson, 2026-09-01); the pump therefore logs +`FramePump stats: n/target frames per 5s from R renders, avg render Xms (resolve R2), avg submit Yms` so a slow stage names itself (the `frames … from R renders` pair and the render-count denominators arrived with UNIT 0 — see "Frame cadence" above; before that every average was divided by the SUBMITTED count, which deadline padding always fills to the full target, so a pump fed one webcam frame per 200ms still logged "300/300 frames" and its render average was diluted ~an order of magnitude) — and since 2026-09-04 so does the BUILD: every build gets a GUID stamped +into `Helpers/BuildStamp` (csproj `GenerateBuildStamp` target, fresh per compile — no lying incremental build), logged at startup +("Build xxxxxxxx (compiled ...)"). **Take 6 (stamp-less, ambiguous): render 35-41ms with slice 3 built or not unknown — attribution is why the stamp exists; takes 7+ are +readable.** If the breakdown says resolve dominates, the suspects are pre-cached chat/config thrash or the web frame; if blit dominates, the general-path element sizes. **Pattern — everything is a constructor-injected +seam:** `Func`, `Func` resolver, `Func`, +`Func`, `Func`, `Action` log, and an injectable pacing delay +(default `Task.Delay`; tests inject `Task.Yield`). The pump is free of WPF and of the capture managers. + +- **`StartAsync` never throws** — the VM fires-and-forgets it from the sync command handler; failures log + + surface via the `Failed` event. `BuildEncoderOptions` now always returns options (TASK 18) with + `StreamEnabled`/`RecordEnabled`/`RecordPath` baked from the pills + `_activeRecordPath`; the pump skips the + encoder only when `options == null`, i.e. "no output configured". `MainViewModel._rtmpUrlProvider` is that + seam — a `Func` that now yields the reusable stream's ingest URL (TASK 9, shipped 2026-08-16): + loaded from the `LayoutStore` cache at startup and set fresh by `PrepareAndStartLiveAsync` before the pump + starts. **The pump reads the URL once at startup**, which is why go-live ensures the stream BEFORE + `StartAsync`. +- **Deadline pacing + row-blit render (take-3 starvation fix, 2026-09-03):** take 3 recorded 30s into a + 2.1s file and the 5s stats line named the cause in one number — `17/300 frames per 5s, avg render + 258.1ms, avg submit 1.5ms`. Two defects, both fixed: (1) the pump slept the FULL interval after each + render, so period = render + interval — OBS's `video_thread` (libobs/media-io/video-io.c) pattern + replaces it: absolute `nextTick += intervalTicks` deadline, sleep only the remainder, and on overrun + skip the wait. (NOTE, slice 9: the original "skip the missed ticks (rebase)" half of this fix was + wrong — see the slice 9 correction below.) (2) the compositor did + per-pixel float sampling + `Math.Round` blending over all 2.07M master pixels (backdrop) and scanned + the whole destination per overlay (a 64px social-bar strip cost 2M iterations). `SceneCompositor` now + follows the libyuv pattern (BSD-3, chromium.googlesource.com/libyuv/libyuv — cited per the + derivative-work rule): 1:1 aligned blits take a row-walk fast path (bilinear at scale 1 + integer + offset is the identity) with per-pixel alpha branch and integer fixed-point blend; `BlitOverlay` + clips to the intersection rect; a full-cover live backdrop skips the opaque-black pre-fill. The + general per-pixel path handled scaled/round/mirrored elements. Test: + `Pump_Paces_To_The_Deadline_Compensating_Render_Cost` records the requested wait + via the pacing seam (a seam fake must genuinely await — a synchronous completed task runs the whole + pump loop on `StartAsync`'s continuation and hangs the run; MyMistakes). +- **Slice 2 of the render fix (2026-09-04, take 4):** the pacing held (file no longer truncated at + loop level) but render stayed at 58.9ms — the per-pixel row walk was still 2M managed iterations and + every tick allocated a fresh 8.3MB master buffer. Three changes: (1) **`VideoFrame.IsOpaque`** — a + producer-contract flag (only the screen-capture and webcam paths set it; DWM/MediaCapture fill alpha + 255 by contract); a full-canvas, aligned, opacity-1 blit of an opaque frame is now ONE + `Buffer.BlockCopy` (~1.5ms) instead of the loop, and the black pre-fill is skipped when the backdrop + covers. (2) **integer fixed-point bilinear** in the general `BlitContent` path (row-hoisted invariants, + no divisions, no `Math.Round`) — within ±1 of the float reference, inside the ±2 test tolerance. + (3) **scratch pool in the pump** — `AcquireScratch`/`ReleaseScratch` recycle the master buffer + (max 4, keyed by length, owned-by-reference so bake-cache/social-bar/static-art arrays can never be + captured); release happens strictly AFTER `SubmitFrameAsync` returns (the write to stdin copies), + and the free-list Contains guard makes the transition Cut path safe (`BlendFrame` returns the + to-frame itself, aliasing the scratch). The per-tick `fromScene` render in the transition branch was + **dead weight** (BlendFrame uses `TransitionService.FromFrame` captured at `Start`, never the pump's) — + removed, and the pump's `fromSceneProvider` seam + `MainViewModel` call site went with it. Tests: + `Pump_Pools_ScratchBuffers_Across_Frames_Without_Stale_Pixels` (the ONE: alternating backdrop colors + pin every frame's content, repeated backing-array identity proves the pool recycles), + `Composite_OpaqueFullCover_Backdrop_CopiesEveryPixel_Into_Scratch` (memcpy branch + scratch + sentinel). `FakeEncoder.SubmitFrameAsync` now snapshots bytes like the real stdin write — holding the + reference would race legitimate recycling. Take 5 must show `avg render ≤ ~10ms, ≈300/300`; the + vertical tier's final 1080×1920 `BilinearScale` still allocates fresh per frame (same GC lesson when + someone streams vertical — recorded as a follow-up, not silently "done"). +- **Slice 3 — chat raster-on-change (2026-09-04, take 5):** take 5 measured `138/300, avg render + 25.5ms` — the per-tick blits were cheap now, but `ResolveOutputFrame` → `RenderChatBox` ran a FULL + WPF raster (`ChatBoxRenderer`: FormattedText + `RenderTargetBitmap` + CopyPixels + channel swap) + EVERY tick whenever the message buffer was non-empty — and the buffer survives between sessions, + so even a signed-out record-only take paid it. Established answer, same as OBS text sources: + **raster on change, blit the cache every tick.** `ChatOverlayLayer` now keeps a content version + (`Messages.CollectionChanged` → `_contentVersion++`) plus a config key (box size + all Chat* + props); `RenderFrame` returns the cached `VideoFrame` by identity until either changes (the + compositor only ever reads a cached frame). Test: `ChatOverlayLayerCacheTests` (RealApp, real + renderer — `Same()` for unchanged inputs, `NotSame()` on message/config change, null on empty). + Accepted cost pending take 6: one slow tick (~15-25ms) per arriving message; if chat-burst frame + loss shows up, the next slice moves the re-render off-tick (debounced, dispatcher-side). +- **Slice 5 — paste cache for non-opaque layers (2026-09-04, take 7/8 data):** the render/resolve + split (added in the stamp commit) finally named the last thing honestly: `resolve ≈ 0` but + `render 26-27ms` (124-135/300, ≈2.2x) — the chat fix HAD worked; the remaining cost was the + compositor re-rasterizing EVERY layer per tick even when its pixels never changed (this creator's + Live scene: chat 159k + web widget 271k + image 95k + webcam 156k px ≈ 680k samples @ ~38ns each). + OBS's actual shape: sources cache their surface, the compositor pastes. `SceneCompositor` now + routes non-opaque layers through `BlitCachedLayer`: a layer rasterizes ONCE into an element-space, + TRANSPARENT-based frame keyed by (source-array identity, source W×H, ceil'd dst rect, round, + mirror), then every later tick PASTES it (integer position, row alpha-blend, opacity applied at + paste). **Correction (2026-09-12, the web-widget "black box"):** the raster builds on a + TRANSPARENT base, but the sampler's PARTIAL-alpha branch used the opaque-dst blend — it + premultiplied the color into RGB and forced `alpha=255`. A translucent widget pixel then pasted + as opaque darkened ink (the box in recordings) while the raw-bitmap preview stayed correct. + `BlitContentRaw` now takes `transparentDst` (the raster call passes `true` and writes straight + color + straight alpha; the paste rows do the source-over). Master paths are bit-identical. + Guard: `PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk`. Producers hand out fresh immutable arrays, so array-identity keys can never serve stale + content; dict bounded at 48, cleared wholesale on overflow. Only changing content (webcam device + frames, web capture ticks, chat messages) resamples; the opaque backdrop keeps its memcpy path. + Position/opacity drags are now near-free (no resample — paste params, not cache keys). + ONE integration test: `PasteCache_RepeatRender_IsByteIdentical_And_ContentChangePropagates` + (raster-vs-paste byte equality incl. round-clip margins + new-array propagation); the whole + existing pixel suite guards sampler semantics. Take 9 must show `≈300/300, avg render ≤ ~8ms`; + if it lands there, the recording saga closes. Follow-ups unchanged: vertical-tier scale alloc, + debounced chat re-render under message bursts. +- **Slice 6 — the sleep quantum WAS the ceiling (take 9, 2026-09-04):** period measured ~37ms while + work (render+submit) was ~25 — the missing ~12ms is `Task.Delay` rounding EVERY request up to the + Windows clock tick (documented ~15.6ms default; learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.task.delay). + A frame finishing 3ms early requests a 3ms wait and sleeps a full 15.6 — capping the producer at + ~27fps NO MATTER how fast the compositor ran. This is why takes 7→9 showed zero playback change + despite real render wins: the sleep floor dominated everything above it. Established fix (game-loop + canon — stackoverflow.com/questions/5441464; timeBeginPeriod — learn.microsoft.com/en-us/windows/win32/api/timeapi/nf-timeapi-timebeginperiod): + `timeBeginPeriod(1)` for the pump's lifetime (paired `timeEndPeriod` in the finally), sleep only the + BULK of the remainder (request minus 2ms), `Thread.SpinWait` the last ~2ms across the deadline; + blown deadlines still rebase. (SUPERSEDED in part: slice 9 deleted the rebase — deadlines only + advance; slice 10 replaced the burst re-write with one fresh composite per iteration.) Stats gained + `avg wait Xms` so render+submit+wait must ≈ period — + the accounting is closed, no stage can hide again. Same slice: the webcam's `IsOpaque` paste-cache + bypass was removed (it re-sampled ~156k px every tick even between identical device frames; the + cached paste beats the sampler on hits and costs the same on misses). Take 10 verdict: `≈300/300` + honest fps — if short, the wait/render split names the remaining term with no ambiguity left. +- **Slice 7 — the loop was ON the UI thread the whole time (take-8 numbers, 2026-09-04):** with + the wait finally measured, take 8 (59a02a5b) showed the impossible pair — `render 22ms, wait 10ms` + against a 16.7ms deadline: a blown deadline rebases with NO wait. The wait was the producer + **queued behind the live preview on the dispatcher**: `PumpAsync` starts from a UI command handler + and its `await` continuations inherit the UI `SynchronizationContext`, so the "WPF-free" compositor + actually rendered on the UI thread and ran only when WPF let it. OBS runs its media loops on + dedicated threads for exactly this reason. Fix: `_pumpTask = Task.Run(() => PumpAsync(...))` + (no sync context inside → continuations stay on the pool). Side effects handled, not ignored: + `StaticPixelCache.Get` now locks (pool miss-decodes race UI callers); `ChatOverlayLayer.RenderFrame` + checks its cache off-thread but MARSHALS the rare raster miss to the dispatcher (DrawingVisual/ + RenderTargetBitmap are UI-thread objects) and re-validates there; pump events already marshalled. + Also: `GCLatencyMode.SustainedLowLatency` for the pump's life, stats gained `worst render Xms` + (spike visibility — bimodal averages hid them), webcam now routes through the paste cache (bypass + re-sampled 156k px even between identical device frames). Tests: `Pump_Produces_OffTheStartingContext` + (the ONE — inline-pumping SyncContext proves continuations never return to the starting thread) + + 70/70. Take 9 verdict: wait ≈ true remainder (period → 16.7, n → ~300); `worst render` names any + remaining raster-miss spikes; render+submit+wait still == period — the accounting holds. +- **Slice 8 — capture buffer ring + paste-cache Epoch (take 11, 2026-09-04):** take 11 validated + the architecture — typical frames now land `work ~10ms + wait ~6.8ms = 16.7`, exactly the deadline + (212/300). The whole remaining gap is PERIODIC 35-65ms worst-render spikes that worsened across + the take (189→147 frames/5s) — the signature of gen2 GC pauses, now provable via `gen2 +N` in + every stats window. Biggest churn was structural: `ScreenCaptureFrameSource` minted a fresh +~8.3MB `byte[]` per DWM frame (~500MB/s LOH). It now rotates a shared-frame ring (4-deep at slice 8, + deepened to 8-deep after the "flash" finding — consumer holds must never outlive depth × source + period), size-matched per slot, with reused downscale row scratch. Recycled + arrays would poison the paste cache (it keys on array IDENTITY), so `VideoFrame.Epoch` — + monotonic per producer frame, 0 for fresh-array producers — joins the key. Regression test + `PasteCache_RecycledArrayWithNewEpoch_ReRasterizes_NotStaleHits` fails on the old key by + construction. The camera producer carried the same fresh-array churn (110-220MB/s at 30-60fps): + it now rotates its own 8-deep ring + Epoch (same identity rule), and the WebView2 capture + reuses a canvas scratch + 8-deep output ring + a reused WriteableBitmap instead of minting two + fresh arrays + a new bitmap per 10Hz tick. Next suspect if gen2 stays hot: the WPF preview load + itself driving gen2 — recorded as follow-up, untouched this slice. +- **Slice 9 — the "rebase" WAS the recording time-lapse (2026-09-10, the recording playback-timing + take):** slice 6's "blown deadlines still rebase" was the recording-timing bug all along. In a real + take the pump emitted `215-219/300 per 5s` (~43fps) while `render+wait == period` still looked closed — + the rebase ERASED every missed slot (deadline = wall-now again), so the missed ticks never showed in + the stats. rawvideo has no per-frame timestamps: ffmpeg muxes by frame count at `-framerate 60`, so a + 43fps reality was authored into a 60fps container and **every recording played ~1.4x fast** (takes + confirmed with a WSL ticker visible in the recording: file duration 11.44s vs ~15.5s wall). Two + defects, both fixed the OBS way (`libobs/media-io/video-io.c` — the video thread NEVER resets its + deadline; every interval tick outputs ONE frame, and a late render means repeated content — judder — + never a skipped timestamp): + (1) **count-based CFR emission:** `while (GetTimestamp() >= nextTick) { SubmitFrame(frame); nextTick += intervalTicks; statFrames++; }` + submits exactly one frame per crossed slot, re-writing the current composite when the render overruns + (duplicated footage = correct duration, not a time-lapse), and `nextTick` is NEVER reset to wall-now; + (2) **`-re` removed from `FfmpegArgs`** — it was a second, fighting pacer on the rawvideo demux + (its "Resumed reading … after a lag" grew 0.79s→4.82s across the take, leaving the pump behind its own + honest-stats count). One fix covers the game background AND the webcam — both flow through this one + pump. Muxed duration is now frame-count ÷ fps == wall time by construction. Test: the pump suite stays + green unchanged — the pacing test's `` (cap 120 ≈ + 2s at 60fps) owned by a dedicated drain task with the stdin write + flush; the caller NEVER blocks on + the pipe. Pixels are copied into an `ArrayPool` buffer before enqueue (the write now happens later on + the drain thread, so the pump's scratch reusable the moment submit returns — same contract as before). + (2) **drop-on-overflow (creator's choice: freshness over coverage):** when the queue is full the NEWEST + frame is dropped and counted (`IFfmpegEncoder.DroppedFrames`, `Interlocked`); the session never freezes + or smears — it drops. Stop flushes the whole queue then EOF (`Channel.TryComplete` → drain writes + leftovers → closes stdin → ffmpeg finalizes+exits), so no accepted frame is ever lost at stop. + (3) **pump loops ONE submit per iteration** (replacing slice 9's burst): every due slot gets a FRESH + composite — no catch-up slot ever repeats frozen content; missed slots vanish as a count-based gap. + The deadline counter is still never reset to wall-now (slice 9 ruling). + (4) **burned-in frame counter (the new judge):** `_outputIndex` is burned into a 6-digit dot-matrix + strip (white box + black 5×7 glyphs, bottom-right corner) of EVERY composite before submit. The WSL + ticker is demoted because its own timers smear under Windows host load; decoding the recording and + reading the strip is clock-independent: the number advances +1 per frame and jumps by exactly the + counted drops (queue overflow OR skipped catch-up slots). Stats gained `worst submit Xms`, `dropped N`, + `stalls K`; a stall logger names any iteration > 2× interval with its render/submit split — with the + queue, submit is ~1ms, so a stall means render/resolve. The strip sits above the social bar (bar is + composited, then burned over) — a debug judge, tiny at 1080p. Test: + `Backpressure_QueueOverflow_DropsFrames_AndNeverBlocks` (the ONE: a 40ms-per-frame sink fake makes + every enqueue fill the queue; submit must return instantly, drops are counted, stop flushes exactly + submitted − dropped bytes). Full suite 289/290 (the pre-existing compositor pixel failure unchanged). + Audio untouched (follow-up). WSL ticker-under-load reliability check still to run (informational — the + burned counter is the judge). +- **Slice 11 — web-layer animation ran at ~1/6 speed; capture cadence 10Hz→~30Hz + de-throttle + (2026-09-10, the "widget animates too slow" take):** the recording is 60fps but the WebView2 + capture loop was a blind 100ms `DispatcherTimer` = 10Hz — a 60fps-designed widget was sampled 6× + under its native rate (repeated-footage slow-mo). Two stacked throttles: + > Superseded by **Slice 14** (below): `CaptureScheduler` and the cadence hooks were the polling + > design; frame-driven composition capture replaced them — nothing here is current architecture. + (1) **the 10Hz cap** (unconditional, code-proven) and + (2) **Chromium hidden-page throttling** (`requestAnimationFrame` parked, JS timers clamped to ~1s + per WebView2Feedback#1172/#3070 + Chrome-88 timer throttling) whenever the app window is unfocused + or covered — the page is off-screen at (-5000,-5000), so it is a hidden page the moment the host + window loses occlusion. + Fixed: + (1) **`CaptureScheduler` (new, `Services/CaptureScheduler.cs`)** replaces the per-session + `DispatcherTimer`: a dispatcher timer that DROPS ticks while a capture is in flight (latest-wins, + never queues) — the in-flight drop is what made raising the cadence safe (concurrent full-HD PNG + `CapturePreviewAsync` calls would stack ~10-30ms encodes and publish stale-after-fresh). Interval is + owner-configurable: recording 33ms (~30Hz), idle 200ms. Unit-tested without a WebView2 runtime + (the ONE test: `CaptureScheduler_Drops_Ticks_While_Capture_InFlight_And_Resumes` — overlapping + ticks dropped, capture resumes when idle, driven by a TCS so it is deterministic). + (2) **de-throttle browser args**: the shared `CoreWebView2Environment` is created with + `--disable-backgrounding-occluded-windows --disable-renderer-backgrounding + --disable-features=CalculateNativeWinOcclusion` (the Electron/Streamlabs-class embedder answer for + occluded-window animation throttling) BEFORE `EnsureCoreWebView2Async`. + (3) **cadence hook**: `MainViewModel` sets `SetCaptureInterval(33)` on record/stream start and + `(200)` on stop (`MainViewModel.Streaming.Operations.cs`). + (4) **capture-cost telemetry**: first 30 captures per session log elapsed ms to startup.log — + PNG-encode+decode cost decides whether ~30Hz stays or drops to ~20Hz; the FramePump drops frames + (never time-lapses, slice 10) if UI-thread GC churn starves it, so the cost is observable. +Full suite 290/291 passing, the sole failure the pre-existing compositor pixel test. The web-overlay + transparency verification (the first-capture diagnostic bound to about:blank) and the audio-silence + item remain open push-gate items — both untouched by this slice. +- **Slice 12 (audio diagnostics, 2026-09-10):** the recorded audio is full-length silent AAC + (−91dB, 1124 frames / 23.95s in the latest take) — the named-pipe delivered ~9.1MB of zeros to + ffmpeg for the entire recording, so the loop ran and connected, but both WASAPI capture sources + delivered nothing. Sources started fine (`Audio: using system default mic...` logged), no failure + callbacks fired, and the mixer's existing integration test (`Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe`) + proves the loop→pipe math is sound. The fault is capture-side: either the default render endpoint + carried nothing (audio played on a non-default device — common), or both endpoints were held + exclusive, or genuinely nothing played. Added permanent per-5s live-loop telemetry to startup.log + (`Audio live: pipe connected=, dropped writes=, micLevel=, loopLevel=, drained, peakMix`) and + `NamedPipeAudioWriter.DroppedWrites` — names the exact stage on the next take without new + code. `AudioMixer.FillAndMix` now returns `(MicRms, MicDrained, LoopDrained)` for the telemetry + accumulation. Full suite 290/291, same pre-existing sole failure. +- **Slice 13 (web transparency diagnostic re-point, 2026-09-10):** the "?" in MyMistakes' + "RESOLVED (? VERIFY)" is being closed. The one-shot widget dump + alpha stats bound to the + FIRST capture ever = the initial about:blank document — a blind instrument. NavigationCompleted + for the REAL widget URL now arms `WidgetDumpRemaining = 5`; those 5 post-paint captures dump + `%TEMP%\ytLive-web--w1..5.png` + the shared `AlphaStats(...)` line (alpha[min/max/mean/zero%]) + + FindContentBounds result to startup.log. That line alone decides the fix branch: zero% ⇒ the + pre-parse injection did not hold for this widget (opaque capture → chroma-key / stronger DOM + injection); large zero% + tight contentBounds ⇒ the capture IS transparent and the recording's + black block lives downstream (compositor blend/underlay). `AlphaStats` extracted as the shared + sampler. No new tests (WebView2 runtime not instantiable in tests; the re-point is log-only). +- **Slice 14 — web frames are now COMPOSITION-CAPTURED; the PNG poll + `CaptureScheduler` are gone + (2026-09-14, the "still not 60fps" take):** slice 11's ~30Hz was still a `CapturePreviewAsync` PNG + poll — every 1920×1080 full-HD encode+decode cost 35–165ms, so the session budget capped real + cadence at ~20Hz for a 60fps-designed widget. Root fix = frame-driven capture, not faster polling: + the WebView2 renderer now feeds `Windows.Graphics.Capture` directly through a + `CoreWebView2CompositionController` (the mechanism `WebView2CompositionControl` and Flutter's + `webview_windows` use — see `graphics_context.cc` `CreateGraphicsCaptureItemFromVisual`, which + captures the root `surface_` visual). + - **Pipeline (per session):** `CreateCoreWebView2CompositionControllerAsync(WindowHandle)` — the + parent HWND must exist, so `MainWindow.InitWebView2()` moved from the ctor to **Loaded** (was + `InitWebView2(WebViewHostPanel)`; the hidden XAML `WebViewHostPanel` overlay is deleted). + `controller.RootVisualTarget` = a child `ContainerVisual` (`RelativeSizeAdjustment = 1,1`) under a + root `ContainerVisual` (1920×1080, `IsVisible=true`) built on ONE `Windows.UI.Composition.Compositor` + (created once on the UI thread); `Bounds = 0,0,1920,1080`, `BoundsMode=UseRawPixels`, + `ShouldDetectMonitorScaleChanges=false`, `RasterizationScale=1.0`, `IsVisible=true`, + `DefaultBackgroundColor=Transparent`; then `GraphicsCaptureItem.CreateFromVisual(root)`. + Frames pull from a free-threaded `Direct3D11CaptureFramePool` (2 buffers) → + `SoftwareBitmap.CreateCopyFromSurfaceAsync` (BGRA, `BitmapAlphaMode.Straight` — the compositor + blends STRAIGHT alpha, so premultiplied readback would wreck corner anti-aliasing) → per-frame + `FindContentBounds` alpha-bbox on the capture worker → `VideoFrame.CropBounds` stamped → ring up to + 2 fresh + Epoch (reuse; GC lesson slice 8) → dispatcher-coalesced copy (Render priority) into ONE + crop-sized shared `WriteableBitmap`; `PreviewBitmapChanged` raised only on bitmap (re)creation. + Also unpumped: the OBS-side de-throttle flags (slice 11) stay — they fix the hidden-page JS + throttling; the capture path itself no longer polls. + - **Implemented in:** new `Services/WebCaptureFrameSource.cs` (owns item+pool+session — mirrors + `ScreenCaptureFrameSource`, which servers as the copy template) + reworked `Services/WebView2Manager.cs` + (owns controllers, the visual tree, nav/script wiring, preview; internal seam ctor + `(Dispatcher, Func?)` so the tests instantiate zero WinRT; public + ctor `(Dispatcher)`). + - **CoreMessaging DQ recipe (record-once):** the 19041 CsWinRT projection has NO + `DispatcherQueueController.CreateOnCurrentThread()` (CS0117 — only `CreateOnDedicatedThread` + + `FromAbi(IntPtr)`). P/Invoke `coreMessaging.dll!CreateDispatcherQueueController` with struct + `DispatcherQueueOptions { DwSize, ThreadType = 2 (DQTYPE_THREAD_CURRENT), ApartmentType = 2 (DQTAT_COM_STA) }`, + wrap via `DispatcherQueueController.FromAbi(ptr)` (mirror of `CaptureInterop`), only then + `new Compositor()` — all once on the WPF UI thread (the app dispatches Render there). + - **Compile notes (each cost a build cycle):** `Compositor` collides with the repo's OWN + `ytLive.Services.Compositor` namespace → fully-qualify `Windows.UI.Composition.Compositor`; + `CoreWebView2CompositionController` exposes **`Close()`**, not `Dispose()`; `Color` is ambiguous + (`System.Drawing` vs `System.Windows.Media`) → `System.Drawing.Color.Transparent`. + - **Dropped:** `Services/CaptureScheduler.cs` (DELETED — no cadence to schedule), all three + `SetCaptureInterval` hooks in `MainViewModel.Streaming.Operations.cs`, the XAML `WebViewHostPanel`. + `TransparentBackgroundScript` const is unchanged (a test pins it). + - **Test model (Good Dog):** reworked `ytLive.Tests/WebView2ManagerTests.cs` — dropped the 4 + control-size tests + the `CaptureScheduler_Drops…` test; `FindContentBounds` tests moved to + `WebCaptureFrameSource.FindContentBounds`; the ONE integration test + `Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds` drives the internal seam with + `FakeWebSource : IScreenCaptureSource` + a real background-STA `DispatcherPump` (copied from + `ScreenCaptureManagerTests`): register → one crop-sized preview bitmap published → `CropBounds` + + `IsOpaque=false` on `GetLatestFrame` → back-to-back pumps coalesce to latest. Byte assertion compares + the DENSE 2×2 crop (`CropBytes` helper — slices source rows with stride gaps, a contiguous range + spans rows wrongly). + - Full suite **293/293 green, 0 warnings**. Audio untouched. **NOT YET VERIFIED ON DEVICE** — the + composition path needs a real 60fps-widget run (open item; see HANDOFF). Web work commits stay + LOCAL (no push) until the user greenlights. +- **Slice 15 — one frame per deadline SLOT: duplicate-on-lag, never skip (2026-09-14, device takes + ty-1723/1726):** slice 10's freshness choice — a render overrun VANISHES the missed slots from the + file — authored ACCELERATED playback: ~35ms render cost vs the 16.6ms slot, one fresh frame per + 35ms (log wall-1726: `FramePump stall: iteration 33ms (> 2× the 17ms interval): worst render + 33-46ms`), and a 60fps container muxed on arrival → 1723: 697 frames = 11.62s video vs 11.84s + audio; 1726: 163 frames = 2.72s vs 2.93s. The video also ended 0.21–0.24s before the audio ("audio + speeds up, cuts off at the end"). OBS's answer is duplicate-on-lag: `libobs/media-io/video-io.c` + emits one frame per tick and a lagger DUPLICATES the newest frame, counted as "lagged frames due + to rendering lag/stalls" (obs-output.c) — a wall-time hole never exists (docs.obsproject.com/ + backend-design: "If the video frame queue is full, it will duplicate the last frame"). The pump's + submit is now a bounded catch-up: `while (now >= nextTick) { submit; nextTick += intervalTicks; }` + over a `deadlineNow` captured once per iteration — the LATEST composite, fresh on the first missed + slot, repeated (OBS's duplication) for the rest. Duration == wall under any render load, at the + cost of a short judder during a stall. The burst is safe because `Channel.TryWrite` + never blocks (slice 10's queue — the take-9 smear was the BLOCKING pipe-write re-copying a stale buffer + during a long freeze; here each emit is nanoseconds). The burned `_outputIndex` (slice 10 judge) + moved INSIDE the submit loop so every emitted slot carries its own +1 (and the old unconditional + pre-gate bump no longer gaps the sequence on non-submitting fast-render iterations). **Good Dog + test** `Pump_Overrun_Renders_EmitsEverySlot_NotSkipped`: 60fps, resolver sleeps 35ms, asserts ≥0.65 + of the wall slots are emitted (the skip-pump wrote ~1/35ms ≈ 200 in 7s; the slot pump ~415). Full + suite **294/294 green, 0 warnings**. The +0.6s audio-late clap reading on 1726 was confounded by + the 1.1x acceleration — re-measure on device; if a real residual remains it is the audio pipeline. + No push (web/A/V work stays local). +- **Slice 16 — the desktop capture conversion was the bottleneck: fast downscale + + cadence throttle + reuse-distance ring (2026-09-14, device take ty-1742):** slice 15 + fixed pacing but the 1742 desktop layer was still jerky/frozen with a horizontal + tear. Decoded to raw frames and audited: the **desktop band was frozen 21s of 23.35s + (90%)** — ~6.1 content updates/s, freeze runs up to 2.28-2.78s, dup-run max 90 + frames (1.5s). The render stat (`worst render 33-36ms`) was real but MOOT: the + capture CONVERSION was the wall. The monitor delivers at the **240Hz DWM cadence** + (~4.2ms); with one-in-flight conversions and each 2560×1440→1920×1080 + `DownscaleBgra` at ~150ms under load, `LatestFrame` updated a handful of times/s — + the desktop feed inside a 60fps file read ~90% frozen. (Webcam + audio were their + own paths — fine, matching the report.) Three changes, all in + `Services/ScreenCaptureFrameSource.cs` (+ `Services/FrameRingBuffer.cs`): + (1) `DownscaleBgra` is now **integer 8.8 fixed-point, "shift only at the end"** — + the exact two-stage math of `SceneCompositor.Bilinear` (MyMistakes take-4 rule), + dropping double-per-pixel to a row-walk of integer ops (the capture downscale had + stayed the naive float twin of the 258ms disaster); (2) a **10ms conversion floor** + (`MinConvertInterval`) so the 240Hz tail stops queuing ~150ms of serialized + conversion per second — the open edge sits just above the ~60/s the pump can use; + (3) the ring is a **reuse-distance pool** (`FrameRingBuffer`, depth 8, redLine 4): + a buffer is only rewritten ≥4 rents after its last hand-out else a fresh allocation, + replacing the blind round-robin — since `ScreenCaptureManager` keeps + `session.LatestFrame` across conversions and the dispatcher preview copy lags, + "who released the buffer" needs a consumer API that doesn't exist; a reuse + DISTANCE needs no cooperation (the 1742 new-top/old-bottom tear is that + read-under-write, structural now). Telemetry added: a startup.log line every 2s + (`frames/s, conv avg/max ms, skip busy/cadence, ring allocs`) so the device take + can be judged numerically. **Good Dog test** + `Ring_NoLap_ReusesOnlyAfterRedLineRents` (depth 3/redLine 4 exercises the red-line + skip; the 8/4 config settles at 8 buffers and never grows). Full suite **295/295 + green, 0 warnings**. C4 (compositor Epoch-cached downscale / blit-on-change) was + DEFERRED: the slice-15 approved plan assumed a native-res relocation; the + measurement recast it — relocating a ~30ms float downscale to the render thread + just moves the same cost into the slot budget. Re-measure on device; if render + still >16.6ms slots after capture feeds ≤60 real updates/s, add C4. No push. +- **Slice 17 — the OS readback was the real wall: overlapping conversions + monotonic + publish + deeper pool (2026-09-14, device take ty-1824):** slice 16's downscale fix + landed but the desktop was still ~90% frozen on the 1824 take (band 6.8/s updates, max + freeze 4.85s). The new 2s telemetry was decisive: `conv avg 46-50ms max ~61ms` with + `skip busy 37-83` — the **GPU→CPU readback (`CreateCopyFromSurfaceAsync`), not + `DownscaleBgra`**, is the ~47ms wall (240Hz HDR compositing + encoder + 2 capture + devices share the GPU); at one-in-flight that caps the desktop feed at ~17-20 + updates/s, which is the file's whole-frame ~7 content-moments/s. Two facts reshaped + the fix: **(a)** shrinking the pool size does NOT scale the desktop (Microsoft docs: + "If content is larger than the frame, the contents are **clipped**") — readback stays + at native 2560×1440; **(b)** delivery is healthy (60-100 arrivals/s), so the lever is + conversion throughput, not the pool size. Changes in `Services/ScreenCaptureFrameSource.cs`: + conversions overlap up to **MaxConcurrentConversions = 3** (pool deepened to 5 buffers + so in-flight frames fit), each completion publishes ONLY if its Epoch is strictly + newer than the last published (`MonotonicGate` — a slow older completion must never + overwrite a newer LatestFrame), Epoch increments via `Interlocked`, and the + DownscaleBgra row-scratch became per-conversion locals (concurrent callers). The + render side (33-42ms → ~30 unique composites/s) is the NEXT cap after capture speeds + up — that's the C4 slice, queued right after this re-measure. **Good Dog test** + `PublishGate_TryPublish_OnlyStrictlyNewerWins`. Full suite **296/296 green, 0 + warnings**. NOT YET DEVICE-VERIFIED; target: telemetry frames/s jumps ≥ ~30-40 and the + band audit drops below ~50% frozen. No push. +- **Slice 18 — C4: the FramePump blit-on-change composite cache (2026-09-15, take ty-1841 + verdict):** the 1841 take proved capture fixed (band ~20 fresh updates/s, no tears, pacing + clean) but logged a FramePump **stall on EVERY iteration** (`totalMs 21-44`, + `render=full-render split=0 elements=6 dynamic=4`, worst render 166ms startup spike) → + ~22-28 composites/s hard-caps the desktop inside the 60fps file. The SceneGraph split + CANNOT fix this wired: `GetSplitPoint` returns **0** because the live-capture backdrop is + element 0 and must stay dynamic — a baked capture goes stale — so every tick runs the full + re-composite even when no input changed (the compositor's per-layer paste cache already makes + identical frames identical per-element; the waste is re-COMPOSITING the whole 8.3MB frame). + Same shape as OBS (sources cache their surface, the compositor renders on update only — + the pattern this file has cited since the 2026-09-04 paste-cache slice) at the FRAME level. + `FramePump.RenderFull` now wraps both full-render call sites (no-SceneGraph + split=0 + fallback): it computes `BuildFullRenderSignature` (a deterministic hash of the crop/output + dims, the social bar identity + top, and — per element, MIRRORING the compositor's own + resolution — the element ref, layout/visual bits, and the frame each element would resolve + through the SAME resolver seam, using buffer-array identity + Epoch + CropBounds; a changed + frame forces a re-composite, an unchanged one never serves stale bytes) and, when it matches + the last composite, reuses the cache with ONE `Buffer.BlockCopy` (~3ms) instead. The cache + buffer is a **separate long-lived array** — never the scratch pool — because the caller burns + the frame counter and recycles the scratch AFTER `RenderScene` returns; the cache is written + from the rendered scratch BEFORE returning (pre-burn, pre-recycle). Engagement is gated on + the 1:1 config (`SourceRect == Output`, the only deployed tier); the vertical tier's cached + fresh-scale-buffer follow-up stays exactly where it was. Telemetry: internal `CacheHits`/ + `CacheRenders` counters surface as `cache {renders}R/{hits}H` on the 5s stats line, and the + ticks' key is an internal `OutputIndex` (burned counter, stable across one tick's resolver + passes). **Good Dog test** + `FullRenderCache_StaticInputs_RenderOnce_Then_Reuse_UntilInputChanges`: static scene → + exactly ONE composite (renders==1, hits>0, byte-identical content above the burn strip), + then a new frame (new array + monotonic Epoch) invalidates and propagates. Existing pool test + now passes a STABLE scene (production hands `StagedScene`; a fresh scene per tick churns the + element refs inside the signature and hides the cache — the tests that care about cache + behavior must mirror production) and keys its red/blue alternation on `OutputIndex`/frame + count so the tick's two resolver passes (signature + render) don't double-advance a call-count + flip. Full suite **297/297 green, 0 warnings**; verify.sh scope-locked to FramePump.cs + + FramePumpTests.cs + docs. Target next take: `cache` hits dominate on TV holds, `avg render` + drops toward the BlockCopy, stalls vanish, then the band audit + clap re-measure decide push. + No push. +- **Stop ordering matters:** `StopAsync` stops the encoder — since slice 10 it FLUSHES the pending + queue (`Channel.TryComplete` → drain writes the leftovers, closes stdin → EOF → ffmpeg finalizes+exits; + an accepted frame is never lost) — **before** awaiting the loop. The old reverse-order deadlock was + a submit stuck on pipe backpressure; the drain thread owns that wait now, and stopping the encoder + first keeps the pump's own (non-blocking) submits from racing the completed queue as a spurious drop. + `ProcessFailed` self-stops the pump. `Failed` while live flips + `StreamStatus.Error` (minimal). +- **Health stats (ship step 6, shipped 2026-08-13):** `HealthUpdated` is bound to the bottom bar — + `MainViewModel.OnFramePumpHealthUpdated` marshals to the UI thread (the encoder's stderr loop raises on + a background thread) and copies into `CurrentHealth` (the bottom bar's existing `CurrentHealth.*` + bindings). `ResetHealth(status)` zeroes dropped/duration on go-live and on End so stats never linger + from a previous session; bitrate/FPS stay on the tier's targets (`ApplyStreamQuality`). The bar shows + real encoder values since TASK 9 (2026-08-16) wired the reusable stream URL into `_rtmpUrlProvider`. +- **`MainViewModel` owns the resolver** (`ResolveOutputFrame`): `WebcamSceneConfig` → + `CameraManager.GetLatestFrame(DeviceKeyForWebcam(WebcamId))` — **device-keyed** (configs carry the + identity GUID; sessions key on the Windows device id — the old direct-GUID lookup returned null + forever and the webcam was structurally absent from output; take-2 fix 2026-09-01, + `WebcamOutputKeyTests`), `Source { IsLiveCapture, CaptureKey }` → + `ScreenCaptureManager.GetLatestFrame(CaptureKey)` (the new accessor mirroring `CameraManager`), image/ + background → `StaticPixelCache.Get(AssetId)`. `BuildCompositorOptions` rounds the VM's `OutputRect*` + doubles to ints — the vertical 607.5 half-pixel crop rounds to a perfectly-centered **608** (`Math.Round`, + ToEven); `BuildEncoderOptions` fills W×H/FPS/bitrate from the tier once the URL provider yields one. +- **Media source decoder (TASK 21, Increment B — 2026-08-31):** `MediaVideoSource` (`Services/MediaVideoSource.cs`) + decodes a local file to `VideoFrame`s by spawning ffmpeg with + `-f rawvideo -pix_fmt bgra -vf scale=WxH -an` and draining the raw BGRA stdout pipe through the pure + `RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raising `FrameAvailable` per frame. Implements + `IMediaFrameSource` (`Services/IMediaFrameSource.cs`: `Key`, `FrameAvailable`, `Completed`, `StartAsync`/`StopAsync`). + The subprocess sits behind `IDecodeProcess`/`FfmpegDecodeProcess` (binary-stdout mirror of the encoder's + `IEncoderProcess`), and the path comes from `IFfmpegLocator` — codec-agnostic, uses the ffmpeg already shipped, + testable with a fake process. Decode sessions are owned app-wide by `MediaVideoSourceManager` + (`Services/MediaVideoSourceManager.cs`): refcounted by `MediaPath`, `Func` factory + seam (mirror of `ScreenCaptureManager`), `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`, + coalescing each file's frames onto the UI dispatcher onto one shared `WriteableBitmap`. Frames emit as fast as + the pipe produces them. Wired into the live pipeline (slice 1 step 4): `ResolveOutputFrame` reads + `_mediaManager.GetLatestFrame(MediaPath)` for a `MediaSource`, and `Source.DisplaySource` + (`Models/Source.cs`) routes `VideoImageSource` for `MediaSource` so the preview/canvas show decoded video. + Slice 2a shipped the probe seam: `FfmpegFrameRateParser` (pure, `avg_frame_rate=` then + `r_frame_rate=` rational N/N/M, unknown→null) + `IFrameRateProbe`/`FfmpegFrameRateProbe` (derives sibling `ffprobe.exe` + from the located ffmpeg dir, reuses the `IDecodeProcess` seam for the ffprobe subprocess text; null if + ffprobe absent) + `FfmpegLocator.ProbeFileName` now also extracts `ffprobe.exe` from the pinned archive + (conditional — old caches without it just get no pacing). Slice 2b wired pacing: `MediaVideoSource` + takes optional `IFrameRateProbe?` + `Func? delay` (default `Task.Delay`) + seams, probes FPS once in `RunAsync`, and delays by 1/fps after each emitted frame; no probe/unknown → no + pacing (ffmpeg's own pipe backpressure already throttles the decode). Slice 3 added loop control: + `IMediaFrameSource.Looping` (bool); `MediaVideoSource` takes a `Func` process factory (a single + `Process` can't be re-`Start()`ed, so each loop pass creates a fresh decoder) and wraps the decode in a + `do…while (Looping)` — restart on natural EOF instead of raising `Completed`. Production wiring (MainViewModel + media factory): passes `() => new FfmpegDecodeProcess()` as the factory, `FfmpegFrameRateProbe(new FfmpegLocator(), + () => new FfmpegDecodeProcess())` as the probe. + Still open: session acquisition on add/remove (UI picker) + wiring `Source.MediaIsLooping` into + `IMediaFrameSource.Looping` (needs a manager-level per-path loop provider, comes with the picker slice). +- **Social bar on the output (bar bug-fix branch):** the `FramePump` takes an optional + `socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam, re-read **every frame** (so a + mid-stream position flip applies immediately). The strip is pre-rasterized by `Compositor/SocialBarRenderer.cs` + (WPF glue — WPF glue here is fine because the strip is rendered once per config change on the UI thread, + the resulting immutable frame is then composited pure-CPU by `SceneCompositor`), and `SceneCompositor.Render` + blits it **last — above the branding flash** at `socialBarTop` (0 = top, `SourceRectHeight − barHeight` = + bottom) in master space. `MainViewModel` owns the frame (`_socialBarFrame`, rebuilt by `RenderSocialBarFrame` + on load/save/`NotifySocialsChanged`). +- Known consideration: the pump reads the active scene on a background thread while the UI can still edit + it; a concurrent-mutation exception is contained (logged + `Failed` + the pump stops) rather than + crashing. The background thread + video pipeline is the new reality since this step. + +--- + +## Related in [`ai.md`](../../ai.md) + +This file owns the compositor and the pump. The neighbours stay in `ai.md`: + +- **Live encoder + RTMP push** — `IFfmpegEncoder`/`FfmpegEncoder`, `FfmpegArgs`, locator and bundle. +- **Media source** — `MediaVideoSource` (ffmpeg `rawvideo` decode) feeding the pump's resolver. +- **Social bar** and **Native events & alerts** — the overlay strips the compositor blits last. +- **Run** and **Tests** — how to launch in Release and how the suite is run. diff --git a/docs/ai/index.md b/docs/ai/index.md new file mode 100644 index 0000000..a31c524 --- /dev/null +++ b/docs/ai/index.md @@ -0,0 +1,18 @@ +# docs/ai — index + +Subsystems split out of [`ai.md`](../../ai.md) (2026-10-26) so a session pays only for the +subsystem its task touches. The split is a **verbatim move**: history, traps and measurements are +unchanged, and `ai.md` keeps a short stub carrying the load-bearing invariants plus a link here, so +no rule is reachable only from a file you forgot to open. + +Read the file whose row matches the task. There is no required reading order. + +| File | Covers | Read when | +|------|--------|-----------| +| [`frame-pipeline.md`](frame-pipeline.md) | Render budget & measurement (`FrameCadence`, per-element composite timing, scene dump), `SceneCompositor` design + `StripPool`, and `FramePump` from scene snapshot to encoder (pacing, pool/paste caches, backpressure, teardown, health stats) | Touching the compositor, the render loop, per-frame cost, `Services/Encoder/*`, or debugging dropped/juddering frames | + +## Split-out yet to come + +The remaining `ai.md` blocks are being moved in staged units (capture, YouTube/alerts/auth, product, +UI patterns/theming, audio/recording/encoder). Add the row when the file lands; do not create empty +placeholders. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..06c313e --- /dev/null +++ b/docs/index.md @@ -0,0 +1,11 @@ +# docs — index + +Reference material that is too large for [`ai.md`](../ai.md)'s front half. Nothing here is +mandatory reading on session start; open the file whose subsystem your task touches. + +| Path | Purpose | +|------|---------| +| [`ai/`](ai/index.md) | Split-out subsystems of the AI guide — each started life verbatim inside `ai.md`, which keeps a short invariant stub pointing here | +| `ytLlive-preview.jpg` | Preview screenshot used by [`README.md`](../README.md) | + +See [`schema.md`](../schema.md) for the memory-map conventions. diff --git a/schema.md b/schema.md index a9d9486..fb0fb64 100644 --- a/schema.md +++ b/schema.md @@ -24,6 +24,7 @@ that every other memory file follows. Read `ai.md` first; this file explains | `MyMistakes.md` | Per-task failure log **+ RECIPES REGISTRY** (derived-solution rule): one-off solutions / how-tos get recorded here the same session, then grepp'd before ever re-deriving them (see `AGENTS.md` 🔬) | grep first on any "did this before?" wall | | `schema.md` | This file: the conventions below | when in doubt | | `/index.md` | Per-directory map (progressive disclosure): what lives there + links | when diving into code | +| `docs/ai/*.md` | **Split-out subsystems** of `ai.md` — verbatim moves (from 2026-10-26); `ai.md` keeps a short invariant stub + link, and [`docs/ai/index.md`](docs/ai/index.md) routes | when the task touches a moved subsystem | | `Views/` | Reserved for Views; currently empty | — | ## Conventions