LlamaCasty/docs/ai/frame-pipeline.md

Frame pipeline — compositor, render budget, measurement, and the live pump

Split out of ai.md (2026-10-26) so a session reads only what its task touches; ai.md keeps a short invariant stub for this file. Directory map: index.md.

This is the subsystem that turns scenes into the encoder's frames: the CPU compositor that renders the master, the budget it must fit inside, the instrumentation that measures it, and the FramePump that paces frames into ffmpeg. Everything here is current architecture plus the measured history that produced it; superseded designs are marked inline.

---

Budget, measurement & the compositor

Frame cadence — how a dropped frame became measurable (UNIT 0, 2026-10-05)

The problem this exists to solve. The recorded file cannot distinguish a dropped camera frame from a padded repeat. -f rawvideo carries no per-frame timestamps — ffmpeg assigns pts from frame order at the DECLARED fps — so a webcam delivering 25 of its 30 frames still muxes a full-length, correctly-timed 60fps file. Measured on take ty-20261005-1001-821-0001.mp4 (85.6s, 1080p60): the lower-right webcam region held 25.6 distinct images/s against the camera's 30fps, with 1,153 normal 33ms intervals, 281 gaps of 50–100ms, and 96 bunched 17ms gaps — roughly one hitch in five, which is what makes a face judder while the file looks flawless. (The "webcam renders at ~18fps" figure that used to sit in this map was WRONG: it came from a different session and counted every newly rendered layer, not webcam arrivals. Corrected 2026-10-05.)

Helpers/FrameCadence.cs — one pure counter, three feeds. Add(timestamp) on any shared monotonic clock; Summarize() → frames, rate (gaps ÷ span, NOT frames ÷ span — N frames span N-1 periods), gaps, hitches (default threshold 50ms ≈ 1.5× a 30fps period), longest and average gap; Describe(label) gives every producer the same one-line shape so two lines can be read against each other. An out-of-order stamp counts as a frame but produces no gap AND does not become the reference for the next interval — otherwise one bad stamp invents a multi-second "hitch" twice.

Three measurement points, and what each separates:

Line	Fed by	Answers
Camera cadence WxH: arrivals … | camera …	MediaCaptureFrameSource	device/USB/driver cadence (MediaFrame.SystemRelativeTime, QPC domain, nullable) vs what the reader handed us. TryAcquireLatestFrame keeps only the LATEST frame, so falling behind is silent by design — this is where that shows
webcam … on the pump's stats line	FramePump.TimedResolver	distinct webcam VideoFrame.Epoch values that reached a composite — what actually got into the output
frames per 5s from R renders	FramePump	submits vs REAL composites — the duplicate-on-lag ratio, the only honest reading of how much of the stream is a repeat

Why the stats-line denominators changed (the actual bug). Every average was divided by statFrames, the number of frames SUBMITTED — which deadline padding always fills to the full 60fps target. A pump fed one webcam frame per 200ms logged "300/300 frames per 5s, avg render 12.4ms" while producing 60 distinct composites: the render average was diluted ~5× and the loss was invisible. Render/resolve/wait are now divided by statRenders; submit stays per-frame (it really happens per submit). Regression test: FramePumpTests.Stats_ReportRealRendersAndDistinctWebcamFrames_NotJustPaddedSubmits.

Reading the three together: camera 30/s steady + arrivals 25/s = WE dropped it (reader or compositor). Ragged in both = the device, the USB link, or the driver. Pump webcam below arrivals = the compositor missed frames the reader had already delivered.

✅ WHAT THE FIRST REAL READING SAID (2026-10-05) — the camera was never the problem. ty-20261005-1201-556-0001.mp4 + its log: the C920 delivers 30.0/s with ZERO gaps ≥50ms, and the device's own stamps are identical, so the camera, USB link, and driver are all clean. The pump did 34 composites per 5s (6.8/s) at ~141ms each (worst 294ms), and the file holds 6.8 distinct webcam images/s at 88.7% duplicates with no arrival hitches at all. The judder is entirely us holding one face for 140ms and padding the rest of a 60fps container. resolve is 0.1ms, so it is pure pixel work in the composite, not frame lookup. Two consequences, both load-bearing:

• 720p is a sharpness win, NOT a smoothness win — it triples the bytes stretched per render on a path already 4.4× over budget. Schedule the exclusive lock AFTER the compositor work or it makes the judder measurably worse.
• The 10:01 take hit 25.6 images/s and the 12:01 take 6.8, same scene shape (full-render split=0 elements=6 dynamic=5), same 640×480 camera. Composites went 29–34ms → 143–155ms between sessions with nothing logged explaining it. That comparison was made by EYE on two takes because the log carried no per-element breakdown and no scene dump — both now exist (see "Per-element composite timing + scene dump" below); naming the culprit is a reading, not a guess, and guessing is explicitly out of bounds. Superseded 2026-10-16: "same scene shape" was an eye comparison of takes whose geometry was never logged, and the next bad take still on disk is the vertical tier (below). A Release 1920×1080 take now measures 1.0ms with no element over 0.4ms — see "Per-element composite timing + scene dump" → what the first instrumented take said.

Counter subtlety, so it is not re-derived: the pump's webcam line read 13/s while the file held 6.8/s. Not a bug — the counter stamps a frame when the resolver FIRST sees it, and one composite takes 141ms, so ~4 camera frames arrive during it (signature pass stamps one epoch, the actual draw uses a later one). The file is the truth; the pump counter is a pipeline-health signal and overcounts by design.

Still unmeasured: the camera is stuck at the C920 default 640×480 YUY2 @30 because all 185 SetMediaStreamPropertiesAsync attempts are refused under SharedReadOnly, so 720p MJPG is available — but see above, it is a quality win only.

Per-element composite timing + scene dump (UNIT 0 follow-up, 2026-10-16 — MEASUREMENT ONLY)

The gap the cadence work left open is now closed by instrumentation, and closed by nothing else: the 141ms take was never compared to the 30ms take because no line in the log said which element cost what, and no line said what the scene WAS. Both now exist. No fix has been attempted, and none is allowed until a real reading names the culprit.

• Services/Compositor/CompositeTimings.cs — a collector owned by the pump, threaded through SceneCompositor.Render / CompositeLayers / BakeStaticBase / BlitBakedElement / BlitCachedLayer (and SceneGraph.GetBakedBase → bake path) as an optional last parameter, so every other caller (thumbnails, tests, transition preview) is untouched and costs nothing. Add(stage, ticks) accumulates; Describe(renders, minMs = 0.1) renders the averaged breakdown slowest-first, dropping sub-0.1ms noise.
• Stage names (stable, greppable): backdrop (prefill + live capture + background art + their resolves), base (the cached-bake BlockCopy), flash, social, ticker, zoom, scale, overlay (the fully-static path's per-frame overlay copies), bake backdrop, bake overlays, and one entry per element as #<z-index> <name> — the index IS the z-order, so the log names the layer by both position and name. Bake-pass elements are prefixed bake #… so a one-off rebake cannot masquerade as a per-composite cost.
• #<i> <name> raster nests inside its element's total and is the paste-cache rasterize miss (BlitCachedLayer's raster == null branch) — the direct answer to "is one element re-sampling every tick, or is the paste itself slow". A stage's time includes that stage's frame resolve; resolve is separately totalled in the existing stats line.
• FramePump parts (N renders/5s): … — logged once per 5s window (same window as FramePump stats), averaged per render so the numbers compare directly with avg render, then the collector is reset. No parts line, or a short one, is a reading too: the fully-static shortcut and C4 cache hits composite no elements (cache 92R/130H beside a missing parts line is expected).
• FramePump scene dump: '<name>' elements=N split=S followed by one line per element ([i] name kind=… type=… vis=… rect=x,y WxH op=… clip=…), logged when the pump first runs and whenever the active scene changes. This is the record the 10:01/12:01 comparison never had.
• Gate test: ytLive.Tests/CompositeTimingTests.cs (one [Fact]) — a REAL composite of a full-canvas element plus a 64×64 one must rank the full-canvas element first, carry the raster sub-stage and the fixed stages, and the dump must name both elements with z-index/type/rect and echo the split.
• scripts/webcam-cadence.py — the other half of a reading: what the FILE holds (distinct webcam images/s + hold histogram), beside what the pump says it did. Recipe and its three traps (H.264 re-encoding, the round-clip crop, scene switches): MyMistakes.md → "Measuring webcam CAMERA cadence in a recording".

Read takes in Release. The creator launches dotnet run -c Release; Debug pixel paths are several× slower, so a Debug number is not a reading of this pipeline (see environment.md → Run).

Derivative check (AGENTS.md): this is not invented — OBS's own answer to "which source is slow" is the Source Profiler, which reports the average execution time of each source's tick and render functions (docs.obsproject.com → Source Profiler, per-source timers via util/profiler.h). Same shape here: one timer per element per composite, reported as an average, plus the stage split. The #<i> <name> raster sub-stage is the extra this case needs — it separates "re-sampled every tick" from "slow paste" without another session.

✅ What the first instrumented take said (2026-10-16, Release) — the judder DID NOT reproduce

Take ty-20261016-1841-559-0002.mp4 (174.5s, 1920×1080@60, Release build e95154d9 — a recording, not preview; scene Live, switched to Chat at 28s, back to Live at 44s) and its log window:

Reading	This take
composites	301 per 5s = 60/s, every slot; 0 dropped, 0 stalls after warm-up
avg render	1.0–1.1ms, worst 2.5ms — ~7% of the 16.67ms budget
parts	#5 HD Pro Webcam C920 0.4ms, #5 … raster 0.2ms, backdrop 0.2ms — the other four elements are under the 0.1ms print floor
C4 full-composite cache	151R/150H per window — half the ticks were a whole-frame BlockCopy
webcam at the resolver	30.0/s, longest gap 50.0ms (one extra 60Hz frame = sampling phase, not a drop)
the file itself	29.6 distinct webcam images/s, 82% of images held exactly 2 frames, longest hold 267ms (scripts/webcam-cadence.py)

Nothing in this take is over budget and no element is expensive — there is no culprit here to fix. The two stalls in the take are not per-composite cost: 124ms on the very first render (warm-up) and 53ms on the switch into Chat (a rebake).

Scale of the disagreement: the historic slow readings are 141ms (Oct 5 12:01) and 45ms (Oct 9 19:48) — 40–140× the ~1ms this take needed per composite. Three facts about those takes, all read off the same log:

• Oct 9 19:48 (ty-20261009-1948-121-0000-2.mp4) was the VERTICAL tier, not a 16:9 one: FramePump started (1080×1920 @ 60 fps) scene='Live' comp=608x1080 — the 607×1080 centre crop bilinear-upscaled to 1080×1920 (2M px), 92 renders/5s, avg render 45.0ms, worst 278ms, 92 stalls, cache 92R/0H. The file on disk is 11.6s and every frame is identical (a static desktop), so it cannot be read for cadence — only its cost.
• Oct 5 12:01 (…1201-556-0001.mp4, no longer on disk) logged FramePump started (1920×1080 @ 60 fps) with NO scene=/comp= suffix — that suffix did not exist yet, so its source rect was never recorded and the map's "21:9 → 1920×803" explanation stays a hypothesis.
• A take can latch mid-take. Oct 5 12:29 goes 1.5ms (avg render) → 3.0ms → 45.2ms within 15 seconds and stays there (cache 10R/0H, then 98R/0H). So the slow state is not a fixed property of a scene geometry. The consistent signature of every slow window is cache xR/**0H**: the C4 whole-frame cache never reused once, so every tick composited. The healthy take shows the other stable state (composite 1ms → every second tick is byte-identical → whole-frame reuse). The latch is the thing to test on a reproducing take — it is a hypothesis, not a reading; the parts line is what will decide it.

What this does NOT say. This take is the 1:1 master tier and its scene logs full-render split=0 elements=6 dynamic=4 (the bad-period takes logged dynamic=5), so it never exercises the failing geometry. "The healthy path is healthy" is not "the judder is gone". Reproducing it needs a Release take in the failing configuration (vertical tier first, then the 21:9 display config) — that take's parts line names the stage, and only then is a fix unit allowed.

Reading the parts line's denominators: like avg render, it is divided by statRenders, which counts C4 REUSES as well as real composites (the 301 above = 151 real + 150 whole-frame copies). So in a healthy 30fps-camera take both averages are ~2× optimistic; the cache NR/NH pair on the same line says how much of the window was real composite work. CompositeTimings.Describe drops stages under 0.1ms by default, and a missing or short parts line is itself a reading — a fully-static window (every tick a C4 hit) composites no elements at all.

Render budget — getting a composite inside 16.67ms (UNITs 4-5, 2026-10-05)

The pump emits one encoder frame per 16.67ms slot and pads with a duplicate when a composite overruns, so the whole problem is "render + submit + untimed work ≤ 16.67ms, ideally ≤12ms to leave headroom". FramePump now reports per-stage timings (social/ticker/flash/acquireScratch) so a slow stage names itself, and logs the scene name + compositor source rect at pump start.

Why a 21:9 monitor cost 141ms and a 16:9 one 33ms — the fast path is EXACTLY 1920×1080. A 21:9 display shrinks the source rect to 1920×803 ("fit", not "fill"), so the opaque-backdrop 1:1 row walk is skipped, the base is prefilled with opaque black (~2M byte writes), and the backdrop is resampled per-pixel (~1.54M pixels with an alpha branch). Every recorded A/B lines up: 21:9 → 141ms, 16:9 → 30-33ms, 2560×1440 (fits to 1920×1080 exactly) → fast path. Corrected 2026-10-16 — the "never Debug-vs-Release" half of that is withdrawn. A Release 1920×1080 take measures 1.0–1.1ms where an Oct 9 take of the same tier measured 10.4ms, and Build <hash> carries no build configuration, so the config of every pre-2026-10-16 take is UNKNOWN: no historic render number may be compared against a Release take. The 21:9 explanation itself stays a hypothesis — the 141ms take's start line predates the scene=/comp= suffix, so its source rect was never recorded. A take can also latch mid-take (see the instrumentation section).

Unit 4 — cheap wins, all pixel-identical:

• Opaque 1:1 fast paths expanded to the whole-canvas opaque case (a BlockCopy, not a per-pixel alpha branch).
• Prefill only the uncovered rows. When the backdrop does not cover the canvas, the letterbox bands are filled instead of the whole buffer; with no backdrop at all it still fills everything.
• Per-column LUT for the scaled blit (BlitContentRaw): the source column, its neighbour index and the 8.8 weights are computed once per column instead of a divide + clamp per pixel. The table comes from ArrayPool<float>.Shared and is returned in a finally — a single static buffer is a data race the moment strips run in parallel (see below).

Unit 5 — strip-parallel composite: Helpers/StripPool.cs. One composite splits into four horizontal row strips; the calling thread takes a strip too, so four workers never means five-way oversubscription. Below 64 rows the fan-out costs more than it saves and runs inline. Each strip must write only its own rows — the blits already partition by row, which is why the output stays byte-identical.

The demotion must come from DEDICATED threads, not the shared pool. The obvious Thread.CurrentThread.Priority = BelowNormal inside Parallel.For is wrong: Parallel.For runs on shared ThreadPool threads, so the demotion leaks into every unrelated work item that thread later picks up. Research (https://stackoverflow.com/questions/524483/net-backgroundworker-objects-thread-priority — "you have next to no control of which thread pool thread will actually run your task"; https://stackoverflow.com/questions/1578599/c-sharp-lower-thread-priority-in-thread-pool — "You shouldn't mess with the priority of the regular ThreadPool, since you aren't the only consumer ... write your own producer/consumer queue, using your own Thread(s) as the consumer(s) - setting the thread priority yourself") says exactly that, and a real failed attempt confirmed it: a full test run after demoting the pool threads broke an unrelated audio-mix timing test. StripPool IS that producer/consumer queue — four Threads with Priority = BelowNormal, IsBackground = true, started lazily on first use, each fed a Queue<Strip> + SemaphoreSlim. Four is the measured sweet spot: fewer leaves render over budget, more steals cycles from the game and the encoder and made render worse.

Gate test: SceneCompositorTests.StripParallel_RepeatedRenders_AreByteIdentical — the same scene (scaled round webcam + mirrored image + master-sized flash over a short backdrop, 240 rows so all three fanned-out paths run) must come out byte-for-byte identical across 8 renders, and must differ from a backdrop-only render so it cannot pass vacuously. It earned its keep immediately: it caught a loop-variable capture (new Thread(() => Worker(i)) where i was Workers by the time the thread ran) that crashed the test host on first run.

Scene compositor (TASK 4 ship step 1 — shipped 2026-08-10, plan in TASKS.md)

The encoder needs the master 1920×1080 frame without the preview's editing chrome (SelectionOverlay, DimRects, output-rect outline, badge, placeholder). WPF's RenderTargetBitmap is software-rendered and captures the visual tree including chrome, so the preview can't be captured — the output is a second, parallel software compositor over the VideoFrame (BGRA8) seam, and the XAML preview (MainWindow.xaml CanvasGrid + element DataTemplate) is the rendering contract it replicates. Two renderers must agree: geometry, UniformToFill cover-crop, round clip, mirror, border, z-order. Preview stays XAML (editing view); the compositor is the output view.

• The DimRects shading is itself preview chrome, so it renders as a SIBLING of CanvasGrid (above it, below the ZoomBox), never a descendant (2026-10-08, creator ruling): the zoom box's VisualBrush samples all of CanvasGrid, and if the dims were inside it the box would show the dim shading — especially after the box is dragged away and the fixed capture slides under the dims.

• Composite the active tier's crop at crop size (CompositorOptions {SourceRectX/Y/W/H, OutputWidth, OutputHeight}, fed from MainViewModel.OutputRect*): 16:9 = full 1920×1080; the vertical 9:16 tier = centered 608×1080 crop. FFmpeg's scale filter resizes the raw BGRA frame to the selected output tier after it leaves the frame pump; setsar=1 keeps square pixels.

• CPU posture: with FFmpeg as a subprocess the master crosses a CPU readback to the pipe every frame anyway, so GPU compositing buys little at this layer count (2-3 live layers; static layers pre-composite once into a cached base). GPU effort belongs to NVENC (the encoder), not composition; if composition grows (wipes, filters, many layers), a D3D11 compositor can replace this one behind the same seam — the CPU master buffer stays the contract.

• Every global overlay must be composited on EVERY render path (bug found 2026-09-26). The frame pump has three exits: RenderFull (no graph / no static base), CompositeLayers (static base

  • dynamic tail), and the fully-static shortcut, which stretches the cached bake and returned it directly. That third path silently discarded every per-frame overlay — the social bar, the alert ticker and now the brand flash. A creator whose scene was one static background saw none of them, and the flash looked "preview-only forever" for a reason unrelated to the tier. SceneCompositor.Overlay is therefore internal static (it copies before blitting, so the cached bake is never mutated) and the shortcut now composites overlays onto a copy before stretching. When adding an overlay, grep for all three paths — a fourth exit that forgets it fails the same silent way.

• VideoFrame buffers are recycled by some producers, and Epoch is not optional. A producer that hands out the same array across frames MUST bump Epoch, or FramePump's buffer-identity paste cache false-hits and freezes stale content (the take-11 class of bug). The C4 cache signature mixes Epoch, so a per-frame overlay naturally invalidates it. BrandFlashPresenter reuses one 8MB master buffer and stamps Epoch per read for exactly this reason — allocating a fresh 8MB frame per tick is not an option at 1080p30.

• Dev-only: two instances side by side (Helpers/InstanceProfile.cs). Pre-1.0 the creator runs one instance to stream and another to screen-capture it. Nothing prevented a second instance (no mutex, no port); what broke it was shared state — the whole-scene read/write layout DB (clobber), Chromium's exclusive lock on the WebView2 user data folder (second instance won't start), the auth token store (a test instance overwrites the real YouTube sign-in), and startup.log. Set YTLIVE_INSTANCE=<id> and that process gets a private root at %APPDATA%\ytLlive\instances\<id>\ for the DB, the auth file, the log and the WebView2 folder. Recording folder and the ffmpeg tools\ cache stay shared on purpose; global hotkeys stay un-namespaced (Windows refusing the second RegisterHotKey is the correct answer). The entire implementation is inside #if DEBUG — a Release build compiles to DataRoot => DefaultRoot and WebViewDataFolder => null, and the call sites are unconditional so Release cannot drift. Proof + harness: ytLive.Tests/InstanceIsolationTests.cs. ⚠️ Grepping the binary for YTLIVE_INSTANCE proves nothing — a const is inlined and appears in neither build; check the InstanceVariable field in metadata instead.

• Branding flash is composited by the output path too (it's on the live output, per Monetization), passed in as a pre-rendered VideoFrame? — the compositor core stays pure byte-math, no WPF. Likely a bundled asset rather than runtime text rendering (deterministic, no font/layout risk).

• Frame sources are injected via a Func<SceneElement, VideoFrame?> resolver (SceneCompositor.Render(scene, frameFor, flashFrame, options)) — the caller maps each element to its frame (webcam → DeviceId, image → AssetId via StaticPixelCache, backdrop → CaptureKey), so the compositor is pure, WPF-free, and hermetic to test. The capture managers wire into that resolver in the encoder step, not the compositor step. The master buffer (the compositor's return value) is the seam a future D3D11 compositor would honor identically.

---

The live pump — frame producer to encoder

Live frame pipeline (TASK 4 ship step 5 — shipped 2026-08-12, health stats 2026-08-13, plan in TASKS.md)

The FramePump (Services/Encoder/) is the live frame producer: while live it snapshots the active scene each tick, resolves every element to its latest frame, composites it into the tier's output frame, and paces frames into the encoder at the tier's FPS. rawvideo is stamped by ARRIVAL: ffmpeg assigns pts from frame order at the declared fps — supply rate = output speed. A starved producer (pump < fps) ships a time-lapse, truncated file with NO error (take-2 lesson, 2026-09-01); the pump therefore logs FramePump stats: n/target frames per 5s from R renders, avg render Xms (resolve R2), avg submit Yms so a slow stage names itself (the frames … from R renders pair and the render-count denominators arrived with UNIT 0 — see "Frame cadence" above; before that every average was divided by the SUBMITTED count, which deadline padding always fills to the full target, so a pump fed one webcam frame per 200ms still logged "300/300 frames" and its render average was diluted ~an order of magnitude) — and since 2026-09-04 so does the BUILD: every build gets a GUID stamped into Helpers/BuildStamp (csproj GenerateBuildStamp target, fresh per compile — no lying incremental build), logged at startup ("Build xxxxxxxx (compiled ...)"). Take 6 (stamp-less, ambiguous): render 35-41ms with slice 3 built or not unknown — attribution is why the stamp exists; takes 7+ are readable. If the breakdown says resolve dominates, the suspects are pre-cached chat/config thrash or the web frame; if blit dominates, the general-path element sizes. Pattern — everything is a constructor-injected seam: Func<Scene?>, Func<SceneElement, VideoFrame?> resolver, Func<CompositorOptions>, Func<EncoderOptions?>, Func<IFfmpegEncoder>, Action<string> log, and an injectable pacing delay (default Task.Delay; tests inject Task.Yield). The pump is free of WPF and of the capture managers.

• StartAsync never throws — the VM fires-and-forgets it from the sync command handler; failures log
  • surface via the Failed event. BuildEncoderOptions now always returns options (TASK 18) with StreamEnabled/RecordEnabled/RecordPath baked from the pills + _activeRecordPath; the pump skips the encoder only when options == null, i.e. "no output configured". MainViewModel._rtmpUrlProvider is that seam — a Func<string?> that now yields the reusable stream's ingest URL (TASK 9, shipped 2026-08-16): loaded from the LayoutStore cache at startup and set fresh by PrepareAndStartLiveAsync before the pump starts. The pump reads the URL once at startup, which is why go-live ensures the stream BEFORE StartAsync.
• Deadline pacing + row-blit render (take-3 starvation fix, 2026-09-03): take 3 recorded 30s into a 2.1s file and the 5s stats line named the cause in one number — 17/300 frames per 5s, avg render 258.1ms, avg submit 1.5ms. Two defects, both fixed: (1) the pump slept the FULL interval after each render, so period = render + interval — OBS's video_thread (libobs/media-io/video-io.c) pattern replaces it: absolute nextTick += intervalTicks deadline, sleep only the remainder, and on overrun skip the wait. (NOTE, slice 9: the original "skip the missed ticks (rebase)" half of this fix was wrong — see the slice 9 correction below.) (2) the compositor did per-pixel float sampling + Math.Round blending over all 2.07M master pixels (backdrop) and scanned the whole destination per overlay (a 64px social-bar strip cost 2M iterations). SceneCompositor now follows the libyuv pattern (BSD-3, chromium.googlesource.com/libyuv/libyuv — cited per the derivative-work rule): 1:1 aligned blits take a row-walk fast path (bilinear at scale 1 + integer offset is the identity) with per-pixel alpha branch and integer fixed-point blend; BlitOverlay clips to the intersection rect; a full-cover live backdrop skips the opaque-black pre-fill. The general per-pixel path handled scaled/round/mirrored elements. Test: Pump_Paces_To_The_Deadline_Compensating_Render_Cost records the requested wait via the pacing seam (a seam fake must genuinely await — a synchronous completed task runs the whole pump loop on StartAsync's continuation and hangs the run; MyMistakes).
• Slice 2 of the render fix (2026-09-04, take 4): the pacing held (file no longer truncated at loop level) but render stayed at 58.9ms — the per-pixel row walk was still 2M managed iterations and every tick allocated a fresh 8.3MB master buffer. Three changes: (1) VideoFrame.IsOpaque — a producer-contract flag (only the screen-capture and webcam paths set it; DWM/MediaCapture fill alpha 255 by contract); a full-canvas, aligned, opacity-1 blit of an opaque frame is now ONE Buffer.BlockCopy (~1.5ms) instead of the loop, and the black pre-fill is skipped when the backdrop covers. (2) integer fixed-point bilinear in the general BlitContent path (row-hoisted invariants, no divisions, no Math.Round) — within ±1 of the float reference, inside the ±2 test tolerance. (3) scratch pool in the pump — AcquireScratch/ReleaseScratch recycle the master buffer (max 4, keyed by length, owned-by-reference so bake-cache/social-bar/static-art arrays can never be captured); release happens strictly AFTER SubmitFrameAsync returns (the write to stdin copies), and the free-list Contains guard makes the transition Cut path safe (BlendFrame returns the to-frame itself, aliasing the scratch). The per-tick fromScene render in the transition branch was dead weight (BlendFrame uses TransitionService.FromFrame captured at Start, never the pump's) — removed, and the pump's fromSceneProvider seam + MainViewModel call site went with it. Tests: Pump_Pools_ScratchBuffers_Across_Frames_Without_Stale_Pixels (the ONE: alternating backdrop colors pin every frame's content, repeated backing-array identity proves the pool recycles), Composite_OpaqueFullCover_Backdrop_CopiesEveryPixel_Into_Scratch (memcpy branch + scratch sentinel). FakeEncoder.SubmitFrameAsync now snapshots bytes like the real stdin write — holding the reference would race legitimate recycling. Take 5 must show avg render ≤ ~10ms, ≈300/300. Vertical/output scaling moved to bundled FFmpeg (2026-11-04): the C# compositor no longer enlarges the 608×1080 crop to 1080×1920 on every frame. The bundled FFmpeg scale filter processed 300 synthetic BGRA frames in 0.447s on this machine; that is a scaler-only microbenchmark, not a live-stream result.
• Slice 3 — chat raster-on-change (2026-09-04, take 5): take 5 measured 138/300, avg render 25.5ms — the per-tick blits were cheap now, but ResolveOutputFrame → RenderChatBox ran a FULL WPF raster (ChatBoxRenderer: FormattedText + RenderTargetBitmap + CopyPixels + channel swap) EVERY tick whenever the message buffer was non-empty — and the buffer survives between sessions, so even a signed-out record-only take paid it. Established answer, same as OBS text sources: raster on change, blit the cache every tick. ChatOverlayLayer now keeps a content version (Messages.CollectionChanged → _contentVersion++) plus a config key (box size + all Chat* props); RenderFrame returns the cached VideoFrame by identity until either changes (the compositor only ever reads a cached frame). Test: ChatOverlayLayerCacheTests (RealApp, real renderer — Same() for unchanged inputs, NotSame() on message/config change, null on empty). Accepted cost pending take 6: one slow tick (~15-25ms) per arriving message; if chat-burst frame loss shows up, the next slice moves the re-render off-tick (debounced, dispatcher-side).
• TASK 3.17 — static TextOverlay raster-on-change: TextOverlayRenderer.UpdatePreview runs WPF FormattedText + RenderTargetBitmap only on the dispatcher when text, formatting, or box size changes. It publishes one immutable straight-alpha BGRA VideoFrame and a WriteableBitmap copied from those same pixels; preview and output therefore share the raster. FramePump calls only RenderFrame (a concurrent cache read), never WPF. Text is a static Source, so its raster participates in the static bake or above-split path; property changes invalidate the bake, and the full-render signature includes the resolved text frame. Its premultiplied WPF readback is unpremultiplied before entering VideoFrame, whose compositor contract is straight-alpha BGRA.
• Slice 5 — paste cache for non-opaque layers (2026-09-04, take 7/8 data): the render/resolve split (added in the stamp commit) finally named the last thing honestly: resolve ≈ 0 but render 26-27ms (124-135/300, ≈2.2x) — the chat fix HAD worked; the remaining cost was the compositor re-rasterizing EVERY layer per tick even when its pixels never changed (this creator's Live scene: chat 159k + web widget 271k + image 95k + webcam 156k px ≈ 680k samples @ ~38ns each). OBS's actual shape: sources cache their surface, the compositor pastes. SceneCompositor now routes non-opaque layers through BlitCachedLayer: a layer rasterizes ONCE into an element-space, TRANSPARENT-based frame keyed by (source-array identity, source W×H, ceil'd dst rect, round, mirror), then every later tick PASTES it (integer position, row alpha-blend, opacity applied at paste). Correction (2026-09-12, the web-widget "black box"): the raster builds on a TRANSPARENT base, but the sampler's PARTIAL-alpha branch used the opaque-dst blend — it premultiplied the color into RGB and forced alpha=255. A translucent widget pixel then pasted as opaque darkened ink (the box in recordings) while the raw-bitmap preview stayed correct. BlitContentRaw now takes transparentDst (the raster call passes true and writes straight color + straight alpha; the paste rows do the source-over). Master paths are bit-identical. Guard: PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk. Producers hand out fresh immutable arrays, so array-identity keys can never serve stale content; dict bounded at 48, cleared wholesale on overflow. Only changing content (webcam device frames, web capture ticks, chat messages) resamples; the opaque backdrop keeps its memcpy path. Position/opacity drags are now near-free (no resample — paste params, not cache keys). ONE integration test: PasteCache_RepeatRender_IsByteIdentical_And_ContentChangePropagates (raster-vs-paste byte equality incl. round-clip margins + new-array propagation); the whole existing pixel suite guards sampler semantics. Take 9 must show ≈300/300, avg render ≤ ~8ms; if it lands there, the recording saga closes. Follow-ups unchanged: vertical-tier scale alloc, debounced chat re-render under message bursts.
• Slice 6 — the sleep quantum WAS the ceiling (take 9, 2026-09-04): period measured ~37ms while work (render+submit) was ~25 — the missing ~12ms is Task.Delay rounding EVERY request up to the Windows clock tick (documented ~15.6ms default; learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.task.delay). A frame finishing 3ms early requests a 3ms wait and sleeps a full 15.6 — capping the producer at ~27fps NO MATTER how fast the compositor ran. This is why takes 7→9 showed zero playback change despite real render wins: the sleep floor dominated everything above it. Established fix (game-loop canon — stackoverflow.com/questions/5441464; timeBeginPeriod — learn.microsoft.com/en-us/windows/win32/api/timeapi/nf-timeapi-timebeginperiod): timeBeginPeriod(1) for the pump's lifetime (paired timeEndPeriod in the finally), sleep only the BULK of the remainder (request minus 2ms), Thread.SpinWait the last ~2ms across the deadline; blown deadlines still rebase. (SUPERSEDED in part: slice 9 deleted the rebase — deadlines only advance; slice 10 replaced the burst re-write with one fresh composite per iteration.) Stats gained avg wait Xms so render+submit+wait must ≈ period — the accounting is closed, no stage can hide again. Same slice: the webcam's IsOpaque paste-cache bypass was removed (it re-sampled ~156k px every tick even between identical device frames; the cached paste beats the sampler on hits and costs the same on misses). Take 10 verdict: ≈300/300 honest fps — if short, the wait/render split names the remaining term with no ambiguity left.
• Slice 7 — the loop was ON the UI thread the whole time (take-8 numbers, 2026-09-04): with the wait finally measured, take 8 (59a02a5b) showed the impossible pair — render 22ms, wait 10ms against a 16.7ms deadline: a blown deadline rebases with NO wait. The wait was the producer queued behind the live preview on the dispatcher: PumpAsync starts from a UI command handler and its await continuations inherit the UI SynchronizationContext, so the "WPF-free" compositor actually rendered on the UI thread and ran only when WPF let it. OBS runs its media loops on dedicated threads for exactly this reason. Fix: _pumpTask = Task.Run(() => PumpAsync(...)) (no sync context inside → continuations stay on the pool). Side effects handled, not ignored: StaticPixelCache.Get now locks (pool miss-decodes race UI callers); ChatOverlayLayer.RenderFrame checks its cache off-thread but MARSHALS the rare raster miss to the dispatcher (DrawingVisual/ RenderTargetBitmap are UI-thread objects) and re-validates there; pump events already marshalled. Also: GCLatencyMode.SustainedLowLatency for the pump's life, stats gained worst render Xms (spike visibility — bimodal averages hid them), webcam now routes through the paste cache (bypass re-sampled 156k px even between identical device frames). Tests: Pump_Produces_OffTheStartingContext (the ONE — inline-pumping SyncContext proves continuations never return to the starting thread) + 70/70. Take 9 verdict: wait ≈ true remainder (period → 16.7, n → ~300); worst render names any remaining raster-miss spikes; render+submit+wait still == period — the accounting holds.
• Slice 8 — capture buffer ring + paste-cache Epoch (take 11, 2026-09-04): take 11 validated the architecture — typical frames now land work ~10ms + wait ~6.8ms = 16.7, exactly the deadline (212/300). The whole remaining gap is PERIODIC 35-65ms worst-render spikes that worsened across the take (189→147 frames/5s) — the signature of gen2 GC pauses, now provable via gen2 +N in every stats window. Biggest churn was structural: ScreenCaptureFrameSource minted a fresh ~8.3MB byte[] per DWM frame (~500MB/s LOH). It now rotates a shared-frame ring (4-deep at slice 8, deepened to 8-deep after the "flash" finding — consumer holds must never outlive depth × source period), size-matched per slot, with reused downscale row scratch. Recycled arrays would poison the paste cache (it keys on array IDENTITY), so VideoFrame.Epoch — monotonic per producer frame, 0 for fresh-array producers — joins the key. Regression test PasteCache_RecycledArrayWithNewEpoch_ReRasterizes_NotStaleHits fails on the old key by construction. The camera producer carried the same fresh-array churn (110-220MB/s at 30-60fps): it now rotates its own 8-deep ring + Epoch (same identity rule), and the WebView2 capture reuses a canvas scratch + 8-deep output ring + a reused WriteableBitmap instead of minting two fresh arrays + a new bitmap per 10Hz tick. Next suspect if gen2 stays hot: the WPF preview load itself driving gen2 — recorded as follow-up, untouched this slice.
• Slice 9 — the "rebase" WAS the recording time-lapse (2026-09-10, the recording playback-timing take): slice 6's "blown deadlines still rebase" was the recording-timing bug all along. In a real take the pump emitted 215-219/300 per 5s (~43fps) while render+wait == period still looked closed — the rebase ERASED every missed slot (deadline = wall-now again), so the missed ticks never showed in the stats. rawvideo has no per-frame timestamps: ffmpeg muxes by frame count at -framerate 60, so a 43fps reality was authored into a 60fps container and every recording played ~1.4x fast (takes confirmed with a WSL ticker visible in the recording: file duration 11.44s vs ~15.5s wall). Two defects, both fixed the OBS way (libobs/media-io/video-io.c — the video thread NEVER resets its deadline; every interval tick outputs ONE frame, and a late render means repeated content — judder — never a skipped timestamp): (1) count-based CFR emission: while (GetTimestamp() >= nextTick) { SubmitFrame(frame); nextTick += intervalTicks; statFrames++; } submits exactly one frame per crossed slot, re-writing the current composite when the render overruns (duplicated footage = correct duration, not a time-lapse), and nextTick is NEVER reset to wall-now; (2) -re removed from FfmpegArgs — it was a second, fighting pacer on the rawvideo demux (its "Resumed reading … after a lag" grew 0.79s→4.82s across the take, leaving the pump behind its own honest-stats count). One fix covers the game background AND the webcam — both flow through this one pump. Muxed duration is now frame-count ÷ fps == wall time by construction. Test: the pump suite stays green unchanged — the pacing test's <interval wait assertion still holds because the frame emits at the next slot boundary, not immediately. (Unrelated pre-existing failure found the same day: Composite_FullScene_MasterPixels pixel (1380,700) cyan-vs-magenta — reproduces with this fix stashed, untouched by it, recorded as a follow-up.)
• Slice 10 — bounded encoder queue + drop policy + burned-in frame counter (2026-09-10, the playback-pacing take): slice 9 made the DURATION right but the CREATOR still read the recorded ticker as "1...23...4...56..." — variable pacing. The aggregates (301/300, uniform PTS, 15.6s wall vs 15.74s file) could not see it, and the cause was finally measured in FfmpegEncoder.SubmitFrameAsync: it BLOCKED on WriteAsync(8.3MB) + FlushAsync whenever ffmpeg lagged the pipe, and slice 9's burst while loop then re-wrote that SAME composite for every slot that ticked past — frozen content runs (the "smeared ticker"). Fixed the OBS way (the encoder queue in libobs/obs-encoder.c: the encoder's thread NEVER couples back into the video thread; overload = dropped content, never a frozen video thread): (1) FfmpegEncoder.SubmitFrameAsync is now an ENQUEUE into a bounded Channel<byte[]> (cap 120 ≈ 2s at 60fps) owned by a dedicated drain task with the stdin write + flush; the caller NEVER blocks on the pipe. Pixels are copied into an ArrayPool buffer before enqueue (the write now happens later on the drain thread, so the pump's scratch reusable the moment submit returns — same contract as before). (2) drop-on-overflow (creator's choice: freshness over coverage): when the queue is full the NEWEST frame is dropped and counted (IFfmpegEncoder.DroppedFrames, Interlocked); the session never freezes or smears — it drops. Stop flushes the whole queue then EOF (Channel.TryComplete → drain writes leftovers → closes stdin → ffmpeg finalizes+exits), so no accepted frame is ever lost at stop. (3) pump loops ONE submit per iteration (replacing slice 9's burst): every due slot gets a FRESH composite — no catch-up slot ever repeats frozen content; missed slots vanish as a count-based gap. The deadline counter is still never reset to wall-now (slice 9 ruling). (4) burned-in frame counter (the new judge): _outputIndex is burned into a 6-digit dot-matrix strip (white box + black 5×7 glyphs, bottom-right corner) of EVERY composite before submit. The WSL ticker is demoted because its own timers smear under Windows host load; decoding the recording and reading the strip is clock-independent: the number advances +1 per frame and jumps by exactly the counted drops (queue overflow OR skipped catch-up slots). Stats gained worst submit Xms, dropped N, stalls K; a stall logger names any iteration > 2× interval with its render/submit split — with the queue, submit is ~1ms, so a stall means render/resolve. The strip sits above the social bar (bar is composited, then burned over) — a debug judge, tiny at 1080p. Test: Backpressure_QueueOverflow_DropsFrames_AndNeverBlocks (the ONE: a 40ms-per-frame sink fake makes every enqueue fill the queue; submit must return instantly, drops are counted, stop flushes exactly submitted − dropped bytes). Full suite 289/290 (the pre-existing compositor pixel failure unchanged). Audio untouched (follow-up). WSL ticker-under-load reliability check still to run (informational — the burned counter is the judge).
• Slice 11 — web-layer animation ran at ~1/6 speed; capture cadence 10Hz→~30Hz + de-throttle (2026-09-10, the "widget animates too slow" take): the recording is 60fps but the WebView2 capture loop was a blind 100ms DispatcherTimer = 10Hz — a 60fps-designed widget was sampled 6× under its native rate (repeated-footage slow-mo). Two stacked throttles:

  Superseded by Slice 14 (below): CaptureScheduler and the cadence hooks were the polling design; frame-driven composition capture replaced them — nothing here is current architecture. (1) the 10Hz cap (unconditional, code-proven) and (2) Chromium hidden-page throttling (requestAnimationFrame parked, JS timers clamped to ~1s per WebView2Feedback#1172/#3070 + Chrome-88 timer throttling) whenever the app window is unfocused or covered — the page is off-screen at (-5000,-5000), so it is a hidden page the moment the host window loses occlusion. Fixed: (1) CaptureScheduler (new, Services/CaptureScheduler.cs) replaces the per-session DispatcherTimer: a dispatcher timer that DROPS ticks while a capture is in flight (latest-wins, never queues) — the in-flight drop is what made raising the cadence safe (concurrent full-HD PNG CapturePreviewAsync calls would stack ~10-30ms encodes and publish stale-after-fresh). Interval is owner-configurable: recording 33ms (~30Hz), idle 200ms. Unit-tested without a WebView2 runtime (the ONE test: CaptureScheduler_Drops_Ticks_While_Capture_InFlight_And_Resumes — overlapping ticks dropped, capture resumes when idle, driven by a TCS so it is deterministic). (2) de-throttle browser args: the shared CoreWebView2Environment is created with --disable-backgrounding-occluded-windows --disable-renderer-backgrounding --disable-features=CalculateNativeWinOcclusion (the Electron/Streamlabs-class embedder answer for occluded-window animation throttling) BEFORE EnsureCoreWebView2Async. (3) cadence hook: MainViewModel sets SetCaptureInterval(33) on record/stream start and (200) on stop (MainViewModel.Streaming.Operations.cs). (4) capture-cost telemetry: first 30 captures per session log elapsed ms to startup.log — PNG-encode+decode cost decides whether ~30Hz stays or drops to ~20Hz; the FramePump drops frames (never time-lapses, slice 10) if UI-thread GC churn starves it, so the cost is observable. Full suite 290/291 passing, the sole failure the pre-existing compositor pixel test. The web-overlay transparency verification (the first-capture diagnostic bound to about:blank) and the audio-silence item remain open push-gate items — both untouched by this slice.

• Slice 12 (audio diagnostics, 2026-09-10): the recorded audio is full-length silent AAC (−91dB, 1124 frames / 23.95s in the latest take) — the named-pipe delivered ~9.1MB of zeros to ffmpeg for the entire recording, so the loop ran and connected, but both WASAPI capture sources delivered nothing. Sources started fine (Audio: using system default mic... logged), no failure callbacks fired, and the mixer's existing integration test (Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe) proves the loop→pipe math is sound. The fault is capture-side: either the default render endpoint carried nothing (audio played on a non-default device — common), or both endpoints were held exclusive, or genuinely nothing played. Added permanent per-5s live-loop telemetry to startup.log (Audio live: pipe connected=, dropped writes=, micLevel=, loopLevel=, drained, peakMix) and NamedPipeAudioWriter.DroppedWrites — names the exact stage on the next take without new code. AudioMixer.FillAndMix now returns (MicRms, MicDrained, LoopDrained) for the telemetry accumulation. Full suite 290/291, same pre-existing sole failure.
• Slice 13 (web transparency diagnostic re-point, 2026-09-10): the "?" in MyMistakes' "RESOLVED (? VERIFY)" is being closed. The one-shot widget dump + alpha stats bound to the FIRST capture ever = the initial about:blank document — a blind instrument. NavigationCompleted for the REAL widget URL now arms WidgetDumpRemaining = 5; those 5 post-paint captures dump %TEMP%\ytLive-web-<id>-w1..5.png + the shared AlphaStats(...) line (alpha[min/max/mean/zero%])
  • FindContentBounds result to startup.log. That line alone decides the fix branch: zero% ⇒ the pre-parse injection did not hold for this widget (opaque capture → chroma-key / stronger DOM injection); large zero% + tight contentBounds ⇒ the capture IS transparent and the recording's black block lives downstream (compositor blend/underlay). AlphaStats extracted as the shared sampler. No new tests (WebView2 runtime not instantiable in tests; the re-point is log-only).
• Slice 14 — web frames are now COMPOSITION-CAPTURED; the PNG poll + CaptureScheduler are gone (2026-09-14, the "still not 60fps" take): slice 11's ~30Hz was still a CapturePreviewAsync PNG poll — every 1920×1080 full-HD encode+decode cost 35–165ms, so the session budget capped real cadence at ~20Hz for a 60fps-designed widget. Root fix = frame-driven capture, not faster polling: the WebView2 renderer now feeds Windows.Graphics.Capture directly through a CoreWebView2CompositionController (the mechanism WebView2CompositionControl and Flutter's webview_windows use — see graphics_context.cc CreateGraphicsCaptureItemFromVisual, which captures the root surface_ visual).
  • Pipeline (per session): CreateCoreWebView2CompositionControllerAsync(WindowHandle) — the parent HWND must exist, so MainWindow.InitWebView2() moved from the ctor to Loaded (was InitWebView2(WebViewHostPanel); the hidden XAML WebViewHostPanel overlay is deleted). controller.RootVisualTarget = a child ContainerVisual (RelativeSizeAdjustment = 1,1) under a root ContainerVisual (1920×1080, IsVisible=true) built on ONE Windows.UI.Composition.Compositor (created once on the UI thread); Bounds = 0,0,1920,1080, BoundsMode=UseRawPixels, ShouldDetectMonitorScaleChanges=false, RasterizationScale=1.0, IsVisible=true, DefaultBackgroundColor=Transparent; then GraphicsCaptureItem.CreateFromVisual(root). Frames pull from a free-threaded Direct3D11CaptureFramePool (2 buffers) → SoftwareBitmap.CreateCopyFromSurfaceAsync (BGRA, BitmapAlphaMode.Straight — the compositor blends STRAIGHT alpha, so premultiplied readback would wreck corner anti-aliasing) → per-frame FindContentBounds alpha-bbox on the capture worker → VideoFrame.CropBounds stamped → ring up to 2 fresh + Epoch (reuse; GC lesson slice 8) → dispatcher-coalesced copy (Render priority) into ONE crop-sized shared WriteableBitmap; PreviewBitmapChanged raised only on bitmap (re)creation. Also unpumped: the OBS-side de-throttle flags (slice 11) stay — they fix the hidden-page JS throttling; the capture path itself no longer polls.
  • Implemented in: new Services/WebCaptureFrameSource.cs (owns item+pool+session — mirrors ScreenCaptureFrameSource, which servers as the copy template) + reworked Services/WebView2Manager.cs (owns controllers, the visual tree, nav/script wiring, preview; internal seam ctor (Dispatcher, Func<string, IScreenCaptureSource>?) so the tests instantiate zero WinRT; public ctor (Dispatcher)).
  • CoreMessaging DQ recipe (record-once): the 19041 CsWinRT projection has NO DispatcherQueueController.CreateOnCurrentThread() (CS0117 — only CreateOnDedicatedThread + FromAbi(IntPtr)). P/Invoke coreMessaging.dll!CreateDispatcherQueueController with struct DispatcherQueueOptions { DwSize, ThreadType = 2 (DQTYPE_THREAD_CURRENT), ApartmentType = 2 (DQTAT_COM_STA) }, wrap via DispatcherQueueController.FromAbi(ptr) (mirror of CaptureInterop), only then new Compositor() — all once on the WPF UI thread (the app dispatches Render there).
  • Compile notes (each cost a build cycle): Compositor collides with the repo's OWN ytLive.Services.Compositor namespace → fully-qualify Windows.UI.Composition.Compositor; CoreWebView2CompositionController exposes Close(), not Dispose(); Color is ambiguous (System.Drawing vs System.Windows.Media) → System.Drawing.Color.Transparent.
  • Dropped: Services/CaptureScheduler.cs (DELETED — no cadence to schedule), all three SetCaptureInterval hooks in MainViewModel.Streaming.Operations.cs, the XAML WebViewHostPanel. TransparentBackgroundScript const is unchanged (a test pins it).
  • Test model (Good Dog): reworked ytLive.Tests/WebView2ManagerTests.cs — dropped the 4 control-size tests + the CaptureScheduler_Drops… test; FindContentBounds tests moved to WebCaptureFrameSource.FindContentBounds; the ONE integration test Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds drives the internal seam with FakeWebSource : IScreenCaptureSource + a real background-STA DispatcherPump (copied from ScreenCaptureManagerTests): register → one crop-sized preview bitmap published → CropBounds + IsOpaque=false on GetLatestFrame → back-to-back pumps coalesce to latest. Byte assertion compares the DENSE 2×2 crop (CropBytes helper — slices source rows with stride gaps, a contiguous range spans rows wrongly).
  • Full suite 293/293 green, 0 warnings. Audio untouched. NOT YET VERIFIED ON DEVICE — the composition path needs a real 60fps-widget run (open item; see HANDOFF). Web work commits stay LOCAL (no push) until the user greenlights.
• Slice 15 — one frame per deadline SLOT: duplicate-on-lag, never skip (2026-09-14, device takes ty-1723/1726): slice 10's freshness choice — a render overrun VANISHES the missed slots from the file — authored ACCELERATED playback: ~35ms render cost vs the 16.6ms slot, one fresh frame per 35ms (log wall-1726: FramePump stall: iteration 33ms (> 2× the 17ms interval): worst render 33-46ms), and a 60fps container muxed on arrival → 1723: 697 frames = 11.62s video vs 11.84s audio; 1726: 163 frames = 2.72s vs 2.93s. The video also ended 0.21–0.24s before the audio ("audio speeds up, cuts off at the end"). OBS's answer is duplicate-on-lag: libobs/media-io/video-io.c emits one frame per tick and a lagger DUPLICATES the newest frame, counted as "lagged frames due to rendering lag/stalls" (obs-output.c) — a wall-time hole never exists (docs.obsproject.com/ backend-design: "If the video frame queue is full, it will duplicate the last frame"). The pump's submit is now a bounded catch-up: while (now >= nextTick) { submit; nextTick += intervalTicks; } over a deadlineNow captured once per iteration — the LATEST composite, fresh on the first missed slot, repeated (OBS's duplication) for the rest. Duration == wall under any render load, at the cost of a short judder during a stall. The burst is safe because Channel.TryWrite never blocks (slice 10's queue — the take-9 smear was the BLOCKING pipe-write re-copying a stale buffer during a long freeze; here each emit is nanoseconds). The burned _outputIndex (slice 10 judge) moved INSIDE the submit loop so every emitted slot carries its own +1 (and the old unconditional pre-gate bump no longer gaps the sequence on non-submitting fast-render iterations). Good Dog test Pump_Overrun_Renders_EmitsEverySlot_NotSkipped: 60fps, resolver sleeps 35ms, asserts ≥0.65 of the wall slots are emitted (the skip-pump wrote ~1/35ms ≈ 200 in 7s; the slot pump ~415). Full suite 294/294 green, 0 warnings. The +0.6s audio-late clap reading on 1726 was confounded by the 1.1x acceleration — re-measure on device; if a real residual remains it is the audio pipeline. No push (web/A/V work stays local).
• Slice 16 — the desktop capture conversion was the bottleneck: fast downscale + cadence throttle + reuse-distance ring (2026-09-14, device take ty-1742): slice 15 fixed pacing but the 1742 desktop layer was still jerky/frozen with a horizontal tear. Decoded to raw frames and audited: the desktop band was frozen 21s of 23.35s (90%) — ~6.1 content updates/s, freeze runs up to 2.28-2.78s, dup-run max 90 frames (1.5s). The render stat (worst render 33-36ms) was real but MOOT: the capture CONVERSION was the wall. The monitor delivers at the 240Hz DWM cadence (~4.2ms); with one-in-flight conversions and each 2560×1440→1920×1080 DownscaleBgra at ~150ms under load, LatestFrame updated a handful of times/s — the desktop feed inside a 60fps file read ~90% frozen. (Webcam + audio were their own paths — fine, matching the report.) Three changes, all in Services/ScreenCaptureFrameSource.cs (+ Services/FrameRingBuffer.cs): (1) DownscaleBgra is now integer 8.8 fixed-point, "shift only at the end" — the exact two-stage math of SceneCompositor.Bilinear (MyMistakes take-4 rule), dropping double-per-pixel to a row-walk of integer ops (the capture downscale had stayed the naive float twin of the 258ms disaster); (2) a 10ms conversion floor (MinConvertInterval) so the 240Hz tail stops queuing ~150ms of serialized conversion per second — the open edge sits just above the ~60/s the pump can use; (3) the ring is a reuse-distance pool (FrameRingBuffer, depth 8, redLine 4): a buffer is only rewritten ≥4 rents after its last hand-out else a fresh allocation, replacing the blind round-robin — since ScreenCaptureManager keeps session.LatestFrame across conversions and the dispatcher preview copy lags, "who released the buffer" needs a consumer API that doesn't exist; a reuse DISTANCE needs no cooperation (the 1742 new-top/old-bottom tear is that read-under-write, structural now). Telemetry added: a startup.log line every 2s (frames/s, conv avg/max ms, skip busy/cadence, ring allocs) so the device take can be judged numerically. Good Dog test Ring_NoLap_ReusesOnlyAfterRedLineRents (depth 3/redLine 4 exercises the red-line skip; the 8/4 config settles at 8 buffers and never grows). Full suite 295/295 green, 0 warnings. C4 (compositor Epoch-cached downscale / blit-on-change) was DEFERRED: the slice-15 approved plan assumed a native-res relocation; the measurement recast it — relocating a ~30ms float downscale to the render thread just moves the same cost into the slot budget. Re-measure on device; if render still >16.6ms slots after capture feeds ≤60 real updates/s, add C4. No push.
• Slice 17 — the OS readback was the real wall: overlapping conversions + monotonic publish + deeper pool (2026-09-14, device take ty-1824): slice 16's downscale fix landed but the desktop was still ~90% frozen on the 1824 take (band 6.8/s updates, max freeze 4.85s). The new 2s telemetry was decisive: conv avg 46-50ms max ~61ms with skip busy 37-83 — the GPU→CPU readback (CreateCopyFromSurfaceAsync), not DownscaleBgra, is the ~47ms wall (240Hz HDR compositing + encoder + 2 capture devices share the GPU); at one-in-flight that caps the desktop feed at ~17-20 updates/s, which is the file's whole-frame ~7 content-moments/s. Two facts reshaped the fix: (a) shrinking the pool size does NOT scale the desktop (Microsoft docs: "If content is larger than the frame, the contents are clipped") — readback stays at native 2560×1440; (b) delivery is healthy (60-100 arrivals/s), so the lever is conversion throughput, not the pool size. Changes in Services/ScreenCaptureFrameSource.cs: conversions overlap up to MaxConcurrentConversions = 3 (pool deepened to 5 buffers so in-flight frames fit), each completion publishes ONLY if its Epoch is strictly newer than the last published (MonotonicGate — a slow older completion must never overwrite a newer LatestFrame), Epoch increments via Interlocked, and the DownscaleBgra row-scratch became per-conversion locals (concurrent callers). The render side (33-42ms → ~30 unique composites/s) is the NEXT cap after capture speeds up — that's the C4 slice, queued right after this re-measure. Good Dog test PublishGate_TryPublish_OnlyStrictlyNewerWins. Full suite 296/296 green, 0 warnings. NOT YET DEVICE-VERIFIED; target: telemetry frames/s jumps ≥ ~30-40 and the band audit drops below ~50% frozen. No push.
• Slice 18 — C4: the FramePump blit-on-change composite cache (2026-09-15, take ty-1841 verdict): the 1841 take proved capture fixed (band ~20 fresh updates/s, no tears, pacing clean) but logged a FramePump stall on EVERY iteration (totalMs 21-44, render=full-render split=0 elements=6 dynamic=4, worst render 166ms startup spike) → ~22-28 composites/s hard-caps the desktop inside the 60fps file. The SceneGraph split CANNOT fix this wired: GetSplitPoint returns 0 because the live-capture backdrop is element 0 and must stay dynamic — a baked capture goes stale — so every tick runs the full re-composite even when no input changed (the compositor's per-layer paste cache already makes identical frames identical per-element; the waste is re-COMPOSITING the whole 8.3MB frame). Same shape as OBS (sources cache their surface, the compositor renders on update only — the pattern this file has cited since the 2026-09-04 paste-cache slice) at the FRAME level. FramePump.RenderFull now wraps both full-render call sites (no-SceneGraph + split=0 fallback): it computes BuildFullRenderSignature (a deterministic hash of the crop/output dims, the social bar identity + top, and — per element, MIRRORING the compositor's own resolution — the element ref, layout/visual bits, and the frame each element would resolve through the SAME resolver seam, using buffer-array identity + Epoch + CropBounds; a changed frame forces a re-composite, an unchanged one never serves stale bytes) and, when it matches the last composite, reuses the cache with ONE Buffer.BlockCopy (~3ms) instead. The cache buffer is a separate long-lived array — never the scratch pool — because the caller burns the frame counter and recycles the scratch AFTER RenderScene returns; the cache is written from the rendered scratch BEFORE returning (pre-burn, pre-recycle). Engagement is gated on the 1:1 config (SourceRect == Output, the only deployed tier); the vertical tier's cached fresh-scale-buffer follow-up stays exactly where it was. Telemetry: internal CacheHits/ CacheRenders counters surface as cache {renders}R/{hits}H on the 5s stats line, and the ticks' key is an internal OutputIndex (burned counter, stable across one tick's resolver passes). Good Dog test FullRenderCache_StaticInputs_RenderOnce_Then_Reuse_UntilInputChanges: static scene → exactly ONE composite (renders==1, hits>0, byte-identical content above the burn strip), then a new frame (new array + monotonic Epoch) invalidates and propagates. Existing pool test now passes a STABLE scene (production hands StagedScene; a fresh scene per tick churns the element refs inside the signature and hides the cache — the tests that care about cache behavior must mirror production) and keys its red/blue alternation on OutputIndex/frame count so the tick's two resolver passes (signature + render) don't double-advance a call-count flip. Full suite 297/297 green, 0 warnings; verify.sh scope-locked to FramePump.cs + FramePumpTests.cs + docs. Target next take: cache hits dominate on TV holds, avg render drops toward the BlockCopy, stalls vanish, then the band audit + clap re-measure decide push. No push.
• Stop ordering matters: StopAsync stops the encoder — since slice 10 it FLUSHES the pending queue (Channel.TryComplete → drain writes the leftovers, closes stdin → EOF → ffmpeg finalizes+exits; an accepted frame is never lost) — before awaiting the loop. The old reverse-order deadlock was a submit stuck on pipe backpressure; the drain thread owns that wait now, and stopping the encoder first keeps the pump's own (non-blocking) submits from racing the completed queue as a spurious drop. ProcessFailed self-stops the pump. Failed while live flips StreamStatus.Error (minimal).
• Health stats (ship step 6, shipped 2026-08-13): HealthUpdated is bound to the bottom bar — MainViewModel.OnFramePumpHealthUpdated marshals to the UI thread (the encoder's stderr loop raises on a background thread) and copies into CurrentHealth (the bottom bar's existing CurrentHealth.* bindings). ResetHealth(status) zeroes dropped/duration on go-live and on End so stats never linger from a previous session; bitrate/FPS stay on the tier's targets (ApplyStreamQuality). The bar shows real encoder values since TASK 9 (2026-08-16) wired the reusable stream URL into _rtmpUrlProvider.
• MainViewModel owns the resolver (ResolveOutputFrame): WebcamSceneConfig → CameraManager.GetLatestFrame(DeviceKeyForWebcam(WebcamId)) — device-keyed (configs carry the identity GUID; sessions key on the Windows device id — the old direct-GUID lookup returned null forever and the webcam was structurally absent from output; take-2 fix 2026-09-01, WebcamOutputKeyTests), Source { IsLiveCapture, CaptureKey } → ScreenCaptureManager.GetLatestFrame(CaptureKey) (the new accessor mirroring CameraManager), image/ background → StaticPixelCache.Get(AssetId). BuildCompositorOptions rounds the VM's OutputRect* doubles to ints — the vertical 607.5 half-pixel crop rounds to a perfectly-centered 608 (Math.Round, ToEven); BuildEncoderOptions fills W×H/FPS/bitrate from the tier once the URL provider yields one.
• Media source decoder (TASK 21, Increment B — 2026-08-31): MediaVideoSource (Services/MediaVideoSource.cs) decodes a local file to VideoFrames by spawning ffmpeg with -f rawvideo -pix_fmt bgra -vf scale=WxH -an and draining the raw BGRA stdout pipe through the pure RawVideoFrameReader (Services/RawVideoFrameReader.cs), raising FrameAvailable per frame. Implements IMediaFrameSource (Services/IMediaFrameSource.cs: Key, FrameAvailable, Completed, StartAsync/StopAsync). The subprocess sits behind IDecodeProcess/FfmpegDecodeProcess (binary-stdout mirror of the encoder's IEncoderProcess), and the path comes from IFfmpegLocator — codec-agnostic, uses the ffmpeg already shipped, testable with a fake process. Decode sessions are owned app-wide by MediaVideoSourceManager (Services/MediaVideoSourceManager.cs): refcounted by MediaPath, Func<string, IMediaFrameSource?> factory seam (mirror of ScreenCaptureManager), AcquireAsync/ReleaseAsync/ReleaseAllAsync/GetLatestFrame, coalescing each file's frames onto the UI dispatcher onto one shared WriteableBitmap. Frames emit as fast as the pipe produces them. Wired into the live pipeline (slice 1 step 4): ResolveOutputFrame reads _mediaManager.GetLatestFrame(MediaPath) for a MediaSource, and Source.DisplaySource (Models/Source.cs) routes VideoImageSource for MediaSource so the preview/canvas show decoded video. Slice 2a shipped the probe seam: FfmpegFrameRateParser (pure, avg_frame_rate= then r_frame_rate= rational N/N/M, unknown→null) + IFrameRateProbe/FfmpegFrameRateProbe (derives sibling ffprobe.exe from the located ffmpeg dir, reuses the IDecodeProcess seam for the ffprobe subprocess text; null if ffprobe absent) + FfmpegLocator.ProbeFileName now also extracts ffprobe.exe from the pinned archive (conditional — old caches without it just get no pacing). Slice 2b wired pacing: MediaVideoSource takes optional IFrameRateProbe? + Func<TimeSpan,CancellationToken,Task>? delay (default Task.Delay) seams, probes FPS once in RunAsync, and delays by 1/fps after each emitted frame; no probe/unknown → no pacing (ffmpeg's own pipe backpressure already throttles the decode). Slice 3 added loop control: IMediaFrameSource.Looping (bool); MediaVideoSource takes a Func<IDecodeProcess> process factory (a single Process can't be re-Start()ed, so each loop pass creates a fresh decoder) and wraps the decode in a do…while (Looping) — restart on natural EOF instead of raising Completed. Production wiring (MainViewModel media factory): passes () => new FfmpegDecodeProcess() as the factory, FfmpegFrameRateProbe(new FfmpegLocator(), () => new FfmpegDecodeProcess()) as the probe. Still open: session acquisition on add/remove (UI picker) + wiring Source.MediaIsLooping into IMediaFrameSource.Looping (needs a manager-level per-path loop provider, comes with the picker slice). Acquisition shipped 2026-11-05: AddMediaCommand (MainViewModel.Media.cs) opens a video-filtered OpenFileDialog and AddMediaSource(path) adds a MediaSource layer through the ordinary AddSource path, defaulting to a full-canvas 1920x1080 rect, then RegisterMediaSource subscribes the layer and EnsureMediaSession starts one decoder per distinct path (HasSession guard keeps refcounts at 0/1). Remove (RemoveElement) unsubscribes and releases only when no scene references the file any more; ReacquireMediaSources runs after LoadLayout so persisted layers get sessions and stale files are released (GetActiveKeys). Loop is a manager-level per-path answer: the ctor takes Func<string,bool>? loopProvider (production: "any live media reference loops"), applied before StartAsync, and SetLooping(key, bool) applies a live toggle. A file swap releases the old path after the layer no longer references it. Sound ships with it (2026-11-05): both MediaVideoSource and AlertClipDecoder are now thin wrappers over one Services/VideoClipPipe.cs (video pipe + f32le audio pipe, both real-time paced, loop restarts both), so a clip is a clip. The manager applies the layer's Level/Mute per session and raises AudioAvailable, and MainViewModel feeds it to AudioMixer.EnqueueDesktopMediaAudio — the DESKTOP/GAME ring, so the desktop fader and the voice ducker apply. The alert clip keeps its own unity-gain ring (EnqueueAlertAudio, never ducked). Green/blue screen (TASK 64) rides the same pipe as a decode option: VideoFilterChain.BuildVideo appends colorkey=… + despill=… + format=bgra after the scale, so the key is computed by ffmpeg and both consumers (shared preview bitmap, compositor frame) carry the same alpha — no app-side per-pixel pass, no second render path. A media session is therefore keyed by path + look (MainViewModel.MediaSessionKey), so two layers with identical settings share one decoder and two different looks get two. The offer comes from ChromaKeyDetector over one probed frame (FfmpegChromaProbe); settings persist on the Source row.
• Social bar on the output (bar bug-fix branch): the FramePump takes an optional socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>? seam, re-read every frame (so a mid-stream position flip applies immediately). The strip is pre-rasterized by Compositor/SocialBarRenderer.cs (WPF glue — WPF glue here is fine because the strip is rendered once per config change on the UI thread, the resulting immutable frame is then composited pure-CPU by SceneCompositor), and SceneCompositor.Render blits it last — above the branding flash at socialBarTop (0 = top, SourceRectHeight − barHeight = bottom) in master space. MainViewModel owns the frame (_socialBarFrame, rebuilt by RenderSocialBarFrame on load/save/NotifySocialsChanged).
• Privacy blur layer (2026-11-01): SourceType.Blur is a Live-scene element that blurs the composed frame inside a rect or ellipse. It is universal: SceneCompositor applies the blur group AFTER every layer has been composited (before the flash/social/ticker overlays), so it can cover the webcam, chat, alerts — anything — which is the creator's "moment of privacy" case. Its position in the layer list is cosmetic for that reason (it is added at the top), and the earlier "never above the brand" rule is superseded for blurs: a blur over the logo does obscure it. SceneCompositor treats all visible blurs as ONE group the first time one is reached in the z-order (ApplyBlurLayers), sorts them strongest first and passes a shared coverage map to RegionBlur.ApplyInPlace, so an overlap resolves to the strongest single blur rather than a blurred blur. RegionBlur shares the webcam's radius curve and constants (Downscale 4, MinRadius 1, MaxRadius 6, RadiusFor(strength)), so a strength means the same thing in both features; the webcam's BackgroundEffectProcessor delegates to it. Blur is ElementKind.Dynamic, so it is never baked into the static base and every render path (pump, preview, thumbnails, scene dump) gets it from the one compositor.
• Known consideration: the pump reads the active scene on a background thread while the UI can still edit it; a concurrent-mutation exception is contained (logged + Failed + the pump stops) rather than crashing. The background thread + video pipeline is the new reality since this step.

---

Related files

This file owns the compositor and the pump. The neighbours live elsewhere:

• output.md — live encoder + RTMP push, IFfmpegEncoder/FfmpegEncoder, FfmpegArgs, locator and bundle.
• TASKS/task-21-media-source.md — media source, MediaVideoSource (ffmpeg rawvideo decode) feeding the pump's resolver.
• overlays.md — social bar and native events & alerts, the overlay strips the compositor blits last.
• environment.md — Run and Tests, how to launch in Release and how the suite is run.