Creator's Release take `ty-20261109-1142-036-0005.mp4` (1080×1920 @ 30, 300 s) measured by cropping the webcam tile and diffing frames: 29.8 distinct images/s overall, median 30 s window 30.0/s, worst 24/s, one 4-frame repeat run in 9026 frames. The app's own lines agree (`webcam 28.4–30.0/s` against camera `arrivals 29.9–30.1/s`). - The 25.6/s failure recorded on the 2026-10-05 16:9 take does **not** reproduce in the vertical tier, so the vertical take no longer blocks the cadence units. - Recorded the measurement trap in `frame-pipeline.md`: a threshold of 1.0 on a downscaled tile reports long "0.0/s" stretches during low motion — duplicates sit below 0.02, a still face at 0.3–0.7. Read the histogram before believing a count. - Vertical-tier camera check reported 1–6 pass; both creator-hands items in TASKS.md are closed. (The mic being off for the first 2:42 does not affect video cadence.)
862 lines
75 KiB
Markdown
862 lines
75 KiB
Markdown
# Frame pipeline — compositor, render budget, measurement, and the live pump
|
||
|
||
> Split out of [`ai.md`](../../ai.md) (2026-10-26) so a session reads only what its task touches;
|
||
> `ai.md` keeps a short invariant stub for this file. Directory map: [`index.md`](index.md).
|
||
|
||
This is the subsystem that turns scenes into the encoder's frames: the CPU compositor that renders
|
||
the master, the budget it must fit inside, the instrumentation that measures it, and the
|
||
`FramePump` that paces frames into ffmpeg. Everything here is current architecture plus the
|
||
measured history that produced it; superseded designs are marked inline.
|
||
|
||
---
|
||
|
||
## Budget, measurement & the compositor
|
||
|
||
### Frame cadence — how a dropped frame became measurable (UNIT 0, 2026-10-05)
|
||
|
||
**The problem this exists to solve.** The recorded file cannot distinguish a dropped camera frame
|
||
from a padded repeat. `-f rawvideo` carries no per-frame timestamps — ffmpeg assigns pts from frame
|
||
order at the DECLARED fps — so a webcam delivering 25 of its 30 frames still muxes a full-length,
|
||
correctly-timed 60fps file. Measured on take `ty-20261005-1001-821-0001.mp4` (85.6s, 1080p60): the
|
||
lower-right webcam region held 25.6 distinct images/s against the camera's 30fps, with 1,153 normal
|
||
33ms intervals, 281 gaps of 50–100ms, and 96 bunched 17ms gaps — roughly one hitch in five, which is
|
||
what makes a face judder while the file looks flawless. (The "webcam renders at ~18fps" figure that
|
||
used to sit in this map was WRONG: it came from a different session and counted every newly rendered
|
||
layer, not webcam arrivals. Corrected 2026-10-05.)
|
||
|
||
**Vertical-tier take (2026-11-09).** `ty-20261109-1142-036-0005.mp4`, Release, 1080×1920 @ 30 fps,
|
||
300 s, webcam in its default bottom-right tile. Cropped the tile, downscaled to 160×88 grey and
|
||
diffed frame to frame: **29.8 distinct images/s overall, median 30-second window 30.0/s, worst 24/s,
|
||
one 4-frame repeat run in 9026 frames.** The app's own lines agree — `webcam 28.4–30.0/s` against
|
||
`arrivals 29.9–30.1/s`. So the 25.6/s failure above does **not** reproduce in the vertical tier, and
|
||
the vertical take therefore does not block the cadence units.
|
||
|
||
⚠️ Threshold matters. A naive `mean abs diff > 1.0` on a downscaled tile reports long "0.0/s"
|
||
stretches during low motion — that is the measurement failing, not the camera. Duplicates encode to
|
||
near-zero (<0.02); a still face with sensor noise sits around 0.3–0.7. Use ~0.05 and read the
|
||
histogram before believing a count.
|
||
|
||
**`Helpers/FrameCadence.cs`** — one pure counter, three feeds. `Add(timestamp)` on any shared
|
||
monotonic clock; `Summarize()` → frames, rate (gaps ÷ span, NOT frames ÷ span — N frames span N-1
|
||
periods), gaps, hitches (default threshold 50ms ≈ 1.5× a 30fps period), longest and average gap;
|
||
`Describe(label)` gives every producer the same one-line shape so two lines can be read against each
|
||
other. An out-of-order stamp counts as a frame but produces no gap AND does not become the reference
|
||
for the next interval — otherwise one bad stamp invents a multi-second "hitch" twice.
|
||
|
||
**Three measurement points, and what each separates:**
|
||
|
||
| Line | Fed by | Answers |
|
||
|---|---|---|
|
||
| `Camera cadence WxH: arrivals … \| camera …` | `MediaCaptureFrameSource` | device/USB/driver cadence (`MediaFrame.SystemRelativeTime`, QPC domain, nullable) vs what the reader handed us. `TryAcquireLatestFrame` keeps only the LATEST frame, so falling behind is silent by design — this is where that shows |
|
||
| `webcam …` on the pump's stats line | `FramePump.TimedResolver` | distinct webcam `VideoFrame.Epoch` values that reached a composite — what actually got into the output |
|
||
| `frames per 5s from R renders` | `FramePump` | submits vs REAL composites — the duplicate-on-lag ratio, the only honest reading of how much of the stream is a repeat |
|
||
|
||
**Why the stats-line denominators changed (the actual bug).** Every average was divided by
|
||
`statFrames`, the number of frames SUBMITTED — which deadline padding always fills to the full 60fps
|
||
target. A pump fed one webcam frame per 200ms logged "300/300 frames per 5s, avg render 12.4ms"
|
||
while producing 60 distinct composites: the render average was diluted ~5× and the loss was
|
||
invisible. Render/resolve/wait are now divided by `statRenders`; submit stays per-frame (it really
|
||
happens per submit). Regression test:
|
||
`FramePumpTests.Stats_ReportRealRendersAndDistinctWebcamFrames_NotJustPaddedSubmits`.
|
||
|
||
**Reading the three together:** camera 30/s steady + arrivals 25/s = WE dropped it (reader or
|
||
compositor). Ragged in both = the device, the USB link, or the driver. Pump `webcam` below
|
||
`arrivals` = the compositor missed frames the reader had already delivered.
|
||
|
||
**✅ WHAT THE FIRST REAL READING SAID (2026-10-05) — the camera was never the problem.**
|
||
`ty-20261005-1201-556-0001.mp4` + its log: the C920 delivers **30.0/s with ZERO gaps ≥50ms**, and the
|
||
device's own stamps are identical, so the camera, USB link, and driver are all clean. The pump did
|
||
**34 composites per 5s (6.8/s) at ~141ms each** (worst 294ms), and the file holds **6.8 distinct
|
||
webcam images/s at 88.7% duplicates with no arrival hitches at all**. The judder is entirely us
|
||
holding one face for 140ms and padding the rest of a 60fps container. `resolve` is 0.1ms, so it is
|
||
pure pixel work in the composite, not frame lookup. Two consequences, both load-bearing:
|
||
|
||
- **720p is a sharpness win, NOT a smoothness win** — it triples the bytes stretched per render on a
|
||
path already 4.4× over budget. Schedule the exclusive lock AFTER the compositor work or it makes
|
||
the judder measurably worse.
|
||
- **The 10:01 take hit 25.6 images/s and the 12:01 take 6.8, same scene shape
|
||
(`full-render split=0 elements=6 dynamic=5`), same 640×480 camera.** Composites went 29–34ms →
|
||
143–155ms between sessions with nothing logged explaining it. That comparison was made by EYE on
|
||
two takes because the log carried no per-element breakdown and no scene dump — both now exist
|
||
(see "Per-element composite timing + scene dump" below); naming the culprit is a reading, not a
|
||
guess, and guessing is explicitly out of bounds. **Superseded 2026-10-16:** "same scene shape" was
|
||
an eye comparison of takes whose geometry was never logged, and the next bad take still on disk is
|
||
the *vertical* tier (below). A Release 1920×1080 take now measures 1.0ms with no element over
|
||
0.4ms — see "Per-element composite timing + scene dump" → what the first instrumented take said.
|
||
|
||
**Counter subtlety, so it is not re-derived:** the pump's `webcam` line read 13/s while the file held
|
||
6.8/s. Not a bug — the counter stamps a frame when the resolver FIRST sees it, and one composite takes
|
||
141ms, so ~4 camera frames arrive during it (signature pass stamps one epoch, the actual draw uses a
|
||
later one). The file is the truth; the pump counter is a pipeline-health signal and overcounts by
|
||
design.
|
||
|
||
**Still unmeasured:** the camera is stuck at the C920 default 640×480 YUY2 @30 because all 185
|
||
`SetMediaStreamPropertiesAsync` attempts are refused under `SharedReadOnly`, so 720p MJPG is available
|
||
— but see above, it is a quality win only.
|
||
|
||
### Per-element composite timing + scene dump (UNIT 0 follow-up, 2026-10-16 — MEASUREMENT ONLY)
|
||
|
||
The gap the cadence work left open is now closed by instrumentation, and closed by nothing else: the
|
||
141ms take was never compared to the 30ms take because **no line in the log said which element cost
|
||
what, and no line said what the scene WAS.** Both now exist. No fix has been attempted, and none is
|
||
allowed until a real reading names the culprit.
|
||
|
||
- **`Services/Compositor/CompositeTimings.cs`** — a collector owned by the pump, threaded through
|
||
`SceneCompositor.Render` / `CompositeLayers` / `BakeStaticBase` / `BlitBakedElement` /
|
||
`BlitCachedLayer` (and `SceneGraph.GetBakedBase` → bake path) as an optional last parameter, so
|
||
every other caller (thumbnails, tests, transition preview) is untouched and costs nothing.
|
||
`Add(stage, ticks)` accumulates; `Describe(renders, minMs = 0.1)` renders the averaged breakdown
|
||
slowest-first, dropping sub-0.1ms noise.
|
||
- **Stage names** (stable, greppable): `backdrop` (prefill + live capture + background art + their
|
||
resolves), `base` (the cached-bake `BlockCopy`), `flash`, `social`, `ticker`, `zoom`, `scale`,
|
||
`overlay` (the fully-static path's per-frame overlay copies), `bake backdrop`, `bake overlays`, and
|
||
**one entry per element as `#<z-index> <name>`** — the index IS the z-order, so the log names the
|
||
layer by both position and name. Bake-pass elements are prefixed `bake #…` so a one-off rebake
|
||
cannot masquerade as a per-composite cost.
|
||
- **`#<i> <name> raster` nests inside its element's total** and is the paste-cache rasterize miss
|
||
(`BlitCachedLayer`'s `raster == null` branch) — the direct answer to "is one element re-sampling
|
||
every tick, or is the paste itself slow". A stage's time includes that stage's frame resolve;
|
||
`resolve` is separately totalled in the existing stats line.
|
||
- **`FramePump parts (N renders/5s): …`** — logged once per 5s window (same window as
|
||
`FramePump stats`), averaged per render so the numbers compare directly with `avg render`, then the
|
||
collector is reset. **No parts line, or a short one, is a reading too:** the fully-static shortcut
|
||
and C4 cache hits composite no elements (`cache 92R/130H` beside a missing parts line is expected).
|
||
- **`FramePump scene dump: '<name>' elements=N split=S`** followed by one line per element
|
||
(`[i] name kind=… type=… vis=… rect=x,y WxH op=… clip=…`), logged when the pump first runs and
|
||
whenever the active scene changes. This is the record the 10:01/12:01 comparison never had.
|
||
- Gate test: `ytLive.Tests/CompositeTimingTests.cs` (one `[Fact]`) — a REAL composite of a full-canvas
|
||
element plus a 64×64 one must rank the full-canvas element first, carry the `raster` sub-stage and
|
||
the fixed stages, and the dump must name both elements with z-index/type/rect and echo the split.
|
||
- **`scripts/webcam-cadence.py`** — the other half of a reading: what the FILE holds (distinct webcam
|
||
images/s + hold histogram), beside what the pump says it did. Recipe and its three traps (H.264
|
||
re-encoding, the round-clip crop, scene switches): `MyMistakes.md` → "Measuring webcam CAMERA cadence
|
||
in a recording".
|
||
|
||
**Read takes in Release.** The creator launches `dotnet run -c Release`; Debug pixel paths are
|
||
several× slower, so a Debug number is not a reading of this pipeline (see
|
||
[`environment.md` → Run](environment.md#run)).
|
||
|
||
**Derivative check (AGENTS.md):** this is not invented — OBS's own answer to "which source is slow" is
|
||
the **Source Profiler**, which reports the *average execution time of each source's tick and render
|
||
functions* ([docs.obsproject.com → Source Profiler](https://docs.obsproject.com/reference-libobs-util-source-profiler),
|
||
per-source timers via `util/profiler.h`). Same shape here: one timer per element per composite, reported
|
||
as an average, plus the stage split. The `#<i> <name> raster` sub-stage is the extra this case needs —
|
||
it separates "re-sampled every tick" from "slow paste" without another session.
|
||
|
||
### ✅ What the first instrumented take said (2026-10-16, Release) — the judder DID NOT reproduce
|
||
|
||
Take `ty-20261016-1841-559-0002.mp4` (174.5s, 1920×1080@60, Release build `e95154d9` — a **recording**,
|
||
not preview; scene `Live`, switched to `Chat` at 28s, back to `Live` at 44s) and its log window:
|
||
|
||
| Reading | This take |
|
||
|---|---|
|
||
| composites | **301 per 5s = 60/s**, every slot; 0 dropped, 0 stalls after warm-up |
|
||
| avg render | **1.0–1.1ms**, worst 2.5ms — ~7% of the 16.67ms budget |
|
||
| `parts` | `#5 HD Pro Webcam C920 0.4ms, #5 … raster 0.2ms, backdrop 0.2ms` — the other four elements are under the 0.1ms print floor |
|
||
| C4 full-composite cache | **151R/150H** per window — half the ticks were a whole-frame `BlockCopy` |
|
||
| webcam at the resolver | **30.0/s**, longest gap **50.0ms** (one extra 60Hz frame = sampling phase, not a drop) |
|
||
| the file itself | **29.6 distinct webcam images/s, 82% of images held exactly 2 frames**, longest hold 267ms (`scripts/webcam-cadence.py`) |
|
||
|
||
Nothing in this take is over budget and **no element is expensive — there is no culprit here to fix.**
|
||
The two stalls in the take are not per-composite cost: 124ms on the very first render (warm-up) and
|
||
53ms on the switch into `Chat` (a rebake).
|
||
|
||
**Scale of the disagreement:** the historic slow readings are 141ms (Oct 5 12:01) and 45ms (Oct 9
|
||
19:48) — 40–140× the ~1ms this take needed per composite. Three facts about those takes, all read off
|
||
the same log:
|
||
|
||
- **Oct 9 19:48 (`ty-20261009-1948-121-0000-2.mp4`) was the VERTICAL tier**, not a 16:9 one:
|
||
`FramePump started (1080×1920 @ 60 fps) scene='Live' comp=608x1080` — the 607×1080 centre crop
|
||
bilinear-upscaled to 1080×1920 (2M px), 92 renders/5s, avg render 45.0ms, worst 278ms, 92 stalls,
|
||
`cache 92R/0H`. The file on disk is 11.6s and every frame is identical (a static desktop), so it
|
||
cannot be read for cadence — only its cost.
|
||
- **Oct 5 12:01 (`…1201-556-0001.mp4`, no longer on disk) logged `FramePump started (1920×1080 @ 60
|
||
fps)` with NO `scene=/comp=` suffix** — that suffix did not exist yet, so its source rect was never
|
||
recorded and the map's "21:9 → 1920×803" explanation stays a hypothesis.
|
||
- **A take can latch mid-take.** Oct 5 12:29 goes 1.5ms (avg render) → 3.0ms → **45.2ms within 15
|
||
seconds and stays there** (`cache 10R/0H`, then `98R/0H`). So the slow state is not a fixed property
|
||
of a scene geometry. The consistent signature of every slow window is `cache xR/**0H**`: the C4
|
||
whole-frame cache never reused once, so every tick composited. The healthy take shows the other
|
||
stable state (composite 1ms → every second tick is byte-identical → whole-frame reuse). **The latch
|
||
is the thing to test on a reproducing take** — it is a hypothesis, not a reading; the `parts` line
|
||
is what will decide it.
|
||
|
||
**What this does NOT say.** This take is the 1:1 master tier and its scene logs
|
||
`full-render split=0 elements=6 dynamic=4` (the bad-period takes logged `dynamic=5`), so it never
|
||
exercises the failing geometry. "The healthy path is healthy" is not "the judder is gone". Reproducing
|
||
it needs a Release take in the failing configuration (vertical tier first, then the 21:9 display
|
||
config) — that take's `parts` line names the stage, and only then is a fix unit allowed.
|
||
|
||
**Reading the `parts` line's denominators:** like `avg render`, it is divided by `statRenders`, which
|
||
counts C4 REUSES as well as real composites (the 301 above = 151 real + 150 whole-frame copies). So in
|
||
a healthy 30fps-camera take both averages are ~2× optimistic; the `cache NR/NH` pair on the same line
|
||
says how much of the window was real composite work. `CompositeTimings.Describe` drops stages under
|
||
0.1ms by default, and **a missing or short parts line is itself a reading** — a fully-static window
|
||
(every tick a C4 hit) composites no elements at all.
|
||
|
||
### Render budget — getting a composite inside 16.67ms (UNITs 4-5, 2026-10-05)
|
||
|
||
The pump emits one encoder frame per 16.67ms slot and **pads with a duplicate when a composite
|
||
overruns**, so the whole problem is "render + submit + untimed work ≤ 16.67ms, ideally ≤12ms to
|
||
leave headroom". `FramePump` now reports per-stage timings (`social`/`ticker`/`flash`/`acquireScratch`)
|
||
so a slow stage names itself, and logs the scene name + compositor source rect at pump start.
|
||
|
||
**Why a 21:9 monitor cost 141ms and a 16:9 one 33ms — the fast path is EXACTLY 1920×1080.** A
|
||
21:9 display shrinks the source rect to 1920×803 ("fit", not "fill"), so the opaque-backdrop 1:1 row
|
||
walk is skipped, the base is prefilled with opaque black (~2M byte writes), and the backdrop is
|
||
resampled per-pixel (~1.54M pixels with an alpha branch). Every recorded A/B lines up:
|
||
21:9 → 141ms, 16:9 → 30-33ms, 2560×1440 (fits to 1920×1080 exactly) → fast path.
|
||
**Corrected 2026-10-16 — the "never Debug-vs-Release" half of that is withdrawn.** A Release
|
||
1920×1080 take measures **1.0–1.1ms** where an Oct 9 take of the same tier measured 10.4ms, and
|
||
`Build <hash>` carries no build configuration, so the config of every pre-2026-10-16 take is
|
||
UNKNOWN: no historic render number may be compared against a Release take. The 21:9 explanation
|
||
itself stays a hypothesis — the 141ms take's start line predates the `scene=/comp=` suffix, so its
|
||
source rect was never recorded. A take can also latch mid-take (see the instrumentation section).
|
||
|
||
**Unit 4 — cheap wins, all pixel-identical:**
|
||
- Opaque 1:1 fast paths expanded to the whole-canvas opaque case (a `BlockCopy`, not a per-pixel
|
||
alpha branch).
|
||
- **Prefill only the uncovered rows.** When the backdrop does not cover the canvas, the letterbox
|
||
bands are filled instead of the whole buffer; with no backdrop at all it still fills everything.
|
||
- **Per-column LUT** for the scaled blit (`BlitContentRaw`): the source column, its neighbour index
|
||
and the 8.8 weights are computed once per column instead of a divide + clamp per pixel.
|
||
The table comes from `ArrayPool<float>.Shared` and is returned in a `finally` — a single static
|
||
buffer is a **data race** the moment strips run in parallel (see below).
|
||
|
||
**Unit 5 — strip-parallel composite: `Helpers/StripPool.cs`.** One composite splits into four
|
||
horizontal row strips; the calling thread takes a strip too, so four workers never means five-way
|
||
oversubscription. Below 64 rows the fan-out costs more than it saves and runs inline. Each strip must
|
||
write only its own rows — the blits already partition by row, which is why the output stays
|
||
byte-identical.
|
||
|
||
**The demotion must come from DEDICATED threads, not the shared pool.** The obvious
|
||
`Thread.CurrentThread.Priority = BelowNormal` inside `Parallel.For` is wrong: `Parallel.For` runs on
|
||
shared ThreadPool threads, so the demotion leaks into every unrelated work item that thread later
|
||
picks up. Research (https://stackoverflow.com/questions/524483/net-backgroundworker-objects-thread-priority
|
||
— "you have next to no control of which thread pool thread will actually run your task";
|
||
https://stackoverflow.com/questions/1578599/c-sharp-lower-thread-priority-in-thread-pool — "You
|
||
shouldn't mess with the priority of the regular ThreadPool, since you aren't the only consumer ...
|
||
write your own producer/consumer queue, using your own Thread(s) as the consumer(s) - setting the
|
||
thread priority yourself") says exactly that, and **a real failed attempt confirmed it**: a full test
|
||
run after demoting the pool threads broke an unrelated audio-mix timing test. `StripPool` IS that
|
||
producer/consumer queue — four `Thread`s with `Priority = BelowNormal`, `IsBackground = true`,
|
||
started lazily on first use, each fed a `Queue<Strip>` + `SemaphoreSlim`. Four is the measured sweet
|
||
spot: fewer leaves render over budget, more steals cycles from the game and the encoder and made
|
||
render **worse**.
|
||
|
||
Gate test: `SceneCompositorTests.StripParallel_RepeatedRenders_AreByteIdentical` — the same scene
|
||
(scaled round webcam + mirrored image + master-sized flash over a short backdrop, 240 rows so all
|
||
three fanned-out paths run) must come out byte-for-byte identical across 8 renders, and must differ
|
||
from a backdrop-only render so it cannot pass vacuously. It earned its keep immediately: it caught a
|
||
loop-variable capture (`new Thread(() => Worker(i))` where `i` was `Workers` by the time the thread
|
||
ran) that crashed the test host on first run.
|
||
|
||
### Scene compositor (TASK 4 ship step 1 — shipped 2026-08-10, plan in TASKS.md)
|
||
|
||
The encoder needs the master 1920×1080 frame **without** the preview's editing chrome (SelectionOverlay,
|
||
DimRects, output-rect outline, badge, placeholder). WPF's `RenderTargetBitmap` is software-rendered and
|
||
captures the visual tree *including* chrome, so the preview can't be captured — the output is a **second,
|
||
parallel software compositor** over the `VideoFrame` (BGRA8) seam, and the XAML preview
|
||
(`MainWindow.xaml` CanvasGrid + element DataTemplate) is the rendering contract it replicates. Two
|
||
renderers must agree: geometry, `UniformToFill` cover-crop, round clip, mirror, border, z-order. Preview
|
||
stays XAML (editing view); the compositor is the output view.
|
||
|
||
- **The `DimRects` shading is itself preview chrome, so it renders as a SIBLING of `CanvasGrid`
|
||
(above it, below the `ZoomBox`), never a descendant** (2026-10-08, creator ruling): the zoom box's
|
||
`VisualBrush` samples all of `CanvasGrid`, and if the dims were inside it the box would show the
|
||
dim shading — especially after the box is dragged away and the fixed capture slides under the dims.
|
||
|
||
- **Composite the active tier's crop at crop size** (`CompositorOptions {SourceRectX/Y/W/H,
|
||
OutputWidth, OutputHeight}`, fed from `MainViewModel.OutputRect*`): 16:9 = full 1920×1080; the
|
||
vertical 9:16 tier = centered 608×1080 crop. FFmpeg's `scale` filter resizes the raw BGRA frame
|
||
to the selected output tier after it leaves the frame pump; `setsar=1` keeps square pixels.
|
||
- **CPU posture:** with FFmpeg as a subprocess the master crosses a CPU readback to the pipe every frame
|
||
anyway, so GPU compositing buys little at this layer count (2-3 live layers; static layers
|
||
pre-composite once into a cached base). GPU effort belongs to **NVENC** (the encoder), not composition;
|
||
if composition grows (wipes, filters, many layers), a D3D11 compositor can replace this one **behind
|
||
the same seam** — the CPU master buffer stays the contract.
|
||
- **Every global overlay must be composited on EVERY render path (bug found 2026-09-26).** The
|
||
frame pump has three exits: `RenderFull` (no graph / no static base), `CompositeLayers` (static base
|
||
+ dynamic tail), and the **fully-static shortcut**, which stretches the cached bake and returned it
|
||
directly. That third path silently **discarded every per-frame overlay** — the social bar, the alert
|
||
ticker and now the brand flash. A creator whose scene was one static background saw none of them,
|
||
and the flash looked "preview-only forever" for a reason unrelated to the tier. `SceneCompositor.Overlay`
|
||
is therefore `internal static` (it copies before blitting, so the cached bake is never mutated) and
|
||
the shortcut now composites overlays onto a copy before stretching. **When adding an overlay, grep
|
||
for all three paths** — a fourth exit that forgets it fails the same silent way.
|
||
- **`VideoFrame` buffers are recycled by some producers, and `Epoch` is not optional.** A producer that
|
||
hands out the same array across frames MUST bump `Epoch`, or `FramePump`'s buffer-identity paste cache
|
||
false-hits and freezes stale content (the take-11 class of bug). The `C4` cache signature mixes
|
||
`Epoch`, so a per-frame overlay naturally invalidates it. `BrandFlashPresenter` reuses one 8MB
|
||
master buffer and stamps `Epoch` per read for exactly this reason — allocating a fresh 8MB frame per
|
||
tick is not an option at 1080p30.
|
||
- **Dev-only: two instances side by side (`Helpers/InstanceProfile.cs`).** Pre-1.0 the creator runs one
|
||
instance to stream and another to screen-capture it. Nothing prevented a second instance (no mutex, no
|
||
port); what broke it was shared state — the whole-scene read/write layout DB (clobber), Chromium's
|
||
**exclusive** lock on the WebView2 user data folder (second instance won't start), the auth token store
|
||
(a test instance overwrites the real YouTube sign-in), and `startup.log`. Set
|
||
`YTLIVE_INSTANCE=<id>` and that process gets a private root at `%APPDATA%\ytLlive\instances\<id>\`
|
||
for the DB, the auth file, the log and the WebView2 folder. Recording folder and the ffmpeg `tools\`
|
||
cache stay shared on purpose; global hotkeys stay un-namespaced (Windows refusing the second
|
||
`RegisterHotKey` is the correct answer). The **entire implementation is inside `#if DEBUG`** — a
|
||
Release build compiles to `DataRoot => DefaultRoot` and `WebViewDataFolder => null`, and the call
|
||
sites are unconditional so Release cannot drift. Proof + harness: `ytLive.Tests/InstanceIsolationTests.cs`.
|
||
⚠️ Grepping the binary for `YTLIVE_INSTANCE` proves nothing — a `const` is inlined and appears in
|
||
neither build; check the `InstanceVariable` **field** in metadata instead.
|
||
- **Branding flash is composited by the output path too** (it's on the live output, per Monetization),
|
||
passed in as a pre-rendered `VideoFrame?` — the compositor core stays pure byte-math, no WPF. Likely a
|
||
bundled asset rather than runtime text rendering (deterministic, no font/layout risk).
|
||
- **Frame sources are injected** via a `Func<SceneElement, VideoFrame?>` resolver
|
||
(`SceneCompositor.Render(scene, frameFor, flashFrame, options)`) — the caller maps each element to
|
||
its frame (webcam → `DeviceId`, image → `AssetId` via `StaticPixelCache`, backdrop → `CaptureKey`),
|
||
so the compositor is pure, WPF-free, and hermetic to test. The capture managers wire into that
|
||
resolver in the encoder step, not the compositor step. The master buffer (the compositor's return
|
||
value) is the seam a future D3D11 compositor would honor identically.
|
||
|
||
---
|
||
|
||
## The live pump — frame producer to encoder
|
||
|
||
### Live frame pipeline (TASK 4 ship step 5 — shipped 2026-08-12, health stats 2026-08-13, plan in TASKS.md)
|
||
|
||
The **`FramePump`** (`Services/Encoder/`) is the live frame producer: while live it snapshots the active
|
||
scene each tick, resolves every element to its latest frame, composites it into the tier's output frame,
|
||
and paces frames into the encoder at the tier's FPS. **rawvideo is stamped by ARRIVAL:** ffmpeg assigns
|
||
pts from frame order at the declared fps — supply rate = output speed. A starved producer (pump < fps)
|
||
ships a time-lapse, truncated file with NO error (take-2 lesson, 2026-09-01); the pump therefore logs
|
||
`FramePump stats: n/target frames per 5s from R renders, avg render Xms (resolve R2), avg submit Yms` so a slow stage names itself (the `frames … from R renders` pair and the render-count denominators arrived with UNIT 0 — see "Frame cadence" above; before that every average was divided by the SUBMITTED count, which deadline padding always fills to the full target, so a pump fed one webcam frame per 200ms still logged "300/300 frames" and its render average was diluted ~an order of magnitude) — and since 2026-09-04 so does the BUILD: every build gets a GUID stamped
|
||
into `Helpers/BuildStamp` (csproj `GenerateBuildStamp` target, fresh per compile — no lying incremental build), logged at startup
|
||
("Build xxxxxxxx (compiled ...)"). **Take 6 (stamp-less, ambiguous): render 35-41ms with slice 3 built or not unknown — attribution is why the stamp exists; takes 7+ are
|
||
readable.** If the breakdown says resolve dominates, the suspects are pre-cached chat/config thrash or the web frame; if blit dominates, the general-path element sizes. **Pattern — everything is a constructor-injected
|
||
seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<CompositorOptions>`,
|
||
`Func<EncoderOptions?>`, `Func<IFfmpegEncoder>`, `Action<string>` log, and an injectable pacing delay
|
||
(default `Task.Delay`; tests inject `Task.Yield`). The pump is free of WPF and of the capture managers.
|
||
|
||
- **`StartAsync` never throws** — the VM fires-and-forgets it from the sync command handler; failures log
|
||
+ surface via the `Failed` event. `BuildEncoderOptions` now always returns options (TASK 18) with
|
||
`StreamEnabled`/`RecordEnabled`/`RecordPath` baked from the pills + `_activeRecordPath`; the pump skips the
|
||
encoder only when `options == null`, i.e. "no output configured". `MainViewModel._rtmpUrlProvider` is that
|
||
seam — a `Func<string?>` that now yields the reusable stream's ingest URL (TASK 9, shipped 2026-08-16):
|
||
loaded from the `LayoutStore` cache at startup and set fresh by `PrepareAndStartLiveAsync` before the pump
|
||
starts. **The pump reads the URL once at startup**, which is why go-live ensures the stream BEFORE
|
||
`StartAsync`.
|
||
- **Deadline pacing + row-blit render (take-3 starvation fix, 2026-09-03):** take 3 recorded 30s into a
|
||
2.1s file and the 5s stats line named the cause in one number — `17/300 frames per 5s, avg render
|
||
258.1ms, avg submit 1.5ms`. Two defects, both fixed: (1) the pump slept the FULL interval after each
|
||
render, so period = render + interval — OBS's `video_thread` (libobs/media-io/video-io.c) pattern
|
||
replaces it: absolute `nextTick += intervalTicks` deadline, sleep only the remainder, and on overrun
|
||
skip the wait. (NOTE, slice 9: the original "skip the missed ticks (rebase)" half of this fix was
|
||
wrong — see the slice 9 correction below.) (2) the compositor did
|
||
per-pixel float sampling + `Math.Round` blending over all 2.07M master pixels (backdrop) and scanned
|
||
the whole destination per overlay (a 64px social-bar strip cost 2M iterations). `SceneCompositor` now
|
||
follows the libyuv pattern (BSD-3, chromium.googlesource.com/libyuv/libyuv — cited per the
|
||
derivative-work rule): 1:1 aligned blits take a row-walk fast path (bilinear at scale 1 + integer
|
||
offset is the identity) with per-pixel alpha branch and integer fixed-point blend; `BlitOverlay`
|
||
clips to the intersection rect; a full-cover live backdrop skips the opaque-black pre-fill. The
|
||
general per-pixel path handled scaled/round/mirrored elements. Test:
|
||
`Pump_Paces_To_The_Deadline_Compensating_Render_Cost` records the requested wait
|
||
via the pacing seam (a seam fake must genuinely await — a synchronous completed task runs the whole
|
||
pump loop on `StartAsync`'s continuation and hangs the run; MyMistakes).
|
||
- **Slice 2 of the render fix (2026-09-04, take 4):** the pacing held (file no longer truncated at
|
||
loop level) but render stayed at 58.9ms — the per-pixel row walk was still 2M managed iterations and
|
||
every tick allocated a fresh 8.3MB master buffer. Three changes: (1) **`VideoFrame.IsOpaque`** — a
|
||
producer-contract flag (only the screen-capture and webcam paths set it; DWM/MediaCapture fill alpha
|
||
255 by contract); a full-canvas, aligned, opacity-1 blit of an opaque frame is now ONE
|
||
`Buffer.BlockCopy` (~1.5ms) instead of the loop, and the black pre-fill is skipped when the backdrop
|
||
covers. (2) **integer fixed-point bilinear** in the general `BlitContent` path (row-hoisted invariants,
|
||
no divisions, no `Math.Round`) — within ±1 of the float reference, inside the ±2 test tolerance.
|
||
(3) **scratch pool in the pump** — `AcquireScratch`/`ReleaseScratch` recycle the master buffer
|
||
(max 4, keyed by length, owned-by-reference so bake-cache/social-bar/static-art arrays can never be
|
||
captured); release happens strictly AFTER `SubmitFrameAsync` returns (the write to stdin copies),
|
||
and the free-list Contains guard makes the transition Cut path safe (`BlendFrame` returns the
|
||
to-frame itself, aliasing the scratch). The per-tick `fromScene` render in the transition branch was
|
||
**dead weight** (BlendFrame uses `TransitionService.FromFrame` captured at `Start`, never the pump's) —
|
||
removed, and the pump's `fromSceneProvider` seam + `MainViewModel` call site went with it. Tests:
|
||
`Pump_Pools_ScratchBuffers_Across_Frames_Without_Stale_Pixels` (the ONE: alternating backdrop colors
|
||
pin every frame's content, repeated backing-array identity proves the pool recycles),
|
||
`Composite_OpaqueFullCover_Backdrop_CopiesEveryPixel_Into_Scratch` (memcpy branch + scratch
|
||
sentinel). `FakeEncoder.SubmitFrameAsync` now snapshots bytes like the real stdin write — holding the
|
||
reference would race legitimate recycling. Take 5 must show `avg render ≤ ~10ms, ≈300/300`.
|
||
**Vertical/output scaling moved to bundled FFmpeg (2026-11-04):** the C# compositor no longer
|
||
enlarges the 608×1080 crop to 1080×1920 on every frame. The bundled FFmpeg `scale` filter processed
|
||
300 synthetic BGRA frames in 0.447s on this machine; that is a scaler-only microbenchmark, not a
|
||
live-stream result.
|
||
- **Slice 3 — chat raster-on-change (2026-09-04, take 5):** take 5 measured `138/300, avg render
|
||
25.5ms` — the per-tick blits were cheap now, but `ResolveOutputFrame` → `RenderChatBox` ran a FULL
|
||
WPF raster (`ChatBoxRenderer`: FormattedText + `RenderTargetBitmap` + CopyPixels + channel swap)
|
||
EVERY tick whenever the message buffer was non-empty — and the buffer survives between sessions,
|
||
so even a signed-out record-only take paid it. Established answer, same as OBS text sources:
|
||
**raster on change, blit the cache every tick.** `ChatOverlayLayer` now keeps a content version
|
||
(`Messages.CollectionChanged` → `_contentVersion++`) plus a config key (box size + all Chat*
|
||
props); `RenderFrame` returns the cached `VideoFrame` by identity until either changes (the
|
||
compositor only ever reads a cached frame). Test: `ChatOverlayLayerCacheTests` (RealApp, real
|
||
renderer — `Same()` for unchanged inputs, `NotSame()` on message/config change, null on empty).
|
||
Accepted cost pending take 6: one slow tick (~15-25ms) per arriving message; if chat-burst frame
|
||
loss shows up, the next slice moves the re-render off-tick (debounced, dispatcher-side).
|
||
- **TASK 3.17 — static TextOverlay raster-on-change:** `TextOverlayRenderer.UpdatePreview` runs WPF
|
||
`FormattedText` + `RenderTargetBitmap` only on the dispatcher when text, formatting, or box size
|
||
changes. It publishes one immutable straight-alpha BGRA `VideoFrame` and a `WriteableBitmap`
|
||
copied from those same pixels; preview and output therefore share the raster. `FramePump` calls
|
||
only `RenderFrame` (a concurrent cache read), never WPF. Text is a static `Source`, so its raster
|
||
participates in the static bake or above-split path; property changes invalidate the bake, and the
|
||
full-render signature includes the resolved text frame. Its premultiplied WPF readback is
|
||
unpremultiplied before entering `VideoFrame`, whose compositor contract is straight-alpha BGRA.
|
||
- **Slice 5 — paste cache for non-opaque layers (2026-09-04, take 7/8 data):** the render/resolve
|
||
split (added in the stamp commit) finally named the last thing honestly: `resolve ≈ 0` but
|
||
`render 26-27ms` (124-135/300, ≈2.2x) — the chat fix HAD worked; the remaining cost was the
|
||
compositor re-rasterizing EVERY layer per tick even when its pixels never changed (this creator's
|
||
Live scene: chat 159k + web widget 271k + image 95k + webcam 156k px ≈ 680k samples @ ~38ns each).
|
||
OBS's actual shape: sources cache their surface, the compositor pastes. `SceneCompositor` now
|
||
routes non-opaque layers through `BlitCachedLayer`: a layer rasterizes ONCE into an element-space,
|
||
TRANSPARENT-based frame keyed by (source-array identity, source W×H, ceil'd dst rect, round,
|
||
mirror), then every later tick PASTES it (integer position, row alpha-blend, opacity applied at
|
||
paste). **Correction (2026-09-12, the web-widget "black box"):** the raster builds on a
|
||
TRANSPARENT base, but the sampler's PARTIAL-alpha branch used the opaque-dst blend — it
|
||
premultiplied the color into RGB and forced `alpha=255`. A translucent widget pixel then pasted
|
||
as opaque darkened ink (the box in recordings) while the raw-bitmap preview stayed correct.
|
||
`BlitContentRaw` now takes `transparentDst` (the raster call passes `true` and writes straight
|
||
color + straight alpha; the paste rows do the source-over). Master paths are bit-identical.
|
||
Guard: `PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk`. Producers hand out fresh immutable arrays, so array-identity keys can never serve stale
|
||
content; dict bounded at 48, cleared wholesale on overflow. Only changing content (webcam device
|
||
frames, web capture ticks, chat messages) resamples; the opaque backdrop keeps its memcpy path.
|
||
Position/opacity drags are now near-free (no resample — paste params, not cache keys).
|
||
ONE integration test: `PasteCache_RepeatRender_IsByteIdentical_And_ContentChangePropagates`
|
||
(raster-vs-paste byte equality incl. round-clip margins + new-array propagation); the whole
|
||
existing pixel suite guards sampler semantics. Take 9 must show `≈300/300, avg render ≤ ~8ms`;
|
||
if it lands there, the recording saga closes. Follow-ups unchanged: vertical-tier scale alloc,
|
||
debounced chat re-render under message bursts.
|
||
- **Slice 6 — the sleep quantum WAS the ceiling (take 9, 2026-09-04):** period measured ~37ms while
|
||
work (render+submit) was ~25 — the missing ~12ms is `Task.Delay` rounding EVERY request up to the
|
||
Windows clock tick (documented ~15.6ms default; learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.task.delay).
|
||
A frame finishing 3ms early requests a 3ms wait and sleeps a full 15.6 — capping the producer at
|
||
~27fps NO MATTER how fast the compositor ran. This is why takes 7→9 showed zero playback change
|
||
despite real render wins: the sleep floor dominated everything above it. Established fix (game-loop
|
||
canon — stackoverflow.com/questions/5441464; timeBeginPeriod — learn.microsoft.com/en-us/windows/win32/api/timeapi/nf-timeapi-timebeginperiod):
|
||
`timeBeginPeriod(1)` for the pump's lifetime (paired `timeEndPeriod` in the finally), sleep only the
|
||
BULK of the remainder (request minus 2ms), `Thread.SpinWait` the last ~2ms across the deadline;
|
||
blown deadlines still rebase. (SUPERSEDED in part: slice 9 deleted the rebase — deadlines only
|
||
advance; slice 10 replaced the burst re-write with one fresh composite per iteration.) Stats gained
|
||
`avg wait Xms` so render+submit+wait must ≈ period —
|
||
the accounting is closed, no stage can hide again. Same slice: the webcam's `IsOpaque` paste-cache
|
||
bypass was removed (it re-sampled ~156k px every tick even between identical device frames; the
|
||
cached paste beats the sampler on hits and costs the same on misses). Take 10 verdict: `≈300/300`
|
||
honest fps — if short, the wait/render split names the remaining term with no ambiguity left.
|
||
- **Slice 7 — the loop was ON the UI thread the whole time (take-8 numbers, 2026-09-04):** with
|
||
the wait finally measured, take 8 (59a02a5b) showed the impossible pair — `render 22ms, wait 10ms`
|
||
against a 16.7ms deadline: a blown deadline rebases with NO wait. The wait was the producer
|
||
**queued behind the live preview on the dispatcher**: `PumpAsync` starts from a UI command handler
|
||
and its `await` continuations inherit the UI `SynchronizationContext`, so the "WPF-free" compositor
|
||
actually rendered on the UI thread and ran only when WPF let it. OBS runs its media loops on
|
||
dedicated threads for exactly this reason. Fix: `_pumpTask = Task.Run(() => PumpAsync(...))`
|
||
(no sync context inside → continuations stay on the pool). Side effects handled, not ignored:
|
||
`StaticPixelCache.Get` now locks (pool miss-decodes race UI callers); `ChatOverlayLayer.RenderFrame`
|
||
checks its cache off-thread but MARSHALS the rare raster miss to the dispatcher (DrawingVisual/
|
||
RenderTargetBitmap are UI-thread objects) and re-validates there; pump events already marshalled.
|
||
Also: `GCLatencyMode.SustainedLowLatency` for the pump's life, stats gained `worst render Xms`
|
||
(spike visibility — bimodal averages hid them), webcam now routes through the paste cache (bypass
|
||
re-sampled 156k px even between identical device frames). Tests: `Pump_Produces_OffTheStartingContext`
|
||
(the ONE — inline-pumping SyncContext proves continuations never return to the starting thread) +
|
||
70/70. Take 9 verdict: wait ≈ true remainder (period → 16.7, n → ~300); `worst render` names any
|
||
remaining raster-miss spikes; render+submit+wait still == period — the accounting holds.
|
||
- **Slice 8 — capture buffer ring + paste-cache Epoch (take 11, 2026-09-04):** take 11 validated
|
||
the architecture — typical frames now land `work ~10ms + wait ~6.8ms = 16.7`, exactly the deadline
|
||
(212/300). The whole remaining gap is PERIODIC 35-65ms worst-render spikes that worsened across
|
||
the take (189→147 frames/5s) — the signature of gen2 GC pauses, now provable via `gen2 +N` in
|
||
every stats window. Biggest churn was structural: `ScreenCaptureFrameSource` minted a fresh
|
||
~8.3MB `byte[]` per DWM frame (~500MB/s LOH). It now rotates a shared-frame ring (4-deep at slice 8,
|
||
deepened to 8-deep after the "flash" finding — consumer holds must never outlive depth × source
|
||
period), size-matched per slot, with reused downscale row scratch. Recycled
|
||
arrays would poison the paste cache (it keys on array IDENTITY), so `VideoFrame.Epoch` —
|
||
monotonic per producer frame, 0 for fresh-array producers — joins the key. Regression test
|
||
`PasteCache_RecycledArrayWithNewEpoch_ReRasterizes_NotStaleHits` fails on the old key by
|
||
construction. The camera producer carried the same fresh-array churn (110-220MB/s at 30-60fps):
|
||
it now rotates its own 8-deep ring + Epoch (same identity rule), and the WebView2 capture
|
||
reuses a canvas scratch + 8-deep output ring + a reused WriteableBitmap instead of minting two
|
||
fresh arrays + a new bitmap per 10Hz tick. Next suspect if gen2 stays hot: the WPF preview load
|
||
itself driving gen2 — recorded as follow-up, untouched this slice.
|
||
- **Slice 9 — the "rebase" WAS the recording time-lapse (2026-09-10, the recording playback-timing
|
||
take):** slice 6's "blown deadlines still rebase" was the recording-timing bug all along. In a real
|
||
take the pump emitted `215-219/300 per 5s` (~43fps) while `render+wait == period` still looked closed —
|
||
the rebase ERASED every missed slot (deadline = wall-now again), so the missed ticks never showed in
|
||
the stats. rawvideo has no per-frame timestamps: ffmpeg muxes by frame count at `-framerate 60`, so a
|
||
43fps reality was authored into a 60fps container and **every recording played ~1.4x fast** (takes
|
||
confirmed with a WSL ticker visible in the recording: file duration 11.44s vs ~15.5s wall). Two
|
||
defects, both fixed the OBS way (`libobs/media-io/video-io.c` — the video thread NEVER resets its
|
||
deadline; every interval tick outputs ONE frame, and a late render means repeated content — judder —
|
||
never a skipped timestamp):
|
||
(1) **count-based CFR emission:** `while (GetTimestamp() >= nextTick) { SubmitFrame(frame); nextTick += intervalTicks; statFrames++; }`
|
||
submits exactly one frame per crossed slot, re-writing the current composite when the render overruns
|
||
(duplicated footage = correct duration, not a time-lapse), and `nextTick` is NEVER reset to wall-now;
|
||
(2) **`-re` removed from `FfmpegArgs`** — it was a second, fighting pacer on the rawvideo demux
|
||
(its "Resumed reading … after a lag" grew 0.79s→4.82s across the take, leaving the pump behind its own
|
||
honest-stats count). One fix covers the game background AND the webcam — both flow through this one
|
||
pump. Muxed duration is now frame-count ÷ fps == wall time by construction. Test: the pump suite stays
|
||
green unchanged — the pacing test's `<interval` wait assertion still holds because the frame emits at
|
||
the next slot boundary, not immediately. (Unrelated pre-existing failure found the same day:
|
||
`Composite_FullScene_MasterPixels` pixel (1380,700) cyan-vs-magenta — reproduces with this fix stashed,
|
||
untouched by it, recorded as a follow-up.)
|
||
- **Slice 10 — bounded encoder queue + drop policy + burned-in frame counter (2026-09-10, the
|
||
playback-pacing take):** slice 9 made the DURATION right but the CREATOR still read the recorded
|
||
ticker as "1...23...4...56..." — variable pacing. The aggregates (301/300, uniform PTS, 15.6s wall vs
|
||
15.74s file) could not see it, and the cause was finally measured in `FfmpegEncoder.SubmitFrameAsync`:
|
||
it BLOCKED on `WriteAsync(8.3MB) + FlushAsync` whenever ffmpeg lagged the pipe, and slice 9's
|
||
burst `while` loop then re-wrote that SAME composite for every slot that ticked past — frozen
|
||
content runs (the "smeared ticker"). Fixed the OBS way (the encoder queue in `libobs/obs-encoder.c`:
|
||
the encoder's thread NEVER couples back into the video thread; overload = dropped content, never a
|
||
frozen video thread):
|
||
(1) **`FfmpegEncoder.SubmitFrameAsync` is now an ENQUEUE** into a bounded `Channel<byte[]>` (cap 120 ≈
|
||
2s at 60fps) owned by a dedicated drain task with the stdin write + flush; the caller NEVER blocks on
|
||
the pipe. Pixels are copied into an `ArrayPool` buffer before enqueue (the write now happens later on
|
||
the drain thread, so the pump's scratch reusable the moment submit returns — same contract as before).
|
||
(2) **drop-on-overflow (creator's choice: freshness over coverage):** when the queue is full the NEWEST
|
||
frame is dropped and counted (`IFfmpegEncoder.DroppedFrames`, `Interlocked`); the session never freezes
|
||
or smears — it drops. Stop flushes the whole queue then EOF (`Channel.TryComplete` → drain writes
|
||
leftovers → closes stdin → ffmpeg finalizes+exits), so no accepted frame is ever lost at stop.
|
||
(3) **pump loops ONE submit per iteration** (replacing slice 9's burst): every due slot gets a FRESH
|
||
composite — no catch-up slot ever repeats frozen content; missed slots vanish as a count-based gap.
|
||
The deadline counter is still never reset to wall-now (slice 9 ruling).
|
||
(4) **burned-in frame counter (the new judge):** `_outputIndex` is burned into a 6-digit dot-matrix
|
||
strip (white box + black 5×7 glyphs, bottom-right corner) of EVERY composite before submit. The WSL
|
||
ticker is demoted because its own timers smear under Windows host load; decoding the recording and
|
||
reading the strip is clock-independent: the number advances +1 per frame and jumps by exactly the
|
||
counted drops (queue overflow OR skipped catch-up slots). Stats gained `worst submit Xms`, `dropped N`,
|
||
`stalls K`; a stall logger names any iteration > 2× interval with its render/submit split — with the
|
||
queue, submit is ~1ms, so a stall means render/resolve. The strip sits above the social bar (bar is
|
||
composited, then burned over) — a debug judge, tiny at 1080p. Test:
|
||
`Backpressure_QueueOverflow_DropsFrames_AndNeverBlocks` (the ONE: a 40ms-per-frame sink fake makes
|
||
every enqueue fill the queue; submit must return instantly, drops are counted, stop flushes exactly
|
||
submitted − dropped bytes). Full suite 289/290 (the pre-existing compositor pixel failure unchanged).
|
||
Audio untouched (follow-up). WSL ticker-under-load reliability check still to run (informational — the
|
||
burned counter is the judge).
|
||
- **Slice 11 — web-layer animation ran at ~1/6 speed; capture cadence 10Hz→~30Hz + de-throttle
|
||
(2026-09-10, the "widget animates too slow" take):** the recording is 60fps but the WebView2
|
||
capture loop was a blind 100ms `DispatcherTimer` = 10Hz — a 60fps-designed widget was sampled 6×
|
||
under its native rate (repeated-footage slow-mo). Two stacked throttles:
|
||
> Superseded by **Slice 14** (below): `CaptureScheduler` and the cadence hooks were the polling
|
||
> design; frame-driven composition capture replaced them — nothing here is current architecture.
|
||
(1) **the 10Hz cap** (unconditional, code-proven) and
|
||
(2) **Chromium hidden-page throttling** (`requestAnimationFrame` parked, JS timers clamped to ~1s
|
||
per WebView2Feedback#1172/#3070 + Chrome-88 timer throttling) whenever the app window is unfocused
|
||
or covered — the page is off-screen at (-5000,-5000), so it is a hidden page the moment the host
|
||
window loses occlusion.
|
||
Fixed:
|
||
(1) **`CaptureScheduler` (new, `Services/CaptureScheduler.cs`)** replaces the per-session
|
||
`DispatcherTimer`: a dispatcher timer that DROPS ticks while a capture is in flight (latest-wins,
|
||
never queues) — the in-flight drop is what made raising the cadence safe (concurrent full-HD PNG
|
||
`CapturePreviewAsync` calls would stack ~10-30ms encodes and publish stale-after-fresh). Interval is
|
||
owner-configurable: recording 33ms (~30Hz), idle 200ms. Unit-tested without a WebView2 runtime
|
||
(the ONE test: `CaptureScheduler_Drops_Ticks_While_Capture_InFlight_And_Resumes` — overlapping
|
||
ticks dropped, capture resumes when idle, driven by a TCS so it is deterministic).
|
||
(2) **de-throttle browser args**: the shared `CoreWebView2Environment` is created with
|
||
`--disable-backgrounding-occluded-windows --disable-renderer-backgrounding
|
||
--disable-features=CalculateNativeWinOcclusion` (the Electron/Streamlabs-class embedder answer for
|
||
occluded-window animation throttling) BEFORE `EnsureCoreWebView2Async`.
|
||
(3) **cadence hook**: `MainViewModel` sets `SetCaptureInterval(33)` on record/stream start and
|
||
`(200)` on stop (`MainViewModel.Streaming.Operations.cs`).
|
||
(4) **capture-cost telemetry**: first 30 captures per session log elapsed ms to startup.log —
|
||
PNG-encode+decode cost decides whether ~30Hz stays or drops to ~20Hz; the FramePump drops frames
|
||
(never time-lapses, slice 10) if UI-thread GC churn starves it, so the cost is observable.
|
||
Full suite 290/291 passing, the sole failure the pre-existing compositor pixel test. The web-overlay
|
||
transparency verification (the first-capture diagnostic bound to about:blank) and the audio-silence
|
||
item remain open push-gate items — both untouched by this slice.
|
||
- **Slice 12 (audio diagnostics, 2026-09-10):** the recorded audio is full-length silent AAC
|
||
(−91dB, 1124 frames / 23.95s in the latest take) — the named-pipe delivered ~9.1MB of zeros to
|
||
ffmpeg for the entire recording, so the loop ran and connected, but both WASAPI capture sources
|
||
delivered nothing. Sources started fine (`Audio: using system default mic...` logged), no failure
|
||
callbacks fired, and the mixer's existing integration test (`Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe`)
|
||
proves the loop→pipe math is sound. The fault is capture-side: either the default render endpoint
|
||
carried nothing (audio played on a non-default device — common), or both endpoints were held
|
||
exclusive, or genuinely nothing played. Added permanent per-5s live-loop telemetry to startup.log
|
||
(`Audio live: pipe connected=, dropped writes=, micLevel=, loopLevel=, drained, peakMix`) and
|
||
`NamedPipeAudioWriter.DroppedWrites` — names the exact stage on the next take without new
|
||
code. `AudioMixer.FillAndMix` now returns `(MicRms, MicDrained, LoopDrained)` for the telemetry
|
||
accumulation. Full suite 290/291, same pre-existing sole failure.
|
||
- **Slice 13 (web transparency diagnostic re-point, 2026-09-10):** the "?" in MyMistakes'
|
||
"RESOLVED (? VERIFY)" is being closed. The one-shot widget dump + alpha stats bound to the
|
||
FIRST capture ever = the initial about:blank document — a blind instrument. NavigationCompleted
|
||
for the REAL widget URL now arms `WidgetDumpRemaining = 5`; those 5 post-paint captures dump
|
||
`%TEMP%\ytLive-web-<id>-w1..5.png` + the shared `AlphaStats(...)` line (alpha[min/max/mean/zero%])
|
||
+ FindContentBounds result to startup.log. That line alone decides the fix branch: zero% ⇒ the
|
||
pre-parse injection did not hold for this widget (opaque capture → chroma-key / stronger DOM
|
||
injection); large zero% + tight contentBounds ⇒ the capture IS transparent and the recording's
|
||
black block lives downstream (compositor blend/underlay). `AlphaStats` extracted as the shared
|
||
sampler. No new tests (WebView2 runtime not instantiable in tests; the re-point is log-only).
|
||
- **Slice 14 — web frames are now COMPOSITION-CAPTURED; the PNG poll + `CaptureScheduler` are gone
|
||
(2026-09-14, the "still not 60fps" take):** slice 11's ~30Hz was still a `CapturePreviewAsync` PNG
|
||
poll — every 1920×1080 full-HD encode+decode cost 35–165ms, so the session budget capped real
|
||
cadence at ~20Hz for a 60fps-designed widget. Root fix = frame-driven capture, not faster polling:
|
||
the WebView2 renderer now feeds `Windows.Graphics.Capture` directly through a
|
||
`CoreWebView2CompositionController` (the mechanism `WebView2CompositionControl` and Flutter's
|
||
`webview_windows` use — see `graphics_context.cc` `CreateGraphicsCaptureItemFromVisual`, which
|
||
captures the root `surface_` visual).
|
||
- **Pipeline (per session):** `CreateCoreWebView2CompositionControllerAsync(WindowHandle)` — the
|
||
parent HWND must exist, so `MainWindow.InitWebView2()` moved from the ctor to **Loaded** (was
|
||
`InitWebView2(WebViewHostPanel)`; the hidden XAML `WebViewHostPanel` overlay is deleted).
|
||
`controller.RootVisualTarget` = a child `ContainerVisual` (`RelativeSizeAdjustment = 1,1`) under a
|
||
root `ContainerVisual` (1920×1080, `IsVisible=true`) built on ONE `Windows.UI.Composition.Compositor`
|
||
(created once on the UI thread); `Bounds = 0,0,1920,1080`, `BoundsMode=UseRawPixels`,
|
||
`ShouldDetectMonitorScaleChanges=false`, `RasterizationScale=1.0`, `IsVisible=true`,
|
||
`DefaultBackgroundColor=Transparent`; then `GraphicsCaptureItem.CreateFromVisual(root)`.
|
||
Frames pull from a free-threaded `Direct3D11CaptureFramePool` (2 buffers) →
|
||
`SoftwareBitmap.CreateCopyFromSurfaceAsync` (BGRA, `BitmapAlphaMode.Straight` — the compositor
|
||
blends STRAIGHT alpha, so premultiplied readback would wreck corner anti-aliasing) → per-frame
|
||
`FindContentBounds` alpha-bbox on the capture worker → `VideoFrame.CropBounds` stamped → ring up to
|
||
2 fresh + Epoch (reuse; GC lesson slice 8) → dispatcher-coalesced copy (Render priority) into ONE
|
||
crop-sized shared `WriteableBitmap`; `PreviewBitmapChanged` raised only on bitmap (re)creation.
|
||
Also unpumped: the OBS-side de-throttle flags (slice 11) stay — they fix the hidden-page JS
|
||
throttling; the capture path itself no longer polls.
|
||
- **Implemented in:** new `Services/WebCaptureFrameSource.cs` (owns item+pool+session — mirrors
|
||
`ScreenCaptureFrameSource`, which servers as the copy template) + reworked `Services/WebView2Manager.cs`
|
||
(owns controllers, the visual tree, nav/script wiring, preview; internal seam ctor
|
||
`(Dispatcher, Func<string, IScreenCaptureSource>?)` so the tests instantiate zero WinRT; public
|
||
ctor `(Dispatcher)`).
|
||
- **CoreMessaging DQ recipe (record-once):** the 19041 CsWinRT projection has NO
|
||
`DispatcherQueueController.CreateOnCurrentThread()` (CS0117 — only `CreateOnDedicatedThread` +
|
||
`FromAbi(IntPtr)`). P/Invoke `coreMessaging.dll!CreateDispatcherQueueController` with struct
|
||
`DispatcherQueueOptions { DwSize, ThreadType = 2 (DQTYPE_THREAD_CURRENT), ApartmentType = 2 (DQTAT_COM_STA) }`,
|
||
wrap via `DispatcherQueueController.FromAbi(ptr)` (mirror of `CaptureInterop`), only then
|
||
`new Compositor()` — all once on the WPF UI thread (the app dispatches Render there).
|
||
- **Compile notes (each cost a build cycle):** `Compositor` collides with the repo's OWN
|
||
`ytLive.Services.Compositor` namespace → fully-qualify `Windows.UI.Composition.Compositor`;
|
||
`CoreWebView2CompositionController` exposes **`Close()`**, not `Dispose()`; `Color` is ambiguous
|
||
(`System.Drawing` vs `System.Windows.Media`) → `System.Drawing.Color.Transparent`.
|
||
- **Dropped:** `Services/CaptureScheduler.cs` (DELETED — no cadence to schedule), all three
|
||
`SetCaptureInterval` hooks in `MainViewModel.Streaming.Operations.cs`, the XAML `WebViewHostPanel`.
|
||
`TransparentBackgroundScript` const is unchanged (a test pins it).
|
||
- **Test model (Good Dog):** reworked `ytLive.Tests/WebView2ManagerTests.cs` — dropped the 4
|
||
control-size tests + the `CaptureScheduler_Drops…` test; `FindContentBounds` tests moved to
|
||
`WebCaptureFrameSource.FindContentBounds`; the ONE integration test
|
||
`Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds` drives the internal seam with
|
||
`FakeWebSource : IScreenCaptureSource` + a real background-STA `DispatcherPump` (copied from
|
||
`ScreenCaptureManagerTests`): register → one crop-sized preview bitmap published → `CropBounds` +
|
||
`IsOpaque=false` on `GetLatestFrame` → back-to-back pumps coalesce to latest. Byte assertion compares
|
||
the DENSE 2×2 crop (`CropBytes` helper — slices source rows with stride gaps, a contiguous range
|
||
spans rows wrongly).
|
||
- Full suite **293/293 green, 0 warnings**. Audio untouched. **NOT YET VERIFIED ON DEVICE** — the
|
||
composition path needs a real 60fps-widget run (open item; see HANDOFF). Web work commits stay
|
||
LOCAL (no push) until the user greenlights.
|
||
- **Slice 15 — one frame per deadline SLOT: duplicate-on-lag, never skip (2026-09-14, device takes
|
||
ty-1723/1726):** slice 10's freshness choice — a render overrun VANISHES the missed slots from the
|
||
file — authored ACCELERATED playback: ~35ms render cost vs the 16.6ms slot, one fresh frame per
|
||
35ms (log wall-1726: `FramePump stall: iteration 33ms (> 2× the 17ms interval): worst render
|
||
33-46ms`), and a 60fps container muxed on arrival → 1723: 697 frames = 11.62s video vs 11.84s
|
||
audio; 1726: 163 frames = 2.72s vs 2.93s. The video also ended 0.21–0.24s before the audio ("audio
|
||
speeds up, cuts off at the end"). OBS's answer is duplicate-on-lag: `libobs/media-io/video-io.c`
|
||
emits one frame per tick and a lagger DUPLICATES the newest frame, counted as "lagged frames due
|
||
to rendering lag/stalls" (obs-output.c) — a wall-time hole never exists (docs.obsproject.com/
|
||
backend-design: "If the video frame queue is full, it will duplicate the last frame"). The pump's
|
||
submit is now a bounded catch-up: `while (now >= nextTick) { submit; nextTick += intervalTicks; }`
|
||
over a `deadlineNow` captured once per iteration — the LATEST composite, fresh on the first missed
|
||
slot, repeated (OBS's duplication) for the rest. Duration == wall under any render load, at the
|
||
cost of a short judder during a stall. The burst is safe because `Channel.TryWrite`
|
||
never blocks (slice 10's queue — the take-9 smear was the BLOCKING pipe-write re-copying a stale buffer
|
||
during a long freeze; here each emit is nanoseconds). The burned `_outputIndex` (slice 10 judge)
|
||
moved INSIDE the submit loop so every emitted slot carries its own +1 (and the old unconditional
|
||
pre-gate bump no longer gaps the sequence on non-submitting fast-render iterations). **Good Dog
|
||
test** `Pump_Overrun_Renders_EmitsEverySlot_NotSkipped`: 60fps, resolver sleeps 35ms, asserts ≥0.65
|
||
of the wall slots are emitted (the skip-pump wrote ~1/35ms ≈ 200 in 7s; the slot pump ~415). Full
|
||
suite **294/294 green, 0 warnings**. The +0.6s audio-late clap reading on 1726 was confounded by
|
||
the 1.1x acceleration — re-measure on device; if a real residual remains it is the audio pipeline.
|
||
No push (web/A/V work stays local).
|
||
- **Slice 16 — the desktop capture conversion was the bottleneck: fast downscale +
|
||
cadence throttle + reuse-distance ring (2026-09-14, device take ty-1742):** slice 15
|
||
fixed pacing but the 1742 desktop layer was still jerky/frozen with a horizontal
|
||
tear. Decoded to raw frames and audited: the **desktop band was frozen 21s of 23.35s
|
||
(90%)** — ~6.1 content updates/s, freeze runs up to 2.28-2.78s, dup-run max 90
|
||
frames (1.5s). The render stat (`worst render 33-36ms`) was real but MOOT: the
|
||
capture CONVERSION was the wall. The monitor delivers at the **240Hz DWM cadence**
|
||
(~4.2ms); with one-in-flight conversions and each 2560×1440→1920×1080
|
||
`DownscaleBgra` at ~150ms under load, `LatestFrame` updated a handful of times/s —
|
||
the desktop feed inside a 60fps file read ~90% frozen. (Webcam + audio were their
|
||
own paths — fine, matching the report.) Three changes, all in
|
||
`Services/ScreenCaptureFrameSource.cs` (+ `Services/FrameRingBuffer.cs`):
|
||
(1) `DownscaleBgra` is now **integer 8.8 fixed-point, "shift only at the end"** —
|
||
the exact two-stage math of `SceneCompositor.Bilinear` (MyMistakes take-4 rule),
|
||
dropping double-per-pixel to a row-walk of integer ops (the capture downscale had
|
||
stayed the naive float twin of the 258ms disaster); (2) a **10ms conversion floor**
|
||
(`MinConvertInterval`) so the 240Hz tail stops queuing ~150ms of serialized
|
||
conversion per second — the open edge sits just above the ~60/s the pump can use;
|
||
(3) the ring is a **reuse-distance pool** (`FrameRingBuffer`, depth 8, redLine 4):
|
||
a buffer is only rewritten ≥4 rents after its last hand-out else a fresh allocation,
|
||
replacing the blind round-robin — since `ScreenCaptureManager` keeps
|
||
`session.LatestFrame` across conversions and the dispatcher preview copy lags,
|
||
"who released the buffer" needs a consumer API that doesn't exist; a reuse
|
||
DISTANCE needs no cooperation (the 1742 new-top/old-bottom tear is that
|
||
read-under-write, structural now). Telemetry added: a startup.log line every 2s
|
||
(`frames/s, conv avg/max ms, skip busy/cadence, ring allocs`) so the device take
|
||
can be judged numerically. **Good Dog test**
|
||
`Ring_NoLap_ReusesOnlyAfterRedLineRents` (depth 3/redLine 4 exercises the red-line
|
||
skip; the 8/4 config settles at 8 buffers and never grows). Full suite **295/295
|
||
green, 0 warnings**. C4 (compositor Epoch-cached downscale / blit-on-change) was
|
||
DEFERRED: the slice-15 approved plan assumed a native-res relocation; the
|
||
measurement recast it — relocating a ~30ms float downscale to the render thread
|
||
just moves the same cost into the slot budget. Re-measure on device; if render
|
||
still >16.6ms slots after capture feeds ≤60 real updates/s, add C4. No push.
|
||
- **Slice 17 — the OS readback was the real wall: overlapping conversions + monotonic
|
||
publish + deeper pool (2026-09-14, device take ty-1824):** slice 16's downscale fix
|
||
landed but the desktop was still ~90% frozen on the 1824 take (band 6.8/s updates, max
|
||
freeze 4.85s). The new 2s telemetry was decisive: `conv avg 46-50ms max ~61ms` with
|
||
`skip busy 37-83` — the **GPU→CPU readback (`CreateCopyFromSurfaceAsync`), not
|
||
`DownscaleBgra`**, is the ~47ms wall (240Hz HDR compositing + encoder + 2 capture
|
||
devices share the GPU); at one-in-flight that caps the desktop feed at ~17-20
|
||
updates/s, which is the file's whole-frame ~7 content-moments/s. Two facts reshaped
|
||
the fix: **(a)** shrinking the pool size does NOT scale the desktop (Microsoft docs:
|
||
"If content is larger than the frame, the contents are **clipped**") — readback stays
|
||
at native 2560×1440; **(b)** delivery is healthy (60-100 arrivals/s), so the lever is
|
||
conversion throughput, not the pool size. Changes in `Services/ScreenCaptureFrameSource.cs`:
|
||
conversions overlap up to **MaxConcurrentConversions = 3** (pool deepened to 5 buffers
|
||
so in-flight frames fit), each completion publishes ONLY if its Epoch is strictly
|
||
newer than the last published (`MonotonicGate` — a slow older completion must never
|
||
overwrite a newer LatestFrame), Epoch increments via `Interlocked`, and the
|
||
DownscaleBgra row-scratch became per-conversion locals (concurrent callers). The
|
||
render side (33-42ms → ~30 unique composites/s) is the NEXT cap after capture speeds
|
||
up — that's the C4 slice, queued right after this re-measure. **Good Dog test**
|
||
`PublishGate_TryPublish_OnlyStrictlyNewerWins`. Full suite **296/296 green, 0
|
||
warnings**. NOT YET DEVICE-VERIFIED; target: telemetry frames/s jumps ≥ ~30-40 and the
|
||
band audit drops below ~50% frozen. No push.
|
||
- **Slice 18 — C4: the FramePump blit-on-change composite cache (2026-09-15, take ty-1841
|
||
verdict):** the 1841 take proved capture fixed (band ~20 fresh updates/s, no tears, pacing
|
||
clean) but logged a FramePump **stall on EVERY iteration** (`totalMs 21-44`,
|
||
`render=full-render split=0 elements=6 dynamic=4`, worst render 166ms startup spike) →
|
||
~22-28 composites/s hard-caps the desktop inside the 60fps file. The SceneGraph split
|
||
CANNOT fix this wired: `GetSplitPoint` returns **0** because the live-capture backdrop is
|
||
element 0 and must stay dynamic — a baked capture goes stale — so every tick runs the full
|
||
re-composite even when no input changed (the compositor's per-layer paste cache already makes
|
||
identical frames identical per-element; the waste is re-COMPOSITING the whole 8.3MB frame).
|
||
Same shape as OBS (sources cache their surface, the compositor renders on update only —
|
||
the pattern this file has cited since the 2026-09-04 paste-cache slice) at the FRAME level.
|
||
`FramePump.RenderFull` now wraps both full-render call sites (no-SceneGraph + split=0
|
||
fallback): it computes `BuildFullRenderSignature` (a deterministic hash of the crop/output
|
||
dims, the social bar identity + top, and — per element, MIRRORING the compositor's own
|
||
resolution — the element ref, layout/visual bits, and the frame each element would resolve
|
||
through the SAME resolver seam, using buffer-array identity + Epoch + CropBounds; a changed
|
||
frame forces a re-composite, an unchanged one never serves stale bytes) and, when it matches
|
||
the last composite, reuses the cache with ONE `Buffer.BlockCopy` (~3ms) instead. The cache
|
||
buffer is a **separate long-lived array** — never the scratch pool — because the caller burns
|
||
the frame counter and recycles the scratch AFTER `RenderScene` returns; the cache is written
|
||
from the rendered scratch BEFORE returning (pre-burn, pre-recycle). Engagement is gated on
|
||
the 1:1 config (`SourceRect == Output`, the only deployed tier); the vertical tier's cached
|
||
fresh-scale-buffer follow-up stays exactly where it was. Telemetry: internal `CacheHits`/
|
||
`CacheRenders` counters surface as `cache {renders}R/{hits}H` on the 5s stats line, and the
|
||
ticks' key is an internal `OutputIndex` (burned counter, stable across one tick's resolver
|
||
passes). **Good Dog test**
|
||
`FullRenderCache_StaticInputs_RenderOnce_Then_Reuse_UntilInputChanges`: static scene →
|
||
exactly ONE composite (renders==1, hits>0, byte-identical content above the burn strip),
|
||
then a new frame (new array + monotonic Epoch) invalidates and propagates. Existing pool test
|
||
now passes a STABLE scene (production hands `StagedScene`; a fresh scene per tick churns the
|
||
element refs inside the signature and hides the cache — the tests that care about cache
|
||
behavior must mirror production) and keys its red/blue alternation on `OutputIndex`/frame
|
||
count so the tick's two resolver passes (signature + render) don't double-advance a call-count
|
||
flip. Full suite **297/297 green, 0 warnings**; verify.sh scope-locked to FramePump.cs +
|
||
FramePumpTests.cs + docs. Target next take: `cache` hits dominate on TV holds, `avg render`
|
||
drops toward the BlockCopy, stalls vanish, then the band audit + clap re-measure decide push.
|
||
No push.
|
||
- **Stop ordering matters:** `StopAsync` stops the encoder — since slice 10 it FLUSHES the pending
|
||
queue (`Channel.TryComplete` → drain writes the leftovers, closes stdin → EOF → ffmpeg finalizes+exits;
|
||
an accepted frame is never lost) — **before** awaiting the loop. The old reverse-order deadlock was
|
||
a submit stuck on pipe backpressure; the drain thread owns that wait now, and stopping the encoder
|
||
first keeps the pump's own (non-blocking) submits from racing the completed queue as a spurious drop.
|
||
`ProcessFailed` self-stops the pump. `Failed` while live flips
|
||
`StreamStatus.Error` (minimal).
|
||
- **Health stats (ship step 6, shipped 2026-08-13):** `HealthUpdated` is bound to the bottom bar —
|
||
`MainViewModel.OnFramePumpHealthUpdated` marshals to the UI thread (the encoder's stderr loop raises on
|
||
a background thread) and copies into `CurrentHealth` (the bottom bar's existing `CurrentHealth.*`
|
||
bindings). `ResetHealth(status)` zeroes dropped/duration on go-live and on End so stats never linger
|
||
from a previous session; bitrate/FPS stay on the tier's targets (`ApplyStreamQuality`). The bar shows
|
||
real encoder values since TASK 9 (2026-08-16) wired the reusable stream URL into `_rtmpUrlProvider`.
|
||
- **`MainViewModel` owns the resolver** (`ResolveOutputFrame`): `WebcamSceneConfig` →
|
||
`CameraManager.GetLatestFrame(DeviceKeyForWebcam(WebcamId))` — **device-keyed** (configs carry the
|
||
identity GUID; sessions key on the Windows device id — the old direct-GUID lookup returned null
|
||
forever and the webcam was structurally absent from output; take-2 fix 2026-09-01,
|
||
`WebcamOutputKeyTests`), `Source { IsLiveCapture, CaptureKey }` →
|
||
`ScreenCaptureManager.GetLatestFrame(CaptureKey)` (the new accessor mirroring `CameraManager`), image/
|
||
background → `StaticPixelCache.Get(AssetId)`. `BuildCompositorOptions` rounds the VM's `OutputRect*`
|
||
doubles to ints — the vertical 607.5 half-pixel crop rounds to a perfectly-centered **608** (`Math.Round`,
|
||
ToEven); `BuildEncoderOptions` fills W×H/FPS/bitrate from the tier once the URL provider yields one.
|
||
- **Media source decoder (TASK 21, Increment B — 2026-08-31):** `MediaVideoSource` (`Services/MediaVideoSource.cs`)
|
||
decodes a local file to `VideoFrame`s by spawning ffmpeg with
|
||
`-f rawvideo -pix_fmt bgra -vf scale=WxH -an` and draining the raw BGRA stdout pipe through the pure
|
||
`RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raising `FrameAvailable` per frame. Implements
|
||
`IMediaFrameSource` (`Services/IMediaFrameSource.cs`: `Key`, `FrameAvailable`, `Completed`, `StartAsync`/`StopAsync`).
|
||
The subprocess sits behind `IDecodeProcess`/`FfmpegDecodeProcess` (binary-stdout mirror of the encoder's
|
||
`IEncoderProcess`), and the path comes from `IFfmpegLocator` — codec-agnostic, uses the ffmpeg already shipped,
|
||
testable with a fake process. Decode sessions are owned app-wide by `MediaVideoSourceManager`
|
||
(`Services/MediaVideoSourceManager.cs`): refcounted by `MediaPath`, `Func<string, IMediaFrameSource?>` factory
|
||
seam (mirror of `ScreenCaptureManager`), `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`,
|
||
coalescing each file's frames onto the UI dispatcher onto one shared `WriteableBitmap`. Frames emit as fast as
|
||
the pipe produces them. Wired into the live pipeline (slice 1 step 4): `ResolveOutputFrame` reads
|
||
`_mediaManager.GetLatestFrame(MediaPath)` for a `MediaSource`, and `Source.DisplaySource`
|
||
(`Models/Source.cs`) routes `VideoImageSource` for `MediaSource` so the preview/canvas show decoded video.
|
||
Slice 2a shipped the probe seam: `FfmpegFrameRateParser` (pure, `avg_frame_rate=` then
|
||
`r_frame_rate=` rational N/N/M, unknown→null) + `IFrameRateProbe`/`FfmpegFrameRateProbe` (derives sibling `ffprobe.exe`
|
||
from the located ffmpeg dir, reuses the `IDecodeProcess` seam for the ffprobe subprocess text; null if
|
||
ffprobe absent) + `FfmpegLocator.ProbeFileName` now also extracts `ffprobe.exe` from the pinned archive
|
||
(conditional — old caches without it just get no pacing). Slice 2b wired pacing: `MediaVideoSource`
|
||
takes optional `IFrameRateProbe?` + `Func<TimeSpan,CancellationToken,Task>? delay` (default `Task.Delay`)
|
||
seams, probes FPS once in `RunAsync`, and delays by 1/fps after each emitted frame; no probe/unknown → no
|
||
pacing (ffmpeg's own pipe backpressure already throttles the decode). Slice 3 added loop control:
|
||
`IMediaFrameSource.Looping` (bool); `MediaVideoSource` takes a `Func<IDecodeProcess>` process factory (a single
|
||
`Process` can't be re-`Start()`ed, so each loop pass creates a fresh decoder) and wraps the decode in a
|
||
`do…while (Looping)` — restart on natural EOF instead of raising `Completed`. Production wiring (MainViewModel
|
||
media factory): passes `() => new FfmpegDecodeProcess()` as the factory, `FfmpegFrameRateProbe(new FfmpegLocator(),
|
||
() => new FfmpegDecodeProcess())` as the probe.
|
||
Still open: session acquisition on add/remove (UI picker) + wiring `Source.MediaIsLooping` into
|
||
`IMediaFrameSource.Looping` (needs a manager-level per-path loop provider, comes with the picker slice).
|
||
**Acquisition shipped 2026-11-05:** `AddMediaCommand` (`MainViewModel.Media.cs`) opens a video-filtered
|
||
`OpenFileDialog` and `AddMediaSource(path)` adds a `MediaSource` layer through the ordinary `AddSource`
|
||
path, defaulting to a full-canvas 1920x1080 rect, then `RegisterMediaSource` subscribes the layer and
|
||
`EnsureMediaSession` starts one decoder per **distinct path** (`HasSession` guard keeps refcounts at 0/1).
|
||
Remove (`RemoveElement`) unsubscribes and releases only when no scene references the file any more;
|
||
`ReacquireMediaSources` runs after `LoadLayout` so persisted layers get sessions and stale files are
|
||
released (`GetActiveKeys`). Loop is a manager-level per-path answer: the ctor takes
|
||
`Func<string,bool>? loopProvider` (production: "any live media reference loops"), applied before
|
||
`StartAsync`, and `SetLooping(key, bool)` applies a live toggle. A file swap releases the old path after
|
||
the layer no longer references it. **Sound ships with it (2026-11-05):** both `MediaVideoSource` and
|
||
`AlertClipDecoder` are now thin wrappers over one `Services/VideoClipPipe.cs` (video pipe + `f32le`
|
||
audio pipe, both real-time paced, loop restarts both), so a clip is a clip. The manager applies the
|
||
layer's Level/Mute per session and raises `AudioAvailable`, and `MainViewModel` feeds it to
|
||
`AudioMixer.EnqueueDesktopMediaAudio` — the DESKTOP/GAME ring, so the desktop fader and the voice
|
||
ducker apply. The alert clip keeps its own unity-gain ring (`EnqueueAlertAudio`, never ducked).
|
||
**Green/blue screen (TASK 64)** rides the same pipe as a decode option: `VideoFilterChain.BuildVideo`
|
||
appends `colorkey=…` + `despill=…` + `format=bgra` after the scale, so the key is computed by ffmpeg
|
||
and both consumers (shared preview bitmap, compositor frame) carry the same alpha — no app-side
|
||
per-pixel pass, no second render path. A media session is therefore keyed by **path + look**
|
||
(`MainViewModel.MediaSessionKey`), so two layers with identical settings share one decoder and two
|
||
different looks get two. The offer comes from `ChromaKeyDetector` over one probed frame
|
||
(`FfmpegChromaProbe`); settings persist on the `Source` row.
|
||
- **Social bar on the output (bar bug-fix branch):** the `FramePump` takes an optional
|
||
`socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam, re-read **every frame** (so a
|
||
mid-stream position flip applies immediately). The strip is pre-rasterized by `Compositor/SocialBarRenderer.cs`
|
||
(WPF glue — WPF glue here is fine because the strip is rendered once per config change on the UI thread,
|
||
the resulting immutable frame is then composited pure-CPU by `SceneCompositor`), and `SceneCompositor.Render`
|
||
blits it **last — above the branding flash** at `socialBarTop` (0 = top, `SourceRectHeight − barHeight` =
|
||
bottom) in master space. `MainViewModel` owns the frame (`_socialBarFrame`, rebuilt by `RenderSocialBarFrame`
|
||
on load/save/`NotifySocialsChanged`).
|
||
- **Privacy blur layer (2026-11-01):** `SourceType.Blur` is a Live-scene element that blurs the
|
||
composed frame inside a rect or ellipse. **It is universal**: `SceneCompositor` applies the blur
|
||
group AFTER every layer has been composited (before the flash/social/ticker overlays), so it can
|
||
cover the webcam, chat, alerts — anything — which is the creator's "moment of privacy" case. Its
|
||
position in the layer list is cosmetic for that reason (it is added at the top), and the earlier
|
||
"never above the brand" rule is superseded for blurs: a blur over the logo does obscure it. `SceneCompositor` treats all visible blurs as ONE group the first time one is
|
||
reached in the z-order (`ApplyBlurLayers`), sorts them **strongest first** and passes a shared
|
||
coverage map to `RegionBlur.ApplyInPlace`, so an overlap resolves to the strongest single blur
|
||
rather than a blurred blur. `RegionBlur` shares the webcam's radius curve and constants
|
||
(`Downscale 4`, `MinRadius 1`, `MaxRadius 6`, `RadiusFor(strength)`), so a strength means the
|
||
same thing in both features; the webcam's `BackgroundEffectProcessor` delegates to it.
|
||
Blur is `ElementKind.Dynamic`, so it is never baked into the static base and every render path
|
||
(pump, preview, thumbnails, scene dump) gets it from the one compositor.
|
||
- Known consideration: the pump reads the active scene on a background thread while the UI can still edit
|
||
it; a concurrent-mutation exception is contained (logged + `Failed` + the pump stops) rather than
|
||
crashing. The background thread + video pipeline is the new reality since this step.
|
||
|
||
---
|
||
|
||
## Related files
|
||
|
||
This file owns the compositor and the pump. The neighbours live elsewhere:
|
||
|
||
- [`output.md`](output.md) — **live encoder + RTMP push**, `IFfmpegEncoder`/`FfmpegEncoder`, `FfmpegArgs`, locator and bundle.
|
||
- [`TASKS/task-21-media-source.md`](../../TASKS/task-21-media-source.md) — **media source**, `MediaVideoSource` (ffmpeg `rawvideo` decode) feeding the pump's resolver.
|
||
- [`overlays.md`](overlays.md) — **social bar** and **native events & alerts**, the overlay strips the compositor blits last.
|
||
- [`environment.md`](environment.md) — **Run** and **Tests**, how to launch in Release and how the suite is run.
|