perf(capture): slice 8 — buffer ring + paste-cache Epoch; gen2 visibility (take-11 spikes)
Take 11 (c10ce06c) validated the off-UI architecture: typical frames land work ~10ms + wait ~6.8ms = 16.7 exactly on the deadline; 212/300 best yet. The ENTIRE remaining gap is periodic 35-65ms render spikes that WORSENED across the take (189 -> 147) — the signature of gen2 GC pauses. Biggest churner is structural: the screen capture minted a fresh ~8.3MB byte[] per DWM frame (~500MB/s of LOH), a producer OBS never does (it owns fixed surface pools). - ScreenCaptureFrameSource: 4-deep buffer ring with size-matched slots (a <=17ms consumer cannot be lapped at 60Hz) + reused downscale row scratch. - VideoFrame.Epoch: monotonic per producer frame. The paste cache keys on array IDENTITY, so recycled arrays MUST be distinguished — epoch joins the PasteKey. Producers handing fresh arrays leave it 0 (key unchanged effect). - Stats print 'gen2 +N' per 5s window: next take acquits or convicts GC without another guess (rule: prove the stage). - Test (the ONE): PasteCache_RecycledArrayWithNewEpoch_ReRasterizes_NotStaleHits — same array, new content, bumped epoch; fails on the old key by construction. 37/37 compositor/pump, clean build. - Next suspect if gen2 stays hot: the 10Hz WebView2 capture (full-canvas PNG decode + fresh arrays on the UI thread) — recorded, untouched. Creator audio ask queued in the same working session (+40% post-mix master gain before the -1dBFS limiter) lands as its own commit next.
This commit is contained in:
+18
-6
@@ -60,12 +60,24 @@ RealApp boot-smoke. Scope-check passed.
|
|||||||
`avg wait` so render+submit+wait ≈ period (accounting closed — nothing can hide). Webcam now
|
`avg wait` so render+submit+wait ≈ period (accounting closed — nothing can hide). Webcam now
|
||||||
routes through the paste cache too (bypass re-sampled 156k px even between identical device
|
routes through the paste cache too (bypass re-sampled 156k px even between identical device
|
||||||
frames). 52/52 per-class green, clean build 0 warnings, committed this slice.
|
frames). 52/52 per-class green, clean build 0 warnings, committed this slice.
|
||||||
4. **Take 11 (user, ~30s record-only):** read the superscript; expect `≈300/300 frames, wait ≈ the true remainder, worst render` now visible. If ~300: playback must be honest 1x — saga CLOSED, Unit B starts. If still ~250-280 with worst-render spikes: raster-miss spikes (web capture ~ every second) — next slice is pre-rasterizing on content change rather than on first-tick-after-change (cache the miss behind a swap-in). If wait is STILL large: the context theory was wrong and I have egg to eat — re-instrument, don't guess.
|
4. **Take 11 ran + slice 8 (2026-09-04, committed with the audio unit below):** off-UI loop
|
||||||
~22, wait ~0-3` — honest 60fps IF render+submit ≤ ~16.7. If wait is near zero and n/300 sits at
|
WORKED — typical frames land exactly on the deadline (work ~10 + wait ~6.8 = 16.7; 212/300).
|
||||||
~200, the remaining gap is pure render 22ms → next slice = per-phase compositor timing (the stats
|
Remaining gap = periodic 35-65ms spikes growing across the take = gen2 GC pauses; the capture
|
||||||
can split blit phases the same way they split resolve; that is the honest path, not a guess).
|
path minted a fresh ~8.3MB byte[] per DWM frame (~500MB/s LOH). Slice 8: 4-deep buffer ring
|
||||||
If take 10 lands at ~300/300 → recording saga CLOSED.
|
in ScreenCaptureFrameSource (size-matched slots, reused row scratch) + `VideoFrame.Epoch`
|
||||||
5. **Unit B — the top bar + session logic (user spec 2026-09-04 re-sent twice + decisions settled in Q&A):**
|
joins the paste-cache key so recycled arrays can never false-hit + `gen2 +N` per stats window
|
||||||
|
(suspect-or-acquit — never guess at the tail again). Test `PasteCache_RecycledArrayWithNewEpoch_
|
||||||
|
ReRasterizes_NotStaleHits` fails on the old key by construction. 37/37 compositor/pump green.
|
||||||
|
4b. **AUDIO (creator ask 2026-09-04):** post-mix volume up ~40% — constant master gain applied
|
||||||
|
in AudioMixer.FillAndMix BEFORE the −1 dBFS limiter (limiter still owns the ceiling: the
|
||||||
|
gain can't add clipping, it just makes the limiter bind sooner on hot material). If the
|
||||||
|
creator reports pumping on loud game audio → the constant is the tuning knob (or per-input
|
||||||
|
defaults need raising instead — discuss with data).
|
||||||
|
5. **Take 12 (user, ~30s record + talk):** stats: `gen2 +N` should be ~0-1/window, `worst render`
|
||||||
|
→ ~16-20ms, n/300 → 300; playback honest speed; file audio ~40% hotter than before. If gen2
|
||||||
|
still >2/window: next suspect is the 10Hz WebView2 capture (full-canvas PNG decode + fresh
|
||||||
|
arrays on the UI thread) — throttle or move it, cited pattern first.
|
||||||
|
6. **Unit B — the top bar + session logic (user spec 2026-09-04 re-sent twice + decisions settled in Q&A):**
|
||||||
- Two-line top bar. Line 1: center = REC + **LIVE** pills (text renamed from ON-AIR; pills become
|
- Two-line top bar. Line 1: center = REC + **LIVE** pills (text renamed from ON-AIR; pills become
|
||||||
mutually-exclusive RADIOS — record-OR-stream ruling), right = avatar + **Login/Logout** button
|
mutually-exclusive RADIOS — record-OR-stream ruling), right = avatar + **Login/Logout** button
|
||||||
(no account status light). Line 2: centered primary **Start** (grayed while NO pill armed —
|
(no account status light). Line 2: centered primary **Start** (grayed while NO pill armed —
|
||||||
|
|||||||
@@ -19,10 +19,11 @@ public sealed class SceneCompositor
|
|||||||
// chat/web/image layers are identical frame after frame. Same shape as OBS (the
|
// chat/web/image layers are identical frame after frame. Same shape as OBS (the
|
||||||
// source surface is cached, the compositor blits): a scaled/round-clipped/mirrored
|
// source surface is cached, the compositor blits): a scaled/round-clipped/mirrored
|
||||||
// layer rasterizes once and is pasted (row-copy/blend) on every later tick.
|
// layer rasterizes once and is pasted (row-copy/blend) on every later tick.
|
||||||
// Keyed by the SOURCE FRAME IDENTITY — every producer (capture, webcam, chat
|
// Keyed by the SOURCE FRAME IDENTITY (array + Epoch) — producers hand out fresh
|
||||||
// renderer, web capture, static cache) hands out immutable byte[], so "same pixels
|
// immutable byte[], or recycled-ring arrays whose monotonic Epoch distinguishes
|
||||||
// array + same target rect" ⇒ same raster. Bounds-checked eviction clears wholesale.
|
// each generation (screen capture, take-11 fix), so "same array+epoch + same
|
||||||
private sealed record PasteKey(byte[] Pixels, int SrcW, int SrcH, int DstW, int DstH, bool Round, bool Mirror);
|
// target rect" ⇒ same raster. Bounds-checked eviction clears wholesale.
|
||||||
|
private sealed record PasteKey(byte[] Pixels, long Epoch, int SrcW, int SrcH, int DstW, int DstH, bool Round, bool Mirror);
|
||||||
private static readonly Dictionary<PasteKey, VideoFrame> _pasteCache = new();
|
private static readonly Dictionary<PasteKey, VideoFrame> _pasteCache = new();
|
||||||
private static readonly object _pasteGate = new();
|
private static readonly object _pasteGate = new();
|
||||||
private const int MaxPasteEntries = 48;
|
private const int MaxPasteEntries = 48;
|
||||||
@@ -284,7 +285,7 @@ public sealed class SceneCompositor
|
|||||||
VideoFrame? raster = null;
|
VideoFrame? raster = null;
|
||||||
lock (_pasteGate)
|
lock (_pasteGate)
|
||||||
{
|
{
|
||||||
key = new PasteKey(frame.BgraPixels, frame.Width, frame.Height, dw, dh, isRound, mirror);
|
key = new PasteKey(frame.BgraPixels, frame.Epoch, frame.Width, frame.Height, dw, dh, isRound, mirror);
|
||||||
_pasteCache.TryGetValue(key, out raster);
|
_pasteCache.TryGetValue(key, out raster);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -30,6 +30,18 @@ public sealed class ScreenCaptureFrameSource : IScreenCaptureSource
|
|||||||
private bool _framePending;
|
private bool _framePending;
|
||||||
private DateTime _lastErrorLog = DateTime.MinValue;
|
private DateTime _lastErrorLog = DateTime.MinValue;
|
||||||
|
|
||||||
|
// Buffer recycling (take-11 spikes, 2026-09-04): a fresh ~8.3MB byte[] per DWM
|
||||||
|
// frame ≈ 500MB/s of LOH churn — the gen2 pauses it forces surfaced as the
|
||||||
|
// "worst render 35-65ms" spikes that capped fps at ~42 long after the compositor
|
||||||
|
// itself was fast. A 4-deep ring rotated round-robin is never lapped by a
|
||||||
|
// ≤17ms consumer at 60Hz; each hand-out carries an Epoch so identity-keyed
|
||||||
|
// consumers (the compositor's paste cache) cannot false-hit a recycled array.
|
||||||
|
private readonly byte[]?[] _frameRing = new byte[4][];
|
||||||
|
private int _ringNext;
|
||||||
|
private long _epoch;
|
||||||
|
private byte[]? _row0;
|
||||||
|
private byte[]? _row1;
|
||||||
|
|
||||||
// The composition master frame (see ai.md "Resolution tiers"): the background
|
// The composition master frame (see ai.md "Resolution tiers"): the background
|
||||||
// is an input layer, so we never hold a CPU frame bigger than the master.
|
// is an input layer, so we never hold a CPU frame bigger than the master.
|
||||||
private const int MaxBackgroundWidth = 1920;
|
private const int MaxBackgroundWidth = 1920;
|
||||||
@@ -157,7 +169,26 @@ public sealed class ScreenCaptureFrameSource : IScreenCaptureSource
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private static VideoFrame CopyToVideoFrame(SoftwareBitmap bitmap)
|
private byte[] RentRingBuffer(int size)
|
||||||
|
{
|
||||||
|
for (var tries = 0; tries < _frameRing.Length; tries++)
|
||||||
|
{
|
||||||
|
var idx = (_ringNext + tries) % _frameRing.Length;
|
||||||
|
var buf = _frameRing[idx];
|
||||||
|
if (buf is { Length: var len } && len == size)
|
||||||
|
{
|
||||||
|
_ringNext = (idx + 1) % _frameRing.Length;
|
||||||
|
return buf;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
var slot = _ringNext;
|
||||||
|
_ringNext = (slot + 1) % _frameRing.Length;
|
||||||
|
var fresh = new byte[size];
|
||||||
|
_frameRing[slot] = fresh;
|
||||||
|
return fresh;
|
||||||
|
}
|
||||||
|
|
||||||
|
private VideoFrame CopyToVideoFrame(SoftwareBitmap bitmap)
|
||||||
{
|
{
|
||||||
var sw = bitmap.PixelWidth;
|
var sw = bitmap.PixelWidth;
|
||||||
var sh = bitmap.PixelHeight;
|
var sh = bitmap.PixelHeight;
|
||||||
@@ -178,21 +209,23 @@ public sealed class ScreenCaptureFrameSource : IScreenCaptureSource
|
|||||||
var dw = Math.Max(1, (int)(sw * scale));
|
var dw = Math.Max(1, (int)(sw * scale));
|
||||||
var dh = Math.Max(1, (int)(sh * scale));
|
var dh = Math.Max(1, (int)(sh * scale));
|
||||||
// DWM delivers an opaque surface (alpha 255); bilinear keeps it 255.
|
// DWM delivers an opaque surface (alpha 255); bilinear keeps it 255.
|
||||||
return new VideoFrame(dw, dh, DownscaleBgra(data, sw, sh, srcStride, dw, dh)) { IsOpaque = true };
|
var scaled = RentRingBuffer(dw * dh * 4);
|
||||||
|
return new VideoFrame(dw, dh, DownscaleBgra(data, sw, sh, srcStride, dw, dh, scaled))
|
||||||
|
{ IsOpaque = true, Epoch = ++_epoch };
|
||||||
}
|
}
|
||||||
|
|
||||||
var pixels = new byte[count];
|
var pixels = RentRingBuffer(count);
|
||||||
Marshal.Copy(data, pixels, 0, pixels.Length);
|
Marshal.Copy(data, pixels, 0, pixels.Length);
|
||||||
return new VideoFrame(sw, sh, pixels) { IsOpaque = true };
|
return new VideoFrame(sw, sh, pixels) { IsOpaque = true, Epoch = ++_epoch };
|
||||||
}
|
}
|
||||||
|
|
||||||
// Bilinear downscale to the master frame. Reads each source row pair through
|
// Bilinear downscale to the master frame. Reads each source row pair through
|
||||||
// Marshal.Copy (no unsafe), writing tightly packed BGRA output.
|
// Marshal.Copy (no unsafe), writing tightly packed BGRA output.
|
||||||
private static byte[] DownscaleBgra(IntPtr src, int sw, int sh, int srcStride, int dw, int dh)
|
private byte[] DownscaleBgra(IntPtr src, int sw, int sh, int srcStride, int dw, int dh, byte[] dst)
|
||||||
{
|
{
|
||||||
var row0 = new byte[srcStride];
|
// Row scratch is per-capture-thread and reused across frames (same churn lesson).
|
||||||
var row1 = new byte[srcStride];
|
var row0 = _row0 != null && _row0.Length >= srcStride ? _row0 : (_row0 = new byte[srcStride]);
|
||||||
var dst = new byte[dw * dh * 4];
|
var row1 = _row1 != null && _row1.Length >= srcStride ? _row1 : (_row1 = new byte[srcStride]);
|
||||||
var xs = sw / (double)dw;
|
var xs = sw / (double)dw;
|
||||||
var ys = sh / (double)dh;
|
var ys = sh / (double)dh;
|
||||||
|
|
||||||
|
|||||||
@@ -13,6 +13,13 @@ public sealed class VideoFrame
|
|||||||
public byte[] BgraPixels { get; }
|
public byte[] BgraPixels { get; }
|
||||||
public int Stride => Width * 4;
|
public int Stride => Width * 4;
|
||||||
|
|
||||||
|
/// <summary>Producer frame epoch — a monotonically increasing number stamped by
|
||||||
|
/// sources that RECYCLE their pixel buffers (the screen-capture ring, take-11 spike
|
||||||
|
/// fix). Consumers that key caches on buffer identity (the compositor's paste cache)
|
||||||
|
/// MUST include it, or a recycled array false-hits with stale content. Producers
|
||||||
|
/// that hand out fresh arrays per frame leave it 0 — identity alone is then enough.</summary>
|
||||||
|
public long Epoch { get; init; }
|
||||||
|
|
||||||
/// <summary>Producer contract: every alpha byte in <see cref="BgraPixels"/> is 255
|
/// <summary>Producer contract: every alpha byte in <see cref="BgraPixels"/> is 255
|
||||||
/// (DWM capture surfaces and MediaCapture video carry no alpha — the OS fills 255).
|
/// (DWM capture surfaces and MediaCapture video carry no alpha — the OS fills 255).
|
||||||
/// Lets the compositor take a straight-copy fast path for a full-canvas opaque
|
/// Lets the compositor take a straight-copy fast path for a full-canvas opaque
|
||||||
|
|||||||
@@ -794,6 +794,18 @@ seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<Compos
|
|||||||
(the ONE — inline-pumping SyncContext proves continuations never return to the starting thread) +
|
(the ONE — inline-pumping SyncContext proves continuations never return to the starting thread) +
|
||||||
70/70. Take 9 verdict: wait ≈ true remainder (period → 16.7, n → ~300); `worst render` names any
|
70/70. Take 9 verdict: wait ≈ true remainder (period → 16.7, n → ~300); `worst render` names any
|
||||||
remaining raster-miss spikes; render+submit+wait still == period — the accounting holds.
|
remaining raster-miss spikes; render+submit+wait still == period — the accounting holds.
|
||||||
|
- **Slice 8 — capture buffer ring + paste-cache Epoch (take 11, 2026-09-04):** take 11 validated
|
||||||
|
the architecture — typical frames now land `work ~10ms + wait ~6.8ms = 16.7`, exactly the deadline
|
||||||
|
(212/300). The whole remaining gap is PERIODIC 35-65ms worst-render spikes that worsened across
|
||||||
|
the take (189→147 frames/5s) — the signature of gen2 GC pauses, now provable via `gen2 +N` in
|
||||||
|
every stats window. Biggest churn was structural: `ScreenCaptureFrameSource` minted a fresh
|
||||||
|
~8.3MB `byte[]` per DWM frame (~500MB/s LOH). It now rotates a 4-deep ring (a ≤17ms consumer can
|
||||||
|
never be lapped at 60Hz), size-matched per slot, with reused downscale row scratch. Recycled
|
||||||
|
arrays would poison the paste cache (it keys on array IDENTITY), so `VideoFrame.Epoch` —
|
||||||
|
monotonic per producer frame, 0 for fresh-array producers — joins the key. Regression test
|
||||||
|
`PasteCache_RecycledArrayWithNewEpoch_ReRasterizes_NotStaleHits` fails on the old key by
|
||||||
|
construction. Next suspect if gen2 stays hot: the 10Hz WebView2 capture loop (full-canvas PNG
|
||||||
|
decode + fresh arrays on the UI thread) — recorded as follow-up, untouched this slice.
|
||||||
- **Stop ordering matters:** `StopAsync` stops the encoder (closes stdin → EOF → ffmpeg finalizes+exits)
|
- **Stop ordering matters:** `StopAsync` stops the encoder (closes stdin → EOF → ffmpeg finalizes+exits)
|
||||||
**before** awaiting the loop, because closing stdin unblocks a write stuck on pipe backpressure — the
|
**before** awaiting the loop, because closing stdin unblocks a write stuck on pipe backpressure — the
|
||||||
reverse order would deadlock. `ProcessFailed` self-stops the pump. `Failed` while live flips
|
reverse order would deadlock. `ProcessFailed` self-stops the pump. `Failed` while live flips
|
||||||
|
|||||||
@@ -211,6 +211,49 @@ public class SceneCompositorTests
|
|||||||
AssertColor(third, 101, 101, 255, 0, 0);
|
AssertColor(third, 101, 101, 255, 0, 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>The ONE integration test for the capture buffer ring + paste-cache epoch
|
||||||
|
/// (take-11 spike fix): a recycled capture buffer hands out the SAME byte[] with
|
||||||
|
/// different content; only VideoFrame.Epoch can keep the paste cache honest. Without
|
||||||
|
/// the epoch in the key, frame 2 false-hits frame 1's raster and the recording shows
|
||||||
|
/// stale pixels for as long as the array keeps cycling — the exact bug this pins.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void PasteCache_RecycledArrayWithNewEpoch_ReRasterizes_NotStaleHits()
|
||||||
|
{
|
||||||
|
var pixels = new byte[64 * 32 * 4];
|
||||||
|
for (var i = 0; i < pixels.Length; i += 4)
|
||||||
|
{
|
||||||
|
pixels[i] = 0; pixels[i + 1] = 0; pixels[i + 2] = 255; pixels[i + 3] = 255; // red
|
||||||
|
}
|
||||||
|
var gen1 = new VideoFrame(64, 32, pixels) { IsOpaque = true, Epoch = 1 };
|
||||||
|
|
||||||
|
var webcam = new WebcamSceneConfig { X = 4, Y = 4, Width = 32, Height = 16 };
|
||||||
|
var scene = new Scene { Name = "Live" };
|
||||||
|
scene.Elements.Add(webcam);
|
||||||
|
var options = new CompositorOptions
|
||||||
|
{
|
||||||
|
SourceRectX = 0, SourceRectY = 0, SourceRectWidth = 128, SourceRectHeight = 64,
|
||||||
|
OutputWidth = 128, OutputHeight = 64,
|
||||||
|
};
|
||||||
|
|
||||||
|
var compositor = new SceneCompositor();
|
||||||
|
var out1 = compositor.Render(scene, _ => gen1, null, options);
|
||||||
|
AssertColor(out1, 20, 12, 255, 0, 0);
|
||||||
|
|
||||||
|
// Generation 2: the SAME array, new content, bumped epoch (what the capture
|
||||||
|
// ring does 60 times a second).
|
||||||
|
for (var i = 0; i < pixels.Length; i += 4)
|
||||||
|
{
|
||||||
|
pixels[i] = 0; pixels[i + 1] = 255; pixels[i + 2] = 0; pixels[i + 3] = 255; // green
|
||||||
|
}
|
||||||
|
var gen2 = new VideoFrame(64, 32, pixels) { IsOpaque = true, Epoch = 2 };
|
||||||
|
var out2 = compositor.Render(scene, _ => gen2, null, options);
|
||||||
|
AssertColor(out2, 20, 12, 0, 255, 0); // a stale hit would still be red
|
||||||
|
|
||||||
|
// And the same epoch twice still hits the cache (idempotent paste).
|
||||||
|
var out3 = compositor.Render(scene, _ => gen2, null, options);
|
||||||
|
Assert.Equal(out2.BgraPixels, out3.BgraPixels);
|
||||||
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void Render_VerticalTier_Outputs_1080x1920_From_The_Center_Crop()
|
public void Render_VerticalTier_Outputs_1080x1920_From_The_Center_Crop()
|
||||||
{
|
{
|
||||||
|
|||||||
Reference in New Issue
Block a user