Per-element composite timing + scene dump (measurement only)
The cadence work ended with a question the log could not answer: the compositor
manages 6.8 composites/s at ~141ms while the 10:03 take did 29-34ms on the "same
scene shape" (full-render split=0 elements=6 dynamic=5). Nothing in the stats line
said WHICH of the six elements cost what, and nothing recorded the scene itself, so
the two takes could only be compared by eye. This closes that gap and nothing else:
no fix is attempted until a real take names the culprit.
- Services/Compositor/CompositeTimings.cs (new): per-window collector.
Add(stage, ticks) accumulates; Describe(renders, minMs = 0.1) prints the averaged
breakdown slowest-first, dropping sub-0.1ms noise.
- SceneCompositor.Render / CompositeLayers / BakeStaticBase / BlitBakedElement /
BlitCachedLayer + SceneGraph.GetBakedBase: trailing optional CompositeTimings?
Every stage is timed (backdrop, base, flash, social, ticker, zoom, scale, overlay,
bake backdrop, bake overlays) and EVERY element gets its own entry keyed by
z-index + name ("#2 Webcam"), with the nested "raster" sub-stage recording the
paste-cache rasterize miss - the split between "re-samples every tick" and "slow
paste" without a second session. Bake-pass elements are prefixed "bake #..." so a
one-off rebake cannot masquerade as a per-composite cost. Callers that pass
nothing (thumbnails, existing tests) are unaffected and pay nothing.
- FramePump: owns the collector, threads it through RenderScene/RenderFull/
GetBakedBase/CompositeLayers, and logs "FramePump parts (N renders/5s): ..." once
per 5s window beside the stats line (averaged per render, so it compares directly
with "avg render"), then resets it. Also dumps the scene - "FramePump scene dump:"
header plus one line per element (z-index, name, kind, source type, visibility,
rect, opacity, clip) - at pump start and on every scene change.
- One Good Dog integration test (ytLive.Tests/CompositeTimingTests.cs): a real
composite of a full-canvas element plus a 64x64 one through the real collector
must rank the full-canvas element first, carry the raster sub-stage and the fixed
stages, and the dump must name both elements with z-index/type/rect and echo the
split point.
Derivative work (AGENTS.md): OBS's answer to "which source is slow" is the Source
Profiler - average execution time of each source's tick and render functions
(https://docs.obsproject.com/reference-libobs-util-source-profiler, util/profiler.h).
Same shape here: one timer per element per composite, reported as an average, plus
the stage split this case needs.
Docs in the same commit: ai.md (new subsection "Per-element composite timing + scene
dump", the closed gap corrected in "Frame cadence"), Services/index.md (new file row
+ FramePump/SceneCompositor rows), TASKS.md (unit marked shipped, next = creator
take), HANDOFF.md (rewritten: unpushed commit, next step, verify.sh whitelist hole).
Also recorded this session (from the creator): the launch command is
`dotnet run -c Release` - Release is the DEFAULT, Debug only when explicitly named,
and perf takes must read Release. ai.md -> Run updated.
Full suite 428 tests, 0 warnings; the only failures are the known audio env flakes
(both pass in isolation). Scope check passed.
This commit is contained in:
@@ -0,0 +1,76 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Diagnostics;
|
||||
using System.Linq;
|
||||
|
||||
namespace ytLive.Services.Compositor;
|
||||
|
||||
/// <summary>
|
||||
/// Per-stage composite timing (webcam-cadence unit, 2026-10-16): wall time per render
|
||||
/// pass, split into named stages — the backdrop, every element (keyed by z-index +
|
||||
/// name), the global overlays, and the final output scale. The pump owns one instance
|
||||
/// for the whole session, passes it into <see cref="SceneCompositor.Render"/> /
|
||||
/// <c>CompositeLayers</c> / <c>BakeStaticBase</c>, and logs an averaged breakdown every
|
||||
/// stats window — the "which of the 6 elements eats ~140ms" reading the old stats line
|
||||
/// could not give (it reported render/acquire/resolve, never the culprit layer).
|
||||
/// <para>Stage names are stable and greppable: <c>backdrop</c>, <c>base</c>,
|
||||
/// <c>flash</c>, <c>social</c>, <c>ticker</c>, <c>zoom</c>, <c>scale</c>,
|
||||
/// <c>bake backdrop</c>, <c>bake overlays</c>, and <c>#<index> <name></c> per
|
||||
/// element (bake-pass elements are prefixed <c>bake</c> so a one-off rebake cannot
|
||||
/// masquerade as a per-composite cost). A <c>#<i> <name> raster</c> sub-stage NESTS inside its parent's total
|
||||
/// (it is the paste-cache rasterize miss — the prime suspect when one element costs a
|
||||
/// full resample every tick). Stage time INCLUDES that stage's frame resolve; resolve
|
||||
/// is separately totalled in the stats line.</para>
|
||||
/// <para>Single writer in production (the pump's render thread — every blit joins its
|
||||
/// <see cref="StripPool"/> workers before returning); the lock only makes a test able
|
||||
/// to share one across threads.</para>
|
||||
/// </summary>
|
||||
public sealed class CompositeTimings
|
||||
{
|
||||
private readonly object _gate = new();
|
||||
private readonly Dictionary<string, long> _stages = new(StringComparer.Ordinal);
|
||||
|
||||
/// <summary>Accumulate one stage sample. Non-positive ticks are dropped — a
|
||||
/// skipped layer must not create a zero-cost entry that dilutes the reading.</summary>
|
||||
public void Add(string stage, long ticks)
|
||||
{
|
||||
if (ticks <= 0 || stage.Length == 0) return;
|
||||
lock (_gate)
|
||||
{
|
||||
_stages.TryGetValue(stage, out var cur);
|
||||
_stages[stage] = cur + ticks;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Clear every stage — called once per stats window after the parts line
|
||||
/// is logged, so each window reports its own breakdown.</summary>
|
||||
public void Reset()
|
||||
{
|
||||
lock (_gate) _stages.Clear();
|
||||
}
|
||||
|
||||
/// <summary>Average milliseconds per render for every stage at or above
|
||||
/// <paramref name="minMs"/>, sorted slowest-first:
|
||||
/// <c>"#2 Webcam 88.1ms, backdrop 61.2ms, #2 Webcam raster 84.0ms"</c>.
|
||||
/// <paramref name="renders"/> is the divisor (the window's render count, so the
|
||||
/// numbers compare directly with the stats line's <c>avg render</c>). The floor
|
||||
/// keeps a healthy pipeline from burying the contender in zeroes; the empty string
|
||||
/// means nothing was composited (the fully-static shortcut and C4 cache hits do no
|
||||
/// per-element work — an empty parts line there is the correct reading).</summary>
|
||||
public string Describe(int renders, double minMs = 0.1)
|
||||
{
|
||||
List<KeyValuePair<string, long>> snap;
|
||||
lock (_gate)
|
||||
{
|
||||
if (_stages.Count == 0) return string.Empty;
|
||||
snap = _stages.ToList();
|
||||
}
|
||||
|
||||
var freq = (double)Stopwatch.Frequency;
|
||||
var parts = snap
|
||||
.Select(kv => (Stage: kv.Key, Ms: kv.Value / freq * 1000.0 / System.Math.Max(1, renders)))
|
||||
.Where(p => p.Ms >= minMs)
|
||||
.OrderByDescending(p => p.Ms)
|
||||
.Select(p => $"{p.Stage} {p.Ms:F1}ms");
|
||||
return string.Join(", ", parts);
|
||||
}
|
||||
}
|
||||
@@ -54,7 +54,8 @@ public sealed class SceneCompositor
|
||||
VideoFrame? staticBase = null,
|
||||
int split = -1,
|
||||
byte[]? scratch = null,
|
||||
VideoFrame? tickerFrame = null)
|
||||
VideoFrame? tickerFrame = null,
|
||||
CompositeTimings? timings = null)
|
||||
{
|
||||
if (scene == null) throw new ArgumentNullException(nameof(scene));
|
||||
if (frameFor == null) throw new ArgumentNullException(nameof(frameFor));
|
||||
@@ -76,12 +77,15 @@ public sealed class SceneCompositor
|
||||
{
|
||||
if (staticBase.Width != cropW || staticBase.Height != cropH)
|
||||
throw new ArgumentException("The static base must match the source rect.", nameof(staticBase));
|
||||
var tBase = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
Buffer.BlockCopy(staticBase.BgraPixels, 0, buffer, 0, buffer.Length);
|
||||
timings?.Add("base", System.Diagnostics.Stopwatch.GetTimestamp() - tBase);
|
||||
}
|
||||
else
|
||||
{
|
||||
if (split < 0) split = scene.Elements.Count;
|
||||
|
||||
var tBackdrop = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
var elements = scene.Elements;
|
||||
var liveCapture = elements.OfType<Source>().FirstOrDefault(s => s.IsBackground);
|
||||
var liveCaptureFrame = liveCapture != null && liveCapture.IsVisible ? frameFor(liveCapture) : null;
|
||||
@@ -149,8 +153,11 @@ public sealed class SceneCompositor
|
||||
if (backgroundFrame != null)
|
||||
BlitContent(buffer, cropW, cropH, 0, 0, cropW, cropH, backgroundFrame, 1f, false, false);
|
||||
|
||||
foreach (var element in elements)
|
||||
timings?.Add("backdrop", System.Diagnostics.Stopwatch.GetTimestamp() - tBackdrop);
|
||||
|
||||
for (var ei = 0; ei < elements.Count; ei++)
|
||||
{
|
||||
var element = elements[ei];
|
||||
if (!element.IsVisible) continue;
|
||||
switch (element)
|
||||
{
|
||||
@@ -162,7 +169,7 @@ public sealed class SceneCompositor
|
||||
}
|
||||
var fullFrame = frameFor(element);
|
||||
if (fullFrame == null) continue;
|
||||
BlitBakedElement(buffer, cropW, cropH, options, element, fullFrame);
|
||||
BlitBakedElement(buffer, cropW, cropH, options, element, fullFrame, timings, $"#{ei} {element.Name}");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -181,17 +188,29 @@ public sealed class SceneCompositor
|
||||
}
|
||||
var frame = frameFor(element);
|
||||
if (frame == null) continue;
|
||||
BlitBakedElement(buffer, cropW, cropH, options, element, frame);
|
||||
BlitBakedElement(buffer, cropW, cropH, options, element, frame, timings, $"#{i} {element.Name}");
|
||||
}
|
||||
|
||||
if (flashFrame != null)
|
||||
{
|
||||
var tFlash = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
BlitOverlay(buffer, cropW, cropH, options, flashFrame, 0, 0);
|
||||
timings?.Add("flash", System.Diagnostics.Stopwatch.GetTimestamp() - tFlash);
|
||||
}
|
||||
|
||||
if (socialBarFrame != null && socialBarFrame.Width > 0 && socialBarFrame.Height > 0)
|
||||
{
|
||||
var tSocial = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
BlitOverlay(buffer, cropW, cropH, options, socialBarFrame, 0, socialBarTop);
|
||||
timings?.Add("social", System.Diagnostics.Stopwatch.GetTimestamp() - tSocial);
|
||||
}
|
||||
|
||||
if (tickerFrame != null && tickerFrame.Width > 0 && tickerFrame.Height > 0)
|
||||
{
|
||||
var tTicker = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
BlitOverlay(buffer, cropW, cropH, options, tickerFrame, tickerFrame.OriginX, tickerFrame.OriginY);
|
||||
timings?.Add("ticker", System.Diagnostics.Stopwatch.GetTimestamp() - tTicker);
|
||||
}
|
||||
|
||||
// The zoom box is layer n+1: a live 2x view of a FIXED capture window sampled
|
||||
// from the frame just composited (all layers). Drawn LAST — above everything.
|
||||
@@ -199,10 +218,17 @@ public sealed class SceneCompositor
|
||||
// the box; the content it shows never changes (creator ruling 2026-10-08).
|
||||
var zoom = scene.Elements.OfType<ZoomSceneElement>().FirstOrDefault(z => z.IsZoomActive);
|
||||
if (zoom != null)
|
||||
{
|
||||
var tZoom = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
BlitZoom(buffer, cropW, cropH, options, zoom);
|
||||
timings?.Add("zoom", System.Diagnostics.Stopwatch.GetTimestamp() - tZoom);
|
||||
}
|
||||
|
||||
return StretchMath.BilinearScale(
|
||||
var tScale = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
var scaled = StretchMath.BilinearScale(
|
||||
new VideoFrame(cropW, cropH, buffer), options.OutputWidth, options.OutputHeight);
|
||||
timings?.Add("scale", System.Diagnostics.Stopwatch.GetTimestamp() - tScale);
|
||||
return scaled;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -220,12 +246,14 @@ public sealed class SceneCompositor
|
||||
Func<CompositorOptions> options,
|
||||
VideoFrame? socialBarFrame = null,
|
||||
int socialBarTop = 0,
|
||||
VideoFrame? flashFrame = null)
|
||||
VideoFrame? flashFrame = null,
|
||||
CompositeTimings? timings = null)
|
||||
{
|
||||
if (split <= 0) return null;
|
||||
var opts = options();
|
||||
if (opts.SourceRectWidth <= 0 || opts.SourceRectHeight <= 0) return null;
|
||||
|
||||
var tBake = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
var cropW = opts.SourceRectWidth;
|
||||
var cropH = opts.SourceRectHeight;
|
||||
var buffer = new byte[cropW * cropH * 4];
|
||||
@@ -250,6 +278,8 @@ public sealed class SceneCompositor
|
||||
BlitContent(buffer, cropW, cropH, 0, 0, cropW, cropH, backgroundFrame, 1f, false, false);
|
||||
}
|
||||
|
||||
timings?.Add("bake backdrop", System.Diagnostics.Stopwatch.GetTimestamp() - tBake);
|
||||
|
||||
for (var i = 0; i < split; i++)
|
||||
{
|
||||
var element = scene.Elements[i];
|
||||
@@ -264,14 +294,19 @@ public sealed class SceneCompositor
|
||||
}
|
||||
var frame = frameFor(element);
|
||||
if (frame == null) continue;
|
||||
BlitBakedElement(buffer, cropW, cropH, opts, element, frame);
|
||||
BlitBakedElement(buffer, cropW, cropH, opts, element, frame, timings, $"bake #{i} {element.Name}");
|
||||
}
|
||||
|
||||
var baked = new VideoFrame(cropW, cropH, buffer);
|
||||
if (flashFrame != null)
|
||||
baked = Overlay(baked, opts, flashFrame, 0, 0);
|
||||
if (socialBarFrame != null && socialBarFrame.Width > 0 && socialBarFrame.Height > 0)
|
||||
baked = Overlay(baked, opts, socialBarFrame, 0, socialBarTop);
|
||||
if (flashFrame != null || (socialBarFrame != null && socialBarFrame.Width > 0 && socialBarFrame.Height > 0))
|
||||
{
|
||||
var tOverlays = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
if (flashFrame != null)
|
||||
baked = Overlay(baked, opts, flashFrame, 0, 0);
|
||||
if (socialBarFrame != null && socialBarFrame.Width > 0 && socialBarFrame.Height > 0)
|
||||
baked = Overlay(baked, opts, socialBarFrame, 0, socialBarTop);
|
||||
timings?.Add("bake overlays", System.Diagnostics.Stopwatch.GetTimestamp() - tOverlays);
|
||||
}
|
||||
return baked;
|
||||
}
|
||||
|
||||
@@ -291,7 +326,8 @@ public sealed class SceneCompositor
|
||||
int socialBarTop = 0,
|
||||
VideoFrame? flashFrame = null,
|
||||
byte[]? scratch = null,
|
||||
VideoFrame? tickerFrame = null)
|
||||
VideoFrame? tickerFrame = null,
|
||||
CompositeTimings? timings = null)
|
||||
{
|
||||
var cropW = options.SourceRectWidth;
|
||||
var cropH = options.SourceRectHeight;
|
||||
@@ -299,7 +335,9 @@ public sealed class SceneCompositor
|
||||
var buffer = scratch != null && scratch.Length == baseFrame.BgraPixels.Length
|
||||
? scratch
|
||||
: new byte[baseFrame.BgraPixels.Length];
|
||||
var tBase = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
Buffer.BlockCopy(baseFrame.BgraPixels, 0, buffer, 0, baseFrame.BgraPixels.Length);
|
||||
timings?.Add("base", System.Diagnostics.Stopwatch.GetTimestamp() - tBase);
|
||||
|
||||
for (var i = split; i < scene.Elements.Count; i++)
|
||||
{
|
||||
@@ -315,22 +353,35 @@ public sealed class SceneCompositor
|
||||
}
|
||||
var frame = frameFor(element);
|
||||
if (frame == null) continue;
|
||||
BlitBakedElement(buffer, cropW, cropH, options, element, frame);
|
||||
BlitBakedElement(buffer, cropW, cropH, options, element, frame, timings, $"#{i} {element.Name}");
|
||||
}
|
||||
|
||||
if (tickerFrame != null && tickerFrame.Width > 0 && tickerFrame.Height > 0)
|
||||
{
|
||||
var tTicker = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
BlitOverlay(buffer, cropW, cropH, options, tickerFrame, tickerFrame.OriginX, tickerFrame.OriginY);
|
||||
timings?.Add("ticker", System.Diagnostics.Stopwatch.GetTimestamp() - tTicker);
|
||||
}
|
||||
|
||||
var zoom = scene.Elements.OfType<ZoomSceneElement>().FirstOrDefault(z => z.IsZoomActive);
|
||||
if (zoom != null)
|
||||
{
|
||||
var tZoom = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
BlitZoom(buffer, cropW, cropH, options, zoom);
|
||||
timings?.Add("zoom", System.Diagnostics.Stopwatch.GetTimestamp() - tZoom);
|
||||
}
|
||||
|
||||
return StretchMath.BilinearScale(new VideoFrame(cropW, cropH, buffer), options.OutputWidth, options.OutputHeight);
|
||||
var tScale = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
var scaled = StretchMath.BilinearScale(new VideoFrame(cropW, cropH, buffer), options.OutputWidth, options.OutputHeight);
|
||||
timings?.Add("scale", System.Diagnostics.Stopwatch.GetTimestamp() - tScale);
|
||||
return scaled;
|
||||
}
|
||||
|
||||
private static void BlitBakedElement(
|
||||
byte[] buffer, int cropW, int cropH, CompositorOptions options, SceneElement element, VideoFrame frame)
|
||||
byte[] buffer, int cropW, int cropH, CompositorOptions options, SceneElement element, VideoFrame frame,
|
||||
CompositeTimings? timings, string stage)
|
||||
{
|
||||
var tElement = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
var ex = (float)(element.X - options.SourceRectX);
|
||||
var ey = (float)(element.Y - options.SourceRectY);
|
||||
var ew = (float)element.Width;
|
||||
@@ -342,9 +393,10 @@ public sealed class SceneCompositor
|
||||
// the device frame had not changed; an opaque source rasterizes to an opaque
|
||||
// raster whose paste is a per-pixel copy branch, so hits beat the sampler and
|
||||
// misses cost exactly what the bypass cost.
|
||||
BlitCachedLayer(buffer, cropW, cropH, ex, ey, ew, eh, frame, element, isRound);
|
||||
BlitCachedLayer(buffer, cropW, cropH, ex, ey, ew, eh, frame, element, isRound, timings, stage);
|
||||
if (element.HasBorder)
|
||||
DrawBorder(buffer, cropW, cropH, ex, ey, ew, eh, element, isRound);
|
||||
timings?.Add(stage, System.Diagnostics.Stopwatch.GetTimestamp() - tElement);
|
||||
}
|
||||
|
||||
/// <summary>Non-opaque layer blit through the paste cache: the element-space raster
|
||||
@@ -356,7 +408,8 @@ public sealed class SceneCompositor
|
||||
private static void BlitCachedLayer(
|
||||
byte[] buffer, int cropW, int cropH,
|
||||
float ex, float ey, float ew, float eh,
|
||||
VideoFrame frame, SceneElement element, bool isRound)
|
||||
VideoFrame frame, SceneElement element, bool isRound,
|
||||
CompositeTimings? timings, string stage)
|
||||
{
|
||||
if (ew <= 0 || eh <= 0) return;
|
||||
var dw = (int)System.Math.Ceiling(ew);
|
||||
@@ -374,6 +427,7 @@ public sealed class SceneCompositor
|
||||
|
||||
if (raster == null)
|
||||
{
|
||||
var tRaster = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
// Rasterize in ELEMENT space on a TRANSPARENT base: the sampler writes 255
|
||||
// alpha only where it actually draws, so untouched margins (outside the
|
||||
// round clip, past the UniformToFill edges) stay alpha 0 and the paste
|
||||
@@ -398,6 +452,7 @@ public sealed class SceneCompositor
|
||||
_pasteCache[key!] = raster;
|
||||
_elementRasters[element] = raster;
|
||||
}
|
||||
timings?.Add(stage + " raster", System.Diagnostics.Stopwatch.GetTimestamp() - tRaster);
|
||||
}
|
||||
|
||||
// Paste at integer position; opacity blends. The raster is never flagged
|
||||
|
||||
@@ -35,6 +35,12 @@ public sealed class FramePump : IDisposable
|
||||
private readonly SceneGraph? _sceneGraph;
|
||||
private readonly SceneCompositor _compositor = new();
|
||||
|
||||
/// <summary>Per-stage composite timing for the CURRENT stats window (webcam-cadence
|
||||
/// unit, 2026-10-16): the compositor records each stage into this collector and
|
||||
/// <see cref="ReportStats"/> logs the averaged breakdown, then clears it — the
|
||||
/// "which of the 6 elements eats ~140ms" reading (see the parts line).</summary>
|
||||
private readonly CompositeTimings _timings = new();
|
||||
|
||||
// Master-buffer scratch pool (take-4 starvation fix, slice 2 of 2): a fresh
|
||||
// 8.3MB byte[] every tick is ~500MB/s of LOH churn — GC stalls masquerading
|
||||
// as render cost. The pump pools ONLY buffers it handed out (reference-equality
|
||||
@@ -402,6 +408,19 @@ public sealed class FramePump : IDisposable
|
||||
+ $"worst submit {worstSubmit / freq * 1000:F1}ms, "
|
||||
+ $"dropped {dropped}, stalls {stalls}{cacheDetail}, {webcamCadence.Summarize().Describe("webcam")}"
|
||||
+ (detail.Length > 0 ? " | " + detail : ""));
|
||||
|
||||
// Per-stage breakdown (webcam-cadence unit, 2026-10-16): the stats line above
|
||||
// says WHAT a composite cost; this says WHERE it went — one entry per stage,
|
||||
// slowest first, averaged per render so the numbers compare with `avg render`.
|
||||
// No entries (fully-static shortcut, or every render a C4 cache hit) means no
|
||||
// per-element work happened — that IS the reading. The sub-stage
|
||||
// `#i <name> raster` is the paste-cache rasterize miss and nests in its
|
||||
// element's total.
|
||||
var parts = _timings.Describe(statRenders);
|
||||
if (parts.Length > 0)
|
||||
_log?.Invoke($"FramePump parts ({statRenders} renders/5s): {parts}");
|
||||
_timings.Reset();
|
||||
|
||||
worstRender = 0;
|
||||
worstSubmit = 0;
|
||||
renderTicks = submitTicks = resolveTicks = waitTicks = 0;
|
||||
@@ -430,6 +449,8 @@ public sealed class FramePump : IDisposable
|
||||
return frame;
|
||||
}
|
||||
|
||||
Scene? lastDumpedScene = null;
|
||||
|
||||
try
|
||||
{
|
||||
while (!ct.IsCancellationRequested)
|
||||
@@ -437,6 +458,18 @@ public sealed class FramePump : IDisposable
|
||||
var scene = _sceneProvider();
|
||||
if (scene != null)
|
||||
{
|
||||
// Scene dump (webcam-cadence unit, 2026-10-16): the 141ms regression
|
||||
// could not be compared take-to-take because the log recorded no scene
|
||||
// shape at all. Dump it when the pump starts on a scene and whenever
|
||||
// the active scene changes — one header line plus one line per element
|
||||
// (z-index, name, kind, source type, visibility, rect, opacity, clip).
|
||||
if (!ReferenceEquals(scene, lastDumpedScene))
|
||||
{
|
||||
lastDumpedScene = scene;
|
||||
foreach (var line in FormatSceneDump(scene, _sceneGraph?.GetSplitPoint(scene) ?? -1))
|
||||
_log?.Invoke(line);
|
||||
}
|
||||
|
||||
var iterStart = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
var compositorOptions = _compositorOptions();
|
||||
VideoFrame? socialBarFrame = null;
|
||||
@@ -470,7 +503,7 @@ public sealed class FramePump : IDisposable
|
||||
sw.Restart();
|
||||
var scratch = AcquireScratch(scratchSize);
|
||||
acquireTicks += sw.ElapsedTicks;
|
||||
frame = RenderScene(scene, compositorOptions, socialBarFrame, socialBarTop, tickerFrame, flashFrame, scratch, TimedResolver);
|
||||
frame = RenderScene(scene, compositorOptions, socialBarFrame, socialBarTop, tickerFrame, flashFrame, scratch, TimedResolver, _timings);
|
||||
if (_transition is { Active: true } transition)
|
||||
{
|
||||
frame = transition.BlendFrame(frame);
|
||||
@@ -614,13 +647,14 @@ public sealed class FramePump : IDisposable
|
||||
VideoFrame? tickerFrame,
|
||||
VideoFrame? flashFrame,
|
||||
byte[]? scratch = null,
|
||||
Func<SceneElement, VideoFrame?>? resolver = null)
|
||||
Func<SceneElement, VideoFrame?>? resolver = null,
|
||||
CompositeTimings? timings = null)
|
||||
{
|
||||
resolver ??= _frameResolver;
|
||||
if (_sceneGraph == null)
|
||||
{
|
||||
var sw = System.Diagnostics.Stopwatch.StartNew();
|
||||
var frame = RenderFull(scene, options, socialBarFrame, socialBarTop, tickerFrame, flashFrame, scratch, resolver);
|
||||
var frame = RenderFull(scene, options, socialBarFrame, socialBarTop, tickerFrame, flashFrame, scratch, resolver, timings);
|
||||
ProbeRender("full-no-graph", scene, -1, sw.ElapsedMilliseconds, 0);
|
||||
return frame;
|
||||
}
|
||||
@@ -630,7 +664,7 @@ public sealed class FramePump : IDisposable
|
||||
{
|
||||
// Fully static scene: bake once, reuse.
|
||||
var sw = System.Diagnostics.Stopwatch.StartNew();
|
||||
var baked = _sceneGraph.GetBakedBase(scene, _frameResolver, _compositorOptions);
|
||||
var baked = _sceneGraph.GetBakedBase(scene, _frameResolver, _compositorOptions, timings: timings);
|
||||
if (baked != null)
|
||||
{
|
||||
// BUG (pre-existing, found while wiring the branding credit 2026-09-26):
|
||||
@@ -639,31 +673,35 @@ public sealed class FramePump : IDisposable
|
||||
// cached bake is not mutated and the flash still cannot get sticky in it.
|
||||
if (socialBarFrame != null || tickerFrame != null || flashFrame != null)
|
||||
{
|
||||
var tOverlay = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
var withOverlays = baked;
|
||||
if (flashFrame != null) withOverlays = SceneCompositor.Overlay(withOverlays, options, flashFrame, 0, 0);
|
||||
if (tickerFrame != null) withOverlays = SceneCompositor.Overlay(withOverlays, options, tickerFrame, tickerFrame.OriginX, tickerFrame.OriginY);
|
||||
if (socialBarFrame != null) withOverlays = SceneCompositor.Overlay(withOverlays, options, socialBarFrame, socialBarTop, 0);
|
||||
baked = withOverlays;
|
||||
timings?.Add("overlay", System.Diagnostics.Stopwatch.GetTimestamp() - tOverlay);
|
||||
}
|
||||
var tScale = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||
var stretched = StretchMath.BilinearScale(baked, options.OutputWidth, options.OutputHeight);
|
||||
timings?.Add("scale", System.Diagnostics.Stopwatch.GetTimestamp() - tScale);
|
||||
ProbeRender("fully-static", scene, split, sw.ElapsedMilliseconds, (int)sw.ElapsedMilliseconds);
|
||||
return stretched;
|
||||
}
|
||||
}
|
||||
|
||||
var swBase = System.Diagnostics.Stopwatch.StartNew();
|
||||
var baseFrame = _sceneGraph.GetBakedBase(scene, _frameResolver, _compositorOptions);
|
||||
var baseFrame = _sceneGraph.GetBakedBase(scene, _frameResolver, _compositorOptions, timings: timings);
|
||||
if (baseFrame != null)
|
||||
{
|
||||
var frame = SceneCompositor.CompositeLayers(
|
||||
baseFrame, scene, split, resolver, options, socialBarFrame, socialBarTop, flashFrame: flashFrame, tickerFrame: tickerFrame, scratch: scratch);
|
||||
baseFrame, scene, split, resolver, options, socialBarFrame, socialBarTop, flashFrame: flashFrame, tickerFrame: tickerFrame, scratch: scratch, timings: timings);
|
||||
ProbeRender("base+layers", scene, split, swBase.ElapsedMilliseconds, 0);
|
||||
return frame;
|
||||
}
|
||||
|
||||
// No static base (first layer is dynamic or empty scene) — full render.
|
||||
var swFull = System.Diagnostics.Stopwatch.StartNew();
|
||||
var frame2 = RenderFull(scene, options, socialBarFrame, socialBarTop, tickerFrame, flashFrame, scratch, resolver);
|
||||
var frame2 = RenderFull(scene, options, socialBarFrame, socialBarTop, tickerFrame, flashFrame, scratch, resolver, timings);
|
||||
ProbeRender("full-render", scene, split, swFull.ElapsedMilliseconds, 0);
|
||||
return frame2;
|
||||
}
|
||||
@@ -676,7 +714,8 @@ public sealed class FramePump : IDisposable
|
||||
private VideoFrame RenderFull(
|
||||
Scene scene, CompositorOptions options, VideoFrame? socialBarFrame, int socialBarTop,
|
||||
VideoFrame? tickerFrame, VideoFrame? flashFrame,
|
||||
byte[]? scratch, Func<SceneElement, VideoFrame?> resolver)
|
||||
byte[]? scratch, Func<SceneElement, VideoFrame?> resolver,
|
||||
CompositeTimings? timings = null)
|
||||
{
|
||||
// 1:1 guard: the cache stores the OUTPUT bytes and scratch is source-sized —
|
||||
// at 1:1 they are the same buffer size. Off-size tiers (a vertical 1080×1920
|
||||
@@ -697,7 +736,7 @@ public sealed class FramePump : IDisposable
|
||||
return new VideoFrame(options.SourceRectWidth, options.SourceRectHeight, scratch);
|
||||
}
|
||||
|
||||
var rendered = _compositor.Render(scene, resolver, flashFrame, options, socialBarFrame, socialBarTop, scratch: scratch, tickerFrame: tickerFrame);
|
||||
var rendered = _compositor.Render(scene, resolver, flashFrame, options, socialBarFrame, socialBarTop, scratch: scratch, tickerFrame: tickerFrame, timings: timings);
|
||||
CacheRenders++;
|
||||
if (useCache)
|
||||
{
|
||||
@@ -718,6 +757,33 @@ public sealed class FramePump : IDisposable
|
||||
return rendered;
|
||||
}
|
||||
|
||||
/// <summary>The pump's scene dump (webcam-cadence unit, 2026-10-16): one header line
|
||||
/// plus one line per element, in z-order, carrying the shape two takes must be
|
||||
/// compared by — index, name, kind, source type, visibility, rect in master pixels,
|
||||
/// opacity, clip. <paramref name="split"/> is the SceneGraph split point (-1 when no
|
||||
/// graph is wired: the full-render path). The 10:03 vs 12:01 regression
|
||||
/// (29–34ms → 143–155ms at "the same scene shape") could not be compared because the
|
||||
/// log recorded no scene at all; this is that record.</summary>
|
||||
internal static string[] FormatSceneDump(Scene scene, int split)
|
||||
{
|
||||
var lines = new List<string>
|
||||
{
|
||||
$"FramePump scene dump: '{scene.Name}' elements={scene.Elements.Count}"
|
||||
+ (split >= 0 ? $" split={split}" : "")
|
||||
};
|
||||
for (var i = 0; i < scene.Elements.Count; i++)
|
||||
{
|
||||
var e = scene.Elements[i];
|
||||
var source = e is Source s
|
||||
? $" type={s.Type}{(s.IsBackground ? " background" : "")}"
|
||||
: "";
|
||||
lines.Add($" [{i}] {e.Name} kind={e.Kind}{source}"
|
||||
+ $" vis={(e.IsVisible ? 1 : 0)} rect={e.X:F0},{e.Y:F0} {e.Width:F0}x{e.Height:F0}"
|
||||
+ $" op={e.Opacity:F2} clip={e.ClipShape}");
|
||||
}
|
||||
return lines.ToArray();
|
||||
}
|
||||
|
||||
/// <summary>Hash the complete set of inputs the compositor's FULL render consumes:
|
||||
/// the crop/output dimensions, the social bar, and — in element order, mirroring
|
||||
/// <c>SceneCompositor.Render</c>'s iteration — every element's layout/visual fields
|
||||
|
||||
@@ -111,7 +111,8 @@ public sealed class SceneGraph
|
||||
Func<CompositorOptions> options,
|
||||
VideoFrame? socialBarFrame = null,
|
||||
int socialBarTop = 0,
|
||||
VideoFrame? flashFrame = null)
|
||||
VideoFrame? flashFrame = null,
|
||||
CompositeTimings? timings = null)
|
||||
{
|
||||
var split = GetSplitPoint(scene);
|
||||
if (split == 0) return null;
|
||||
@@ -124,7 +125,7 @@ public sealed class SceneGraph
|
||||
}
|
||||
|
||||
var baked = SceneCompositor.BakeStaticBase(
|
||||
scene, split, frameFor, options, socialBarFrame, socialBarTop, flashFrame);
|
||||
scene, split, frameFor, options, socialBarFrame, socialBarTop, flashFrame, timings);
|
||||
if (baked == null) return null;
|
||||
|
||||
lock (_gate)
|
||||
|
||||
+3
-2
@@ -41,7 +41,8 @@ External-facing logic: YouTube API, persistence. See
|
||||
| `ScreenCaptureSourceFactory.cs` | `Resolve(key)` parses `monitor:<n>` / `window:<hwnd>` / `picker:<name>` into a source; `PickAsync()` shows the OS `GraphicsCapturePicker` and returns the `picker:` key (transient — a reload falls back to auto-detection) |
|
||||
| `WebView2Manager.cs` | **Web-source ownership (TASK 17, composition-capture since 2026-09-14):** per registered `Source { Type: WebSource }` it owns a `CoreWebView2CompositionController` + its visual tree (`RootVisualTarget` child under a 1920×1080 root `ContainerVisual` on one `Windows.UI.Composition.Compositor` created on the UI thread via the CoreMessaging `CreateDispatcherQueueController` P/Invoke), a wrapped `WebCaptureFrameSource`, and the shared crop-sized `WriteableBitmap` preview (dispatcher-coalesced copy at Render priority, `PreviewBitmapChanged` raised only on recreation). Nav + `TransparentBackgroundScript` injection unchanged. Public ctor `(Dispatcher)`; internal seam ctor `(Dispatcher, Func<string, IScreenCaptureSource>?)` for hermetic tests (no WinRT). Caller: `MainViewModel.InitWebView2()` (now invoked from `MainWindow_Loaded` — the parent HWND must exist). See `../ai.md` "Slice 14" |
|
||||
| `WebCaptureFrameSource.cs` | The `IScreenCaptureSource` for a web session: owns the `GraphicsCaptureItem` (`CreateFromVisual`) + a free-threaded `Direct3D11CaptureFramePool` (2 buffers) + session; readback via `SoftwareBitmap.CreateCopyFromSurfaceAsync` (BGRA, **`BitmapAlphaMode.Straight`** — the compositor blends straight alpha); per-frame `FindContentBounds` alpha-bbox (internal static, unit-tested) stamps `VideoFrame.CropBounds`; ring up to 2 fresh frames + Epoch reuse; `IsOpaque=false`. Mirror of `ScreenCaptureFrameSource` |
|
||||
| `Compositor/SceneCompositor.cs` | **The output compositor (TASK 4 ship step 1)**: renders a scene into the encoder's master `VideoFrame` (tightly-packed BGRA8), mirroring the XAML preview minus editing chrome — backdrop → background → elements (`UniformToFill` cover-crop, round clip, mirror, opacity, border) → branding flash → social bar (optional `socialBarFrame` + `socialBarTop` in master space, blitted last so the bar sits above the flash; the old `BlitFlash` generalized to `BlitOverlay` with source offsets). **TASK 47:** `Render` + `CompositeLayers` take a trailing `tickerFrame` blitted at (0,0) after the bar — the alert ticker is a message-scrolling dynamic overlay, never baked. Pure and WPF-free: frames injected via a `Func<SceneElement, VideoFrame?>` resolver (webcam → DeviceId, image → AssetId, backdrop → CaptureKey); output sized by `CompositorOptions` (16:9 = full master 1:1; vertical 9:16 = 607×1080 crop → 1080×1920 bilinear). Preview stays XAML (editing view); this is the output view — see `ai.md` "Scene compositor" |
|
||||
| `Compositor/SceneCompositor.cs` | **The output compositor (TASK 4 ship step 1)**: renders a scene into the encoder's master `VideoFrame` (tightly-packed BGRA8), mirroring the XAML preview minus editing chrome — backdrop → background → elements (`UniformToFill` cover-crop, round clip, mirror, opacity, border) → branding flash → social bar (optional `socialBarFrame` + `socialBarTop` in master space, blitted last so the bar sits above the flash; the old `BlitFlash` generalized to `BlitOverlay` with source offsets). **TASK 47:** `Render` + `CompositeLayers` take a trailing `tickerFrame` blitted at (0,0) after the bar — the alert ticker is a message-scrolling dynamic overlay, never baked. Pure and WPF-free: frames injected via a `Func<SceneElement, VideoFrame?>` resolver (webcam → DeviceId, image → AssetId, backdrop → CaptureKey); output sized by `CompositorOptions` (16:9 = full master 1:1; vertical 9:16 = 607×1080 crop → 1080×1920 bilinear). Preview stays XAML (editing view); this is the output view — see `ai.md` "Scene compositor". **2026-10-16 (measurement only):** `Render`/`CompositeLayers`/`BakeStaticBase` take a trailing optional `CompositeTimings?` — every stage and every element timed (`#<i> <name>`, nested `raster`) and reported by the pump's parts line; zero cost for callers that pass nothing |
|
||||
| `Compositor/CompositeTimings.cs` | **Per-element composite timing (2026-10-16, measurement only)**: the collector the pump owns for a stats window — `Add(stage, ticks)` accumulates, `Describe(renders, minMs = 0.1)` prints the averaged breakdown slowest-first. Stage names: `backdrop`/`base`/`flash`/`social`/`ticker`/`zoom`/`scale`/`overlay`, `bake backdrop`/`bake overlays`/`bake #i name`, and **one entry per element `#<z-index> <name>`** with the nested `raster` sub-stage (paste-cache rasterize miss) — the line that names WHICH of the six elements eats the render budget. See `ai.md` "Per-element composite timing + scene dump" |
|
||||
| `Compositor/CompositorOptions.cs` | The active tier's output rect (source space over the 1920×1080 master, integer-aligned — `MainViewModel.OutputRectX` can be 656.5) + target W×H |
|
||||
| `Compositor/StretchMath.cs` | Pure pixel math: the WPF `UniformToFill` cover-crop, clamped bilinear sample/scale (unit-tested half of the compositor) |
|
||||
| `Compositor/StaticPixelCache.cs` | Asset bytes → cached BGRA8 `VideoFrame`, decoded once per content hash — the output path's raw-pixel source (the preview uses ImageCache's `BitmapImage`) |
|
||||
@@ -57,7 +58,7 @@ External-facing logic: YouTube API, persistence. See
|
||||
| `Encoder/FfmpegArgs.cs` | Pure FFmpeg command-line builder: rawvideo `pipe:0` input, `anullsrc` silent audio (WASAPI replaces it), H.264+AAC, closed GOP (`-g fps×4 -keyint_min -sc_threshold 0 -bf 0`, `yuv420p`), FLV → RTMP |
|
||||
| `Encoder/FfmpegProgressParser.cs` | Pure parser for `frame=/fps=/size=/time=/bitrate=` stats lines → `FfmpegProgress` |
|
||||
| `Encoder/FfmpegEncoderPicker.cs` | Pure H.264 encoder picker from `-encoders` output: NVENC → QSV → AMF → OpenH264; **never returns libx264** (GPL) |
|
||||
| `Encoder/FramePump.cs` | **TASK 4 ship step 5**: the live frame producer — while live it snapshots the active scene each tick, resolves every element to its latest frame (`Func<SceneElement, VideoFrame?>` resolver), composites it into the tier's output frame, and paces frames into the encoder at the tier's FPS. All collaborators constructor-injected seams; free of WPF and the capture managers. `StartAsync` never throws (failures log + `Failed`); no RTMP URL = encoder skipped; `StopAsync` stops the encoder before awaiting the loop (backpressure deadlock); `ProcessFailed` self-stops. Since the bar bug-fix branch it takes an optional `socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam — a pre-rendered bar strip composited last (above the flash) at the top edge or `SourceRectHeight − bar height`. **TASK 47:** adds an `alertTicker: Func<VideoFrame?>?` ctor seam (parallel to `socialBar`), read per tick and threaded through `RenderScene`/`RenderFull` + **mixed into `BuildFullRenderSignature`** so the ticker scroll changes the cache key. See `ai.md` "Live frame pipeline" |
|
||||
| `Encoder/FramePump.cs` | **TASK 4 ship step 5**: the live frame producer — while live it snapshots the active scene each tick, resolves every element to its latest frame (`Func<SceneElement, VideoFrame?>` resolver), composites it into the tier's output frame, and paces frames into the encoder at the tier's FPS. All collaborators constructor-injected seams; free of WPF and the capture managers. `StartAsync` never throws (failures log + `Failed`); no RTMP URL = encoder skipped; `StopAsync` stops the encoder before awaiting the loop (backpressure deadlock); `ProcessFailed` self-stops. Since the bar bug-fix branch it takes an optional `socialBar: Func<(VideoFrame? Frame, SocialBarPosition Position)>?` seam — a pre-rendered bar strip composited last (above the flash) at the top edge or `SourceRectHeight − bar height`. **TASK 47:** adds an `alertTicker: Func<VideoFrame?>?` ctor seam (parallel to `socialBar`), read per tick and threaded through `RenderScene`/`RenderFull` + **mixed into `BuildFullRenderSignature`** so the ticker scroll changes the cache key. See `ai.md` "Live frame pipeline". **2026-10-16:** owns a `CompositeTimings` collector, passes it through `RenderScene`/`RenderFull`/`GetBakedBase`/`CompositeLayers`, logs `FramePump parts (N renders/5s): …` (per-stage breakdown, slowest first) beside the stats line each 5s window, and dumps the scene shape — `FramePump scene dump:` header + one line per element (z-index, name, kind, type, visibility, rect, opacity, clip) — at pump start and on every scene change |
|
||||
| `Encoder/FfmpegLocator.cs` | The default locator: PATH probe wins, then the cache, then pull+extract (`ffmpeg.exe` **+ the `libav*.dll` family** via a staging dir, failures via `AppLog`). Pinned to BtbN `autobuild-2026-08-09-13-03` **lgpl-shared** (dynamic linking = LGPL compliance without static-relink §6 material; no GPL libx264; NVENC/QSV/AMF + libopenh264 + native AAC — see `ai.md` → Licensing). Search dirs / tools dir / downloader constructor-injected for hermetic tests. Wired into the encoder since ship step 5 |
|
||||
| `INotificationService.cs` | **Toast seam (TASK 24)**: `Show(title, message, AppNotificationSeverity)` + `AppNotificationSeverity { Info, Success, Warning, Error }` + `Info/Success/Warning/Error` extension methods — non-blocking user-facing messages; VMs/services raise through this so failures surface without MessageBox blocking the UI thread |
|
||||
| `NotificationService.cs` | The default seam impl (Notification.Wpf 11.0.0, MIT): wraps a library `NotificationManager`; routes to the in-window `NotificationArea` named `ToastArea` (MainWindow bottom-right above the footer); lifetimes via the pure `BuildRequest` mapping + `Cards` tint table — Info/Success ~4s, Warning ~8s auto-dismiss, Error sticky (`NeverExpires()` = `TimeSpan.MaxValue`, never null); dark card tints matching the health-banner language; all Show calls marshal to the captured Dispatcher. See [`../ai.md`](../ai.md) "Toast notifications" |
|
||||
|
||||
Reference in New Issue
Block a user