Scene compositor (TASK 4 ship step 1): encoder master-frame scaffolding
Software compositor that renders a scene into the encoder's master VideoFrame, mirroring the XAML preview minus editing chrome: backdrop -> background -> elements (UniformToFill cover-crop, round clip, mirror, opacity, border) -> branding flash. NOTE FOR USERS: this change shows NO difference in the app's UI — it is pure backend scaffolding laying the groundwork for live video capture/streaming. The preview you see is unchanged. - Services/Compositor/: SceneCompositor (Render(scene, frameFor resolver, flashFrame, CompositorOptions)), CompositorOptions (source rect + output size; 16:9 full master, vertical 607x1080 -> 1080x1920), StretchMath (pure UniformToFill + bilinear), StaticPixelCache (asset bytes -> BGRA8 frame) - Frame sources injected via Func<SceneElement, VideoFrame?> resolver, so the compositor is pure, WPF-free, and hermetic to test (D3D11 upgrade behind the same seam later) - SceneElement.TryGetBorderColor public (shared hex parse), stale MainViewModel comment fixed, pre-existing CS1998 in YouTubeAuthServiceTests cleaned up - Tests: SceneCompositorTests integration (full scene + vertical tier + flash) + StretchMath units, docs updated (72 tests passing, 0 warnings)
This commit is contained in:
@@ -0,0 +1,19 @@
|
||||
namespace ytLive.Services.Compositor;
|
||||
|
||||
/// <summary>
|
||||
/// Where the compositor draws from (the active quality tier's output rect over the
|
||||
/// 1920×1080 master) and at what size. 16:9 tiers = the full master 1:1; the vertical
|
||||
/// 9:16 tier = the centered 607×1080 crop scaled up to 1080×1920. The source rect is
|
||||
/// integer-aligned — MainViewModel's <c>OutputRectX</c> can be 656.5, so the caller
|
||||
/// rounds before building these options.
|
||||
/// </summary>
|
||||
public sealed class CompositorOptions
|
||||
{
|
||||
public int SourceRectX { get; init; }
|
||||
public int SourceRectY { get; init; }
|
||||
public int SourceRectWidth { get; init; }
|
||||
public int SourceRectHeight { get; init; }
|
||||
|
||||
public int OutputWidth { get; init; }
|
||||
public int OutputHeight { get; init; }
|
||||
}
|
||||
@@ -0,0 +1,198 @@
|
||||
using ytLive.Models;
|
||||
|
||||
namespace ytLive.Services.Compositor;
|
||||
|
||||
/// <summary>
|
||||
/// The output compositor: renders a scene into the encoder's master frame (tightly-packed
|
||||
/// BGRA8 <see cref="VideoFrame"/>) exactly as the XAML preview renders it, minus the
|
||||
/// editing chrome (SelectionOverlay, DimRects, badge, placeholder). The preview stays the
|
||||
/// editing view; this is the output view — see TASKS.md "Ship step 1 — Scene compositor".
|
||||
///
|
||||
/// Frame sources are supplied by a <see cref="Func{SceneElement, VideoFrame}"/> resolver
|
||||
/// (the caller maps webcam → DeviceId, images → AssetId, backdrop → CaptureKey), keeping
|
||||
/// the compositor pure and free of WPF and of the capture managers.
|
||||
/// </summary>
|
||||
public sealed class SceneCompositor
|
||||
{
|
||||
/// <summary>
|
||||
/// Composite <paramref name="scene"/> into the tier's output frame. Layer order (back →
|
||||
/// front): live backdrop (the scene's <c>IsBackdrop</c> source) → background image →
|
||||
/// visible elements (z-order = <c>Elements</c> order, mirroring the XAML DataTemplate) →
|
||||
/// branding flash. Transparent regions read opaque black.
|
||||
/// </summary>
|
||||
public VideoFrame Render(
|
||||
Scene scene,
|
||||
Func<SceneElement, VideoFrame?> frameFor,
|
||||
VideoFrame? flashFrame,
|
||||
CompositorOptions options)
|
||||
{
|
||||
if (scene == null) throw new ArgumentNullException(nameof(scene));
|
||||
if (frameFor == null) throw new ArgumentNullException(nameof(frameFor));
|
||||
if (options == null) throw new ArgumentNullException(nameof(options));
|
||||
if (options.SourceRectWidth <= 0 || options.SourceRectHeight <= 0)
|
||||
throw new ArgumentException("The source rect must be positive.", nameof(options));
|
||||
if (options.OutputWidth <= 0 || options.OutputHeight <= 0)
|
||||
throw new ArgumentException("The output size must be positive.", nameof(options));
|
||||
|
||||
var cropW = options.SourceRectWidth;
|
||||
var cropH = options.SourceRectHeight;
|
||||
var buffer = new byte[cropW * cropH * 4];
|
||||
for (var i = 3; i < buffer.Length; i += 4)
|
||||
buffer[i] = 255; // opaque black base — video frames are never transparent
|
||||
|
||||
var elements = scene.Elements;
|
||||
|
||||
var backdrop = elements.OfType<Source>().FirstOrDefault(s => s.IsBackdrop);
|
||||
var backdropFrame = backdrop != null ? frameFor(backdrop) : null;
|
||||
if (backdropFrame != null)
|
||||
BlitContent(buffer, cropW, cropH, 0, 0, cropW, cropH, backdropFrame, 1f, false, false);
|
||||
|
||||
var background = elements.OfType<Source>().FirstOrDefault(s => s.Type == SourceType.Background);
|
||||
var backgroundFrame = background != null ? frameFor(background) : null;
|
||||
if (backgroundFrame != null)
|
||||
BlitContent(buffer, cropW, cropH, 0, 0, cropW, cropH, backgroundFrame, 1f, false, false);
|
||||
|
||||
foreach (var element in elements)
|
||||
{
|
||||
if (!element.IsVisible) continue;
|
||||
switch (element)
|
||||
{
|
||||
case Source { IsBackdrop: true }:
|
||||
case Source { Type: SourceType.Background }:
|
||||
case Source { Type: SourceType.TextOverlay }:
|
||||
continue; // backdrop/background are their own layers; Text isn't shipped
|
||||
}
|
||||
|
||||
var frame = frameFor(element);
|
||||
if (frame == null) continue;
|
||||
|
||||
var ex = (float)(element.X - options.SourceRectX);
|
||||
var ey = (float)(element.Y - options.SourceRectY);
|
||||
var ew = (float)element.Width;
|
||||
var eh = (float)element.Height;
|
||||
var isRound = element.ClipShape == ClipShape.Round;
|
||||
|
||||
BlitContent(buffer, cropW, cropH, ex, ey, ew, eh, frame,
|
||||
(float)element.Opacity, isRound, element.IsMirrored);
|
||||
if (element.HasBorder)
|
||||
DrawBorder(buffer, cropW, cropH, ex, ey, ew, eh, element, isRound);
|
||||
}
|
||||
|
||||
if (flashFrame != null)
|
||||
BlitFlash(buffer, cropW, cropH, options, flashFrame);
|
||||
|
||||
return StretchMath.BilinearScale(
|
||||
new VideoFrame(cropW, cropH, buffer), options.OutputWidth, options.OutputHeight);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// UniformToFill blit of a source frame into element rect (ex, ey, ew, eh), with
|
||||
/// straight-alpha source-over, optional round clip (true circle, hard edge) and
|
||||
/// horizontal mirror around the element center.
|
||||
/// </summary>
|
||||
private static void BlitContent(
|
||||
byte[] dst, int dstW, int dstH,
|
||||
float ex, float ey, float ew, float eh,
|
||||
VideoFrame src, float opacity, bool isRound, bool isMirror)
|
||||
{
|
||||
if (ew <= 0 || eh <= 0) return;
|
||||
|
||||
var x0 = Math.Max(0, (int)Math.Floor(ex));
|
||||
var y0 = Math.Max(0, (int)Math.Floor(ey));
|
||||
var x1 = Math.Min(dstW - 1, (int)Math.Ceiling(ex + ew));
|
||||
var y1 = Math.Min(dstH - 1, (int)Math.Ceiling(ey + eh));
|
||||
if (x0 > x1 || y0 > y1) return;
|
||||
|
||||
var (scale, ox, oy) = StretchMath.UniformToFill(ew, eh, src.Width, src.Height);
|
||||
var drawnW = src.Width * scale;
|
||||
var drawnH = src.Height * scale;
|
||||
var radius = Math.Min(ew, eh) / 2f;
|
||||
var cx = ew / 2f;
|
||||
var cy = eh / 2f;
|
||||
|
||||
for (var y = y0; y <= y1; y++)
|
||||
{
|
||||
for (var x = x0; x <= x1; x++)
|
||||
{
|
||||
var px = x - ex; // element space
|
||||
var py = y - ey;
|
||||
if (px < ox || px > ox + drawnW || py < oy || py > oy + drawnH) continue;
|
||||
if (isRound)
|
||||
{
|
||||
var dx = px - cx;
|
||||
var dy = py - cy;
|
||||
if (dx * dx + dy * dy > radius * radius) continue;
|
||||
}
|
||||
|
||||
var sx = (px - ox) / scale;
|
||||
var sy = (py - oy) / scale;
|
||||
if (isMirror) sx = src.Width - 1 - sx;
|
||||
var sample = StretchMath.SampleBgra(src.BgraPixels, src.Width, src.Height, sx, sy);
|
||||
BlendPixel(dst, (y * dstW + x) * 4, sample, opacity);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Centered OBS-style border stroke: rect ring (Traditional) or circle ring (Round).</summary>
|
||||
private static void DrawBorder(
|
||||
byte[] dst, int dstW, int dstH,
|
||||
float ex, float ey, float ew, float eh,
|
||||
SceneElement element, bool isRound)
|
||||
{
|
||||
if (!element.TryGetBorderColor(out var r, out var g, out var b)) return;
|
||||
var half = element.BorderWidth / 2f;
|
||||
var alpha = (float)(element.Opacity * element.BorderOpacity);
|
||||
if (half <= 0 || alpha <= 0) return;
|
||||
|
||||
var cx = ex + ew / 2f;
|
||||
var cy = ey + eh / 2f;
|
||||
var radius = Math.Min(ew, eh) / 2f;
|
||||
|
||||
var x0 = Math.Max(0, (int)Math.Floor(ex - half));
|
||||
var y0 = Math.Max(0, (int)Math.Floor(ey - half));
|
||||
var x1 = Math.Min(dstW - 1, (int)Math.Ceiling(ex + ew + half));
|
||||
var y1 = Math.Min(dstH - 1, (int)Math.Ceiling(ey + eh + half));
|
||||
|
||||
for (var y = y0; y <= y1; y++)
|
||||
{
|
||||
for (var x = x0; x <= x1; x++)
|
||||
{
|
||||
float d = isRound
|
||||
? MathF.Sqrt((x - cx) * (x - cx) + (y - cy) * (y - cy)) - radius
|
||||
: MathF.Max(MathF.Max(ex - x, x - (ex + ew)), MathF.Max(ey - y, y - (ey + eh)));
|
||||
if (MathF.Abs(d) <= half)
|
||||
BlendPixel(dst, (y * dstW + x) * 4, (b, g, r, (byte)255), alpha);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>1:1 copy of the master-sized branding flash, cropped to the active source rect.</summary>
|
||||
private static void BlitFlash(byte[] dst, int dstW, int dstH, CompositorOptions options, VideoFrame flash)
|
||||
{
|
||||
for (var y = 0; y < dstH; y++)
|
||||
{
|
||||
for (var x = 0; x < dstW; x++)
|
||||
{
|
||||
var sx = x + options.SourceRectX;
|
||||
var sy = y + options.SourceRectY;
|
||||
if (sx >= flash.Width || sy >= flash.Height) continue;
|
||||
var sample = StretchMath.SampleBgra(flash.BgraPixels, flash.Width, flash.Height, sx, sy);
|
||||
BlendPixel(dst, (y * dstW + x) * 4, sample, 1f);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Straight-alpha source-over blend; the frame's alpha (and the layer opacity) drives coverage.</summary>
|
||||
private static void BlendPixel(byte[] dst, int di, (byte B, byte G, byte R, byte A) src, float opacity)
|
||||
{
|
||||
var sa = src.A / 255f * opacity;
|
||||
if (sa <= 0) return;
|
||||
var da = dst[di + 3] / 255f;
|
||||
var outA = sa + da * (1 - sa);
|
||||
if (outA <= 0) return;
|
||||
dst[di] = (byte)Math.Round((src.B * sa + dst[di] * da * (1 - sa)) / outA);
|
||||
dst[di + 1] = (byte)Math.Round((src.G * sa + dst[di + 1] * da * (1 - sa)) / outA);
|
||||
dst[di + 2] = (byte)Math.Round((src.R * sa + dst[di + 2] * da * (1 - sa)) / outA);
|
||||
dst[di + 3] = (byte)Math.Round(outA * 255);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
using System.IO;
|
||||
using System.Windows.Media;
|
||||
using System.Windows.Media.Imaging;
|
||||
|
||||
namespace ytLive.Services.Compositor;
|
||||
|
||||
/// <summary>
|
||||
/// Decodes a layout asset's bytes into a tightly-packed BGRA8 <see cref="VideoFrame"/>
|
||||
/// once per content-hash asset id. The preview uses the WPF <c>BitmapImage</c> in
|
||||
/// ImageCache; the output path needs raw pixels, so assets decode here instead.
|
||||
/// </summary>
|
||||
public static class StaticPixelCache
|
||||
{
|
||||
private static readonly Dictionary<string, VideoFrame> Cache = new(StringComparer.Ordinal);
|
||||
|
||||
public static VideoFrame? Get(string assetId)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(assetId)) return null;
|
||||
if (Cache.TryGetValue(assetId, out var frame)) return frame;
|
||||
|
||||
var bytes = LayoutStore.Instance?.GetAssetBytes(assetId);
|
||||
if (bytes == null || bytes.Length == 0) return null;
|
||||
|
||||
var decoded = Decode(bytes);
|
||||
if (decoded != null) Cache[assetId] = decoded;
|
||||
return decoded;
|
||||
}
|
||||
|
||||
public static VideoFrame? Decode(byte[] bytes)
|
||||
{
|
||||
try
|
||||
{
|
||||
using var stream = new MemoryStream(bytes, writable: false);
|
||||
var decoder = BitmapDecoder.Create(stream, BitmapCreateOptions.PreservePixelFormat, BitmapCacheOption.OnLoad);
|
||||
var source = decoder.Frames[0];
|
||||
var bgra = new FormatConvertedBitmap(source, PixelFormats.Bgra32, null, 0);
|
||||
var width = bgra.PixelWidth;
|
||||
var height = bgra.PixelHeight;
|
||||
var pixels = new byte[width * height * 4];
|
||||
bgra.CopyPixels(pixels, width * 4, 0);
|
||||
return new VideoFrame(width, height, pixels);
|
||||
}
|
||||
catch
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
namespace ytLive.Services.Compositor;
|
||||
|
||||
/// <summary>
|
||||
/// Pure pixel math shared by the compositor: the WPF "UniformToFill" cover-crop
|
||||
/// (what the preview's <c>Stretch="UniformToFill"</c> does) and a clamped bilinear
|
||||
/// sample/scale. Pure and deterministic — the unit-tested half of the compositor.
|
||||
/// </summary>
|
||||
public static class StretchMath
|
||||
{
|
||||
/// <summary>
|
||||
/// UniformToFill: the drawn content covers the destination while preserving the
|
||||
/// source aspect, centered; overflow is cropped. Returns the scale and the
|
||||
/// source-origin offset (in destination pixels) of the drawn content within the
|
||||
/// destination rect.
|
||||
/// </summary>
|
||||
public static (float Scale, float OffsetX, float OffsetY) UniformToFill(float dstW, float dstH, int srcW, int srcH)
|
||||
{
|
||||
var scale = Math.Max(dstW / srcW, dstH / srcH);
|
||||
var drawnW = srcW * scale;
|
||||
var drawnH = srcH * scale;
|
||||
return (scale, (dstW - drawnW) / 2f, (dstH - drawnH) / 2f);
|
||||
}
|
||||
|
||||
/// <summary>Clamped bilinear scale into a new tightly-packed BGRA8 frame; returns the input unchanged when the sizes already match.</summary>
|
||||
public static VideoFrame BilinearScale(VideoFrame src, int outW, int outH)
|
||||
{
|
||||
if (outW == src.Width && outH == src.Height) return src;
|
||||
|
||||
var outPixels = new byte[outW * outH * 4];
|
||||
for (var y = 0; y < outH; y++)
|
||||
{
|
||||
var sy = (y + 0.5f) / outH * src.Height - 0.5f;
|
||||
for (var x = 0; x < outW; x++)
|
||||
{
|
||||
var sx = (x + 0.5f) / outW * src.Width - 0.5f;
|
||||
var (b, g, r, a) = SampleBgra(src.BgraPixels, src.Width, src.Height, sx, sy);
|
||||
var di = (y * outW + x) * 4;
|
||||
outPixels[di] = b;
|
||||
outPixels[di + 1] = g;
|
||||
outPixels[di + 2] = r;
|
||||
outPixels[di + 3] = a;
|
||||
}
|
||||
}
|
||||
return new VideoFrame(outW, outH, outPixels);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Clamped bilinear sample of one BGRA8 pixel at fractional (sx, sy). Coordinates
|
||||
/// outside [0, w-1]×[0, h-1] clamp to the edge, so a caller can sample freely
|
||||
/// without bounds checks.
|
||||
/// </summary>
|
||||
public static (byte B, byte G, byte R, byte A) SampleBgra(byte[] src, int w, int h, float sx, float sy)
|
||||
{
|
||||
sx = Math.Clamp(sx, 0, w - 1);
|
||||
sy = Math.Clamp(sy, 0, h - 1);
|
||||
var x0 = (int)sx;
|
||||
var y0 = (int)sy;
|
||||
var x1 = Math.Min(x0 + 1, w - 1);
|
||||
var y1 = Math.Min(y0 + 1, h - 1);
|
||||
var fx = sx - x0;
|
||||
var fy = sy - y0;
|
||||
|
||||
var p00 = (y0 * w + x0) * 4;
|
||||
var p10 = (y0 * w + x1) * 4;
|
||||
var p01 = (y1 * w + x0) * 4;
|
||||
var p11 = (y1 * w + x1) * 4;
|
||||
|
||||
float r = Lerp(Lerp(src[p00 + 2], src[p10 + 2], fx), Lerp(src[p01 + 2], src[p11 + 2], fx), fy);
|
||||
float g = Lerp(Lerp(src[p00 + 1], src[p10 + 1], fx), Lerp(src[p01 + 1], src[p11 + 1], fx), fy);
|
||||
float b = Lerp(Lerp(src[p00], src[p10], fx), Lerp(src[p01], src[p11], fx), fy);
|
||||
float a = Lerp(Lerp(src[p00 + 3], src[p10 + 3], fx), Lerp(src[p01 + 3], src[p11 + 3], fx), fy);
|
||||
return ((byte)Math.Round(b), (byte)Math.Round(g), (byte)Math.Round(r), (byte)Math.Round(a));
|
||||
}
|
||||
|
||||
private static float Lerp(float a, float b, float t) => a + (b - a) * t;
|
||||
}
|
||||
@@ -27,6 +27,10 @@ External-facing logic: YouTube API, persistence. See
|
||||
| `ScreenCaptureFrameSource.cs` | One `Direct3D11CaptureFramePool` (free-threaded, 2 buffers) + session per target; frames → `SoftwareBitmap.CreateCopyFromSurfaceAsync` (BGRA, alpha ignored) → `VideoFrame`, bytes read via `WindowsRuntimeMarshal.TryGetDataUnsafe` (CsWinRT-safe — the `IMemoryBufferByteAccess` ComImport cast fails on every frame and is gone). Surfaces > 1920×1080 downscaled bilinearly to the master; conversion failures logged ≤ once/5 s. DRM content = black frames (OS limit). `CreateForMonitor`/`CreateForWindow`/`CreateForPicker` |
|
||||
| `ScreenCaptureManager.cs` | Screen-capture ownership mirroring `CameraManager`: refcounted by target key, one shared `WriteableBitmap`, dispatcher-coalesced latest-frame copies; `PreviewBitmapChanged`/`CaptureFailed` events; `ReleaseAllAsync` used on re-designation |
|
||||
| `ScreenCaptureSourceFactory.cs` | `Resolve(key)` parses `monitor:<n>` / `window:<hwnd>` / `picker:<name>` into a source; `PickAsync()` shows the OS `GraphicsCapturePicker` and returns the `picker:` key (transient — a reload falls back to auto-detection) |
|
||||
| `Compositor/SceneCompositor.cs` | **The output compositor (TASK 4 ship step 1)**: renders a scene into the encoder's master `VideoFrame` (tightly-packed BGRA8), mirroring the XAML preview minus editing chrome — backdrop → background → elements (`UniformToFill` cover-crop, round clip, mirror, opacity, border) → branding flash. Pure and WPF-free: frames injected via a `Func<SceneElement, VideoFrame?>` resolver (webcam → DeviceId, image → AssetId, backdrop → CaptureKey); output sized by `CompositorOptions` (16:9 = full master 1:1; vertical 9:16 = 607×1080 crop → 1080×1920 bilinear). Preview stays XAML (editing view); this is the output view — see `ai.md` "Scene compositor" |
|
||||
| `Compositor/CompositorOptions.cs` | The active tier's output rect (source space over the 1920×1080 master, integer-aligned — `MainViewModel.OutputRectX` can be 656.5) + target W×H |
|
||||
| `Compositor/StretchMath.cs` | Pure pixel math: the WPF `UniformToFill` cover-crop, clamped bilinear sample/scale (unit-tested half of the compositor) |
|
||||
| `Compositor/StaticPixelCache.cs` | Asset bytes → cached BGRA8 `VideoFrame`, decoded once per content hash — the output path's raw-pixel source (the preview uses ImageCache's `BitmapImage`) |
|
||||
|
||||
Related: constructed in [`ViewModels/MainViewModel.cs`](../ViewModels/MainViewModel.cs)
|
||||
(no DI container yet). Models in [`Models/index.md`](../Models/index.md).
|
||||
|
||||
Reference in New Issue
Block a user