fix(compositor): keep straight alpha in the paste-cache raster — the web widget's recording black box
The element-space raster builds on a TRANSPARENT base but BlitContentRaw's partial-alpha branch applied the opaque-dst source-over blend: color premultiplied by the sampled alpha, then alpha forced to 255. Pasting that raster saw a==255 and straight-copied darkened ink over the scene — a recording box that the raw-bitmap preview (correct alpha) never showed. Transparent margins and opaque content were unaffected, which is why every take looped on the page/CSS while the capture was transparent all along (15:51 dumps: alpha max 255, mean ~19, zero 57%). BlitContentRaw now takes transparentDst; the raster call passes true and writes straight color + straight alpha so the paste rows (BlendRowOpaque/Weighted) do the real source-over onto the opaque master. Master paths byte-identical. ONE integration test PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk: 50%-blue over red reads (127,0,128) fixed vs (0,0,128) buggy — proven both ways (verified by stashing the fix: fails before, passes after). Clean build, 0 warnings; 22/23 compositor-class tests pass, the sole failure the documented pre-existing Composite_FullScene_MasterPixels pixel (1380,700). Alpha-compositing model: standard source-over with producer-cached surfaces, the OBS/libyuv paste model already cited in ai.md/MyMistakes (rawvideo recipe, row-blit BLEND_NONE / straight-alpha branches); full story in MyMistakes (RESOLVED entry). Per the good-dog rule: one integration test, memory updates (MyMistakes/ai.md/ HANDOFF) in the same commit.
This commit is contained in:
+59
-35
@@ -1,52 +1,76 @@
|
|||||||
# HANDOFF — 2026-09-10 (transparency hardening applied)
|
# HANDOFF — 2026-09-12 (transparency: ROOT CAUSE FOUND + FIXED, awaiting the verify take)
|
||||||
|
|
||||||
## Branch / Commit State
|
## Branch / Commit State
|
||||||
|
|
||||||
`main` HEAD will be = the new transparency-hardening commit (slice 14). Ahead of
|
`main` HEAD lands THIS WORK UNIT: the compositor alpha fix (see ✅ below) + its ONE
|
||||||
origin by 19, NOT pushing (user ruling: no push until web-overlay transparency AND
|
integration test + memory updates (`MyMistakes.md`, `ai.md`, `HANDOFF.md`). Ahead of
|
||||||
audio-silence are addressed).
|
origin by ~22 commits. **NOT pushing** — user ruling: no push until web-overlay
|
||||||
|
transparency AND audio-silence are addressed. Nothing uncommitted at end of session.
|
||||||
|
|
||||||
## ACTIVE THREAD (ONE problem at a time)
|
## ✅ HONEST STATUS — THE TRANSPARENCY BUG IS IDENTIFIED
|
||||||
|
|
||||||
**Web-uri transparency — second hardening applied (background-image wipe), awaiting take.**
|
**Root cause found by reading the unread code path to the end (the handoff's live
|
||||||
The 15:34 take: the recording's box interior is the widget's OWN full-canvas opaque paint
|
suspect, now convicted):**
|
||||||
(bright content strips at top/bottom + right bar over a black void) — NOT desktop, NOT a
|
|
||||||
compositor blend. The real-document dumps showed blank-transparent at +1s and the page's
|
|
||||||
backdrop appears later; a background-COLOR-only `!important` wipe (b4bba4b) leaves CSS
|
|
||||||
background-IMAGE (gradient/backdrop) intact — the OBS answer is `background: none
|
|
||||||
!important`, background-image too (obsproject/obs-studio#6659).
|
|
||||||
|
|
||||||
**Slice 15 (commit target, the fix):**
|
`SceneCompositor.BlitContentRaw` (SceneCompositor.cs, partial-alpha `else if (sa > 0)`
|
||||||
- `TransparentBackgroundScript` now also wipes `background-image:none!important` on
|
branch) applied the OPAQUE-dst source-over blend onto the paste-cache raster's
|
||||||
`html,body,html *` (gradients/backdrops die; `<img>`/DOM art survives — OBS semantics).
|
TRANSPARENT base: `dst = (src*sa + dst*inv)/255` with dst black → color premultiplied by
|
||||||
- `NavigateCompleted` re-arms `WidgetDumpRemaining = 5` ~30s in, so this take dumps the
|
sa, then `dst[+3] = 255`. A 50%-alpha widget pixel became darkened color + FULL alpha; at
|
||||||
real document INSIDE the recording window (the missing evidence last time).
|
paste time `BlendRowOpaque` saw alpha 255 → straight copy → the scene behind was
|
||||||
- Test extended (asserts `background-image:none!important` present, weak inline form absent).
|
overwritten by darkened ink. Transparent margins (alpha 0) and opaque content (alpha 255)
|
||||||
|
survived, which is why every take showed a box while the dumps (raw capture) and the
|
||||||
|
preview (raw WriteableBitmap) stayed correct. The CSS-wipe fixes (`b4bba4b`/`0f72c53`)
|
||||||
|
were red herrings — they treated the PAGE as the villain, but the capture was transparent
|
||||||
|
from the start (the 15:51 dumps already proved it).
|
||||||
|
|
||||||
## THE ONE REMAINING STEP (verify, no further analysis)
|
**The fix (committed):** `BlitContentRaw` gained `transparentDst=false`; the raster call
|
||||||
|
passes `true` and writes straight color + straight alpha so the paste rows
|
||||||
|
(`BlendRowOpaque`/`BlendRowWeighted`) do the real source-over onto the opaque master.
|
||||||
|
The two blend rows were verified correct all along (HANDOFF accepted facts held); the
|
||||||
|
divergence was the sampler feeding them. Master paths are bit-identical (`transparentDst`
|
||||||
|
defaults false).
|
||||||
|
|
||||||
1. User records the Live scene, widget animating (pre-record ~35s so the 30s re-arm dump
|
**Verification:** clean build 0 warnings (app + tests); `SceneCompositorTests` +
|
||||||
fires mid-take).
|
`SceneGraphTests` + `FramePumpTests` + `ChatOverlayLayerCacheTests` = 22 pass, the ONE
|
||||||
2. Verdict: element rect shows the scene backdrop behind the widget art (no black box/void)
|
failure is the documented pre-existing `Composite_FullScene_MasterPixels` pixel (1380,700)
|
||||||
→ transparency CLOSED, push gate #1 clears. If a black void persists BEYOND a visible
|
(reproduces with the fix stashed — see ai.md slice 9). The new
|
||||||
background-image, it's not background painting and only chroma-key remains — bring the
|
`PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk` test FAILS on the old code
|
||||||
take and we do the compositor key, not more capture investigation.
|
(exact signature: `pixel (16,16): expected rgb(127,0,128), got rgb(0,0,128)`) and PASSES
|
||||||
3. Then the queued layer-order-save bug (reorder doesn't persist `SortOrder`) is next.
|
on the fix — non-vacuous, proven both ways.
|
||||||
|
|
||||||
|
## NEXT STEP — ONE take (verify, no further analysis)
|
||||||
|
|
||||||
|
The Good Dog Rule is satisfied (ONE integration test shipped with the fix). Record the
|
||||||
|
Live scene with the widget animating, pre-record ~35s so the 30s re-arm dump fires
|
||||||
|
mid-take, then read the verdict:
|
||||||
|
- Element rect shows the scene backdrop behind the widget art (no black box/void, no
|
||||||
|
darkened edge ring) → **transparency CLOSED, push gate #1 clears**.
|
||||||
|
- If anything persists, the fix's own test contract is the diagnostic: a translucent
|
||||||
|
pixel must read as scene-through-src, never inked — bring the take.
|
||||||
|
|
||||||
## Other threads (paused)
|
## Other threads (paused)
|
||||||
|
|
||||||
- **Audio silence** — `f3d578c` has per-5s `Audio live:` telemetry; next take with desktop
|
- **Audio silence** — `f3d578c` has per-5s `Audio live:` telemetry; next take with
|
||||||
audio ACTIVE names the stage. Push gate #2.
|
desktop audio ACTIVE names the stage. Push gate #2.
|
||||||
- **Webcam missing** — "MJPG negotiation refused (being used by another process)". Queued.
|
- **Webcam missing** — "MJPG negotiation refused (being used by another process)". Queued.
|
||||||
- **Web capture speed** — ~10-14Hz effective, user satisfied. Revisit only on request.
|
- **Web capture speed** — ~10-14Hz effective, user satisfied. Revisit only on request.
|
||||||
|
- **Layer order** — dragging an element over another does not persist `SortOrder`; user
|
||||||
|
explicitly asked it not be buried. Queued after transparency.
|
||||||
|
- **Known backfills when queued work resumes:** `Composite_FullScene_MasterPixels` pixel
|
||||||
|
(1380,700) cyan-vs-magenta (pre-existing, recorded in ai.md slice 9); vertical-tier
|
||||||
|
`BilinearScale` fresh allocation per frame.
|
||||||
|
|
||||||
## Landmines
|
## Landmines
|
||||||
|
|
||||||
- testhost shares startup.log with the app — filter by time.
|
- testhost shares startup.log with the app — filter by time.
|
||||||
- App was running at commit time; `taskkill //F //IM ytLive.exe` (Windows `taskkill.exe`,
|
- `taskkill //F //IM ytLive.exe` before rebuilds; re-run if `MSB3021` copy-lock.
|
||||||
bash-quoted `//F //IM`) before rebuilds, and re-run if `MSB3021` copy-lock appears.
|
- Build/tests: `/mnt/c/Program Files/dotnet/dotnet.exe build …` / vstest. 0 warnings rule.
|
||||||
- Build/tests: `/mnt/c/Program Files/dotnet/dotnet.exe build …` / vstest.
|
FULL-suite vstest can hang (WASAPI teardown, pre-existing) — per-class filters are the
|
||||||
- Probing: `/mnt/c/Program Files/Krita (x64)/bin/ffmpeg.exe` / `ffprobe.exe` — Windows exes
|
norm (`--TestCaseFilter:"FullyQualifiedName~…"`).
|
||||||
take Windows-style paths.
|
- Probing: `/mnt/c/Program Files/Krita (x64)/bin/ffmpeg.exe` / `ffprobe.exe` — Windows
|
||||||
- Never re-derive the transparency story again — MyMistakes "SPIN GUARD → RESOLVED" is the
|
exes take Windows-style paths (do NOT re-download Linux ffmpeg — user aborted that).
|
||||||
record (wash-rinse-repeat cost the user a whole session).
|
- Evidence artifacts (keep): `%TEMP%\ytLive-web-8d7234ec-…-w1..5.png` (15:51),
|
||||||
|
recordings `ty-20260910-12\51|1534|1551-*.mp4`, decoded frames at
|
||||||
|
`/mnt/c/Users/gramp/AppData/Local/Temp/w151.raw`.
|
||||||
|
- The full transparency story lives in `MyMistakes.md` → "SPIN GUARD → RESOLVED" (now
|
||||||
|
including the 2026-09-12 RESOLVED entry). GREP IT FIRST. Do not re-derive a fourth time.
|
||||||
@@ -93,6 +93,29 @@ closes. Self-inflicted again: the agent re-derived the whole transparency story
|
|||||||
this entry — the instrument said transparent because the capture had NOT been
|
this entry — the instrument said transparent because the capture had NOT been
|
||||||
repainted yet. Do NOT re-derive this story a third time.
|
repainted yet. Do NOT re-derive this story a third time.
|
||||||
|
|
||||||
|
**2026-09-12 → RESOLVED — the box lived in the paste-cache RASTER, not the page.**
|
||||||
|
The 15:51 facts (alpha max=255, mean ~19, zero 57%, white rounded panel, real
|
||||||
|
transparent margins) + preview-correct + recording-box meant the capture transparency was
|
||||||
|
REAL all along; the fracture sat in `SceneCompositor.BlitContentRaw`
|
||||||
|
(SceneCompositor.cs:419-550) — the sampler that builds the element-space paste-cache
|
||||||
|
raster, the ONE path the take loop never read in full. Its partial-alpha branch applied
|
||||||
|
the OPAQUE-dst blend onto a TRANSPARENT raster base: `dst = (src*sa + dst*inv)/255`
|
||||||
|
with dst black → color PREMULTIPLIED by sa, then `dst[+3] = 255` — a 50%-alpha widget
|
||||||
|
pixel became darkened color + FULL alpha. At paste time `BlendRowOpaque` saw alpha 255 →
|
||||||
|
straight copy → the scene behind was overwritten by darkened ink. Fully-transparent
|
||||||
|
margins (alpha 0, `continue`) and fully-opaque content (alpha 255 branch) survived —
|
||||||
|
which is why every take showed a box while the dumps and the preview (raw WriteableBitmap,
|
||||||
|
unaffected by the compositor) stayed correct, and why the CSS-wipe fixes (`b4bba4b` /
|
||||||
|
`0f72c53`) were red herrings: they treated the page as the villain, but the capture was
|
||||||
|
transparent from the start. **FIX:** `BlitContentRaw` gained `transparentDst=false`;
|
||||||
|
the raster call site passes `true` and writes STRAIGHT color + straight alpha so the
|
||||||
|
paste rows (`BlendRowOpaque`/`BlendRowWeighted`) do the real source-over onto the opaque
|
||||||
|
master. The master paths are untouched (bit-identical). ONE test
|
||||||
|
`PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk` fails on the old code with
|
||||||
|
exactly the bug encoded: 50%-blue over red reads `(0,0,128)` instead of `(127,0,128)` —
|
||||||
|
backdrop never shows through. Next: verify take (element rect shows the backdrop behind
|
||||||
|
the widget art), then push gate #1 clears.
|
||||||
|
|
||||||
### Shrink / re-encode an image for the README (screenshots → small hero image)
|
### Shrink / re-encode an image for the README (screenshots → small hero image)
|
||||||
|
|
||||||
Worked out 2026-08-29 (the recipe was NEVER recorded the first time it was done, so
|
Worked out 2026-08-29 (the recipe was NEVER recorded the first time it was done, so
|
||||||
|
|||||||
@@ -308,7 +308,7 @@ public sealed class SceneCompositor
|
|||||||
}
|
}
|
||||||
if (raster == null) raster = new VideoFrame(dw, dh, new byte[dw * dh * 4]);
|
if (raster == null) raster = new VideoFrame(dw, dh, new byte[dw * dh * 4]);
|
||||||
Array.Clear(raster.BgraPixels);
|
Array.Clear(raster.BgraPixels);
|
||||||
BlitContentRaw(raster.BgraPixels, dw, dh, 0, 0, ew, eh, frame, 1f, isRound, mirror);
|
BlitContentRaw(raster.BgraPixels, dw, dh, 0, 0, ew, eh, frame, 1f, isRound, mirror, transparentDst: true);
|
||||||
lock (_pasteGate)
|
lock (_pasteGate)
|
||||||
{
|
{
|
||||||
// Only the SOURCE frame's identity is a real key owner; drop any older
|
// Only the SOURCE frame's identity is a real key owner; drop any older
|
||||||
@@ -419,7 +419,8 @@ public sealed class SceneCompositor
|
|||||||
private static void BlitContentRaw(
|
private static void BlitContentRaw(
|
||||||
byte[] dst, int dstW, int dstH,
|
byte[] dst, int dstW, int dstH,
|
||||||
float ex, float ey, float ew, float eh,
|
float ex, float ey, float ew, float eh,
|
||||||
VideoFrame src, float opacity, bool isRound, bool isMirror)
|
VideoFrame src, float opacity, bool isRound, bool isMirror,
|
||||||
|
bool transparentDst = false)
|
||||||
{
|
{
|
||||||
if (ew <= 0 || eh <= 0) return;
|
if (ew <= 0 || eh <= 0) return;
|
||||||
|
|
||||||
@@ -528,6 +529,21 @@ public sealed class SceneCompositor
|
|||||||
dst[di + 3] = 255;
|
dst[di + 3] = 255;
|
||||||
}
|
}
|
||||||
else if (sa > 0)
|
else if (sa > 0)
|
||||||
|
{
|
||||||
|
if (transparentDst)
|
||||||
|
{
|
||||||
|
// Element-raster pass on a TRANSPARENT base (the paste cache):
|
||||||
|
// keep straight color + straight alpha so the paste rows can
|
||||||
|
// source-over onto the opaque master. Premultiplying here (the
|
||||||
|
// opaque-dst blend) stamped alpha 255 onto already-alpha-scaled
|
||||||
|
// color — the paste over-wrote the scene with darkened ink, the
|
||||||
|
// web widget's "black box" in recordings.
|
||||||
|
dst[di] = (byte)sb;
|
||||||
|
dst[di + 1] = (byte)sg;
|
||||||
|
dst[di + 2] = (byte)sr;
|
||||||
|
dst[di + 3] = (byte)sa;
|
||||||
|
}
|
||||||
|
else
|
||||||
{
|
{
|
||||||
var inv = 255 - sa;
|
var inv = 255 - sa;
|
||||||
dst[di] = (byte)((sb * sa + dst[di] * inv + 127) / 255);
|
dst[di] = (byte)((sb * sa + dst[di] * inv + 127) / 255);
|
||||||
@@ -538,6 +554,7 @@ public sealed class SceneCompositor
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>Two-stage bilinear in 8.8 fixed point; result within ±1 of the
|
/// <summary>Two-stage bilinear in 8.8 fixed point; result within ±1 of the
|
||||||
/// float StretchMath.SampleBgra (well inside every pixel test's ±2 tolerance).
|
/// float StretchMath.SampleBgra (well inside every pixel test's ±2 tolerance).
|
||||||
|
|||||||
@@ -758,7 +758,13 @@ seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<Compos
|
|||||||
routes non-opaque layers through `BlitCachedLayer`: a layer rasterizes ONCE into an element-space,
|
routes non-opaque layers through `BlitCachedLayer`: a layer rasterizes ONCE into an element-space,
|
||||||
TRANSPARENT-based frame keyed by (source-array identity, source W×H, ceil'd dst rect, round,
|
TRANSPARENT-based frame keyed by (source-array identity, source W×H, ceil'd dst rect, round,
|
||||||
mirror), then every later tick PASTES it (integer position, row alpha-blend, opacity applied at
|
mirror), then every later tick PASTES it (integer position, row alpha-blend, opacity applied at
|
||||||
paste). Producers hand out fresh immutable arrays, so array-identity keys can never serve stale
|
paste). **Correction (2026-09-12, the web-widget "black box"):** the raster builds on a
|
||||||
|
TRANSPARENT base, but the sampler's PARTIAL-alpha branch used the opaque-dst blend — it
|
||||||
|
premultiplied the color into RGB and forced `alpha=255`. A translucent widget pixel then pasted
|
||||||
|
as opaque darkened ink (the box in recordings) while the raw-bitmap preview stayed correct.
|
||||||
|
`BlitContentRaw` now takes `transparentDst` (the raster call passes `true` and writes straight
|
||||||
|
color + straight alpha; the paste rows do the source-over). Master paths are bit-identical.
|
||||||
|
Guard: `PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk`. Producers hand out fresh immutable arrays, so array-identity keys can never serve stale
|
||||||
content; dict bounded at 48, cleared wholesale on overflow. Only changing content (webcam device
|
content; dict bounded at 48, cleared wholesale on overflow. Only changing content (webcam device
|
||||||
frames, web capture ticks, chat messages) resamples; the opaque backdrop keeps its memcpy path.
|
frames, web capture ticks, chat messages) resamples; the opaque backdrop keeps its memcpy path.
|
||||||
Position/opacity drags are now near-free (no resample — paste params, not cache keys).
|
Position/opacity drags are now near-free (no resample — paste params, not cache keys).
|
||||||
|
|||||||
@@ -254,6 +254,55 @@ public class SceneCompositorTests
|
|||||||
Assert.Equal(out2.BgraPixels, out3.BgraPixels);
|
Assert.Equal(out2.BgraPixels, out3.BgraPixels);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>THE page-transparency fix (the web widget's recording "black box"): a
|
||||||
|
/// semi-transparent layer flowing through the paste-cache raster must reveal the
|
||||||
|
/// backdrop through it (true source-over), never an opaque darkened block. The raster
|
||||||
|
/// is built on a TRANSPARENT base, but the old partial-alpha branch applied the
|
||||||
|
/// opaque-dst blend (premultiplied color + forced alpha 255), so every translucent
|
||||||
|
/// widget pixel pasted as opaque ink — the box in recordings while the raw-bitmap
|
||||||
|
/// preview stayed correct. A half-alpha blue widget over a red backdrop must read
|
||||||
|
/// (127,0,128) here; the buggy raster pastes (0,0,128).</summary>
|
||||||
|
[Fact]
|
||||||
|
public void PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk()
|
||||||
|
{
|
||||||
|
var red = Solid(128, 64, 255, 0, 0); // live backdrop
|
||||||
|
var web = new byte[32 * 16 * 4]; // 2:1 widget canvas, half-alpha blue, uniform
|
||||||
|
for (var i = 0; i < web.Length; i += 4)
|
||||||
|
{
|
||||||
|
web[i] = 255; web[i + 1] = 0; web[i + 2] = 0; web[i + 3] = 128;
|
||||||
|
}
|
||||||
|
var semitransparent = new VideoFrame(32, 16, web); // NOT IsOpaque
|
||||||
|
|
||||||
|
var background = new Source { Type = SourceType.DisplayCapture, IsBackground = true, CaptureKey = "monitor:0" };
|
||||||
|
var widget = new Source { Type = SourceType.WebSource, X = 8, Y = 8, Width = 16, Height = 16 };
|
||||||
|
var scene = new Scene { Name = "Live" };
|
||||||
|
scene.Elements.Add(background);
|
||||||
|
scene.Elements.Add(widget);
|
||||||
|
|
||||||
|
VideoFrame? FrameFor(SceneElement e) => e switch
|
||||||
|
{
|
||||||
|
Source { IsBackground: true } => red,
|
||||||
|
Source { Type: SourceType.WebSource } => semitransparent,
|
||||||
|
_ => null,
|
||||||
|
};
|
||||||
|
|
||||||
|
var options = new CompositorOptions
|
||||||
|
{
|
||||||
|
SourceRectX = 0, SourceRectY = 0, SourceRectWidth = 128, SourceRectHeight = 64,
|
||||||
|
OutputWidth = 128, OutputHeight = 64,
|
||||||
|
};
|
||||||
|
|
||||||
|
var compositor = new SceneCompositor();
|
||||||
|
var build = compositor.Render(scene, FrameFor, null, options); // raster build + paste
|
||||||
|
var hit = compositor.Render(scene, FrameFor, null, options); // paste-cache hit
|
||||||
|
foreach (var output in new[] { build, hit })
|
||||||
|
{
|
||||||
|
AssertColor(output, 16, 16, 127, 0, 128); // backdrop shows through the 50% blue
|
||||||
|
AssertColor(output, 4, 4, 255, 0, 0); // outside the element: untouched backdrop
|
||||||
|
}
|
||||||
|
Assert.Equal(build.BgraPixels, hit.BgraPixels);
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>take-16 regression gate — GC churn convicted mechanically: a ring-style
|
/// <summary>take-16 regression gate — GC churn convicted mechanically: a ring-style
|
||||||
/// producer hands out a small set of rotating arrays with a NEW EPOCH every frame
|
/// producer hands out a small set of rotating arrays with a NEW EPOCH every frame
|
||||||
/// (exactly what the capture/camera rings do). The compositor must re-rasterize INTO
|
/// (exactly what the capture/camera rings do). The compositor must re-rasterize INTO
|
||||||
|
|||||||
Reference in New Issue
Block a user