fix(compositor): keep straight alpha in the paste-cache raster — the web widget's recording black box
The element-space raster builds on a TRANSPARENT base but BlitContentRaw's partial-alpha branch applied the opaque-dst source-over blend: color premultiplied by the sampled alpha, then alpha forced to 255. Pasting that raster saw a==255 and straight-copied darkened ink over the scene — a recording box that the raw-bitmap preview (correct alpha) never showed. Transparent margins and opaque content were unaffected, which is why every take looped on the page/CSS while the capture was transparent all along (15:51 dumps: alpha max 255, mean ~19, zero 57%). BlitContentRaw now takes transparentDst; the raster call passes true and writes straight color + straight alpha so the paste rows (BlendRowOpaque/Weighted) do the real source-over onto the opaque master. Master paths byte-identical. ONE integration test PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk: 50%-blue over red reads (127,0,128) fixed vs (0,0,128) buggy — proven both ways (verified by stashing the fix: fails before, passes after). Clean build, 0 warnings; 22/23 compositor-class tests pass, the sole failure the documented pre-existing Composite_FullScene_MasterPixels pixel (1380,700). Alpha-compositing model: standard source-over with producer-cached surfaces, the OBS/libyuv paste model already cited in ai.md/MyMistakes (rawvideo recipe, row-blit BLEND_NONE / straight-alpha branches); full story in MyMistakes (RESOLVED entry). Per the good-dog rule: one integration test, memory updates (MyMistakes/ai.md/ HANDOFF) in the same commit.
This commit is contained in:
+59
-35
@@ -1,52 +1,76 @@
|
||||
# HANDOFF — 2026-09-10 (transparency hardening applied)
|
||||
# HANDOFF — 2026-09-12 (transparency: ROOT CAUSE FOUND + FIXED, awaiting the verify take)
|
||||
|
||||
## Branch / Commit State
|
||||
|
||||
`main` HEAD will be = the new transparency-hardening commit (slice 14). Ahead of
|
||||
origin by 19, NOT pushing (user ruling: no push until web-overlay transparency AND
|
||||
audio-silence are addressed).
|
||||
`main` HEAD lands THIS WORK UNIT: the compositor alpha fix (see ✅ below) + its ONE
|
||||
integration test + memory updates (`MyMistakes.md`, `ai.md`, `HANDOFF.md`). Ahead of
|
||||
origin by ~22 commits. **NOT pushing** — user ruling: no push until web-overlay
|
||||
transparency AND audio-silence are addressed. Nothing uncommitted at end of session.
|
||||
|
||||
## ACTIVE THREAD (ONE problem at a time)
|
||||
## ✅ HONEST STATUS — THE TRANSPARENCY BUG IS IDENTIFIED
|
||||
|
||||
**Web-uri transparency — second hardening applied (background-image wipe), awaiting take.**
|
||||
The 15:34 take: the recording's box interior is the widget's OWN full-canvas opaque paint
|
||||
(bright content strips at top/bottom + right bar over a black void) — NOT desktop, NOT a
|
||||
compositor blend. The real-document dumps showed blank-transparent at +1s and the page's
|
||||
backdrop appears later; a background-COLOR-only `!important` wipe (b4bba4b) leaves CSS
|
||||
background-IMAGE (gradient/backdrop) intact — the OBS answer is `background: none
|
||||
!important`, background-image too (obsproject/obs-studio#6659).
|
||||
**Root cause found by reading the unread code path to the end (the handoff's live
|
||||
suspect, now convicted):**
|
||||
|
||||
**Slice 15 (commit target, the fix):**
|
||||
- `TransparentBackgroundScript` now also wipes `background-image:none!important` on
|
||||
`html,body,html *` (gradients/backdrops die; `<img>`/DOM art survives — OBS semantics).
|
||||
- `NavigateCompleted` re-arms `WidgetDumpRemaining = 5` ~30s in, so this take dumps the
|
||||
real document INSIDE the recording window (the missing evidence last time).
|
||||
- Test extended (asserts `background-image:none!important` present, weak inline form absent).
|
||||
`SceneCompositor.BlitContentRaw` (SceneCompositor.cs, partial-alpha `else if (sa > 0)`
|
||||
branch) applied the OPAQUE-dst source-over blend onto the paste-cache raster's
|
||||
TRANSPARENT base: `dst = (src*sa + dst*inv)/255` with dst black → color premultiplied by
|
||||
sa, then `dst[+3] = 255`. A 50%-alpha widget pixel became darkened color + FULL alpha; at
|
||||
paste time `BlendRowOpaque` saw alpha 255 → straight copy → the scene behind was
|
||||
overwritten by darkened ink. Transparent margins (alpha 0) and opaque content (alpha 255)
|
||||
survived, which is why every take showed a box while the dumps (raw capture) and the
|
||||
preview (raw WriteableBitmap) stayed correct. The CSS-wipe fixes (`b4bba4b`/`0f72c53`)
|
||||
were red herrings — they treated the PAGE as the villain, but the capture was transparent
|
||||
from the start (the 15:51 dumps already proved it).
|
||||
|
||||
## THE ONE REMAINING STEP (verify, no further analysis)
|
||||
**The fix (committed):** `BlitContentRaw` gained `transparentDst=false`; the raster call
|
||||
passes `true` and writes straight color + straight alpha so the paste rows
|
||||
(`BlendRowOpaque`/`BlendRowWeighted`) do the real source-over onto the opaque master.
|
||||
The two blend rows were verified correct all along (HANDOFF accepted facts held); the
|
||||
divergence was the sampler feeding them. Master paths are bit-identical (`transparentDst`
|
||||
defaults false).
|
||||
|
||||
1. User records the Live scene, widget animating (pre-record ~35s so the 30s re-arm dump
|
||||
fires mid-take).
|
||||
2. Verdict: element rect shows the scene backdrop behind the widget art (no black box/void)
|
||||
→ transparency CLOSED, push gate #1 clears. If a black void persists BEYOND a visible
|
||||
background-image, it's not background painting and only chroma-key remains — bring the
|
||||
take and we do the compositor key, not more capture investigation.
|
||||
3. Then the queued layer-order-save bug (reorder doesn't persist `SortOrder`) is next.
|
||||
**Verification:** clean build 0 warnings (app + tests); `SceneCompositorTests` +
|
||||
`SceneGraphTests` + `FramePumpTests` + `ChatOverlayLayerCacheTests` = 22 pass, the ONE
|
||||
failure is the documented pre-existing `Composite_FullScene_MasterPixels` pixel (1380,700)
|
||||
(reproduces with the fix stashed — see ai.md slice 9). The new
|
||||
`PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk` test FAILS on the old code
|
||||
(exact signature: `pixel (16,16): expected rgb(127,0,128), got rgb(0,0,128)`) and PASSES
|
||||
on the fix — non-vacuous, proven both ways.
|
||||
|
||||
## NEXT STEP — ONE take (verify, no further analysis)
|
||||
|
||||
The Good Dog Rule is satisfied (ONE integration test shipped with the fix). Record the
|
||||
Live scene with the widget animating, pre-record ~35s so the 30s re-arm dump fires
|
||||
mid-take, then read the verdict:
|
||||
- Element rect shows the scene backdrop behind the widget art (no black box/void, no
|
||||
darkened edge ring) → **transparency CLOSED, push gate #1 clears**.
|
||||
- If anything persists, the fix's own test contract is the diagnostic: a translucent
|
||||
pixel must read as scene-through-src, never inked — bring the take.
|
||||
|
||||
## Other threads (paused)
|
||||
|
||||
- **Audio silence** — `f3d578c` has per-5s `Audio live:` telemetry; next take with desktop
|
||||
audio ACTIVE names the stage. Push gate #2.
|
||||
- **Audio silence** — `f3d578c` has per-5s `Audio live:` telemetry; next take with
|
||||
desktop audio ACTIVE names the stage. Push gate #2.
|
||||
- **Webcam missing** — "MJPG negotiation refused (being used by another process)". Queued.
|
||||
- **Web capture speed** — ~10-14Hz effective, user satisfied. Revisit only on request.
|
||||
- **Layer order** — dragging an element over another does not persist `SortOrder`; user
|
||||
explicitly asked it not be buried. Queued after transparency.
|
||||
- **Known backfills when queued work resumes:** `Composite_FullScene_MasterPixels` pixel
|
||||
(1380,700) cyan-vs-magenta (pre-existing, recorded in ai.md slice 9); vertical-tier
|
||||
`BilinearScale` fresh allocation per frame.
|
||||
|
||||
## Landmines
|
||||
|
||||
- testhost shares startup.log with the app — filter by time.
|
||||
- App was running at commit time; `taskkill //F //IM ytLive.exe` (Windows `taskkill.exe`,
|
||||
bash-quoted `//F //IM`) before rebuilds, and re-run if `MSB3021` copy-lock appears.
|
||||
- Build/tests: `/mnt/c/Program Files/dotnet/dotnet.exe build …` / vstest.
|
||||
- Probing: `/mnt/c/Program Files/Krita (x64)/bin/ffmpeg.exe` / `ffprobe.exe` — Windows exes
|
||||
take Windows-style paths.
|
||||
- Never re-derive the transparency story again — MyMistakes "SPIN GUARD → RESOLVED" is the
|
||||
record (wash-rinse-repeat cost the user a whole session).
|
||||
- `taskkill //F //IM ytLive.exe` before rebuilds; re-run if `MSB3021` copy-lock.
|
||||
- Build/tests: `/mnt/c/Program Files/dotnet/dotnet.exe build …` / vstest. 0 warnings rule.
|
||||
FULL-suite vstest can hang (WASAPI teardown, pre-existing) — per-class filters are the
|
||||
norm (`--TestCaseFilter:"FullyQualifiedName~…"`).
|
||||
- Probing: `/mnt/c/Program Files/Krita (x64)/bin/ffmpeg.exe` / `ffprobe.exe` — Windows
|
||||
exes take Windows-style paths (do NOT re-download Linux ffmpeg — user aborted that).
|
||||
- Evidence artifacts (keep): `%TEMP%\ytLive-web-8d7234ec-…-w1..5.png` (15:51),
|
||||
recordings `ty-20260910-12\51|1534|1551-*.mp4`, decoded frames at
|
||||
`/mnt/c/Users/gramp/AppData/Local/Temp/w151.raw`.
|
||||
- The full transparency story lives in `MyMistakes.md` → "SPIN GUARD → RESOLVED" (now
|
||||
including the 2026-09-12 RESOLVED entry). GREP IT FIRST. Do not re-derive a fourth time.
|
||||
@@ -93,6 +93,29 @@ closes. Self-inflicted again: the agent re-derived the whole transparency story
|
||||
this entry — the instrument said transparent because the capture had NOT been
|
||||
repainted yet. Do NOT re-derive this story a third time.
|
||||
|
||||
**2026-09-12 → RESOLVED — the box lived in the paste-cache RASTER, not the page.**
|
||||
The 15:51 facts (alpha max=255, mean ~19, zero 57%, white rounded panel, real
|
||||
transparent margins) + preview-correct + recording-box meant the capture transparency was
|
||||
REAL all along; the fracture sat in `SceneCompositor.BlitContentRaw`
|
||||
(SceneCompositor.cs:419-550) — the sampler that builds the element-space paste-cache
|
||||
raster, the ONE path the take loop never read in full. Its partial-alpha branch applied
|
||||
the OPAQUE-dst blend onto a TRANSPARENT raster base: `dst = (src*sa + dst*inv)/255`
|
||||
with dst black → color PREMULTIPLIED by sa, then `dst[+3] = 255` — a 50%-alpha widget
|
||||
pixel became darkened color + FULL alpha. At paste time `BlendRowOpaque` saw alpha 255 →
|
||||
straight copy → the scene behind was overwritten by darkened ink. Fully-transparent
|
||||
margins (alpha 0, `continue`) and fully-opaque content (alpha 255 branch) survived —
|
||||
which is why every take showed a box while the dumps and the preview (raw WriteableBitmap,
|
||||
unaffected by the compositor) stayed correct, and why the CSS-wipe fixes (`b4bba4b` /
|
||||
`0f72c53`) were red herrings: they treated the page as the villain, but the capture was
|
||||
transparent from the start. **FIX:** `BlitContentRaw` gained `transparentDst=false`;
|
||||
the raster call site passes `true` and writes STRAIGHT color + straight alpha so the
|
||||
paste rows (`BlendRowOpaque`/`BlendRowWeighted`) do the real source-over onto the opaque
|
||||
master. The master paths are untouched (bit-identical). ONE test
|
||||
`PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk` fails on the old code with
|
||||
exactly the bug encoded: 50%-blue over red reads `(0,0,128)` instead of `(127,0,128)` —
|
||||
backdrop never shows through. Next: verify take (element rect shows the backdrop behind
|
||||
the widget art), then push gate #1 clears.
|
||||
|
||||
### Shrink / re-encode an image for the README (screenshots → small hero image)
|
||||
|
||||
Worked out 2026-08-29 (the recipe was NEVER recorded the first time it was done, so
|
||||
|
||||
@@ -308,7 +308,7 @@ public sealed class SceneCompositor
|
||||
}
|
||||
if (raster == null) raster = new VideoFrame(dw, dh, new byte[dw * dh * 4]);
|
||||
Array.Clear(raster.BgraPixels);
|
||||
BlitContentRaw(raster.BgraPixels, dw, dh, 0, 0, ew, eh, frame, 1f, isRound, mirror);
|
||||
BlitContentRaw(raster.BgraPixels, dw, dh, 0, 0, ew, eh, frame, 1f, isRound, mirror, transparentDst: true);
|
||||
lock (_pasteGate)
|
||||
{
|
||||
// Only the SOURCE frame's identity is a real key owner; drop any older
|
||||
@@ -419,7 +419,8 @@ public sealed class SceneCompositor
|
||||
private static void BlitContentRaw(
|
||||
byte[] dst, int dstW, int dstH,
|
||||
float ex, float ey, float ew, float eh,
|
||||
VideoFrame src, float opacity, bool isRound, bool isMirror)
|
||||
VideoFrame src, float opacity, bool isRound, bool isMirror,
|
||||
bool transparentDst = false)
|
||||
{
|
||||
if (ew <= 0 || eh <= 0) return;
|
||||
|
||||
@@ -529,11 +530,27 @@ public sealed class SceneCompositor
|
||||
}
|
||||
else if (sa > 0)
|
||||
{
|
||||
var inv = 255 - sa;
|
||||
dst[di] = (byte)((sb * sa + dst[di] * inv + 127) / 255);
|
||||
dst[di + 1] = (byte)((sg * sa + dst[di + 1] * inv + 127) / 255);
|
||||
dst[di + 2] = (byte)((sr * sa + dst[di + 2] * inv + 127) / 255);
|
||||
dst[di + 3] = 255;
|
||||
if (transparentDst)
|
||||
{
|
||||
// Element-raster pass on a TRANSPARENT base (the paste cache):
|
||||
// keep straight color + straight alpha so the paste rows can
|
||||
// source-over onto the opaque master. Premultiplying here (the
|
||||
// opaque-dst blend) stamped alpha 255 onto already-alpha-scaled
|
||||
// color — the paste over-wrote the scene with darkened ink, the
|
||||
// web widget's "black box" in recordings.
|
||||
dst[di] = (byte)sb;
|
||||
dst[di + 1] = (byte)sg;
|
||||
dst[di + 2] = (byte)sr;
|
||||
dst[di + 3] = (byte)sa;
|
||||
}
|
||||
else
|
||||
{
|
||||
var inv = 255 - sa;
|
||||
dst[di] = (byte)((sb * sa + dst[di] * inv + 127) / 255);
|
||||
dst[di + 1] = (byte)((sg * sa + dst[di + 1] * inv + 127) / 255);
|
||||
dst[di + 2] = (byte)((sr * sa + dst[di + 2] * inv + 127) / 255);
|
||||
dst[di + 3] = 255;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -758,7 +758,13 @@ seam:** `Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<Compos
|
||||
routes non-opaque layers through `BlitCachedLayer`: a layer rasterizes ONCE into an element-space,
|
||||
TRANSPARENT-based frame keyed by (source-array identity, source W×H, ceil'd dst rect, round,
|
||||
mirror), then every later tick PASTES it (integer position, row alpha-blend, opacity applied at
|
||||
paste). Producers hand out fresh immutable arrays, so array-identity keys can never serve stale
|
||||
paste). **Correction (2026-09-12, the web-widget "black box"):** the raster builds on a
|
||||
TRANSPARENT base, but the sampler's PARTIAL-alpha branch used the opaque-dst blend — it
|
||||
premultiplied the color into RGB and forced `alpha=255`. A translucent widget pixel then pasted
|
||||
as opaque darkened ink (the box in recordings) while the raw-bitmap preview stayed correct.
|
||||
`BlitContentRaw` now takes `transparentDst` (the raster call passes `true` and writes straight
|
||||
color + straight alpha; the paste rows do the source-over). Master paths are bit-identical.
|
||||
Guard: `PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk`. Producers hand out fresh immutable arrays, so array-identity keys can never serve stale
|
||||
content; dict bounded at 48, cleared wholesale on overflow. Only changing content (webcam device
|
||||
frames, web capture ticks, chat messages) resamples; the opaque backdrop keeps its memcpy path.
|
||||
Position/opacity drags are now near-free (no resample — paste params, not cache keys).
|
||||
|
||||
@@ -254,6 +254,55 @@ public class SceneCompositorTests
|
||||
Assert.Equal(out2.BgraPixels, out3.BgraPixels);
|
||||
}
|
||||
|
||||
/// <summary>THE page-transparency fix (the web widget's recording "black box"): a
|
||||
/// semi-transparent layer flowing through the paste-cache raster must reveal the
|
||||
/// backdrop through it (true source-over), never an opaque darkened block. The raster
|
||||
/// is built on a TRANSPARENT base, but the old partial-alpha branch applied the
|
||||
/// opaque-dst blend (premultiplied color + forced alpha 255), so every translucent
|
||||
/// widget pixel pasted as opaque ink — the box in recordings while the raw-bitmap
|
||||
/// preview stayed correct. A half-alpha blue widget over a red backdrop must read
|
||||
/// (127,0,128) here; the buggy raster pastes (0,0,128).</summary>
|
||||
[Fact]
|
||||
public void PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk()
|
||||
{
|
||||
var red = Solid(128, 64, 255, 0, 0); // live backdrop
|
||||
var web = new byte[32 * 16 * 4]; // 2:1 widget canvas, half-alpha blue, uniform
|
||||
for (var i = 0; i < web.Length; i += 4)
|
||||
{
|
||||
web[i] = 255; web[i + 1] = 0; web[i + 2] = 0; web[i + 3] = 128;
|
||||
}
|
||||
var semitransparent = new VideoFrame(32, 16, web); // NOT IsOpaque
|
||||
|
||||
var background = new Source { Type = SourceType.DisplayCapture, IsBackground = true, CaptureKey = "monitor:0" };
|
||||
var widget = new Source { Type = SourceType.WebSource, X = 8, Y = 8, Width = 16, Height = 16 };
|
||||
var scene = new Scene { Name = "Live" };
|
||||
scene.Elements.Add(background);
|
||||
scene.Elements.Add(widget);
|
||||
|
||||
VideoFrame? FrameFor(SceneElement e) => e switch
|
||||
{
|
||||
Source { IsBackground: true } => red,
|
||||
Source { Type: SourceType.WebSource } => semitransparent,
|
||||
_ => null,
|
||||
};
|
||||
|
||||
var options = new CompositorOptions
|
||||
{
|
||||
SourceRectX = 0, SourceRectY = 0, SourceRectWidth = 128, SourceRectHeight = 64,
|
||||
OutputWidth = 128, OutputHeight = 64,
|
||||
};
|
||||
|
||||
var compositor = new SceneCompositor();
|
||||
var build = compositor.Render(scene, FrameFor, null, options); // raster build + paste
|
||||
var hit = compositor.Render(scene, FrameFor, null, options); // paste-cache hit
|
||||
foreach (var output in new[] { build, hit })
|
||||
{
|
||||
AssertColor(output, 16, 16, 127, 0, 128); // backdrop shows through the 50% blue
|
||||
AssertColor(output, 4, 4, 255, 0, 0); // outside the element: untouched backdrop
|
||||
}
|
||||
Assert.Equal(build.BgraPixels, hit.BgraPixels);
|
||||
}
|
||||
|
||||
/// <summary>take-16 regression gate — GC churn convicted mechanically: a ring-style
|
||||
/// producer hands out a small set of rotating arrays with a NEW EPOCH every frame
|
||||
/// (exactly what the capture/camera rings do). The compositor must re-rasterize INTO
|
||||
|
||||
Reference in New Issue
Block a user