From bccdb4892c0181941477603ef6a60dd71553338b Mon Sep 17 00:00:00 2001 From: gramps Date: Fri, 28 Aug 2026 11:53:36 -0700 Subject: [PATCH] =?UTF-8?q?Web=20source:=20crop=20to=20the=20frame's=20ALP?= =?UTF-8?q?HA=20bounding=20box=20=E2=80=94=20the=20widget's=20true=20rende?= =?UTF-8?q?red=20extent,=20no=20DOM=20timing,=20can't=20break=20the=20rend?= =?UTF-8?q?er?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - FindContentBounds scans the Bgra32 capture for the bounding box of non-transparent pixels (background is injected transparent, so alpha = widget extent). Crop to that rect + Stretch=Fill maps the widget flush under the selection box: full-canvas widgets fall through to the full frame (identical to the confirmed-good image); floating widgets get tight-fitted with no dead space. - Removes QueryContentBoundsAsync + the JS script entirely — the DOM-union attempt (a5b9952) broke rendering by measuring an unsettled layout on NavigationCompleted; alpha measurement is immune. - 3 new unit tests: tight box, no-alpha no-crop, full-alpha no-crop. 7/7 pass, 0 warnings. --- HANDOFF.md | 49 ++++++++++--------- MyMistakes.md | 16 ++++--- Services/WebView2Manager.cs | 72 +++++++++++----------------- TASKS.md | 16 ++++--- ytLive.Tests/WebView2ManagerTests.cs | 44 +++++++++++++++++ 5 files changed, 117 insertions(+), 80 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index 90a8f5a..09911db 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -6,34 +6,36 @@ working). **NOT pushed — user rule: never push without explicit instruction (2026-08-24).** No feature branches pre-1.0: all work lands on `main` per work unit. -## The web-source rendering work (2026-08-28) — CURRENT MODEL: canvas-size viewport + content crop +## The web-source rendering work (2026-08-28) — CURRENT MODEL: canvas-size viewport + ALPHA-BBOX crop -**Root causes finally isolated:** (1) hosting the page at the element-box size clips full-bleed -widgets to their upper-left chunk (fixed — page now renders at the master canvas 1920×1080, -stable). (2) The content-bounds crop that was supposed to kill right/bottom dead space never ran: -the script returned `JSON.stringify(...)` (a string), and `ExecuteScriptAsync` JSON-encodes the -result, so `JsonDocument.Parse` hit a string node and `GetProperty` threw → bounds stayed unset → -no crop. ALSO the intended measure was wrong: scroll extents ≠ widget rect (they include the -page's own margins). Fixed together: the script now returns a real OBJECT (union of every visible -`body *` `getBoundingClientRect`), giving the widget's true rect INCLUDING its top-left offset, -and `CaptureFrame` crops AT that offset — so the widget anchors at (0,0) and the box is tight on -all four sides. `Stretch="Fill"` maps it flush under the box. Creator confirmed render+resize are -solid at the canvas-size change; this commit delivers the tight/0,0 bounding box. +**History of failures (see MyMistakes.md for the full log):** box-size rendering clipped full-bleed +widgets to their upper-left chunk (fixed — page renders at the master canvas 1920×1080, stable, the +one render the creator confirmed solid). Then four crop attempts failed in different ways: (1) the +`JSON.stringify` return double-encoded under `ExecuteScriptAsync`, silently disabling the crop; +(2) `scrollWidth/scrollHeight` at canvas size returns the whole viewport for full-canvas wrapper +bodies → no trim (ghost boundary); (3) union-of-DOM-elements measured an unsettled layout on +NavigationCompleted and BROKE the image (creator's "don't break the rendering" boundary crossed). +(4) Reverting to f39ebd9's scroll-crop was another dead end (#2 again). -**The CURRENT MODEL:** -- `Services/WebView2Manager.cs` — page renders at 1920×1080 (stable); `overflow:hidden` + - transparent-bg injection; `QueryContentBoundsAsync` measures real widget bounds (object return - from JS, NOT JSON.stringify); `CaptureFrame` crops at the widget's (x,y) offset → anchored 0,0. -- `MainWindow.xaml` — web `` `Stretch="Fill"`. -- Invariant: **render at canvas size; crop to the widget's true bounds; Fill to the box.** +**The CURRENT MODEL (this commit):** +- `Services/WebView2Manager.cs` — page renders at 1920×1080 (stable, never tracked); `overflow:hidden` + + transparent-bg injection. `FindContentBounds` scans the Bgra32 capture per frame and crops to the + bounding box of NON-TRANSPARENT pixels — the widget's true rendered extent, measured from the frame + itself. No DOM query, no layout timing: it can never truncate visible content (worst case = full + frame = the confirmed-good image). `Stretch="Fill"` maps the cropped frame flush under the selection + box → the box hugs the widget on all four sides. `QueryContentBoundsAsync` + the JS script are + DELETED. +- `MainWindow.xaml` — web `` Stretch set to Fill via the IsWebSource DataTrigger (unchanged). +- Invariant: **render at canvas size; crop to the alpha extent; Fill to the box.** +- Unit tests: 3 new for `FindContentBounds` (tight box, no-alpha no-crop, full-alpha no-crop). **Property panel / sliders (2026-08-28):** web URI ✕ clears the textbox; web ✓/✕ icon buttons `IsTabStop="False"` so Tab walks X→Y→W→H→URI; Slider style `IsMoveToPointEnabled="True"` restores click-anywhere-on-bar. All three confirmed fixed by the creator. -Build: 0 warnings; tests 4/4. Rejected for the record: box-sized viewport (upper-left truncation), -fixed 1920×1080 + `Uniform` (letterbox dead space), CSS margin/100% reset (enlarged gap), -viewport==box (reflow/truncation/scrollbars). +Build: 0 warnings; tests 7/7 (WebView2Manager). Rejected for the record: box-sized viewport (upper-left +truncation), fixed 1920×1080 + `Uniform` (letterbox dead space), CSS margin/100% reset (enlarged gap), +viewport==box (reflow/truncation/scrollbars), DOM-union crop (broke the image). ## Key facts / landmines @@ -50,7 +52,8 @@ viewport==box (reflow/truncation/scrollbars). - No `CODING.md`; conventions live in `AGENTS.md` + `ai.md`. ## Next steps -1. AWAIT creator's verdict on viewport==box (geometry-agnostic) rendering — visual check with a - widget URL at several box sizes; the selection frame must snap flush with no padding. +1. AWAIT creator's verdict on the ALPHA-BBOX crop — reload a widget URL at several box sizes; the + selection frame must snap flush to the widget with no dead space, image must stay identical for + full-bleed widgets. If an animated widget causes visible jitter, add crop hysteresis as a follow-up. 2. Follow-up only if the creator asks: graceful handling of navigation errors / invalid URIs (TASK 17 ×9). 3. Continue queue: TASK 18 local recording, TASK 21 media source, TASK 22 audio sync offset. \ No newline at end of file diff --git a/MyMistakes.md b/MyMistakes.md index 784cf93..4f0588c 100644 --- a/MyMistakes.md +++ b/MyMistakes.md @@ -31,15 +31,17 @@ Stream widget bounding box: hug the widget on all four sides, pinned to canvas ( extents regardless. Lesson: verify declared mechanics actually execute; assume nothing ran. ## Current iteration (web bounding box — UNCOMMITTED, verification pending) -RENDER RESTORED to the last creator-verified-good state (`70fe2b8`): full-frame capture (no crop), -1920×1080 canvas-size viewport, plain `Stretch="Fill"`. Image rendering is back to "renders and -resizes beautifully". The bounding-box dead-space problem is STILL OPEN — do NOT reintroduce any -crop until the creator explicitly approves the approach. +ALPHA-BBOX CROP: `FindContentBounds` scans the captured Bgra32 frame and returns the bounding box +of non-transparent pixels (the background is injected transparent, so alpha = the widget's extent). +Capture is cropped to that rect and `Stretch="Fill"` maps it flush under the box. No DOM query, no +layout timing — the crop rect IS the rendered widget by construction, so it can NEVER truncate +visible content. Worst case (widget fills the canvas) = full frame = the confirmed-good render. +Canvas-size viewport + Fill untouched. `QueryContentBoundsAsync` + the JS script are DELETED. -## Iteration that just failed (`a5b9952` — wrapper-excluded content-union crop + image alignment) +## Iteration that just failed (`a5b9952` — wrapper-excluded DOM-union crop + image alignment) Broke the image rendering — creator could not evaluate the box because the image was gone/ruined. Why: the visible-element-union rect is measured ONCE on NavigationCompleted; widget layout isn't settled then (async fonts/images, load animations, iframes), so the crop rect comes out tiny/wrong and the Fill-stretched frame renders a broken/zoomed sliver. This was against the creator's explicit -"don't break the rendering" instruction. Lesson: the canvas-size render + Fill is the ONLY verified -good base; any crop must be built on it one tested step at a time, never shipped blind. \ No newline at end of file +"don't break the rendering" instruction. Lesson: measure actual RENDERED PIXELS (alpha), never +transient DOM state; and never ship a crop blind. \ No newline at end of file diff --git a/Services/WebView2Manager.cs b/Services/WebView2Manager.cs index 7a2ec62..6c0eb95 100644 --- a/Services/WebView2Manager.cs +++ b/Services/WebView2Manager.cs @@ -27,10 +27,6 @@ public sealed class WebView2Manager : IDisposable public required WebView2 Control { get; init; } public required DispatcherTimer CaptureTimer { get; init; } public VideoFrame? LatestFrame; - public double ViewportCssW; - public double ViewportCssH; - public double ContentCssW; - public double ContentCssH; public bool Disposed; public bool Initialized; } @@ -48,9 +44,11 @@ public sealed class WebView2Manager : IDisposable // OBS model: the page renders at the MASTER CANVAS size (1920×1080), // stable, never tracked. Full-bleed widgets (designed for the canvas) // render fully instead of being clipped to a small box-sized viewport; - // dragging the box never reflows the page. The capture is cropped to - // the widget's actual content bounds (QueryContentBoundsAsync) and the - // display (Stretch=Fill) maps it flush under the box, anchored at (0,0). + // dragging the box never reflows the page. The capture is tightly + // cropped to the widget's ALPHA bounding box (FindContentBounds) and + // the display (Stretch=Fill) maps it flush under the box, anchored at + // (0,0) — no DOM query, so no timing fragility; worst case is the + // full frame, which is the confirmed-good rendering. var webView = new WebView2 { Width = 1920, @@ -132,7 +130,6 @@ public sealed class WebView2Manager : IDisposable "document.documentElement.style.overflow='hidden';" + "document.body.style.background='transparent';" + "document.body.style.overflow='hidden';"); - _ = QueryContentBoundsAsync(source.Id); _ = CaptureFrame(source.Id); }; @@ -159,33 +156,30 @@ public sealed class WebView2Manager : IDisposable catch (Exception ex) { AppLog.Write($"WebView2Manager: navigate failed: {ex.Message}"); } } - // The page's real content rect (scrollWidth/Height) can be smaller than the - // viewport for widgets that don't fill it — the leftover area is dead space - // that must not stretch across the selection box. Crop the capture to the - // content bounds (anchored at 0,0) so the box shows exactly the widget. - private async Task QueryContentBoundsAsync(string sourceId) + // The widget's extent is measured from the captured frame itself: the + // bounding box of non-transparent pixels (the background is injected + // transparent, so the alpha channel IS the widget). This is immune to + // DOM timing (fonts/images/animations), needs no JS round-trip, and + // CANNOT break rendering: if the widget fills the canvas the box equals + // the full frame (identical to the confirmed-good uncropped display); if + // it floats in transparent margin, the crop maps it flush under the box. + internal static (int X, int Y, int W, int H) FindContentBounds(byte[] pixels, int pixW, int pixH, int stride) { - if (!_sessions.TryGetValue(sourceId, out var session) || session.Disposed) return; - try + int minX = pixW, minY = pixH, maxX = -1, maxY = -1; + for (var y = 0; y < pixH; y++) { - if (session.Control.CoreWebView2 is not { } cws) return; - var json = await cws.ExecuteScriptAsync( - "JSON.stringify({w:document.documentElement.scrollWidth||0," + - "h:document.documentElement.scrollHeight||0," + - "vw:window.innerWidth||0,vh:window.innerHeight||0})"); - if (string.IsNullOrEmpty(json) || json == "null") return; - using var doc = JsonDocument.Parse(json); - var root = doc.RootElement; - session.ViewportCssW = root.GetProperty("vw").GetDouble(); - session.ViewportCssH = root.GetProperty("vh").GetDouble(); - session.ContentCssW = root.GetProperty("w").GetDouble(); - session.ContentCssH = root.GetProperty("h").GetDouble(); - _ = CaptureFrame(sourceId); - } - catch (Exception ex) - { - AppLog.Write($"WebView2Manager: content bounds query failed: {ex.Message}"); + var row = y * stride; + for (var x = 0; x < pixW; x++) + { + if (pixels[row + x * 4 + 3] == 0) continue; + if (x < minX) minX = x; + if (x > maxX) maxX = x; + if (y < minY) minY = y; + if (y > maxY) maxY = y; + } } + if (maxX < minX || maxY < minY) return (0, 0, pixW, pixH); + return (minX, minY, maxX - minX + 1, maxY - minY + 1); } private async Task CaptureFrame(string sourceId) @@ -223,23 +217,15 @@ public sealed class WebView2Manager : IDisposable var stride = pixW * 4; - int cropW = pixW, cropH = pixH; - if (session.ViewportCssW >= 1 && session.ViewportCssH >= 1 && - session.ContentCssW >= 1 && session.ContentCssH >= 1) - { - var sx = pixW / session.ViewportCssW; - var sy = pixH / session.ViewportCssH; - cropW = (int)Math.Clamp(Math.Round(session.ContentCssW * sx), 1, pixW); - cropH = (int)Math.Clamp(Math.Round(session.ContentCssH * sy), 1, pixH); - } + var (cropX, cropY, cropW, cropH) = FindContentBounds(pixels, pixW, pixH, stride); var outPixels = pixels; - if (cropW != pixW || cropH != pixH) + if (cropW != pixW || cropH != pixH || cropX != 0 || cropY != 0) { var dstStride = cropW * 4; outPixels = new byte[dstStride * cropH]; for (var r = 0; r < cropH; r++) - Buffer.BlockCopy(pixels, r * stride, outPixels, r * dstStride, dstStride); + Buffer.BlockCopy(pixels, (cropY + r) * stride + cropX * 4, outPixels, r * dstStride, dstStride); } var wb = new WriteableBitmap(cropW, cropH, 96, 96, PixelFormats.Bgra32, null); diff --git a/TASKS.md b/TASKS.md index 3983441..a43ef13 100644 --- a/TASKS.md +++ b/TASKS.md @@ -917,14 +917,16 @@ capturing the primary display (which causes the infinity mirror effect). 8. ✅ Preview shows live web content (not just a blank rectangle) 9. ☐ Handle navigation errors, invalid URIs, timeout gracefully -**Rendering model (2026-08-28, canvas-size viewport, NO crop — the only creator-verified state):** +**Rendering model (2026-08-28, canvas-size viewport + ALPHA-BBOX crop):** the page renders at the MASTER CANVAS size (1920×1080), stable, never tracked (no reflow/truncation; -scrollbars suppressed via `overflow:hidden`). Capture is the FULL frame; `Stretch="Fill"` maps it -under the box. Image rendering verified good ("renders and resizes beautifully"). WEB BOX DEAD SPACE -STILL OPEN: previous cropping attempts all failed (JSON double-encode silently disabled the crop; -scrollWidth returns the full canvas for full-canvas wrappers; content-union measured an unsettled -layout on NavigationCompleted and BROKE the image). Any further crop needs creator approval first. -⚠️ `ExecuteScriptAsync` JSON-encodes the returned value — never return `JSON.stringify` from script. +scrollbars suppressed via `overflow:hidden`). `FindContentBounds` scans the Bgra32 capture and crops +to the bounding box of non-transparent pixels — the widget's true rendered extent, measured from the +frame itself (NO DOM query, immune to layout timing; can never truncate content; full-canvas widgets +fall through to the full frame = prior verified-good image). `Stretch="Fill"` maps the cropped frame +flush under the selection box → box hugs the widget on all four sides. `QueryContentBoundsAsync` + +the JS content-bounds script are DELETED. ⚠️ Lesson bank: never return `JSON.stringify` from +`ExecuteScriptAsync` (double-encodes); never measure DOM stuff on NavigationCompleted (unsettled +layout broke the image — `a5b9952`); measure rendered pixels instead. **Properties panel (2026-08-28):** web URI ✓/✕ icon buttons are `IsTabStop="False"` so Tab flows X→Y→W→H→URI; the ✕ button now clears the URI textbox (was reverting to the pre-accept snapshot). diff --git a/ytLive.Tests/WebView2ManagerTests.cs b/ytLive.Tests/WebView2ManagerTests.cs index 783dec4..867dd48 100644 --- a/ytLive.Tests/WebView2ManagerTests.cs +++ b/ytLive.Tests/WebView2ManagerTests.cs @@ -110,4 +110,48 @@ public sealed class WebView2ManagerTests Assert.Equal(System.Drawing.Color.Transparent, webView.DefaultBackgroundColor); }); } + + [Fact] + public void FindContentBounds_Returns_Bounding_Box_Of_NonTransparent_Pixels() + { + const int w = 10, h = 6; + var pixels = new byte[w * h * 4]; + for (int yy = 2; yy <= 4; yy++) + for (int xx = 3; xx <= 6; xx++) + { + var i = (yy * w + xx) * 4; + pixels[i] = 255; // B + pixels[i + 1] = 0; // G + pixels[i + 2] = 0; // R + pixels[i + 3] = 255; // A + } + + var (x, y, bw, bh) = WebView2Manager.FindContentBounds(pixels, w, h, w * 4); + + Assert.Equal((3, 2, 4, 3), (x, y, bw, bh)); + } + + [Fact] + public void FindContentBounds_No_Alpha_Means_No_Crop() + { + const int w = 10, h = 6; + var pixels = new byte[w * h * 4]; + + var (x, y, bw, bh) = WebView2Manager.FindContentBounds(pixels, w, h, w * 4); + + Assert.Equal((0, 0, w, h), (x, y, bw, bh)); + } + + [Fact] + public void FindContentBounds_Full_Opacity_Means_No_Crop() + { + const int w = 10, h = 6; + var pixels = new byte[w * h * 4]; + for (var i = 3; i < pixels.Length; i += 4) + pixels[i] = 255; + + var (x, y, bw, bh) = WebView2Manager.FindContentBounds(pixels, w, h, w * 4); + + Assert.Equal((0, 0, w, h), (x, y, bw, bh)); + } }