gramps/LlamaCasty
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

TASK 51 slice 1: background-effects engine — bundled MediaPipe Selfie Segmenter (ONNX) + Remove (Unit: bg-effects engine)

Browse Source 
- Model: Google's official MediaPipe Selfie Segmenter (landscape, Apache-2.0 per model card),
  converted to ONNX by tools/selfie-seg/convert_tflite_to_onnx.py (custom op
  Convolution2DTransposeBias -> ConvTranspose) and verified against MediaPipe's own runtime
  (verify_against_mediapipe.py: max |diff| 5e-5, 100% binary agreement). Embedded resource, 430 KB.
- Services/BackgroundEffects: IPersonSegmenter seam, OnnxPersonSegmenter (onnxruntime 1.31.0 CPU,
  never throws), BackgroundEffectProcessor (None passthrough, Remove = alpha mask). Not wired yet.
- Tests: Remove integration test (fake segmenter) + bundled-model contract test vs MediaPipe
  reference mask. Full suite 407/408 — the one failure is the documented interactive-desktop flake
  RealMouseDrag_OnTheLayerList_PersistsTheReorder (passed in the run before; code path untouched).
- Docs: ai.md reversal + section + licensing guard, THIRD-PARTY-NOTICES 11-12, TASK 51, indexes,
  MyMistakes recipe, HANDOFF.

References: obs-backgroundremoval (approach only — ONNX Runtime on CPU, 2 threads; GPL-3.0, no code
used) https://github.com/royshil/obs-backgroundremoval ; MediaPipe image segmenter
https://ai.google.dev/edge/mediapipe/solutions/vision/image_segmenter
This commit is contained in:
gramps
2026-10-09 09:24:13 -07:00
parent f8dd1079f3
commit 9f3c154d84
20 changed files with 796 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files
HANDOFF.md
+32 -50
View File
@@ -1,60 +1,42 @@
# HANDOFF — 2026-10-08 (zoom box shipped, READY FOR PUSH)
# HANDOFF — 2026-10-09 (TASK 51 slice 1 shipped: background-effects engine)
## Branch
- main (no feature branch)
- main (no feature branch). Pushed through `f8dd107`; the TASK 51 slice-1 commit sits on top,
**not pushed** (push at the user's call).
## Push readiness
- **5 commits ahead of origin/main, clean to push** (fetch confirmed origin has not moved):
`966e718` zoom box visuals → `4b3d82f` zoom as layered + TASK 51 booking → `6bff21b`
anchored-capture rework → `948dd08` dims-shading fix → (docs unit: TASK 52 record +
numbering fixes + this HANDOFF).
- The pinned-chat work unit below is **uncommitted on purpose** — `git push` sends only
commits, so pushing is safe; the dirty files stay local.
## Dirty files (NOT mine — do NOT commit or co-edit)
- The in-flight "pinned chat" work unit: `Models/YouTube.cs`, `ViewModels/MainViewModel.Chat.cs`,
`Controls/LeftPanel.xaml`, untracked `Helpers/RelayCommandGeneric.cs`.
- The creator's untracked logo art in `Assets/` (`llamacasty text.*`, `llamacasty-text-nobg*.png`,
`thin logo wordmark.png`).
## Dirty files (NOT mine — the in-flight "pinned chat" work unit; do NOT commit or co-edit)
- `Models/YouTube.cs`
- `ViewModels/MainViewModel.Chat.cs`
- `Controls/LeftPanel.xaml` (pinned-message UI)
- Untracked: `Helpers/RelayCommandGeneric.cs`
## What just happened (2026-10-08, third zoom pass — dims shading)
The zoom box shipped as a **loupe** (`4b3d82f`), then was corrected to an **anchored-capture
magnifier over the full composite drawn last** (commit `6bff21b`). The final correction: as the box
is dragged, the dim rectangles (55% black "div shading" around the box) were sliding under the
capture region — shown inside the box because its `VisualBrush` samples all of `CanvasGrid`, and the
`DimRects` ItemsControl previously lived INSIDE `CanvasGrid`.
Fix: the `DimRects` overlay is now a **sibling of `CanvasGrid`** (renders above the composed
content, below the `ZoomBox`), never a descendant — so the zoom sample can never pick up the shading
and the captured area stays clear no matter where the box is dragged. Behavior consequence to note:
in non-zoom ("quality dims") mode the dims now also shade the global overlays (brand flash, alert
ticker, social bar) that sit above them in the old tree — consistent with "chrome outside the crop
is dimmed". Structural invariant added to `ZoomBoxInteractionTests`: `DimRectsLayer` is a direct
child of `PreviewCanvas`, not inside `CanvasGrid`, and renders below the `ZoomBox`.
## Zoom feature state
- **TASK 52 (zoom box) is now a first-class record** — see [`TASKS/task-52-zoom-box.md`](TASKS/task-52-zoom-box.md):
feature provenance + the three creator corrections, shapes, implementation map, tests.
TASK 51 (webcam background Remove + Blur) is unblocked for the next unit.
- Models: `ZoomState.cs`, `ZoomShape.cs` (Circle/VerticalRectangle/HorizontalRectangle), `ZoomSceneElement.cs`.
- ViewModel (`MainViewModel.Zoom.cs`): `ActivateZoom`/`ToggleZoom`/`DismissZoom`/`StartZoomMove`/`UpdateZoomMove`/`EndZoomMove` + `RebuildDimRectsForZoom`; transient element healed/mirrored by `HealZoomLayers`/`EnsureZoomLayer`.
- Output: box composited last via `BlitZoom`/`ZoomMagnifier`; output crop = tier (box never the crop).
- Preview: `ZoomBox` + `DimRectsLayer` are both SIBLINGS of `CanvasGrid` (box above dims above content); box samples the full composite at the fixed capture, 2x; circle→ellipse clip, rect/square outline per shape; `LeftPanel` hides the placeholder row via `IsZoomLayer`.
- Rectangle shape = 4:3; Circle square; Phone 9:16. Capture anchored on click (`CaptureX/Y`); dragging the box never changes it.
## What just happened — TASK 51 slice 1 (engine)
- **Model:** Google's official MediaPipe Selfie Segmenter (landscape, 144×256, Apache-2.0),
converted to ONNX by our own `tools/selfie-seg/convert_tflite_to_onnx.py` and **verified against
MediaPipe's own runtime** (`verify_against_mediapipe.py`, max |diff| 5e-5). Embedded in the exe
(`Assets/Models/selfie_segmenter_landscape.onnx`, 430 KB, in git).
- **Code:** `Services/BackgroundEffects/` — `IPersonSegmenter` seam, `OnnxPersonSegmenter`
(onnxruntime 1.31.0 CPU, never throws), `BackgroundEffectProcessor` (None passthrough, Remove =
alpha mask, Blur = passthrough until slice 2). `Models/BackgroundEffect.cs` enum.
- **Not wired:** nothing calls it yet — no config, UI, or camera hookup. Zero risk to the live path.
- **Docs:** ai.md (stale "background removal PERMANENTLY OUT" bullet replaced by the reversal + a
"Webcam background effects" section + a model-provenance licensing guard), THIRD-PARTY-NOTICES
items 11 (ONNX Runtime, MIT) + 12 (MediaPipe model, Apache-2.0 with §4b change note),
TASK 51 file + TASKS.md row, Services/Models indexes, MyMistakes conversion recipe.
## Tests
- Full suite: **406 passed / 0 failed** on a clean full run (the pre-existing audio-loopback timing
flake and `RealMouseDrag`'s interactive-desktop requirement recur occasionally — not zoom
regressions). Zoom tests 3/3 green, 0 build warnings.
- Full suite **408 passed / 0 failed**, 0 build warnings. New: `BackgroundEffectEngineTests` (Remove
integration test with a fake segmenter; bundled-model contract test vs a MediaPipe reference mask,
fixtures in `ytLive.Tests/Fixtures/`).
## Known issues / follow-ups
- **Background "Use default background" toggle still not working** (carried from 2026-10-07):
binding exists (`UseDefaultBackground`), VM handler invalidates bake; investigate when time permits.
- The 3px selection / crop-outline strokes could still show in the zoom box IF the capture overlaps
them in edit mode — not shading, thin outlines only; the dims (the user's complaint) are gone.
- `onnxruntime.dll` is ~17 MB; an RID-less build copies every RID's native folder — TASK 48's MSIX
publish must pin win-x64.
- `THIRD-PARTY-NOTICES.txt` header still says "paid, closed-source product" (pre free-forever) — noted,
not changed (out of scope).
- Carried: "Use default background" toggle still not working (from 2026-10-07).
## Next step
- User runs the app live to eyeball: drag the zoom box anywhere and the captured content stays clean
(no dim shading in the box; the scene outside the box stays dimmed). Then TASK 51 (webcam
background Remove + Blur) per `TASKS/task-51-webcam-bg-effects.md`, and the background-toggle
investigation if still open.
- TASK 51 slice 2: **Blur** in `BackgroundEffectProcessor` (separable box/gaussian, strength), one
integration test. Then slice 3 config + persistence, slice 4 camera-path hookup + perf check on the
creator's box, slice 5 properties-panel UI.