gramps
7827d239e2
The commercial plan was spread over five documents written in five different weeks, and they contradicted each other: product.md sold the Store, decisions.md called the Store dead then re-opened, Distribution.md (736 lines) still described Polar as a download host, MARCOM.md quoted "no subscription" next to a paid tier, and README.md promised "free forever". An outsider could not tell what ships. docs/ai/commercial.md is now the only place money is decided: free = whole app at a 921,600 px / 30 fps output profile covering the recording as well as the stream; Pro = $10/mo or $400 lifetime through Polar, key bound to the buyer's YouTube channel ID via a claim page, 2 activations, 7–14 day lease with offline grace, fail closed, never downgrade a running stream; one portable zip + winget primary, Steam and the Store parked; Azure Trusted Signing (~$9.99/mo), no EV; Founding 200 promo; ~$135/yr costs; anti-piracy posture and what is permanently rejected. Everything else now points at it instead of restating it: - product.md — money sections replaced with pointers; packaging rewritten to winget-first, Store parked; signing updated to Trusted Signing. - decisions.md — graveyard with cause of death for free-forever, the Store ordering, the updater, and the 2026-10-30 reversal of the quality-ceiling rejection. - Distribution.md — rewritten to release mechanics only (artifact, GitHub Releases, winget manifest, signing, checklist, takedown): 736 → 120 lines. - TASKS.md — TASK 54 rewritten, TASK 48's route settled, TASK 10 superseded, five new rows (entitlement seam, output-profile gate, licence UI, system messages, packaging). - README.md — "free forever" replaced with the real split. - capture.md — obs-backgroundremoval is GPL-3.0 and so licence-INCOMPATIBLE since the proprietary EULA; the old "compatible" line was falsified by that change. - TASKS/task-10, task-48, task-51, task-54 — superseded-banner at the top of each, so none of them reads as current policy. - HANDOFF.md — rewritten: plan settled, next code unit is the entitlement seam. MARCOM.md and MONETIZATION.md (gitignored) were corrected in place: policy pointer at the top of each, the flash removed from every live claim, the Polar fee break-even corrected to ~$900/mo, and the projections flagged as written against a dead model.
146 lines
10 KiB
Markdown
146 lines
10 KiB
Markdown
# TASK 51 — Webcam background effects: Remove + Blur (v1 requirement)
|
||
|
||
> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here.
|
||
> **Booked 2026-10-08 by creator ruling** — *"we're doing the bg removal and blur for v1 - book it, Danno"*.
|
||
> This **reverses** the 2026-10-05 "background removal: out of product" exclusion (TASK 3 item 19).
|
||
|
||
**Status:** ✅ **Built 2026-10-09 — all five slices shipped** (engine, Blur, config + persistence,
|
||
camera hookup, UI). **Creator's real-camera check done 2026-10-09 and both findings addressed:**
|
||
🎯 **Remove "jittery" — FIXED** (temporal mask smoothing, `MaskSmoother` — 0.75×new + 0.25×previous
|
||
momentum, weights from the OBS/Jitsi consensus); 🎯 **Blur "heavy" — FIXED** (quadratic radius curve,
|
||
max 12→6 small px, default 0.5 = 2 — StreamFX-consensus scale). **Pending: creator's recheck of both
|
||
on a real camera.**
|
||
|
||
**Goal:** per-webcam **Background Remove** and **Background Blur** on the live webcam layer,
|
||
selected per element in its properties panel, composited into stream + recording like any
|
||
other layer. No external downloads, no hardware requirement, works on any machine.
|
||
Replacement backgrounds are **free via layers** — drop any element behind the webcam.
|
||
|
||
---
|
||
|
||
## Why this is now IN (and what changed)
|
||
|
||
The 2026-10-05 exclusion was based on: an in-app AI model = a runtime, a GPU/driver matrix,
|
||
per-scene lighting tuning, and "a vcam app does it free." The ruling is reversed because the
|
||
delivery shape changed:
|
||
|
||
1. **Bundled small ONNX model (~2–5 MB)** — MediaPipe Selfie-Segmentation family —
|
||
shipped in the installer/fixture (TASK 48 MSIX), executed by **onnxruntime** (CPU default;
|
||
DirectML optional later). No NVIDIA SDK, no GPU requirement, no per-client download.
|
||
2. **Quality reality (creator):** a 720p webcam displays at ~15% of a 1080p frame — premium
|
||
segmentation quality is invisible at that size; commodity class is sufficient.
|
||
3. **⛔ Ruled out: NVIDIA Video Effects SDK (VFX / Maxine, AI Green Screen + Background Blur).**
|
||
Works on RTX 2000+/R570+ only, and needs a ~1 GB user-side redistributable on first use.
|
||
Creator: *"I don't think I'm putting a 1gb download requirement on a client box."*
|
||
|
||
## Constraints — non-negotiable
|
||
|
||
1. **No external download at runtime.** Model ships in the installer. (Store policy 10.2.2 —
|
||
same posture as the ffmpeg bundling in TASK 48 item 1.)
|
||
2. **No hardware requirement.** CPU path must work; DirectML is an optional accelerator.
|
||
3. **Per-element (Meld-style), not global.** Each webcam element carries its own effect.
|
||
4. **No feature gating.** Background effects are in the free tier (2026-10-31 freemium keeps every
|
||
feature free; the paid delta is output quality only — `docs/ai/commercial.md`). The old "only the
|
||
branding flash is paid" line here was superseded in 2026-10-08, and the flash delta itself died
|
||
on 2026-10-30.
|
||
5. **Scene change / webcam add/remove must not crash or stall the pump** — a failing model
|
||
run skips to passthrough, never drops the layer.
|
||
|
||
## Out of scope for v1 (same engine later)
|
||
|
||
- ❌ **"Remove chair"-style object selection / quality tiers** — not a picker engine
|
||
- ❌ **Super resolution / denoise / prettify passes**
|
||
- ❌ **Virtual camera output** — permanently out of product (no vcam)
|
||
|
||
## 1. Engine
|
||
|
||
- ✅ **Model bundled (slice 1, 2026-10-09):** Google's official MediaPipe Selfie Segmenter
|
||
**landscape** (144×256, Apache-2.0 per the model card — license + changes recorded in
|
||
THIRD-PARTY-NOTICES item 12). Converted to ONNX by `tools/selfie-seg/convert_tflite_to_onnx.py`
|
||
(custom op `Convolution2DTransposeBias` → `ConvTranspose`); verified against MediaPipe's own
|
||
runtime by `tools/selfie-seg/verify_against_mediapipe.py` (max |diff| 5e-5). 430 KB,
|
||
`EmbeddedResource` in the exe. Rejected sources: PINTO #109 (ML Kit-extracted, unclear license),
|
||
obs-backgroundremoval code (GPL-3.0) — referenced for approach only (ONNX Runtime, CPU, 2 threads).
|
||
- ✅ `Microsoft.ML.OnnxRuntime` 1.31.0, CPU EP (`OnnxPersonSegmenter`). Failing load/run →
|
||
logs once, returns null, caller passes through. DirectML still optional/later.
|
||
- ✅ **Blur (slice 2, 2026-10-09):** background box-averaged to ¼ res, 3 separable box-blur passes
|
||
(≈ gaussian, radius by strength), mask-blended back; person pixels untouched (`BackgroundEffectProcessor.ApplyBlur`).
|
||
- ✅ Remove = alpha = mask outside the person (`BackgroundEffectProcessor`), `IsOpaque = false`.
|
||
The compositor already alpha-blends per element — no compositor change expected (confirm in
|
||
the hookup slice).
|
||
- ☐ Feed the **VideoFrame seam** (`Services/VideoFrame.cs` doc explicitly anticipates
|
||
*"every future capture source (screen, background-removed webcam) feeds the same seam"*):
|
||
the frame handed to BOTH the preview bitmap path and the compositor's
|
||
`_cameraManager.GetLatestFrame` is the processed frame. One processing point, two consumers.
|
||
|
||
## 2. ☐ Config surface + persistence
|
||
|
||
- ✅ `WebcamSceneConfig` gains `BackgroundEffect` (`None | Remove | Blur`) + `BackgroundStrength` (slice 3).
|
||
- ✅ Persist via the existing `WebcamSceneConfig` table path in `LayoutStore.Save.cs`/
|
||
`Load.cs` (new columns + migration). Follow `LayoutStore.Migrations.cs` conventions.
|
||
- ✅ **Properties UI (creator spec 2026-10-09, slice 5):** in the webcam's properties in the **Layers
|
||
panel**, two **pill switches** — "Remove background" and "Blur background" — using the existing
|
||
`PillToggle` style (`Themes/Controls.xaml`, already used in `Controls/LeftPanel.xaml`). They act
|
||
as **radio buttons**: turning one on turns the other off; at most one is active; both off =
|
||
`None`. Bound to the element's `BackgroundEffect`, per webcam element. (Supersedes the old
|
||
"effect dropdown" idea.) **Plus a strength slider** (creator: "keep", 2026-10-09) — Blur
|
||
radius / Remove edge softness via `Strength`.
|
||
- ☐ Replace-background affordance: **none in v1** — the creator drops any layer behind the
|
||
webcam. Effect preview must not obscure the fact that layers below still show through.
|
||
|
||
## 3. ☐ Test seam (Good Dog — one integration test per unit)
|
||
|
||
- ✅ Model-runner interface seam (`IPersonSegmenter`) so tests inject a fake segmenter.
|
||
- ✅ The **processed frame reaches both consumers** (preview bitmap + compositor frame), raw stays raw (`CameraManagerTests`, slice 4).
|
||
- ✅ `None` = same-instance passthrough of the raw frame (`BackgroundEffectEngineTests`).
|
||
- ✅ `Remove` = alpha 0 outside the person, RGB untouched, non-opaque (`BackgroundEffectEngineTests`);
|
||
✅ `Blur` = background smoothed, person pixel-identical (`BackgroundEffectEngineTests`).
|
||
- ✅ **Bundled-model contract:** the embedded ONNX model through the C# preprocessing reproduces
|
||
MediaPipe's reference mask on a fixture (≥99.5% agreement). This is a fixed, deterministic
|
||
contract check of the shipped bytes — not a camera/lighting quality test.
|
||
- ✅ Config → persistence roundtrip per scene + legacy-DB migration (`LayoutStorePersistenceTests`).
|
||
- ☐ Engine failure → passthrough + single log, layer never dropped (code path exists; dedicated
|
||
test lands with the hookup slice).
|
||
- ☐ Real inference: **manual on the creator's box** (camera + lighting), not in CI.
|
||
- ✅ **Perf, synthetic 720p (Debug):** segment 1.6 ms, Remove 3.3 ms, Blur 17.5 ms (after making the
|
||
upsampling separable — naive was 46/80 ms). Runs on its own worker, off the raw path.
|
||
☐ **Real numbers on the creator's box:** `startup.log` → "background effects avg … ms/frame".
|
||
|
||
## 4. Sequencing
|
||
|
||
- ✅ Separate work unit after the zoom build (zoom shipped 2026-10-08).
|
||
- Slice order: 1 engine ✅ → 2 Blur ✅ → 3 config + persistence ✅ → 4 camera-path hookup ✅ (one
|
||
processing point, both consumers) + perf check → 5 properties-panel UI ✅.
|
||
|
||
## 5. Creator's real-camera check (2026-10-09) — jitter FIXED, blur heavy deferred
|
||
|
||
- 🎯 **Finding: "removal is jittery"** (the edge flickers frame to frame). Root cause: the segmenter
|
||
classifies each frame independently — 256×144 mask, bilinear upsample — zero temporal coherence.
|
||
**Fix shipped: `MaskSmoother`** blends each raw mask with the previous smoothed one
|
||
(`CurrentWeight` = 0.75 × new + 0.25 × previous), wired at the ONE production point
|
||
(`CameraManager.RunEffectWorker`, per `CameraSession`). Why per-session and not in the segmenter:
|
||
the `OnnxPersonSegmenter` is shared app-wide (`MainViewModel:284-285`, one processor), and the
|
||
bundled-model contract test pins its raw per-frame output. Why 0.75/0.25: OBS's
|
||
obs-backgroundremoval smoke-tests `temporal_smooth_factor` at 0.85 default; Jitsi's webcam
|
||
background filter (jitsi/jitsi-meet#17080) found 0.7/0.3 best (0.5/0.5 smears on motion). Null mask
|
||
resets history; first frame passes through; no per-frame allocation (two ping-pong buffers — a
|
||
256×144 mask is 147 KB > the 85 KB LOH). Integration test: `CameraManagerTests.Temporal_Smoothing_…`
|
||
(two consecutive masks disagree at an edge → second frame keeps partial alpha where raw would cut).
|
||
- 🎯 **Finding: "blur is heavy"** — the OLD curve was linear 1→12 downscaled px (≈ 26–50 full-res px),
|
||
far past any moderate background blur. **Fix shipped: quadratic radius** —
|
||
`BackgroundEffectProcessor.BlurRadius(strength)` = 1 + 5·s²: default 0.5 = 2 small px (≈ a third of
|
||
the old default's spread), max = 6 small px (≈ 24 full px). Why these numbers: StreamFX defaults its
|
||
Box blur Size to 5 and its author stops downscale-based blurs around 8 px (cited). Integration test
|
||
`BackgroundEffectEngineTests.Blur_default_strength_is_gentle_and_full_strength_is_strong` pins the
|
||
mapping (BlurRadius(0)=1, 0.5→2, 1→6) and the step-edge behaviour: default smears light ~9px into
|
||
the dark half vs ~26px at full strength. The existing slice-2 blur test (person sharp + checkerboard
|
||
smoothed) passes unchanged.
|
||
- ✅ `startup.log` perf line present on the real box ("background effects avg … ms/frame").
|
||
|
||
## Related (not duplicated here)
|
||
|
||
- [`task-48-distribution-msix.md`](task-48-distribution-msix.md) — the model ships in the
|
||
MSIX; bundled-binary posture established by the ffmpeg work.
|
||
- `TASKS.md` Webcam cadence section — the "⛔ no background removal" line is superseded
|
||
by this ruling; the **judder/compositor cost work is unaffected and still heads its lane.**
|
||
- `research-store-certification.md` — 11.12/10.2.2 posture; nothing here sends data off-box. |