TASK 51 slice 1: background-effects engine — bundled MediaPipe Selfie Segmenter (ONNX) + Remove (Unit: bg-effects engine)

- Model: Google's official MediaPipe Selfie Segmenter (landscape, Apache-2.0 per model card),
  converted to ONNX by tools/selfie-seg/convert_tflite_to_onnx.py (custom op
  Convolution2DTransposeBias -> ConvTranspose) and verified against MediaPipe's own runtime
  (verify_against_mediapipe.py: max |diff| 5e-5, 100% binary agreement). Embedded resource, 430 KB.
- Services/BackgroundEffects: IPersonSegmenter seam, OnnxPersonSegmenter (onnxruntime 1.31.0 CPU,
  never throws), BackgroundEffectProcessor (None passthrough, Remove = alpha mask). Not wired yet.
- Tests: Remove integration test (fake segmenter) + bundled-model contract test vs MediaPipe
  reference mask. Full suite 407/408 — the one failure is the documented interactive-desktop flake
  RealMouseDrag_OnTheLayerList_PersistsTheReorder (passed in the run before; code path untouched).
- Docs: ai.md reversal + section + licensing guard, THIRD-PARTY-NOTICES 11-12, TASK 51, indexes,
  MyMistakes recipe, HANDOFF.

References: obs-backgroundremoval (approach only — ONNX Runtime on CPU, 2 threads; GPL-3.0, no code
used) https://github.com/royshil/obs-backgroundremoval ; MediaPipe image segmenter
https://ai.google.dev/edge/mediapipe/solutions/vision/image_segmenter
This commit is contained in:
2026-10-09 09:24:13 -07:00
parent f8dd1079f3
commit 9f3c154d84
20 changed files with 796 additions and 78 deletions
+16
View File
@@ -15,6 +15,22 @@
## 🔬 Recipes registry
### CONVERTING A MEDIAPIPE .tflite (WITH CUSTOM OPS) TO ONNX — AND PROVING IT (RECIPE, TASK 51)
**Problem:** MediaPipe segmentation models end in the custom op `Convolution2DTransposeBias`;
tf2onnx and tflite2onnx refuse it. **Recipe:** `tools/selfie-seg/convert_tflite_to_onnx.py` —
parse the flatbuffer with the `tflite` pip package, emit ONNX with `onnx.helper`, transpose
NHWC→NCHW once at input/output, fold every `DEQUANTIZE` (fp16 weights) into fp32 initializers.
Weight layouts: CONV OHWI→OIHW; DEPTHWISE [1,H,W,C]→[C,1,H,W] with `group=C`; transpose-conv
[O,H,W,I]→[I,O,H,W]. TF `SAME` padding = ONNX `auto_pad=SAME_UPPER`.
- ⚠️ **Enum trap:** the custom op's options are the C-API `TfLiteTransposeConvParams`, whose
`padding` uses `TfLitePadding` (Unknown=0, **SAME=1**, VALID=2) — NOT the flatbuffer `Padding`
enum (SAME=0). Comparing against `Padding.SAME` rejects the real model.
- **Proof, not hope:** `pip install mediapipe` ships the TFLite runtime WITH the custom op. Feed
an image already at the model's exact input size (no resampling on either side) to MediaPipe's
`ImageSegmenter` and to onnxruntime; compare masks (`verify_against_mediapipe.py`, exit 0 at
max |diff| < 1e-3 — we got 5e-5). Work in a throwaway venv under `/tmp/opencode/`.
### ⛔ NEVER PRINT CREDENTIALS.md — NOT EVEN "FILTERED" (2026-10-09 incident)
**What happened:** to look for an API key, a `sed -n '1,40p' CREDENTIALS.md` piped through a