gramps
21db8a17b8
TASK 64 units B and C — the two pure halves, no wiring yet. - `Services/ChromaKeyDetector.cs`: samples the frame's outer ring (the screen is at the edges, the subject in the middle), reports Green/Blue/None plus the colour actually shot and a confidence. Confidence is coverage × hue tightness, which is what keeps a lawn or a football pitch from being offered as a keyable screen. Letterbox bars are excluded from the denominator rather than counted against the screen. Never decides anything: an offer is the creator's to accept. - `Services/VideoFilterChain.cs`: options → `-vf` (scale → colorkey → despill → format=bgra) plus a `Signature` so two layers keyed identically can share one decoder. Numbers clamp and an unparseable colour falls back to the app default, so ffmpeg is never handed a filter it rejects at spawn. - Keying is done by the bundled ffmpeg, not by us: verified against the shipped binary that `colorkey=…,format=bgra` emits straight-alpha BGRA (alpha 0 on keyed pixels), which is what the compositor already blends. No per-pixel loop lands in the 16.67 ms render budget. Test: `ChromaKeyDetectorTests` — green, blue, letterboxed, lawn (must say None), mixed scene, dark frame, plus the chain strings and signature sharing. Docs: TASK 64 queued in TASKS.md with its slice table; the decisions.md chroma-key reversal recorded as a reversal (it was buried for webcam background removal, not for supplied clips).
64 lines
4.4 KiB
Markdown
64 lines
4.4 KiB
Markdown
# TASK 64 — Green/blue screen (chroma) for video clips
|
|
|
|
> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here.
|
|
> Queued 2026-11-05 at the creator's direction, straight after TASK 21's audio slice.
|
|
|
|
**Goal:** a creator who drops in a clip shot against a green or blue screen gets the background
|
|
removed — without knowing what "chroma key", "similarity" or "spill" mean. The app *notices* the
|
|
screen and offers to remove it; the creator never has to pick a colour.
|
|
|
|
**Why this is not the retired "colour-key" item:** `docs/ai/decisions.md` buried chroma key in the
|
|
webcam background-removal context (ML model, lighting, NVIDIA gives it free — support hell). This is a
|
|
different job: the creator supplies the clip, ffmpeg does the arithmetic, there is no model and no
|
|
lighting to blame. The reversal is recorded there.
|
|
|
|
---
|
|
|
|
## 0. What already exists (verified 2026-11-05, bundled ffmpeg N-126342)
|
|
|
|
- `colorkey` / `chromakey` (colour, `similarity`, `blend`) and `despill` (`green`/`blue`, `mix`) are all
|
|
present in the shipped binary and slice-threaded.
|
|
- A probe of the bundled binary showed keyed pixels come out of the pipe with **alpha 0** and straight
|
|
(un-premultiplied) colour — `colorkey=…,format=bgra` → `63 175 0 0` vs `63 175 0 255` unkeyed. The
|
|
compositor already blends straight alpha, and media frames are not flagged `IsOpaque`, so no
|
|
compositor change is needed.
|
|
- Consequence: keying costs **zero app-side CPU** — it is one more term in the decode pipe's `-vf`,
|
|
which is also the only place that feeds **both** consumers (the shared preview bitmap and the
|
|
compositor's `LatestFrame`). A C# post-process would have to be wired twice.
|
|
|
|
## 1. Decisions (creator, 2026-11-05)
|
|
|
|
- **Suggest, never override.** Detection offers; the creator clicks. Opt-in setting for auto-apply.
|
|
- **Green and blue.** The keyed colour is the *detected* screen colour (real screens read 0x00B140,
|
|
not 0x00FF00), not a canonical green.
|
|
- **"Auto Green/Blue Screen"** is the setting name. No engineering words in the UI (never "chroma",
|
|
never "colour key"): the panel section is **Green/Blue Screen**, the sliders are **Amount**,
|
|
**Softness**, **Fringe**.
|
|
- **A clip is a clip** — alert clips get the same treatment from the same settings block.
|
|
- **High confidence only** may auto-apply; medium shows the offer; low says nothing.
|
|
- Settings are **per layer** (they belong to the layer in the UI). Sessions are keyed by
|
|
**path + settings signature**, so two layers with identical settings still share one decoder.
|
|
|
|
## 2. Slices (one Good Dog each)
|
|
|
|
| # | Slice | Test |
|
|
|---|---|---|
|
|
| A | ✅ media audio + shared clip pipe (TASK 21) | shipped `2670ac0` |
|
|
| B | ✅ Pure `Services/ChromaKeyDetector.cs`: sample a `VideoFrame` → green / blue / none + suggested colour + confidence | synthetic frames: green, blue, gradient, letterboxed, lawn (must say none) |
|
|
| C | ✅ Pure `Services/VideoFilterChain.cs`: options → `-vf` string (`scale → colorkey → despill → format=bgra`) | exact string assertions (encoder `FfmpegArgs` style) |
|
|
| D | Detect on add: one `-frames:v 1` probe (mirror `IFrameRateProbe`) → suggestion in the panel; no rendering change | add a green-screen file → suggestion appears with its colour |
|
|
| E | Apply: session key `path|key-settings`, filter chain in the pipe, level/mute untouched | keyed layer over art shows the art through keyed pixels **and** the preview bitmap carries alpha |
|
|
| F | Alert clips (falls out of E), tuning sliders, "Auto Green/Blue Screen" setting (App Settings), spill | slider reaches the chain and the restart preserves loop state |
|
|
|
|
## 3. Landmines for whoever builds this
|
|
|
|
- **Two consumers.** The preview bitmap and `LatestFrame` fork inside `MediaVideoSourceManager`; keying
|
|
must happen upstream (in the pipe), or one path shows green.
|
|
- **Parameter change = decoder restart** (`-vf` is fixed at spawn). Debounce, and carry loop + refcount
|
|
across the restart instead of tearing the session down.
|
|
- **Session identity.** `ResolveOutputFrame` and the preview-bitmap adoption must derive the session key
|
|
from the same helper as the panel, or a keyed and an unkeyed layer on one file fight over one decoder.
|
|
- **Detection is a second reader of the file** — one frame, async, never on the UI thread; a failed
|
|
probe means "no suggestion", never a broken layer.
|
|
- **Letterboxing** (black bars) breaks naive border sampling — detect and exclude them first.
|