Files

T

gramps 21db8a17b8 feat(chroma): detect a green or blue screen and build the key filter

TASK 64 units B and C — the two pure halves, no wiring yet.

- `Services/ChromaKeyDetector.cs`: samples the frame's outer ring (the screen is at the
  edges, the subject in the middle), reports Green/Blue/None plus the colour actually shot
  and a confidence. Confidence is coverage × hue tightness, which is what keeps a lawn or
  a football pitch from being offered as a keyable screen. Letterbox bars are excluded from
  the denominator rather than counted against the screen. Never decides anything: an offer
  is the creator's to accept.
- `Services/VideoFilterChain.cs`: options → `-vf` (scale → colorkey → despill → format=bgra)
  plus a `Signature` so two layers keyed identically can share one decoder. Numbers clamp and
  an unparseable colour falls back to the app default, so ffmpeg is never handed a filter it
  rejects at spawn.
- Keying is done by the bundled ffmpeg, not by us: verified against the shipped binary that
  `colorkey=…,format=bgra` emits straight-alpha BGRA (alpha 0 on keyed pixels), which is what
  the compositor already blends. No per-pixel loop lands in the 16.67 ms render budget.

Test: `ChromaKeyDetectorTests` — green, blue, letterboxed, lawn (must say None), mixed scene,
dark frame, plus the chain strings and signature sharing.

Docs: TASK 64 queued in TASKS.md with its slice table; the decisions.md chroma-key reversal
recorded as a reversal (it was buried for webcam background removal, not for supplied clips).

2026-11-05 08:05:18 -08:00

4.4 KiB

Raw Blame History

TASK 64 — Green/blue screen (chroma) for video clips

Catalog: TASKS.md — status and requirements live here. Queued 2026-11-05 at the creator's direction, straight after TASK 21's audio slice.

Goal: a creator who drops in a clip shot against a green or blue screen gets the background removed — without knowing what "chroma key", "similarity" or "spill" mean. The app notices the screen and offers to remove it; the creator never has to pick a colour.

Why this is not the retired "colour-key" item: docs/ai/decisions.md buried chroma key in the webcam background-removal context (ML model, lighting, NVIDIA gives it free — support hell). This is a different job: the creator supplies the clip, ffmpeg does the arithmetic, there is no model and no lighting to blame. The reversal is recorded there.

0. What already exists (verified 2026-11-05, bundled ffmpeg N-126342)

colorkey / chromakey (colour, similarity, blend) and despill (green/blue, mix) are all present in the shipped binary and slice-threaded.
A probe of the bundled binary showed keyed pixels come out of the pipe with alpha 0 and straight (un-premultiplied) colour — colorkey=…,format=bgra → 63 175 0 0 vs 63 175 0 255 unkeyed. The compositor already blends straight alpha, and media frames are not flagged IsOpaque, so no compositor change is needed.
Consequence: keying costs zero app-side CPU — it is one more term in the decode pipe's -vf, which is also the only place that feeds both consumers (the shared preview bitmap and the compositor's LatestFrame). A C# post-process would have to be wired twice.

1. Decisions (creator, 2026-11-05)

Suggest, never override. Detection offers; the creator clicks. Opt-in setting for auto-apply.
Green and blue. The keyed colour is the detected screen colour (real screens read 0x00B140, not 0x00FF00), not a canonical green.
"Auto Green/Blue Screen" is the setting name. No engineering words in the UI (never "chroma", never "colour key"): the panel section is Green/Blue Screen, the sliders are Amount, Softness, Fringe.
A clip is a clip — alert clips get the same treatment from the same settings block.
High confidence only may auto-apply; medium shows the offer; low says nothing.
Settings are per layer (they belong to the layer in the UI). Sessions are keyed by path + settings signature, so two layers with identical settings still share one decoder.

2. Slices (one Good Dog each)

#	Slice	Test
A	✅ media audio + shared clip pipe (TASK 21)	shipped `2670ac0`
B	✅ Pure `Services/ChromaKeyDetector.cs`: sample a `VideoFrame` → green / blue / none + suggested colour + confidence	synthetic frames: green, blue, gradient, letterboxed, lawn (must say none)
C	✅ Pure `Services/VideoFilterChain.cs`: options → `-vf` string (`scale → colorkey → despill → format=bgra`)	exact string assertions (encoder `FfmpegArgs` style)
D	Detect on add: one `-frames:v 1` probe (mirror `IFrameRateProbe`) → suggestion in the panel; no rendering change	add a green-screen file → suggestion appears with its colour
E	Apply: session key `path	key-settings`, filter chain in the pipe, level/mute untouched
F	Alert clips (falls out of E), tuning sliders, "Auto Green/Blue Screen" setting (App Settings), spill	slider reaches the chain and the restart preserves loop state

3. Landmines for whoever builds this

Two consumers. The preview bitmap and LatestFrame fork inside MediaVideoSourceManager; keying must happen upstream (in the pipe), or one path shows green.
Parameter change = decoder restart (-vf is fixed at spawn). Debounce, and carry loop + refcount across the restart instead of tearing the session down.
Session identity. ResolveOutputFrame and the preview-bitmap adoption must derive the session key from the same helper as the panel, or a keyed and an unkeyed layer on one file fight over one decoder.
Detection is a second reader of the file — one frame, async, never on the UI thread; a failed probe means "no suggestion", never a broken layer.
Letterboxing (black bars) breaks naive border sampling — detect and exclude them first.

4.4 KiB Raw Blame History