# TASK 64 — Green/blue screen (chroma) for video clips

> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here.
> Queued 2026-11-05 at the creator's direction, straight after TASK 21's audio slice.

**Goal:** a creator who drops in a clip shot against a green or blue screen gets the background
removed — without knowing what "chroma key", "similarity" or "spill" mean. The app *notices* the
screen and offers to remove it; the creator never has to pick a colour.

**Why this is not the retired "colour-key" item:** `docs/ai/decisions.md` buried chroma key in the
webcam background-removal context (ML model, lighting, NVIDIA gives it free — support hell). This is a
different job: the creator supplies the clip, ffmpeg does the arithmetic, there is no model and no
lighting to blame. The reversal is recorded there.

---

## 0. What already exists (verified 2026-11-05, bundled ffmpeg N-126342)

- `colorkey` / `chromakey` (colour, `similarity`, `blend`) and `despill` (`green`/`blue`, `mix`) are all
  present in the shipped binary and slice-threaded.
- A probe of the bundled binary showed keyed pixels come out of the pipe with **alpha 0** and straight
  (un-premultiplied) colour — `colorkey=…,format=bgra` → `63 175 0 0` vs `63 175 0 255` unkeyed. The
  compositor already blends straight alpha, and media frames are not flagged `IsOpaque`, so no
  compositor change is needed.
- Consequence: keying costs **zero app-side CPU** — it is one more term in the decode pipe's `-vf`,
  which is also the only place that feeds **both** consumers (the shared preview bitmap and the
  compositor's `LatestFrame`). A C# post-process would have to be wired twice.

## 1. Decisions (creator, 2026-11-05)

- **Suggest, never override.** Detection offers; the creator clicks. Opt-in setting for auto-apply.
- **Green and blue.** The keyed colour is the *detected* screen colour (real screens read 0x00B140,
  not 0x00FF00), not a canonical green.
- **"Auto Green/Blue Screen"** is the setting name. No engineering words in the UI (never "chroma",
  never "colour key"): the panel section is **Green/Blue Screen**, the sliders are **Amount**,
  **Softness**, **Fringe**.
- **A clip is a clip** — alert clips get the same treatment from the same settings block.
- **High confidence only** may auto-apply; medium shows the offer; low says nothing.
- Settings are **per layer** (they belong to the layer in the UI). Sessions are keyed by
  **path + settings signature**, so two layers with identical settings still share one decoder.

## 2. Slices (one Good Dog each)

| # | Slice | Test |
|---|---|---|
| A | ✅ media audio + shared clip pipe (TASK 21) | shipped `2670ac0` |
| B | ✅ Pure `Services/ChromaKeyDetector.cs`: sample a `VideoFrame` → green / blue / none + suggested colour + confidence | synthetic frames: green, blue, gradient, letterboxed, lawn (must say none) |
| C | ✅ Pure `Services/VideoFilterChain.cs`: options → `-vf` string (`scale → colorkey → despill → format=bgra`) | exact string assertions (encoder `FfmpegArgs` style) |
| D | ✅ Detect on add: one `-frames:v 1` probe (mirror `IFrameRateProbe`) → suggestion in the panel; no rendering change | add a green-screen file → suggestion appears with its colour |
| E | ✅ Apply: session key `path|key-settings`, filter chain in the pipe, level/mute untouched | keyed layer over art shows the art through keyed pixels **and** the preview bitmap carries alpha |
| F | ✅ "Auto Green/Blue Screen" opt-in (App Settings; high confidence only, and the panel says it was automatic), alert clips keyed through the same settings block, spill shipped as the **Fringe** switch | slider reaches the chain and the restart preserves loop state |

## 3. Landmines for whoever builds this

- **Two consumers.** The preview bitmap and `LatestFrame` fork inside `MediaVideoSourceManager`; keying
  must happen upstream (in the pipe), or one path shows green.
- **Parameter change = decoder restart** (`-vf` is fixed at spawn). Debounce, and carry loop + refcount
  across the restart instead of tearing the session down.
- **Session identity.** `ResolveOutputFrame` and the preview-bitmap adoption must derive the session key
  from the same helper as the panel, or a keyed and an unkeyed layer on one file fight over one decoder.
- **Detection is a second reader of the file** — one frame, async, never on the UI thread; a failed
  probe means "no suggestion", never a broken layer.
- **Letterboxing** (black bars) breaks naive border sampling — detect and exclude them first.

## 4. Known limits (honest)

- The offer is made **when a clip is added or changed** — not at every launch. The settings persist,
  so a keyed layer comes back keyed; it just does not re-offer the probe.
- Auto-apply fires only at high confidence. Medium confidence always asks, low says nothing.
- The key is the layer's, but the decode is shared per **path + look**: two layers on one file with
  different Amount/Softness values are two decoders (and therefore two audio streams, muted upstream
  unless enabled) — rare, and the alternative is silently sharing the wrong picture.