gramps
7bb1207fb2
TASK 64 slice F — the two things left after the feature worked by hand. - App Settings → **Video clips → Auto Green/Blue Screen** (off by default). When on, a high-confidence detected screen is keyed as soon as the clip is added; the panel says "Removed automatically" so an automatic change is never silent. Medium confidence still asks, low says nothing — a wrong automatic key eats the picture. - **A clip is a clip:** the alert box's own replacement clip keys through the same settings block (one alert box per layout, so its settings are the ones the next alert uses), and choosing an alert clip runs the same detector. The Green/Blue Screen panel now appears for a video layer *or* an alert box with a custom clip (`Source.ShowsChromaKey`). - Spill ships as the creator-facing **Fringe** switch; nothing further to tune behind it. Test: `ChromaKeyApplyTests` gains the auto-apply case (no click, still explicit in the panel) and an alert-clip case (its pipe asks ffmpeg for colorkey+despill with the blue screen type). Docs: TASK 64 complete in TASKS.md, known limits recorded in the task file, HANDOFF rewritten. verify.sh: 546 passed, 0 failed, 0 warnings.
73 lines
5.0 KiB
Markdown
73 lines
5.0 KiB
Markdown
# TASK 64 — Green/blue screen (chroma) for video clips
|
|
|
|
> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here.
|
|
> Queued 2026-11-05 at the creator's direction, straight after TASK 21's audio slice.
|
|
|
|
**Goal:** a creator who drops in a clip shot against a green or blue screen gets the background
|
|
removed — without knowing what "chroma key", "similarity" or "spill" mean. The app *notices* the
|
|
screen and offers to remove it; the creator never has to pick a colour.
|
|
|
|
**Why this is not the retired "colour-key" item:** `docs/ai/decisions.md` buried chroma key in the
|
|
webcam background-removal context (ML model, lighting, NVIDIA gives it free — support hell). This is a
|
|
different job: the creator supplies the clip, ffmpeg does the arithmetic, there is no model and no
|
|
lighting to blame. The reversal is recorded there.
|
|
|
|
---
|
|
|
|
## 0. What already exists (verified 2026-11-05, bundled ffmpeg N-126342)
|
|
|
|
- `colorkey` / `chromakey` (colour, `similarity`, `blend`) and `despill` (`green`/`blue`, `mix`) are all
|
|
present in the shipped binary and slice-threaded.
|
|
- A probe of the bundled binary showed keyed pixels come out of the pipe with **alpha 0** and straight
|
|
(un-premultiplied) colour — `colorkey=…,format=bgra` → `63 175 0 0` vs `63 175 0 255` unkeyed. The
|
|
compositor already blends straight alpha, and media frames are not flagged `IsOpaque`, so no
|
|
compositor change is needed.
|
|
- Consequence: keying costs **zero app-side CPU** — it is one more term in the decode pipe's `-vf`,
|
|
which is also the only place that feeds **both** consumers (the shared preview bitmap and the
|
|
compositor's `LatestFrame`). A C# post-process would have to be wired twice.
|
|
|
|
## 1. Decisions (creator, 2026-11-05)
|
|
|
|
- **Suggest, never override.** Detection offers; the creator clicks. Opt-in setting for auto-apply.
|
|
- **Green and blue.** The keyed colour is the *detected* screen colour (real screens read 0x00B140,
|
|
not 0x00FF00), not a canonical green.
|
|
- **"Auto Green/Blue Screen"** is the setting name. No engineering words in the UI (never "chroma",
|
|
never "colour key"): the panel section is **Green/Blue Screen**, the sliders are **Amount**,
|
|
**Softness**, **Fringe**.
|
|
- **A clip is a clip** — alert clips get the same treatment from the same settings block.
|
|
- **High confidence only** may auto-apply; medium shows the offer; low says nothing.
|
|
- Settings are **per layer** (they belong to the layer in the UI). Sessions are keyed by
|
|
**path + settings signature**, so two layers with identical settings still share one decoder.
|
|
|
|
## 2. Slices (one Good Dog each)
|
|
|
|
| # | Slice | Test |
|
|
|---|---|---|
|
|
| A | ✅ media audio + shared clip pipe (TASK 21) | shipped `2670ac0` |
|
|
| B | ✅ Pure `Services/ChromaKeyDetector.cs`: sample a `VideoFrame` → green / blue / none + suggested colour + confidence | synthetic frames: green, blue, gradient, letterboxed, lawn (must say none) |
|
|
| C | ✅ Pure `Services/VideoFilterChain.cs`: options → `-vf` string (`scale → colorkey → despill → format=bgra`) | exact string assertions (encoder `FfmpegArgs` style) |
|
|
| D | ✅ Detect on add: one `-frames:v 1` probe (mirror `IFrameRateProbe`) → suggestion in the panel; no rendering change | add a green-screen file → suggestion appears with its colour |
|
|
| E | ✅ Apply: session key `path|key-settings`, filter chain in the pipe, level/mute untouched | keyed layer over art shows the art through keyed pixels **and** the preview bitmap carries alpha |
|
|
| F | ✅ "Auto Green/Blue Screen" opt-in (App Settings; high confidence only, and the panel says it was automatic), alert clips keyed through the same settings block, spill shipped as the **Fringe** switch | slider reaches the chain and the restart preserves loop state |
|
|
|
|
## 3. Landmines for whoever builds this
|
|
|
|
- **Two consumers.** The preview bitmap and `LatestFrame` fork inside `MediaVideoSourceManager`; keying
|
|
must happen upstream (in the pipe), or one path shows green.
|
|
- **Parameter change = decoder restart** (`-vf` is fixed at spawn). Debounce, and carry loop + refcount
|
|
across the restart instead of tearing the session down.
|
|
- **Session identity.** `ResolveOutputFrame` and the preview-bitmap adoption must derive the session key
|
|
from the same helper as the panel, or a keyed and an unkeyed layer on one file fight over one decoder.
|
|
- **Detection is a second reader of the file** — one frame, async, never on the UI thread; a failed
|
|
probe means "no suggestion", never a broken layer.
|
|
- **Letterboxing** (black bars) breaks naive border sampling — detect and exclude them first.
|
|
|
|
## 4. Known limits (honest)
|
|
|
|
- The offer is made **when a clip is added or changed** — not at every launch. The settings persist,
|
|
so a keyed layer comes back keyed; it just does not re-offer the probe.
|
|
- Auto-apply fires only at high confidence. Medium confidence always asks, low says nothing.
|
|
- The key is the layer's, but the decode is shared per **path + look**: two layers on one file with
|
|
different Amount/Softness values are two decoders (and therefore two audio streams, muted upstream
|
|
unless enabled) — rare, and the alternative is silently sharing the wrong picture.
|