Two follow-ups from the creator's lip-sync observation.
1. Audio still slept a fixed 50 ms per chunk — the same "sleep on top of the work" fault fixed for
video an hour ago, left in place. Every chunk's read cost was added, so a clip's audio fell
progressively behind its picture: small per chunk, unbounded over a long clip. It now uses the
same deadline pacer, so audio and video advance on the same wall clock.
2. The preview pane is NOT the compositor. "Preview looks great" proves the decode and the preview
bitmap; the recording and stream are blitted by SceneCompositor. `MediaCompositeOutputTests`
renders a keyed video layer over a background through the real compositor and asserts the two
things that matter: keyed (alpha 0) pixels let the background through, and the subject half
covers it. So "green screen removed" is now proven in the output path, not just the pane.
Also: a round-clipped video layer had no preview trigger (only webcam+Round existed), so it would
render in the output and be invisible in the pane. Same unit-circle path now covers it.
Tests: 16 scoped (pacer, real-clip rate + keying, composite output, decode pacing, alert pacing)
green. FULL SUITE NOT RUN — the machine is in use: two runs under load aborted in the WPF test host
(HWND subclass crash) and one hit the 15-minute cap. Re-run scripts/verify.sh in a quiet window.
Reported as "the default video and the green screen video are playing at a hideously slow
rate… I thought the playback timing issues were resolved?!" They were not: two independent
faults, both in the shared clip pipe.
1. QUADRATIC COPY IN THE FRAME READER. `RawVideoFrameReader` re-concatenated the whole pending
buffer on every `Feed`, and the pipe feeds it 64 KB at a time. A 1080p BGRA frame is 8 MB, so
one frame cost ~127 copies of a growing buffer — roughly half a gigabyte of memcpy per frame.
Measured on a 30 fps clip: 6 frames in 3 seconds (2 fps). The reader now fills one preallocated
frame buffer and hands that array to the VideoFrame, so a frame is copied exactly once. Pipe
reads are 256 KB. Same clip: 29 fps.
2. PACING SLEPT ON TOP OF THE WORK. A fixed 1/fps sleep after each frame adds the frame's own
cost, so 20 ms of work on a 30 fps clip yields 1/(33+20) ≈ 19 fps — measured 20.9. Pacing is
now deadline-based (`Helpers/FramePacer.cs`): wait only what is left of the frame's interval,
resync when a frame overruns instead of bursting, never sleep longer than one interval.
The alert clip shares this pipe, so it gained both fixes.
Tests: `FramePacerTests` (subtract work, no drift, late → resync, zero duration, reset);
`ChromaKeyRealClipTests.RealClip_PlaysAtItsNativeRate` (throughput floor of 15 fps on the real
30 fps clip — the old code sat at 2); the two pacing assertions now check "at most one interval"
rather than an exact sleep, which is what deadline pacing means. Full suite: 574 passed,
0 failed, clean build.
Reported 2026-11-05: "when I click Loop, the video doesn't play in the preview pane". Two
causes, both real:
- `VideoClipPipe.StartAsync` bailed out once it had run once (`_run != null`), so a clip that
reached EOF could never be started again — it held its last frame forever.
- Loop is only consulted at end-of-file by a LIVE decoder, so flipping it on a finished clip
did nothing at all.
Now: the pipe reports running state and is restartable after a natural end; the manager marks a
session finished (keeping its last frame) and exposes `IsRunning`/`RestartAsync`; the layer shows
`IsMediaRunning`/`IsMediaFinished`; flipping Loop on a finished clip restarts it.
UI: `▶ Play` sits to the right of the **Video File** label and is enabled only while Loop is off
(creator ruling: with Loop on the clip already runs continuously, so there is nothing to start).
The panel says "Clip finished — holding the last frame." so a frozen picture explains itself.
Also: verify.sh no longer runs physical-input tests by default. That test moves the real cursor
(SetCursorPos + mouse_event); running it mid-gate hijacks the machine someone is using, which is
exactly how it felt. Opt in with PHYSICAL_TESTS=1 when the desktop is idle; the header documents
the scoped-run command for working hours.
Test: `MediaPlaybackRestartTests` (pipe restart, finished state, Play again, Loop-on-finished).
8 scoped tests green. FULL SUITE NOT RUN — the machine was in use; run scripts/verify.sh in a
quiet window before the next milestone.
TASK 64 slices D and E: the feature is usable end to end.
- Add a video file → one frame is probed (new `IChromaProbe`/`FfmpegChromaProbe`, one ffmpeg
pass at 160x90, off the UI thread) and the layer gets an offer: "Looks like a green screen —
Remove it", with the measured colour as a swatch.
- Accepting it turns the offer into settings (Amount / Softness / Fringe) that reach ffmpeg as
`colorkey` + `despill` + `format=bgra`, so the key is computed in the pipe and both consumers
(the shared preview bitmap and the compositor's frame) carry the same alpha.
- Media sessions are now keyed by **path + look** (`MainViewModel.MediaSessionKey`), so two
layers with the same settings still share one decoder, and changing the look releases the old
session and starts the new one. Removal, load reconciliation, loop and audio settings all move
to the same key.
- Settings persist on the `Source` row (six ALTER TABLE guards + SELECT/INSERT), so a keyed layer
comes back keyed.
- Panel section is plain creator words: Green/Blue Screen, Amount, Softness, Fringe.
Test: `ChromaKeyApplyTests` — the pipe asks ffmpeg to key (and does not when off); a green-screen
clip is offered, accepting it starts a keyed session, the layer's resolver then reads that session
rather than the old one; the setting survives a save.
Docs: TASK 64 D/E ticked with slice F (auto-apply setting + alert clips) left open; frame-pipeline
records the decode-option design; HANDOFF rewritten. verify.sh: 544 passed, 0 failed, 0 warnings.
A video file was picture-only: the decode pipe passed `-an`, so a starting-soon
or BRB clip with music played silent everywhere. Adds sound to every clip and
collapses the two decoders into one.
- `Services/VideoClipPipe.cs`: the shared pipe behind `MediaVideoSource` (TASK 21)
and `AlertClipDecoder` (TASK 47) — video (`rawvideo bgra`, scaled) + audio
(`f32le -ac 2 -ar 48000`, 50 ms chunks, byte-carry across reads), both paced to
real time, loop restarts both so picture and sound stay together. Alert
behaviour is unchanged.
- `MediaVideoSourceManager` applies the layer's Level/Mute per session and raises
`AudioAvailable`; `MainViewModel` feeds it to the new
`AudioMixer.EnqueueDesktopMediaAudio`, which joins the DESKTOP/GAME ring so the
desktop fader and the mic auto-ducker apply exactly as they do for game audio
and TRAX (creator ruling 2026-11-05: audio joins the group).
- Per-layer `MediaMuted` + existing `MediaVolume` surfaced as Mute/Level in the
properties panel; `MediaMuted` persisted (ALTER TABLE guard + SELECT/INSERT).
- Level/mute are per SESSION because one decoder is shared per file path — the
last layer touched wins; one decoder per layer would play the file twice.
Test: `MediaSourceAudioTests` — the pipe emits 50 ms chunks at 48k stereo, the
manager scales by level and drops when muted, and a media layer's sound moves the
desktop meter. Alert/decode suites unchanged and green.
Docs: task-21 closed on item 10, the "silent soundtrack" facts corrected in
frame-pipeline/output/overlays, HANDOFF rewritten. verify.sh green (529 passed;
the two documented pre-existing failures only).
gramps