TASK 21 Increment B: FFmpeg rawvideo video decoder -> VideoFrame

Adds the codec-agnostic decoder half of the media source. Spawns ffmpeg
with -f rawvideo -pix_fmt bgra (reusing the already-shipped ffmpeg via
IFfmpegLocator) and drains the raw BGRA stdout pipe into VideoFrames.

- Services/RawVideoFrameReader.cs: pure rawvideo BGRA stream -> frames
  (partial reads kept across Feed; no ffmpeg needed to test)
- Services/IDecodeProcess.cs + FfmpegDecodeProcess.cs: binary-stdout
  subprocess seam, mirror of the encoder's IEncoderProcess
- Services/MediaVideoSource.cs: owns the decode, raises FrameReady/Completed
- MediaVideoSourceTests: 3 pure reader + 1 integration (fake decode
  process through the real source loop, frames in order)

Reference (external scan): ffmpeg rawvideo pipe decode is the canonical
codec-agnostic frame feeds pattern (ffmpeg docs -f rawvideo; how OBS/media
pipelines push frames to a compositor). Verified: 4/4 tests, build 0 warnings.
Native-FPS pacing + resolver/compositor wiring are the next slice.
This commit is contained in:
2026-08-31 18:57:29 -07:00
parent f89b9f9ffa
commit 8f490102ee
8 changed files with 383 additions and 40 deletions
+38 -36
View File
@@ -2,54 +2,56 @@
## Branch / Commit State
**`main`**, working toward TASK 22 (audio sync offset). Prior milestone **TASK 31**
(SceneGraph + baked-crust compositor) is fully done, committed (`670fe3a`), pushed
(`origin/main` `3bf053a..670fe3a`), and verified by the user on native Windows
PowerShell (only the 2 known pre-existing failures remained).
**`main`**, working through TASK 21 (media source). Local commits ahead of
`origin/main` (not yet pushed — user pushes at agreed checkpoints):
- `f89b9f9` — docs: default-device audio assumption (README + startup log)
- `e8ff4df` — TASK 22: global audio sync offset (positive delay at the mixer out)
- `670fe3a` — TASK 31 (already pushed; listed for context)
Prior milestones pushed through `origin/main`: TASK 31 (`3bf053a..670fe3a`), plus
the full refactor series (Commit F `3107f92`, Commit G `85893ea`).
## What's In Flight
**TASK 22 (audio sync offset) — implemented, uncommitted, awaiting commit + user
verification on Windows.**
**TASK 21 (media source) — Increment B (the decoder) implemented, uncommitted-or-pending-commit, awaiting next unit.**
Scope (Good-Dog single use-case; user chose positive-only slider at the mixer out):
a global `AudioSyncOffsetMs` (0..500 ms, default 0) that delays the whole
interleaved-stereo mix so audio lands on the video when it runs ahead — OBS's
documented lip-sync fix. Positive-only: advancing audio needs a video-side delay
(out of the audio layer's scope; tracked for v1.1).
Increment A (the `Source` model: `MediaSourceType`, `SourceType.MediaSource`,
`MediaPath`/`MediaIsLooping`/`MediaVolume`/`MediaPlaybackState` + LayoutStore
persistence) shipped earlier. This session shipped **Increment B — the decoder**:
Files touched (this work unit):
- `Services/Audio/AudioSyncDelay.cs` (new) — pure delay line, flushed on `Configure`.
- `Services/Audio/AudioMixer.cs` — `Func<int> syncOffsetMs` seam + `_syncDelay`
applied post-limiter in `FillAndMix`.
- `Services/LayoutStore.Settings.cs` — `LoadAudioSyncOffsetMs`/`SaveAudioSyncOffsetMs`
(`Audio.SyncOffsetMs`).
- `ViewModels/MainViewModel.cs` — load offset at startup + save in `SaveLayoutNow`.
- `ViewModels/MainViewModel.Audio.cs` — `AudioSyncOffsetMs` property (clamped, `ScheduleSave`).
- `Controls/PreviewPane.xaml` — SYNC slider (0..500) + status dot on the mic bar.
- `Helpers/IntToSyncBrushConverter.cs` (new) + registered in `Themes/Controls.xaml`.
- `ytLive.Tests/AudioSyncDelayTests.cs` (new) — 3 tests.
- `ai.md`, `TASKS.md`, `HANDOFF.md`.
New files:
- `Services/RawVideoFrameReader.cs` — pure: parses a rawvideo BGRA stream into
`VideoFrame`s (exactly W×H×4 per frame; partial reads kept across `Feed`).
- `Services/IDecodeProcess.cs` + `Services/FfmpegDecodeProcess.cs` — binary-stdout
subprocess seam (mirror of `IEncoderProcess`).
- `Services/MediaVideoSource.cs` — spawns ffmpeg `-f rawvideo -pix_fmt bgra -an`,
drains the pipe, raises `FrameReady` per frame; `Completed` on EOF.
- `ytLive.Tests/MediaVideoSourceTests.cs` — 3 pure reader tests + 1 integration
(fake `IDecodeProcess`+`IFfmpegLocator` through the real source loop, frames in
order). **Verified: 4/4 pass, build 0 warnings.**
**Verified:** `ytLive.csproj` and `ytLive.Tests.csproj` build with 0 warnings;
`AudioSyncDelayTests` (3/3) pass via vstest filter. The full audio pipeline /
RealAppHost suites hang headless (pre-existing) — must be verified on native
Windows PowerShell by the user.
Still open (next slices): native-FPS pacing (ffprobe probe), loop control, volume,
the `ResolveOutputFrame` resolver case + wire into `SceneCompositor`, the file-picker
UI, and the remaining tests.
**Landmine:** a stale `testhost` can lock `ytLive.Tests`'s `ytLive.dll` and break
`dotnet build` of the test project — kill it first (`cmd.exe /c "taskkill /PID N /F"`)
if the build hits MSB3027.
on MSB3027. Full-suite vstest hangs headless (RealAppHost/WASAPI) — only filtered
pure tests run in WSL; user verifies the GUI suites on native Windows PowerShell
(which may also hang if WASAPI startup blocks — pre-existing, not this change).
## Next
1. User runs `scripts/verify.sh` (or native Windows vstest) to confirm the full
suite — expect only the 2 known pre-existing failures.
2. On confirmation, run `scripts/scope-check.sh` with the declared file list,
commit as one work unit, await the user's push go-ahead (sub-milestone).
1. Commit Increment B (decoder) as one work unit (scope-check + docs already
staged in ai.md/TASKS.md/HANDOFF.md).
2. Next slice: native-FPS pacing + wire media into `ResolveOutputFrame` and the
compositor, per TASK 21 steps 6–7.
3. Await user push checkpoint for the TASK 22/21 commits.
## The directive (2026-08-31, user)
Rewrite the project into functional components to aid AI retrieval — the
`Services/ChatOverlayLayer.cs` style (owner-state extraction), not line-count
chasing. TASK 31's `SceneGraph.cs` continued that pattern (an owner object owning
the scene collection + mutation surface).
Rewrite the project into functional components to aid AI retrieval —
`Services/ChatOverlayLayer.cs` / `SceneGraph.cs` style (owner-state extraction),
not line-count chasing. Audio assumption is now an explicit contract (README
"Audio Assumption"): the app uses system defaults, never fights Windows device
locking, and does not debug user audio issues — OS's problem, not ours.