TASK 21 Increment B: FFmpeg rawvideo video decoder -> VideoFrame
Adds the codec-agnostic decoder half of the media source. Spawns ffmpeg with -f rawvideo -pix_fmt bgra (reusing the already-shipped ffmpeg via IFfmpegLocator) and drains the raw BGRA stdout pipe into VideoFrames. - Services/RawVideoFrameReader.cs: pure rawvideo BGRA stream -> frames (partial reads kept across Feed; no ffmpeg needed to test) - Services/IDecodeProcess.cs + FfmpegDecodeProcess.cs: binary-stdout subprocess seam, mirror of the encoder's IEncoderProcess - Services/MediaVideoSource.cs: owns the decode, raises FrameReady/Completed - MediaVideoSourceTests: 3 pure reader + 1 integration (fake decode process through the real source loop, frames in order) Reference (external scan): ffmpeg rawvideo pipe decode is the canonical codec-agnostic frame feeds pattern (ffmpeg docs -f rawvideo; how OBS/media pipelines push frames to a compositor). Verified: 4/4 tests, build 0 warnings. Native-FPS pacing + resolver/compositor wiring are the next slice.
This commit is contained in:
+38
-36
@@ -2,54 +2,56 @@
|
||||
|
||||
## Branch / Commit State
|
||||
|
||||
**`main`**, working toward TASK 22 (audio sync offset). Prior milestone **TASK 31**
|
||||
(SceneGraph + baked-crust compositor) is fully done, committed (`670fe3a`), pushed
|
||||
(`origin/main` `3bf053a..670fe3a`), and verified by the user on native Windows
|
||||
PowerShell (only the 2 known pre-existing failures remained).
|
||||
**`main`**, working through TASK 21 (media source). Local commits ahead of
|
||||
`origin/main` (not yet pushed — user pushes at agreed checkpoints):
|
||||
- `f89b9f9` — docs: default-device audio assumption (README + startup log)
|
||||
- `e8ff4df` — TASK 22: global audio sync offset (positive delay at the mixer out)
|
||||
- `670fe3a` — TASK 31 (already pushed; listed for context)
|
||||
|
||||
Prior milestones pushed through `origin/main`: TASK 31 (`3bf053a..670fe3a`), plus
|
||||
the full refactor series (Commit F `3107f92`, Commit G `85893ea`).
|
||||
|
||||
## What's In Flight
|
||||
|
||||
**TASK 22 (audio sync offset) — implemented, uncommitted, awaiting commit + user
|
||||
verification on Windows.**
|
||||
**TASK 21 (media source) — Increment B (the decoder) implemented, uncommitted-or-pending-commit, awaiting next unit.**
|
||||
|
||||
Scope (Good-Dog single use-case; user chose positive-only slider at the mixer out):
|
||||
a global `AudioSyncOffsetMs` (0..500 ms, default 0) that delays the whole
|
||||
interleaved-stereo mix so audio lands on the video when it runs ahead — OBS's
|
||||
documented lip-sync fix. Positive-only: advancing audio needs a video-side delay
|
||||
(out of the audio layer's scope; tracked for v1.1).
|
||||
Increment A (the `Source` model: `MediaSourceType`, `SourceType.MediaSource`,
|
||||
`MediaPath`/`MediaIsLooping`/`MediaVolume`/`MediaPlaybackState` + LayoutStore
|
||||
persistence) shipped earlier. This session shipped **Increment B — the decoder**:
|
||||
|
||||
Files touched (this work unit):
|
||||
- `Services/Audio/AudioSyncDelay.cs` (new) — pure delay line, flushed on `Configure`.
|
||||
- `Services/Audio/AudioMixer.cs` — `Func<int> syncOffsetMs` seam + `_syncDelay`
|
||||
applied post-limiter in `FillAndMix`.
|
||||
- `Services/LayoutStore.Settings.cs` — `LoadAudioSyncOffsetMs`/`SaveAudioSyncOffsetMs`
|
||||
(`Audio.SyncOffsetMs`).
|
||||
- `ViewModels/MainViewModel.cs` — load offset at startup + save in `SaveLayoutNow`.
|
||||
- `ViewModels/MainViewModel.Audio.cs` — `AudioSyncOffsetMs` property (clamped, `ScheduleSave`).
|
||||
- `Controls/PreviewPane.xaml` — SYNC slider (0..500) + status dot on the mic bar.
|
||||
- `Helpers/IntToSyncBrushConverter.cs` (new) + registered in `Themes/Controls.xaml`.
|
||||
- `ytLive.Tests/AudioSyncDelayTests.cs` (new) — 3 tests.
|
||||
- `ai.md`, `TASKS.md`, `HANDOFF.md`.
|
||||
New files:
|
||||
- `Services/RawVideoFrameReader.cs` — pure: parses a rawvideo BGRA stream into
|
||||
`VideoFrame`s (exactly W×H×4 per frame; partial reads kept across `Feed`).
|
||||
- `Services/IDecodeProcess.cs` + `Services/FfmpegDecodeProcess.cs` — binary-stdout
|
||||
subprocess seam (mirror of `IEncoderProcess`).
|
||||
- `Services/MediaVideoSource.cs` — spawns ffmpeg `-f rawvideo -pix_fmt bgra -an`,
|
||||
drains the pipe, raises `FrameReady` per frame; `Completed` on EOF.
|
||||
- `ytLive.Tests/MediaVideoSourceTests.cs` — 3 pure reader tests + 1 integration
|
||||
(fake `IDecodeProcess`+`IFfmpegLocator` through the real source loop, frames in
|
||||
order). **Verified: 4/4 pass, build 0 warnings.**
|
||||
|
||||
**Verified:** `ytLive.csproj` and `ytLive.Tests.csproj` build with 0 warnings;
|
||||
`AudioSyncDelayTests` (3/3) pass via vstest filter. The full audio pipeline /
|
||||
RealAppHost suites hang headless (pre-existing) — must be verified on native
|
||||
Windows PowerShell by the user.
|
||||
Still open (next slices): native-FPS pacing (ffprobe probe), loop control, volume,
|
||||
the `ResolveOutputFrame` resolver case + wire into `SceneCompositor`, the file-picker
|
||||
UI, and the remaining tests.
|
||||
|
||||
**Landmine:** a stale `testhost` can lock `ytLive.Tests`'s `ytLive.dll` and break
|
||||
`dotnet build` of the test project — kill it first (`cmd.exe /c "taskkill /PID N /F"`)
|
||||
if the build hits MSB3027.
|
||||
on MSB3027. Full-suite vstest hangs headless (RealAppHost/WASAPI) — only filtered
|
||||
pure tests run in WSL; user verifies the GUI suites on native Windows PowerShell
|
||||
(which may also hang if WASAPI startup blocks — pre-existing, not this change).
|
||||
|
||||
## Next
|
||||
|
||||
1. User runs `scripts/verify.sh` (or native Windows vstest) to confirm the full
|
||||
suite — expect only the 2 known pre-existing failures.
|
||||
2. On confirmation, run `scripts/scope-check.sh` with the declared file list,
|
||||
commit as one work unit, await the user's push go-ahead (sub-milestone).
|
||||
1. Commit Increment B (decoder) as one work unit (scope-check + docs already
|
||||
staged in ai.md/TASKS.md/HANDOFF.md).
|
||||
2. Next slice: native-FPS pacing + wire media into `ResolveOutputFrame` and the
|
||||
compositor, per TASK 21 steps 6–7.
|
||||
3. Await user push checkpoint for the TASK 22/21 commits.
|
||||
|
||||
## The directive (2026-08-31, user)
|
||||
|
||||
Rewrite the project into functional components to aid AI retrieval — the
|
||||
`Services/ChatOverlayLayer.cs` style (owner-state extraction), not line-count
|
||||
chasing. TASK 31's `SceneGraph.cs` continued that pattern (an owner object owning
|
||||
the scene collection + mutation surface).
|
||||
Rewrite the project into functional components to aid AI retrieval —
|
||||
`Services/ChatOverlayLayer.cs` / `SceneGraph.cs` style (owner-state extraction),
|
||||
not line-count chasing. Audio assumption is now an explicit contract (README
|
||||
"Audio Assumption"): the app uses system defaults, never fights Windows device
|
||||
locking, and does not debug user audio issues — OS's problem, not ours.
|
||||
|
||||
Reference in New Issue
Block a user