TASK 9 audio milestone: real stream audio + voice filters + auto-duck + free TRAX music — the mixer now feeds the encoder real audio (ffmpeg reads a Windows named pipe, -f f32le -ar 48000 -ac 2 -i \.\pipe\ytllive_audio, replacing anullsrc silence; explicit -map 0:v -map 1:a via EncoderOptions.AudioPipeName), with honest gains reaching the stream (MicVolume scales mic, GameAudioVolume + mute scale loopback, Func<double> seams on AudioMixer), an always-on pure-C# voice chain on the mic BEFORE the meter and mix (LowShelfFilter 120Hz +4dB → HighShelfFilter 8kHz +3dB → NoiseGate 0.005/hysteresis 0.5 → Compressor 0.5 4:1, all TDF2), AutoDucker (mic RMS>0.02 → loopback ×0.25, attack 0.05/release 0.005), and TRAX free background music (MusicPlayer = MediaFoundationReader → VolumeWaveProvider16 at fixed 0.20 → WaveOutEvent via the new sibling NAudio.WinMM 2.2.1 package; plays to the default device so the existing loopback carries it, ducked with game; footer TRAX button with red/yellow/green status dot, left-click toggles/picks, right-click opens the OpenFileDialog picker, tooltip shows the track name; persisted via schema v9 single-row Music). Build-time deviations from the plan (recorded in ai.md + TASKS.md): WasapiCapture in NAudio 2.2.1 exposes no overridable GetDefaultMixFormat so sources run device mix format and the mixer's TinyResampler normalizes any rate to 48kHz (resampler IS the design); FfmpegEncoder.cs untouched — MainViewModel.StopStream calls _audioMixer.StopLive() (pipe EOF) before the frame pump stops; sound-bar relabelled 'Desktop Audio', IsGameAudioBarVisible = game sound OR music playing. Tests: new AudioPipelineTests (DSP/ring-buffer/ducker/resampler units + the ONE integration test Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe reading real pipe bytes; ring-buffer overwrite bug found + fixed), FfmpegEncoderTests/LayoutStorePersistenceTests updated — 196 tests passing, 0 warnings

This commit is contained in:
2026-08-14 18:09:26 -07:00
parent 3d92bd0225
commit 6d71acef8c
22 changed files with 1605 additions and 123 deletions
+59 -33
View File
@@ -91,7 +91,7 @@ C# / WPF (.NET 8) following MVVM:
|------|------|
| `Models/` | Plain data types — Scene, Source (incl. `ClipShape`, `IsMirrored`, `VideoImageSource`), QualityOption, StreamConfig, StreamHealth, YouTubeChannel, ChatMessage, **Socials (`SocialService` enum + `SocialEntry`/`SocialsConfig` + `SocialServiceIcons`) — the social bar** |
| `ViewModels/` | MainViewModel — exposes collections + commands for the UI; GoLiveViewModel, ReuseImageViewModel, CameraPickerViewModel, **SocialsDialogViewModel** |
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **compositor: `SceneCompositor` + `CompositorOptions` + pure `StretchMath` + `StaticPixelCache` (see "Scene compositor")**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat` + game bar: `IGameAudioDetector`/`GameAudioHysteresis`/`GameAudioDetector` (see "Live audio capture")**, **encoder: `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/`FfmpegEncoderProcess` + `IFfmpegLocator`/`FfmpegLocator` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` + the `FramePump` frame producer (see "Live encoder" + "Live frame pipeline")** |
| `Services/` | YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), **SocialValidator (`ISocialValidator` seam + `HttpSocialValidator` default)**, **webcam: `VideoFrame` seam + `CameraDeviceInfo`/`ICameraEnumerator`/`ICameraFrameSource` interfaces + `MediaCaptureCameraEnumerator`/`MediaCaptureFrameSource` (WinRT) + `CameraManager`**, **screen capture: `IFullScreenDetector`/`Win32FullScreenDetector` + `IScreenCaptureSource`/`ScreenCaptureFrameSource` (WinRT GraphicsCapture) + `ScreenCaptureManager` + `ScreenCaptureSourceFactory` + `Direct3D11Helper`/`CaptureInterop` (COM bridges)**, **compositor: `SceneCompositor` + `CompositorOptions` + pure `StretchMath` + `StaticPixelCache` (see "Scene compositor")**, **audio: `IAudioSource` seam + `WasapiLoopbackAudioSource`/`WasapiMicAudioSource` (NAudio WASAPI) + `AudioMixer` + pure `AudioLevelMeter`/`WaveToFloat`/`VoiceFilterChain`/`LowShelfFilter`/`HighShelfFilter`/`NoiseGate`/`Compressor`/`AutoDucker`/`AudioRingBuffer`/`TinyResampler` + `MusicPlayer` + `IAudioPipeWriter`/`NamedPipeAudioWriter` + game bar: `IGameAudioDetector`/`GameAudioHysteresis`/`GameAudioDetector` (see "Live audio capture")**, **encoder: `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/`FfmpegEncoderProcess` + `IFfmpegLocator`/`FfmpegLocator` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` + the `FramePump` frame producer (see "Live encoder" + "Live frame pipeline")** |
| `Helpers/` | ViewModelBase (INotifyPropertyChanged), RelayCommand, ImageCache, AppLog (file logger), FocusPreservingListBox, OAuthCredentials, **TokenStore (DPAPI session persistence)**, visibility converters |
| `Themes/` | `Controls.xaml` — the single dark-theme source, merged once in `App.xaml` (see `Themes/index.md`) |
| `MainWindow.xaml` | Dark theme; layout: top bar (controls), center (preview), left (scenes/sources), right (chat), bottom (health) |
@@ -353,8 +353,10 @@ integration test fakes the whole subprocess (probe + encoder) with a Channel-bac
`Complete()` is EOF (`null`), never a `ChannelClosedException`.
**Decisions (locked):** args are pure (`FfmpegArgs.Build`, no string building in the encoder):
`-re -f rawvideo -pix_fmt bgra -video_size WxH -framerate FPS -i pipe:0` + a **silent placeholder
`-f lavfi -i anullsrc`** track (WASAPI capture, ship step 4, replaces it) + `-c:v <enc> -b:v K
`-re -f rawvideo -pix_fmt bgra -video_size WxH -framerate FPS -i pipe:0` + a **real audio input — the
mixer writes IEEE-float stereo to a Windows named pipe** (`-f f32le -ar 48000 -ac 2
-i \\.\pipe\ytllive_audio`, name via `EncoderOptions.AudioPipeName`; replaced the old `-f lavfi -i
anullsrc` silence in the TASK 9 audio milestone) + explicit `-map 0:v -map 1:a` + `-c:v <enc> -b:v K
-maxrate K -bufsize 2K` + **`-g fps×4 -keyint_min fps×4 -sc_threshold 0 -bf 0 -pix_fmt yuv420p`**
(≤4s keyframes, closed GOP, H.264 compliance) + `-c:a aac -ar 48000 -ac 2 -f flv <rtmpUrl>`.
**Encoder choice is probed from the binary's `-encoders` listing** (`FfmpegEncoderPicker`, pure):
@@ -363,7 +365,7 @@ forces one and skips the probe. `EncoderOptions` also carries W×H/FPS/bitrate f
`GopSize` = FPS×4. Constructed by `MainViewModel` since ship step 5 (`new FfmpegEncoder(new FfmpegLocator())`),
driven by the `FramePump` below.
### Live audio capture (TASK 4 ship step 4 — shipped 2026-08-12, game audio bar follow-up 2026-08-13, plans in TASKS.md)
### Live audio capture (TASK 4 ship step 4 — shipped 2026-08-12; game audio bar 2026-08-13; **TASK 9 audio milestone: real stream audio + filters + duck + TRAX, shipped 2026-08-14**, plan in TASKS.md)
**Capture runs for the app's lifetime and is KISS by rule**: desktop/game audio is automatic (WASAPI
loopback), the mic is the creator's only audio control (meter/mute/volume already shipped). The whole
@@ -371,28 +373,52 @@ layer sits behind an **`IAudioSource` seam** (`Services/Audio/`: `Start`/`Stop`/
`Failed`, IDisposable) so the app never touches NAudio directly and tests inject hermetic fakes (no
devices, no timers).
- **Sources (NAudio `NAudio.Wasapi` 2.2.1, MIT — item 9 in `THIRD-PARTY-NOTICES.txt`):**
`WasapiLoopbackAudioSource` = `WasapiLoopbackCapture` on the default render device;
`WasapiMicAudioSource` = `WasapiCapture` with the device resolved by `FriendlyName` matching
`MicSourceName` (the app only persists the **DisplayName**), falling back to the default capture
endpoint. Mic device resolution re-reads the name provider `Func<string?>` at each `Start`, so a mic
picked mid-session takes effect **immediately** (the mixer restarts the mic on pick). Both sources
raise `Started` once their capture loop actually begins — the mixer turns that into `MicConnected`.
- **Sources (NAudio 2.2.1 — `NAudio.Wasapi` + `NAudio.WinMM` (for `WaveOutEvent`), MIT — item 9 in
`THIRD-PARTY-NOTICES.txt`):** `WasapiLoopbackAudioSource` = `WasapiLoopbackCapture` on the default
render device; `WasapiMicAudioSource` = `WasapiCapture` with the device resolved by `FriendlyName`
matching `MicSourceName` (the app only persists the **DisplayName**), falling back to the default
capture endpoint. Both run at the **device's own mix format** — NAudio 2.2.1 exposes no overridable
`GetDefaultMixFormat` on `WasapiCapture`, so there is no forced 48 kHz path; the mixer's
`TinyResampler` normalizes any rate/channel layout to 48 kHz stereo (this replaced the earlier
"force IEEE-float 48k" plan — the resampler IS the design). Mic device resolution re-reads the name
provider `Func<string?>` at each `Start`, so a mic picked mid-session takes effect **immediately**
(the mixer restarts the mic on pick). Both sources raise `Started` once their capture loop actually
begins — the mixer turns that into `MicConnected`.
- **`AudioMixer`** owns both sources; **`StartMicCaptureAsync` starts the mixer once at startup** and
`Shutdown` disposes it — NOT go-live — so the meters preview live (`BeginGoLive`/`StopStream`
no longer touch the mixer). Mic samples feed a pure **`AudioLevelMeter`** (RMS with 0.2 exponential
smoothing) → `MicLevelChanged` → marshalled to the UI thread → `AudioLevel`; loopback samples feed a
second meter → `LoopbackLevelChanged` → the game bar's `GameAudioLevel`. The raw linear level is
mapped to the meter's display scale by `AudioLevelMeter.ToDisplay` (60..0 dBFS → 0..1 with **+10 dB
input amplification**): real speech/game RMS is ~0.01..0.1 linear, which would leave a flat scale
looking dead; the amplification (added 2026-08-14) pushes real speech peaks (~0.2 RMS, 14 dBFS) to
~0.93 at maxed volume so the meter uses the whole bar proportionately instead of hovering at the red
edge. **Mic connection state
surfaces as events:** `MicConnected` (source `Started`), `MicFailed` (source `Failed`), and
`RestartMic()` re-resolves + restarts just the mic (loopback keeps running). Failures log via
`AppLog`; a mic failure zeroes the meter, a loopback failure never kills the mic. Meter `Push` is
unconditional (a `?.` on the event would skip the argument — and the meter update — when nothing is
subscribed yet).
`Shutdown` disposes it — NOT go-live — so the meters preview live. Mic samples: downmix to mono →
`TinyResampler` → **`VoiceFilterChain` per sample** (bass `LowShelfFilter` 120 Hz +4 dB → treble
`HighShelfFilter` 8 kHz +3 dB → `NoiseGate` (threshold 0.005, hysteresis 0.5, attack 0.5, release
0.0005) → `Compressor` (threshold 0.5, ratio 4:1); all pure TDF2 DSP) → ring buffer → the
**post-filter** `AudioLevelMeter` (RMS, 0.2 exponential smoothing) → `MicLevelChanged` →
marshalled to the UI thread → `AudioLevel`. Loopback samples: resample → ring buffer → second meter
→ `LoopbackLevelChanged` → the game bar's `GameAudioLevel`. The raw linear level maps to the meter's
display scale by `AudioLevelMeter.ToDisplay` (60..0 dBFS → 0..1 with **+10 dB input amplification**).
**Mic connection state surfaces as events:** `MicConnected` (source `Started`), `MicFailed` (source
`Failed`), and `RestartMic()` re-resolves + restarts just the mic (loopback keeps running). Failures
log via `AppLog`; a mic failure zeroes the meter, a loopback failure never kills the mic. Meter `Push`
is unconditional (a `?.` on the event would skip the meter update when nothing is subscribed yet).
- **Go-live audio (TASK 9):** `BeginGoLive` → `StartLive(pipeName)` (idempotent) creates the named pipe
server (`NamedPipeAudioWriter`, behind the `IAudioPipeWriter` seam) and a live loop task; every
`mixInterval` (default 10 ms) `FillAndMix` reads a chunk from each ring buffer (silence-fills
underruns), computes the post-filter mic RMS → `AutoDucker` (threshold 0.02, duck ×0.25 / 12 dB,
attack 0.05, release 0.005, always on), applies the **honest gains** via `Func<double>` seams
(`micGain = MicVolume` so mute = 0; `loopbackGain = GameMuted ? 0 : GameAudioVolume`, × duck), mixes
mono mic + stereo loopback into interleaved stereo (no clamp — gains are user-owned), and writes the
floats to the pipe. `StopStream` → `StopLive()` (closes the pipe → ffmpeg audio EOF) **BEFORE**
stopping the frame pump (video EOF) — the reverse order stalls on pipe backpressure. `FfmpegEncoder`
itself is untouched; the VM owns the pipe lifecycle.
- **TRAX — free background music (TASK 9):** `MusicPlayer` = NAudio `MediaFoundationReader`
(mp3/wav/m4a) → `VolumeWaveProvider16` at the hardcoded **0.20** (no slider) → `WaveOutEvent` on the
default device, looping on natural end. It plays to the **desktop**, so the existing loopback captures
it: the creator hears it in headphones, the sound-bar meter bounces, and the stream carries it through
the loopback channel — **ducked with the game when the mic is hot**. No third mixer input. Footer
**TRAX** button beside MIC: status dot (**red** no track / **yellow** loaded stopped / **green**
playing) + "TRAX"; **left-click** toggles play/pause (opens the in-app picker via `OpenFileDialog`
when no track is loaded); **right-click** always opens the picker (`TraxButton_PreviewMouseRightButtonUp`,
`e.Handled = true`, code-behind pattern); tooltip shows the loaded/playing track name or "No track —
right-click to choose background music". The picked track persists via **schema v9** single-row
`Music` (`TrackPath`/`IsEnabled`) in `LayoutStore`. Known wrinkle (out of scope, future feature):
YouTube mutes VODs carrying copyrighted music — "music on live, off VOD" is tabled.
- **Mic status dot (`Models/MicStatus.cs`)** on the footer's MIC button: green = `MicConnected`,
yellow = `MicFailed` (in use/unplugged), red = no mic device at startup (the mixer is never started,
so loopback and the game bar can't run either — no capture devices at all).
@@ -401,17 +427,17 @@ devices, no timers).
~500ms of fullscreen + sound, HIDE after ~1s away from fullscreen, **silence never hides an active
bar**), and the default `GameAudioDetector` composing `IFullScreenDetector` + the live loopback level
(floor 0.5%). WPF-free — the VM owns a 250ms `DispatcherTimer` that polls it and flips
`IsGameAudioBarVisible`. The bar is **overlaid at the bottom of the preview window** (bottom-center,
dark translucent chip, a mirror of the mic bar: meter + mute + volume slider). It's monitoring UI
only — it lives in the XAML preview (`MainWindow.xaml`, the PreviewGrid) and never reaches the live
output (the `SceneCompositor` doesn't know about it).
`IsGameAudioBarVisible` (`_gameAudioBarActive || IsMusicPlaying` — music bounces the bar even with no
game). The bar is **overlaid at the bottom of the preview window** (bottom-center, dark translucent
chip, a mirror of the mic bar: meter + mute + volume slider). It's monitoring UI only — it lives in
the XAML preview (`MainWindow.xaml`, the PreviewGrid) and never reaches the live output. Relabelled
**"Desktop Audio"** (TASK 9) since TRAX rides the same channel.
- **`WaveToFloat`** (pure, shared): WASAPI mix formats → interleaved float — IEEE float 32-bit direct,
PCM 16-bit normalized to -1..1, `WaveFormatExtensible` with the IEEE-float subformat GUID
(`NAudio.Dmo.AudioMediaSubtypes.MEDIASUBTYPE_IEEE_FLOAT`), trailing partial samples ignored.
- Build **0 warnings**; **169 passing** (mixer/hysteresis/game-detector/meter-scale unit tests).
**Deferred:** the loopback→encoder AAC mix wiring (a later step — the encoder construction + frame
pipeline shipped in ship step 5).
- Build **0 warnings**; **196 passing** (DSP/ring-buffer/ducker/resampler/pipe + mixer/hysteresis/
game-detector/meter-scale unit tests + the ONE integration test
`AudioPipelineTests.Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe`).
### Live frame pipeline (TASK 4 ship step 5 — shipped 2026-08-12, health stats 2026-08-13, plan in TASKS.md)