docs: restructure TASKS.md into a catalog — one file per task in TASKS/
TASKS.md is now the index (status table, open items, research pointer). 33 files: 32 task files + 1 research facts file. The full take-saga narrative and all design decisions are preserved verbatim; the catalog makes the queue readable without opening every task body. Schema and AGENTS.md updated to reflect the new layout.
This commit is contained in:
@@ -0,0 +1,339 @@
|
||||
# TASK 4 — RTMP Ingest to YouTube
|
||||
|
||||
> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here.
|
||||
|
||||
|
||||
**Goal:** Push encoded video to YouTube's RTMP ingest.
|
||||
|
||||
### Status: ✅ Done — all 7 items shipped
|
||||
|
||||
1. ✅ **Ship step 1 — the output compositor SHIPPED** (2026-08-10)
|
||||
2. ✅ **Ship step 2 — the FFmpeg locator SHIPPED** (2026-08-10)
|
||||
3. ✅ **Encoder + RTMP push SHIPPED** (2026-08-12) — the FFmpeg subprocess: raw BGRA frames via stdin, stderr health parsing, FLV mux + push to the ingestion URL (see the ship step 3 plan below)
|
||||
4. ✅ **WASAPI audio capture SHIPPED** (2026-08-12) — NAudio loopback (desktop/game) + the picked mic feeding `AudioLevel`, so the realtime meter comes alive (see the ship step 4 plan below)
|
||||
5. ✅ **Frame-pipeline wiring SHIPPED** (2026-08-12) — `CameraManager`/`ScreenCaptureManager` → compositor resolver → encoder, driven by a paced `FramePump` (see the ship step 5 plan below)
|
||||
6. ✅ **Health stats SHIPPED** (2026-08-13) — `FramePump.HealthUpdated` (encoder's parsed bitrate/FPS/dropped/duration, already forwarded from `FfmpegEncoder.OnStderrLine`) now lands in the bottom bar: `MainViewModel.OnFramePumpHealthUpdated` marshals to the UI thread (the stderr loop raises on a background thread) and copies into `CurrentHealth` (the bottom bar's existing binding); `ResetHealth` zeroes dropped/duration on go-live and on End so stats never linger from a previous session (bitrate/FPS stay on the tier's targets). The bar lights up with real values once TASK 9 supplies the RTMP URL (until then the pump skips the encoder and the bar shows the tier's targets)
|
||||
7. ✅ **One-click go live + private-only enforcement SHIPPED** (2026-08-14) — the Go Live dialog is **locked to Private** (no dropdown, `GoLiveViewModel.Visibility` is a get-only "Private"); `YouTubeStreamService.CreateBroadcast` **always sends `privacyStatus = "private"`** (dialog + service enforcement, requirement 8 — nothing can go out non-private) and gained an injectable `HttpClient? http = null` seam for tests; `BeginGoLive` now calls `CreateBroadcastAsync` and remembers `_currentBroadcastId` for TASK 9's bind/transition (failure → `StreamStatus.Error`, never a crash; `StopStream` clears the ID); the REC sign shows a **PRIVATE badge** (dark-red border, next to REC, `IsLivePrivate`) when the live stream is private; settings' dead "Default Visibility" dropdown + `MainViewModel.Visibilities`/`DefaultStreamVisibility` removed. The broadcast-insert integration test asserts the request body carries `"privacyStatus":"private"` (2 new tests → 173 passing, 0 warnings)
|
||||
|
||||
The pipeline chain the encoder needs doesn't exist yet: **scene compositing** (the master 1920×1080 frame
|
||||
without the preview's editing chrome) → **audio capture** (WASAPI, feeds the meter) → **H.264+AAC encode**
|
||||
→ **vertical-tier crop/scale** → **RTMP push** → **health stats** into the bottom bar. Nothing can encode
|
||||
until a frame source exists, so the compositor is ship step 1. The pipeline is
|
||||
`CameraManager + ScreenCaptureManager → compositor resolver → compositor → encoder → RTMP`.
|
||||
|
||||
### Requirements:
|
||||
|
||||
1. **Encoding** — H.264 (hardware via NVENC/AMD, fallback x264) + AAC audio; **must comply**: keyframes ≤ 4s (gopSizeLong), closed GOP, AAC/MP3 @ 44.1/48kHz, mono/stereo only. **License posture (decided): GPL-free build** — NVENC (NVIDIA) / QSV (Intel) / AMF (AMD) + OpenH264 software fallback + built-in AAC; no libx264 (GPL contaminates a paid product). Output containers are identical either way (H.264+AAC in `.flv` for RTMP, `.mp4`/`.ts` for VOD) — the format is NOT the differentiator, the license and per-GPU quality are. **License guardrails (never violate — see `ai.md` → "Licensing — do not violate"):** only BtbN `lgpl`/`lgpl-shared` builds; never GPL (gyan.dev) or `nonfree` (fdk-aac); never static for distribution (LGPL §6 relink material); never link FFmpeg into the app; never drop `THIRD-PARTY-NOTICES.txt` from the app/About screen.
|
||||
2. **RTMP push** — **FFmpeg subprocess (decided)**: app feeds raw frames via stdin, parses stderr for health; one battle-tested binary does encode + FLV mux + push. **No self-heal (claim-check 2026-09-01): ffmpeg's reconnect flags are input-side; an RTMP *output* push does not reconnect itself — retry is app-side, owned by TASK 32.** **Binary distribution (decided): check-then-pull** — probe `where ffmpeg`/PATH at first go-live; if absent, download a **pinned** build (**BtbN LGPL-shared win64** zip, ~75 MB — gyan.dev's builds are GPLv3 and ship libx264, which violates the license posture; BtbN's LGPL variant drops x264/x265 while keeping NVENC/QSV/AMF + libopenh264 + native AAC) to `%APPDATA%\ytLlive\tools\ffmpeg.exe` (extract `ffmpeg.exe` **plus the `libav*.dll` family**) and cache it, offline-friendly. Behind an `IFfmpegLocator` seam so tests fake it (ship step 2, below). Push goes to the cached reusable stream's ingestion URL
|
||||
3. **Quality ladder** — the offered tiers, with **1080p60 @ 8 Mbps as the standard/default**:
|
||||
1. 720p30 @ 6 Mbps
|
||||
2. 720p60 @ 6 Mbps
|
||||
3. 1080p30 @ 8 Mbps
|
||||
4. **1080p60 @ 8 Mbps** (default — mainstream ceiling, GPU hardware-encoded so the gaming
|
||||
machine never notices; upload headroom stays comfortable)
|
||||
5. Vertical 1080×1920 @ 60fps @ 8 Mbps (9:16 phone tier)
|
||||
The composition master is always 1920×1080; a tier is an output rect + target resolution
|
||||
(see `ai.md` "Resolution tiers"). Vertical output = the centered 607×1080 crop of the master
|
||||
scaled to 1080×1920 (semi-crop preview is already implemented; the encoder applies the same rect).
|
||||
1080p60 is the ceiling by design — "if you want 1440 or 4K or 8K → OBS is your solution"; the app
|
||||
targets the most mainstream creator, not power users.
|
||||
Ladder is sculpted by a **cached probe** (IP-only TCP vs public ingest host; no auth required).
|
||||
Quality is greyed out while live because the declared resolution can't change mid-stream — but
|
||||
with `variable`, we can **auto step-down** bitrate/resolution on the fly with zero API calls
|
||||
(no stream recreation); 60fps presumes a hardware encoder — no hardware encoder → auto
|
||||
fallback to 720p60/1080p30
|
||||
4. **Stream key management** — reuse the cached reusable stream (one per channel) instead of creating a new one per go-live; prefill default YouTube ingest URL `rtmp://a.rtmp.youtube.com/live2`
|
||||
5. **Health stats** — bitrate, FPS, dropped frames reported live in the bottom bar (encoder-side)
|
||||
6. **One-click go live** — defaults that work out of the box
|
||||
7. **Audio capture (feeds the meter — this task ships the wiring)** — WASAPI loopback (desktop/game at unity, zero UI — "it just is") + the picked mic (`MicSourceName` from the `MicPickerDialog`). The mic capture feeds `AudioLevel` so the realtime meter comes alive (today it reads 0 — the mixer feed is pending, see `ai.md` audio notes). AAC mono/stereo @ 48 kHz per the compliance rules.
|
||||
8. **Private-only go live until v1 (reputation guard, decided 2026-08-10)** — until the v1 release, go-live is **locked to private streams only** so a software error can never publish something public/unlisted that damages the creator's reputation. RTMP push itself has no privacy — privacy lives on the YouTube **live broadcast object**, which this app already controls via its OAuth API calls. So the lock is purely API-side: the Go Live flow always creates/updates the broadcast with `privacyStatus = "private"` and a guard **refuses** to set anything else (same spirit as the Live-only backdrop policy). The UI shows a clear "PRIVATE" badge next to the stream state so the creator always knows who can see them. Enforcement must be verifiable in the auth-service tests (fake the broadcast-insert/update call, assert `privacyStatus` is forced to private).
|
||||
9. **v1 release gate: bundle the full license texts (decided 2026-08-10)** — `THIRD-PARTY-NOTICES.txt` currently links the canonical license texts rather than embedding them. At the **v1 (GA) release**, the full texts of every license it names (LGPL v2.1+, BSD-2-Clause, MIT, Apache-2.0) MUST be bundled alongside it (shipped in the app output, e.g. a `licenses/` folder next to the notices file, still reachable from the About screen). This is a **release blocker for v1, not a task to queue early** — do it in the release pass. The repo should treat this like the private-only go-live gate: a checkbox that cannot silently lapse.
|
||||
|
||||
#### Ship step 1 — Scene compositor (the frame source)
|
||||
|
||||
**Goal:** a pure-CPU software compositor producing the encoder's master frame (BGRA8, the `VideoFrame`
|
||||
seam) from the scene model. The preview stays XAML (the editing view); the compositor is the **output
|
||||
view** — WPF's `RenderTargetBitmap` can't be used (software-rendered + captures chrome). Two renderers
|
||||
must agree, so the XAML (`MainWindow.xaml` CanvasGrid + element DataTemplate) is the contract.
|
||||
|
||||
**Decisions (locked 2026-08-10):** **Path A CPU blitter** — GPU effort belongs to NVENC (the encoder),
|
||||
not composition; with an FFmpeg subprocess the master crosses a CPU readback to the pipe every frame
|
||||
anyway, so GPU compositing buys ~nothing at this layer count (2-3 live layers; static layers
|
||||
pre-composite once). A D3D11 compositor can replace this one later **behind the same seam** (the CPU
|
||||
master buffer stays the contract). **Render the output rect directly**: compositor is constructed with
|
||||
`CompositorOptions {SourceRectX/Y/W/H, OutputWidth, OutputHeight}`; 16:9 tiers = full 1920×1080 1:1;
|
||||
vertical (9:16) = composite the centered 607×1080 crop then bilinear-upscale to 1080×1920. Reuses
|
||||
`MainViewModel.OutputRectX/Y/W/H` (note `(1920−607)/2 = 656.5` → align to integer pixels for output).
|
||||
|
||||
**Render spec (back → front, mirror the XAML exactly):**
|
||||
1. Backdrop — the Live scene's `IsBackdrop` Source (`CaptureKey` → live frame), `UniformToFill`
|
||||
full-frame (XAML's separate `BackdropImage` layer; the backdrop *element* renders nothing — its
|
||||
DataTemplate Image is Collapsed for DisplayCapture).
|
||||
2. Background — the scene's `Background` Source, `UniformToFill` full-frame (the `ActiveBackgroundImage`
|
||||
layer, not per-element).
|
||||
3. Elements in `Scene.Elements` order (back→front), skip `IsVisible=false`. What actually renders:
|
||||
1. `Source` Type `Image` → static asset, `UniformToFill` cover-crop into (X, Y, W, H)
|
||||
2. `WebcamSceneConfig` → latest frame by `DeviceId`: Traditional = `UniformToFill` rect; Round = circle
|
||||
diameter `min(W,H)` (alpha 0 outside — true circle, not oval); mirror = horizontal flip around
|
||||
element center (`MirrorScale`); opacity = per-pixel multiply (content + border); border = stroked
|
||||
rect / centered circle at `RoundBorderSize`, width `BorderWidth`, alpha `BorderOpacity`
|
||||
3. `Background` / `IsBackdrop` / `TextOverlay` are NOT per-element (layers above; Text not shipped)
|
||||
4. Branding flash — pre-rendered full-frame "made with ytLlive!" at 25% alpha when live +
|
||||
`BrandFlashEnabled` + timer active. Passed in as a `VideoFrame?` (compositor core stays pure byte-math,
|
||||
no WPF; likely a bundled asset rather than runtime text rendering).
|
||||
5. NOT in output (preview chrome only): SelectionOverlay, DimRects, output-rect outline, badge, placeholder.
|
||||
|
||||
**New files (all in `Services/Compositor/`):**
|
||||
1. `SceneCompositor.cs` — `Render(Scene, frameFor: Func<SceneElement, VideoFrame?>, flashFrame:
|
||||
VideoFrame?, CompositorOptions) → VideoFrame` (output-sized). The caller's `frameFor` resolver maps
|
||||
each element to its frame (webcam → DeviceId, image → AssetId via `StaticPixelCache`, backdrop →
|
||||
CaptureKey) — the compositor stays pure/hermetic/no WPF.
|
||||
2. `CompositorOptions.cs` — source-rect + output W×H.
|
||||
3. `StretchMath.cs` — `UniformToFill` cover-crop, ellipse mask, bilinear scale (pure, unit-tested).
|
||||
4. `StaticPixelCache.cs` — asset `byte[]` → cached BGRA `VideoFrame` (WPF `BitmapDecoder` + `CopyPixels`,
|
||||
decode once per content hash).
|
||||
|
||||
**Test plan (Good Dog Rule — ONE integration test):** `SceneCompositorTests` — a scene with backdrop
|
||||
(solid red fake frame) + round webcam (solid green) + image (solid blue) → render 16:9 master → assert
|
||||
per-layer probe pixels (corner = backdrop color, element center = webcam color, outside the round clip =
|
||||
backdrop color, mirrored element swaps left/right); a vertical-tier variant asserts 1080×1920 output +
|
||||
crop fidelity. Focused unit tests on `StretchMath`. Tests push frames directly — no capture managers
|
||||
involved (they wire in a later step).
|
||||
|
||||
**Same-PR housekeeping:** fix the stale comment `MainViewModel.cs:324` ("shown under the meter on line 2"
|
||||
→ "shown left-justified INSIDE the meter bar" — `ai.md` is the authority); this task's requirements now
|
||||
include the explicit audio-capture/meter wiring (#7 above).
|
||||
|
||||
**Out of scope (later ship steps):** FFmpeg locator + license posture (covered in requirements 1-2),
|
||||
encoder + RTMP push, WASAPI audio capture (loopback + mic) feeding `AudioLevel`, wiring
|
||||
`CameraManager`/`ScreenCaptureManager` into the frame pipeline, brand-flash timer wiring, health stats
|
||||
(bitrate/FPS/dropped).
|
||||
|
||||
**Built (2026-08-10):** all four files shipped in `Services/Compositor/`, `SceneElement.TryGetBorderColor`
|
||||
made public (shared hex parse with the compositor — no duplicated color parsing), the stale
|
||||
`MainViewModel.cs:324` comment corrected, and the pre-existing CS1998 in `YouTubeAuthServiceTests`
|
||||
cleaned up — build **0 warnings**. Tests: the `SceneCompositorTests` integration test (full-scene master
|
||||
pixels, vertical tier, flash) + 4 `StretchMath` units — **72 passing**.
|
||||
|
||||
#### Ship step 2 — FFmpeg locator (the encoder's binary)
|
||||
|
||||
**Goal:** resolve a usable `ffmpeg.exe` on demand (the encoder's one external dependency), never shipping
|
||||
a binary in the repo. Returns an absolute path; downloads only when neither PATH nor the local cache
|
||||
provides one.
|
||||
|
||||
**Decisions (locked 2026-08-10):**
|
||||
1. **BtbN LGPL-shared win64 build** — not gyan.dev (gyan's "essentials" is GPLv3 and ships libx264, which
|
||||
violates requirement 1's license posture) and **not the static lgpl build**: LGPLv2.1 §6 wants
|
||||
relinkable object files for static linking, but the **shared** (dynamic-DLL) variant sidesteps that —
|
||||
compliance is "license text + source offer + unmodified binaries" (see `THIRD-PARTY-NOTICES.txt` and
|
||||
`ai.md` → Licensing). Drops libx264/libx265 while keeping NVENC/QSV/AMF, libopenh264 (the LGPL-legal
|
||||
H.264 software fallback) and native AAC — exactly the requirement-1 encoder profile.
|
||||
2. **Pinned URL** — **RE-PINNED 2026-09-01**: `https://github.com/BtbN/FFmpeg-Builds/releases/download/autobuild-2026-08-31-13-27/ffmpeg-N-126342-gf88b741dbf-win64-lgpl-shared.zip`
|
||||
(BtbN **lgpl-shared** variant, ffmpeg N-126342; ~75 MB zip — earlier "~30 MB" estimate corrected). A dated autobuild tag is immutable; BtbN retention
|
||||
keeps the last 14 daily builds + each month-end build for 2 years, so a cold cache after retention
|
||||
expiry 404s — a logged, recoverable failure (the seam throws; the encoder step surfaces it). The
|
||||
first pin (`autobuild-2026-08-09-13-03`, a daily) aged out on 2026-09-01 — the first real recording
|
||||
attempt — proving the rule; the new pin is deliberately the **month-end** build (2-year retention).
|
||||
Dated tags carry versioned asset names (`ffmpeg-N-…-win64-lgpl-shared.zip`), extraction is
|
||||
name-agnostic; download failures now wrap in `IOException` with an actionable message
|
||||
("refresh `FfmpegLocator.PinnedUrl` or install ffmpeg on PATH") instead of leaking a raw 404. Once
|
||||
cached, the URL is never touched again. The pin is a single `const`, bumpable in one place — and must
|
||||
always stay on the **shared** variant (never `gpl`, `nonfree`, or static; see ai.md Licensing).
|
||||
3. **Check-then-pull order** — (1) PATH probe (the user's own install wins), (2) cached
|
||||
`%APPDATA%\ytLlive\tools\ffmpeg.exe`, (3) download + extract. Extract `ffmpeg.exe` **plus the
|
||||
`libav*.dll` family** (the shared build's bin/ folder; Windows resolves the DLLs from the exe's own
|
||||
directory) into a staging dir then move into place — a crash never leaves a corrupt or partial cache.
|
||||
4. **Seam** — `IFfmpegLocator.LocateAsync(CancellationToken)`: search dirs, tools dir, and the downloader
|
||||
(`Func<string, CancellationToken, Task<byte[]>>`) are constructor-injected with production defaults, so
|
||||
tests fake the network (feeding a real in-memory zip) and never touch disk outside a temp dir.
|
||||
|
||||
**New files (all in `Services/Encoder/`):**
|
||||
1. `IFfmpegLocator.cs` — the seam.
|
||||
2. `FfmpegLocator.cs` — the impl (PATH probe → cache → pull+extract exe + DLLs), failures logged via `AppLog`.
|
||||
3. `THIRD-PARTY-NOTICES.txt` (repo root) — the LGPL/BSD/MIT notices + source offer, copied to the build
|
||||
output. *(Surfacing changed on 2026-08-13: the top-bar About button that opened the file in the OS
|
||||
viewer is GONE — the notices are reachable in-app via the logo → About overlay instead.)*
|
||||
|
||||
**Test plan:** the hermetic integration test drives the full decision ladder against a temp tools dir and
|
||||
a fake downloader returning a real in-memory zip (`.../bin/ffmpeg.exe` entry): PATH hit wins without
|
||||
downloading, cache hit skips the network, cold cache downloads → extracts → `ffmpeg.exe` lands in the
|
||||
tools dir, and a second call serves the cache (downloader invoked exactly once). Focused unit tests:
|
||||
**shared-build DLLs extract alongside the exe**, empty zip throws, missing entry throws, empty download
|
||||
throws, downloader failure propagates, zero-byte cache is refreshed.
|
||||
|
||||
**Same-PR housekeeping:** requirement 2's stale binary facts corrected in this plan (~30 MB → ~75 MB zip;
|
||||
"gyan.dev/BtB N" → BtbN LGPL-shared only, with the why); the "never do" licensing guardrails recorded in
|
||||
`ai.md` so the reasoning survives.
|
||||
|
||||
**Out of scope (later ship steps):** the FFmpeg subprocess encoder (frames in via stdin, stderr health
|
||||
parsing), RTMP push, WASAPI audio capture, the frame-pipeline wiring, health stats.
|
||||
|
||||
**Built (2026-08-10):** `IFfmpegLocator` + `FfmpegLocator` shipped in `Services/Encoder/`, pinned to the
|
||||
**lgpl-shared** build `autobuild-2026-08-09-13-03` (extracts `ffmpeg.exe` + the `libav*.dll` family via a
|
||||
staging dir). `THIRD-PARTY-NOTICES.txt` (repo root) ships to the build output; the "never do" licensing
|
||||
guardrails are recorded in `ai.md` — build **0 warnings**.
|
||||
Tests: the hermetic `FfmpegLocatorTests` integration test (PATH → cache → download decision ladder with a
|
||||
fake downloader serving a real in-memory zip) + edge/unit cases (shared-build DLL extraction, zero-byte
|
||||
cache refresh, empty payload, missing zip entry, downloader failure) — **78 passing**.
|
||||
|
||||
#### Ship step 3 — Encoder + RTMP push (the FFmpeg subprocess)
|
||||
|
||||
**Goal:** encode raw BGRA master frames into H.264+AAC FLV and push them to the reusable stream's RTMP
|
||||
ingestion URL — one battle-tested subprocess doing encode + mux + push (retry is app-side, TASK 32 — ffmpeg does not self-reconnect an RTMP output), the app feeding
|
||||
frames via stdin and parsing stderr for health (req 2).
|
||||
|
||||
**Decisions (locked):** the encoder is a thin orchestrator over `ffmpeg.exe` — no H.264/AAC code in the
|
||||
app. Arguments (pure `FfmpegArgs.Build`): `-re -f rawvideo -pix_fmt bgra -video_size WxH -framerate FPS
|
||||
-i pipe:0` (frames in), a **silent placeholder audio track** via `-f lavfi -i anullsrc` (the WASAPI
|
||||
capture step replaces this input), `-c:v <encoder> -b:v K -maxrate K -bufsize 2K` + **`-g fps×4`
|
||||
`-keyint_min fps×4` `-sc_threshold 0` `-bf 0` `-pix_fmt yuv420p`** (the keyframe ≤4s / closed-GOP /
|
||||
H.264 compliance), `-c:a aac -ar 48000 -ac 2`, `-f flv <rtmpUrl>`. Encoder choice is **probed from the
|
||||
binary's `-encoders` listing** (`FfmpegEncoderPicker`, pure): hardware NVENC → QSV → AMF, then OpenH264
|
||||
software fallback — **never libx264** (GPL; see `ai.md` → Licensing). The seam (`IFfmpegEncoder` +
|
||||
`IEncoderProcess`, constructor-injected locator + process factory) keeps it hermetic — tests fake the
|
||||
whole subprocess (probe + encoder), no real binary.
|
||||
|
||||
**Behavior:** `StartAsync` (locate → probe → spawn → stderr loop), `SubmitFrameAsync` (serialized BGRA
|
||||
stdin writes, ~2 Hz health via `HealthUpdated`/`StreamHealth` — bitrate/FPS/duration/dropped-from-frame-
|
||||
count), `StopAsync` (stdin EOF → ffmpeg finalizes + exits by itself; 10s watchdog kill), `ProcessFailed`
|
||||
on a non-zero unexpected exit.
|
||||
|
||||
**Built (2026-08-12):** `EncoderOptions` + `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/
|
||||
`FfmpegEncoderProcess` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` in
|
||||
`Services/Encoder/`. Not yet constructed by the app (the frame-pipeline wiring, ship step 5, owns it).
|
||||
Tests: `FfmpegEncoderTests` integration (probe → spawn with NVENC preferred → frames into stdin →
|
||||
progress parsed → graceful stop, no kill) + units (args compliance/GOP, progress parser, picker
|
||||
preference + GPL guard, no-URL/not-running/noop stops, process-death `ProcessFailed`) — **122 passing**.
|
||||
|
||||
#### Ship step 4 — WASAPI audio capture (the meter comes alive)
|
||||
|
||||
**Goal:** capture desktop/game audio (loopback) and the picked mic, feed the mic level into `AudioLevel`
|
||||
so the realtime meter reads something other than 0, run capture **only while live** (req 7).
|
||||
|
||||
**Decisions (locked):**
|
||||
1. **NAudio `NAudio.Wasapi` 2.2.1** — the wasapi feature package (not the `NAudio` meta-package): it
|
||||
carries the capture types (`WasapiCapture`/`WasapiLoopbackCapture` + the MMDevice enumeration) with
|
||||
`NAudio.Core`/`NAudio.Asio` pulled in transitively. MIT — recorded in `THIRD-PARTY-NOTICES.txt` (item 9).
|
||||
2. **`IAudioSource` seam** (`Start`/`Stop`/`SampleReady`/`Failed`, IDisposable) — the app consumes the
|
||||
seam; the two WASAPI implementations wrap NAudio; tests inject hermetic fakes (no real audio devices,
|
||||
no timers). Loopback = `WasapiLoopbackCapture` on the default render device; mic = `WasapiCapture`
|
||||
with the NAudio device resolved by `FriendlyName` matching `MicSourceName` (the app only persists the
|
||||
DisplayName), falling back to the default capture endpoint. Mic device resolution is re-read at each
|
||||
`Start` via a name provider so a mic picked mid-session takes effect next go-live.
|
||||
3. **`AudioMixer` owns both sources** — starts/stops both with go-live (`BeginGoLive` success → `Start`,
|
||||
`StopStream` → `Stop`). Mic samples feed a pure `AudioLevelMeter` (RMS, exponential smoothing) and
|
||||
raise `MicLevelChanged`, marshalled to the UI thread into `AudioLevel`; desktop samples are currently
|
||||
dropped (consumed by the encoder's AAC mix in a later step). Capture failures are logged via `AppLog`
|
||||
(mic failure also zeroes the meter); loopback failure doesn't kill the mic.
|
||||
4. **Byte→float** — pure `WaveToFloat.Convert` handles the WASAPI mix formats: IEEE float 32-bit (direct)
|
||||
and PCM 16-bit (normalized to -1..1), including `WaveFormatExtensible` with the IEEE-float subformat
|
||||
GUID. Trailing partial samples are ignored.
|
||||
|
||||
**Built (2026-08-12):** `Services/Audio/` ships `IAudioSource` + `AudioSample`, `WasapiLoopbackAudioSource`,
|
||||
`WasapiMicAudioSource`, `AudioMixer`, `AudioLevelMeter`, `WaveToFloat`; `MainViewModel` constructs the
|
||||
mixer (mic source fed `() => MicSourceName`), starts it on go-live and stops it on end-stream, and maps
|
||||
`MicLevelChanged` → `AudioLevel`. A pre-existing CS8602 in `FfmpegEncoder.cs:139` surfaced during this
|
||||
step's rebuild and was fixed (`process!`) — build **0 warnings**. Tests: `AudioMixerTests` (mixer
|
||||
lifecycle/forwarding/failure against fakes, meter RMS/smoothing/reset, `WaveToFloat` float/PCM16/
|
||||
extensible/truncation) — **139 passing**.
|
||||
|
||||
**Deferred (later ship steps):** wiring the desktop-capture samples into the encoder's AAC mix (replaces
|
||||
the `-f lavfi -i anullsrc` placeholder; the encoder construction itself shipped in ship step 5).
|
||||
*(The "capture while not live" + "audio UI beyond the mic controls" deferrals were SHIPPED on the
|
||||
2026-08-13 game audio bar branch — capture is now always-on for preview and the game bar is the second
|
||||
audio UI. The mixer's short-circuit meter fix + `Started`/`RestartMic` seams live in the same branch.)*
|
||||
|
||||
#### Ship step 5 — Frame-pipeline wiring (the encoder gets a frame source)
|
||||
|
||||
**Goal:** the chain `CameraManager`/`ScreenCaptureManager` → compositor resolver → encoder, driven while
|
||||
live by a paced frame pump: snapshot the active scene → resolve each element to its latest frame →
|
||||
composite into the tier's output frame → pace into the encoder's stdin at the tier's FPS.
|
||||
|
||||
**Decisions (locked via user Q&A, 2026-08-12):**
|
||||
1. **Video pipeline first** — the `-f lavfi -i anullsrc` silent track stays; mixing the loopback/mic
|
||||
WASAPI samples into the encoder's AAC track is its own later step.
|
||||
2. **RTMP URL via a provider seam** — `MainViewModel._rtmpUrlProvider` is a `Func<string?>` returning
|
||||
null today (the reusable stream's ingest URL lands with TASK 9); when it yields null the pump logs
|
||||
and skips the encoder entirely, so go-live runs the existing visual flow without pushing.
|
||||
|
||||
**Design:**
|
||||
1. `Services/Encoder/FramePump.cs` — the frame producer. All collaborators constructor-injected seams
|
||||
(`Func<Scene?>`, `Func<SceneElement, VideoFrame?>` resolver, `Func<CompositorOptions>`,
|
||||
`Func<EncoderOptions?>`, `Func<IFfmpegEncoder>`, `Action<string>` log, injectable pacing delay) so it
|
||||
stays free of WPF and of the capture managers and is hermetic in tests. `StartAsync` never throws
|
||||
(failures log + surface via `Failed` — the VM fires-and-forgets from the sync command handler);
|
||||
loop = snapshot → render → `SubmitFrameAsync`, paced at `1/options.Fps` (default `Task.Delay`; tests
|
||||
inject `Task.Yield`). `StopAsync` stops the encoder (closes stdin) BEFORE awaiting the loop — closing
|
||||
stdin unblocks a write stuck on pipe backpressure, so stop can't deadlock on the pump. `ProcessFailed`
|
||||
self-stops the pump. `HealthUpdated` forwards the encoder's stats (ship step 6 binds the bottom bar).
|
||||
2. `ScreenCaptureManager.GetLatestFrame(key)` — mirrors `CameraManager.GetLatestFrame(deviceId)`; the
|
||||
backdrop's live frame for the compositor.
|
||||
3. `MainViewModel` — owns the resolver (`WebcamSceneConfig` → `GetLatestFrame(WebcamId)`;
|
||||
`Source.IsLiveCapture` → `GetLatestFrame(CaptureKey)`; image/background → `StaticPixelCache.Get(AssetId)`),
|
||||
builds `CompositorOptions` from the tier + `OutputRect*` (doubles rounded to ints — the vertical
|
||||
607.5 half-pixel crop rounds to a perfectly-centered 608), builds `EncoderOptions` from the tier when
|
||||
the URL provider returns one, constructs the real `FfmpegEncoder(new FfmpegLocator())`, starts the
|
||||
pump on go-live, stops it on end-stream, disposes in `Shutdown`, and flips `StreamStatus.Error` when
|
||||
the pump fails while live (minimal — detailed health surfacing is ship step 6).
|
||||
|
||||
**Test plan (Good Dog Rule — ONE integration test):** `FramePumpTests.Start_CompositesScene_FeedsEncoder_StopsCleanly`
|
||||
drives the full lifecycle against fakes — real `SceneCompositor` + real `FramePump`, fake `IFfmpegEncoder`
|
||||
— asserting the composited red backdrop frame actually reaches the encoder at the tier size and that stop
|
||||
tears everything down. Units: no-URL start skips the encoder, re-entrant start/stop no-ops, encoder
|
||||
start-failure raises `Failed` + disposes, `ProcessFailed` self-stops the pump, `HealthUpdated` forwards.
|
||||
`ScreenCaptureManagerTests.GetLatestFrame_ReturnsLatestPump_UntilReleased` pins the new accessor.
|
||||
|
||||
**Out of scope (later ship steps):** the loopback/mic → AAC mix (replaces `anullsrc`), health stats in the
|
||||
bottom bar (ship step 6), scene-switching transitions, and any flash-frame wiring.
|
||||
|
||||
**Built (2026-08-12):** `FramePump` shipped in `Services/Encoder/`, `ScreenCaptureManager.GetLatestFrame`
|
||||
added, `MainViewModel` wired end-to-end (resolver + both option builders + pump lifecycle), `FramePumpTests`
|
||||
(7) + `GetLatestFrame` test (1) added — build **0 warnings**, **147 tests passing**. Known consideration:
|
||||
the pump reads the active scene on a background thread while the UI can still edit it; a concurrent-mutation
|
||||
exception is contained (logged + `Failed` + pump stops) rather than crashing.
|
||||
|
||||
#### Ship step 5.5 — Social bar bug fixes + the bar on the live output (2026-08-13)
|
||||
|
||||
**Bug 1 — bar wouldn't reliably change position (root cause found, then simplified):** the original
|
||||
drag set `Canvas.SetTop(bar, …)` with a local value, which permanently overrides the
|
||||
`Canvas.Top="{Binding SocialBarTop}"` binding — the release-time `SetSocialBarPosition` →
|
||||
`PropertyChanged(SocialBarTop)` could never beat it. First fix added direction-snapping during the drag
|
||||
(`SocialBarSnap.Decide`, ±6px deadzone) + `bar.ClearValue(Canvas.TopProperty)` on release — but that
|
||||
still misbehaved for shaky hands (jitter around the deadzone: it snapped up reliably, then refused to
|
||||
come back down and snapped back to top). **Superseded by a click-toggle (KISS, user decision):** clicking
|
||||
the bar in the preview flips it top ⇄ bottom (`MainViewModel.ToggleSocialBarPosition` → the existing
|
||||
`SetSocialBarPosition`), the bar rides `{Binding SocialBarTop}` alone (no local values, no deadzone, no
|
||||
jitter sensitivity), and `SocialBarSnap` was removed. The `ClearValue` lesson stands: never set a local
|
||||
value on a property the binding owns.
|
||||
|
||||
**Bug 2 — Mastodon showed the generic 7-star honeycomb (root cause found):** the DB row
|
||||
`@gramps@llamachile.tube` had `Software = NULL` — nodeinfo was only ever resolved against the identity domain
|
||||
(`llamachile.tube`, a landing page), never probed for the real instance at `mastodon.llamachile.tube`.
|
||||
Fixed on three fronts: `HttpSocialValidator` now **probes well-known subdomains** (mastodon. → social. →
|
||||
… `FediverseSubdomainCandidates`) when the identity domain and its redirect both come up empty, under a
|
||||
~15s linked-CTS budget, with an optional `Action<string>` log; `MainViewModel` **heals** any fediverse entry
|
||||
missing a software name on layout load (`HealFediverseSoftwareAsync` — static, testable; instance wrapper
|
||||
runs it off the UI thread, applies via the dispatcher, saves); `SocialEntry.FediverseSoftware` is now
|
||||
**settable** and raises `PropertyChanged` for `LogoData`, so the heal updates the icon in place. If the user
|
||||
later re-saves the entry with the icon fixed, the name persists with it.
|
||||
|
||||
**Compositor rendering (user-approved scope):** the social bar now appears **on the live output**, not just
|
||||
the preview — `Compositor/SocialBarRenderer.cs` rasterizes the entries into a transparent straight-alpha
|
||||
BGRA strip (1920-wide, 40px content + 24px glow pad, green `#2ecc71` glow baked in) via `RenderTargetBitmap`
|
||||
(WPF glue like `StaticPixelCache`; the compositor core stays pure). `SceneCompositor.Render` takes optional
|
||||
`socialBarFrame` + `socialBarTop` (master space) and blits it **last — above the branding flash** (the old
|
||||
`BlitFlash` generalized to offset `BlitOverlay`). `FramePump` gains a `socialBar:` seam
|
||||
(`Func<(VideoFrame?, SocialBarPosition)>`, re-read every frame) and places the bar at `0` or
|
||||
`SourceRectHeight − bar height`. `MainViewModel` owns the frame (`RenderSocialBarFrame`, re-rendered on
|
||||
load/save/notify) and feeds the seam.
|
||||
|
||||
**Tests (+6 → 153 passing, 0 warnings):** settable `FediverseSoftware` updates `LogoData`,
|
||||
subdomain-probe unit + null-when-silent unit, the branch's **one integration test**
|
||||
`Socials_HealMissingFediverseSoftware_RoundTripsThroughDb` (temp-DB roundtrip: NULL software → healed via
|
||||
the validator → persisted), compositor bar overlay (top/bottom + above-flash), `FramePump` bar pass-through
|
||||
(Top then flipped to Bottom mid-run — the seam is re-read each frame), and both `ISocialValidator` fakes
|
||||
(`FakeValidator`/`BlockingValidator`) gained `ResolveFediverseSoftwareAsync`. The drag-snap units
|
||||
(`SocialBarSnap`) were removed with the click-toggle supersession.
|
||||
|
||||
**Not included (say the word):** rewriting the healed entry's `ProfileUrl` to `https://mastodon.llamachile.tube/@gramps`.
|
||||
|
||||
Reference in New Issue
Block a user