diff --git a/AGENTS.md b/AGENTS.md index 5f0bc29..89e5406 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -38,7 +38,7 @@ conventions live here and in `ai.md`. 1. [`schema.md`](schema.md) — the memory-map conventions (what lives where, how to keep it true). 2. [`ai.md`](ai.md) — the AI guide: architecture, patterns, decisions, current state. -3. [`TASKS.md`](TASKS.md) — the task queue + authoritative YouTube API research. +3. [`TASKS.md`](TASKS.md) — the task catalog (index + queue state); open items and API research facts are in [`TASKS/`](TASKS/). 4. [`HANDOFF.md`](HANDOFF.md) — current operational state: what's in flight, landmines, next step. 5. `/index.md` — the index for any directory you're about to touch. diff --git a/TASKS.md b/TASKS.md index 18d8d91..8455dcb 100644 --- a/TASKS.md +++ b/TASKS.md @@ -1,1634 +1,85 @@ # ytLlive — Task List -> Task queue and authoritative research. Memory-map conventions: [`schema.md`](schema.md); -> architecture/decisions: [`ai.md`](ai.md). Update statuses here whenever a task moves. +> **Catalog:** task files live in [`TASKS/`](TASKS/) (one file per task). This file is the +> index. Memory-map conventions: [`schema.md`](schema.md); architecture/decisions: [`ai.md`](ai.md). -> **Checklist markers** — every task's Status list uses the same states: -> 1. ✅ — completed (green check) -> 2. 🔶 — in progress (amber diamond) -> 3. ☐ — not completed / pending (empty box) -> 4. ❌ — exception (blocked, known-issue, or deliberately excluded from this build) +> **Checklist markers** — every task's Status line uses the same states: +> 1. ✅ — completed / shipped +> 2. 🔶 — in progress +> 3. ⏳ — blocked / waiting +> 4. ☐ — not started / queued +> 5. ❌ — exception (known-issue, deliberately excluded, or merged) -> **2026-09-01 — v1 = feature-complete ruling:** there is no v1.x. Everything queues to v1 or to -> **"Out of product — permanently"** (file end). Read those two sections before adding or reviving -> anything here. - -## YouTube Live API — research facts (authoritative, v3 build) - -Lifecycle: `created → ready → [testing] → live → complete` (transitional `liveStarting` / `testStarting`). - -1. **liveBroadcasts.insert** requires: `snippet.title`, `snippet.scheduledStartTime`, `status.privacyStatus`, `status.selfDeclaredMadeForKids` (COPPA). -2. **liveStreams.insert** requires: `snippet.title`, `cdn.frameRate`, `cdn.ingestionType`, `cdn.resolution`. **None of the four (except title) can ever change after creation** — changing them means delete + recreate the stream. This is the hard constraint behind the quality grey-out. -3. **Title / description / privacy**: editable at any time, including while live (`liveBroadcasts.update`, part=`snippet,status`). -4. **contentDetails** (DVR, recordFromStart, monitorStream, embed, latency): editable only in `created` / `ready`. -5. **Transition to live** only allowed when the bound stream's `status.streamStatus == active`. - -### Two features that reshape the design - -1. **enableAutoStart / enableAutoStop** — instant one-click go-live, no transition call. With `enableAutoStart=true` we never call `transition(live)`: the broadcast auto-goes-live the moment the encoder starts. Combined with `enableMonitorStream=false` (our preview pane replaces YouTube's monitor stream — the thing that forces a testing stage), the flow is **create → bind → Start Stream → encoder starts → YouTube brings it live**. No testing, no transition polling, no liveStarting stuck-state handling. -2. **`cdn.resolution=variable` / `cdn.frameRate=variable`** — free auto step-down. YouTube auto-detects what we send; since we ARE the encoder we can drop bitrate/resolution on the fly with zero API calls. Declaring an explicit resolution instead (e.g. 1080p) requires a new stream, which can't happen mid-broadcast. Variable is the enabler for the whole auto step-down feature. **(Claim-check 2026-09-01: the enabler is real, the governor is not built — the drop-side policy ships as TASK 33, v1 per the complete-v1 ruling.)** - -### Compliance gotchas (maps perfectly to report-by-exception) - -1. `liveStreams.status.healthStatus`: `good | ok | bad | noData` plus `configurationIssues[]` with `type` + `severity` (`info|warning|error`). Literally built for report-by-exception — poll it, render nothing on good/ok, surface a banner only on warning/error. No need to invent our own health logic. -2. Encoder must comply or YouTube flags it: keyframes ≤ 4s (`gopSizeLong`), closed GOP, H.264, audio AAC/MP3 @ 44.1/48kHz, mono/stereo only. -3. Error codes to handle: `errorStreamInactive`, `invalidTransition`, `redundantTransition`, `liveStreamDeletionNotAllowed`, `liveStreamModificationNotAllowed`, `liveBroadcastBindingNotAllowed`. - -### Tips we should take advantage of - -1. **Reusable streams** (`isReusable=true`): one stream per channel, cache its ingestion URL + stream name, reuse for every broadcast. No rebinding dance each go-live. This is exactly the manual-stream-key baseline. -2. **Backup ingestion address**: YouTube provides a simultaneous-push backup — future hardening, not v1. -3. **`recordFromStart` + `enableDvr` default true** → every live is auto-recorded and immediately replayable. Free VOD archive, matches the v0.2 recording goal. -4. **`latencyPreference`: `normal | low | ultraLow`** — for homelab streamers talking to chat, `low` (or `ultraLow`, capped at 1080p) is a real feature. -5. **Broadcast ID == Video ID** — one ID to track everything. +> **2026-09-01 — v1 = feature-complete ruling:** there is no v1.x. Everything queues to +> v1 or to **"Out of product — permanently"** (file end). Read those two sections before +> adding or reviving anything here. --- -## TASK 1 — Initial Scaffold +## Task catalog -**Goal:** Working C# / WPF project with MVVM architecture, dark-theme main window, and YouTube service stubs. - -### Status: ✅ Done - -1. ✅ Models: Scene, Source, StreamConfig, StreamHealth, YouTubeChannel, ChatMessage -2. ✅ Services: YouTubeAuthService (OAuth2), YouTubeStreamService (broadcast/health), YouTubeChatService (chat polling) -3. ✅ MainViewModel: scene management, stream controls, chat -4. ✅ MainWindow: scene/source panel, preview area, chat panel, status bar -5. ✅ Clean build, 0 warnings (WSL + Windows) +| # | Task | Status | File | +|---|------|--------|------| +| 01 | Initial Scaffold | ✅ Done | [`TASKS/task-01-initial-scaffold.md`](TASKS/task-01-initial-scaffold.md) | +| 02 | YouTube OAuth2 Authentication | ✅ Done | [`TASKS/task-02-oauth2-auth.md`](TASKS/task-02-oauth2-auth.md) | +| 03 | Capture Pipeline (Scenes/Sources) | 🔶 In progress — 27/30 items done; items 16, 17, 20 open | [`TASKS/task-03-capture-pipeline.md`](TASKS/task-03-capture-pipeline.md) | +| 04 | RTMP Ingest to YouTube | ✅ Done — all 7 items shipped | [`TASKS/task-04-rtmp-ingest.md`](TASKS/task-04-rtmp-ingest.md) | +| 05 | Layout Persistence (SQLite) | ✅ Done | [`TASKS/task-05-layout-persistence.md`](TASKS/task-05-layout-persistence.md) | +| 06 | UI polish batch | ✅ Done | [`TASKS/task-06-ui-polish-batch.md`](TASKS/task-06-ui-polish-batch.md) | +| 07 | Meter scaling amplification | ✅ Done | [`TASKS/task-07-meter-scaling.md`](TASKS/task-07-meter-scaling.md) | +| 08 | Audio milestone | ✅ SHIPPED 2026-08-14 | [`TASKS/task-08-audio-milestone.md`](TASKS/task-08-audio-milestone.md) | +| 09 | YouTube Live Stream Management | ⏳ In progress — items 1–3, 4 shipped; item 5 open; item 6 locked (TASK 36) | [`TASKS/task-09-live-stream-management.md`](TASKS/task-09-live-stream-management.md) | +| 10 | Monetization: watermark-only subscription + Polar billing | 🔶 In progress — steps 1–7 shipped | [`TASKS/task-10-monetization.md`](TASKS/task-10-monetization.md) | +| 11 | Post-pause polish batch (creator's 8 review issues) | ✅ SHIPPED 2026-08-15 | [`TASKS/task-11-polish-batch.md`](TASKS/task-11-polish-batch.md) | +| 12 | Master limiter on the live mix | ☐ Queued | [`TASKS/task-12-master-limiter.md`](TASKS/task-12-master-limiter.md) | +| 13 | Social media launch kit | 🔶 Scoped — queued after v1 | [`TASKS/task-13-social-launch-kit.md`](TASKS/task-13-social-launch-kit.md) | +| 14 | Creator feedback batch | 🔶 In progress — Branch 2 shipped | [`TASKS/task-14-creator-feedback-batch.md`](TASKS/task-14-creator-feedback-batch.md) | +| 15 | Stock scene background images | ✅ Shipped | [`TASKS/task-15-scene-backgrounds.md`](TASKS/task-15-scene-backgrounds.md) | +| 16 | Kill infinity display | ✅ Shipped | [`TASKS/task-16-kill-infinity-display.md`](TASKS/task-16-kill-infinity-display.md) | +| 17 | Web source rendering (WebView2) | ✅ Done | [`TASKS/task-17-web-source-rendering.md`](TASKS/task-17-web-source-rendering.md) | +| 18 | Local recording | ✅ Shipped `a9eb360` (2026-08-29) | [`TASKS/task-18-local-recording.md`](TASKS/task-18-local-recording.md) | +| 19 | Scene transitions | MERGED → TASK 23 (Control Surface UX) | [`TASKS/task-19-scene-transitions.md`](TASKS/task-19-scene-transitions.md) | +| 20 | Hotkeys (keyboard shortcuts) | ✅ Done — shipped 2026-08-26 | [`TASKS/task-20-hotkeys.md`](TASKS/task-20-hotkeys.md) | +| 21 | Media source (video file playback) | 🔶 In progress — UI picker remaining | [`TASKS/task-21-media-source.md`](TASKS/task-21-media-source.md) | +| 22 | Audio sync offset | ✅ Done (2026-08-31) | [`TASKS/task-22-audio-sync-offset.md`](TASKS/task-22-audio-sync-offset.md) | +| 23 | Studio mode | MERGED → TASK 23 (Control Surface UX) | [`TASKS/task-23-studio-mode.md`](TASKS/task-23-studio-mode.md) | +| 24 | Toast notifications | ✅ Done (2026-08-23) | [`TASKS/task-24-toast-notifications.md`](TASKS/task-24-toast-notifications.md) | +| 25 | Background consolidation | ✅ Done (2026-08-23) | [`TASKS/task-25-background-consolidation.md`](TASKS/task-25-background-consolidation.md) | +| 30 | Gear menu cleanup + Default Location for Recordings + preview layout | ✅ Done (2026-08-29) | [`TASKS/task-30-gear-cleanup-preview-layout.md`](TASKS/task-30-gear-cleanup-preview-layout.md) | +| 31 | SceneGraph component + static/dynamic compositor optimization | ✅ Done (2026-08-31) | [`TASKS/task-31-scenegraph-optimization.md`](TASKS/task-31-scenegraph-optimization.md) | +| 32 | Stream resilience | ☐ Queued (2026-09-01) | [`TASKS/task-32-stream-resilience.md`](TASKS/task-32-stream-resilience.md) | +| 33 | Bandwidth auto step-down | ☐ Queued (2026-09-01) | [`TASKS/task-33-bandwidth-step-down.md`](TASKS/task-33-bandwidth-step-down.md) | +| 34 | Scheduled streams (Text-drawer version) | ☐ Queued (2026-09-01) | [`TASKS/task-34-scheduled-streams.md`](TASKS/task-34-scheduled-streams.md) | +| 35 | Scene-linked audio | ☐ Queued (2026-09-01) | [`TASKS/task-35-scene-linked-audio.md`](TASKS/task-35-scene-linked-audio.md) | +| 36 | Gold pass | ☐ Queued (2026-09-01) | [`TASKS/task-36-gold-pass.md`](TASKS/task-36-gold-pass.md) | --- ---- +## Open items (at a glance) -## TASK 2 — YouTube OAuth2 Authentication - -**Goal:** Fully working Google OAuth2 flow — user clicks "YouTube", browser opens, authorization callback lands, channel info is stored. - -### Status: ✅ Done - -1. ✅ Two-state Start/End Stream button, go-live dialog (account + title/description/visibility), red top bar, pulsing LIVE badge + elapsed timer, preview glow, taskbar red dot -2. ✅ Real OAuth2 wiring — baked-in Google credentials (desktop client; loopback callback) + `YouTubeAuthService` complete: browser launch, `HttpListener` callback, token exchange, refresh, channel fetch -3. ✅ Token persistence via Windows DPAPI (`Helpers/TokenStore.cs` → `%APPDATA%\ytLlive\ytLlive.auth`), best-effort reload + proactive refresh at startup, saved after every exchange/refresh -4. ✅ Account sign-in/change surfaced in the GoLive dialog (saved account shown with "Change Account"; "Sign in to YouTube" when none; Start disabled until signed in) -5. ✅ ~~**End Livestream signs out**~~ **SUPERSEDED by TASK 18 (2026-08-29): explicit sign-out only** — `StopStream()` no longer clears the session (sign out via Logout / Change Account, `SignOutYouTubeAsync`); stopping a stream or recording leaves the creator signed in. Originally shipped as: a graceful end completes the session: `StopStream()` calls `YouTubeAuthService.ClearSession()` + `TokenStore.Clear()` + `IsConnected = false`, so the next Start Stream dialog requires a fresh sign-in. A crash never runs End, so the DPAPI token survives and the creator stays signed in. Resume/reconnect after a midstream crash is deliberately deferred to TASK 3: the socket can't be resumed (it dies with the process), so "resume" = fast reconnect with a saved broadcast ID/stream key within YouTube's disconnect-grace window; too slow and `enableAutoStop` ends the broadcast. **Orphan closed 2026-09-01:** this mechanism is owned by **TASK 32 (Stream resilience)** — in-session blip retry inside the grace window; cross-process "resume" of a dead broadcast is officially out-of-product (the API makes it impossible — creator ruling). -6. ✅ Tests in `ytLive.Tests` (xUnit, net8.0-windows): TokenStore DPAPI roundtrip/corrupt/missing/clear + mocked exchange channel-parse + refresh expiry bump + `ClearSession` — 7 passing - -**Design constraint:** Sign-in must NEVER block core exploration. Users can build scenes, add sources, and audition the software without authenticating. But **going live requires authentication** — the "Start Stream" dialog is where the account sign-in lives, alongside all stream metadata. - -**Two-state flow:** There is no separate "Connect" button. The top bar shows a single button — **Start Stream** when idle, **End Stream** when live. Clicking Start Stream opens one dialog that supplies everything: account (previously-saved account shown as default, with a Change Account action) + title/description/visibility. - -**Live indicators (unmissable):** Top bar + window title bar flip red, a pulsing **● LIVE** badge with elapsed timer appears in the top bar, the preview area gets a red glow, and the taskbar icon shows a red overlay dot. Window title bar shows the stream title once validated. - -### Requirements: - -1. **Google Cloud OAuth credentials** — client ID + secret, **baked into `Helpers/OAuthCredentials.cs`** (desktop "Desktop app" OAuth client; loopback callback — no console redirect URI registration needed; creators never configure) -2. **Local HTTP listener** — `HttpListener` on `http://localhost:PORT/oauth2/callback` to catch the redirect -3. **Browser launch** — open the authorization URL in the default browser -4. **Token persistence** — store access/refresh tokens securely (Windows DPAPI), reload on startup -5. **UI state** — account shown in the Start Stream dialog; "Change Account" action triggers re-auth -6. **Go Live gated on auth** — Start Stream dialog requires sign-in to enable the Start button; scene building works without it - -### Tests: - -1. Mock token exchange response, verify channel info parsed -2. Verify token refresh triggers when near expiry -3. Verify credential load/save roundtrip +- **TASK 3** — items 16 (Text source), 17 (Alerts), 20 (Alerts precursor: RewardEvent capture) open +- **TASK 9** — item 5 open (webcam identity key reconciliation) +- **TASK 10** — Velopack update URL pending +- **TASK 12** — queued future +- **TASK 13** — queued post-v1 +- **TASK 14** — Branch 2 shipped; branch 3+ in progress +- **TASK 21** — UI picker slice remaining (Windows-only verification pending) +- **TASK 32–36** — all queued future work (2026-09-01) --- ---- +## YouTube Live API research facts -## TASK 3 — Capture Pipeline (Scenes/Sources) - -**Goal:** Real video preview in the center panel — the minimal source set below, composited per scene. - -### Status: 🔶 In progress — 27 of 30 items done (1 excluded: background removal); items 16, 17, 20 open - -1. ✅ **Milestone 1 — webcam** — MediaCapture (WinRT SDK projection) with device enumeration, CPU-first frame source, refcounted `CameraManager`, picker dialog, clip shapes (Traditional + Round) + mirror, 480×270 default placement — schema v2 -2. ✅ **Schema v3 (Ship Branch A)** — multi-scene webcam (singleton `Webcam` + per-scene `WebcamSceneConfig`), right-click border/context menu, static OBS-style borders, 50%-per-dimension webcam size cap, device-swap (`ReleaseAllAsync`) — 25 tests passing -3. ✅ **Schema v4** — round→rect restore persisted (`WebcamSceneConfig.RectWidth`/`RectHeight`) + one-time legacy-square 16:9 heal on load -4. ✅ **Screen backdrop (ship task #1, schema v5)** — live desktop/game capture as a permanent, non-deletable bottom layer (`Source.IsBackdrop`), auto-detecting the full-screen game at launch/focus (else the **primary display** — never assumed monitor 0) via `Win32FullScreenDetector` (now with `GetDisplays()`/`PrimaryMonitorIndex()` for the in-app display picker), content re-designated via the OS `GraphicsCapturePicker` ("Change Capture…") or the in-app "Capture Display" submenu, refcounted/shared capture sessions in `ScreenCaptureManager` mirroring `CameraManager` — 45 tests passing -5. ✅ **Schema v6 — backdrop Live-only by policy** — `Scene.HasBackdrop`, enforced by scene name on every load (`EnforceBackdropPolicy`: Starting/BRB/Chat/Ending never carry one; the one-time v5→v6 backfill covers all four), the scene context-menu "Backdrop" checkbox is gone (policy owns the flag), preview watermark hides when the backdrop renders, capture changed to `WindowsRuntimeMarshal.TryGetDataUnsafe` (the CsWinRT-safe frame-read) + downscale to the 1920×1080 master + 5s-throttled error logging (was flooding `startup.log` with 5 MB of cast errors and burning CPU), round webcam no longer re-rasterizes an `ImageBrush` every frame (Image + EllipseGeometry clip) — the live-mode stutter fix -6. ✅ **The five-scene catalog (`SceneCatalog`)** — Starting/Live/BRB/Chat/Ending is the product — work with less, never more; the (+) button only shows when a canonical scene is missing and re-adds it (its menu lists only the missing ones) — 62 tests passing -7. ✅ **Webcam-after-session-start fix** — a webcam added to a scene after the camera was already running (e.g. Chat) previously rendered a transparent container — `CameraManager.GetPreviewBitmap` + propagation in `AddWebcamToActiveSceneAsync`/`ReacquireWebcam` now hands the running shared frames to any newly added `WebcamSceneConfig` — 65 tests passing -8. ✅ **Chat scene webcam size cap** — raised from 50%-per-dimension (960×540) to half the screen AREA (~1358×764 @16:9, `MaxWebcamWidthFor`/`MaxWebcamHeightFor` keyed by canonical name) so the viewer sees the creator better -9. ✅ **"Add Webcam" always opens the camera picker** — deleting one scene's webcam then re-adding used to resurrect the old camera when another scene still used it — `SwapWebcamIdentityAsync` now swaps the app-wide identity if a different camera is chosen, same path as "Change Webcam…" -10. ✅ **Webcam resource validation + first-frame proof** — `MediaCaptureFrameSource` validates post-init (VideoDeviceId match, stream properties ≥1, `reader.StartAsync()` status read + throws on non-Success); subscribes `capture.Failed` + `CameraStreamStateChanged` → `SourceFailed` event on the seam; fallback ladder (VideoPreview → VideoRecord). `CameraManager.AcquireAsync` requires first-frame proof (4s timeout): returns true only after a real frame arrives — silent empty box impossible. `MainViewModel` subscribes `CameraFailed` → red `WebcamError` chip in preview + MessageBox names suspect apps (`CameraConflictProbe`). 19041 SDK projection gaps: `Exclusive`/`DeviceLost` not projected; `CameraStreamState.Failed` compared by `(int)2`. 81 tests passing -11. ✅ **Scenes/sources UI** — add/reorder/rename, image + background overlays with move/resize/opacity/reuse -12. ✅ **Audio UX shipped (UI)** — the bottom-bar footer is now two lines (dropped/duration moved under bitrate/fps), with the mic's sound meter + mute button + volume slider grouped CENTERED on the footer's top line, beneath the preview panel (meter: 288px, muted slate track with ruler graduations + muted yellow/red zone tints, green→yellow→red fill; mute = speaker icon → red do-not-symbol when muted, and the slider and speaker stay in sync (volume 0 ⇔ muted — sliding off flips the speaker to muted, sliding up from 0 clears it); mic volume defaults to 80%, muting zeroes the meter and restores the prior volume on unmute (which flashes the meter to the restored position ~300ms before it returns to the live level); the meter is a READ-ONLY realtime level display (fill = live level × volume — volume is a gain on ambient noise; while the slider is dragged the bar previews the slider position and bounces back to the live level on release, which is 0 with no input — clicking the meter does nothing), clicking the MIC label opens a microphone picker whose chosen source shows left-justified inside the meter bar (fill at 75% opacity so the name + ruler markings show through); slim dimensional slider — gradient track/fill, gloss-sphere thumb; the old flat pink 18px-filled one is gone), everything else on line 2 (bitrate/fps/dropped/duration/health left, quality + gear right) — the creator's only audio control, desktop/game audio is automatic (KISS rule). **Post-test follow-up (2026-08-13, game audio bar branch):** the **game audio bar** (desktop/game, a mirror of the mic bar: meter + mute + volume) is **overlaid at the bottom of the preview window** (bottom-center dark chip — monitoring UI only, never on the live output) and appears only while a **full-screen game is producing sound** in the preview (`IGameAudioDetector` seam + default `GameAudioDetector` polling `IFullScreenDetector` + the live loopback level, floor 0.5%, into a pure `GameAudioHysteresis`: SHOW after ~500ms of fullscreen+sound, HIDE ~1s after leaving fullscreen, **silence never hides an active bar**; the VM polls it on a 250ms `DispatcherTimer`); **capture now runs for the app's lifetime** (mixer started once at startup via `StartMicCaptureAsync`, disposed in `Shutdown` — not go-live) so the meters preview live; the **MIC label became a button with a status dot** (`Models/MicStatus`: green = connected via the source `Started` event, yellow = mic problem, red = no device); picking a mic takes effect immediately (`AudioMixer.RestartMic`, loopback keeps running); fixed a latent `?.Invoke(meter.Push(...))` short-circuit that skipped the meter update when nothing was subscribed. **Round 2 (2026-08-13):** the meters were dead on a flat scale (real speech/game RMS is ~0.01..0.1 linear) — the raw level is now mapped via `AudioLevelMeter.ToDisplay` (−60..0 dBFS spread across 0..1) so typical input reads ~1/3..2/3 of the bar at default volume. 169 tests passing, 0 warnings. **Round 3 (2026-08-13):** the mic bar gained a **mic mute icon** (a microphone glyph in the speaker's 16px style, red + slash when muted) between the meter and the speaker — both mutes adjacent with spacing between them, same `ToggleMicMuteCommand`; and the top-center LIVE badge became an **always-visible REC sign** — dark gray dot + dim "REC" offline, bright red (#e94560) + "REC" + elapsed while live, **darker red (#8f1f1f)** when live with a **private** stream (driven by the dialog's chosen visibility) — `RecDotBrush`/`RecTextBrush`/`RecDotOpacity`/`IsLivePrivate`, pulsing while live. **Queued:** task 22 (voice filters). Voice-filter note: the meter's `ToDisplay` input is the pre-filter mic level; when filters land they must apply BEFORE the meter/encoder mix -13. ✅ The connected YouTube account's avatar/name shows in the top bar next to Start Stream (`SyncConnectedAccount`); the scenes list is content-height now (no dead space before SOURCES) -14. ✅ **Social bar v2 (six-slot dialog, sign-in gate, real logos)** — global bar layer (never a Source, no sources-list row), content-sized, centered, GREEN glow when ON, top/bottom snap-drag (default BOTTOM, persisted `SocialBarPosition`; drag clamps to 0/1040, tie→bottom). Footer Social button gets a state dot (green=ON). Dialog "Social Media Site Promotion" (`SocialsDialog` + `ViewModels/SocialsDialogViewModel`, WPF-free + injected `ISocialValidator`/sign-in/sign-out fakes): ON/OFF bar switch (`SocialsConfig.BarEnabled`, schema v8), 6 fixed slots — row 1 always YouTube (signed-in → channel handle; signed-out → sign-in gate → OAuth; delete → confirm sign-out, mirrors `StopStream`), row 2 free, rows 3-6 lock icons on freemium (Premium seam: all six). Validation: `DetectService` (URL domain / fediverse `@user@domain` / bare→Website) → async `ISocialValidator` on confirm/Save; valid snaps to text + real service logo (bundled SVG path data via `LogoDataFor`, Simple Icons CC0 — initials badges gone); invalid → red do-not, stays editable, Save blocked. LCR justify dropped (`BarJustify` unread), per-scene toggle dropped (`Scene.HasSocialBar` back-compat). **Post-test fixes (2026-08-12):** footer label "Socials" (not "Social"); fediverse `@user@domain` validates — the full handle is the identity end-to-end (`DetectService`/`CanonicalUrlFor`/`HttpSocialValidator` build `https://domain/@user`, no domain loss); **Cancel is a hard stop** — `ISocialValidator.LookupAsync` takes a `CancellationToken`, dialog VM owns a CTS, Cancel/X/Save abort in-flight lookups (HTTP request killed, canceled continuations never touch slot state), and `ConfirmEdit` skips re-submitting identical text (LostFocus on dismiss never re-fires a lookup). `HttpSocialValidator` now has real tests (fake `HttpMessageHandler`). 105 tests passing. **Post-test fixes (2026-08-12, round 2):** fediverse `@user@domain` no longer shows a generic chain — it resolves to the instance's actual software via nodeinfo (`/.well-known/nodeinfo` → `software.name`; `SocialService.Fediverse` enum member + `SocialEntry.FediverseSoftware` persisted in a new `SocialEntry.Software` column, schema migration by column-presence) and renders that software's bundled logo (`LogoDataForFediverse`: mastodon/peertube/pixelfed/misskey/lemmy/pleroma/firefish, generic fediverse honeycomb fallback). Dialog row-2 edit/trash icons were too dark — `IconButton` style gains `Foreground=#d0d0d0`; trash overrides `#e94560` (app red). 112 tests passing. **Post-test fixes (2026-08-12, round 3):** a fediverse handle whose identity domain is itself a redirect (e.g. YunoHost default-app subdomains — `@user@llamachile.tube` where the mastodon instance lives at `mastodon.llamachile.tube`) now still resolves its software: nodeinfo on the identity domain is SSO-blocked, so `HttpSocialValidator` follows the bare root `https://domain/` 302 to the real instance host and re-runs the nodeinfo lookup there. -15. ✅ **Window capture** — absorbed into the Screen picker (no separate source type); dedicated window-as-source work is pending -16. ❌ **Scene compositing (D3DImage/MediaElement)** — SUPERSEDED, permanently out (2026-09-01): the software output compositor (TASK 4 step 1) + the XAML preview ARE the design; a D3D11 swap stays a possibility behind the `VideoFrame` seam, not a feature -17. ☐ **Text source** — live text ("Starting soon", "Back in 5", handle, callout) -18. ✅ **Chat box** — YouTube live chat rendered *on* the stream so viewers read along in-video — `ChatBoxRenderer` (WPF FormattedText → VideoFrame), configurable font size/color/badges/timestamps/max-messages via Elements panel, schema v10, persisted across save/load -19. ❌ **Background removal (milestone 2)** — ONNX Runtime + DirectML, MediaPipe Selfie Segmentation — deliberately NOT in this build -20. ☐ **Alerts** — Super Chat / membership / subscribe pop-ins; build after the six; **free and ungated** (the stale "one paid feature" label predates the 2026-08-31 monetization reversal and is corrected 2026-09-01 — the paid swap is the branding flash ONLY; see Monetization in `ai.md`). **v1 IN** per the complete-v1 ruling. **Precursor — reward-event capture (2026-09-01):** Alerts render from the app's canonical `RewardEvent` feed (all 7 `liveChatMessage` reward types, persisted to SQLite + the session report — see TASK 10 → "Related work — monetization awareness"), so it consumes that already-built data rather than re-integrating `liveChatMessages`. That capture is the required prior milestone. -21. ✅ **Show Desktop toggle** — `Source.ShowDesktop` bool persisted in DB (schema migration + LayoutStore read/write). When checked, primary monitor captures regardless of fullscreen game state. Toggling off falls back to running game or releases to static placeholder. `ClearBackdropCaptureAsync` clears `CaptureKey` + `VideoImageSource` so static fallback renders. Toggle on Live backdrop context menu (IsCheckable MenuItem). -22. ✅ **Top bar — status light + avatar + Log In button** — red/green/pulsing ellipse (disconnected/connected/live). Avatar border visible only when connected, loads via `BitmapImage` code-behind. "Log In" button visible when disconnected, calls `StartStreamCommand` → GoLiveWindow. `ShowStartStream` now requires `IsConnected`. -23. ✅ **Chat preview — one-at-a-time mock messages** — `RunMockChatPreviewAsync`: simple async loop, displays `MockChatMessages[n]` via `Take(n+1)`, 1000ms between messages, wraps at end. `ChatPreviewEnabled` property on Source (default true). Stops on real messages, restarts on fade timer clear. -24. ✅ **All backdrops seeded + EnsureDefaultBackdrop** — Live, Chat, Ending backdrops added (ending-backdrop.jpg replaced). `EnsureDefaultBackdrop()` called on every scene switch. `SeedEndingBackdrop` replaces existing (not skip-if-exists). Chat scene no longer skipped. -25. ✅ **SceneCatalog fix** — Chat scene display name corrected to "Chat" (was "YouTube Chat" — that name belongs to the layer/SourceType.ChatBox, not the scene). -26. ✅ **Left panel spacing + context menu reorder** — HR margins matched, Row 4 changed from `*` to `Auto`. Context menu reordered: Border Thickness first, then Opacity, Color, Effect. -21. ✅ **Logo + richer in-app About (2026-08-13, queued → 2026-08-14 SHIPPED)** — the ytLlive wordmark in the top bar opens the About overlay (already wired); the About overlay is now the **creator hub**: the real logo (the "llama fortnite superman logo" from the creator's vault, copied to `Assets/llama-logo.png` — the creator's own art, no third-party license), plus **creator-hub links** — llama chile shop on YouTube (`MainViewModel.ChannelUrl`), Mastodon (`https://mastodon.llamachile.tube/@gramps`), **Buy me a coffee** (`https://buymeacoffee.com/llamachiley` — live), and **Unlock Premium** (greyed "coming soon" — the **billing** product URL is a tabled seam, `PremiumUrl`, until TASK 10 picks the subscription provider). A **Licenses & legal** button flips the About overlay to an in-app scrolling panel that loads the full `THIRD-PARTY-NOTICES.txt` text (`MainViewModel.ShowLicenses` reads the shipped file from the executable directory on first open; graceful "not found" fallback — **never the OS viewer, everything stays in-app**); "← Back to About" returns. The About overlay is also the in-app home of the notices — the top-bar About button that opened the file in the OS viewer was removed on 2026-08-13 for exactly this reason. Integration test (Good Dog Rule — ONE): `AboutHubTests.About_Opens_InApp_Licensing_Loads_Shipped_Notices` drives the real window + VM, asserting the hub opens, the link URLs are real, the licensing panel loads the shipped notices text (contains "Third-Party Notices" + "LGPL"), and back returns to the hub. 174 tests passing, 0 warnings -22. ✅ **Voice filters on the mic channel (2026-08-13 queued → SHIPPED 2026-08-14 inside TASK 9, the audio milestone)** — the standard four applied to the sound input path (before the meter/encoder mix): **bass boost, treble, noise suppression, compressor** (set decided with the creator 2026-08-13). Noise suppression = a **pure-C# noise gate** (creator chose over RNNoise / a second ffmpeg `afftdn` pipe, 2026-08-14 — KISS). Always-on — no UI knobs; the mic stays the creator's single audio control -27. ✅ **Webcam row gates on the app-wide identity (2026-08-24, TASK 26)** — the (+) menu's Webcam item greys out whenever a webcam exists **anywhere** (`CanAddWebcam` = `StagedScene != null && _webcam == null`; renamed from `CanAddWebcamToStagedScene` whose per-scene rule let a second picker run from a scene lacking the config), raised at both `_webcam` mutation sites (create / last-config removal) + scene staging + elements change. Creator's visual pass found it: minis/no-capture-rows/capture-controls all good. ONE integration test `WebcamMenuGateTests.CanAddWebcam_Gates_On_The_AppWide_Webcam_Identity` (real window + temp DB seeded with a webcam in Starting; asserts greyed while Live staged, re-enabled after `RemoveSourceCommand` clears the last config + the `Webcam` DB row). Stale map fixed in the same commit: ai.md's "empty-canvas right-click Show Webcam" claim dropped — that XAML never shipped (`ShowWebcamCommand`/`CanShowWebcamInStagedScene` are wired but unbound dead code, audit item). 224 tests (223 pass; the pre-existing AudioPipelineTests failure is unrelated) -28. ✅ **YouTube Chat layer: one-per-layout gate + legacy label heal (2026-08-24, TASK 27)** — same rule as TASK 26 applied to the chat layer: the (+) menu's YouTube Chat item greys out while any scene carries a ChatBox source (`CanAddYouTubeChat`, raised on staging + elements change; `AddSource` refuses a second), tooltip "One chat layer at a time — it's already in your stream". Plus the creator's label fix: layers added by commit `65641d8` were named "Chat"; LoadLayout now heals exactly that un-renamed default to "YouTube Chat" so the Layers row matches the (+) picklist (creator renames untouched, idempotent). ONE integration test `ChatLayerGateTests.LegacyChatName_Heals_And_CanAddYouTubeChat_Gates_On_The_Existing_Layer` (real window + temp DB with a legacy "Chat" row: healed on load + persisted on save, greyed cross-scene and in-scene, re-enabled after remove). 225 tests (224 pass; pre-existing AudioPipelineTests failure unrelated) -29. ✅ **Broadcast metadata pull-out + launch geometry (2026-08-24)** — creator ask: "a tab-pullout on the right side of the preview pane — white tab, YouTube-red label reading 'Text' — opening a form with all settable liveBroadcast fields; fields that can't be filled before launch greyed out; saving/updating remote content; field data saved and pre-loaded on app run". Shipped as specified with one correction: the API is the inverse of "more fields once live" for contentDetails (those lock in created/ready) — the pull-out carries the always-editable snippet/status set: Title, Description, Tags (csv), Visibility (private/unlisted/public), Made-for-Kids, plus read-only Scheduled Start. **Always visible** (creator revised same day from Live-only gating). White 30px tab, rotated red "Text", right edge vertically centered; click slides a dark 320px drawer left over the preview (200ms CubicEase). Every edit persists to Settings keys (`Broadcast.*`) immediately; **Update Broadcast** button calls the new `YouTubeStreamService.UpdateBroadcast` (PUT `liveBroadcasts?part=snippet,status`, echoes scheduledStartTime because update replaces the whole snippet part). Go Live prefills from the form and captures what was inserted (`CaptureGoLive`). The old un-persisted Default Stream Title/Description fields came OFF the App Settings overlay (replaced by the form); `DefaultStreamTitle/Description` VM properties deleted. **Launch geometry** (same unit): default 1920×1040, MinWidth 1366, MinHeight 768; `WindowStartupLocation=Manual` + window size/position persisted on close via `RestoreBounds` (maximized-safe), restored in ctor clamped to minimums and the primary work area (disconnected-secondary fallback). ONE integration test `BroadcastPullOutTests.Metadata_Persists_WindowRestores_Clamped_And_UpdatePatchesRemote`. 228 tests (227 pass; pre-existing AudioPipelineTests failure unrelated) -30. ✅ **Gear menu cleanup + Default Location for Recordings + single Start button (2026-08-29)** — **KISS: removed** Gear-menu `Save Layout`/`Save Layout As…`/`Open Layout…` (redundant with auto-save; multi-layout = OBS) + dead `SaveLayoutCommand`/`SaveLayoutAsCommand`/`OpenLayoutCommand` + `SaveLayoutAs()`/`OpenLayoutFile()` methods. **App Settings panel:** first real setting — `Default Location for Recordings` (configured folder via `ChooseRecordFolderCommand` + "Use Downloads" reset via `ResetRecordFolderCommand`, resolved path shown). Fallback chain: configured → `%USERPROFILE%\Downloads` → `MyVideos`. `StartRecordFile()` + `ChooseRecordFolder()` `InitialDirectory` updated to use the new fallback. **Single Start button:** `PrimaryButtonText`/`IsStreamingStart` + their `OnPropertyChanged` calls removed; button `Content="Start"` in XAML — pills are the intent surface, button is always just Start. `Choose Record Folder` fast-path kept in the button context menu. Monetization (About links/premium) deferred to a later task. ONE fallback test `DefaultRecordFolder_Falls_Back_To_Downloads`. build 0 warnings, 247 tests (2 known failures, pre-existing). - -### The Minimal Source Set (design decision — do not expand casually) - -ytLlive is YouTube-only and 90% of users are casual. OBS's long source list is off-putting; we ship -the hot few and nothing esoteric. If a user needs more, they've graduated to OBS. - -1. **Webcam** — the face cam. Non-negotiable. -2. **Screen** — the main event (game, slides, browser). One source; a picker chooses a monitor *or* a - window. (Window capture is absorbed here — no separate source type.) -3. **Background** — a full-canvas backdrop image. Fills the whole scene automatically, zero fiddling. - Kept separate from Image on purpose: same pixels, but this one needs no positioning. -4. **Image** — a floating graphic/logo overlay (watermark, badge, corner branding). Free-positioned. -5. **Text** — live text ("Starting soon", "Back in 5", handle, callout). Casual streamers live on this. -6. **Chat box** — YouTube live chat rendered *on* the stream so viewers read along in-video. YT-native. -7. **Alerts** — Super Chat / membership / subscribe pop-ins. The dopamine source. **The one big lift** - (Super Chat event streaming + on-stream rendering/animation); build after the six. **Free and - ungated** — paid = flash removal only (stale "one paid feature" corrected 2026-09-01). Consumes - the canonical `RewardEvent` feed (reward-event capture, TASK 10 → related work). - -Deliberately NOT supported: game capture, browser source, media playlist, VLC, color-key voodoo, MIDI. - -### Source memory model (design decision) - -1. A scene has resources. Resources can be shared across scenes. -2. A resource exists exactly once in memory, no matter how many scenes use it (a logo in five - scenes = one loaded bitmap). -3. Every resource carries a **catalog of scenes**: one usage entry per scene it appears in, each - entry dictating that scene's use — placement (X/Y/Width/Height), opacity, z-order, enabled, - scale mode, crop. -4. Usages are named `{resourceName}.{sceneName}` — whatever the user named the resource, dot, the - scene name: `logo.starting`, `logo.live`, `myPic.brb`. Not a hardcoded "logo". -5. A webcam in two scenes = one capture session, two catalog entries. -6. Refcount by catalog size: the last usage removed → the resource is disposed and evicted. -7. The resource (not a per-scene node) owns everything `IDisposable`. - -### Scene transitions (design decision) - -Scene switching while live must never stutter. Supported types, most → least economical: - -1. **Cut** — instant switch. The default. Zero cost. -2. **Fade** — short crossfade (~300ms). -3. **Move** — a simple, economical move transition, done to perfection and memory-efficient. The - smart streamer's bread and butter. -4. **Custom (media) transitions** — require media elements (video/stinger playback during the - transition). Heavier, but creators pay for these, so we support them. Their media follows the - same resource memory model: loaded once, catalogued by scene. - -Preview shows the transition too (WYSIWYG). No wipes/slides/LUTs beyond the four above. - -### Requirements: - -1. **Screen** — Windows.Graphics.Capture (WinRT), enumerate displays/windows, picker -2. **Webcam** — MediaCapture (WinRT SDK projection) with device enumeration — ✅ **milestone 1 done**: - 1. TFM bumped to `net8.0-windows10.0.19041.0` (app **and** tests) so the WinRT projection resolves from the SDK reference packs — no NuGet package, no capability manifest (unpackaged desktop app) - 2. `MediaCaptureFrameSource` (CPU-first: `MemoryPreference = Cpu`, BGRA8 via `CreateFrameReaderAsync`), `MediaCaptureCameraEnumerator` (`DeviceInformation.FindAllAsync(DeviceClass.VideoCapture)`) - 3. `CameraManager`: refcounted by `DeviceId`, one shared `WriteableBitmap` app-wide, dispatcher-coalesced UI updates (~render rate, latest-frame drop), placeholder/`AppLog` + warning on failure - 4. `CameraPickerDialog` (mirror of `ReuseImageDialog`) — "Searching for cameras…" / list / "No cameras found" states - 5. One webcam app-wide: Add → Webcam greyed out once one exists ("it's already in your stream" tooltip); persisted `DeviceId` re-acquires after layout load - 6. Default placement 16:9 **480×270**, bottom-right, 32px margin; drag/resize/selection shared with Image sources - 7. **Clip shapes: Traditional + Round** (phone view dropped — the 9:16 phone output is the vertical output-crop tier); **mirror**; both persisted in the layout DB (schema v2) and toggled from the source chip - 8. **Background removal = milestone 2** (ONNX Runtime + DirectML, MediaPipe Selfie Segmentation) — not in this build -3. **Background / Image / Text** — static sources positioned/scaled/opacity -4. **Chat box** — rendered from the live chat poll (right panel is the same feed, raw) -5. **Scene compositing** — per-scene source layering (z-order = sources list order, top-to-bottom - back-to-front), preview rendered via D3DImage or MediaElement -6. **Branding flash** — the topmost full-frame "made with ytLlive!" layer at ~25% opacity, ~1s on / - 300s off (see Monetization in `ai.md`), gated on `BrandFlashEnabled` + live/recording. Lives in the - preview compositor now (`BrandFlashLayer` in `MainWindow.xaml` CanvasGrid, driven by - `BrandFlashActive`/`BrandFlashTimer` in `MainViewModel`); the encoder output renders the same layer, - and v0.2 local recordings carry it too -7. **Drag/drop placement & reorder** — intuitive, visual (per design principle): - 1. **Preview:** click-drag a source in the center panel to reposition it; resize via handles - 2. **Scenes list:** drag rows to reorder scenes - 3. **Sources list:** drag rows to reorder sources (this *is* the z-order) — implemented +See [`TASKS/research-youtube-api.md`](TASKS/research-youtube-api.md) — authoritative facts for v3 build. --- ---- +## Out of product — permanently -## TASK 4 — RTMP Ingest to YouTube +These were explicitly ruled out: -**Goal:** Push encoded video to YouTube's RTMP ingest. - -### Status: ✅ Done — all 7 items shipped - -1. ✅ **Ship step 1 — the output compositor SHIPPED** (2026-08-10) -2. ✅ **Ship step 2 — the FFmpeg locator SHIPPED** (2026-08-10) -3. ✅ **Encoder + RTMP push SHIPPED** (2026-08-12) — the FFmpeg subprocess: raw BGRA frames via stdin, stderr health parsing, FLV mux + push to the ingestion URL (see the ship step 3 plan below) -4. ✅ **WASAPI audio capture SHIPPED** (2026-08-12) — NAudio loopback (desktop/game) + the picked mic feeding `AudioLevel`, so the realtime meter comes alive (see the ship step 4 plan below) -5. ✅ **Frame-pipeline wiring SHIPPED** (2026-08-12) — `CameraManager`/`ScreenCaptureManager` → compositor resolver → encoder, driven by a paced `FramePump` (see the ship step 5 plan below) -6. ✅ **Health stats SHIPPED** (2026-08-13) — `FramePump.HealthUpdated` (encoder's parsed bitrate/FPS/dropped/duration, already forwarded from `FfmpegEncoder.OnStderrLine`) now lands in the bottom bar: `MainViewModel.OnFramePumpHealthUpdated` marshals to the UI thread (the stderr loop raises on a background thread) and copies into `CurrentHealth` (the bottom bar's existing binding); `ResetHealth` zeroes dropped/duration on go-live and on End so stats never linger from a previous session (bitrate/FPS stay on the tier's targets). The bar lights up with real values once TASK 9 supplies the RTMP URL (until then the pump skips the encoder and the bar shows the tier's targets) -7. ✅ **One-click go live + private-only enforcement SHIPPED** (2026-08-14) — the Go Live dialog is **locked to Private** (no dropdown, `GoLiveViewModel.Visibility` is a get-only "Private"); `YouTubeStreamService.CreateBroadcast` **always sends `privacyStatus = "private"`** (dialog + service enforcement, requirement 8 — nothing can go out non-private) and gained an injectable `HttpClient? http = null` seam for tests; `BeginGoLive` now calls `CreateBroadcastAsync` and remembers `_currentBroadcastId` for TASK 9's bind/transition (failure → `StreamStatus.Error`, never a crash; `StopStream` clears the ID); the REC sign shows a **PRIVATE badge** (dark-red border, next to REC, `IsLivePrivate`) when the live stream is private; settings' dead "Default Visibility" dropdown + `MainViewModel.Visibilities`/`DefaultStreamVisibility` removed. The broadcast-insert integration test asserts the request body carries `"privacyStatus":"private"` (2 new tests → 173 passing, 0 warnings) - -The pipeline chain the encoder needs doesn't exist yet: **scene compositing** (the master 1920×1080 frame -without the preview's editing chrome) → **audio capture** (WASAPI, feeds the meter) → **H.264+AAC encode** -→ **vertical-tier crop/scale** → **RTMP push** → **health stats** into the bottom bar. Nothing can encode -until a frame source exists, so the compositor is ship step 1. The pipeline is -`CameraManager + ScreenCaptureManager → compositor resolver → compositor → encoder → RTMP`. - -### Requirements: - -1. **Encoding** — H.264 (hardware via NVENC/AMD, fallback x264) + AAC audio; **must comply**: keyframes ≤ 4s (gopSizeLong), closed GOP, AAC/MP3 @ 44.1/48kHz, mono/stereo only. **License posture (decided): GPL-free build** — NVENC (NVIDIA) / QSV (Intel) / AMF (AMD) + OpenH264 software fallback + built-in AAC; no libx264 (GPL contaminates a paid product). Output containers are identical either way (H.264+AAC in `.flv` for RTMP, `.mp4`/`.ts` for VOD) — the format is NOT the differentiator, the license and per-GPU quality are. **License guardrails (never violate — see `ai.md` → "Licensing — do not violate"):** only BtbN `lgpl`/`lgpl-shared` builds; never GPL (gyan.dev) or `nonfree` (fdk-aac); never static for distribution (LGPL §6 relink material); never link FFmpeg into the app; never drop `THIRD-PARTY-NOTICES.txt` from the app/About screen. -2. **RTMP push** — **FFmpeg subprocess (decided)**: app feeds raw frames via stdin, parses stderr for health; one battle-tested binary does encode + FLV mux + push. **No self-heal (claim-check 2026-09-01): ffmpeg's reconnect flags are input-side; an RTMP *output* push does not reconnect itself — retry is app-side, owned by TASK 32.** **Binary distribution (decided): check-then-pull** — probe `where ffmpeg`/PATH at first go-live; if absent, download a **pinned** build (**BtbN LGPL-shared win64** zip, ~75 MB — gyan.dev's builds are GPLv3 and ship libx264, which violates the license posture; BtbN's LGPL variant drops x264/x265 while keeping NVENC/QSV/AMF + libopenh264 + native AAC) to `%APPDATA%\ytLlive\tools\ffmpeg.exe` (extract `ffmpeg.exe` **plus the `libav*.dll` family**) and cache it, offline-friendly. Behind an `IFfmpegLocator` seam so tests fake it (ship step 2, below). Push goes to the cached reusable stream's ingestion URL -3. **Quality ladder** — the offered tiers, with **1080p60 @ 8 Mbps as the standard/default**: - 1. 720p30 @ 6 Mbps - 2. 720p60 @ 6 Mbps - 3. 1080p30 @ 8 Mbps - 4. **1080p60 @ 8 Mbps** (default — mainstream ceiling, GPU hardware-encoded so the gaming - machine never notices; upload headroom stays comfortable) - 5. Vertical 1080×1920 @ 60fps @ 8 Mbps (9:16 phone tier) - The composition master is always 1920×1080; a tier is an output rect + target resolution - (see `ai.md` "Resolution tiers"). Vertical output = the centered 607×1080 crop of the master - scaled to 1080×1920 (semi-crop preview is already implemented; the encoder applies the same rect). - 1080p60 is the ceiling by design — "if you want 1440 or 4K or 8K → OBS is your solution"; the app - targets the most mainstream creator, not power users. - Ladder is sculpted by a **cached probe** (IP-only TCP vs public ingest host; no auth required). - Quality is greyed out while live because the declared resolution can't change mid-stream — but - with `variable`, we can **auto step-down** bitrate/resolution on the fly with zero API calls - (no stream recreation); 60fps presumes a hardware encoder — no hardware encoder → auto - fallback to 720p60/1080p30 -4. **Stream key management** — reuse the cached reusable stream (one per channel) instead of creating a new one per go-live; prefill default YouTube ingest URL `rtmp://a.rtmp.youtube.com/live2` -5. **Health stats** — bitrate, FPS, dropped frames reported live in the bottom bar (encoder-side) -6. **One-click go live** — defaults that work out of the box -7. **Audio capture (feeds the meter — this task ships the wiring)** — WASAPI loopback (desktop/game at unity, zero UI — "it just is") + the picked mic (`MicSourceName` from the `MicPickerDialog`). The mic capture feeds `AudioLevel` so the realtime meter comes alive (today it reads 0 — the mixer feed is pending, see `ai.md` audio notes). AAC mono/stereo @ 48 kHz per the compliance rules. -8. **Private-only go live until v1 (reputation guard, decided 2026-08-10)** — until the v1 release, go-live is **locked to private streams only** so a software error can never publish something public/unlisted that damages the creator's reputation. RTMP push itself has no privacy — privacy lives on the YouTube **live broadcast object**, which this app already controls via its OAuth API calls. So the lock is purely API-side: the Go Live flow always creates/updates the broadcast with `privacyStatus = "private"` and a guard **refuses** to set anything else (same spirit as the Live-only backdrop policy). The UI shows a clear "PRIVATE" badge next to the stream state so the creator always knows who can see them. Enforcement must be verifiable in the auth-service tests (fake the broadcast-insert/update call, assert `privacyStatus` is forced to private). -9. **v1 release gate: bundle the full license texts (decided 2026-08-10)** — `THIRD-PARTY-NOTICES.txt` currently links the canonical license texts rather than embedding them. At the **v1 (GA) release**, the full texts of every license it names (LGPL v2.1+, BSD-2-Clause, MIT, Apache-2.0) MUST be bundled alongside it (shipped in the app output, e.g. a `licenses/` folder next to the notices file, still reachable from the About screen). This is a **release blocker for v1, not a task to queue early** — do it in the release pass. The repo should treat this like the private-only go-live gate: a checkbox that cannot silently lapse. - -#### Ship step 1 — Scene compositor (the frame source) - -**Goal:** a pure-CPU software compositor producing the encoder's master frame (BGRA8, the `VideoFrame` -seam) from the scene model. The preview stays XAML (the editing view); the compositor is the **output -view** — WPF's `RenderTargetBitmap` can't be used (software-rendered + captures chrome). Two renderers -must agree, so the XAML (`MainWindow.xaml` CanvasGrid + element DataTemplate) is the contract. - -**Decisions (locked 2026-08-10):** **Path A CPU blitter** — GPU effort belongs to NVENC (the encoder), -not composition; with an FFmpeg subprocess the master crosses a CPU readback to the pipe every frame -anyway, so GPU compositing buys ~nothing at this layer count (2-3 live layers; static layers -pre-composite once). A D3D11 compositor can replace this one later **behind the same seam** (the CPU -master buffer stays the contract). **Render the output rect directly**: compositor is constructed with -`CompositorOptions {SourceRectX/Y/W/H, OutputWidth, OutputHeight}`; 16:9 tiers = full 1920×1080 1:1; -vertical (9:16) = composite the centered 607×1080 crop then bilinear-upscale to 1080×1920. Reuses -`MainViewModel.OutputRectX/Y/W/H` (note `(1920−607)/2 = 656.5` → align to integer pixels for output). - -**Render spec (back → front, mirror the XAML exactly):** -1. Backdrop — the Live scene's `IsBackdrop` Source (`CaptureKey` → live frame), `UniformToFill` - full-frame (XAML's separate `BackdropImage` layer; the backdrop *element* renders nothing — its - DataTemplate Image is Collapsed for DisplayCapture). -2. Background — the scene's `Background` Source, `UniformToFill` full-frame (the `ActiveBackgroundImage` - layer, not per-element). -3. Elements in `Scene.Elements` order (back→front), skip `IsVisible=false`. What actually renders: - 1. `Source` Type `Image` → static asset, `UniformToFill` cover-crop into (X, Y, W, H) - 2. `WebcamSceneConfig` → latest frame by `DeviceId`: Traditional = `UniformToFill` rect; Round = circle - diameter `min(W,H)` (alpha 0 outside — true circle, not oval); mirror = horizontal flip around - element center (`MirrorScale`); opacity = per-pixel multiply (content + border); border = stroked - rect / centered circle at `RoundBorderSize`, width `BorderWidth`, alpha `BorderOpacity` - 3. `Background` / `IsBackdrop` / `TextOverlay` are NOT per-element (layers above; Text not shipped) -4. Branding flash — pre-rendered full-frame "made with ytLlive!" at 25% alpha when live + - `BrandFlashEnabled` + timer active. Passed in as a `VideoFrame?` (compositor core stays pure byte-math, - no WPF; likely a bundled asset rather than runtime text rendering). -5. NOT in output (preview chrome only): SelectionOverlay, DimRects, output-rect outline, badge, placeholder. - -**New files (all in `Services/Compositor/`):** -1. `SceneCompositor.cs` — `Render(Scene, frameFor: Func, flashFrame: - VideoFrame?, CompositorOptions) → VideoFrame` (output-sized). The caller's `frameFor` resolver maps - each element to its frame (webcam → DeviceId, image → AssetId via `StaticPixelCache`, backdrop → - CaptureKey) — the compositor stays pure/hermetic/no WPF. -2. `CompositorOptions.cs` — source-rect + output W×H. -3. `StretchMath.cs` — `UniformToFill` cover-crop, ellipse mask, bilinear scale (pure, unit-tested). -4. `StaticPixelCache.cs` — asset `byte[]` → cached BGRA `VideoFrame` (WPF `BitmapDecoder` + `CopyPixels`, - decode once per content hash). - -**Test plan (Good Dog Rule — ONE integration test):** `SceneCompositorTests` — a scene with backdrop -(solid red fake frame) + round webcam (solid green) + image (solid blue) → render 16:9 master → assert -per-layer probe pixels (corner = backdrop color, element center = webcam color, outside the round clip = -backdrop color, mirrored element swaps left/right); a vertical-tier variant asserts 1080×1920 output + -crop fidelity. Focused unit tests on `StretchMath`. Tests push frames directly — no capture managers -involved (they wire in a later step). - -**Same-PR housekeeping:** fix the stale comment `MainViewModel.cs:324` ("shown under the meter on line 2" -→ "shown left-justified INSIDE the meter bar" — `ai.md` is the authority); this task's requirements now -include the explicit audio-capture/meter wiring (#7 above). - -**Out of scope (later ship steps):** FFmpeg locator + license posture (covered in requirements 1-2), -encoder + RTMP push, WASAPI audio capture (loopback + mic) feeding `AudioLevel`, wiring -`CameraManager`/`ScreenCaptureManager` into the frame pipeline, brand-flash timer wiring, health stats -(bitrate/FPS/dropped). - -**Built (2026-08-10):** all four files shipped in `Services/Compositor/`, `SceneElement.TryGetBorderColor` -made public (shared hex parse with the compositor — no duplicated color parsing), the stale -`MainViewModel.cs:324` comment corrected, and the pre-existing CS1998 in `YouTubeAuthServiceTests` -cleaned up — build **0 warnings**. Tests: the `SceneCompositorTests` integration test (full-scene master -pixels, vertical tier, flash) + 4 `StretchMath` units — **72 passing**. - -#### Ship step 2 — FFmpeg locator (the encoder's binary) - -**Goal:** resolve a usable `ffmpeg.exe` on demand (the encoder's one external dependency), never shipping -a binary in the repo. Returns an absolute path; downloads only when neither PATH nor the local cache -provides one. - -**Decisions (locked 2026-08-10):** -1. **BtbN LGPL-shared win64 build** — not gyan.dev (gyan's "essentials" is GPLv3 and ships libx264, which - violates requirement 1's license posture) and **not the static lgpl build**: LGPLv2.1 §6 wants - relinkable object files for static linking, but the **shared** (dynamic-DLL) variant sidesteps that — - compliance is "license text + source offer + unmodified binaries" (see `THIRD-PARTY-NOTICES.txt` and - `ai.md` → Licensing). Drops libx264/libx265 while keeping NVENC/QSV/AMF, libopenh264 (the LGPL-legal - H.264 software fallback) and native AAC — exactly the requirement-1 encoder profile. -2. **Pinned URL** — **RE-PINNED 2026-09-01**: `https://github.com/BtbN/FFmpeg-Builds/releases/download/autobuild-2026-08-31-13-27/ffmpeg-N-126342-gf88b741dbf-win64-lgpl-shared.zip` - (BtbN **lgpl-shared** variant, ffmpeg N-126342; ~75 MB zip — earlier "~30 MB" estimate corrected). A dated autobuild tag is immutable; BtbN retention - keeps the last 14 daily builds + each month-end build for 2 years, so a cold cache after retention - expiry 404s — a logged, recoverable failure (the seam throws; the encoder step surfaces it). The - first pin (`autobuild-2026-08-09-13-03`, a daily) aged out on 2026-09-01 — the first real recording - attempt — proving the rule; the new pin is deliberately the **month-end** build (2-year retention). - Dated tags carry versioned asset names (`ffmpeg-N-…-win64-lgpl-shared.zip`), extraction is - name-agnostic; download failures now wrap in `IOException` with an actionable message - ("refresh `FfmpegLocator.PinnedUrl` or install ffmpeg on PATH") instead of leaking a raw 404. Once - cached, the URL is never touched again. The pin is a single `const`, bumpable in one place — and must - always stay on the **shared** variant (never `gpl`, `nonfree`, or static; see ai.md Licensing). -3. **Check-then-pull order** — (1) PATH probe (the user's own install wins), (2) cached - `%APPDATA%\ytLlive\tools\ffmpeg.exe`, (3) download + extract. Extract `ffmpeg.exe` **plus the - `libav*.dll` family** (the shared build's bin/ folder; Windows resolves the DLLs from the exe's own - directory) into a staging dir then move into place — a crash never leaves a corrupt or partial cache. -4. **Seam** — `IFfmpegLocator.LocateAsync(CancellationToken)`: search dirs, tools dir, and the downloader - (`Func>`) are constructor-injected with production defaults, so - tests fake the network (feeding a real in-memory zip) and never touch disk outside a temp dir. - -**New files (all in `Services/Encoder/`):** -1. `IFfmpegLocator.cs` — the seam. -2. `FfmpegLocator.cs` — the impl (PATH probe → cache → pull+extract exe + DLLs), failures logged via `AppLog`. -3. `THIRD-PARTY-NOTICES.txt` (repo root) — the LGPL/BSD/MIT notices + source offer, copied to the build - output. *(Surfacing changed on 2026-08-13: the top-bar About button that opened the file in the OS - viewer is GONE — the notices are reachable in-app via the logo → About overlay instead.)* - -**Test plan:** the hermetic integration test drives the full decision ladder against a temp tools dir and -a fake downloader returning a real in-memory zip (`.../bin/ffmpeg.exe` entry): PATH hit wins without -downloading, cache hit skips the network, cold cache downloads → extracts → `ffmpeg.exe` lands in the -tools dir, and a second call serves the cache (downloader invoked exactly once). Focused unit tests: -**shared-build DLLs extract alongside the exe**, empty zip throws, missing entry throws, empty download -throws, downloader failure propagates, zero-byte cache is refreshed. - -**Same-PR housekeeping:** requirement 2's stale binary facts corrected in this plan (~30 MB → ~75 MB zip; -"gyan.dev/BtB N" → BtbN LGPL-shared only, with the why); the "never do" licensing guardrails recorded in -`ai.md` so the reasoning survives. - -**Out of scope (later ship steps):** the FFmpeg subprocess encoder (frames in via stdin, stderr health -parsing), RTMP push, WASAPI audio capture, the frame-pipeline wiring, health stats. - -**Built (2026-08-10):** `IFfmpegLocator` + `FfmpegLocator` shipped in `Services/Encoder/`, pinned to the -**lgpl-shared** build `autobuild-2026-08-09-13-03` (extracts `ffmpeg.exe` + the `libav*.dll` family via a -staging dir). `THIRD-PARTY-NOTICES.txt` (repo root) ships to the build output; the "never do" licensing -guardrails are recorded in `ai.md` — build **0 warnings**. -Tests: the hermetic `FfmpegLocatorTests` integration test (PATH → cache → download decision ladder with a -fake downloader serving a real in-memory zip) + edge/unit cases (shared-build DLL extraction, zero-byte -cache refresh, empty payload, missing zip entry, downloader failure) — **78 passing**. - -#### Ship step 3 — Encoder + RTMP push (the FFmpeg subprocess) - -**Goal:** encode raw BGRA master frames into H.264+AAC FLV and push them to the reusable stream's RTMP -ingestion URL — one battle-tested subprocess doing encode + mux + push (retry is app-side, TASK 32 — ffmpeg does not self-reconnect an RTMP output), the app feeding -frames via stdin and parsing stderr for health (req 2). - -**Decisions (locked):** the encoder is a thin orchestrator over `ffmpeg.exe` — no H.264/AAC code in the -app. Arguments (pure `FfmpegArgs.Build`): `-re -f rawvideo -pix_fmt bgra -video_size WxH -framerate FPS --i pipe:0` (frames in), a **silent placeholder audio track** via `-f lavfi -i anullsrc` (the WASAPI -capture step replaces this input), `-c:v -b:v K -maxrate K -bufsize 2K` + **`-g fps×4` -`-keyint_min fps×4` `-sc_threshold 0` `-bf 0` `-pix_fmt yuv420p`** (the keyframe ≤4s / closed-GOP / -H.264 compliance), `-c:a aac -ar 48000 -ac 2`, `-f flv `. Encoder choice is **probed from the -binary's `-encoders` listing** (`FfmpegEncoderPicker`, pure): hardware NVENC → QSV → AMF, then OpenH264 -software fallback — **never libx264** (GPL; see `ai.md` → Licensing). The seam (`IFfmpegEncoder` + -`IEncoderProcess`, constructor-injected locator + process factory) keeps it hermetic — tests fake the -whole subprocess (probe + encoder), no real binary. - -**Behavior:** `StartAsync` (locate → probe → spawn → stderr loop), `SubmitFrameAsync` (serialized BGRA -stdin writes, ~2 Hz health via `HealthUpdated`/`StreamHealth` — bitrate/FPS/duration/dropped-from-frame- -count), `StopAsync` (stdin EOF → ffmpeg finalizes + exits by itself; 10s watchdog kill), `ProcessFailed` -on a non-zero unexpected exit. - -**Built (2026-08-12):** `EncoderOptions` + `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/ -`FfmpegEncoderProcess` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` in -`Services/Encoder/`. Not yet constructed by the app (the frame-pipeline wiring, ship step 5, owns it). -Tests: `FfmpegEncoderTests` integration (probe → spawn with NVENC preferred → frames into stdin → -progress parsed → graceful stop, no kill) + units (args compliance/GOP, progress parser, picker -preference + GPL guard, no-URL/not-running/noop stops, process-death `ProcessFailed`) — **122 passing**. - -#### Ship step 4 — WASAPI audio capture (the meter comes alive) - -**Goal:** capture desktop/game audio (loopback) and the picked mic, feed the mic level into `AudioLevel` -so the realtime meter reads something other than 0, run capture **only while live** (req 7). - -**Decisions (locked):** -1. **NAudio `NAudio.Wasapi` 2.2.1** — the wasapi feature package (not the `NAudio` meta-package): it - carries the capture types (`WasapiCapture`/`WasapiLoopbackCapture` + the MMDevice enumeration) with - `NAudio.Core`/`NAudio.Asio` pulled in transitively. MIT — recorded in `THIRD-PARTY-NOTICES.txt` (item 9). -2. **`IAudioSource` seam** (`Start`/`Stop`/`SampleReady`/`Failed`, IDisposable) — the app consumes the - seam; the two WASAPI implementations wrap NAudio; tests inject hermetic fakes (no real audio devices, - no timers). Loopback = `WasapiLoopbackCapture` on the default render device; mic = `WasapiCapture` - with the NAudio device resolved by `FriendlyName` matching `MicSourceName` (the app only persists the - DisplayName), falling back to the default capture endpoint. Mic device resolution is re-read at each - `Start` via a name provider so a mic picked mid-session takes effect next go-live. -3. **`AudioMixer` owns both sources** — starts/stops both with go-live (`BeginGoLive` success → `Start`, - `StopStream` → `Stop`). Mic samples feed a pure `AudioLevelMeter` (RMS, exponential smoothing) and - raise `MicLevelChanged`, marshalled to the UI thread into `AudioLevel`; desktop samples are currently - dropped (consumed by the encoder's AAC mix in a later step). Capture failures are logged via `AppLog` - (mic failure also zeroes the meter); loopback failure doesn't kill the mic. -4. **Byte→float** — pure `WaveToFloat.Convert` handles the WASAPI mix formats: IEEE float 32-bit (direct) - and PCM 16-bit (normalized to -1..1), including `WaveFormatExtensible` with the IEEE-float subformat - GUID. Trailing partial samples are ignored. - -**Built (2026-08-12):** `Services/Audio/` ships `IAudioSource` + `AudioSample`, `WasapiLoopbackAudioSource`, -`WasapiMicAudioSource`, `AudioMixer`, `AudioLevelMeter`, `WaveToFloat`; `MainViewModel` constructs the -mixer (mic source fed `() => MicSourceName`), starts it on go-live and stops it on end-stream, and maps -`MicLevelChanged` → `AudioLevel`. A pre-existing CS8602 in `FfmpegEncoder.cs:139` surfaced during this -step's rebuild and was fixed (`process!`) — build **0 warnings**. Tests: `AudioMixerTests` (mixer -lifecycle/forwarding/failure against fakes, meter RMS/smoothing/reset, `WaveToFloat` float/PCM16/ -extensible/truncation) — **139 passing**. - -**Deferred (later ship steps):** wiring the desktop-capture samples into the encoder's AAC mix (replaces -the `-f lavfi -i anullsrc` placeholder; the encoder construction itself shipped in ship step 5). -*(The "capture while not live" + "audio UI beyond the mic controls" deferrals were SHIPPED on the -2026-08-13 game audio bar branch — capture is now always-on for preview and the game bar is the second -audio UI. The mixer's short-circuit meter fix + `Started`/`RestartMic` seams live in the same branch.)* - -#### Ship step 5 — Frame-pipeline wiring (the encoder gets a frame source) - -**Goal:** the chain `CameraManager`/`ScreenCaptureManager` → compositor resolver → encoder, driven while -live by a paced frame pump: snapshot the active scene → resolve each element to its latest frame → -composite into the tier's output frame → pace into the encoder's stdin at the tier's FPS. - -**Decisions (locked via user Q&A, 2026-08-12):** -1. **Video pipeline first** — the `-f lavfi -i anullsrc` silent track stays; mixing the loopback/mic - WASAPI samples into the encoder's AAC track is its own later step. -2. **RTMP URL via a provider seam** — `MainViewModel._rtmpUrlProvider` is a `Func` returning - null today (the reusable stream's ingest URL lands with TASK 9); when it yields null the pump logs - and skips the encoder entirely, so go-live runs the existing visual flow without pushing. - -**Design:** -1. `Services/Encoder/FramePump.cs` — the frame producer. All collaborators constructor-injected seams - (`Func`, `Func` resolver, `Func`, - `Func`, `Func`, `Action` log, injectable pacing delay) so it - stays free of WPF and of the capture managers and is hermetic in tests. `StartAsync` never throws - (failures log + surface via `Failed` — the VM fires-and-forgets from the sync command handler); - loop = snapshot → render → `SubmitFrameAsync`, paced at `1/options.Fps` (default `Task.Delay`; tests - inject `Task.Yield`). `StopAsync` stops the encoder (closes stdin) BEFORE awaiting the loop — closing - stdin unblocks a write stuck on pipe backpressure, so stop can't deadlock on the pump. `ProcessFailed` - self-stops the pump. `HealthUpdated` forwards the encoder's stats (ship step 6 binds the bottom bar). -2. `ScreenCaptureManager.GetLatestFrame(key)` — mirrors `CameraManager.GetLatestFrame(deviceId)`; the - backdrop's live frame for the compositor. -3. `MainViewModel` — owns the resolver (`WebcamSceneConfig` → `GetLatestFrame(WebcamId)`; - `Source.IsLiveCapture` → `GetLatestFrame(CaptureKey)`; image/background → `StaticPixelCache.Get(AssetId)`), - builds `CompositorOptions` from the tier + `OutputRect*` (doubles rounded to ints — the vertical - 607.5 half-pixel crop rounds to a perfectly-centered 608), builds `EncoderOptions` from the tier when - the URL provider returns one, constructs the real `FfmpegEncoder(new FfmpegLocator())`, starts the - pump on go-live, stops it on end-stream, disposes in `Shutdown`, and flips `StreamStatus.Error` when - the pump fails while live (minimal — detailed health surfacing is ship step 6). - -**Test plan (Good Dog Rule — ONE integration test):** `FramePumpTests.Start_CompositesScene_FeedsEncoder_StopsCleanly` -drives the full lifecycle against fakes — real `SceneCompositor` + real `FramePump`, fake `IFfmpegEncoder` -— asserting the composited red backdrop frame actually reaches the encoder at the tier size and that stop -tears everything down. Units: no-URL start skips the encoder, re-entrant start/stop no-ops, encoder -start-failure raises `Failed` + disposes, `ProcessFailed` self-stops the pump, `HealthUpdated` forwards. -`ScreenCaptureManagerTests.GetLatestFrame_ReturnsLatestPump_UntilReleased` pins the new accessor. - -**Out of scope (later ship steps):** the loopback/mic → AAC mix (replaces `anullsrc`), health stats in the -bottom bar (ship step 6), scene-switching transitions, and any flash-frame wiring. - -**Built (2026-08-12):** `FramePump` shipped in `Services/Encoder/`, `ScreenCaptureManager.GetLatestFrame` -added, `MainViewModel` wired end-to-end (resolver + both option builders + pump lifecycle), `FramePumpTests` -(7) + `GetLatestFrame` test (1) added — build **0 warnings**, **147 tests passing**. Known consideration: -the pump reads the active scene on a background thread while the UI can still edit it; a concurrent-mutation -exception is contained (logged + `Failed` + pump stops) rather than crashing. - -#### Ship step 5.5 — Social bar bug fixes + the bar on the live output (2026-08-13) - -**Bug 1 — bar wouldn't reliably change position (root cause found, then simplified):** the original -drag set `Canvas.SetTop(bar, …)` with a local value, which permanently overrides the -`Canvas.Top="{Binding SocialBarTop}"` binding — the release-time `SetSocialBarPosition` → -`PropertyChanged(SocialBarTop)` could never beat it. First fix added direction-snapping during the drag -(`SocialBarSnap.Decide`, ±6px deadzone) + `bar.ClearValue(Canvas.TopProperty)` on release — but that -still misbehaved for shaky hands (jitter around the deadzone: it snapped up reliably, then refused to -come back down and snapped back to top). **Superseded by a click-toggle (KISS, user decision):** clicking -the bar in the preview flips it top ⇄ bottom (`MainViewModel.ToggleSocialBarPosition` → the existing -`SetSocialBarPosition`), the bar rides `{Binding SocialBarTop}` alone (no local values, no deadzone, no -jitter sensitivity), and `SocialBarSnap` was removed. The `ClearValue` lesson stands: never set a local -value on a property the binding owns. - -**Bug 2 — Mastodon showed the generic 7-star honeycomb (root cause found):** the DB row -`@gramps@llamachile.tube` had `Software = NULL` — nodeinfo was only ever resolved against the identity domain -(`llamachile.tube`, a landing page), never probed for the real instance at `mastodon.llamachile.tube`. -Fixed on three fronts: `HttpSocialValidator` now **probes well-known subdomains** (mastodon. → social. → -… `FediverseSubdomainCandidates`) when the identity domain and its redirect both come up empty, under a -~15s linked-CTS budget, with an optional `Action` log; `MainViewModel` **heals** any fediverse entry -missing a software name on layout load (`HealFediverseSoftwareAsync` — static, testable; instance wrapper -runs it off the UI thread, applies via the dispatcher, saves); `SocialEntry.FediverseSoftware` is now -**settable** and raises `PropertyChanged` for `LogoData`, so the heal updates the icon in place. If the user -later re-saves the entry with the icon fixed, the name persists with it. - -**Compositor rendering (user-approved scope):** the social bar now appears **on the live output**, not just -the preview — `Compositor/SocialBarRenderer.cs` rasterizes the entries into a transparent straight-alpha -BGRA strip (1920-wide, 40px content + 24px glow pad, green `#2ecc71` glow baked in) via `RenderTargetBitmap` -(WPF glue like `StaticPixelCache`; the compositor core stays pure). `SceneCompositor.Render` takes optional -`socialBarFrame` + `socialBarTop` (master space) and blits it **last — above the branding flash** (the old -`BlitFlash` generalized to offset `BlitOverlay`). `FramePump` gains a `socialBar:` seam -(`Func<(VideoFrame?, SocialBarPosition)>`, re-read every frame) and places the bar at `0` or -`SourceRectHeight − bar height`. `MainViewModel` owns the frame (`RenderSocialBarFrame`, re-rendered on -load/save/notify) and feeds the seam. - -**Tests (+6 → 153 passing, 0 warnings):** settable `FediverseSoftware` updates `LogoData`, -subdomain-probe unit + null-when-silent unit, the branch's **one integration test** -`Socials_HealMissingFediverseSoftware_RoundTripsThroughDb` (temp-DB roundtrip: NULL software → healed via -the validator → persisted), compositor bar overlay (top/bottom + above-flash), `FramePump` bar pass-through -(Top then flipped to Bottom mid-run — the seam is re-read each frame), and both `ISocialValidator` fakes -(`FakeValidator`/`BlockingValidator`) gained `ResolveFediverseSoftwareAsync`. The drag-snap units -(`SocialBarSnap`) were removed with the click-toggle supersession. - -**Not included (say the word):** rewriting the healed entry's `ProfileUrl` to `https://mastodon.llamachile.tube/@gramps`. - ---- - ---- - -## TASK 5 — Layout Persistence (SQLite) - -**Goal:** Scenes, sources, and asset bytes survive restarts; assets are always available. - -### Status: ✅ Done - -1. ✅ SQLite database (`Microsoft.Data.Sqlite`) at `%APPDATA%\ytLlive\ytLlive.db`; schema versioned via `PRAGMA user_version` (currently **v8**) -2. ✅ Assets live in the DB (BLOB keyed by SHA-256 content hash), never file paths — deleting the original file never breaks a scene -3. ✅ File-model save/open — the active layout file is tracked (default is the AppData DB); **Save Layout As… / Open Layout…** switch the active file; auto-save writes to whatever is active -4. ✅ Auto-save (invisible) — ~1.5s debounce on scene/source add/remove/reorder/rename/hide + any source transform change; flush on window close -5. ✅ Startup — load the active file; seed the five canonical scenes only when the DB is empty; (+) re-adds a missing canonical scene and is hidden once all five are present; adding beyond the five is rejected -6. ✅ Schema v1 → v9 — webcam columns (v2), singleton `Webcam` + per-scene `WebcamSceneConfig` (v3), `RectWidth`/`RectHeight` round-to-rect restore (v4), `Source.IsBackdrop` + `Source.CaptureKey` (v5), `Scene.HasBackdrop` — backdrop **Live-only by policy** (v6, one-time backfill + `EnforceBackdropPolicy` on every load), `Scene.HasSocialBar` (v7, dropped per-scene toggle — column back-compat, unread), `Socials.BarEnabled` (v8); the `SocialEntry.Software` fediverse-software column is a **column-presence migration** (commented v8→v9, no version bump — `user_version` stays 8); v9 = single-row `Music`. **Load-time rule:** sources load with `IsBackground` derived from `Type` (OR'd with the persisted column so legacy DisplayCapture backdrops keep their flag) — rows saved before the Type setter derived the flag heal on load -7. ✅ WindowHandle stays in-memory (per-session); save = transactional rewrite; orphaned assets pruned. **v9 lands in the TASK 8 audio milestone** (single-row `Music` — `TrackPath`/`IsEnabled`) - -### Design decisions - -1. **SQLite database** (`Microsoft.Data.Sqlite`) at `%APPDATA%\ytLlive\ytLlive.db`; schema versioned - via `PRAGMA user_version`. -2. **Assets live in the DB, not on disk** — `Asset` table stores image bytes (BLOB) keyed by a - SHA-256 content hash (unique). Identical image content collapses to one row regardless of file - name — the 1:M resource memory model, enforced by the database. No file paths; deleting the - original file never breaks a scene. -3. **File-model save/open** — the active layout file is tracked (default is the AppData DB). - **Save Layout As… / Open Layout…** switch the active file; auto-save writes to whatever is active. -4. **Auto-save (invisible)** — ~1.5s debounce on scene add/remove/reorder/rename/hide, source - add/remove/reorder, and any source transform change; flush on window close. -5. **Schema** — `Scene` (Id, Name, IsHidden, IsChatScene, HasBackdrop, HasSocialBar, SortOrder), `Asset` (Id, Hash, Data, - PixelWidth, PixelHeight), `Source` (Id, SceneId FK cascade, AssetId FK, Type, Name, IsEnabled, - X/Y/Width/Height/Opacity, MonitorIndex, DeviceId, ClipShape, IsMirrored, SortOrder), `Socials` (Id, - BarPosition, BarJustify — back-compat, unread, BarEnabled added via `ALTER`), `SocialEntry` (Id, - SocialsId FK cascade, Service, Handle, ProfileUrl, SortOrder) — `user_version` **8** (v1 → v2 = - `ALTER TABLE` adds the two webcam columns; v3 = singleton `Webcam` + per-scene `WebcamSceneConfig`; - v4 = `WebcamSceneConfig.RectWidth`/`RectHeight` for the round-to-rect restore; v5 = `Source.IsBackdrop` - + `Source.CaptureKey` for the live-capture backdrop; v6 = `Scene.HasBackdrop` — the backdrop is - **Live-only by policy** (one-time backfill turns Starting/BRB/Chat/Ending off and drops their backdrop - sources; `EnforceBackdropPolicy` re-normalizes every load); v7 = `Scene.HasSocialBar` (per-scene toggle - dropped — column back-compat, unread); v8 = `Socials.BarEnabled`). The `SocialEntry.Software` - fediverse-software column is a column-presence migration (commented v8→v9, no version bump). - `WindowHandle` stays in-memory (per-session). Save = transactional rewrite; orphaned assets pruned. -6. **Startup** — load the active file; seed the five canonical scenes - (Starting/Live/BRB/Chat/Ending, `SceneCatalog`) only when the DB is empty. The (+) - button re-adds a missing canonical scene and is hidden once all five are present; - adding beyond the five is rejected — work with less, never more. - ---- - ---- - -## TASK 6 — UI polish batch (scenes/sources rows, dedup naming, social bar) - -**Goal:** clean up the two side lists and the social bar per gramps's review. - -### Status: ✅ Done - -1. ✅ Scene rows are pure selection rows — the per-row edit/trash/visibility icons and the inline rename TextBox are gone (`EditSceneCommand`/`RemoveSceneCommand`/`ToggleSceneVisibilityCommand` + handlers + `Scene.IsEditing` removed; `IsHidden` stays persisted + dims hidden rows) -2. ✅ Source rows gained the trio — edit (inline rename via new `EditElementCommand` + `SceneElement.IsEditing`), visibility eye (new `ToggleElementVisibilityCommand` flips `SceneElement.IsVisible`; the eye style now binds `IsVisible`, open/slashed + row dims to 0.45 when hidden), and the existing trash -3. ✅ Duplicate resource names get a no-space incrementing suffix via shared `NextSourceName` (Image, Image2, Image3…) — next free number derived from the names actually in the scene, so deleting a middle resource never collides (`AddSource` + `AddReusedImage` both use it) -4. ✅ Social bar renders the full validated handle — `MaxWidth=200` + `TextTrimming` removed from BOTH `SocialBarRenderer` and the preview template (mastodon `@gramps@…` no longer cuts off) -5. ✅ Side panels stay fixed-width (left 220 / right 300) — deliberate: they never re-layout on resize, the preview absorbs it -6. ✅ Focus-loss capture lag documented as a known OS limit in `ai.md` — deferred by user decision (no code change) - -### Design decisions - -1. **Next free number from names, not type counts** — the old scheme counted elements by `SourceType` (`count == 0 ? baseName : base+count+1`), which collided after deletions; the new helper scans actual names. -2. **One WPF App per AppDomain** — the real-App tests (round-clip + naming) share `RealAppHost` (a dedicated STA thread owning the single `App`) via the `RealApp` serial collection, instead of each calling `new App()`. - ---- - ---- - -## TASK 7 — Meter scaling amplification (voice meter uses the full bar) - -**Goal:** the mic (and game) meters feel light — speech peaks should peg into the red at maxed volume. - -### Status: ✅ Done - -1. ✅ `AudioLevelMeter.ToDisplay` now adds **+10 dB of input amplification** before the −60..0 dBFS → 0..1 log mapping (was raw dB): speech peaks (~0.2 RMS, −14 dBFS) read ~0.93 → red zone; normal speech (~0.05, −26 dBFS) ~0.73 → yellow; background noise ≤0.001 linear (−60 dBFS) still reads 0 (the meter never idles on it) -2. ✅ Unit tests updated to the new mapping + new `ToDisplay_Pushes_Speech_Peaks_Into_Red_At_Maxed_Volume` (0.2 → 0.92..0.95, 0.05 → 0.72..0.75) - -### Design decisions - -1. **Amplify at the mapping, not in the VM** — one knob (`ToDisplay`), shared by the mic bar and the game bar; the `× MicVolume` gain-on-noise behavior is untouched (raising the slider still moves ambient noise up the bar). -2. **+10 dB, not more** — louder boosts push quiet speech into the lower half and make the noise floor visible; +10 puts speech peaks solidly in red while idle stays at zero. - ---- - ---- - -## TASK 8 — Audio milestone: real stream audio + voice filters + TRAX music (2026-08-14) - -**Goal:** every audio issue done and tested in one branch — real mic/game audio reaches the encoder (replacing the `anullsrc` silence), `MicVolume`/`GameAudioVolume`/mute become real pre-AAC gains, TASK 10's voice filters land, auto-duck keeps the creator's voice over game + music, and a free background-music source ("TRAX") plays into the sound bar and the stream. - -### Status: ✅ SHIPPED 2026-08-14 — 196 tests passing, 0 warnings (ONE integration test: `AudioPipelineTests.Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe`) - -1. ✅ **Real audio into the encoder** — the mixer's loopback samples were **dropped today** (`AudioMixer.OnLoopbackSample` metered only) and `MicVolume` was meter-only; `FfmpegArgs.cs:28` ran `-f lavfi -i anullsrc` (silence). Transport: **ffmpeg reads a Windows named pipe** — `-f f32le -ar 48000 -ac 2 -i \\.\pipe\ytllive_audio` replaces the anullsrc block, plus explicit `-map 0:v -map 1:a`; pipe name via `EncoderOptions.AudioPipeName` (`DefaultAudioPipeName = "ytllive_audio"`). Mixer stays **two inputs (mic + loopback)** — no N-source abstraction, no `StereoMixer` class. The VM owns the pipe lifecycle (`BeginGoLive` → `_audioMixer.StartLive(pipeName)`, `StopStream` → `_audioMixer.StopLive()` before the pump stops); `FfmpegEncoder.cs` is untouched. -2. ✅ **Honest gains reach the stream** — `MicVolume` scales the mic channel; `GameAudioVolume` + mute scale the loopback channel (the "mute in preview but stream still plays" footgun dies with it). One knob per input = KISS. Implemented as `Func` gain seams on `AudioMixer` (`micGain`/`loopbackGain`), wired by `MainViewModel`. -3. ✅ **Voice filters (TASK 10, before the meter AND the mix)** — `VoiceFilterChain`: bass (`LowShelfFilter` 120 Hz +4 dB) → treble (`HighShelfFilter` 8 kHz +3 dB) → **noise gate** (pure C#, creator's choice) → compressor (threshold 0.5, 4:1). Pure stateful DSP (TDF2), sine-in unit tests. Meter stays post-filter. -4. ✅ **Auto-duck** — pure envelope (`AutoDucker`): mic RMS > 0.02 → game+music dip ×0.25 (~12 dB), attack 0.05 / release 0.005, recover on release. Always-on, no knobs. Strong form of the creator's "voice always over the game volume" idea. -5. ✅ **TRAX — background music, FREE** (keeps the one paid line = Alerts + flash removal). `MusicPlayer`: NAudio `MediaFoundationReader` (mp3/wav/m4a) → `VolumeWaveProvider16` at the hardcoded **0.20** (fixed, not changeable) → `WaveOutEvent` on the default device (added the sibling `NAudio.WinMM` 2.2.1 package — `WaveOutEvent` isn't in `NAudio.Wasapi`), loop on end. **No 3rd mixer input and no `MusicVolume`** — music plays on the desktop, the existing loopback picks it up: heard in headphones, the sound-bar meter bounces, and the stream carries it through the loopback channel (ducked with game when the voice is hot). Known wrinkle (out of scope): YouTube mutes VODs carrying copyrighted music — future "music on live, off VOD". -6. ✅ **TRAX footer control (final spec, agreed 2026-08-14)** — `YtButtonSecondary`, **status dot + "TRAX"** text, in the center footer stack beside MIC: - - **Status dot** (same 8px Ellipse pattern as MIC/Socials): **red** no track loaded · **yellow** loaded not playing · **green** playing. - - **Left-click** → toggle play/pause; no track loaded → opens the picker instead. - - **Right-click** → always opens the in-app track picker (`TraxButton_PreviewMouseRightButtonUp`, code-behind pattern like `GameSpeaker_MouseLeftButtonUp`, `e.Handled = true` so no context menu). - - **Tooltip** → playing/loaded track name; "No track — right-click to choose background music" when empty. - - **No slider** — volume is the 0.20 constant. Picker = plain OS `OpenFileDialog` filtered to mp3/wav/m4a/aac/flac/ogg. -7. ✅ **Sound bar shows music** — `IsGameAudioBarVisible = gameDetectorProducingSound || isMusicPlaying`; relabelled "Game Audio Capture" → **"Desktop Audio"** (TRAX rides the same channel). **Follow-up (2026-08-15): the gating is gone — the game audio bar is ALWAYS visible** (the `IGameAudioDetector`/`GameAudioDetector`/`GameAudioHysteresis` stack + the VM's 250ms poll timer and their tests were deleted; the bar kept hiding the desktop meter when no full-screen game with sound was up, and the creator wants it pinned). -8. ✅ **Capture hardening (design changed at build time)** — the plan's "force IEEE-float 48 kHz on both WASAPI sources" was **dropped**: NAudio 2.2.1's `WasapiCapture` exposes no overridable `GetDefaultMixFormat`, so both sources capture the device's own mix format and the mixer's pure `TinyResampler` normalizes any rate/channel count to 48 kHz stereo (the resampler IS the design, not a fallback). Stop path disposes the audio pipe (EOF) in `MainViewModel.StopStream` before the pump stops, so both ffmpeg inputs end in order. -9. ✅ **Schema v9** — single-row `Music` (`TrackPath`, `IsEnabled`; volume is the 0.20 constant) + migration in `LayoutStore.cs` (`user_version` 9; `Save` gained an optional `Music? music = null` param so existing 3-arg callers still compile; load query + null reset). -10. ✅ **Docs in the same commit** — `ai.md` audio section rewritten ("one knob per input", 2-in mix, auto-duck, TRAX free, VOD-mute wrinkle), TASK 10's status, this task's status, HANDOFF. - -**Files (new):** `Services/Audio/VoiceFilterChain.cs` (+ `LowShelfFilter`/`HighShelfFilter`/`NoiseGate`/`Compressor`), `Services/Audio/AudioRingBuffer.cs` (async-arrival source buffers), `Services/Audio/AutoDucker.cs`, `Services/Audio/TinyResampler.cs`, `Services/Audio/MusicPlayer.cs`, `Services/Audio/NamedPipeAudioWriter.cs` (+ `IAudioPipeWriter` seam), `Models/Music.cs`. - -**Files (changed):** `AudioMixer.cs` (filter chain before meter+mix, loopback into the mix, gains via `Func` seams, ducker, silence-filler so the pipe never stalls, `StartLive`/`StopLive`), `WasapiMicAudioSource.cs` (device mix format — resampler normalizes; no forced format), `FfmpegArgs.cs` (named-pipe input + `-map`, name via `EncoderOptions.AudioPipeName`), `EncoderOptions.cs` (`AudioPipeName`), `ytLive.csproj` (`NAudio.WinMM` 2.2.1 for `WaveOutEvent`), `LayoutStore.cs` (schema v9), `MainViewModel.cs`/`MainWindow.xaml`/`MainWindow.xaml.cs` (footer TRAX group, sound-bar visibility OR music, go-live starts the pipe, end closes it first). - -**Testing (Good Dog Rule — ONE integration test):** `AudioPipelineTests.Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe` — real filter chain + mixer + ducker + `NamedPipeAudioWriter`, fake `IAudioSource`s, test-side `NamedPipeClientStream` reads the bytes and asserts post-filter/post-gain/post-duck mixed stereo (ring buffers pre-filled so the first pipe tick already carries real audio — avoids a start-of-stream silence race). Units: each DSP stage (known sine-in → expected gain), ring buffer wrap/underflow/overwrite, ducker envelope, resampler (down/up/stateful/identity), `FfmpegArgs` pipe+map, schema v9 roundtrip + no-music-row. Existing tests stay green. - -**Out of scope (this branch):** IP webcam (video-only when it lands — never an audio input), chat box source (TASK 3 item 18), alt-key crop, credits, background removal, music-off-VOD, `PremiumUrl` (TASK 10 seam). - ---- - ---- - -## TASK 9 — YouTube Live Stream Management - -**Goal:** Create/bind broadcasts, monitor YouTube-side stream health — the v3 way. - -### Status: ⏳ In progress — items 1–3 SHIPPED (reusable stream 2026-08-16; report-by-exception health 2026-08-16); item 4 SHIPPED (status reconciled 2026-09-01); item 5 open; item 6 = deliberate lock (TASK 36); item 7 scoped to core fields — advanced tab is out-of-product - -1. ☑ **Broadcast creation** — title/description/privacy/scheduledStartTime via API, with the v3 flags above (SHIPPED: `CreateBroadcast` sends `enableAutoStart/Stop`, `enableMonitorStream=false`, `latencyPreference=low`, `selfDeclaredMadeForKids=false`) -2. ☑ **Reusable stream** — create once, cache + reuse; bind to broadcast (SHIPPED: `GetOrCreateReusableStreamAsync` lists-then-inserts the `variable`/`isReusable` stream, cached via `LayoutStore` Settings, bound at broadcast insert via `boundStreamId`; `_rtmpUrlProvider` yields the ingest URL so go-live actually encodes + pushes) -3. ☑ **Health monitoring** — poll `liveStreams.list` `healthStatus` + `configurationIssues[]`, surface banner only on warning/error (SHIPPED: `GetStreamHealthAsync(streamId)` 30s while live; pure `StreamHealthReporter.BannerFor` = report-by-exception; banner strip under the top bar, amber warning / dark-red error, via `HealthIssueBanner`/`HealthIssueBackground`; poll failures log-only; ONE integration test `GetStreamHealthAsync_Report_By_Exception_Banner_Only_On_Warning_Or_Error`) -4. ☑ Live chat — poll `liveChat/messages`, render in right panel, support Super Chat + membership badges — **SHIPPED** (status reconciled 2026-09-01: `YouTubeChatService` polls + parses `superChatEvent`/`newSponsorEvent` with badge/level on `ChatMessage`; the panel MOVED to the left-panel live state by Control Surface UX, "right panel" superseded; the remaining reward types land with the monetization chain, TASK 10 → related work) -5. ☐ Error handling — the YouTube error codes: `errorStreamInactive`, `invalidTransition`, `redundantTransition`, `liveStreamDeletionNotAllowed`, `liveStreamModificationNotAllowed`, `liveBroadcastBindingNotAllowed` -6. ❌ **Visibility picker** — DELIBERATE TEST-PHASE LOCK (creator decision 2026-09-01), not open work: "always Private" stays during multi-month real-world testing so breakage VODs never clutter the channel. `recordFromStart`/DVR stay on — Private VODs are invisible review tapes; bulk-delete pre-GA. Unlock = TASK 36 gold-pass item (remove the override in `YouTubeStreamService.CreateBroadcast`, wire the dialog selection). See `ai.md` → YouTube Live API constraints. -7. 🔶 **Full broadcast form** — RESCOPED 2026-09-01, not the old two-tab plan. **Core fields (SHIPPED as the Text drawer, 2026-08-24):** title, description, tags, visibility, made-for-kids — live-editable. **Scheduling:** ships as **TASK 34** (Scheduled checkbox + datetime in the same drawer). **Advanced tab is PERMANENTLY OUT** (the 10% margin): latency (locked `low`), DVR/record-from-start (locked on), embed, projection, CC, region restrictions stay fixed at sane defaults, invisible — every exposed field is a support ticket. `categoryId` removal: done (not a `liveBroadcast` field). Monetization enablement (if ever needed) rides the reward-events chain via `liveBroadcasts.update`, not a form field - -### Design decisions (v3) - -1. **One-click go-live** — `liveBroadcasts.insert` with `enableAutoStart=true`, `enableAutoStop=true`, `enableMonitorStream=false`, `selfDeclaredMadeForKids=false`, `latencyPreference=low`. No `transition(live)` call, no testing stage, no liveStarting polling. Encoder starts → YouTube brings it live by itself. -2. **Variable reusable stream** — `cdn.resolution=variable`, `cdn.frameRate=variable`, `isReusable=true`. Create once per channel, cache ingestion URL + stream name, reuse for every broadcast. Any quality tier works without recreation; auto step-down needs no API calls. -3. **Report-by-exception** — poll `liveStreams.list`; banner only on `healthStatus` warning/error issues (`configurationIssues[]`). Bottom strip = YouTube logo + green/red connection dot (clickable → opens the dialog). -4. **One dialog, three states** — `not connected` (sign-in) / `connected-offline` (all editable) / `live` (title + description + visibility editable; quality + account greyed out). Both entry points (Start Stream button + bottom strip) open it; prefilled from saved session profile. -5. **Live edits** — `liveBroadcasts.update` with part=`snippet,status` for title/description/privacy. -6. **End stream** — stop encoder → `transition(complete)`, with `enableAutoStop` as the safety net. **SHIPPED 2026-09-01** (`EndBroadcastAsync`, wired into `StopStream` after RTMP EOF; the call had been specced since TASK 9 but never existed — found during recording verification) -7. **Broadcast ID == Video ID** — one ID to track status, health, and the auto-created VOD (`recordFromStart` + `enableDvr`). - -### Requirements: - -1. **Broadcast creation** — title/description/privacy/scheduledStartTime via API, with the v3 flags above -2. **Reusable stream** — create once, cache + reuse; bind to broadcast -3. **Health monitoring** — poll `liveStreams.list` `healthStatus` + `configurationIssues[]`, surface banner only on warning/error -4. **Live chat** — poll `liveChat/messages`, render in right panel, support Super Chat + membership badges -5. **Error handling** — the YouTube error codes: `errorStreamInactive`, `invalidTransition`, `redundantTransition`, `liveStreamDeletionNotAllowed`, `liveStreamModificationNotAllowed`, `liveBroadcastBindingNotAllowed` -6. **Visibility picker** — Private/Unlisted/Public from the go-live dialog -7. **Full broadcast form** — Core + Advanced tabs with all API-supported fields - ---- - ---- - -## TASK 10 — Monetization: watermark-only subscription + Polar billing - -**Goal:** annual subscription via Polar.sh that removes the branding watermark. All features are -free — the only difference between free and paid is the watermark. - -**Related work — monetization awareness (2026-09-01, ungated, free):** this task's billing is the -*payer* side; the *product* also carries built-in, ungated monetization awareness for every creator — -reward-event capture (all 7 `liveChatMessage` reward event types → canonical `RewardEvent` → SQLite -persistence), per-session/broadcast reports (members, Super Chat $, stickers, gifts, milestones — the -in-app replacement for the emailed "stream activity report"), and a journey tracker (position % + -ETA vs. versioned, date-aware YPP thresholds; the Tier-2 bar doubles for new applicants 2027-02-01). -Details and design notes: `ai.md` → Monetization. This is tracked as related work under this task's -narrative, not a separate task number — it ships in its own code slices with one integration test each. -**Scope: v1** (2026-09-01 complete-v1 ruling — there is no v1.x): slice order = reward-event capture → -session report → journey tracker → Alerts (TASK 3 item 20). - -**Business details (pricing, Polar product/checkout/discounts) in `MONETIZATION.md` (gitignored).** - -**Polar product:** `d105dfa1-497e-423b-8cd4-e0ee2e3abbc0` (LlamaCasty, $99/yr, currently `private`) -**Checkout:** `llamacasty.com` → Polar hosted page (~80% configured) -**Discounts:** `LLAMAFOUNDER` (100% off, 50 uses, `once`), `LLAMA50` (50% off, 12 months, unlimited) -**Org ID:** `c05fb364-b967-4f6c-adf2-8a144e46d085` (org-scoped OAT — `organization_id` can be omitted from API calls) -**Key prefix:** `LCYT-` -**Validate endpoint:** `POST https://api.polar.sh/v1/customer-portal/license-keys/validate` (no auth required for client-side validation) - -### Status: 🔶 In progress — steps 1–7 shipped, Velopack update URL pending - -1. ✅ **Billing provider** — Polar (polar.sh) chosen and configured. Product, price, license key benefit, discounts, checkout link all created. -2. ✅ **License key entry + verification UI** — About overlay swap panel (same pattern as Licenses panel). Text input + "Activate" button + status message. "Enter License Key" button in the hub. -3. ✅ **Offline entitlement** — `LayoutStore` Settings table: `LicenseKey`, `LicenseValidatedAt`, `IsPremium`. 14-day offline grace period. Startup re-validates online, falls back to cache. -4. ✅ **BrandFlash → watermark toggle** — `IsPremium` property setter flips `BrandFlashEnabled = !value`. Flash stops immediately on premium activation. Flash asset VideoFrame rendering deferred to TASK 4 compositor. -5. ✅ **Remove social slot locks** — all 6 social bar slots open to everyone (no `IsPremium` gating on slots 3-6) — shipped as TASK 10c -6. ✅ **"Unlock Premium" button** — `PremiumUrl` set to Polar checkout; `IsPremiumAvailable` is true; button lights up in the About hub. -7. ✅ **Renewal/lapse handling** — startup re-validates against Polar API. Lapse → `IsPremium = false` → flash returns. Grace period: 14 days offline. -8. 🔶 **Velopack auto-updates** — NuGet packages + bootstrap in `App.xaml.cs`. Update URL self-hosted on DO droplet (pending configuration). - ---- - ---- - -## TASK 11 — Post-pause polish batch: the creator's 8 review issues (2026-08-15, shipped) - -**Status: ✅ SHIPPED 2026-08-15 — 197 tests passing, 0 warnings (ONE integration test: -`AudioPipelineTests.Mix_HonorsProviderGains_AndGameMute_KillsTheLoopback`).** - -The creator reviewed the TASK 8 build and filed 8 issues. All fixed in one branch on `main`: - -1. ✅ **Desktop/game audio volume slider had no effect** (headset OR stream) — the `AudioMixer`'s - `micGain`/`loopbackGain` seams defaulted to unity because the VM never passed them; the sliders - were decorative. Fixed: new `AudioGainProvider` (`Services/Audio/`) hands `MicMuted`/`MicVolume`/ - `GameMuted`/`GameAudioVolume` to the mixer at construction, read live each mix tick. -2. ✅ **Desktop/game mute button had no effect** — same root cause; `GameMuted` now zeroes the - loopback on the stream. Locally, `MusicPlayer.LocalGain` (new) is scaled by the game bar on every - volume/mute change + on TRAX load, so the creator **hears** the control work on the music (the - track rides the loopback channel, so it ducks locally exactly as it does on stream). -3. ✅ **TRAX played once then stopped** — `MusicPlayer.OnPlaybackStopped` only looped when - `Position >= Length`, unreliable for `MediaFoundationReader`; now any clean stop - (`e.Exception == null`) rewinds + replays; only errors/explicit stops surface via `PlaybackEnded`. - 4. ✅ **TRAX/MIC buttons swapped** — TRAX no longer sits between MIC and the (mic-specific) meter; - the footer's mic cluster is MIC + meter + mute + volume. - 5. ✅ **Mic source persists across restarts** — `LayoutStore` gained a `Settings` key/value table - (`SaveMicSourceName`/`LoadMicSourceName`); the VM saves on pick and restores before the mixer's - first `Start`, so a restart reconnects the same already-vetted device (green dot) or reports it - missing (yellow) instead of falling back to the default endpoint. - 6. ✅ **TRAX moved into the game audio bar** — left of the "Desktop Audio" label in the preview - overlay (the creator's follow-up pick), no longer in the footer; its tooltip teaches the clicks - ("left-click pauses/plays, right-click chooses a track"). -7. ✅ **Backdrop's icons shifted right** — the trash button's `Collapsed` released its column; a new - `HiddenBoolToVisibilityConverter` keeps the slot reserved (`Hidden`), so edit/eye stay in their - fixed columns for every source row. -8. ✅ **No separation between scenes and sources** — a 1px hairline with top/bottom padding now sits - between the two listboxes in the left panel. - -### Design decisions - -1. **The gain seams are the one source of truth** — `AudioGainProvider` is a thin seam (four Funcs) - so the mixer contract and the mute⇔zero-volume rule are testable without constructing the VM. -2. **Stream-honest AND locally audible** — the game bar's controls now do two things: scale the - loopback on the stream and scale the music in the headphones. Native game audio is untouched - (system output, OBS-style non-monitored); an OBS-style monitor loop is out of scope. -3. **Persist the DisplayName, not the device ID** — the FriendlyName match was already the mic's - identity end-to-end; `Settings` just makes it survive restarts. - ---- - ---- - -## TASK 12 — Master limiter on the live mix (2026-08-15) - -**Queued by the creator during the TRAX discussion:** "should the soundtrack be limited to avoid -squandering resources / should it cap at 20% / how do the three sound events balance?" Review -conclusions (all three instincts checked out, only ONE real gap): - -- **File size — no guard needed.** `MusicPlayer` uses `MediaFoundationReader`, which **streams from - disk** (progressive source): memory is flat (~a few MB) regardless of file size; CPU negligible. -- **20% cap — already enforced by construction.** `MusicPlayer.MusicVolume = 0.20f` + music rides the - SAME WASAPI loopback (and therefore the same gain) as game audio, so music:game is always exactly - **0.20:1 at any slider position** — it literally cannot rise above 20% of the current desktop volume. -- **The gap:** `AudioMixer.FillAndMix` summed mic + loopback with **no output ceiling** — mic 100% + - loud game/music could pass 0 dBFS and clip the AAC encode. - -**Shipped:** -1. ✅ **`Services/Audio/MasterLimiter.cs`** (pure, unit-tested) — a **−1 dBFS ceiling** (`Ceiling = - 0.891`), **instant attack per frame** (a hot frame is scaled exactly to the ceiling — no overshoot), - **smoothed release** toward unity so loud passages don't pump; gain never exceeds 1 (no boosting). - Applied at the end of `AudioMixer.FillAndMix`, right before the pipe write. -2. ✅ **Unit tests** — over-ceiling frames trimmed to ≤ ceiling; sub-ceiling frames never boosted; gain - recovers to unity after the loud frame ends. -3. ✅ **ONE integration test** (`MasterLimiter_CapsTheLiveMix_OnThePipe`) — real mixer + pipe harness: - a 0.95 loopback bed (hotter than the ceiling) is capped to exactly 0.891 on the wire while staying - audible. -4. ✅ **Docs in the same commit** — `ai.md` (go-live audio section), `Services/index.md` (new row). - -No changes to `MusicPlayer`, the 0.20 cap, the ducker, or the meter zones. Build 0 warnings. - ---- - ---- - -## TASK 13 — Social media launch kit - -**Goal:** marcom/social media assets and strategy for v1 launch. - -**Business details (positioning, messaging, platform strategy, launch assets) in `MARCOM.md` (gitignored).** - -### Status: 🔶 Scoped — nothing built; queued after all v1 features ship - -1. ☐ Product positioning & messaging (one-liner, elevator pitch, competitive positioning) -2. ☐ Social media swipe files (pre-written posts for supporters) -3. ☐ Launch day assets (demo video, screenshots, GIFs, social graphics, press kit) -4. ☐ Platform strategy (YouTube, Reddit, indie dev communities, Product Hunt) -5. ☐ Founder story (67-year-old dev building his own streaming app) -6. ☐ Email announcement templates -7. ☐ "Build in public" livestream angle (stream the coding of ytLlive with ytLlive) - ---- - -## TASK 14 — Creator feedback batch: quick fixes (TRAX, meter, backdrop, sliders) - -**Goal:** address creator's immediate UX feedback from testing session. - -### Status: 🔶 In progress — Branch 2 shipped - -1. ✅ TRAX volume — `LocalGain = GameAudioVolume * 0.2` so TRAX is 20% of the desktop audio slider (was 100%, now quieter) -2. ✅ TRAX tooltip — replaced `ToolTip` binding with `ToolTipService.ToolTip` + `Placement="Top"` to fix z-order inside Viewbox overlay -3. ✅ Mic meter boost — `MeterLevel` multiplied by 1.2× before clamping so the bar reads ~20% higher at same input level -4. ✅ Click-to-position sliders — clicking anywhere on the mic/game volume slider track now jumps the thumb to that position before starting the drag -5. ✅ Backdrop visibility — the dedicated `` now binds `Visibility` to `BackdropVisible` (ViewModel property); compositor checks `IsVisible` before blit -6. ✅ Rename "Backdrop" → "Game Capture" in `EnsureBackdrop()` + test assertions -7. ✅ Elements panel — new "ELEMENTS" section beneath Sources list; per-element config with live preview, ✕ revert, ✓ applied indicator -8. ✅ Webcam border config — color hex input + color swatch picker, border thickness 0–10 slider -9. ✅ Countdown source — `SourceType.Countdown`, pick list (Starting/BRB scenes only), timer minutes 1–60 default 15 -10. ✅ Web source — `SourceType.WebSource`, pick list (all scenes), URI text input -11. ✅ Elements scrollbar deselect fix — clicking the Elements panel scrollbar no longer deselects the selected element (scrollbar lives inside the ScrollViewer, not the StackPanel; guard now checks `ElementsScrollViewer` instead of `ElementsPanel`) -12. ✅ Web source URI + buttons inline — merged URI TextBox and check/revert buttons into a single DockPanel row (✓ rightmost, ✕ left); buttons no longer wrap to a separate line -13. ✅ Web source URI hidden for webcam — URI section was showing for webcam due to WPF binding path `SelectedElement.Type` not resolving on `WebcamSceneConfig`; replaced with `IsWebSource` virtual property on `SceneElement` (mirrors existing `IsWebcam` pattern), overridden in `Source` to return `Type == SourceType.WebSource`, bound via `BoolToVis` -14. ✅ Check/revert consistent order + accept — all three element sections (webcam border, countdown timer, web URI) now use consistent [✕][✓] button order; green check buttons enabled with click handlers that save current values as snapshot baseline (so revert undoes to last accepted state, not initial selection) - ---- - -## TASK 15 — Stock scene background images - -**Goal:** branded background images for each canonical scene (Starting, Live, BRB, Chat, Ending) -that both provide working hints to creators and self-promote LlamaCasty. - -### Status: ✅ Shipped — all 5 scenes seeded - -1. ✅ **Starting backdrop** — `Assets/starting-backdrop.jpg` (embedded resource), `SeedStartingBackdrop()` in MainViewModel, idempotent -2. ✅ **BRB backdrop** — `Assets/brb-backdrop.jpg` (user-provided Gemini image), `SeedBrbBackdrop()` in MainViewModel, idempotent -3. ✅ **Live backdrop** — `Assets/live-backdrop.jpg` (user-provided image), seeded via `SeedLiveBackdropAsset()`, used as the static fallback (TASK 16) -4. ✅ **Ending backdrop** — `Assets/ending-backdrop.jpg` (user-provided image), `SeedEndingBackdrop()` replaces existing (not skip-if-exists) -5. ✅ **Chat backdrop** — `Assets/chat-backdrop.jpg` (user-provided image), `SeedChatBackdrop()` at startup -6. ✅ `EnsureDefaultBackdrop()` called on every scene switch — seeds the default backdrop if scene has none - -### Design decisions - -- Images are user-provided, not generated — the creator owns the brand look -- Each scene gets its own image (Starting = "Starting Soon", Live = live branding, BRB = "Be Right Back", etc.) -- The images double as the no-game backdrop fallback (TASK 16), so the Live scene image should work as a desktop replacement - ---- - -## TASK 16 — Kill infinity display (no-game backdrop fallback) - -**Goal:** when no full-screen game is detected, show a static branded placeholder instead of -capturing the primary display (which causes the infinity mirror effect). - -### Status: ✅ Shipped - -**Changes:** -1. `ResolveAutoCaptureKey()` returns `null` when no full-screen game is detected (was: primary monitor) -2. `ReacquireScreenCaptures()` — clears stale `CaptureKey` when `auto == null` (handles DB upgrades); - skips `AcquireAsync` when no keys are set -3. `SeedLiveBackdropAsset()` — sets `AssetId` on the backdrop element itself (NOT a Background source); - called from `ReacquireScreenCaptures()` after `EnsureBackdrop()` guarantees the element exists -4. `Source.DisplaySource` falls back to `_imageSource` when `VideoImageSource` is null for live captures -5. `Assets/live-backdrop.jpg` — user-provided image, added as `` in csproj -6. `ResolveOutputFrame()` in compositor — `Source { IsLiveCapture: true, CaptureKey: not null }` already skips null keys; fallback pattern `Source { AssetId: not null }` picks up the static image - -**Rendering flow:** -| Scenario | Preview | Compositor | -|---|---|---| -| Game detected | Live capture (opaque, covers static asset on backdrop) | `ScreenCaptureManager.GetLatestFrame` | -| No game | Static image via `DisplaySource` → `_imageSource` on backdrop | `StaticPixelCache.Get(assetId)` | -| Manual capture pick | Live capture | Live frame | - ---- - -## TASK 17 — Web source rendering (WebView2) - -**Goal:** make the web source actually render URLs into the preview and stream output. - -### Status: ✅ Done — required for v1 (2026-08-28) - -1. ✅ Add `Microsoft.Web.WebView2` NuGet package -2. ✅ Schema v10: `WebUri TEXT` column on `Source` table + migration in `LayoutStore.cs` -3. ✅ Persist `Source.WebUri` on save/load (currently in-memory only — lost on restart) -4. ✅ Hidden off-screen `WebView2` control per web source — navigates to `WebUri`, renders in-app -5. ✅ Frame capture from WebView2 (`CoreWebView2.CapturePreviewAsync`) → `WriteableBitmap` (BGRA8) -6. ✅ Wire into `FramePump` resolver — `Source { Type: WebSource }` → latest WebView2 frame -7. ✅ Wire into `SceneCompositor` — render web source as an image element at its position/size -8. ✅ Preview shows live web content (not just a blank rectangle) -9. ☐ Handle navigation errors, invalid URIs, timeout gracefully - -**Rendering model (2026-08-28, canvas-size viewport + ALPHA-BBOX crop — DONE, creator-verified):** -the page renders at the MASTER CANVAS size (1920×1080), stable, never tracked (no reflow/truncation; -scrollbars suppressed via `overflow:hidden`). `FindContentBounds` scans the Bgra32 capture and crops -to the bounding box of non-transparent pixels — the widget's true rendered extent, measured from the -frame itself (NO DOM query, immune to layout timing; can never truncate content; full-canvas widgets -fall through to the full frame = prior verified-good image). `Stretch="Fill"` maps the cropped frame -flush under the selection box → box hugs the widget on all four sides. `QueryContentBoundsAsync` + -the JS content-bounds script are DELETED. **RESULT: bounding verified PERFECT by the creator with -two widgets.** Root-cause note on the earlier "remaining defect/gap": that was a WIDGET GLOW EFFECT -(the widget's own CSS glow pushes out its perceived borders), not a code bug — the alpha-bbox was -already hugging the glow halo. Lesson: test with a second, plain widget before changing code. -⚠️ Lesson bank: never return `JSON.stringify` from `ExecuteScriptAsync` (double-encodes); never -measure DOM stuff on NavigationCompleted (unsettled layout broke the image — `a5b9952`); avoid -sizing the container to the crop (`ab29ec8` reverted — made it worse); a widget glow effect can -fake a gap — verify with a plain widget first; measure rendered pixels instead. - -**Properties panel (2026-08-28):** web URI ✓/✕ icon buttons are `IsTabStop="False"` so Tab flows -X→Y→W→H→URI; the ✕ button now clears the URI textbox (was reverting to the pre-accept snapshot). -Slider style gained `IsMoveToPointEnabled="True"` — click-anywhere-on-bar jumps the thumb to the -click (volume sliders keep their manual `SetSliderValueFromClick`, harmless duplication). - -### Design decisions - -- WebView2 is the only option for Windows — it's pre-installed on Windows 10 20H2+ and Windows 11 -- The web source is a standard element — positioned/sized/opacitied like any image source -- Frame capture rate can be lower than video FPS (5-10 fps for web content is fine) -- This enables Streamlabs/StreamElements overlays via web URLs - ---- - -## TASK 18 — Local recording - -**Goal:** record the stream output to a local file, with or without simultaneously streaming. - -### Status: ✅ Shipped `a9eb360` (2026-08-29) — code done (incl. manual-rename modal), build 0 warnings, 244/246 tests; **running-app verification (takes 1–5 done)**: files land (ffmpeg re-pinned month-end), **webcam-in-output + social bar visually CONFIRMED from take 3's extracted frame**, rename modal used for real (take 3 was named via it); take 3 exposed the ~2fps producer starvation and the pump stage-timing (`97ffc42`) named it in one line — `avg render 258.1ms` — **slice 1 fixed 2026-09-03** (deadline pacing per OBS video-io.c + libyuv-style row blits in `SceneCompositor`, test `Pump_Paces_To_The_Deadline_Compensating_Render_Cost`); **take 4: pacing held but render stayed 58.9ms** (2M-iteration row walk + 8.3MB/tick LOH) — **slice 2 shipped 2026-09-04**: `VideoFrame.IsOpaque` producer-contract flag → full-cover backdrop is one `Buffer.BlockCopy`; integer fixed-point bilinear general path; pump scratch pool (release strictly post-submit, owned-by-reference so cache frames are untouchable); dead per-tick `fromScene` render + the `fromSceneProvider` seam removed (BlendFrame uses `TransitionService.FromFrame` — the old render fed nothing). Tests `Pump_Pools_ScratchBuffers_Across_Frames_Without_Stale_Pixels` (the ONE) + `Composite_OpaqueFullCover_Backdrop_CopiesEveryPixel_Into_Scratch`; 59/59 per-class green, clean build 0 warnings. **take 5 ran: render 58.9→25.5ms (`138/300` ≈ 2.2x still) — cause: the per-tick chat raster (`RenderFrame` hit `RenderTargetBitmap` every tick whenever the message buffer was non-empty — the buffer survives sessions); slice 3 shipped 2026-09-04: `ChatOverlayLayer` rasters on message/config change and blits a cached frame every tick (OBS text-source pattern; test `ChatOverlayLayerCacheTests`).** **take 6 ran WITHOUT attribution (35-41ms — build provenance unknown; slice 3 effectiveness UNCONFIRMED) → build-stamp shipped instead of guessing again: `Helpers/BuildStamp` GUID per build (csproj GenerateBuildStamp; incremental builds can no longer lie), wordmark superscript + startup.log line; stats split `render (resolve)` so the next take names the stage. **takes 7–8 (stamped f190587b): chat fix CONFIRMED (`resolve ≈0`) but render stayed 26-27ms — compositor re-rasterizing STATIC layers every tick; slice 5 shipped: `BlitCachedLayer` pastes once-rasterized element-space layers (OBS surface-cache pattern; test `PasteCache_RepeatRender_IsByteIdentical_And_ContentChangePropagates`). **take 9 ran (c65a3cde, paste cache): render 26.5→22.4ms yet period stayed ~37ms — the gap was Task.Delay's 15.6ms sleep quantum padding every sub-tick wait: the true ceiling, hidden until then (why takes 7→9 looked like zero change). Slice 6 shipped: timeBeginPeriod(1) for the pump life + bulk-sleep + 2ms spin tail + `avg wait` stat (accounting closes) + webcam now routes through the paste cache. **take 10 ran (59a02a5b): the new `wait` stat exposed the LAST structural bug — `render 22 + wait 10` against a 16.7ms deadline is impossible for a rebasing pacer: the wait was the producer QUEUED BEHIND THE LIVE PREVIEW — the pump's await-continuations inherit the UI SynchronizationContext (StartAsync fires from a command handler), so the loop had been rendering on the dispatcher all along. Slice 7: Task.Run the loop (OBS keeps media threads off-UI for this exact reason), StaticPixelCache locked + chat raster marshalled to the dispatcher on cache-miss (RTB/DrawingVisual are UI-thread objects), SustainedLowLatency GC, `worst render` stat, webcam routed through the paste cache; test `Pump_Produces_OffTheStartingContext`, 70/70 green. **take 11 ran (c10ce06c): off-UI loop WORKED — typical frames land exactly on the 16.7ms deadline (work ~10 + wait ~6.8; 212/300 best yet); the remaining gap is periodic 35-65ms spikes worsening across a take = gen2 GC pauses, fed by the capture path's fresh ~8.3MB array per DWM frame (~500MB/s). Slice 8: 4-deep capture buffer ring + `VideoFrame.Epoch` in the identity-keyed paste cache + `gen2 +N` printed per stats window; test `PasteCache_RecycledArrayWithNewEpoch_ReRasterizes_NotStaleHits` (fails on the old key), 37/37. Also this session (creator ask): post-mix master gain +40% (before the −1dBFS limiter, so no new clipping). **take 12: `gen2` ~0-1/window, `worst render` → ~16-20ms, n/300 → 300, audio ~40% louder in the file.** - **ROLL-FORWARD (2026-09-04): the two-loop producer (the OBS duplicate-instead-of-starve redesign) and the +40% pre-limiter gain were NOT kept — the two-loop introduced recorded-output TEARING on real hardware and was rolled back with it (full record in HANDOFF). Main sits on the slice-8 serial loop plus this unit: all shared-frame rings 4→8 (screen/camera/web — depth × source period must exceed the worst consumer hold), fresh camera + WebView2 capture pooling, and the wordmark release counter #N replacing the UI GUID (per-build GUID now logs to startup.log only). The tear-proof pacing fix — duplicate-instead-of-starve ON the serial loop — is the next in-flight unit.** - Known remaining churn (follow-ups, not silently done): vertical tier's final `BilinearScale` still allocates per frame; one slow tick (~15-25ms) per arriving chat message — debounced off-tick re-render if take 6 shows burst loss. **User UX spec (2026-09-04) queued behind this**: two-line top bar (LIVE rename, radio pills, Login/Logout button + avatar right-click Change Account, no account light; line 2 centered Start↔Stop grayed-until-armed) + up-front SaveFileDialog for REC (native overwrite prompt; retires the stop-time rename modal) + Go-Live dialog KEPT as preflight confirmation prefilled from the Text drawer — decisions captured in HANDOFF - -1. ✅ `EncoderOptions` extended with `StreamEnabled` / `RecordEnabled` / `RecordPath` (independent intent flags) -2. ✅ `FfmpegArgs.Build` reworked into per-output blocks (stream `-f flv`, record `-f mp4`) via `AddVideoTags` -3. ✅ Two modes via pills: record-only / stream-only (single ffmpeg, one output). ~~stream+record~~ — **REMOVED by creator ruling 2026-09-01: the VOD is already the copy; dual encode drags mid-range hardware and degrades both outputs ("we're not them"). Pending implementation: pills become mutually exclusive radios + `BeginGoLive(alsoRecord)` param dies.** -4. ✅ Top-bar REC + ON-AIR pill toggles, status lights (REC green when recording, ON-AIR green when live), dynamic primary-button text, account status-light tooltip -5. ✅ Record-folder persistence (`LayoutStore` `RecordFolder` key) + picker (`ChooseRecordFolderCommand`) -6. 🟡 Branding flash carries into local recordings — same frame path as streaming (verify in the running app) -7. ✅ Output folder `%APPDATA%\ytLlive\recordings\` default; auto-name `ty---0000.mp4` at start, **rename-on-stop** to `ty-…-.mp4` (numeric suffix on collision) - -Remaining: running-app verification of the rename + dual output. - -### Design decisions - -- **Record OR stream, never both (creator ruling 2026-09-01):** YouTube's VOD already keeps the copy - (downloadable from Studio), and a second concurrent encode degrades BOTH outputs on anything short of - a performant chassis. Recording's real jobs: the offline mux-check bench and the no-account session. - Pills become radios (arming one disarms the other); the one-stop ruling ("Stop ends everything") - already assumes a single session kind. -- FFmpeg supports multiple outputs natively (`-f flv rtmp://... -f mp4 file.mp4`) — kept as generic machinery, but only one block is ever enabled now -- MP4 is the default container; crash-safety arrives as **fragmented MP4** (TASK 32 slice 4 — in record mode the local file IS the archive), retiring the old "MKV as an option" note -- Record-only mode is useful for pre-recorded content or testing without going live -- The branding flash carries into local recordings (free tier billboard extends to recordings) - ---- - -## v1 execution order - -The tasks below are ordered by dependency and risk. Each task builds on the previous. - -1. ✅ **TASK 9.4** — Live chat (right panel) — wire `YouTubeChatService.Start()`, parse messages, render in panel. Foundation for chat box source. -2. ✅ **TASK 3.18** — Chat box source — renders chat ON the stream. Depends on TASK 9.4 (same message parsing). -3. ✅ **TASK 10** — Polar billing — license key entry + watermark toggle + Velopack auto-updates (steps 1-7 shipped; Velopack update URL pending). -4. ✅ **TASK 19/23** — Control Surface UX — director's control room: thumbnails above central monitor, transitions (Cut/Fade/Move), edit mode offline only, left panel two-state (layers/props ↔ chat), right panel eliminated. Verified shipped 2026-08-24. -5. ✅ **TASK 20** — Hotkeys — global keyboard shortcuts. Shipped 2026-08-26: F1-F9 defaults + config UI with modifier chords, persistence, conflict detection, unbinding. -5b. ✅ **Broadcast metadata pull-out + launch geometry** — shipped 2026-08-24 (row 29 above). Live-screen "Text" tab → broadcast form, persistence + remote update; window default/minimums bumped (1920×1040 / 1366×768) with size+position restore. -6. ✅ **TASK 17** — Web source (WebView2) — shipped 2026-08-28; enables alert ecosystem. -7. ✅ **TASK 18** — Local recording — shipped 2026-08-29 (running-app verification pending in handoff). -8. 🔶 **TASK 21** — Media source — video file playback for non-Live scenes. **Remaining: the UI picker slice** (handoff spec in HANDOFF.md). -9. ✅ **TASK 22** — Audio sync offset — shipped 2026-08-31 (status reconciled 2026-09-01). -10. ✅ **TASK 15** — Stock bg images — all 5 scenes seeded (Starting, BRB, Live, Chat, Ending). `EnsureDefaultBackdrop()` runs on every scene switch. -11. **TASK 32** — Stream resilience (blip retry → grace countdown on the ON-AIR sign → one-click Back on air → crash-safe recording → pre-flight). -12. **TASK 33** — Bandwidth auto step-down (depends on TASK 32's restart machinery). -13. **TASK 34** — Scheduled streams (Text-drawer schedule + adoption at Start). -14. **TASK 35** — Scene-linked audio ("scenes remember the room"). -15. **Text source (TASK 3 item 17)** + **the monetization-awareness chain + Alerts (TASK 10 related work → TASK 3 item 20)** — capture → report → journey → alerts, one slice per integration test. -16. **TASK 13** — Social media launch kit (MARCOM) — after features, before gold. -17. **TASK 36** — Gold pass, always last: visibility unlock + flash-live enable + screens/layers audit + native verification suite + expiry reminders + signing/installer/EULA/Velopack. - ---- - -## TASK 19 — Scene transitions → MERGED into TASK 19/23 (Control Surface UX) - -**Superseded by the Control Surface UX task.** Cut/Fade/Move transitions, thumbnails, edit mode, left panel two-state — all shipped as one cohesive feature. See the Control Surface UX section in `ai.md`. - -### Design decisions - -- **Cut is free** — instant switch, no blending, zero CPU cost. This is the default. -- **Fade is the minimum expectation** — 300ms crossfade is universal. Every streaming tool ships this. -- **Move is economical** — simple slide animation, no video decoding needed. Good middle ground. -- **Custom is premium** — media/stinger transitions require video playback. Heavy but expected by mid-tier streamers. -- Transitions happen in the compositor, not post-encode. The preview sees the same blend as the live output. - ---- - -## TASK 20 — Hotkeys (keyboard shortcuts) - -**Goal:** keyboard shortcuts for scene switching and common actions — the single biggest UX gap. - -### Status: ✅ Done — shipped 2026-08-26 (steps 1-2; all 9 checklist items complete; handoff flags it shipped) - -1. ✅ `GlobalHotkeyManager` service — `Services/GlobalHotkeys.cs`: registers OS-level hotkeys on the window HWND via `RegisterHotKey`/`WM_HOTKEY` (`IHotkeyRegistrar` seam for tests; `RegistrarOverride` mirrors `LayoutPathOverride`). Wired in `MainWindow.OnSourceInitialized`. -2. ✅ Scene switching hotkeys — F1-F5 stage/transition the five canonical scenes. -3. ✅ Common action hotkeys — F6 start/end stream, F7 mute mic, F8 mute desktop audio, F9 TRAX play/pause. Dispatch lives in `MainViewModel.HandleHotkey` and honors command gating. -4. ✅ Hotkey configuration UI — `HotkeyConfigDialog` (XAML Window + `HotkeyConfigViewModel`). Click-to-capture, "Unbind" per row, Reset to Defaults. Conflict detection shows a warning; user must unbind the conflicting action first. Modifier chords supported from day 1 (Ctrl+F10, Alt+Shift+F12, etc.). -5. ✅ Global hotkeys — work even when app is not focused (that is what `RegisterHotKey` gives us) -6. ✅ Conflict detection — warns when a chord is already bound to another action; user must unbind first. -7. ✅ Persistence — `LoadHotkeyBindings()` / `SaveHotkeyBindings()` in `LayoutStore`, stored as `"modifiers+vk"` in Settings table. No schema migration needed. -8. ✅ Test: `GlobalHotkeyTests.WmHotkey_StagesScene_And_TogglesMic_And_RevokesOnClose` — real window + fake registrar, a genuine `WM_HOTKEY` posted through the window's own `HwndSource` hook stages the mapped scene and flips mic mute; close revokes every registration. -9. ✅ Test: `HotkeyConfigTests` — round-trip persistence, display string formatting, storage serialization. - -### Design decisions - -- **Global hotkeys are mandatory** — streamers are in-game and cannot alt-tab. F1-F5 must work from anywhere. -- **Modifier chords from day 1** — Ctrl+F10, Alt+F1, etc. for Stream Deck support and power users. -- **Conflict = warning, not swap** — user must unbind the conflicting action before reassigning. No surprise reassignments. -- **Unbinding** — every action can be set to "None" (no hotkey). Explicitly unbound actions are absent from the DB. -- **No Stream Deck yet** — bare keyboard first. Stream Deck support (physical devices) is v1.1+. - ---- - -## TASK 21 — Media source (video file playback) - -**Goal:** play video files (MP4, MOV, AVI) into scenes — starting soon videos, BRB loops, intro/outro clips. - -### Status: 🔶 In progress — Increment A (model + persistence) + Increment B (decoder) + slice 1 (manager + resolver/preview wire-in) + slice 2a/2b (ffprobe probe + native-FPS pacing) + slice 3 (loop mechanism) shipped. **Remaining: the UI picker slice (AddMedia command + file dialog + Acquire/Release wiring + loop-flag wiring) — GUI, handed off to build/verify natively on Windows (see HANDOFF → "UI picker slice — handoff spec").** - -1. ✅ `MediaSourceType` enum: `Video`, `Audio` (audio-only files via media source) -2. ✅ `SourceType.MediaSource` addition to the enum -3. ✅ `MediaSourceModel`: `MediaPath`, `MediaIsLooping`, `MediaVolume` (0-1), `MediaPlaybackState` — persisted in LayoutStore (schema migration + SELECT/INSERT + round-trip test) -4. ✅ `MediaVideoSource` implements `IMediaFrameSource`: FFmpeg raw video decoder → `VideoFrame` pipeline — spawns ffmpeg `-f rawvideo -pix_fmt bgra`, drains the pipe via pure `RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raises `FrameAvailable`/`Completed`; lifecycle is `StartAsync`/`StopAsync`; decode process behind `IDecodeProcess`/`FfmpegDecodeProcess` seam (binary-stdout mirror of `IEncoderProcess`). Shipped 2026-08-31; refactored onto `IMediaFrameSource` with this slice. -4b. ✅ Slice 2a — native-FPS probe seam: `FfmpegFrameRateParser` (pure, prefers `avg_frame_rate=` then `r_frame_rate=`, rational N/N/M, unknown→null) + `IFrameRateProbe`/`FfmpegFrameRateProbe` (derives sibling `ffprobe.exe` from the located ffmpeg dir, reuses the `IDecodeProcess` seam; null if ffprobe absent) + `FfmpegLocator.ProbeFileName` now also extracts `ffprobe.exe` from the pinned archive (conditional). Tests: `FfmpegFrameRateParserTests` (6 pure units + 1 probe integration via fake locator/process) + `FfmpegLocatorTests` still green — 15/15. -5. ✅ `IMediaFrameSource` + `MediaVideoSourceManager` (`Services/IMediaFrameSource.cs`, `Services/MediaVideoSourceManager.cs`): app-wide decode-session ownership refcounted by `MediaPath` with a `Func` factory seam; `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`; coalesces frames onto the UI dispatcher onto a single shared `WriteableBitmap` per file (mirror of `ScreenCaptureManager`); `MediaFailed` + `PreviewBitmapChanged`. Unit tests + one integration test (bitmap share/coalesce) — `MediaVideoSourceManagerTests`, 5/5 pass. -6. ✅ Native-FPS pacing (slice 2b): `MediaVideoSource` takes optional `IFrameRateProbe?` + `Func? delay` seams, probes FPS once in `RunAsync`, and delays by 1/fps after each emitted frame; unknown/absent probe → no pacing. Test: `MediaVideoSource_PacesFramesByProbedFps` (fake probe + recording delay, one delay per frame ≈ 1ms). -7. ☐ Wire into `FramePump` resolver — `Source { Type: MediaSource }` → latest video frame — **resolver + preview routing + manager wired; session acquisition (start/stop on add/remove) still open (comes with the UI picker)** -8. ☐ Wire into `SceneCompositor` — render media source as an image element at its position/size -9. ✅ Loop control (mechanism) — `IMediaFrameSource.Looping`; `MediaVideoSource` takes a `Func` process factory and restarts the decode on natural EOF when `Looping` (fresh process per pass, since a `Process` can't be re-`Start()`ed). Test: `MediaVideoSource_LoopsUntilLoopDisabled` (single frame re-emits across passes, `Completed` only after loop cleared). Wiring `Source.MediaIsLooping` into the flag lands with the UI-picker (acquisition) slice. -10. ☐ Volume control — per-source volume slider for audio playback -11. ☐ UI: file picker (filtered to video formats), loop toggle, volume slider -12. ☐ Schema migration for media source settings (file path, loop, volume) -13. ☐ Tests: video frame extraction, loop behavior, volume scaling, file validation - -### Design decisions - -- **FFmpeg handles all formats** — no codec-specific code. FFmpeg already in the project. -- **Audio plays through the desktop channel** — media source audio is captured by the WASAPI loopback (like TRAX/game audio). No separate audio routing needed. -- **This pairs with TRAX** — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished. -- **Not a full NLE** — no trimming, no multi-track, no effects. Just "play this video in the scene." - -### UI picker slice — spec (imported from HANDOFF 2026-09-01 so it can't be lost by a rewrite) - -Nothing `AcquireAsync`s a media path yet, so **no frames flow in a running app** — decoder, manager, -pacing, and loop mechanism are shipped and unit-tested, but a media Source has no way to start a -session. Deliverables, in order: - -1. **File picker + add/remove commands** (mirror `MainViewModel.Trax.cs:68` `OpenFileDialog` usage). - A "Media" source action opens an `OpenFileDialog` filtered to video (mp4/mov/avi); on OK set - `Source.MediaPath`, add to scene, and **`AcquireAsync(MediaPath)`**; on remove - **`ReleaseAsync(MediaPath)`** (refcounted — not `ReleaseAllAsync`, unless the path leaves the - layout entirely). On `StagedScene` layout loads/teardown, release every media path no longer - present and acquire new ones so sessions track the live layout. -2. **Wire `Source.MediaIsLooping` → `IMediaFrameSource.Looping`** — needs a manager-level per-**path** - loop provider. Ambiguity to rule + record: sessions are refcounted per `MediaPath` and shared - across scenes, but `MediaIsLooping` is per-`Source`. Chosen rule: the path's session `Looping` = - **any** live reference has it on (shared behaviour). Record it in the slice commit. -3. **Loop toggle + volume slider** in the source context menu / inspector (items 10/11). -4. `MediaVolume`/`MediaPlaybackState` wiring — audio path for media is future work (desktop loopback - already carries it, TRAX-style; `MediaVolume` slider can drive the local file's audio separately - later — out of this slice). - -**Design notes to preserve:** `ResolveOutputFrame` reads `_mediaManager.GetLatestFrame(MediaPath)`; -`OnMediaPreviewBitmapChanged` adopts the shared `WriteableBitmap` per path; `MediaFailed` → -`OnMediaFailed` (currently `Debug.WriteLine` — surface in UI via toast, TASK 24 stack). The compositor -scales any frame size — media renders through the generic `frameFor(element)` path, no compositor -change. First GUI smoke test: add a short real mp4 on native Windows, confirm it plays and previews. - ---- - -## TASK 22 — Audio sync offset - -**Goal:** per-source audio delay compensation to prevent lip-sync drift from USB mics and capture cards. - -### Status: ✅ Done (shipped TASK 22, 2026-08-31) - -1. ☑ `AudioSyncOffsetMs` on `MainViewModel` (global, default 0, range 0..500 ms) — positive-only: OBS's documented fix delays the audio so it lands on video when it runs ahead; advancing audio would need a video-side delay (out of the audio layer's scope, v1.1+). -2. ☑ Applied in `AudioMixer` — post-mix interleaved-stereo delay via the pure `AudioSyncDelay` line (`Services/Audio/AudioSyncDelay.cs`), fed through a `Func syncOffsetMs` seam each mix tick. -3. ☑ UI: compact "SYNC" slider (0..500) on the mic bar with a status dot (green = no-op, amber = offset set) via `IntToSyncBrushConverter`. **Provenance (recorded 2026-09-01 after a creator "I never ordered this" scare): the creator explicitly asked for OBS's delay-filter lip-sync fix BUILT NATIVELY — the feature is his, not AI drift. Its placement (preview-rail vs mic-settings/gear) remains an open design question; do not remove the capability.** -4. ☑ Persisted in `LayoutStore.Settings` (`Audio.SyncOffsetMs`) via `LoadAudioSyncOffsetMs`/`SaveAudioSyncOffsetMs`; saved from `SaveLayoutNow`. -5. ☑ Tests: `AudioSyncDelayTests` — zero-delay identity, negative→0 clamp, >500 ms clamp to 500 ms, and 10 ms → 960 interleaved-sample shift. -6. ⚠ **Post-ship regression (found 2026-09-01, first real launch):** this task's sync-delay line `_delayedMix` was declared nullable and never initialized — the first `delayed.Length` deref NRE'd EVERY live-mix tick, silently killing all live/record audio, hanging `AudioPipelineTests`, and being mislabeled a "known failure". Fixed in the recording-verification pass (init + null-check + throttled loop errors logged with stack); `AudioPipelineTests` 25/25 green afterward. Lesson recorded in MyMistakes. - -### Design decisions - -- **Global offset first** — one setting for all audio sources. Per-source is v1.1+. -- **Positive-only (delay audio)** — the physically-correct direction (audio runs ahead of the video). True "advance" needs a video-side delay and is PERMANENTLY OUT with per-source sync (TASKS.md → "Out of product" — v1.x phrasing retired 2026-09-01). -- **Simple slider** — 0 to +500 ms, default 0. No numeric input needed. -- **Visual feedback** — "sync OK" status dot shows when an offset is dialled in. - ---- - -## TASK 23 — Studio mode → MERGED into TASK 19/23 (Control Surface UX) - -**Superseded by the Control Surface UX task.** Studio mode is now the entire app's paradigm — the director's control surface with 5 thumbnails + central monitor. See the Control Surface UX section in `ai.md`. - ---- - -## TASK 24 — Toast notifications (in-app, non-blocking) - -### Status: ✅ Done (shipped 2026-08-23, branch `task24-notifications`) - -Replace blocking MessageBoxes with in-window toasts; promote actionable log-only failures. - -### Requirements: -1. Library: **Notification.Wpf 11.0.0** (Platonenkov fork of Federerer/Notifications.Wpf) — MIT, active, targets net8.0-windows. Notice #10 added to `THIRD-PARTY-NOTICES.txt`. -2. Placement: bottom-right above the footer (`NotificationArea x:Name="ToastArea"`, last child of MainWindow's root grid → topmost z-order, `Grid.RowSpan=4`, `Margin="0,0,12,96"`, outside the preview Viewbox), max 4 stacked. -3. Behavior: Info ~4s / Success ~4s / Warning ~8s auto-dismiss; **Error sticky until dismissed**. -4. Styling: dark card tints matching the health-banner language (slate `#3a3b52` info, green `#1d5c38` success, amber `#b8860b` warning, dark-red `#8f1f1f` error); corner radius 8; keep-visible-on-hover. -5. Threading: all Show calls marshal via a Dispatcher captured at construction (encoder/pump events fire off-thread). - -### Migrations (MessageBox → toast): -- Webcam acquire failure ×2 (`AddWebcamToStagedSceneAsync`, `SwapWebcamIdentityAsync`) → Warning -- Image read failure (`PickImageBytes`) → Warning -- Sign-in unsuccessful → Warning; sign-in failed → Error - -### Promotions (log-only → toast, each keeps its AppLog line): -- Go-live prep ×3: reusable stream unavailable / broadcast insert failed / prep exception → Error -- Frame-pump death while live (`OnFramePumpFailed`) → Error -- Mic missing at startup (`StartMicCaptureAsync`) → Warning (once) -- Premium lapse at startup re-validation → Warning -- Saved-session refresh failure (signed out notice) → Info - -Deliberately left alone: offline license re-validation skip (log-only — would spam every launch); SocialsDialog slot-delete YesNo confirm (stays modal). - -### Tests: -- Unit: pure severity→request mapping (`NotificationServiceTests`) — area routing, lifetimes, tints. -- Integration (the ONE): real window-hosted `ToastArea` + real service — Info auto-dismisses, Error sticks (`NotificationAreaIntegrationTests`, RealApp collection). - -### Library facts (cost a hunt — see also ai.md): -- Area routing matches the area's XAML `Name` against the request's `AreaName`; unknown name silently drops. -- `NeverExpires()` = `ExpirationTime = TimeSpan.MaxValue` (not null); `NotificationColor.ToHex()` returns `#AARRGGBB`. -- Overlay-window shutdown caveat avoided entirely by using the in-window area mode. - ---- - -## TASK 25 — Background consolidation: one locked Background per screen + mini-view rule - -### Status: ✅ Done (shipped 2026-08-23, branch `task25-backgrounds`) - -The creator rejected the multi-concept background model ("why are there multiple background -things?"). Locked model, verbatim intent: - -1. **Exactly ONE background per screen**, named "Background", at position 0. -2. It cannot be re-ordered or deleted; **no Background/Screen item in the (+) menu**. -3. Context-sensitivity exists **only on the Live screen** (Show Desktop toggle + monitor - switching — behavior kept intact). -4. **Exactly five default background images** live in the DB; stale rows are cleaned up. - -### What shipped: -- **Seeder consolidation:** the five near-clone seeders (`Seed{Starting,Brb,Ending,Chat}Background`, - `SeedLiveBackgroundAsset`, `EnsureDefaultBackground`) are gone. One path now: - `EnsureBackground(scene)` → `CreateBackground(name)` — Live = `DisplayCapture` row (capture - machinery untouched), everything else = static `Background` art row; both named "Background". -- **Heal on every load:** `NormalizeBackgrounds(scenes)` keeps the correctly-flavored row, - converts a wrong-flavor survivor in place (`Source.Type`'s setter derives `IsBackground` — - conversions must re-assert the flag), drops duplicates, seeds missing ones, renames, pins to - index 0; non-canonical scenes lose backgrounds + flag. `HealBackgrounds()` stamps default art - (`Assets/{scene}-background.jpg` via `AddAsset`; custom Browse art wins). This fixed the real DB's - rot: four scenes carried a stray "Game Capture" duplicate beside their static row, and Settings - held 16 orphaned `BackgroundUseDefault_{guid}` keys. -- **Settings purge:** `LayoutStore.Save` deletes `BackgroundUseDefault_{id}` / `BackgroundPath_{id}` - keys whose id is no longer a Source row. -- **(+) menu:** Screen + Background items removed → Webcam/Image/Text/Countdown/Web/YouTube Chat; - `AddSource` also hard-refuses DisplayCapture/WindowCapture/Background parameters. -- **Capture controls Live-only:** `CanChangeBackground` = staged scene is Live. The preview - CanvasGrid menu (Show Desktop/Capture Desktop/Refresh Desktop) binds it directly; the layer-row - context menu MultiBindings it with the row's `IsBackground` through a new - `Helpers/AllTrueToVisibilityConverter`. Non-Live screens keep the Use-default pill + Browse. -- **Mini rule:** minis never render live captures. While Live is staged its mini shows the green - placeholder; unstaged it shows `live-background.jpg`. Fixed by making `LoadBackgroundImage` - flavor-blind (`IsBackground`) and gating staged-Live to the placeholder in `RefreshSnapshotsAsync`. - (Real-time rendering stays center-monitor-only — preview lag during live gameplay is a known - unsolved OS-level problem and must not be compounded.) - -### Tests (suite went 221/218 → 223/220; the two stale background-policy landmine tests healed here): -- Integration (the ONE): `BackgroundHealIntegrationTests` — seeds a dirty temp DB (duplicate rows, - misnamed Live row, orphaned keys), drives the real window, asserts one Background per scene at - index 0 with correct flavor/name, save purges the orphaned keys. -- Unit: SceneCatalogTests rewritten for NormalizeBackgrounds/EnsureBackground flavors; - BackgroundTests empty-scene flavor updated; SourceNamingTests excludes the always-present - Background from numbered-name expectations. - ---- - -## TASK 30 — Gear menu cleanup + Default Location for Recordings + preview layout restructure - -### Status: ✅ Done (shipped 2026-08-29, commits `6031a56` → `61b850f` → `455a409`) - -The creator's KISS review: gear menu cleanup, single Start button, record-location setting, -and a full layout restructure moving all live controls into the preview pane. - -### What shipped: - -**Gear menu cleanup:** -- Removed `Save Layout` / `Save Layout As…` / `Open Layout…` from Gear menu + dead VM - commands/methods (`SaveLayoutCommand`, `SaveLayoutAsCommand`, `OpenLayoutCommand`, - `SaveLayoutAs()`, `OpenLayoutFile()`). Multi-layout = use OBS. - -**Default Location for Recordings (App Settings):** -- First real setting panel in the Gear menu: "Default Location for Recordings". -- `DefaultRecordFolder()` (internal static): configured → `%USERPROFILE%\Downloads` → `MyVideos`. -- `RecordFolderDisplay` property, `ResetRecordFolderCommand`, Browse + "Use Downloads" reset. -- `TextBox` binding fixed to `Mode=OneWay` (get-only computed property). -- ONE fallback test: `DefaultRecordFolder_Falls_Back_To_Downloads`. - -**Single Start button:** -- Button `Content="Start"` hardcoded in XAML. Removed `PrimaryButtonText` / `IsStreamingStart` - properties + all `OnPropertyChanged` references. Pills = intent, lights = state, button = trigger. - -**Preview layout restructure — all live controls under preview:** -- Socials button moved from footer (scenes/sources column) to preview bottom row, left of - TRAX button. `MouseEnter`/`MouseLeave` handlers pulse the `SocialBarElement` - `DropShadowEffect` (sine wave BlurRadius 24→48, Opacity 0.7→1.0 at 30ms tick). -- **Mic controls moved from footer into preview pane** (vertical StackPanel below Socials+TRAX): - MIC button + sound meter + mute grid + volume slider — the same footer controls, relocated. -- Preview bottom row: vertical `StackPanel Grid.Row="2"` → Line 1 (Socials + TRAX), Line 2 - (MIC + meter + mute + slider). All centered. - -**Bottom bar restructured:** -- Reduced to single row (mic row removed), tighter padding (`Padding="16,4"`). -- Gear icon moved to far left. Stream stats (Bitrate/FPS/Dropped/Duration) centered under - preview pane. Resolution dropdown stays far right. - -**Avatar fix:** -- Avatar `BitmapImage` no longer loads in the constructor (stale image from last session). -- Only loads when `IsConnected` fires `true` via `OnViewModelPropertyChanged`. - -### Tests: -- Build 0 warnings, 247 tests (2 pre-existing known failures excluded). - ---- - -## TASK 31 — SceneGraph component + static/dynamic compositor optimization - -**Goal:** extract the scene collection, element inventory, and mutation surface into a standalone -`SceneGraph` component. Classify elements as static or dynamic. Enable the compositor to bake -static layers once and only recompose dynamic layers per frame. - -### Status: ✅ Done (2026-08-31) - -What landed — core optimization + SceneGraph component, all in one commit: - -- `ElementKind` (Static/Dynamic) on `SceneElement` base, overridden in `Source` and `WebcamSceneConfig` -- `Services/SceneGraph.cs`: owns `Scenes` collection (ViewModel's `Scenes` property delegates to it); - mutation surface `AddElement`/`InsertElement`/`RemoveElement`/`MoveElement` (each invalidates the bake); - queries `GetBackground`/`GetWebcam`/`GetChatBoxes`/`GetSplitPoint`/`IsStatic`; and the - `GetBakedBase` bake-cache (keyed by scene id + static element identities below the split) -- `SceneCompositor.BakeStaticBase`/`CompositeLayers` + `Render(.., staticBase, split)` overload — - split-aware, builds the cached base in source-rect space, composites dynamic/above-split per frame -- `FramePump` takes an optional `SceneGraph` and uses the optimized path when wired (falls back to full render) -- `MainViewModel`: routes element mutations through the graph; invalidates the bake on static - layout/opacity/visibility/useDefaultBackground changes and after background heal / EnsureBackground -- Integration test `SceneGraphTests.BakedStaticBase_WithDynamicLayer_CompositesCorrectly` (audio-free, - verifies bake-once + cache hit + dynamic-on-top pixels + static-mutation invalidation + - dynamic-only non-invalidation) - -**Defensive deviations from this spec (decision 2026-08-31):** -1. `ChatOverlayLayer` keeps taking `IEnumerable` rather than `SceneGraph.GetChatBoxes()` — it is - deliberately decoupled from the graph (`// without owning the scene graph`). Forcing the graph in would - couple a WPF-bound layer to it and violate that documented seam. -2. Background static helpers (`EnsureBackground`/`CreateBackground`/`NormalizeBackgrounds`) stay on the - ViewModel because `BackgroundTests.cs` unit-tests `MainViewModel.EnsureBackground` directly. Queries - moved to SceneGraph; helpers + their invalidation wiring stayed on the VM. -3. The full ViewModel-as-facade extraction (moving the ~all-binding-surface off Scenes/StagedScene/ - LiveScene) was deliberately NOT done in this unsupervised pass — it is the "touch 11 files across 4 - layers" regression risk the handoff flagged. SceneGraph owns the collection + mutation surface now; - the remaining VM call sites binding to those still work because `Scenes` delegates. Revisit after 1.0. - -**Verification:** SceneCompositorTests 4, StretchMathTests 4, BackgroundTests 16, SceneCatalogTests 18, -LayoutStorePersistenceTests 12, FramePumpTests 9, SceneGraphTests 1 all green. RealAppHost GUI/collection -tests (SourceNaming, RoundClip, BackgroundHeal, ...) construct a real `MainWindow` — **CORRECTED -2026-09-01:** they DO run from WSL when invoked per-class through the Windows `dotnet.exe` vstest host -(the old "cannot run headless" claim conflated them with the full-suite WASAPI hang). Two first-launch -crashes this refactor shipped with were caught on the first real native launch and fixed same day: -ctor-order `SceneGraph` NRE (`null!` field assigned after first ctor use — now field-initialized) and -window-scope `EyeButton`/`EyeIconStyle` consumed via `StaticResource` from the extracted `LeftPanel` -(UserControl namescopes can't see window resources — styles moved to `Themes/Controls.xaml`, the -app-scope rule honored). The RoundClip "known failure" was then root-caused to stale test code -(`window.FindName` across namescopes + `VisualTreeHelper.HitTest` where `UIElement.InputHitTest` -models input) — test green 2026-09-01, the sole remaining known failure is the audio one. - -### Design - -**Element classification:** - -Every `SceneElement` exposes `ElementKind Kind` — `Static` or `Dynamic`: - -| Type | Kind | Why | -|------|------|-----| -| `Source` (Background art) | Static | Pixels don't change at runtime | -| `Source` (Image) | Static | Pixels don't change at runtime | -| `Source` (DisplayCapture) | Dynamic | Live game/desktop feed | -| `Source` (ChatBox) | Dynamic | Live chat messages arrive continuously | -| `Source` (WebSource) | Dynamic | WebView content can change | -| `WebcamSceneConfig` | Dynamic | Camera feed, changes every frame | - -**Split point:** - -The **split point** is the index of the first dynamic element in a scene's z-ordered `Elements`. -Everything below it = baked base. Everything from it upward (including static layers above dynamic -elements) = composited per frame. If zero dynamic elements exist: entire scene is baked, -compositor skipped entirely. - -**BakedSceneCache:** - -``` -BakedSceneCache { - VideoFrame BaseFrame // composited static layers below split point - int Version // incremented on mutation - SceneElement[] BakedElements // for invalidation tracking -} -``` - -Invalidation triggers (re-bake): element added/removed/reordered, static element moved/resized/ -opacity changed, `IsVisible` toggled on a static element, background asset changed, scene switch. - -No invalidation when: only a dynamic element's pixels changed (webcam frame, chat message), -or a dynamic element moved/resized (per-frame compositing concern, not a base rebake). - -**Compositor integration:** - -``` -Render(scene, frameFor, options): - 1. Find split point (first Dynamic element index) - 2. If split == scene.Elements.Count: - → return cached base (or bake if stale) - 3. Otherwise: - → start from cached base (or bake if stale) - → composite elements[split..] on top using frameFor -``` - -**SceneGraph interface:** - -``` -SceneGraph { - // Collection - ObservableCollection Scenes - Scene? StagedScene - Scene? LiveScene - - // Mutation surface (single owner of element operations) - AddElement(scene, element) - RemoveElement(scene, element) - MoveElement(scene, element, newIndex) - - // Queries (replace scattered LINQ) - GetBackground(scene): Source? - GetWebcam(scene): WebcamSceneConfig? - GetChatBoxes(): IEnumerable - GetSplitPoint(scene): int - IsStatic(scene): bool - - // Events - SceneChanged - ElementAdded/Removed/Moved - SplitPointChanged -} -``` - -**What moves into SceneGraph:** - -| Current location | Moves to | -|-----------------|----------| -| `MainViewModel.Scenes.cs` — Scenes collection, StagedScene, scene switching | `SceneGraph` | -| `MainViewModel.Background.cs` — `NormalizeBackgrounds`, `EnsureBackground`, background queries | `SceneGraph` (background helpers) | -| `MainViewModel.Sources.cs` — `AddSource`, `RemoveElement`, element inventory | `SceneGraph` (mutation surface) | -| `MainViewModel.Webcam.cs` — webcam queries across scenes | `SceneGraph.GetWebcam(scene)` | -| `ChatOverlayLayer` — `scenes.SelectMany(...).OfType().Where(ChatBox)` | `SceneGraph.GetChatBoxes()` | - -**What stays on the ViewModel:** - -The binding surface — `StagedScene` setter still raises `OnPropertyChanged` for -`ShowEmptySceneHint`, `CanAddWebcam`, etc. But now it delegates to `SceneGraph` for actual -state queries. ViewModel becomes a thin binding facade over the SceneGraph (same pattern as -`MainViewModel.Chat.cs` over `ChatOverlayLayer`). - -The compositor resolver (`frameFor` callback) stays in the ViewModel/encoder layer — it bridges -WPF concepts (CameraManager, ScreenCaptureManager, ImageCache) into the pure `VideoFrame` seam. -SceneGraph doesn't know about capture sessions; it just knows element kinds. - -**Migration path:** - -1. Extract `SceneGraph` as a standalone class in `Services/` -2. Move `Scenes` collection + `StagedScene`/`LiveScene` + wiring -3. Move element mutation surface (add/remove/move) -4. Move background normalization/queries -5. Add `ElementKind` to `SceneElement` base -6. Add `GetSplitPoint` + `BakedSceneCache` -7. Update compositor to use split point -8. Wire ViewModel as thin facade -9. One integration test: baked static base + dynamic layer composite - -### Tests - -ONE integration test: `SceneGraphTests.BakedStaticBase_WithDynamicLayer_CompositesCorrectly` -— a scene with static background + image + dynamic webcam; assert the baked base is cached -(renders once), dynamic layer composited on top, static element mutation invalidates the cache, -dynamic-only pixel change does not. - ---- - -## TASK 32 — Stream resilience (queued 2026-09-01 — merges the orphaned TASK 2 resume clause into the pills/signs architecture) - -**Goal:** a dropped stream heals itself when physically possible, tells the truth when it isn't, and -gets the creator back on air in one click. The ON-AIR sign + primary button in the top bar are the -surface — they already mean "reality" (pills = intent, lights = reality, TASK 18/30); reality just -isn't two-state. Resumption of a completed broadcast is explicitly **out of scope** (impossible via -the API — creator ruling 2026-09-01): no lying in the copy. - -**Slices (one integration test each, in order):** - -1. ☐ **Blip retry** — app-side encoder restart: push dies while live (ProcessFailed / stdin - backpressure death) → bounded-backoff relaunch (2s/5s/10s, N attempts) against the cached - constant reusable-stream ingest URL. Success = push running again while the broadcast is still - live; viewers never know. Fixes the map's stale "ffmpeg does reconnect" claim (corrected - 2026-09-01: reconnect flags are input-side only). Test: fake process fails once then starts → - pump recovers without surfacing to the VM. -2. ☐ **Measured grace + the sign** — while the push is dead, escalate the existing health poll - (30s → ~5s). ON-AIR dot goes **amber "RECONNECTING"** with a mm:ss countdown to the observed - YouTube cutoff; the grace length is a **measured, pinned constant** — probe it on a real private - test stream, cite the observation (spin-guard rule; do NOT invent a number). Poll reports the - broadcast `complete` → dot **red "OFF AIR"**, primary button relabels **"● Back on air"** and - pulses. Test: fake status provider drives the dot through live → reconnecting → cut-off. -3. ☐ **One-click Back on air** — the relabeled button fires the existing go-live path with saved - `Broadcast.*` + cached stream, dialog skipped: new broadcast, new VOD (accepted cost, stated - in-product). Test: seeded saved form + fake stream service → click creates + pushes, no dialog. -4. ☐ **Crash-safe recording** — fragmented MP4 on the record block (`-movflags - frag_keyframe+empty_moov+default_base_moof`): plain MP4's trailing moov atom means a power loss - kills the WHOLE file. Rename-on-stop unchanged. Closes the TASK 18 "MKV as an option" design-note - remnant (fragmented MP4 chosen over MKV: single container, YouTube-upload-native — verify player/ - editor tolerance, cite the research). Test: `FfmpegArgs` emits the movflags only on the record block. -5. ☐ **Pre-flight** — before broadcast insert, one TCP probe to the ingest host:1935 (~2s timeout). - Unreachable → Error toast, no insert, no ghost broadcast ("check VPN/network"). The "forgot the - VPN" story, solved as prevention instead of recovery. Test: fake probe failure aborts go-live before any API call. - -### Design decisions - -- **The sign is the status, the button is the action** — no new chrome, no log lines, no spinners. -- **Honest windows:** countdown = observed grace; at zero we say OFF AIR, we don't pretend to reconnect. -- **Why us > incumbents:** constant reusable stream + persisted form make retry/restart cheap; OBS - shows a reconnecting log line and Streamlabs silently dies. - ---- - -## TASK 33 — Bandwidth auto step-down (queued 2026-09-01; v1 per the complete-v1 ruling) - -**Goal:** when the upload can't hold the tier, drop it automatically instead of drowning viewers in -freeze frames. The `variable` reusable stream (shipped) makes this zero-API; the governor that decides -WHEN is what TASKS.md:25 always claimed existed and was never built. - -1. ☐ Pure `StepDownPolicy` — sliding window over the encoder's reported dropped-frame rate - (`StreamHealth`, already parsed): sustained overrun → downgrade exactly one 16:9 tier - (1080p60 → 1080p30 → 720p60 → 720p30). Hysteresis cooldown; **one-way within a broadcast** - (no oscillation — a recovered connection doesn't climb back mid-show; the creator may manually - step up). Vertical tier stays a manual choice. -2. ☐ Restart machinery = TASK 32 slice 1 (tier change is an encoder relaunch at new W×H/FPS over the - same pipe/URL; the master canvas is 1920×1080 regardless, so geometry never rewrites). -3. ☐ Surface: the resolution badge + dropdown update to the landed tier; ONE Info toast - ("Quality lowered to hold your stream") — no nagging after that. -4. ☐ Tests: pure-policy units (drop-rate windows, one-way, cooldown) + the ONE integration: fake - health stream with sustained drops → exactly one lower-tier encoder restart. - ---- - -## TASK 34 — Scheduled streams, Text-drawer version (queued 2026-09-01; option (b) with the creator's placement) - -**Scope ruling (2026-09-01):** this is the whole scheduling story — announce a LIVE show (countdown -watch-page + subscriber notifications), then the creator shows up and pushes. "Going live is our -schedule." Scheduled *playout* of pre-recorded files: discussed and DECLINED by the creator. -Premiere/upload scheduling: out of product (see closed list). - -**Goal:** announce "Friday 8pm" from inside the app — YouTube notifies subscribers and runs the -channel countdown; come Friday, one click goes live into the scheduled broadcast. Scheduling is a -"my channel" concern, so it lives in the always-visible **Text drawer**, not the go-live modal. - -1. ☐ Drawer gains ☑ *Scheduled livestream* + date/time (replacing the read-only "Scheduled Start" - row). Checking + Save **creates** the broadcast: insert `liveBroadcasts` with a future - `scheduledStartTime`, bound to the cached reusable stream — the drawer's Save path grows its first - create operation (today it can only update an existing broadcast). -2. ☐ Invariant: **at most one open scheduled broadcast**; unchecking before airtime → - `liveBroadcasts.delete` (legal in created/ready). -3. ☐ **Adoption:** go-live checks for the open scheduled broadcast and pushes into it instead of - inserting a new one. Mid-stream title/desc edits ride the existing Update path (unchanged). -4. ☐ `LiveBroadcastFormViewModel` owns `IsScheduled` + `ScheduledStart` (nullable), persisted with the - other `Broadcast.*` keys; local time in, RFC3339 offset on the wire — the timezone edge is where the - bugs will live: test it (DST boundary, non-UTC creator). -5. ☐ **Research flags (cite before building — spin guard):** (a) pushing *before* the scheduled start - with `enableAutoStart` — live immediately or held? Decides a "starting early — it goes live NOW" - confirmation. (b) delete semantics for a ready broadcast bound to a reusable stream. -6. ☐ Test (the ONE): schedule → persists + inserts (fake service captures a future scheduledStartTime - + boundStreamId); start → adopts instead of inserting; uncheck → delete called. -- **Test-phase note:** dark by design while the visibility lock holds (Private notifies nobody); the - creator can still exercise insert/adopt/cancel. Full shine arrives with TASK 36's unlock — pleasing - coupling, deliberate. - ---- - -## TASK 35 — Scene-linked audio: "scenes remember the room" (queued 2026-09-01; spec approved by the creator same day) - -**Goal:** the BRB scene mutes the mic; the Chat scene brings it back. The creator stops fiddling with -audio at transitions — the scene carries the room state. OBS needs scripts/plugins; native = the -creator-proof pitch. - -1. ☐ `Scene` gains nullable `SceneMutesMic` / `SceneMutesDesktop` — `null` = don't touch (the default; - scenes are passive until told). Schema column bump + roundtrip. -2. ☐ Applied on **live transitions only** (thumbnail click / hotkey), ONCE at entry — never on - offline staging (no surprise stream-state edits while auditioning). -3. ☐ The creator is boss: a manual mute/unmute mid-scene is never overridden until the next entry. -4. ☐ BRB ships `SceneMutesMic = true`; other scenes default `null` — creator's choice. -5. ☐ Edit UI: two checkboxes in the scene's Properties section (left panel, edit mode) — no new chrome. -6. ☐ Test (the ONE): transition applies scene state; `null` leaves state untouched; manual override - survives until the next entry. - ---- - -## TASK 36 — Gold pass (queued 2026-09-01; the going-gate, always the last work unit) - -**Goal:** the single deliberate pass that turns the dev product into the shipped product. Nothing here -may be done opportunistically mid-development — each item has a lock/comment pointing at this task. - -1. ☐ **Visibility unlock** — remove the `CreateBroadcast` Private override (TASK 9 item 6); dialog - selection rules. The channel-protection stance ends at GA, by hand, here. -2. ☐ **Branding flash goes live** — end the preview-only pre-GA posture: composite the escalating - obnoxious-promo flash onto output + recordings for free users. Paid removes it; that stays the - ONLY paid delta. (Escalation curve finalized here, first.) -3. ☐ **Screens/layers settings audit** — the retired 2026-08-22 landmine (ai.md): fine-tooth-comb - every screen's Background layer properties, pill persistence, context-menu visibility, (+) items. -4. ☐ **Native-Windows verification suite** — the checklist TASKS/HANDOFF never had: real webcam, - real mp4 decode/loop/pacing (TASK 21 picker must have landed), real recording file opens + - plays (fragmented MP4), a real private go-live end-to-end, GUI/RealApp test classes green on - the desktop (they've only ever run against fakes). -5. ☐ **License-expiry reminders** — T-14 / T-7 / T-1 toasts via the existing notification stack - ("renews — manage here" → `PremiumUrl`); lapsed: flash returns (the product itself is the - notice) + one Info toast; active paid users see ZERO license UI. Research: does Polar's - validate response carry `expires_at` reliably — cite before building. -6. ☐ **Release engineering** (the `Distribution.md` plan, 644 lines, finally tasked): code signing - (HARD blocker — an unsigned installer is a SmartScreen scarewall), installer + Velopack update URL - (TASK 10 item 8), EULA draft/review, THIRD-PARTY-NOTICES license-texts gate (TASK 4 req 9). - Build posture (agreed ceiling): compile flags max out at `HARDENED` (release-only anti-debug/ - tamper checks) + `MOCK_REWARDS` (fake reward payloads for Alerts/dev) — **additive-only**, never - branching normal code paths, both compiled every CI build so they can't rot. Obfuscation / - assembly-splitting stay GA-time decisions, not compile-time ones. - ---- - -## v1 = the finished product (creator ruling, 2026-09-01) - -**There is no v1.x.** v1 ships feature-complete: everything in the queue above lands, or is -explicitly buried below. The parking lot is closed — nothing is "deferred to a later version" -anymore; the only two states are **v1 task** and **out of product, permanently**. The 10% margin of -error is this list, bounded and written down — not vibes. - -**Monetization posture restated (what "final" means):** free gives everything; the only difference -is the branding flash. License activation flips exactly one bit (`IsPremium` → flash off). Nagware = -the escalating obnoxious self-promo flash itself — never modals, never feature-locks. Pre-GA the -flash is preview-only (see ai.md → Monetization; TASK 36 flips it live). - -## Out of product — permanently (the 10% margin, closed list — do not re-litigate) - -| Idea | Verdict & reason | -|------|------------------| -| Stream Deck / Loupedeck integration | The device sends keystrokes — **global hotkeys already ARE the integration** (TASK 20). Redundant, not deferred. | -| Per-source audio sync offset | Global offset shipped (TASK 22); per-source is the OBS mixer rabbit hole. Condemned by the Audio Assumption (one knob). | -| Profiles / presets (multi-config) | Same hydra as multi-layout, already ruled at TASK 30: **that's what OBS is for.** | -| Chroma key / background removal | Already ❌ (TASK 3 item 19). NVIDIA Broadcast does it free; model+GPU+lighting caveats = support hell for a solo dev. Stays buried. | -| Virtual camera output (Zoom/Discord) | Different product, device-driver swamp. Our output is one YouTube stream. | -| Replay buffer (instant replay) | Game-clipping culture, not our persona. **The most valuable corpse here** — if the list is ever reopened at all, this is candidate #1, and it reopens by creator ruling, not by AI suggestion. | -| Multi-destination restreaming | "Casual streamers outgrow LlamaCasty first" — the map already said it; the per-destination audio/monitoring expectations are a different product. | -| Stream clipping | YouTube killed clips; trimming local MP4 is an editor. | -| Advanced broadcast-form tab (latency/DVR/embed/projection/CC/region) | TASK 9 item 7 rescoped: fixed sane defaults, invisible. Every exposed field is a support ticket. | -| In-app bug-report mechanism | Never built, never will be: support = email + GitHub issues. (ai.md's support line corrected to match, 2026-09-01. Second-place corpse if the list ever reopens: with replay buffer.) | -| D3DImage/GPU preview compositor | TASK 3 item 16 superseded: XAML preview + software output compositor are the design; a D3D11 swap remains a seam-respecting possibility, not a feature. | -| Simultaneous record + stream | Creator ruling 2026-09-01: the VOD is the copy; dual encode drags mediocre hardware and degrades both outputs. "We're not them." Two modes, radio pills, one stop. | -| Premiere / video-upload pipeline | Uploading finished videos is Studio's job (quota-brutal, processing-state polling, API premiere support unproven). Watch-parties are fandom behavior, not our persona. "Going live is our schedule." | -| Scheduled playout (pre-recorded file airs as live) | Discussed fully 2026-09-01 and declined by the creator — "let's not." If it ever returns, it returns by ruling, with machine-on caveats and the media-audio slice as its price of admission. | - -*If a future session is tempted by anything on this list, the answer is already written. New ideas -must survive this page before they get a task number.* +- **D3D11 swap compositor** (superseded by software compositor; TASK 3 item 16) +- **Background removal** (ONNX/DirectML; TASK 3 item 19) +- **OBS-style source expansion** (game capture, browser source, media playlist, VLC, color-key, MIDI — the minimal source set is deliberate; creators who need more have graduated to OBS) +- **Multi-layout** (auto-save, single layout; multi-layout = OBS territory) +- **Cross-process stream resume** (API makes it impossible; creator ruling 2026-09-01) diff --git a/TASKS/research-youtube-api.md b/TASKS/research-youtube-api.md new file mode 100644 index 0000000..9b30ab8 --- /dev/null +++ b/TASKS/research-youtube-api.md @@ -0,0 +1,33 @@ +# ytLlive — YouTube Live API research facts (authoritative) + +> Catalog: [`TASKS.md`](../TASKS.md) · memory map: [`schema.md`](../schema.md). + +## YouTube Live API — research facts (authoritative, v3 build) + +Lifecycle: `created → ready → [testing] → live → complete` (transitional `liveStarting` / `testStarting`). + +1. **liveBroadcasts.insert** requires: `snippet.title`, `snippet.scheduledStartTime`, `status.privacyStatus`, `status.selfDeclaredMadeForKids` (COPPA). +2. **liveStreams.insert** requires: `snippet.title`, `cdn.frameRate`, `cdn.ingestionType`, `cdn.resolution`. **None of the four (except title) can ever change after creation** — changing them means delete + recreate the stream. This is the hard constraint behind the quality grey-out. +3. **Title / description / privacy**: editable at any time, including while live (`liveBroadcasts.update`, part=`snippet,status`). +4. **contentDetails** (DVR, recordFromStart, monitorStream, embed, latency): editable only in `created` / `ready`. +5. **Transition to live** only allowed when the bound stream's `status.streamStatus == active`. + +### Two features that reshape the design + +1. **enableAutoStart / enableAutoStop** — instant one-click go-live, no transition call. With `enableAutoStart=true` we never call `transition(live)`: the broadcast auto-goes-live the moment the encoder starts. Combined with `enableMonitorStream=false` (our preview pane replaces YouTube's monitor stream — the thing that forces a testing stage), the flow is **create → bind → Start Stream → encoder starts → YouTube brings it live**. No testing, no transition polling, no liveStarting stuck-state handling. +2. **`cdn.resolution=variable` / `cdn.frameRate=variable`** — free auto step-down. YouTube auto-detects what we send; since we ARE the encoder we can drop bitrate/resolution on the fly with zero API calls. Declaring an explicit resolution instead (e.g. 1080p) requires a new stream, which can't happen mid-broadcast. Variable is the enabler for the whole auto step-down feature. **(Claim-check 2026-09-01: the enabler is real, the governor is not built — the drop-side policy ships as TASK 33, v1 per the complete-v1 ruling.)** + +### Compliance gotchas (maps perfectly to report-by-exception) + +1. `liveStreams.status.healthStatus`: `good | ok | bad | noData` plus `configurationIssues[]` with `type` + `severity` (`info|warning|error`). Literally built for report-by-exception — poll it, render nothing on good/ok, surface a banner only on warning/error. No need to invent our own health logic. +2. Encoder must comply or YouTube flags it: keyframes ≤ 4s (`gopSizeLong`), closed GOP, H.264, audio AAC/MP3 @ 44.1/48kHz, mono/stereo only. +3. Error codes to handle: `errorStreamInactive`, `invalidTransition`, `redundantTransition`, `liveStreamDeletionNotAllowed`, `liveStreamModificationNotAllowed`, `liveBroadcastBindingNotAllowed`. + +### Tips we should take advantage of + +1. **Reusable streams** (`isReusable=true`): one stream per channel, cache its ingestion URL + stream name, reuse for every broadcast. No rebinding dance each go-live. This is exactly the manual-stream-key baseline. +2. **Backup ingestion address**: YouTube provides a simultaneous-push backup — future hardening, not v1. +3. **`recordFromStart` + `enableDvr` default true** → every live is auto-recorded and immediately replayable. Free VOD archive, matches the v0.2 recording goal. +4. **`latencyPreference`: `normal | low | ultraLow`** — for homelab streamers talking to chat, `low` (or `ultraLow`, capped at 1080p) is a real feature. +5. **Broadcast ID == Video ID** — one ID to track everything. + diff --git a/TASKS/task-01-initial-scaffold.md b/TASKS/task-01-initial-scaffold.md new file mode 100644 index 0000000..0a19ee1 --- /dev/null +++ b/TASKS/task-01-initial-scaffold.md @@ -0,0 +1,15 @@ +# TASK 1 — Initial Scaffold + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** Working C# / WPF project with MVVM architecture, dark-theme main window, and YouTube service stubs. + +### Status: ✅ Done + +1. ✅ Models: Scene, Source, StreamConfig, StreamHealth, YouTubeChannel, ChatMessage +2. ✅ Services: YouTubeAuthService (OAuth2), YouTubeStreamService (broadcast/health), YouTubeChatService (chat polling) +3. ✅ MainViewModel: scene management, stream controls, chat +4. ✅ MainWindow: scene/source panel, preview area, chat panel, status bar +5. ✅ Clean build, 0 warnings (WSL + Windows) + diff --git a/TASKS/task-02-oauth2-auth.md b/TASKS/task-02-oauth2-auth.md new file mode 100644 index 0000000..548dedd --- /dev/null +++ b/TASKS/task-02-oauth2-auth.md @@ -0,0 +1,37 @@ +# TASK 2 — YouTube OAuth2 Authentication + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** Fully working Google OAuth2 flow — user clicks "YouTube", browser opens, authorization callback lands, channel info is stored. + +### Status: ✅ Done + +1. ✅ Two-state Start/End Stream button, go-live dialog (account + title/description/visibility), red top bar, pulsing LIVE badge + elapsed timer, preview glow, taskbar red dot +2. ✅ Real OAuth2 wiring — baked-in Google credentials (desktop client; loopback callback) + `YouTubeAuthService` complete: browser launch, `HttpListener` callback, token exchange, refresh, channel fetch +3. ✅ Token persistence via Windows DPAPI (`Helpers/TokenStore.cs` → `%APPDATA%\ytLlive\ytLlive.auth`), best-effort reload + proactive refresh at startup, saved after every exchange/refresh +4. ✅ Account sign-in/change surfaced in the GoLive dialog (saved account shown with "Change Account"; "Sign in to YouTube" when none; Start disabled until signed in) +5. ✅ ~~**End Livestream signs out**~~ **SUPERSEDED by TASK 18 (2026-08-29): explicit sign-out only** — `StopStream()` no longer clears the session (sign out via Logout / Change Account, `SignOutYouTubeAsync`); stopping a stream or recording leaves the creator signed in. Originally shipped as: a graceful end completes the session: `StopStream()` calls `YouTubeAuthService.ClearSession()` + `TokenStore.Clear()` + `IsConnected = false`, so the next Start Stream dialog requires a fresh sign-in. A crash never runs End, so the DPAPI token survives and the creator stays signed in. Resume/reconnect after a midstream crash is deliberately deferred to TASK 3: the socket can't be resumed (it dies with the process), so "resume" = fast reconnect with a saved broadcast ID/stream key within YouTube's disconnect-grace window; too slow and `enableAutoStop` ends the broadcast. **Orphan closed 2026-09-01:** this mechanism is owned by **TASK 32 (Stream resilience)** — in-session blip retry inside the grace window; cross-process "resume" of a dead broadcast is officially out-of-product (the API makes it impossible — creator ruling). +6. ✅ Tests in `ytLive.Tests` (xUnit, net8.0-windows): TokenStore DPAPI roundtrip/corrupt/missing/clear + mocked exchange channel-parse + refresh expiry bump + `ClearSession` — 7 passing + +**Design constraint:** Sign-in must NEVER block core exploration. Users can build scenes, add sources, and audition the software without authenticating. But **going live requires authentication** — the "Start Stream" dialog is where the account sign-in lives, alongside all stream metadata. + +**Two-state flow:** There is no separate "Connect" button. The top bar shows a single button — **Start Stream** when idle, **End Stream** when live. Clicking Start Stream opens one dialog that supplies everything: account (previously-saved account shown as default, with a Change Account action) + title/description/visibility. + +**Live indicators (unmissable):** Top bar + window title bar flip red, a pulsing **● LIVE** badge with elapsed timer appears in the top bar, the preview area gets a red glow, and the taskbar icon shows a red overlay dot. Window title bar shows the stream title once validated. + +### Requirements: + +1. **Google Cloud OAuth credentials** — client ID + secret, **baked into `Helpers/OAuthCredentials.cs`** (desktop "Desktop app" OAuth client; loopback callback — no console redirect URI registration needed; creators never configure) +2. **Local HTTP listener** — `HttpListener` on `http://localhost:PORT/oauth2/callback` to catch the redirect +3. **Browser launch** — open the authorization URL in the default browser +4. **Token persistence** — store access/refresh tokens securely (Windows DPAPI), reload on startup +5. **UI state** — account shown in the Start Stream dialog; "Change Account" action triggers re-auth +6. **Go Live gated on auth** — Start Stream dialog requires sign-in to enable the Start button; scene building works without it + +### Tests: + +1. Mock token exchange response, verify channel info parsed +2. Verify token refresh triggers when near expiry +3. Verify credential load/save roundtrip + diff --git a/TASKS/task-03-capture-pipeline.md b/TASKS/task-03-capture-pipeline.md new file mode 100644 index 0000000..1c69cff --- /dev/null +++ b/TASKS/task-03-capture-pipeline.md @@ -0,0 +1,116 @@ +# TASK 3 — Capture Pipeline (Scenes/Sources) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** Real video preview in the center panel — the minimal source set below, composited per scene. + +### Status: 🔶 In progress — 27 of 30 items done (1 excluded: background removal); items 16, 17, 20 open + +1. ✅ **Milestone 1 — webcam** — MediaCapture (WinRT SDK projection) with device enumeration, CPU-first frame source, refcounted `CameraManager`, picker dialog, clip shapes (Traditional + Round) + mirror, 480×270 default placement — schema v2 +2. ✅ **Schema v3 (Ship Branch A)** — multi-scene webcam (singleton `Webcam` + per-scene `WebcamSceneConfig`), right-click border/context menu, static OBS-style borders, 50%-per-dimension webcam size cap, device-swap (`ReleaseAllAsync`) — 25 tests passing +3. ✅ **Schema v4** — round→rect restore persisted (`WebcamSceneConfig.RectWidth`/`RectHeight`) + one-time legacy-square 16:9 heal on load +4. ✅ **Screen backdrop (ship task #1, schema v5)** — live desktop/game capture as a permanent, non-deletable bottom layer (`Source.IsBackdrop`), auto-detecting the full-screen game at launch/focus (else the **primary display** — never assumed monitor 0) via `Win32FullScreenDetector` (now with `GetDisplays()`/`PrimaryMonitorIndex()` for the in-app display picker), content re-designated via the OS `GraphicsCapturePicker` ("Change Capture…") or the in-app "Capture Display" submenu, refcounted/shared capture sessions in `ScreenCaptureManager` mirroring `CameraManager` — 45 tests passing +5. ✅ **Schema v6 — backdrop Live-only by policy** — `Scene.HasBackdrop`, enforced by scene name on every load (`EnforceBackdropPolicy`: Starting/BRB/Chat/Ending never carry one; the one-time v5→v6 backfill covers all four), the scene context-menu "Backdrop" checkbox is gone (policy owns the flag), preview watermark hides when the backdrop renders, capture changed to `WindowsRuntimeMarshal.TryGetDataUnsafe` (the CsWinRT-safe frame-read) + downscale to the 1920×1080 master + 5s-throttled error logging (was flooding `startup.log` with 5 MB of cast errors and burning CPU), round webcam no longer re-rasterizes an `ImageBrush` every frame (Image + EllipseGeometry clip) — the live-mode stutter fix +6. ✅ **The five-scene catalog (`SceneCatalog`)** — Starting/Live/BRB/Chat/Ending is the product — work with less, never more; the (+) button only shows when a canonical scene is missing and re-adds it (its menu lists only the missing ones) — 62 tests passing +7. ✅ **Webcam-after-session-start fix** — a webcam added to a scene after the camera was already running (e.g. Chat) previously rendered a transparent container — `CameraManager.GetPreviewBitmap` + propagation in `AddWebcamToActiveSceneAsync`/`ReacquireWebcam` now hands the running shared frames to any newly added `WebcamSceneConfig` — 65 tests passing +8. ✅ **Chat scene webcam size cap** — raised from 50%-per-dimension (960×540) to half the screen AREA (~1358×764 @16:9, `MaxWebcamWidthFor`/`MaxWebcamHeightFor` keyed by canonical name) so the viewer sees the creator better +9. ✅ **"Add Webcam" always opens the camera picker** — deleting one scene's webcam then re-adding used to resurrect the old camera when another scene still used it — `SwapWebcamIdentityAsync` now swaps the app-wide identity if a different camera is chosen, same path as "Change Webcam…" +10. ✅ **Webcam resource validation + first-frame proof** — `MediaCaptureFrameSource` validates post-init (VideoDeviceId match, stream properties ≥1, `reader.StartAsync()` status read + throws on non-Success); subscribes `capture.Failed` + `CameraStreamStateChanged` → `SourceFailed` event on the seam; fallback ladder (VideoPreview → VideoRecord). `CameraManager.AcquireAsync` requires first-frame proof (4s timeout): returns true only after a real frame arrives — silent empty box impossible. `MainViewModel` subscribes `CameraFailed` → red `WebcamError` chip in preview + MessageBox names suspect apps (`CameraConflictProbe`). 19041 SDK projection gaps: `Exclusive`/`DeviceLost` not projected; `CameraStreamState.Failed` compared by `(int)2`. 81 tests passing +11. ✅ **Scenes/sources UI** — add/reorder/rename, image + background overlays with move/resize/opacity/reuse +12. ✅ **Audio UX shipped (UI)** — the bottom-bar footer is now two lines (dropped/duration moved under bitrate/fps), with the mic's sound meter + mute button + volume slider grouped CENTERED on the footer's top line, beneath the preview panel (meter: 288px, muted slate track with ruler graduations + muted yellow/red zone tints, green→yellow→red fill; mute = speaker icon → red do-not-symbol when muted, and the slider and speaker stay in sync (volume 0 ⇔ muted — sliding off flips the speaker to muted, sliding up from 0 clears it); mic volume defaults to 80%, muting zeroes the meter and restores the prior volume on unmute (which flashes the meter to the restored position ~300ms before it returns to the live level); the meter is a READ-ONLY realtime level display (fill = live level × volume — volume is a gain on ambient noise; while the slider is dragged the bar previews the slider position and bounces back to the live level on release, which is 0 with no input — clicking the meter does nothing), clicking the MIC label opens a microphone picker whose chosen source shows left-justified inside the meter bar (fill at 75% opacity so the name + ruler markings show through); slim dimensional slider — gradient track/fill, gloss-sphere thumb; the old flat pink 18px-filled one is gone), everything else on line 2 (bitrate/fps/dropped/duration/health left, quality + gear right) — the creator's only audio control, desktop/game audio is automatic (KISS rule). **Post-test follow-up (2026-08-13, game audio bar branch):** the **game audio bar** (desktop/game, a mirror of the mic bar: meter + mute + volume) is **overlaid at the bottom of the preview window** (bottom-center dark chip — monitoring UI only, never on the live output) and appears only while a **full-screen game is producing sound** in the preview (`IGameAudioDetector` seam + default `GameAudioDetector` polling `IFullScreenDetector` + the live loopback level, floor 0.5%, into a pure `GameAudioHysteresis`: SHOW after ~500ms of fullscreen+sound, HIDE ~1s after leaving fullscreen, **silence never hides an active bar**; the VM polls it on a 250ms `DispatcherTimer`); **capture now runs for the app's lifetime** (mixer started once at startup via `StartMicCaptureAsync`, disposed in `Shutdown` — not go-live) so the meters preview live; the **MIC label became a button with a status dot** (`Models/MicStatus`: green = connected via the source `Started` event, yellow = mic problem, red = no device); picking a mic takes effect immediately (`AudioMixer.RestartMic`, loopback keeps running); fixed a latent `?.Invoke(meter.Push(...))` short-circuit that skipped the meter update when nothing was subscribed. **Round 2 (2026-08-13):** the meters were dead on a flat scale (real speech/game RMS is ~0.01..0.1 linear) — the raw level is now mapped via `AudioLevelMeter.ToDisplay` (−60..0 dBFS spread across 0..1) so typical input reads ~1/3..2/3 of the bar at default volume. 169 tests passing, 0 warnings. **Round 3 (2026-08-13):** the mic bar gained a **mic mute icon** (a microphone glyph in the speaker's 16px style, red + slash when muted) between the meter and the speaker — both mutes adjacent with spacing between them, same `ToggleMicMuteCommand`; and the top-center LIVE badge became an **always-visible REC sign** — dark gray dot + dim "REC" offline, bright red (#e94560) + "REC" + elapsed while live, **darker red (#8f1f1f)** when live with a **private** stream (driven by the dialog's chosen visibility) — `RecDotBrush`/`RecTextBrush`/`RecDotOpacity`/`IsLivePrivate`, pulsing while live. **Queued:** task 22 (voice filters). Voice-filter note: the meter's `ToDisplay` input is the pre-filter mic level; when filters land they must apply BEFORE the meter/encoder mix +13. ✅ The connected YouTube account's avatar/name shows in the top bar next to Start Stream (`SyncConnectedAccount`); the scenes list is content-height now (no dead space before SOURCES) +14. ✅ **Social bar v2 (six-slot dialog, sign-in gate, real logos)** — global bar layer (never a Source, no sources-list row), content-sized, centered, GREEN glow when ON, top/bottom snap-drag (default BOTTOM, persisted `SocialBarPosition`; drag clamps to 0/1040, tie→bottom). Footer Social button gets a state dot (green=ON). Dialog "Social Media Site Promotion" (`SocialsDialog` + `ViewModels/SocialsDialogViewModel`, WPF-free + injected `ISocialValidator`/sign-in/sign-out fakes): ON/OFF bar switch (`SocialsConfig.BarEnabled`, schema v8), 6 fixed slots — row 1 always YouTube (signed-in → channel handle; signed-out → sign-in gate → OAuth; delete → confirm sign-out, mirrors `StopStream`), row 2 free, rows 3-6 lock icons on freemium (Premium seam: all six). Validation: `DetectService` (URL domain / fediverse `@user@domain` / bare→Website) → async `ISocialValidator` on confirm/Save; valid snaps to text + real service logo (bundled SVG path data via `LogoDataFor`, Simple Icons CC0 — initials badges gone); invalid → red do-not, stays editable, Save blocked. LCR justify dropped (`BarJustify` unread), per-scene toggle dropped (`Scene.HasSocialBar` back-compat). **Post-test fixes (2026-08-12):** footer label "Socials" (not "Social"); fediverse `@user@domain` validates — the full handle is the identity end-to-end (`DetectService`/`CanonicalUrlFor`/`HttpSocialValidator` build `https://domain/@user`, no domain loss); **Cancel is a hard stop** — `ISocialValidator.LookupAsync` takes a `CancellationToken`, dialog VM owns a CTS, Cancel/X/Save abort in-flight lookups (HTTP request killed, canceled continuations never touch slot state), and `ConfirmEdit` skips re-submitting identical text (LostFocus on dismiss never re-fires a lookup). `HttpSocialValidator` now has real tests (fake `HttpMessageHandler`). 105 tests passing. **Post-test fixes (2026-08-12, round 2):** fediverse `@user@domain` no longer shows a generic chain — it resolves to the instance's actual software via nodeinfo (`/.well-known/nodeinfo` → `software.name`; `SocialService.Fediverse` enum member + `SocialEntry.FediverseSoftware` persisted in a new `SocialEntry.Software` column, schema migration by column-presence) and renders that software's bundled logo (`LogoDataForFediverse`: mastodon/peertube/pixelfed/misskey/lemmy/pleroma/firefish, generic fediverse honeycomb fallback). Dialog row-2 edit/trash icons were too dark — `IconButton` style gains `Foreground=#d0d0d0`; trash overrides `#e94560` (app red). 112 tests passing. **Post-test fixes (2026-08-12, round 3):** a fediverse handle whose identity domain is itself a redirect (e.g. YunoHost default-app subdomains — `@user@llamachile.tube` where the mastodon instance lives at `mastodon.llamachile.tube`) now still resolves its software: nodeinfo on the identity domain is SSO-blocked, so `HttpSocialValidator` follows the bare root `https://domain/` 302 to the real instance host and re-runs the nodeinfo lookup there. +15. ✅ **Window capture** — absorbed into the Screen picker (no separate source type); dedicated window-as-source work is pending +16. ❌ **Scene compositing (D3DImage/MediaElement)** — SUPERSEDED, permanently out (2026-09-01): the software output compositor (TASK 4 step 1) + the XAML preview ARE the design; a D3D11 swap stays a possibility behind the `VideoFrame` seam, not a feature +17. ☐ **Text source** — live text ("Starting soon", "Back in 5", handle, callout) +18. ✅ **Chat box** — YouTube live chat rendered *on* the stream so viewers read along in-video — `ChatBoxRenderer` (WPF FormattedText → VideoFrame), configurable font size/color/badges/timestamps/max-messages via Elements panel, schema v10, persisted across save/load +19. ❌ **Background removal (milestone 2)** — ONNX Runtime + DirectML, MediaPipe Selfie Segmentation — deliberately NOT in this build +20. ☐ **Alerts** — Super Chat / membership / subscribe pop-ins; build after the six; **free and ungated** (the stale "one paid feature" label predates the 2026-08-31 monetization reversal and is corrected 2026-09-01 — the paid swap is the branding flash ONLY; see Monetization in `ai.md`). **v1 IN** per the complete-v1 ruling. **Precursor — reward-event capture (2026-09-01):** Alerts render from the app's canonical `RewardEvent` feed (all 7 `liveChatMessage` reward types, persisted to SQLite + the session report — see TASK 10 → "Related work — monetization awareness"), so it consumes that already-built data rather than re-integrating `liveChatMessages`. That capture is the required prior milestone. +21. ✅ **Show Desktop toggle** — `Source.ShowDesktop` bool persisted in DB (schema migration + LayoutStore read/write). When checked, primary monitor captures regardless of fullscreen game state. Toggling off falls back to running game or releases to static placeholder. `ClearBackdropCaptureAsync` clears `CaptureKey` + `VideoImageSource` so static fallback renders. Toggle on Live backdrop context menu (IsCheckable MenuItem). +22. ✅ **Top bar — status light + avatar + Log In button** — red/green/pulsing ellipse (disconnected/connected/live). Avatar border visible only when connected, loads via `BitmapImage` code-behind. "Log In" button visible when disconnected, calls `StartStreamCommand` → GoLiveWindow. `ShowStartStream` now requires `IsConnected`. +23. ✅ **Chat preview — one-at-a-time mock messages** — `RunMockChatPreviewAsync`: simple async loop, displays `MockChatMessages[n]` via `Take(n+1)`, 1000ms between messages, wraps at end. `ChatPreviewEnabled` property on Source (default true). Stops on real messages, restarts on fade timer clear. +24. ✅ **All backdrops seeded + EnsureDefaultBackdrop** — Live, Chat, Ending backdrops added (ending-backdrop.jpg replaced). `EnsureDefaultBackdrop()` called on every scene switch. `SeedEndingBackdrop` replaces existing (not skip-if-exists). Chat scene no longer skipped. +25. ✅ **SceneCatalog fix** — Chat scene display name corrected to "Chat" (was "YouTube Chat" — that name belongs to the layer/SourceType.ChatBox, not the scene). +26. ✅ **Left panel spacing + context menu reorder** — HR margins matched, Row 4 changed from `*` to `Auto`. Context menu reordered: Border Thickness first, then Opacity, Color, Effect. +21. ✅ **Logo + richer in-app About (2026-08-13, queued → 2026-08-14 SHIPPED)** — the ytLlive wordmark in the top bar opens the About overlay (already wired); the About overlay is now the **creator hub**: the real logo (the "llama fortnite superman logo" from the creator's vault, copied to `Assets/llama-logo.png` — the creator's own art, no third-party license), plus **creator-hub links** — llama chile shop on YouTube (`MainViewModel.ChannelUrl`), Mastodon (`https://mastodon.llamachile.tube/@gramps`), **Buy me a coffee** (`https://buymeacoffee.com/llamachiley` — live), and **Unlock Premium** (greyed "coming soon" — the **billing** product URL is a tabled seam, `PremiumUrl`, until TASK 10 picks the subscription provider). A **Licenses & legal** button flips the About overlay to an in-app scrolling panel that loads the full `THIRD-PARTY-NOTICES.txt` text (`MainViewModel.ShowLicenses` reads the shipped file from the executable directory on first open; graceful "not found" fallback — **never the OS viewer, everything stays in-app**); "← Back to About" returns. The About overlay is also the in-app home of the notices — the top-bar About button that opened the file in the OS viewer was removed on 2026-08-13 for exactly this reason. Integration test (Good Dog Rule — ONE): `AboutHubTests.About_Opens_InApp_Licensing_Loads_Shipped_Notices` drives the real window + VM, asserting the hub opens, the link URLs are real, the licensing panel loads the shipped notices text (contains "Third-Party Notices" + "LGPL"), and back returns to the hub. 174 tests passing, 0 warnings +22. ✅ **Voice filters on the mic channel (2026-08-13 queued → SHIPPED 2026-08-14 inside TASK 9, the audio milestone)** — the standard four applied to the sound input path (before the meter/encoder mix): **bass boost, treble, noise suppression, compressor** (set decided with the creator 2026-08-13). Noise suppression = a **pure-C# noise gate** (creator chose over RNNoise / a second ffmpeg `afftdn` pipe, 2026-08-14 — KISS). Always-on — no UI knobs; the mic stays the creator's single audio control +27. ✅ **Webcam row gates on the app-wide identity (2026-08-24, TASK 26)** — the (+) menu's Webcam item greys out whenever a webcam exists **anywhere** (`CanAddWebcam` = `StagedScene != null && _webcam == null`; renamed from `CanAddWebcamToStagedScene` whose per-scene rule let a second picker run from a scene lacking the config), raised at both `_webcam` mutation sites (create / last-config removal) + scene staging + elements change. Creator's visual pass found it: minis/no-capture-rows/capture-controls all good. ONE integration test `WebcamMenuGateTests.CanAddWebcam_Gates_On_The_AppWide_Webcam_Identity` (real window + temp DB seeded with a webcam in Starting; asserts greyed while Live staged, re-enabled after `RemoveSourceCommand` clears the last config + the `Webcam` DB row). Stale map fixed in the same commit: ai.md's "empty-canvas right-click Show Webcam" claim dropped — that XAML never shipped (`ShowWebcamCommand`/`CanShowWebcamInStagedScene` are wired but unbound dead code, audit item). 224 tests (223 pass; the pre-existing AudioPipelineTests failure is unrelated) +28. ✅ **YouTube Chat layer: one-per-layout gate + legacy label heal (2026-08-24, TASK 27)** — same rule as TASK 26 applied to the chat layer: the (+) menu's YouTube Chat item greys out while any scene carries a ChatBox source (`CanAddYouTubeChat`, raised on staging + elements change; `AddSource` refuses a second), tooltip "One chat layer at a time — it's already in your stream". Plus the creator's label fix: layers added by commit `65641d8` were named "Chat"; LoadLayout now heals exactly that un-renamed default to "YouTube Chat" so the Layers row matches the (+) picklist (creator renames untouched, idempotent). ONE integration test `ChatLayerGateTests.LegacyChatName_Heals_And_CanAddYouTubeChat_Gates_On_The_Existing_Layer` (real window + temp DB with a legacy "Chat" row: healed on load + persisted on save, greyed cross-scene and in-scene, re-enabled after remove). 225 tests (224 pass; pre-existing AudioPipelineTests failure unrelated) +29. ✅ **Broadcast metadata pull-out + launch geometry (2026-08-24)** — creator ask: "a tab-pullout on the right side of the preview pane — white tab, YouTube-red label reading 'Text' — opening a form with all settable liveBroadcast fields; fields that can't be filled before launch greyed out; saving/updating remote content; field data saved and pre-loaded on app run". Shipped as specified with one correction: the API is the inverse of "more fields once live" for contentDetails (those lock in created/ready) — the pull-out carries the always-editable snippet/status set: Title, Description, Tags (csv), Visibility (private/unlisted/public), Made-for-Kids, plus read-only Scheduled Start. **Always visible** (creator revised same day from Live-only gating). White 30px tab, rotated red "Text", right edge vertically centered; click slides a dark 320px drawer left over the preview (200ms CubicEase). Every edit persists to Settings keys (`Broadcast.*`) immediately; **Update Broadcast** button calls the new `YouTubeStreamService.UpdateBroadcast` (PUT `liveBroadcasts?part=snippet,status`, echoes scheduledStartTime because update replaces the whole snippet part). Go Live prefills from the form and captures what was inserted (`CaptureGoLive`). The old un-persisted Default Stream Title/Description fields came OFF the App Settings overlay (replaced by the form); `DefaultStreamTitle/Description` VM properties deleted. **Launch geometry** (same unit): default 1920×1040, MinWidth 1366, MinHeight 768; `WindowStartupLocation=Manual` + window size/position persisted on close via `RestoreBounds` (maximized-safe), restored in ctor clamped to minimums and the primary work area (disconnected-secondary fallback). ONE integration test `BroadcastPullOutTests.Metadata_Persists_WindowRestores_Clamped_And_UpdatePatchesRemote`. 228 tests (227 pass; pre-existing AudioPipelineTests failure unrelated) +30. ✅ **Gear menu cleanup + Default Location for Recordings + single Start button (2026-08-29)** — **KISS: removed** Gear-menu `Save Layout`/`Save Layout As…`/`Open Layout…` (redundant with auto-save; multi-layout = OBS) + dead `SaveLayoutCommand`/`SaveLayoutAsCommand`/`OpenLayoutCommand` + `SaveLayoutAs()`/`OpenLayoutFile()` methods. **App Settings panel:** first real setting — `Default Location for Recordings` (configured folder via `ChooseRecordFolderCommand` + "Use Downloads" reset via `ResetRecordFolderCommand`, resolved path shown). Fallback chain: configured → `%USERPROFILE%\Downloads` → `MyVideos`. `StartRecordFile()` + `ChooseRecordFolder()` `InitialDirectory` updated to use the new fallback. **Single Start button:** `PrimaryButtonText`/`IsStreamingStart` + their `OnPropertyChanged` calls removed; button `Content="Start"` in XAML — pills are the intent surface, button is always just Start. `Choose Record Folder` fast-path kept in the button context menu. Monetization (About links/premium) deferred to a later task. ONE fallback test `DefaultRecordFolder_Falls_Back_To_Downloads`. build 0 warnings, 247 tests (2 known failures, pre-existing). + +### The Minimal Source Set (design decision — do not expand casually) + +ytLlive is YouTube-only and 90% of users are casual. OBS's long source list is off-putting; we ship +the hot few and nothing esoteric. If a user needs more, they've graduated to OBS. + +1. **Webcam** — the face cam. Non-negotiable. +2. **Screen** — the main event (game, slides, browser). One source; a picker chooses a monitor *or* a + window. (Window capture is absorbed here — no separate source type.) +3. **Background** — a full-canvas backdrop image. Fills the whole scene automatically, zero fiddling. + Kept separate from Image on purpose: same pixels, but this one needs no positioning. +4. **Image** — a floating graphic/logo overlay (watermark, badge, corner branding). Free-positioned. +5. **Text** — live text ("Starting soon", "Back in 5", handle, callout). Casual streamers live on this. +6. **Chat box** — YouTube live chat rendered *on* the stream so viewers read along in-video. YT-native. +7. **Alerts** — Super Chat / membership / subscribe pop-ins. The dopamine source. **The one big lift** + (Super Chat event streaming + on-stream rendering/animation); build after the six. **Free and + ungated** — paid = flash removal only (stale "one paid feature" corrected 2026-09-01). Consumes + the canonical `RewardEvent` feed (reward-event capture, TASK 10 → related work). + +Deliberately NOT supported: game capture, browser source, media playlist, VLC, color-key voodoo, MIDI. + +### Source memory model (design decision) + +1. A scene has resources. Resources can be shared across scenes. +2. A resource exists exactly once in memory, no matter how many scenes use it (a logo in five + scenes = one loaded bitmap). +3. Every resource carries a **catalog of scenes**: one usage entry per scene it appears in, each + entry dictating that scene's use — placement (X/Y/Width/Height), opacity, z-order, enabled, + scale mode, crop. +4. Usages are named `{resourceName}.{sceneName}` — whatever the user named the resource, dot, the + scene name: `logo.starting`, `logo.live`, `myPic.brb`. Not a hardcoded "logo". +5. A webcam in two scenes = one capture session, two catalog entries. +6. Refcount by catalog size: the last usage removed → the resource is disposed and evicted. +7. The resource (not a per-scene node) owns everything `IDisposable`. + +### Scene transitions (design decision) + +Scene switching while live must never stutter. Supported types, most → least economical: + +1. **Cut** — instant switch. The default. Zero cost. +2. **Fade** — short crossfade (~300ms). +3. **Move** — a simple, economical move transition, done to perfection and memory-efficient. The + smart streamer's bread and butter. +4. **Custom (media) transitions** — require media elements (video/stinger playback during the + transition). Heavier, but creators pay for these, so we support them. Their media follows the + same resource memory model: loaded once, catalogued by scene. + +Preview shows the transition too (WYSIWYG). No wipes/slides/LUTs beyond the four above. + +### Requirements: + +1. **Screen** — Windows.Graphics.Capture (WinRT), enumerate displays/windows, picker +2. **Webcam** — MediaCapture (WinRT SDK projection) with device enumeration — ✅ **milestone 1 done**: + 1. TFM bumped to `net8.0-windows10.0.19041.0` (app **and** tests) so the WinRT projection resolves from the SDK reference packs — no NuGet package, no capability manifest (unpackaged desktop app) + 2. `MediaCaptureFrameSource` (CPU-first: `MemoryPreference = Cpu`, BGRA8 via `CreateFrameReaderAsync`), `MediaCaptureCameraEnumerator` (`DeviceInformation.FindAllAsync(DeviceClass.VideoCapture)`) + 3. `CameraManager`: refcounted by `DeviceId`, one shared `WriteableBitmap` app-wide, dispatcher-coalesced UI updates (~render rate, latest-frame drop), placeholder/`AppLog` + warning on failure + 4. `CameraPickerDialog` (mirror of `ReuseImageDialog`) — "Searching for cameras…" / list / "No cameras found" states + 5. One webcam app-wide: Add → Webcam greyed out once one exists ("it's already in your stream" tooltip); persisted `DeviceId` re-acquires after layout load + 6. Default placement 16:9 **480×270**, bottom-right, 32px margin; drag/resize/selection shared with Image sources + 7. **Clip shapes: Traditional + Round** (phone view dropped — the 9:16 phone output is the vertical output-crop tier); **mirror**; both persisted in the layout DB (schema v2) and toggled from the source chip + 8. **Background removal = milestone 2** (ONNX Runtime + DirectML, MediaPipe Selfie Segmentation) — not in this build +3. **Background / Image / Text** — static sources positioned/scaled/opacity +4. **Chat box** — rendered from the live chat poll (right panel is the same feed, raw) +5. **Scene compositing** — per-scene source layering (z-order = sources list order, top-to-bottom + back-to-front), preview rendered via D3DImage or MediaElement +6. **Branding flash** — the topmost full-frame "made with ytLlive!" layer at ~25% opacity, ~1s on / + 300s off (see Monetization in `ai.md`), gated on `BrandFlashEnabled` + live/recording. Lives in the + preview compositor now (`BrandFlashLayer` in `MainWindow.xaml` CanvasGrid, driven by + `BrandFlashActive`/`BrandFlashTimer` in `MainViewModel`); the encoder output renders the same layer, + and v0.2 local recordings carry it too +7. **Drag/drop placement & reorder** — intuitive, visual (per design principle): + 1. **Preview:** click-drag a source in the center panel to reposition it; resize via handles + 2. **Scenes list:** drag rows to reorder scenes + 3. **Sources list:** drag rows to reorder sources (this *is* the z-order) — implemented + diff --git a/TASKS/task-04-rtmp-ingest.md b/TASKS/task-04-rtmp-ingest.md new file mode 100644 index 0000000..6165f4b --- /dev/null +++ b/TASKS/task-04-rtmp-ingest.md @@ -0,0 +1,339 @@ +# TASK 4 — RTMP Ingest to YouTube + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** Push encoded video to YouTube's RTMP ingest. + +### Status: ✅ Done — all 7 items shipped + +1. ✅ **Ship step 1 — the output compositor SHIPPED** (2026-08-10) +2. ✅ **Ship step 2 — the FFmpeg locator SHIPPED** (2026-08-10) +3. ✅ **Encoder + RTMP push SHIPPED** (2026-08-12) — the FFmpeg subprocess: raw BGRA frames via stdin, stderr health parsing, FLV mux + push to the ingestion URL (see the ship step 3 plan below) +4. ✅ **WASAPI audio capture SHIPPED** (2026-08-12) — NAudio loopback (desktop/game) + the picked mic feeding `AudioLevel`, so the realtime meter comes alive (see the ship step 4 plan below) +5. ✅ **Frame-pipeline wiring SHIPPED** (2026-08-12) — `CameraManager`/`ScreenCaptureManager` → compositor resolver → encoder, driven by a paced `FramePump` (see the ship step 5 plan below) +6. ✅ **Health stats SHIPPED** (2026-08-13) — `FramePump.HealthUpdated` (encoder's parsed bitrate/FPS/dropped/duration, already forwarded from `FfmpegEncoder.OnStderrLine`) now lands in the bottom bar: `MainViewModel.OnFramePumpHealthUpdated` marshals to the UI thread (the stderr loop raises on a background thread) and copies into `CurrentHealth` (the bottom bar's existing binding); `ResetHealth` zeroes dropped/duration on go-live and on End so stats never linger from a previous session (bitrate/FPS stay on the tier's targets). The bar lights up with real values once TASK 9 supplies the RTMP URL (until then the pump skips the encoder and the bar shows the tier's targets) +7. ✅ **One-click go live + private-only enforcement SHIPPED** (2026-08-14) — the Go Live dialog is **locked to Private** (no dropdown, `GoLiveViewModel.Visibility` is a get-only "Private"); `YouTubeStreamService.CreateBroadcast` **always sends `privacyStatus = "private"`** (dialog + service enforcement, requirement 8 — nothing can go out non-private) and gained an injectable `HttpClient? http = null` seam for tests; `BeginGoLive` now calls `CreateBroadcastAsync` and remembers `_currentBroadcastId` for TASK 9's bind/transition (failure → `StreamStatus.Error`, never a crash; `StopStream` clears the ID); the REC sign shows a **PRIVATE badge** (dark-red border, next to REC, `IsLivePrivate`) when the live stream is private; settings' dead "Default Visibility" dropdown + `MainViewModel.Visibilities`/`DefaultStreamVisibility` removed. The broadcast-insert integration test asserts the request body carries `"privacyStatus":"private"` (2 new tests → 173 passing, 0 warnings) + +The pipeline chain the encoder needs doesn't exist yet: **scene compositing** (the master 1920×1080 frame +without the preview's editing chrome) → **audio capture** (WASAPI, feeds the meter) → **H.264+AAC encode** +→ **vertical-tier crop/scale** → **RTMP push** → **health stats** into the bottom bar. Nothing can encode +until a frame source exists, so the compositor is ship step 1. The pipeline is +`CameraManager + ScreenCaptureManager → compositor resolver → compositor → encoder → RTMP`. + +### Requirements: + +1. **Encoding** — H.264 (hardware via NVENC/AMD, fallback x264) + AAC audio; **must comply**: keyframes ≤ 4s (gopSizeLong), closed GOP, AAC/MP3 @ 44.1/48kHz, mono/stereo only. **License posture (decided): GPL-free build** — NVENC (NVIDIA) / QSV (Intel) / AMF (AMD) + OpenH264 software fallback + built-in AAC; no libx264 (GPL contaminates a paid product). Output containers are identical either way (H.264+AAC in `.flv` for RTMP, `.mp4`/`.ts` for VOD) — the format is NOT the differentiator, the license and per-GPU quality are. **License guardrails (never violate — see `ai.md` → "Licensing — do not violate"):** only BtbN `lgpl`/`lgpl-shared` builds; never GPL (gyan.dev) or `nonfree` (fdk-aac); never static for distribution (LGPL §6 relink material); never link FFmpeg into the app; never drop `THIRD-PARTY-NOTICES.txt` from the app/About screen. +2. **RTMP push** — **FFmpeg subprocess (decided)**: app feeds raw frames via stdin, parses stderr for health; one battle-tested binary does encode + FLV mux + push. **No self-heal (claim-check 2026-09-01): ffmpeg's reconnect flags are input-side; an RTMP *output* push does not reconnect itself — retry is app-side, owned by TASK 32.** **Binary distribution (decided): check-then-pull** — probe `where ffmpeg`/PATH at first go-live; if absent, download a **pinned** build (**BtbN LGPL-shared win64** zip, ~75 MB — gyan.dev's builds are GPLv3 and ship libx264, which violates the license posture; BtbN's LGPL variant drops x264/x265 while keeping NVENC/QSV/AMF + libopenh264 + native AAC) to `%APPDATA%\ytLlive\tools\ffmpeg.exe` (extract `ffmpeg.exe` **plus the `libav*.dll` family**) and cache it, offline-friendly. Behind an `IFfmpegLocator` seam so tests fake it (ship step 2, below). Push goes to the cached reusable stream's ingestion URL +3. **Quality ladder** — the offered tiers, with **1080p60 @ 8 Mbps as the standard/default**: + 1. 720p30 @ 6 Mbps + 2. 720p60 @ 6 Mbps + 3. 1080p30 @ 8 Mbps + 4. **1080p60 @ 8 Mbps** (default — mainstream ceiling, GPU hardware-encoded so the gaming + machine never notices; upload headroom stays comfortable) + 5. Vertical 1080×1920 @ 60fps @ 8 Mbps (9:16 phone tier) + The composition master is always 1920×1080; a tier is an output rect + target resolution + (see `ai.md` "Resolution tiers"). Vertical output = the centered 607×1080 crop of the master + scaled to 1080×1920 (semi-crop preview is already implemented; the encoder applies the same rect). + 1080p60 is the ceiling by design — "if you want 1440 or 4K or 8K → OBS is your solution"; the app + targets the most mainstream creator, not power users. + Ladder is sculpted by a **cached probe** (IP-only TCP vs public ingest host; no auth required). + Quality is greyed out while live because the declared resolution can't change mid-stream — but + with `variable`, we can **auto step-down** bitrate/resolution on the fly with zero API calls + (no stream recreation); 60fps presumes a hardware encoder — no hardware encoder → auto + fallback to 720p60/1080p30 +4. **Stream key management** — reuse the cached reusable stream (one per channel) instead of creating a new one per go-live; prefill default YouTube ingest URL `rtmp://a.rtmp.youtube.com/live2` +5. **Health stats** — bitrate, FPS, dropped frames reported live in the bottom bar (encoder-side) +6. **One-click go live** — defaults that work out of the box +7. **Audio capture (feeds the meter — this task ships the wiring)** — WASAPI loopback (desktop/game at unity, zero UI — "it just is") + the picked mic (`MicSourceName` from the `MicPickerDialog`). The mic capture feeds `AudioLevel` so the realtime meter comes alive (today it reads 0 — the mixer feed is pending, see `ai.md` audio notes). AAC mono/stereo @ 48 kHz per the compliance rules. +8. **Private-only go live until v1 (reputation guard, decided 2026-08-10)** — until the v1 release, go-live is **locked to private streams only** so a software error can never publish something public/unlisted that damages the creator's reputation. RTMP push itself has no privacy — privacy lives on the YouTube **live broadcast object**, which this app already controls via its OAuth API calls. So the lock is purely API-side: the Go Live flow always creates/updates the broadcast with `privacyStatus = "private"` and a guard **refuses** to set anything else (same spirit as the Live-only backdrop policy). The UI shows a clear "PRIVATE" badge next to the stream state so the creator always knows who can see them. Enforcement must be verifiable in the auth-service tests (fake the broadcast-insert/update call, assert `privacyStatus` is forced to private). +9. **v1 release gate: bundle the full license texts (decided 2026-08-10)** — `THIRD-PARTY-NOTICES.txt` currently links the canonical license texts rather than embedding them. At the **v1 (GA) release**, the full texts of every license it names (LGPL v2.1+, BSD-2-Clause, MIT, Apache-2.0) MUST be bundled alongside it (shipped in the app output, e.g. a `licenses/` folder next to the notices file, still reachable from the About screen). This is a **release blocker for v1, not a task to queue early** — do it in the release pass. The repo should treat this like the private-only go-live gate: a checkbox that cannot silently lapse. + +#### Ship step 1 — Scene compositor (the frame source) + +**Goal:** a pure-CPU software compositor producing the encoder's master frame (BGRA8, the `VideoFrame` +seam) from the scene model. The preview stays XAML (the editing view); the compositor is the **output +view** — WPF's `RenderTargetBitmap` can't be used (software-rendered + captures chrome). Two renderers +must agree, so the XAML (`MainWindow.xaml` CanvasGrid + element DataTemplate) is the contract. + +**Decisions (locked 2026-08-10):** **Path A CPU blitter** — GPU effort belongs to NVENC (the encoder), +not composition; with an FFmpeg subprocess the master crosses a CPU readback to the pipe every frame +anyway, so GPU compositing buys ~nothing at this layer count (2-3 live layers; static layers +pre-composite once). A D3D11 compositor can replace this one later **behind the same seam** (the CPU +master buffer stays the contract). **Render the output rect directly**: compositor is constructed with +`CompositorOptions {SourceRectX/Y/W/H, OutputWidth, OutputHeight}`; 16:9 tiers = full 1920×1080 1:1; +vertical (9:16) = composite the centered 607×1080 crop then bilinear-upscale to 1080×1920. Reuses +`MainViewModel.OutputRectX/Y/W/H` (note `(1920−607)/2 = 656.5` → align to integer pixels for output). + +**Render spec (back → front, mirror the XAML exactly):** +1. Backdrop — the Live scene's `IsBackdrop` Source (`CaptureKey` → live frame), `UniformToFill` + full-frame (XAML's separate `BackdropImage` layer; the backdrop *element* renders nothing — its + DataTemplate Image is Collapsed for DisplayCapture). +2. Background — the scene's `Background` Source, `UniformToFill` full-frame (the `ActiveBackgroundImage` + layer, not per-element). +3. Elements in `Scene.Elements` order (back→front), skip `IsVisible=false`. What actually renders: + 1. `Source` Type `Image` → static asset, `UniformToFill` cover-crop into (X, Y, W, H) + 2. `WebcamSceneConfig` → latest frame by `DeviceId`: Traditional = `UniformToFill` rect; Round = circle + diameter `min(W,H)` (alpha 0 outside — true circle, not oval); mirror = horizontal flip around + element center (`MirrorScale`); opacity = per-pixel multiply (content + border); border = stroked + rect / centered circle at `RoundBorderSize`, width `BorderWidth`, alpha `BorderOpacity` + 3. `Background` / `IsBackdrop` / `TextOverlay` are NOT per-element (layers above; Text not shipped) +4. Branding flash — pre-rendered full-frame "made with ytLlive!" at 25% alpha when live + + `BrandFlashEnabled` + timer active. Passed in as a `VideoFrame?` (compositor core stays pure byte-math, + no WPF; likely a bundled asset rather than runtime text rendering). +5. NOT in output (preview chrome only): SelectionOverlay, DimRects, output-rect outline, badge, placeholder. + +**New files (all in `Services/Compositor/`):** +1. `SceneCompositor.cs` — `Render(Scene, frameFor: Func, flashFrame: + VideoFrame?, CompositorOptions) → VideoFrame` (output-sized). The caller's `frameFor` resolver maps + each element to its frame (webcam → DeviceId, image → AssetId via `StaticPixelCache`, backdrop → + CaptureKey) — the compositor stays pure/hermetic/no WPF. +2. `CompositorOptions.cs` — source-rect + output W×H. +3. `StretchMath.cs` — `UniformToFill` cover-crop, ellipse mask, bilinear scale (pure, unit-tested). +4. `StaticPixelCache.cs` — asset `byte[]` → cached BGRA `VideoFrame` (WPF `BitmapDecoder` + `CopyPixels`, + decode once per content hash). + +**Test plan (Good Dog Rule — ONE integration test):** `SceneCompositorTests` — a scene with backdrop +(solid red fake frame) + round webcam (solid green) + image (solid blue) → render 16:9 master → assert +per-layer probe pixels (corner = backdrop color, element center = webcam color, outside the round clip = +backdrop color, mirrored element swaps left/right); a vertical-tier variant asserts 1080×1920 output + +crop fidelity. Focused unit tests on `StretchMath`. Tests push frames directly — no capture managers +involved (they wire in a later step). + +**Same-PR housekeeping:** fix the stale comment `MainViewModel.cs:324` ("shown under the meter on line 2" +→ "shown left-justified INSIDE the meter bar" — `ai.md` is the authority); this task's requirements now +include the explicit audio-capture/meter wiring (#7 above). + +**Out of scope (later ship steps):** FFmpeg locator + license posture (covered in requirements 1-2), +encoder + RTMP push, WASAPI audio capture (loopback + mic) feeding `AudioLevel`, wiring +`CameraManager`/`ScreenCaptureManager` into the frame pipeline, brand-flash timer wiring, health stats +(bitrate/FPS/dropped). + +**Built (2026-08-10):** all four files shipped in `Services/Compositor/`, `SceneElement.TryGetBorderColor` +made public (shared hex parse with the compositor — no duplicated color parsing), the stale +`MainViewModel.cs:324` comment corrected, and the pre-existing CS1998 in `YouTubeAuthServiceTests` +cleaned up — build **0 warnings**. Tests: the `SceneCompositorTests` integration test (full-scene master +pixels, vertical tier, flash) + 4 `StretchMath` units — **72 passing**. + +#### Ship step 2 — FFmpeg locator (the encoder's binary) + +**Goal:** resolve a usable `ffmpeg.exe` on demand (the encoder's one external dependency), never shipping +a binary in the repo. Returns an absolute path; downloads only when neither PATH nor the local cache +provides one. + +**Decisions (locked 2026-08-10):** +1. **BtbN LGPL-shared win64 build** — not gyan.dev (gyan's "essentials" is GPLv3 and ships libx264, which + violates requirement 1's license posture) and **not the static lgpl build**: LGPLv2.1 §6 wants + relinkable object files for static linking, but the **shared** (dynamic-DLL) variant sidesteps that — + compliance is "license text + source offer + unmodified binaries" (see `THIRD-PARTY-NOTICES.txt` and + `ai.md` → Licensing). Drops libx264/libx265 while keeping NVENC/QSV/AMF, libopenh264 (the LGPL-legal + H.264 software fallback) and native AAC — exactly the requirement-1 encoder profile. +2. **Pinned URL** — **RE-PINNED 2026-09-01**: `https://github.com/BtbN/FFmpeg-Builds/releases/download/autobuild-2026-08-31-13-27/ffmpeg-N-126342-gf88b741dbf-win64-lgpl-shared.zip` + (BtbN **lgpl-shared** variant, ffmpeg N-126342; ~75 MB zip — earlier "~30 MB" estimate corrected). A dated autobuild tag is immutable; BtbN retention + keeps the last 14 daily builds + each month-end build for 2 years, so a cold cache after retention + expiry 404s — a logged, recoverable failure (the seam throws; the encoder step surfaces it). The + first pin (`autobuild-2026-08-09-13-03`, a daily) aged out on 2026-09-01 — the first real recording + attempt — proving the rule; the new pin is deliberately the **month-end** build (2-year retention). + Dated tags carry versioned asset names (`ffmpeg-N-…-win64-lgpl-shared.zip`), extraction is + name-agnostic; download failures now wrap in `IOException` with an actionable message + ("refresh `FfmpegLocator.PinnedUrl` or install ffmpeg on PATH") instead of leaking a raw 404. Once + cached, the URL is never touched again. The pin is a single `const`, bumpable in one place — and must + always stay on the **shared** variant (never `gpl`, `nonfree`, or static; see ai.md Licensing). +3. **Check-then-pull order** — (1) PATH probe (the user's own install wins), (2) cached + `%APPDATA%\ytLlive\tools\ffmpeg.exe`, (3) download + extract. Extract `ffmpeg.exe` **plus the + `libav*.dll` family** (the shared build's bin/ folder; Windows resolves the DLLs from the exe's own + directory) into a staging dir then move into place — a crash never leaves a corrupt or partial cache. +4. **Seam** — `IFfmpegLocator.LocateAsync(CancellationToken)`: search dirs, tools dir, and the downloader + (`Func>`) are constructor-injected with production defaults, so + tests fake the network (feeding a real in-memory zip) and never touch disk outside a temp dir. + +**New files (all in `Services/Encoder/`):** +1. `IFfmpegLocator.cs` — the seam. +2. `FfmpegLocator.cs` — the impl (PATH probe → cache → pull+extract exe + DLLs), failures logged via `AppLog`. +3. `THIRD-PARTY-NOTICES.txt` (repo root) — the LGPL/BSD/MIT notices + source offer, copied to the build + output. *(Surfacing changed on 2026-08-13: the top-bar About button that opened the file in the OS + viewer is GONE — the notices are reachable in-app via the logo → About overlay instead.)* + +**Test plan:** the hermetic integration test drives the full decision ladder against a temp tools dir and +a fake downloader returning a real in-memory zip (`.../bin/ffmpeg.exe` entry): PATH hit wins without +downloading, cache hit skips the network, cold cache downloads → extracts → `ffmpeg.exe` lands in the +tools dir, and a second call serves the cache (downloader invoked exactly once). Focused unit tests: +**shared-build DLLs extract alongside the exe**, empty zip throws, missing entry throws, empty download +throws, downloader failure propagates, zero-byte cache is refreshed. + +**Same-PR housekeeping:** requirement 2's stale binary facts corrected in this plan (~30 MB → ~75 MB zip; +"gyan.dev/BtB N" → BtbN LGPL-shared only, with the why); the "never do" licensing guardrails recorded in +`ai.md` so the reasoning survives. + +**Out of scope (later ship steps):** the FFmpeg subprocess encoder (frames in via stdin, stderr health +parsing), RTMP push, WASAPI audio capture, the frame-pipeline wiring, health stats. + +**Built (2026-08-10):** `IFfmpegLocator` + `FfmpegLocator` shipped in `Services/Encoder/`, pinned to the +**lgpl-shared** build `autobuild-2026-08-09-13-03` (extracts `ffmpeg.exe` + the `libav*.dll` family via a +staging dir). `THIRD-PARTY-NOTICES.txt` (repo root) ships to the build output; the "never do" licensing +guardrails are recorded in `ai.md` — build **0 warnings**. +Tests: the hermetic `FfmpegLocatorTests` integration test (PATH → cache → download decision ladder with a +fake downloader serving a real in-memory zip) + edge/unit cases (shared-build DLL extraction, zero-byte +cache refresh, empty payload, missing zip entry, downloader failure) — **78 passing**. + +#### Ship step 3 — Encoder + RTMP push (the FFmpeg subprocess) + +**Goal:** encode raw BGRA master frames into H.264+AAC FLV and push them to the reusable stream's RTMP +ingestion URL — one battle-tested subprocess doing encode + mux + push (retry is app-side, TASK 32 — ffmpeg does not self-reconnect an RTMP output), the app feeding +frames via stdin and parsing stderr for health (req 2). + +**Decisions (locked):** the encoder is a thin orchestrator over `ffmpeg.exe` — no H.264/AAC code in the +app. Arguments (pure `FfmpegArgs.Build`): `-re -f rawvideo -pix_fmt bgra -video_size WxH -framerate FPS +-i pipe:0` (frames in), a **silent placeholder audio track** via `-f lavfi -i anullsrc` (the WASAPI +capture step replaces this input), `-c:v -b:v K -maxrate K -bufsize 2K` + **`-g fps×4` +`-keyint_min fps×4` `-sc_threshold 0` `-bf 0` `-pix_fmt yuv420p`** (the keyframe ≤4s / closed-GOP / +H.264 compliance), `-c:a aac -ar 48000 -ac 2`, `-f flv `. Encoder choice is **probed from the +binary's `-encoders` listing** (`FfmpegEncoderPicker`, pure): hardware NVENC → QSV → AMF, then OpenH264 +software fallback — **never libx264** (GPL; see `ai.md` → Licensing). The seam (`IFfmpegEncoder` + +`IEncoderProcess`, constructor-injected locator + process factory) keeps it hermetic — tests fake the +whole subprocess (probe + encoder), no real binary. + +**Behavior:** `StartAsync` (locate → probe → spawn → stderr loop), `SubmitFrameAsync` (serialized BGRA +stdin writes, ~2 Hz health via `HealthUpdated`/`StreamHealth` — bitrate/FPS/duration/dropped-from-frame- +count), `StopAsync` (stdin EOF → ffmpeg finalizes + exits by itself; 10s watchdog kill), `ProcessFailed` +on a non-zero unexpected exit. + +**Built (2026-08-12):** `EncoderOptions` + `IFfmpegEncoder`/`FfmpegEncoder` + `IEncoderProcess`/ +`FfmpegEncoderProcess` + pure `FfmpegArgs`/`FfmpegProgressParser`/`FfmpegEncoderPicker` in +`Services/Encoder/`. Not yet constructed by the app (the frame-pipeline wiring, ship step 5, owns it). +Tests: `FfmpegEncoderTests` integration (probe → spawn with NVENC preferred → frames into stdin → +progress parsed → graceful stop, no kill) + units (args compliance/GOP, progress parser, picker +preference + GPL guard, no-URL/not-running/noop stops, process-death `ProcessFailed`) — **122 passing**. + +#### Ship step 4 — WASAPI audio capture (the meter comes alive) + +**Goal:** capture desktop/game audio (loopback) and the picked mic, feed the mic level into `AudioLevel` +so the realtime meter reads something other than 0, run capture **only while live** (req 7). + +**Decisions (locked):** +1. **NAudio `NAudio.Wasapi` 2.2.1** — the wasapi feature package (not the `NAudio` meta-package): it + carries the capture types (`WasapiCapture`/`WasapiLoopbackCapture` + the MMDevice enumeration) with + `NAudio.Core`/`NAudio.Asio` pulled in transitively. MIT — recorded in `THIRD-PARTY-NOTICES.txt` (item 9). +2. **`IAudioSource` seam** (`Start`/`Stop`/`SampleReady`/`Failed`, IDisposable) — the app consumes the + seam; the two WASAPI implementations wrap NAudio; tests inject hermetic fakes (no real audio devices, + no timers). Loopback = `WasapiLoopbackCapture` on the default render device; mic = `WasapiCapture` + with the NAudio device resolved by `FriendlyName` matching `MicSourceName` (the app only persists the + DisplayName), falling back to the default capture endpoint. Mic device resolution is re-read at each + `Start` via a name provider so a mic picked mid-session takes effect next go-live. +3. **`AudioMixer` owns both sources** — starts/stops both with go-live (`BeginGoLive` success → `Start`, + `StopStream` → `Stop`). Mic samples feed a pure `AudioLevelMeter` (RMS, exponential smoothing) and + raise `MicLevelChanged`, marshalled to the UI thread into `AudioLevel`; desktop samples are currently + dropped (consumed by the encoder's AAC mix in a later step). Capture failures are logged via `AppLog` + (mic failure also zeroes the meter); loopback failure doesn't kill the mic. +4. **Byte→float** — pure `WaveToFloat.Convert` handles the WASAPI mix formats: IEEE float 32-bit (direct) + and PCM 16-bit (normalized to -1..1), including `WaveFormatExtensible` with the IEEE-float subformat + GUID. Trailing partial samples are ignored. + +**Built (2026-08-12):** `Services/Audio/` ships `IAudioSource` + `AudioSample`, `WasapiLoopbackAudioSource`, +`WasapiMicAudioSource`, `AudioMixer`, `AudioLevelMeter`, `WaveToFloat`; `MainViewModel` constructs the +mixer (mic source fed `() => MicSourceName`), starts it on go-live and stops it on end-stream, and maps +`MicLevelChanged` → `AudioLevel`. A pre-existing CS8602 in `FfmpegEncoder.cs:139` surfaced during this +step's rebuild and was fixed (`process!`) — build **0 warnings**. Tests: `AudioMixerTests` (mixer +lifecycle/forwarding/failure against fakes, meter RMS/smoothing/reset, `WaveToFloat` float/PCM16/ +extensible/truncation) — **139 passing**. + +**Deferred (later ship steps):** wiring the desktop-capture samples into the encoder's AAC mix (replaces +the `-f lavfi -i anullsrc` placeholder; the encoder construction itself shipped in ship step 5). +*(The "capture while not live" + "audio UI beyond the mic controls" deferrals were SHIPPED on the +2026-08-13 game audio bar branch — capture is now always-on for preview and the game bar is the second +audio UI. The mixer's short-circuit meter fix + `Started`/`RestartMic` seams live in the same branch.)* + +#### Ship step 5 — Frame-pipeline wiring (the encoder gets a frame source) + +**Goal:** the chain `CameraManager`/`ScreenCaptureManager` → compositor resolver → encoder, driven while +live by a paced frame pump: snapshot the active scene → resolve each element to its latest frame → +composite into the tier's output frame → pace into the encoder's stdin at the tier's FPS. + +**Decisions (locked via user Q&A, 2026-08-12):** +1. **Video pipeline first** — the `-f lavfi -i anullsrc` silent track stays; mixing the loopback/mic + WASAPI samples into the encoder's AAC track is its own later step. +2. **RTMP URL via a provider seam** — `MainViewModel._rtmpUrlProvider` is a `Func` returning + null today (the reusable stream's ingest URL lands with TASK 9); when it yields null the pump logs + and skips the encoder entirely, so go-live runs the existing visual flow without pushing. + +**Design:** +1. `Services/Encoder/FramePump.cs` — the frame producer. All collaborators constructor-injected seams + (`Func`, `Func` resolver, `Func`, + `Func`, `Func`, `Action` log, injectable pacing delay) so it + stays free of WPF and of the capture managers and is hermetic in tests. `StartAsync` never throws + (failures log + surface via `Failed` — the VM fires-and-forgets from the sync command handler); + loop = snapshot → render → `SubmitFrameAsync`, paced at `1/options.Fps` (default `Task.Delay`; tests + inject `Task.Yield`). `StopAsync` stops the encoder (closes stdin) BEFORE awaiting the loop — closing + stdin unblocks a write stuck on pipe backpressure, so stop can't deadlock on the pump. `ProcessFailed` + self-stops the pump. `HealthUpdated` forwards the encoder's stats (ship step 6 binds the bottom bar). +2. `ScreenCaptureManager.GetLatestFrame(key)` — mirrors `CameraManager.GetLatestFrame(deviceId)`; the + backdrop's live frame for the compositor. +3. `MainViewModel` — owns the resolver (`WebcamSceneConfig` → `GetLatestFrame(WebcamId)`; + `Source.IsLiveCapture` → `GetLatestFrame(CaptureKey)`; image/background → `StaticPixelCache.Get(AssetId)`), + builds `CompositorOptions` from the tier + `OutputRect*` (doubles rounded to ints — the vertical + 607.5 half-pixel crop rounds to a perfectly-centered 608), builds `EncoderOptions` from the tier when + the URL provider returns one, constructs the real `FfmpegEncoder(new FfmpegLocator())`, starts the + pump on go-live, stops it on end-stream, disposes in `Shutdown`, and flips `StreamStatus.Error` when + the pump fails while live (minimal — detailed health surfacing is ship step 6). + +**Test plan (Good Dog Rule — ONE integration test):** `FramePumpTests.Start_CompositesScene_FeedsEncoder_StopsCleanly` +drives the full lifecycle against fakes — real `SceneCompositor` + real `FramePump`, fake `IFfmpegEncoder` +— asserting the composited red backdrop frame actually reaches the encoder at the tier size and that stop +tears everything down. Units: no-URL start skips the encoder, re-entrant start/stop no-ops, encoder +start-failure raises `Failed` + disposes, `ProcessFailed` self-stops the pump, `HealthUpdated` forwards. +`ScreenCaptureManagerTests.GetLatestFrame_ReturnsLatestPump_UntilReleased` pins the new accessor. + +**Out of scope (later ship steps):** the loopback/mic → AAC mix (replaces `anullsrc`), health stats in the +bottom bar (ship step 6), scene-switching transitions, and any flash-frame wiring. + +**Built (2026-08-12):** `FramePump` shipped in `Services/Encoder/`, `ScreenCaptureManager.GetLatestFrame` +added, `MainViewModel` wired end-to-end (resolver + both option builders + pump lifecycle), `FramePumpTests` +(7) + `GetLatestFrame` test (1) added — build **0 warnings**, **147 tests passing**. Known consideration: +the pump reads the active scene on a background thread while the UI can still edit it; a concurrent-mutation +exception is contained (logged + `Failed` + pump stops) rather than crashing. + +#### Ship step 5.5 — Social bar bug fixes + the bar on the live output (2026-08-13) + +**Bug 1 — bar wouldn't reliably change position (root cause found, then simplified):** the original +drag set `Canvas.SetTop(bar, …)` with a local value, which permanently overrides the +`Canvas.Top="{Binding SocialBarTop}"` binding — the release-time `SetSocialBarPosition` → +`PropertyChanged(SocialBarTop)` could never beat it. First fix added direction-snapping during the drag +(`SocialBarSnap.Decide`, ±6px deadzone) + `bar.ClearValue(Canvas.TopProperty)` on release — but that +still misbehaved for shaky hands (jitter around the deadzone: it snapped up reliably, then refused to +come back down and snapped back to top). **Superseded by a click-toggle (KISS, user decision):** clicking +the bar in the preview flips it top ⇄ bottom (`MainViewModel.ToggleSocialBarPosition` → the existing +`SetSocialBarPosition`), the bar rides `{Binding SocialBarTop}` alone (no local values, no deadzone, no +jitter sensitivity), and `SocialBarSnap` was removed. The `ClearValue` lesson stands: never set a local +value on a property the binding owns. + +**Bug 2 — Mastodon showed the generic 7-star honeycomb (root cause found):** the DB row +`@gramps@llamachile.tube` had `Software = NULL` — nodeinfo was only ever resolved against the identity domain +(`llamachile.tube`, a landing page), never probed for the real instance at `mastodon.llamachile.tube`. +Fixed on three fronts: `HttpSocialValidator` now **probes well-known subdomains** (mastodon. → social. → +… `FediverseSubdomainCandidates`) when the identity domain and its redirect both come up empty, under a +~15s linked-CTS budget, with an optional `Action` log; `MainViewModel` **heals** any fediverse entry +missing a software name on layout load (`HealFediverseSoftwareAsync` — static, testable; instance wrapper +runs it off the UI thread, applies via the dispatcher, saves); `SocialEntry.FediverseSoftware` is now +**settable** and raises `PropertyChanged` for `LogoData`, so the heal updates the icon in place. If the user +later re-saves the entry with the icon fixed, the name persists with it. + +**Compositor rendering (user-approved scope):** the social bar now appears **on the live output**, not just +the preview — `Compositor/SocialBarRenderer.cs` rasterizes the entries into a transparent straight-alpha +BGRA strip (1920-wide, 40px content + 24px glow pad, green `#2ecc71` glow baked in) via `RenderTargetBitmap` +(WPF glue like `StaticPixelCache`; the compositor core stays pure). `SceneCompositor.Render` takes optional +`socialBarFrame` + `socialBarTop` (master space) and blits it **last — above the branding flash** (the old +`BlitFlash` generalized to offset `BlitOverlay`). `FramePump` gains a `socialBar:` seam +(`Func<(VideoFrame?, SocialBarPosition)>`, re-read every frame) and places the bar at `0` or +`SourceRectHeight − bar height`. `MainViewModel` owns the frame (`RenderSocialBarFrame`, re-rendered on +load/save/notify) and feeds the seam. + +**Tests (+6 → 153 passing, 0 warnings):** settable `FediverseSoftware` updates `LogoData`, +subdomain-probe unit + null-when-silent unit, the branch's **one integration test** +`Socials_HealMissingFediverseSoftware_RoundTripsThroughDb` (temp-DB roundtrip: NULL software → healed via +the validator → persisted), compositor bar overlay (top/bottom + above-flash), `FramePump` bar pass-through +(Top then flipped to Bottom mid-run — the seam is re-read each frame), and both `ISocialValidator` fakes +(`FakeValidator`/`BlockingValidator`) gained `ResolveFediverseSoftwareAsync`. The drag-snap units +(`SocialBarSnap`) were removed with the click-toggle supersession. + +**Not included (say the word):** rewriting the healed entry's `ProfileUrl` to `https://mastodon.llamachile.tube/@gramps`. + diff --git a/TASKS/task-05-layout-persistence.md b/TASKS/task-05-layout-persistence.md new file mode 100644 index 0000000..e96692d --- /dev/null +++ b/TASKS/task-05-layout-persistence.md @@ -0,0 +1,47 @@ +# TASK 5 — Layout Persistence (SQLite) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** Scenes, sources, and asset bytes survive restarts; assets are always available. + +### Status: ✅ Done + +1. ✅ SQLite database (`Microsoft.Data.Sqlite`) at `%APPDATA%\ytLlive\ytLlive.db`; schema versioned via `PRAGMA user_version` (currently **v8**) +2. ✅ Assets live in the DB (BLOB keyed by SHA-256 content hash), never file paths — deleting the original file never breaks a scene +3. ✅ File-model save/open — the active layout file is tracked (default is the AppData DB); **Save Layout As… / Open Layout…** switch the active file; auto-save writes to whatever is active +4. ✅ Auto-save (invisible) — ~1.5s debounce on scene/source add/remove/reorder/rename/hide + any source transform change; flush on window close +5. ✅ Startup — load the active file; seed the five canonical scenes only when the DB is empty; (+) re-adds a missing canonical scene and is hidden once all five are present; adding beyond the five is rejected +6. ✅ Schema v1 → v9 — webcam columns (v2), singleton `Webcam` + per-scene `WebcamSceneConfig` (v3), `RectWidth`/`RectHeight` round-to-rect restore (v4), `Source.IsBackdrop` + `Source.CaptureKey` (v5), `Scene.HasBackdrop` — backdrop **Live-only by policy** (v6, one-time backfill + `EnforceBackdropPolicy` on every load), `Scene.HasSocialBar` (v7, dropped per-scene toggle — column back-compat, unread), `Socials.BarEnabled` (v8); the `SocialEntry.Software` fediverse-software column is a **column-presence migration** (commented v8→v9, no version bump — `user_version` stays 8); v9 = single-row `Music`. **Load-time rule:** sources load with `IsBackground` derived from `Type` (OR'd with the persisted column so legacy DisplayCapture backdrops keep their flag) — rows saved before the Type setter derived the flag heal on load +7. ✅ WindowHandle stays in-memory (per-session); save = transactional rewrite; orphaned assets pruned. **v9 lands in the TASK 8 audio milestone** (single-row `Music` — `TrackPath`/`IsEnabled`) + +### Design decisions + +1. **SQLite database** (`Microsoft.Data.Sqlite`) at `%APPDATA%\ytLlive\ytLlive.db`; schema versioned + via `PRAGMA user_version`. +2. **Assets live in the DB, not on disk** — `Asset` table stores image bytes (BLOB) keyed by a + SHA-256 content hash (unique). Identical image content collapses to one row regardless of file + name — the 1:M resource memory model, enforced by the database. No file paths; deleting the + original file never breaks a scene. +3. **File-model save/open** — the active layout file is tracked (default is the AppData DB). + **Save Layout As… / Open Layout…** switch the active file; auto-save writes to whatever is active. +4. **Auto-save (invisible)** — ~1.5s debounce on scene add/remove/reorder/rename/hide, source + add/remove/reorder, and any source transform change; flush on window close. +5. **Schema** — `Scene` (Id, Name, IsHidden, IsChatScene, HasBackdrop, HasSocialBar, SortOrder), `Asset` (Id, Hash, Data, + PixelWidth, PixelHeight), `Source` (Id, SceneId FK cascade, AssetId FK, Type, Name, IsEnabled, + X/Y/Width/Height/Opacity, MonitorIndex, DeviceId, ClipShape, IsMirrored, SortOrder), `Socials` (Id, + BarPosition, BarJustify — back-compat, unread, BarEnabled added via `ALTER`), `SocialEntry` (Id, + SocialsId FK cascade, Service, Handle, ProfileUrl, SortOrder) — `user_version` **8** (v1 → v2 = + `ALTER TABLE` adds the two webcam columns; v3 = singleton `Webcam` + per-scene `WebcamSceneConfig`; + v4 = `WebcamSceneConfig.RectWidth`/`RectHeight` for the round-to-rect restore; v5 = `Source.IsBackdrop` + + `Source.CaptureKey` for the live-capture backdrop; v6 = `Scene.HasBackdrop` — the backdrop is + **Live-only by policy** (one-time backfill turns Starting/BRB/Chat/Ending off and drops their backdrop + sources; `EnforceBackdropPolicy` re-normalizes every load); v7 = `Scene.HasSocialBar` (per-scene toggle + dropped — column back-compat, unread); v8 = `Socials.BarEnabled`). The `SocialEntry.Software` + fediverse-software column is a column-presence migration (commented v8→v9, no version bump). + `WindowHandle` stays in-memory (per-session). Save = transactional rewrite; orphaned assets pruned. +6. **Startup** — load the active file; seed the five canonical scenes + (Starting/Live/BRB/Chat/Ending, `SceneCatalog`) only when the DB is empty. The (+) + button re-adds a missing canonical scene and is hidden once all five are present; + adding beyond the five is rejected — work with less, never more. + diff --git a/TASKS/task-06-ui-polish-batch.md b/TASKS/task-06-ui-polish-batch.md new file mode 100644 index 0000000..5173124 --- /dev/null +++ b/TASKS/task-06-ui-polish-batch.md @@ -0,0 +1,21 @@ +# TASK 6 — UI polish batch (scenes/sources rows, dedup naming, social bar) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** clean up the two side lists and the social bar per gramps's review. + +### Status: ✅ Done + +1. ✅ Scene rows are pure selection rows — the per-row edit/trash/visibility icons and the inline rename TextBox are gone (`EditSceneCommand`/`RemoveSceneCommand`/`ToggleSceneVisibilityCommand` + handlers + `Scene.IsEditing` removed; `IsHidden` stays persisted + dims hidden rows) +2. ✅ Source rows gained the trio — edit (inline rename via new `EditElementCommand` + `SceneElement.IsEditing`), visibility eye (new `ToggleElementVisibilityCommand` flips `SceneElement.IsVisible`; the eye style now binds `IsVisible`, open/slashed + row dims to 0.45 when hidden), and the existing trash +3. ✅ Duplicate resource names get a no-space incrementing suffix via shared `NextSourceName` (Image, Image2, Image3…) — next free number derived from the names actually in the scene, so deleting a middle resource never collides (`AddSource` + `AddReusedImage` both use it) +4. ✅ Social bar renders the full validated handle — `MaxWidth=200` + `TextTrimming` removed from BOTH `SocialBarRenderer` and the preview template (mastodon `@gramps@…` no longer cuts off) +5. ✅ Side panels stay fixed-width (left 220 / right 300) — deliberate: they never re-layout on resize, the preview absorbs it +6. ✅ Focus-loss capture lag documented as a known OS limit in `ai.md` — deferred by user decision (no code change) + +### Design decisions + +1. **Next free number from names, not type counts** — the old scheme counted elements by `SourceType` (`count == 0 ? baseName : base+count+1`), which collided after deletions; the new helper scans actual names. +2. **One WPF App per AppDomain** — the real-App tests (round-clip + naming) share `RealAppHost` (a dedicated STA thread owning the single `App`) via the `RealApp` serial collection, instead of each calling `new App()`. + diff --git a/TASKS/task-07-meter-scaling.md b/TASKS/task-07-meter-scaling.md new file mode 100644 index 0000000..95fdedf --- /dev/null +++ b/TASKS/task-07-meter-scaling.md @@ -0,0 +1,17 @@ +# TASK 7 — Meter scaling amplification (voice meter uses the full bar) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** the mic (and game) meters feel light — speech peaks should peg into the red at maxed volume. + +### Status: ✅ Done + +1. ✅ `AudioLevelMeter.ToDisplay` now adds **+10 dB of input amplification** before the −60..0 dBFS → 0..1 log mapping (was raw dB): speech peaks (~0.2 RMS, −14 dBFS) read ~0.93 → red zone; normal speech (~0.05, −26 dBFS) ~0.73 → yellow; background noise ≤0.001 linear (−60 dBFS) still reads 0 (the meter never idles on it) +2. ✅ Unit tests updated to the new mapping + new `ToDisplay_Pushes_Speech_Peaks_Into_Red_At_Maxed_Volume` (0.2 → 0.92..0.95, 0.05 → 0.72..0.75) + +### Design decisions + +1. **Amplify at the mapping, not in the VM** — one knob (`ToDisplay`), shared by the mic bar and the game bar; the `× MicVolume` gain-on-noise behavior is untouched (raising the slider still moves ambient noise up the bar). +2. **+10 dB, not more** — louder boosts push quiet speech into the lower half and make the noise floor visible; +10 puts speech peaks solidly in red while idle stays at zero. + diff --git a/TASKS/task-08-audio-milestone.md b/TASKS/task-08-audio-milestone.md new file mode 100644 index 0000000..29618d8 --- /dev/null +++ b/TASKS/task-08-audio-milestone.md @@ -0,0 +1,33 @@ +# TASK 8 — Audio milestone: real stream audio + voice filters + TRAX music (2026-08-14) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** every audio issue done and tested in one branch — real mic/game audio reaches the encoder (replacing the `anullsrc` silence), `MicVolume`/`GameAudioVolume`/mute become real pre-AAC gains, TASK 10's voice filters land, auto-duck keeps the creator's voice over game + music, and a free background-music source ("TRAX") plays into the sound bar and the stream. + +### Status: ✅ SHIPPED 2026-08-14 — 196 tests passing, 0 warnings (ONE integration test: `AudioPipelineTests.Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe`) + +1. ✅ **Real audio into the encoder** — the mixer's loopback samples were **dropped today** (`AudioMixer.OnLoopbackSample` metered only) and `MicVolume` was meter-only; `FfmpegArgs.cs:28` ran `-f lavfi -i anullsrc` (silence). Transport: **ffmpeg reads a Windows named pipe** — `-f f32le -ar 48000 -ac 2 -i \\.\pipe\ytllive_audio` replaces the anullsrc block, plus explicit `-map 0:v -map 1:a`; pipe name via `EncoderOptions.AudioPipeName` (`DefaultAudioPipeName = "ytllive_audio"`). Mixer stays **two inputs (mic + loopback)** — no N-source abstraction, no `StereoMixer` class. The VM owns the pipe lifecycle (`BeginGoLive` → `_audioMixer.StartLive(pipeName)`, `StopStream` → `_audioMixer.StopLive()` before the pump stops); `FfmpegEncoder.cs` is untouched. +2. ✅ **Honest gains reach the stream** — `MicVolume` scales the mic channel; `GameAudioVolume` + mute scale the loopback channel (the "mute in preview but stream still plays" footgun dies with it). One knob per input = KISS. Implemented as `Func` gain seams on `AudioMixer` (`micGain`/`loopbackGain`), wired by `MainViewModel`. +3. ✅ **Voice filters (TASK 10, before the meter AND the mix)** — `VoiceFilterChain`: bass (`LowShelfFilter` 120 Hz +4 dB) → treble (`HighShelfFilter` 8 kHz +3 dB) → **noise gate** (pure C#, creator's choice) → compressor (threshold 0.5, 4:1). Pure stateful DSP (TDF2), sine-in unit tests. Meter stays post-filter. +4. ✅ **Auto-duck** — pure envelope (`AutoDucker`): mic RMS > 0.02 → game+music dip ×0.25 (~12 dB), attack 0.05 / release 0.005, recover on release. Always-on, no knobs. Strong form of the creator's "voice always over the game volume" idea. +5. ✅ **TRAX — background music, FREE** (keeps the one paid line = Alerts + flash removal). `MusicPlayer`: NAudio `MediaFoundationReader` (mp3/wav/m4a) → `VolumeWaveProvider16` at the hardcoded **0.20** (fixed, not changeable) → `WaveOutEvent` on the default device (added the sibling `NAudio.WinMM` 2.2.1 package — `WaveOutEvent` isn't in `NAudio.Wasapi`), loop on end. **No 3rd mixer input and no `MusicVolume`** — music plays on the desktop, the existing loopback picks it up: heard in headphones, the sound-bar meter bounces, and the stream carries it through the loopback channel (ducked with game when the voice is hot). Known wrinkle (out of scope): YouTube mutes VODs carrying copyrighted music — future "music on live, off VOD". +6. ✅ **TRAX footer control (final spec, agreed 2026-08-14)** — `YtButtonSecondary`, **status dot + "TRAX"** text, in the center footer stack beside MIC: + - **Status dot** (same 8px Ellipse pattern as MIC/Socials): **red** no track loaded · **yellow** loaded not playing · **green** playing. + - **Left-click** → toggle play/pause; no track loaded → opens the picker instead. + - **Right-click** → always opens the in-app track picker (`TraxButton_PreviewMouseRightButtonUp`, code-behind pattern like `GameSpeaker_MouseLeftButtonUp`, `e.Handled = true` so no context menu). + - **Tooltip** → playing/loaded track name; "No track — right-click to choose background music" when empty. + - **No slider** — volume is the 0.20 constant. Picker = plain OS `OpenFileDialog` filtered to mp3/wav/m4a/aac/flac/ogg. +7. ✅ **Sound bar shows music** — `IsGameAudioBarVisible = gameDetectorProducingSound || isMusicPlaying`; relabelled "Game Audio Capture" → **"Desktop Audio"** (TRAX rides the same channel). **Follow-up (2026-08-15): the gating is gone — the game audio bar is ALWAYS visible** (the `IGameAudioDetector`/`GameAudioDetector`/`GameAudioHysteresis` stack + the VM's 250ms poll timer and their tests were deleted; the bar kept hiding the desktop meter when no full-screen game with sound was up, and the creator wants it pinned). +8. ✅ **Capture hardening (design changed at build time)** — the plan's "force IEEE-float 48 kHz on both WASAPI sources" was **dropped**: NAudio 2.2.1's `WasapiCapture` exposes no overridable `GetDefaultMixFormat`, so both sources capture the device's own mix format and the mixer's pure `TinyResampler` normalizes any rate/channel count to 48 kHz stereo (the resampler IS the design, not a fallback). Stop path disposes the audio pipe (EOF) in `MainViewModel.StopStream` before the pump stops, so both ffmpeg inputs end in order. +9. ✅ **Schema v9** — single-row `Music` (`TrackPath`, `IsEnabled`; volume is the 0.20 constant) + migration in `LayoutStore.cs` (`user_version` 9; `Save` gained an optional `Music? music = null` param so existing 3-arg callers still compile; load query + null reset). +10. ✅ **Docs in the same commit** — `ai.md` audio section rewritten ("one knob per input", 2-in mix, auto-duck, TRAX free, VOD-mute wrinkle), TASK 10's status, this task's status, HANDOFF. + +**Files (new):** `Services/Audio/VoiceFilterChain.cs` (+ `LowShelfFilter`/`HighShelfFilter`/`NoiseGate`/`Compressor`), `Services/Audio/AudioRingBuffer.cs` (async-arrival source buffers), `Services/Audio/AutoDucker.cs`, `Services/Audio/TinyResampler.cs`, `Services/Audio/MusicPlayer.cs`, `Services/Audio/NamedPipeAudioWriter.cs` (+ `IAudioPipeWriter` seam), `Models/Music.cs`. + +**Files (changed):** `AudioMixer.cs` (filter chain before meter+mix, loopback into the mix, gains via `Func` seams, ducker, silence-filler so the pipe never stalls, `StartLive`/`StopLive`), `WasapiMicAudioSource.cs` (device mix format — resampler normalizes; no forced format), `FfmpegArgs.cs` (named-pipe input + `-map`, name via `EncoderOptions.AudioPipeName`), `EncoderOptions.cs` (`AudioPipeName`), `ytLive.csproj` (`NAudio.WinMM` 2.2.1 for `WaveOutEvent`), `LayoutStore.cs` (schema v9), `MainViewModel.cs`/`MainWindow.xaml`/`MainWindow.xaml.cs` (footer TRAX group, sound-bar visibility OR music, go-live starts the pipe, end closes it first). + +**Testing (Good Dog Rule — ONE integration test):** `AudioPipelineTests.Mix_WithFiltersDuckAndGain_Lands_On_AudioPipe` — real filter chain + mixer + ducker + `NamedPipeAudioWriter`, fake `IAudioSource`s, test-side `NamedPipeClientStream` reads the bytes and asserts post-filter/post-gain/post-duck mixed stereo (ring buffers pre-filled so the first pipe tick already carries real audio — avoids a start-of-stream silence race). Units: each DSP stage (known sine-in → expected gain), ring buffer wrap/underflow/overwrite, ducker envelope, resampler (down/up/stateful/identity), `FfmpegArgs` pipe+map, schema v9 roundtrip + no-music-row. Existing tests stay green. + +**Out of scope (this branch):** IP webcam (video-only when it lands — never an audio input), chat box source (TASK 3 item 18), alt-key crop, credits, background removal, music-off-VOD, `PremiumUrl` (TASK 10 seam). + diff --git a/TASKS/task-09-live-stream-management.md b/TASKS/task-09-live-stream-management.md new file mode 100644 index 0000000..f027dc1 --- /dev/null +++ b/TASKS/task-09-live-stream-management.md @@ -0,0 +1,37 @@ +# TASK 9 — YouTube Live Stream Management + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** Create/bind broadcasts, monitor YouTube-side stream health — the v3 way. + +### Status: ⏳ In progress — items 1–3 SHIPPED (reusable stream 2026-08-16; report-by-exception health 2026-08-16); item 4 SHIPPED (status reconciled 2026-09-01); item 5 open; item 6 = deliberate lock (TASK 36); item 7 scoped to core fields — advanced tab is out-of-product + +1. ☑ **Broadcast creation** — title/description/privacy/scheduledStartTime via API, with the v3 flags above (SHIPPED: `CreateBroadcast` sends `enableAutoStart/Stop`, `enableMonitorStream=false`, `latencyPreference=low`, `selfDeclaredMadeForKids=false`) +2. ☑ **Reusable stream** — create once, cache + reuse; bind to broadcast (SHIPPED: `GetOrCreateReusableStreamAsync` lists-then-inserts the `variable`/`isReusable` stream, cached via `LayoutStore` Settings, bound at broadcast insert via `boundStreamId`; `_rtmpUrlProvider` yields the ingest URL so go-live actually encodes + pushes) +3. ☑ **Health monitoring** — poll `liveStreams.list` `healthStatus` + `configurationIssues[]`, surface banner only on warning/error (SHIPPED: `GetStreamHealthAsync(streamId)` 30s while live; pure `StreamHealthReporter.BannerFor` = report-by-exception; banner strip under the top bar, amber warning / dark-red error, via `HealthIssueBanner`/`HealthIssueBackground`; poll failures log-only; ONE integration test `GetStreamHealthAsync_Report_By_Exception_Banner_Only_On_Warning_Or_Error`) +4. ☑ Live chat — poll `liveChat/messages`, render in right panel, support Super Chat + membership badges — **SHIPPED** (status reconciled 2026-09-01: `YouTubeChatService` polls + parses `superChatEvent`/`newSponsorEvent` with badge/level on `ChatMessage`; the panel MOVED to the left-panel live state by Control Surface UX, "right panel" superseded; the remaining reward types land with the monetization chain, TASK 10 → related work) +5. ☐ Error handling — the YouTube error codes: `errorStreamInactive`, `invalidTransition`, `redundantTransition`, `liveStreamDeletionNotAllowed`, `liveStreamModificationNotAllowed`, `liveBroadcastBindingNotAllowed` +6. ❌ **Visibility picker** — DELIBERATE TEST-PHASE LOCK (creator decision 2026-09-01), not open work: "always Private" stays during multi-month real-world testing so breakage VODs never clutter the channel. `recordFromStart`/DVR stay on — Private VODs are invisible review tapes; bulk-delete pre-GA. Unlock = TASK 36 gold-pass item (remove the override in `YouTubeStreamService.CreateBroadcast`, wire the dialog selection). See `ai.md` → YouTube Live API constraints. +7. 🔶 **Full broadcast form** — RESCOPED 2026-09-01, not the old two-tab plan. **Core fields (SHIPPED as the Text drawer, 2026-08-24):** title, description, tags, visibility, made-for-kids — live-editable. **Scheduling:** ships as **TASK 34** (Scheduled checkbox + datetime in the same drawer). **Advanced tab is PERMANENTLY OUT** (the 10% margin): latency (locked `low`), DVR/record-from-start (locked on), embed, projection, CC, region restrictions stay fixed at sane defaults, invisible — every exposed field is a support ticket. `categoryId` removal: done (not a `liveBroadcast` field). Monetization enablement (if ever needed) rides the reward-events chain via `liveBroadcasts.update`, not a form field + +### Design decisions (v3) + +1. **One-click go-live** — `liveBroadcasts.insert` with `enableAutoStart=true`, `enableAutoStop=true`, `enableMonitorStream=false`, `selfDeclaredMadeForKids=false`, `latencyPreference=low`. No `transition(live)` call, no testing stage, no liveStarting polling. Encoder starts → YouTube brings it live by itself. +2. **Variable reusable stream** — `cdn.resolution=variable`, `cdn.frameRate=variable`, `isReusable=true`. Create once per channel, cache ingestion URL + stream name, reuse for every broadcast. Any quality tier works without recreation; auto step-down needs no API calls. +3. **Report-by-exception** — poll `liveStreams.list`; banner only on `healthStatus` warning/error issues (`configurationIssues[]`). Bottom strip = YouTube logo + green/red connection dot (clickable → opens the dialog). +4. **One dialog, three states** — `not connected` (sign-in) / `connected-offline` (all editable) / `live` (title + description + visibility editable; quality + account greyed out). Both entry points (Start Stream button + bottom strip) open it; prefilled from saved session profile. +5. **Live edits** — `liveBroadcasts.update` with part=`snippet,status` for title/description/privacy. +6. **End stream** — stop encoder → `transition(complete)`, with `enableAutoStop` as the safety net. **SHIPPED 2026-09-01** (`EndBroadcastAsync`, wired into `StopStream` after RTMP EOF; the call had been specced since TASK 9 but never existed — found during recording verification) +7. **Broadcast ID == Video ID** — one ID to track status, health, and the auto-created VOD (`recordFromStart` + `enableDvr`). + +### Requirements: + +1. **Broadcast creation** — title/description/privacy/scheduledStartTime via API, with the v3 flags above +2. **Reusable stream** — create once, cache + reuse; bind to broadcast +3. **Health monitoring** — poll `liveStreams.list` `healthStatus` + `configurationIssues[]`, surface banner only on warning/error +4. **Live chat** — poll `liveChat/messages`, render in right panel, support Super Chat + membership badges +5. **Error handling** — the YouTube error codes: `errorStreamInactive`, `invalidTransition`, `redundantTransition`, `liveStreamDeletionNotAllowed`, `liveStreamModificationNotAllowed`, `liveBroadcastBindingNotAllowed` +6. **Visibility picker** — Private/Unlisted/Public from the go-live dialog +7. **Full broadcast form** — Core + Advanced tabs with all API-supported fields + diff --git a/TASKS/task-10-monetization.md b/TASKS/task-10-monetization.md new file mode 100644 index 0000000..9af4da8 --- /dev/null +++ b/TASKS/task-10-monetization.md @@ -0,0 +1,39 @@ +# TASK 10 — Monetization: watermark-only subscription + Polar billing + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** annual subscription via Polar.sh that removes the branding watermark. All features are +free — the only difference between free and paid is the watermark. + +**Related work — monetization awareness (2026-09-01, ungated, free):** this task's billing is the +*payer* side; the *product* also carries built-in, ungated monetization awareness for every creator — +reward-event capture (all 7 `liveChatMessage` reward event types → canonical `RewardEvent` → SQLite +persistence), per-session/broadcast reports (members, Super Chat $, stickers, gifts, milestones — the +in-app replacement for the emailed "stream activity report"), and a journey tracker (position % + +ETA vs. versioned, date-aware YPP thresholds; the Tier-2 bar doubles for new applicants 2027-02-01). +Details and design notes: `ai.md` → Monetization. This is tracked as related work under this task's +narrative, not a separate task number — it ships in its own code slices with one integration test each. +**Scope: v1** (2026-09-01 complete-v1 ruling — there is no v1.x): slice order = reward-event capture → +session report → journey tracker → Alerts (TASK 3 item 20). + +**Business details (pricing, Polar product/checkout/discounts) in `MONETIZATION.md` (gitignored).** + +**Polar product:** `d105dfa1-497e-423b-8cd4-e0ee2e3abbc0` (LlamaCasty, $99/yr, currently `private`) +**Checkout:** `llamacasty.com` → Polar hosted page (~80% configured) +**Discounts:** `LLAMAFOUNDER` (100% off, 50 uses, `once`), `LLAMA50` (50% off, 12 months, unlimited) +**Org ID:** `c05fb364-b967-4f6c-adf2-8a144e46d085` (org-scoped OAT — `organization_id` can be omitted from API calls) +**Key prefix:** `LCYT-` +**Validate endpoint:** `POST https://api.polar.sh/v1/customer-portal/license-keys/validate` (no auth required for client-side validation) + +### Status: 🔶 In progress — steps 1–7 shipped, Velopack update URL pending + +1. ✅ **Billing provider** — Polar (polar.sh) chosen and configured. Product, price, license key benefit, discounts, checkout link all created. +2. ✅ **License key entry + verification UI** — About overlay swap panel (same pattern as Licenses panel). Text input + "Activate" button + status message. "Enter License Key" button in the hub. +3. ✅ **Offline entitlement** — `LayoutStore` Settings table: `LicenseKey`, `LicenseValidatedAt`, `IsPremium`. 14-day offline grace period. Startup re-validates online, falls back to cache. +4. ✅ **BrandFlash → watermark toggle** — `IsPremium` property setter flips `BrandFlashEnabled = !value`. Flash stops immediately on premium activation. Flash asset VideoFrame rendering deferred to TASK 4 compositor. +5. ✅ **Remove social slot locks** — all 6 social bar slots open to everyone (no `IsPremium` gating on slots 3-6) — shipped as TASK 10c +6. ✅ **"Unlock Premium" button** — `PremiumUrl` set to Polar checkout; `IsPremiumAvailable` is true; button lights up in the About hub. +7. ✅ **Renewal/lapse handling** — startup re-validates against Polar API. Lapse → `IsPremium = false` → flash returns. Grace period: 14 days offline. +8. 🔶 **Velopack auto-updates** — NuGet packages + bootstrap in `App.xaml.cs`. Update URL self-hosted on DO droplet (pending configuration). + diff --git a/TASKS/task-11-polish-batch.md b/TASKS/task-11-polish-batch.md new file mode 100644 index 0000000..075526f --- /dev/null +++ b/TASKS/task-11-polish-batch.md @@ -0,0 +1,46 @@ +# TASK 11 — Post-pause polish batch: the creator's 8 review issues (2026-08-15, shipped) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Status: ✅ SHIPPED 2026-08-15 — 197 tests passing, 0 warnings (ONE integration test: +`AudioPipelineTests.Mix_HonorsProviderGains_AndGameMute_KillsTheLoopback`).** + +The creator reviewed the TASK 8 build and filed 8 issues. All fixed in one branch on `main`: + +1. ✅ **Desktop/game audio volume slider had no effect** (headset OR stream) — the `AudioMixer`'s + `micGain`/`loopbackGain` seams defaulted to unity because the VM never passed them; the sliders + were decorative. Fixed: new `AudioGainProvider` (`Services/Audio/`) hands `MicMuted`/`MicVolume`/ + `GameMuted`/`GameAudioVolume` to the mixer at construction, read live each mix tick. +2. ✅ **Desktop/game mute button had no effect** — same root cause; `GameMuted` now zeroes the + loopback on the stream. Locally, `MusicPlayer.LocalGain` (new) is scaled by the game bar on every + volume/mute change + on TRAX load, so the creator **hears** the control work on the music (the + track rides the loopback channel, so it ducks locally exactly as it does on stream). +3. ✅ **TRAX played once then stopped** — `MusicPlayer.OnPlaybackStopped` only looped when + `Position >= Length`, unreliable for `MediaFoundationReader`; now any clean stop + (`e.Exception == null`) rewinds + replays; only errors/explicit stops surface via `PlaybackEnded`. + 4. ✅ **TRAX/MIC buttons swapped** — TRAX no longer sits between MIC and the (mic-specific) meter; + the footer's mic cluster is MIC + meter + mute + volume. + 5. ✅ **Mic source persists across restarts** — `LayoutStore` gained a `Settings` key/value table + (`SaveMicSourceName`/`LoadMicSourceName`); the VM saves on pick and restores before the mixer's + first `Start`, so a restart reconnects the same already-vetted device (green dot) or reports it + missing (yellow) instead of falling back to the default endpoint. + 6. ✅ **TRAX moved into the game audio bar** — left of the "Desktop Audio" label in the preview + overlay (the creator's follow-up pick), no longer in the footer; its tooltip teaches the clicks + ("left-click pauses/plays, right-click chooses a track"). +7. ✅ **Backdrop's icons shifted right** — the trash button's `Collapsed` released its column; a new + `HiddenBoolToVisibilityConverter` keeps the slot reserved (`Hidden`), so edit/eye stay in their + fixed columns for every source row. +8. ✅ **No separation between scenes and sources** — a 1px hairline with top/bottom padding now sits + between the two listboxes in the left panel. + +### Design decisions + +1. **The gain seams are the one source of truth** — `AudioGainProvider` is a thin seam (four Funcs) + so the mixer contract and the mute⇔zero-volume rule are testable without constructing the VM. +2. **Stream-honest AND locally audible** — the game bar's controls now do two things: scale the + loopback on the stream and scale the music in the headphones. Native game audio is untouched + (system output, OBS-style non-monitored); an OBS-style monitor loop is out of scope. +3. **Persist the DisplayName, not the device ID** — the FriendlyName match was already the mic's + identity end-to-end; `Settings` just makes it survive restarts. + diff --git a/TASKS/task-12-master-limiter.md b/TASKS/task-12-master-limiter.md new file mode 100644 index 0000000..9c71fa1 --- /dev/null +++ b/TASKS/task-12-master-limiter.md @@ -0,0 +1,31 @@ +# TASK 12 — Master limiter on the live mix (2026-08-15) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Queued by the creator during the TRAX discussion:** "should the soundtrack be limited to avoid +squandering resources / should it cap at 20% / how do the three sound events balance?" Review +conclusions (all three instincts checked out, only ONE real gap): + +- **File size — no guard needed.** `MusicPlayer` uses `MediaFoundationReader`, which **streams from + disk** (progressive source): memory is flat (~a few MB) regardless of file size; CPU negligible. +- **20% cap — already enforced by construction.** `MusicPlayer.MusicVolume = 0.20f` + music rides the + SAME WASAPI loopback (and therefore the same gain) as game audio, so music:game is always exactly + **0.20:1 at any slider position** — it literally cannot rise above 20% of the current desktop volume. +- **The gap:** `AudioMixer.FillAndMix` summed mic + loopback with **no output ceiling** — mic 100% + + loud game/music could pass 0 dBFS and clip the AAC encode. + +**Shipped:** +1. ✅ **`Services/Audio/MasterLimiter.cs`** (pure, unit-tested) — a **−1 dBFS ceiling** (`Ceiling = + 0.891`), **instant attack per frame** (a hot frame is scaled exactly to the ceiling — no overshoot), + **smoothed release** toward unity so loud passages don't pump; gain never exceeds 1 (no boosting). + Applied at the end of `AudioMixer.FillAndMix`, right before the pipe write. +2. ✅ **Unit tests** — over-ceiling frames trimmed to ≤ ceiling; sub-ceiling frames never boosted; gain + recovers to unity after the loud frame ends. +3. ✅ **ONE integration test** (`MasterLimiter_CapsTheLiveMix_OnThePipe`) — real mixer + pipe harness: + a 0.95 loopback bed (hotter than the ceiling) is capped to exactly 0.891 on the wire while staying + audible. +4. ✅ **Docs in the same commit** — `ai.md` (go-live audio section), `Services/index.md` (new row). + +No changes to `MusicPlayer`, the 0.20 cap, the ducker, or the meter zones. Build 0 warnings. + diff --git a/TASKS/task-13-social-launch-kit.md b/TASKS/task-13-social-launch-kit.md new file mode 100644 index 0000000..821cbec --- /dev/null +++ b/TASKS/task-13-social-launch-kit.md @@ -0,0 +1,19 @@ +# TASK 13 — Social media launch kit + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** marcom/social media assets and strategy for v1 launch. + +**Business details (positioning, messaging, platform strategy, launch assets) in `MARCOM.md` (gitignored).** + +### Status: 🔶 Scoped — nothing built; queued after all v1 features ship + +1. ☐ Product positioning & messaging (one-liner, elevator pitch, competitive positioning) +2. ☐ Social media swipe files (pre-written posts for supporters) +3. ☐ Launch day assets (demo video, screenshots, GIFs, social graphics, press kit) +4. ☐ Platform strategy (YouTube, Reddit, indie dev communities, Product Hunt) +5. ☐ Founder story (67-year-old dev building his own streaming app) +6. ☐ Email announcement templates +7. ☐ "Build in public" livestream angle (stream the coding of ytLlive with ytLlive) + diff --git a/TASKS/task-14-creator-feedback-batch.md b/TASKS/task-14-creator-feedback-batch.md new file mode 100644 index 0000000..8173bc6 --- /dev/null +++ b/TASKS/task-14-creator-feedback-batch.md @@ -0,0 +1,24 @@ +# TASK 14 — Creator feedback batch: quick fixes (TRAX, meter, backdrop, sliders) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** address creator's immediate UX feedback from testing session. + +### Status: 🔶 In progress — Branch 2 shipped + +1. ✅ TRAX volume — `LocalGain = GameAudioVolume * 0.2` so TRAX is 20% of the desktop audio slider (was 100%, now quieter) +2. ✅ TRAX tooltip — replaced `ToolTip` binding with `ToolTipService.ToolTip` + `Placement="Top"` to fix z-order inside Viewbox overlay +3. ✅ Mic meter boost — `MeterLevel` multiplied by 1.2× before clamping so the bar reads ~20% higher at same input level +4. ✅ Click-to-position sliders — clicking anywhere on the mic/game volume slider track now jumps the thumb to that position before starting the drag +5. ✅ Backdrop visibility — the dedicated `` now binds `Visibility` to `BackdropVisible` (ViewModel property); compositor checks `IsVisible` before blit +6. ✅ Rename "Backdrop" → "Game Capture" in `EnsureBackdrop()` + test assertions +7. ✅ Elements panel — new "ELEMENTS" section beneath Sources list; per-element config with live preview, ✕ revert, ✓ applied indicator +8. ✅ Webcam border config — color hex input + color swatch picker, border thickness 0–10 slider +9. ✅ Countdown source — `SourceType.Countdown`, pick list (Starting/BRB scenes only), timer minutes 1–60 default 15 +10. ✅ Web source — `SourceType.WebSource`, pick list (all scenes), URI text input +11. ✅ Elements scrollbar deselect fix — clicking the Elements panel scrollbar no longer deselects the selected element (scrollbar lives inside the ScrollViewer, not the StackPanel; guard now checks `ElementsScrollViewer` instead of `ElementsPanel`) +12. ✅ Web source URI + buttons inline — merged URI TextBox and check/revert buttons into a single DockPanel row (✓ rightmost, ✕ left); buttons no longer wrap to a separate line +13. ✅ Web source URI hidden for webcam — URI section was showing for webcam due to WPF binding path `SelectedElement.Type` not resolving on `WebcamSceneConfig`; replaced with `IsWebSource` virtual property on `SceneElement` (mirrors existing `IsWebcam` pattern), overridden in `Source` to return `Type == SourceType.WebSource`, bound via `BoolToVis` +14. ✅ Check/revert consistent order + accept — all three element sections (webcam border, countdown timer, web URI) now use consistent [✕][✓] button order; green check buttons enabled with click handlers that save current values as snapshot baseline (so revert undoes to last accepted state, not initial selection) + diff --git a/TASKS/task-15-scene-backgrounds.md b/TASKS/task-15-scene-backgrounds.md new file mode 100644 index 0000000..3bdbd03 --- /dev/null +++ b/TASKS/task-15-scene-backgrounds.md @@ -0,0 +1,23 @@ +# TASK 15 — Stock scene background images + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** branded background images for each canonical scene (Starting, Live, BRB, Chat, Ending) +that both provide working hints to creators and self-promote LlamaCasty. + +### Status: ✅ Shipped — all 5 scenes seeded + +1. ✅ **Starting backdrop** — `Assets/starting-backdrop.jpg` (embedded resource), `SeedStartingBackdrop()` in MainViewModel, idempotent +2. ✅ **BRB backdrop** — `Assets/brb-backdrop.jpg` (user-provided Gemini image), `SeedBrbBackdrop()` in MainViewModel, idempotent +3. ✅ **Live backdrop** — `Assets/live-backdrop.jpg` (user-provided image), seeded via `SeedLiveBackdropAsset()`, used as the static fallback (TASK 16) +4. ✅ **Ending backdrop** — `Assets/ending-backdrop.jpg` (user-provided image), `SeedEndingBackdrop()` replaces existing (not skip-if-exists) +5. ✅ **Chat backdrop** — `Assets/chat-backdrop.jpg` (user-provided image), `SeedChatBackdrop()` at startup +6. ✅ `EnsureDefaultBackdrop()` called on every scene switch — seeds the default backdrop if scene has none + +### Design decisions + +- Images are user-provided, not generated — the creator owns the brand look +- Each scene gets its own image (Starting = "Starting Soon", Live = live branding, BRB = "Be Right Back", etc.) +- The images double as the no-game backdrop fallback (TASK 16), so the Live scene image should work as a desktop replacement + diff --git a/TASKS/task-16-kill-infinity-display.md b/TASKS/task-16-kill-infinity-display.md new file mode 100644 index 0000000..a4ed9c1 --- /dev/null +++ b/TASKS/task-16-kill-infinity-display.md @@ -0,0 +1,27 @@ +# TASK 16 — Kill infinity display (no-game backdrop fallback) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** when no full-screen game is detected, show a static branded placeholder instead of +capturing the primary display (which causes the infinity mirror effect). + +### Status: ✅ Shipped + +**Changes:** +1. `ResolveAutoCaptureKey()` returns `null` when no full-screen game is detected (was: primary monitor) +2. `ReacquireScreenCaptures()` — clears stale `CaptureKey` when `auto == null` (handles DB upgrades); + skips `AcquireAsync` when no keys are set +3. `SeedLiveBackdropAsset()` — sets `AssetId` on the backdrop element itself (NOT a Background source); + called from `ReacquireScreenCaptures()` after `EnsureBackdrop()` guarantees the element exists +4. `Source.DisplaySource` falls back to `_imageSource` when `VideoImageSource` is null for live captures +5. `Assets/live-backdrop.jpg` — user-provided image, added as `` in csproj +6. `ResolveOutputFrame()` in compositor — `Source { IsLiveCapture: true, CaptureKey: not null }` already skips null keys; fallback pattern `Source { AssetId: not null }` picks up the static image + +**Rendering flow:** +| Scenario | Preview | Compositor | +|---|---|---| +| Game detected | Live capture (opaque, covers static asset on backdrop) | `ScreenCaptureManager.GetLatestFrame` | +| No game | Static image via `DisplaySource` → `_imageSource` on backdrop | `StaticPixelCache.Get(assetId)` | +| Manual capture pick | Live capture | Live frame | + diff --git a/TASKS/task-17-web-source-rendering.md b/TASKS/task-17-web-source-rendering.md new file mode 100644 index 0000000..1e301bb --- /dev/null +++ b/TASKS/task-17-web-source-rendering.md @@ -0,0 +1,47 @@ +# TASK 17 — Web source rendering (WebView2) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** make the web source actually render URLs into the preview and stream output. + +### Status: ✅ Done — required for v1 (2026-08-28) + +1. ✅ Add `Microsoft.Web.WebView2` NuGet package +2. ✅ Schema v10: `WebUri TEXT` column on `Source` table + migration in `LayoutStore.cs` +3. ✅ Persist `Source.WebUri` on save/load (currently in-memory only — lost on restart) +4. ✅ Hidden off-screen `WebView2` control per web source — navigates to `WebUri`, renders in-app +5. ✅ Frame capture from WebView2 (`CoreWebView2.CapturePreviewAsync`) → `WriteableBitmap` (BGRA8) +6. ✅ Wire into `FramePump` resolver — `Source { Type: WebSource }` → latest WebView2 frame +7. ✅ Wire into `SceneCompositor` — render web source as an image element at its position/size +8. ✅ Preview shows live web content (not just a blank rectangle) +9. ☐ Handle navigation errors, invalid URIs, timeout gracefully + +**Rendering model (2026-08-28, canvas-size viewport + ALPHA-BBOX crop — DONE, creator-verified):** +the page renders at the MASTER CANVAS size (1920×1080), stable, never tracked (no reflow/truncation; +scrollbars suppressed via `overflow:hidden`). `FindContentBounds` scans the Bgra32 capture and crops +to the bounding box of non-transparent pixels — the widget's true rendered extent, measured from the +frame itself (NO DOM query, immune to layout timing; can never truncate content; full-canvas widgets +fall through to the full frame = prior verified-good image). `Stretch="Fill"` maps the cropped frame +flush under the selection box → box hugs the widget on all four sides. `QueryContentBoundsAsync` + +the JS content-bounds script are DELETED. **RESULT: bounding verified PERFECT by the creator with +two widgets.** Root-cause note on the earlier "remaining defect/gap": that was a WIDGET GLOW EFFECT +(the widget's own CSS glow pushes out its perceived borders), not a code bug — the alpha-bbox was +already hugging the glow halo. Lesson: test with a second, plain widget before changing code. +⚠️ Lesson bank: never return `JSON.stringify` from `ExecuteScriptAsync` (double-encodes); never +measure DOM stuff on NavigationCompleted (unsettled layout broke the image — `a5b9952`); avoid +sizing the container to the crop (`ab29ec8` reverted — made it worse); a widget glow effect can +fake a gap — verify with a plain widget first; measure rendered pixels instead. + +**Properties panel (2026-08-28):** web URI ✓/✕ icon buttons are `IsTabStop="False"` so Tab flows +X→Y→W→H→URI; the ✕ button now clears the URI textbox (was reverting to the pre-accept snapshot). +Slider style gained `IsMoveToPointEnabled="True"` — click-anywhere-on-bar jumps the thumb to the +click (volume sliders keep their manual `SetSliderValueFromClick`, harmless duplication). + +### Design decisions + +- WebView2 is the only option for Windows — it's pre-installed on Windows 10 20H2+ and Windows 11 +- The web source is a standard element — positioned/sized/opacitied like any image source +- Frame capture rate can be lower than video FPS (5-10 fps for web content is fine) +- This enables Streamlabs/StreamElements overlays via web URLs + diff --git a/TASKS/task-18-local-recording.md b/TASKS/task-18-local-recording.md new file mode 100644 index 0000000..e96ce3e --- /dev/null +++ b/TASKS/task-18-local-recording.md @@ -0,0 +1,58 @@ +# TASK 18 — Local recording + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** record the stream output to a local file, with or without simultaneously streaming. + +### Status: ✅ Shipped `a9eb360` (2026-08-29) — code done (incl. manual-rename modal), build 0 warnings, 244/246 tests; **running-app verification (takes 1–5 done)**: files land (ffmpeg re-pinned month-end), **webcam-in-output + social bar visually CONFIRMED from take 3's extracted frame**, rename modal used for real (take 3 was named via it); take 3 exposed the ~2fps producer starvation and the pump stage-timing (`97ffc42`) named it in one line — `avg render 258.1ms` — **slice 1 fixed 2026-09-03** (deadline pacing per OBS video-io.c + libyuv-style row blits in `SceneCompositor`, test `Pump_Paces_To_The_Deadline_Compensating_Render_Cost`); **take 4: pacing held but render stayed 58.9ms** (2M-iteration row walk + 8.3MB/tick LOH) — **slice 2 shipped 2026-09-04**: `VideoFrame.IsOpaque` producer-contract flag → full-cover backdrop is one `Buffer.BlockCopy`; integer fixed-point bilinear general path; pump scratch pool (release strictly post-submit, owned-by-reference so cache frames are untouchable); dead per-tick `fromScene` render + the `fromSceneProvider` seam removed (BlendFrame uses `TransitionService.FromFrame` — the old render fed nothing). Tests `Pump_Pools_ScratchBuffers_Across_Frames_Without_Stale_Pixels` (the ONE) + `Composite_OpaqueFullCover_Backdrop_CopiesEveryPixel_Into_Scratch`; 59/59 per-class green, clean build 0 warnings. **take 5 ran: render 58.9→25.5ms (`138/300` ≈ 2.2x still) — cause: the per-tick chat raster (`RenderFrame` hit `RenderTargetBitmap` every tick whenever the message buffer was non-empty — the buffer survives sessions); slice 3 shipped 2026-09-04: `ChatOverlayLayer` rasters on message/config change and blits a cached frame every tick (OBS text-source pattern; test `ChatOverlayLayerCacheTests`).** **take 6 ran WITHOUT attribution (35-41ms — build provenance unknown; slice 3 effectiveness UNCONFIRMED) → build-stamp shipped instead of guessing again: `Helpers/BuildStamp` GUID per build (csproj GenerateBuildStamp; incremental builds can no longer lie), wordmark superscript + startup.log line; stats split `render (resolve)` so the next take names the stage. **takes 7–8 (stamped f190587b): chat fix CONFIRMED (`resolve ≈0`) but render stayed 26-27ms — compositor re-rasterizing STATIC layers every tick; slice 5 shipped: `BlitCachedLayer` pastes once-rasterized element-space layers (OBS surface-cache pattern; test `PasteCache_RepeatRender_IsByteIdentical_And_ContentChangePropagates`). **take 9 ran (c65a3cde, paste cache): render 26.5→22.4ms yet period stayed ~37ms — the gap was Task.Delay's 15.6ms sleep quantum padding every sub-tick wait: the true ceiling, hidden until then (why takes 7→9 looked like zero change). Slice 6 shipped: timeBeginPeriod(1) for the pump life + bulk-sleep + 2ms spin tail + `avg wait` stat (accounting closes) + webcam now routes through the paste cache. **take 10 ran (59a02a5b): the new `wait` stat exposed the LAST structural bug — `render 22 + wait 10` against a 16.7ms deadline is impossible for a rebasing pacer: the wait was the producer QUEUED BEHIND THE LIVE PREVIEW — the pump's await-continuations inherit the UI SynchronizationContext (StartAsync fires from a command handler), so the loop had been rendering on the dispatcher all along. Slice 7: Task.Run the loop (OBS keeps media threads off-UI for this exact reason), StaticPixelCache locked + chat raster marshalled to the dispatcher on cache-miss (RTB/DrawingVisual are UI-thread objects), SustainedLowLatency GC, `worst render` stat, webcam routed through the paste cache; test `Pump_Produces_OffTheStartingContext`, 70/70 green. **take 11 ran (c10ce06c): off-UI loop WORKED — typical frames land exactly on the 16.7ms deadline (work ~10 + wait ~6.8; 212/300 best yet); the remaining gap is periodic 35-65ms spikes worsening across a take = gen2 GC pauses, fed by the capture path's fresh ~8.3MB array per DWM frame (~500MB/s). Slice 8: 4-deep capture buffer ring + `VideoFrame.Epoch` in the identity-keyed paste cache + `gen2 +N` printed per stats window; test `PasteCache_RecycledArrayWithNewEpoch_ReRasterizes_NotStaleHits` (fails on the old key), 37/37. Also this session (creator ask): post-mix master gain +40% (before the −1dBFS limiter, so no new clipping). **take 12: `gen2` ~0-1/window, `worst render` → ~16-20ms, n/300 → 300, audio ~40% louder in the file.** + **ROLL-FORWARD (2026-09-04): the two-loop producer (the OBS duplicate-instead-of-starve redesign) and the +40% pre-limiter gain were NOT kept — the two-loop introduced recorded-output TEARING on real hardware and was rolled back with it (full record in HANDOFF). Main sits on the slice-8 serial loop plus this unit: all shared-frame rings 4→8 (screen/camera/web — depth × source period must exceed the worst consumer hold), fresh camera + WebView2 capture pooling, and the wordmark release counter #N replacing the UI GUID (per-build GUID now logs to startup.log only). The tear-proof pacing fix — duplicate-instead-of-starve ON the serial loop — is the next in-flight unit.** + Known remaining churn (follow-ups, not silently done): vertical tier's final `BilinearScale` still allocates per frame; one slow tick (~15-25ms) per arriving chat message — debounced off-tick re-render if take 6 shows burst loss. **User UX spec (2026-09-04) queued behind this**: two-line top bar (LIVE rename, radio pills, Login/Logout button + avatar right-click Change Account, no account light; line 2 centered Start↔Stop grayed-until-armed) + up-front SaveFileDialog for REC (native overwrite prompt; retires the stop-time rename modal) + Go-Live dialog KEPT as preflight confirmation prefilled from the Text drawer — decisions captured in HANDOFF + +1. ✅ `EncoderOptions` extended with `StreamEnabled` / `RecordEnabled` / `RecordPath` (independent intent flags) +2. ✅ `FfmpegArgs.Build` reworked into per-output blocks (stream `-f flv`, record `-f mp4`) via `AddVideoTags` +3. ✅ Two modes via pills: record-only / stream-only (single ffmpeg, one output). ~~stream+record~~ — **REMOVED by creator ruling 2026-09-01: the VOD is already the copy; dual encode drags mid-range hardware and degrades both outputs ("we're not them"). Pending implementation: pills become mutually exclusive radios + `BeginGoLive(alsoRecord)` param dies.** +4. ✅ Top-bar REC + ON-AIR pill toggles, status lights (REC green when recording, ON-AIR green when live), dynamic primary-button text, account status-light tooltip +5. ✅ Record-folder persistence (`LayoutStore` `RecordFolder` key) + picker (`ChooseRecordFolderCommand`) +6. 🟡 Branding flash carries into local recordings — same frame path as streaming (verify in the running app) +7. ✅ Output folder `%APPDATA%\ytLlive\recordings\` default; auto-name `ty---0000.mp4` at start, **rename-on-stop** to `ty-…-.mp4` (numeric suffix on collision) + +Remaining: running-app verification of the rename + dual output. + +### Design decisions + +- **Record OR stream, never both (creator ruling 2026-09-01):** YouTube's VOD already keeps the copy + (downloadable from Studio), and a second concurrent encode degrades BOTH outputs on anything short of + a performant chassis. Recording's real jobs: the offline mux-check bench and the no-account session. + Pills become radios (arming one disarms the other); the one-stop ruling ("Stop ends everything") + already assumes a single session kind. +- FFmpeg supports multiple outputs natively (`-f flv rtmp://... -f mp4 file.mp4`) — kept as generic machinery, but only one block is ever enabled now +- MP4 is the default container; crash-safety arrives as **fragmented MP4** (TASK 32 slice 4 — in record mode the local file IS the archive), retiring the old "MKV as an option" note +- Record-only mode is useful for pre-recorded content or testing without going live +- The branding flash carries into local recordings (free tier billboard extends to recordings) + +--- + +## v1 execution order + +The tasks below are ordered by dependency and risk. Each task builds on the previous. + +1. ✅ **TASK 9.4** — Live chat (right panel) — wire `YouTubeChatService.Start()`, parse messages, render in panel. Foundation for chat box source. +2. ✅ **TASK 3.18** — Chat box source — renders chat ON the stream. Depends on TASK 9.4 (same message parsing). +3. ✅ **TASK 10** — Polar billing — license key entry + watermark toggle + Velopack auto-updates (steps 1-7 shipped; Velopack update URL pending). +4. ✅ **TASK 19/23** — Control Surface UX — director's control room: thumbnails above central monitor, transitions (Cut/Fade/Move), edit mode offline only, left panel two-state (layers/props ↔ chat), right panel eliminated. Verified shipped 2026-08-24. +5. ✅ **TASK 20** — Hotkeys — global keyboard shortcuts. Shipped 2026-08-26: F1-F9 defaults + config UI with modifier chords, persistence, conflict detection, unbinding. +5b. ✅ **Broadcast metadata pull-out + launch geometry** — shipped 2026-08-24 (row 29 above). Live-screen "Text" tab → broadcast form, persistence + remote update; window default/minimums bumped (1920×1040 / 1366×768) with size+position restore. +6. ✅ **TASK 17** — Web source (WebView2) — shipped 2026-08-28; enables alert ecosystem. +7. ✅ **TASK 18** — Local recording — shipped 2026-08-29 (running-app verification pending in handoff). +8. 🔶 **TASK 21** — Media source — video file playback for non-Live scenes. **Remaining: the UI picker slice** (handoff spec in HANDOFF.md). +9. ✅ **TASK 22** — Audio sync offset — shipped 2026-08-31 (status reconciled 2026-09-01). +10. ✅ **TASK 15** — Stock bg images — all 5 scenes seeded (Starting, BRB, Live, Chat, Ending). `EnsureDefaultBackdrop()` runs on every scene switch. +11. **TASK 32** — Stream resilience (blip retry → grace countdown on the ON-AIR sign → one-click Back on air → crash-safe recording → pre-flight). +12. **TASK 33** — Bandwidth auto step-down (depends on TASK 32's restart machinery). +13. **TASK 34** — Scheduled streams (Text-drawer schedule + adoption at Start). +14. **TASK 35** — Scene-linked audio ("scenes remember the room"). +15. **Text source (TASK 3 item 17)** + **the monetization-awareness chain + Alerts (TASK 10 related work → TASK 3 item 20)** — capture → report → journey → alerts, one slice per integration test. +16. **TASK 13** — Social media launch kit (MARCOM) — after features, before gold. +17. **TASK 36** — Gold pass, always last: visibility unlock + flash-live enable + screens/layers audit + native verification suite + expiry reminders + signing/installer/EULA/Velopack. + diff --git a/TASKS/task-19-scene-transitions.md b/TASKS/task-19-scene-transitions.md new file mode 100644 index 0000000..6fa9f29 --- /dev/null +++ b/TASKS/task-19-scene-transitions.md @@ -0,0 +1,15 @@ +# TASK 19 — Scene transitions → MERGED into TASK 19/23 (Control Surface UX) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Superseded by the Control Surface UX task.** Cut/Fade/Move transitions, thumbnails, edit mode, left panel two-state — all shipped as one cohesive feature. See the Control Surface UX section in `ai.md`. + +### Design decisions + +- **Cut is free** — instant switch, no blending, zero CPU cost. This is the default. +- **Fade is the minimum expectation** — 300ms crossfade is universal. Every streaming tool ships this. +- **Move is economical** — simple slide animation, no video decoding needed. Good middle ground. +- **Custom is premium** — media/stinger transitions require video playback. Heavy but expected by mid-tier streamers. +- Transitions happen in the compositor, not post-encode. The preview sees the same blend as the live output. + diff --git a/TASKS/task-20-hotkeys.md b/TASKS/task-20-hotkeys.md new file mode 100644 index 0000000..66491be --- /dev/null +++ b/TASKS/task-20-hotkeys.md @@ -0,0 +1,27 @@ +# TASK 20 — Hotkeys (keyboard shortcuts) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** keyboard shortcuts for scene switching and common actions — the single biggest UX gap. + +### Status: ✅ Done — shipped 2026-08-26 (steps 1-2; all 9 checklist items complete; handoff flags it shipped) + +1. ✅ `GlobalHotkeyManager` service — `Services/GlobalHotkeys.cs`: registers OS-level hotkeys on the window HWND via `RegisterHotKey`/`WM_HOTKEY` (`IHotkeyRegistrar` seam for tests; `RegistrarOverride` mirrors `LayoutPathOverride`). Wired in `MainWindow.OnSourceInitialized`. +2. ✅ Scene switching hotkeys — F1-F5 stage/transition the five canonical scenes. +3. ✅ Common action hotkeys — F6 start/end stream, F7 mute mic, F8 mute desktop audio, F9 TRAX play/pause. Dispatch lives in `MainViewModel.HandleHotkey` and honors command gating. +4. ✅ Hotkey configuration UI — `HotkeyConfigDialog` (XAML Window + `HotkeyConfigViewModel`). Click-to-capture, "Unbind" per row, Reset to Defaults. Conflict detection shows a warning; user must unbind the conflicting action first. Modifier chords supported from day 1 (Ctrl+F10, Alt+Shift+F12, etc.). +5. ✅ Global hotkeys — work even when app is not focused (that is what `RegisterHotKey` gives us) +6. ✅ Conflict detection — warns when a chord is already bound to another action; user must unbind first. +7. ✅ Persistence — `LoadHotkeyBindings()` / `SaveHotkeyBindings()` in `LayoutStore`, stored as `"modifiers+vk"` in Settings table. No schema migration needed. +8. ✅ Test: `GlobalHotkeyTests.WmHotkey_StagesScene_And_TogglesMic_And_RevokesOnClose` — real window + fake registrar, a genuine `WM_HOTKEY` posted through the window's own `HwndSource` hook stages the mapped scene and flips mic mute; close revokes every registration. +9. ✅ Test: `HotkeyConfigTests` — round-trip persistence, display string formatting, storage serialization. + +### Design decisions + +- **Global hotkeys are mandatory** — streamers are in-game and cannot alt-tab. F1-F5 must work from anywhere. +- **Modifier chords from day 1** — Ctrl+F10, Alt+F1, etc. for Stream Deck support and power users. +- **Conflict = warning, not swap** — user must unbind the conflicting action before reassigning. No surprise reassignments. +- **Unbinding** — every action can be set to "None" (no hotkey). Explicitly unbound actions are absent from the DB. +- **No Stream Deck yet** — bare keyboard first. Stream Deck support (physical devices) is v1.1+. + diff --git a/TASKS/task-21-media-source.md b/TASKS/task-21-media-source.md new file mode 100644 index 0000000..b4d07e2 --- /dev/null +++ b/TASKS/task-21-media-source.md @@ -0,0 +1,58 @@ +# TASK 21 — Media source (video file playback) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** play video files (MP4, MOV, AVI) into scenes — starting soon videos, BRB loops, intro/outro clips. + +### Status: 🔶 In progress — Increment A (model + persistence) + Increment B (decoder) + slice 1 (manager + resolver/preview wire-in) + slice 2a/2b (ffprobe probe + native-FPS pacing) + slice 3 (loop mechanism) shipped. **Remaining: the UI picker slice (AddMedia command + file dialog + Acquire/Release wiring + loop-flag wiring) — GUI, handed off to build/verify natively on Windows (see HANDOFF → "UI picker slice — handoff spec").** + +1. ✅ `MediaSourceType` enum: `Video`, `Audio` (audio-only files via media source) +2. ✅ `SourceType.MediaSource` addition to the enum +3. ✅ `MediaSourceModel`: `MediaPath`, `MediaIsLooping`, `MediaVolume` (0-1), `MediaPlaybackState` — persisted in LayoutStore (schema migration + SELECT/INSERT + round-trip test) +4. ✅ `MediaVideoSource` implements `IMediaFrameSource`: FFmpeg raw video decoder → `VideoFrame` pipeline — spawns ffmpeg `-f rawvideo -pix_fmt bgra`, drains the pipe via pure `RawVideoFrameReader` (`Services/RawVideoFrameReader.cs`), raises `FrameAvailable`/`Completed`; lifecycle is `StartAsync`/`StopAsync`; decode process behind `IDecodeProcess`/`FfmpegDecodeProcess` seam (binary-stdout mirror of `IEncoderProcess`). Shipped 2026-08-31; refactored onto `IMediaFrameSource` with this slice. +4b. ✅ Slice 2a — native-FPS probe seam: `FfmpegFrameRateParser` (pure, prefers `avg_frame_rate=` then `r_frame_rate=`, rational N/N/M, unknown→null) + `IFrameRateProbe`/`FfmpegFrameRateProbe` (derives sibling `ffprobe.exe` from the located ffmpeg dir, reuses the `IDecodeProcess` seam; null if ffprobe absent) + `FfmpegLocator.ProbeFileName` now also extracts `ffprobe.exe` from the pinned archive (conditional). Tests: `FfmpegFrameRateParserTests` (6 pure units + 1 probe integration via fake locator/process) + `FfmpegLocatorTests` still green — 15/15. +5. ✅ `IMediaFrameSource` + `MediaVideoSourceManager` (`Services/IMediaFrameSource.cs`, `Services/MediaVideoSourceManager.cs`): app-wide decode-session ownership refcounted by `MediaPath` with a `Func` factory seam; `AcquireAsync`/`ReleaseAsync`/`ReleaseAllAsync`/`GetLatestFrame`; coalesces frames onto the UI dispatcher onto a single shared `WriteableBitmap` per file (mirror of `ScreenCaptureManager`); `MediaFailed` + `PreviewBitmapChanged`. Unit tests + one integration test (bitmap share/coalesce) — `MediaVideoSourceManagerTests`, 5/5 pass. +6. ✅ Native-FPS pacing (slice 2b): `MediaVideoSource` takes optional `IFrameRateProbe?` + `Func? delay` seams, probes FPS once in `RunAsync`, and delays by 1/fps after each emitted frame; unknown/absent probe → no pacing. Test: `MediaVideoSource_PacesFramesByProbedFps` (fake probe + recording delay, one delay per frame ≈ 1ms). +7. ☐ Wire into `FramePump` resolver — `Source { Type: MediaSource }` → latest video frame — **resolver + preview routing + manager wired; session acquisition (start/stop on add/remove) still open (comes with the UI picker)** +8. ☐ Wire into `SceneCompositor` — render media source as an image element at its position/size +9. ✅ Loop control (mechanism) — `IMediaFrameSource.Looping`; `MediaVideoSource` takes a `Func` process factory and restarts the decode on natural EOF when `Looping` (fresh process per pass, since a `Process` can't be re-`Start()`ed). Test: `MediaVideoSource_LoopsUntilLoopDisabled` (single frame re-emits across passes, `Completed` only after loop cleared). Wiring `Source.MediaIsLooping` into the flag lands with the UI-picker (acquisition) slice. +10. ☐ Volume control — per-source volume slider for audio playback +11. ☐ UI: file picker (filtered to video formats), loop toggle, volume slider +12. ☐ Schema migration for media source settings (file path, loop, volume) +13. ☐ Tests: video frame extraction, loop behavior, volume scaling, file validation + +### Design decisions + +- **FFmpeg handles all formats** — no codec-specific code. FFmpeg already in the project. +- **Audio plays through the desktop channel** — media source audio is captured by the WASAPI loopback (like TRAX/game audio). No separate audio routing needed. +- **This pairs with TRAX** — TRAX is background music, media source is background video. Together they make non-Live scenes (Starting/BRB/Ending) feel polished. +- **Not a full NLE** — no trimming, no multi-track, no effects. Just "play this video in the scene." + +### UI picker slice — spec (imported from HANDOFF 2026-09-01 so it can't be lost by a rewrite) + +Nothing `AcquireAsync`s a media path yet, so **no frames flow in a running app** — decoder, manager, +pacing, and loop mechanism are shipped and unit-tested, but a media Source has no way to start a +session. Deliverables, in order: + +1. **File picker + add/remove commands** (mirror `MainViewModel.Trax.cs:68` `OpenFileDialog` usage). + A "Media" source action opens an `OpenFileDialog` filtered to video (mp4/mov/avi); on OK set + `Source.MediaPath`, add to scene, and **`AcquireAsync(MediaPath)`**; on remove + **`ReleaseAsync(MediaPath)`** (refcounted — not `ReleaseAllAsync`, unless the path leaves the + layout entirely). On `StagedScene` layout loads/teardown, release every media path no longer + present and acquire new ones so sessions track the live layout. +2. **Wire `Source.MediaIsLooping` → `IMediaFrameSource.Looping`** — needs a manager-level per-**path** + loop provider. Ambiguity to rule + record: sessions are refcounted per `MediaPath` and shared + across scenes, but `MediaIsLooping` is per-`Source`. Chosen rule: the path's session `Looping` = + **any** live reference has it on (shared behaviour). Record it in the slice commit. +3. **Loop toggle + volume slider** in the source context menu / inspector (items 10/11). +4. `MediaVolume`/`MediaPlaybackState` wiring — audio path for media is future work (desktop loopback + already carries it, TRAX-style; `MediaVolume` slider can drive the local file's audio separately + later — out of this slice). + +**Design notes to preserve:** `ResolveOutputFrame` reads `_mediaManager.GetLatestFrame(MediaPath)`; +`OnMediaPreviewBitmapChanged` adopts the shared `WriteableBitmap` per path; `MediaFailed` → +`OnMediaFailed` (currently `Debug.WriteLine` — surface in UI via toast, TASK 24 stack). The compositor +scales any frame size — media renders through the generic `frameFor(element)` path, no compositor +change. First GUI smoke test: add a short real mp4 on native Windows, confirm it plays and previews. + diff --git a/TASKS/task-22-audio-sync-offset.md b/TASKS/task-22-audio-sync-offset.md new file mode 100644 index 0000000..f5f5e57 --- /dev/null +++ b/TASKS/task-22-audio-sync-offset.md @@ -0,0 +1,23 @@ +# TASK 22 — Audio sync offset + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** per-source audio delay compensation to prevent lip-sync drift from USB mics and capture cards. + +### Status: ✅ Done (shipped TASK 22, 2026-08-31) + +1. ☑ `AudioSyncOffsetMs` on `MainViewModel` (global, default 0, range 0..500 ms) — positive-only: OBS's documented fix delays the audio so it lands on video when it runs ahead; advancing audio would need a video-side delay (out of the audio layer's scope, v1.1+). +2. ☑ Applied in `AudioMixer` — post-mix interleaved-stereo delay via the pure `AudioSyncDelay` line (`Services/Audio/AudioSyncDelay.cs`), fed through a `Func syncOffsetMs` seam each mix tick. +3. ☑ UI: compact "SYNC" slider (0..500) on the mic bar with a status dot (green = no-op, amber = offset set) via `IntToSyncBrushConverter`. **Provenance (recorded 2026-09-01 after a creator "I never ordered this" scare): the creator explicitly asked for OBS's delay-filter lip-sync fix BUILT NATIVELY — the feature is his, not AI drift. Its placement (preview-rail vs mic-settings/gear) remains an open design question; do not remove the capability.** +4. ☑ Persisted in `LayoutStore.Settings` (`Audio.SyncOffsetMs`) via `LoadAudioSyncOffsetMs`/`SaveAudioSyncOffsetMs`; saved from `SaveLayoutNow`. +5. ☑ Tests: `AudioSyncDelayTests` — zero-delay identity, negative→0 clamp, >500 ms clamp to 500 ms, and 10 ms → 960 interleaved-sample shift. +6. ⚠ **Post-ship regression (found 2026-09-01, first real launch):** this task's sync-delay line `_delayedMix` was declared nullable and never initialized — the first `delayed.Length` deref NRE'd EVERY live-mix tick, silently killing all live/record audio, hanging `AudioPipelineTests`, and being mislabeled a "known failure". Fixed in the recording-verification pass (init + null-check + throttled loop errors logged with stack); `AudioPipelineTests` 25/25 green afterward. Lesson recorded in MyMistakes. + +### Design decisions + +- **Global offset first** — one setting for all audio sources. Per-source is v1.1+. +- **Positive-only (delay audio)** — the physically-correct direction (audio runs ahead of the video). True "advance" needs a video-side delay and is PERMANENTLY OUT with per-source sync (TASKS.md → "Out of product" — v1.x phrasing retired 2026-09-01). +- **Simple slider** — 0 to +500 ms, default 0. No numeric input needed. +- **Visual feedback** — "sync OK" status dot shows when an offset is dialled in. + diff --git a/TASKS/task-23-studio-mode.md b/TASKS/task-23-studio-mode.md new file mode 100644 index 0000000..ddc8043 --- /dev/null +++ b/TASKS/task-23-studio-mode.md @@ -0,0 +1,7 @@ +# TASK 23 — Studio mode → MERGED into TASK 19/23 (Control Surface UX) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Superseded by the Control Surface UX task.** Studio mode is now the entire app's paradigm — the director's control surface with 5 thumbnails + central monitor. See the Control Surface UX section in `ai.md`. + diff --git a/TASKS/task-24-toast-notifications.md b/TASKS/task-24-toast-notifications.md new file mode 100644 index 0000000..ef17a2f --- /dev/null +++ b/TASKS/task-24-toast-notifications.md @@ -0,0 +1,39 @@ +# TASK 24 — Toast notifications (in-app, non-blocking) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +### Status: ✅ Done (shipped 2026-08-23, branch `task24-notifications`) + +Replace blocking MessageBoxes with in-window toasts; promote actionable log-only failures. + +### Requirements: +1. Library: **Notification.Wpf 11.0.0** (Platonenkov fork of Federerer/Notifications.Wpf) — MIT, active, targets net8.0-windows. Notice #10 added to `THIRD-PARTY-NOTICES.txt`. +2. Placement: bottom-right above the footer (`NotificationArea x:Name="ToastArea"`, last child of MainWindow's root grid → topmost z-order, `Grid.RowSpan=4`, `Margin="0,0,12,96"`, outside the preview Viewbox), max 4 stacked. +3. Behavior: Info ~4s / Success ~4s / Warning ~8s auto-dismiss; **Error sticky until dismissed**. +4. Styling: dark card tints matching the health-banner language (slate `#3a3b52` info, green `#1d5c38` success, amber `#b8860b` warning, dark-red `#8f1f1f` error); corner radius 8; keep-visible-on-hover. +5. Threading: all Show calls marshal via a Dispatcher captured at construction (encoder/pump events fire off-thread). + +### Migrations (MessageBox → toast): +- Webcam acquire failure ×2 (`AddWebcamToStagedSceneAsync`, `SwapWebcamIdentityAsync`) → Warning +- Image read failure (`PickImageBytes`) → Warning +- Sign-in unsuccessful → Warning; sign-in failed → Error + +### Promotions (log-only → toast, each keeps its AppLog line): +- Go-live prep ×3: reusable stream unavailable / broadcast insert failed / prep exception → Error +- Frame-pump death while live (`OnFramePumpFailed`) → Error +- Mic missing at startup (`StartMicCaptureAsync`) → Warning (once) +- Premium lapse at startup re-validation → Warning +- Saved-session refresh failure (signed out notice) → Info + +Deliberately left alone: offline license re-validation skip (log-only — would spam every launch); SocialsDialog slot-delete YesNo confirm (stays modal). + +### Tests: +- Unit: pure severity→request mapping (`NotificationServiceTests`) — area routing, lifetimes, tints. +- Integration (the ONE): real window-hosted `ToastArea` + real service — Info auto-dismisses, Error sticks (`NotificationAreaIntegrationTests`, RealApp collection). + +### Library facts (cost a hunt — see also ai.md): +- Area routing matches the area's XAML `Name` against the request's `AreaName`; unknown name silently drops. +- `NeverExpires()` = `ExpirationTime = TimeSpan.MaxValue` (not null); `NotificationColor.ToHex()` returns `#AARRGGBB`. +- Overlay-window shutdown caveat avoided entirely by using the in-window area mode. + diff --git a/TASKS/task-25-background-consolidation.md b/TASKS/task-25-background-consolidation.md new file mode 100644 index 0000000..d76be9a --- /dev/null +++ b/TASKS/task-25-background-consolidation.md @@ -0,0 +1,50 @@ +# TASK 25 — Background consolidation: one locked Background per screen + mini-view rule + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +### Status: ✅ Done (shipped 2026-08-23, branch `task25-backgrounds`) + +The creator rejected the multi-concept background model ("why are there multiple background +things?"). Locked model, verbatim intent: + +1. **Exactly ONE background per screen**, named "Background", at position 0. +2. It cannot be re-ordered or deleted; **no Background/Screen item in the (+) menu**. +3. Context-sensitivity exists **only on the Live screen** (Show Desktop toggle + monitor + switching — behavior kept intact). +4. **Exactly five default background images** live in the DB; stale rows are cleaned up. + +### What shipped: +- **Seeder consolidation:** the five near-clone seeders (`Seed{Starting,Brb,Ending,Chat}Background`, + `SeedLiveBackgroundAsset`, `EnsureDefaultBackground`) are gone. One path now: + `EnsureBackground(scene)` → `CreateBackground(name)` — Live = `DisplayCapture` row (capture + machinery untouched), everything else = static `Background` art row; both named "Background". +- **Heal on every load:** `NormalizeBackgrounds(scenes)` keeps the correctly-flavored row, + converts a wrong-flavor survivor in place (`Source.Type`'s setter derives `IsBackground` — + conversions must re-assert the flag), drops duplicates, seeds missing ones, renames, pins to + index 0; non-canonical scenes lose backgrounds + flag. `HealBackgrounds()` stamps default art + (`Assets/{scene}-background.jpg` via `AddAsset`; custom Browse art wins). This fixed the real DB's + rot: four scenes carried a stray "Game Capture" duplicate beside their static row, and Settings + held 16 orphaned `BackgroundUseDefault_{guid}` keys. +- **Settings purge:** `LayoutStore.Save` deletes `BackgroundUseDefault_{id}` / `BackgroundPath_{id}` + keys whose id is no longer a Source row. +- **(+) menu:** Screen + Background items removed → Webcam/Image/Text/Countdown/Web/YouTube Chat; + `AddSource` also hard-refuses DisplayCapture/WindowCapture/Background parameters. +- **Capture controls Live-only:** `CanChangeBackground` = staged scene is Live. The preview + CanvasGrid menu (Show Desktop/Capture Desktop/Refresh Desktop) binds it directly; the layer-row + context menu MultiBindings it with the row's `IsBackground` through a new + `Helpers/AllTrueToVisibilityConverter`. Non-Live screens keep the Use-default pill + Browse. +- **Mini rule:** minis never render live captures. While Live is staged its mini shows the green + placeholder; unstaged it shows `live-background.jpg`. Fixed by making `LoadBackgroundImage` + flavor-blind (`IsBackground`) and gating staged-Live to the placeholder in `RefreshSnapshotsAsync`. + (Real-time rendering stays center-monitor-only — preview lag during live gameplay is a known + unsolved OS-level problem and must not be compounded.) + +### Tests (suite went 221/218 → 223/220; the two stale background-policy landmine tests healed here): +- Integration (the ONE): `BackgroundHealIntegrationTests` — seeds a dirty temp DB (duplicate rows, + misnamed Live row, orphaned keys), drives the real window, asserts one Background per scene at + index 0 with correct flavor/name, save purges the orphaned keys. +- Unit: SceneCatalogTests rewritten for NormalizeBackgrounds/EnsureBackground flavors; + BackgroundTests empty-scene flavor updated; SourceNamingTests excludes the always-present + Background from numbered-name expectations. + diff --git a/TASKS/task-30-gear-cleanup-preview-layout.md b/TASKS/task-30-gear-cleanup-preview-layout.md new file mode 100644 index 0000000..70e0747 --- /dev/null +++ b/TASKS/task-30-gear-cleanup-preview-layout.md @@ -0,0 +1,49 @@ +# TASK 30 — Gear menu cleanup + Default Location for Recordings + preview layout restructure + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +### Status: ✅ Done (shipped 2026-08-29, commits `6031a56` → `61b850f` → `455a409`) + +The creator's KISS review: gear menu cleanup, single Start button, record-location setting, +and a full layout restructure moving all live controls into the preview pane. + +### What shipped: + +**Gear menu cleanup:** +- Removed `Save Layout` / `Save Layout As…` / `Open Layout…` from Gear menu + dead VM + commands/methods (`SaveLayoutCommand`, `SaveLayoutAsCommand`, `OpenLayoutCommand`, + `SaveLayoutAs()`, `OpenLayoutFile()`). Multi-layout = use OBS. + +**Default Location for Recordings (App Settings):** +- First real setting panel in the Gear menu: "Default Location for Recordings". +- `DefaultRecordFolder()` (internal static): configured → `%USERPROFILE%\Downloads` → `MyVideos`. +- `RecordFolderDisplay` property, `ResetRecordFolderCommand`, Browse + "Use Downloads" reset. +- `TextBox` binding fixed to `Mode=OneWay` (get-only computed property). +- ONE fallback test: `DefaultRecordFolder_Falls_Back_To_Downloads`. + +**Single Start button:** +- Button `Content="Start"` hardcoded in XAML. Removed `PrimaryButtonText` / `IsStreamingStart` + properties + all `OnPropertyChanged` references. Pills = intent, lights = state, button = trigger. + +**Preview layout restructure — all live controls under preview:** +- Socials button moved from footer (scenes/sources column) to preview bottom row, left of + TRAX button. `MouseEnter`/`MouseLeave` handlers pulse the `SocialBarElement` + `DropShadowEffect` (sine wave BlurRadius 24→48, Opacity 0.7→1.0 at 30ms tick). +- **Mic controls moved from footer into preview pane** (vertical StackPanel below Socials+TRAX): + MIC button + sound meter + mute grid + volume slider — the same footer controls, relocated. +- Preview bottom row: vertical `StackPanel Grid.Row="2"` → Line 1 (Socials + TRAX), Line 2 + (MIC + meter + mute + slider). All centered. + +**Bottom bar restructured:** +- Reduced to single row (mic row removed), tighter padding (`Padding="16,4"`). +- Gear icon moved to far left. Stream stats (Bitrate/FPS/Dropped/Duration) centered under + preview pane. Resolution dropdown stays far right. + +**Avatar fix:** +- Avatar `BitmapImage` no longer loads in the constructor (stale image from last session). +- Only loads when `IsConnected` fires `true` via `OnViewModelPropertyChanged`. + +### Tests: +- Build 0 warnings, 247 tests (2 pre-existing known failures excluded). + diff --git a/TASKS/task-31-scenegraph-optimization.md b/TASKS/task-31-scenegraph-optimization.md new file mode 100644 index 0000000..ac794ea --- /dev/null +++ b/TASKS/task-31-scenegraph-optimization.md @@ -0,0 +1,170 @@ +# TASK 31 — SceneGraph component + static/dynamic compositor optimization + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** extract the scene collection, element inventory, and mutation surface into a standalone +`SceneGraph` component. Classify elements as static or dynamic. Enable the compositor to bake +static layers once and only recompose dynamic layers per frame. + +### Status: ✅ Done (2026-08-31) + +What landed — core optimization + SceneGraph component, all in one commit: + +- `ElementKind` (Static/Dynamic) on `SceneElement` base, overridden in `Source` and `WebcamSceneConfig` +- `Services/SceneGraph.cs`: owns `Scenes` collection (ViewModel's `Scenes` property delegates to it); + mutation surface `AddElement`/`InsertElement`/`RemoveElement`/`MoveElement` (each invalidates the bake); + queries `GetBackground`/`GetWebcam`/`GetChatBoxes`/`GetSplitPoint`/`IsStatic`; and the + `GetBakedBase` bake-cache (keyed by scene id + static element identities below the split) +- `SceneCompositor.BakeStaticBase`/`CompositeLayers` + `Render(.., staticBase, split)` overload — + split-aware, builds the cached base in source-rect space, composites dynamic/above-split per frame +- `FramePump` takes an optional `SceneGraph` and uses the optimized path when wired (falls back to full render) +- `MainViewModel`: routes element mutations through the graph; invalidates the bake on static + layout/opacity/visibility/useDefaultBackground changes and after background heal / EnsureBackground +- Integration test `SceneGraphTests.BakedStaticBase_WithDynamicLayer_CompositesCorrectly` (audio-free, + verifies bake-once + cache hit + dynamic-on-top pixels + static-mutation invalidation + + dynamic-only non-invalidation) + +**Defensive deviations from this spec (decision 2026-08-31):** +1. `ChatOverlayLayer` keeps taking `IEnumerable` rather than `SceneGraph.GetChatBoxes()` — it is + deliberately decoupled from the graph (`// without owning the scene graph`). Forcing the graph in would + couple a WPF-bound layer to it and violate that documented seam. +2. Background static helpers (`EnsureBackground`/`CreateBackground`/`NormalizeBackgrounds`) stay on the + ViewModel because `BackgroundTests.cs` unit-tests `MainViewModel.EnsureBackground` directly. Queries + moved to SceneGraph; helpers + their invalidation wiring stayed on the VM. +3. The full ViewModel-as-facade extraction (moving the ~all-binding-surface off Scenes/StagedScene/ + LiveScene) was deliberately NOT done in this unsupervised pass — it is the "touch 11 files across 4 + layers" regression risk the handoff flagged. SceneGraph owns the collection + mutation surface now; + the remaining VM call sites binding to those still work because `Scenes` delegates. Revisit after 1.0. + +**Verification:** SceneCompositorTests 4, StretchMathTests 4, BackgroundTests 16, SceneCatalogTests 18, +LayoutStorePersistenceTests 12, FramePumpTests 9, SceneGraphTests 1 all green. RealAppHost GUI/collection +tests (SourceNaming, RoundClip, BackgroundHeal, ...) construct a real `MainWindow` — **CORRECTED +2026-09-01:** they DO run from WSL when invoked per-class through the Windows `dotnet.exe` vstest host +(the old "cannot run headless" claim conflated them with the full-suite WASAPI hang). Two first-launch +crashes this refactor shipped with were caught on the first real native launch and fixed same day: +ctor-order `SceneGraph` NRE (`null!` field assigned after first ctor use — now field-initialized) and +window-scope `EyeButton`/`EyeIconStyle` consumed via `StaticResource` from the extracted `LeftPanel` +(UserControl namescopes can't see window resources — styles moved to `Themes/Controls.xaml`, the +app-scope rule honored). The RoundClip "known failure" was then root-caused to stale test code +(`window.FindName` across namescopes + `VisualTreeHelper.HitTest` where `UIElement.InputHitTest` +models input) — test green 2026-09-01, the sole remaining known failure is the audio one. + +### Design + +**Element classification:** + +Every `SceneElement` exposes `ElementKind Kind` — `Static` or `Dynamic`: + +| Type | Kind | Why | +|------|------|-----| +| `Source` (Background art) | Static | Pixels don't change at runtime | +| `Source` (Image) | Static | Pixels don't change at runtime | +| `Source` (DisplayCapture) | Dynamic | Live game/desktop feed | +| `Source` (ChatBox) | Dynamic | Live chat messages arrive continuously | +| `Source` (WebSource) | Dynamic | WebView content can change | +| `WebcamSceneConfig` | Dynamic | Camera feed, changes every frame | + +**Split point:** + +The **split point** is the index of the first dynamic element in a scene's z-ordered `Elements`. +Everything below it = baked base. Everything from it upward (including static layers above dynamic +elements) = composited per frame. If zero dynamic elements exist: entire scene is baked, +compositor skipped entirely. + +**BakedSceneCache:** + +``` +BakedSceneCache { + VideoFrame BaseFrame // composited static layers below split point + int Version // incremented on mutation + SceneElement[] BakedElements // for invalidation tracking +} +``` + +Invalidation triggers (re-bake): element added/removed/reordered, static element moved/resized/ +opacity changed, `IsVisible` toggled on a static element, background asset changed, scene switch. + +No invalidation when: only a dynamic element's pixels changed (webcam frame, chat message), +or a dynamic element moved/resized (per-frame compositing concern, not a base rebake). + +**Compositor integration:** + +``` +Render(scene, frameFor, options): + 1. Find split point (first Dynamic element index) + 2. If split == scene.Elements.Count: + → return cached base (or bake if stale) + 3. Otherwise: + → start from cached base (or bake if stale) + → composite elements[split..] on top using frameFor +``` + +**SceneGraph interface:** + +``` +SceneGraph { + // Collection + ObservableCollection Scenes + Scene? StagedScene + Scene? LiveScene + + // Mutation surface (single owner of element operations) + AddElement(scene, element) + RemoveElement(scene, element) + MoveElement(scene, element, newIndex) + + // Queries (replace scattered LINQ) + GetBackground(scene): Source? + GetWebcam(scene): WebcamSceneConfig? + GetChatBoxes(): IEnumerable + GetSplitPoint(scene): int + IsStatic(scene): bool + + // Events + SceneChanged + ElementAdded/Removed/Moved + SplitPointChanged +} +``` + +**What moves into SceneGraph:** + +| Current location | Moves to | +|-----------------|----------| +| `MainViewModel.Scenes.cs` — Scenes collection, StagedScene, scene switching | `SceneGraph` | +| `MainViewModel.Background.cs` — `NormalizeBackgrounds`, `EnsureBackground`, background queries | `SceneGraph` (background helpers) | +| `MainViewModel.Sources.cs` — `AddSource`, `RemoveElement`, element inventory | `SceneGraph` (mutation surface) | +| `MainViewModel.Webcam.cs` — webcam queries across scenes | `SceneGraph.GetWebcam(scene)` | +| `ChatOverlayLayer` — `scenes.SelectMany(...).OfType().Where(ChatBox)` | `SceneGraph.GetChatBoxes()` | + +**What stays on the ViewModel:** + +The binding surface — `StagedScene` setter still raises `OnPropertyChanged` for +`ShowEmptySceneHint`, `CanAddWebcam`, etc. But now it delegates to `SceneGraph` for actual +state queries. ViewModel becomes a thin binding facade over the SceneGraph (same pattern as +`MainViewModel.Chat.cs` over `ChatOverlayLayer`). + +The compositor resolver (`frameFor` callback) stays in the ViewModel/encoder layer — it bridges +WPF concepts (CameraManager, ScreenCaptureManager, ImageCache) into the pure `VideoFrame` seam. +SceneGraph doesn't know about capture sessions; it just knows element kinds. + +**Migration path:** + +1. Extract `SceneGraph` as a standalone class in `Services/` +2. Move `Scenes` collection + `StagedScene`/`LiveScene` + wiring +3. Move element mutation surface (add/remove/move) +4. Move background normalization/queries +5. Add `ElementKind` to `SceneElement` base +6. Add `GetSplitPoint` + `BakedSceneCache` +7. Update compositor to use split point +8. Wire ViewModel as thin facade +9. One integration test: baked static base + dynamic layer composite + +### Tests + +ONE integration test: `SceneGraphTests.BakedStaticBase_WithDynamicLayer_CompositesCorrectly` +— a scene with static background + image + dynamic webcam; assert the baked base is cached +(renders once), dynamic layer composited on top, static element mutation invalidates the cache, +dynamic-only pixel change does not. + diff --git a/TASKS/task-32-stream-resilience.md b/TASKS/task-32-stream-resilience.md new file mode 100644 index 0000000..e82c070 --- /dev/null +++ b/TASKS/task-32-stream-resilience.md @@ -0,0 +1,44 @@ +# TASK 32 — Stream resilience (queued 2026-09-01 — merges the orphaned TASK 2 resume clause into the pills/signs architecture) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** a dropped stream heals itself when physically possible, tells the truth when it isn't, and +gets the creator back on air in one click. The ON-AIR sign + primary button in the top bar are the +surface — they already mean "reality" (pills = intent, lights = reality, TASK 18/30); reality just +isn't two-state. Resumption of a completed broadcast is explicitly **out of scope** (impossible via +the API — creator ruling 2026-09-01): no lying in the copy. + +**Slices (one integration test each, in order):** + +1. ☐ **Blip retry** — app-side encoder restart: push dies while live (ProcessFailed / stdin + backpressure death) → bounded-backoff relaunch (2s/5s/10s, N attempts) against the cached + constant reusable-stream ingest URL. Success = push running again while the broadcast is still + live; viewers never know. Fixes the map's stale "ffmpeg does reconnect" claim (corrected + 2026-09-01: reconnect flags are input-side only). Test: fake process fails once then starts → + pump recovers without surfacing to the VM. +2. ☐ **Measured grace + the sign** — while the push is dead, escalate the existing health poll + (30s → ~5s). ON-AIR dot goes **amber "RECONNECTING"** with a mm:ss countdown to the observed + YouTube cutoff; the grace length is a **measured, pinned constant** — probe it on a real private + test stream, cite the observation (spin-guard rule; do NOT invent a number). Poll reports the + broadcast `complete` → dot **red "OFF AIR"**, primary button relabels **"● Back on air"** and + pulses. Test: fake status provider drives the dot through live → reconnecting → cut-off. +3. ☐ **One-click Back on air** — the relabeled button fires the existing go-live path with saved + `Broadcast.*` + cached stream, dialog skipped: new broadcast, new VOD (accepted cost, stated + in-product). Test: seeded saved form + fake stream service → click creates + pushes, no dialog. +4. ☐ **Crash-safe recording** — fragmented MP4 on the record block (`-movflags + frag_keyframe+empty_moov+default_base_moof`): plain MP4's trailing moov atom means a power loss + kills the WHOLE file. Rename-on-stop unchanged. Closes the TASK 18 "MKV as an option" design-note + remnant (fragmented MP4 chosen over MKV: single container, YouTube-upload-native — verify player/ + editor tolerance, cite the research). Test: `FfmpegArgs` emits the movflags only on the record block. +5. ☐ **Pre-flight** — before broadcast insert, one TCP probe to the ingest host:1935 (~2s timeout). + Unreachable → Error toast, no insert, no ghost broadcast ("check VPN/network"). The "forgot the + VPN" story, solved as prevention instead of recovery. Test: fake probe failure aborts go-live before any API call. + +### Design decisions + +- **The sign is the status, the button is the action** — no new chrome, no log lines, no spinners. +- **Honest windows:** countdown = observed grace; at zero we say OFF AIR, we don't pretend to reconnect. +- **Why us > incumbents:** constant reusable stream + persisted form make retry/restart cheap; OBS + shows a reconnecting log line and Streamlabs silently dies. + diff --git a/TASKS/task-33-bandwidth-step-down.md b/TASKS/task-33-bandwidth-step-down.md new file mode 100644 index 0000000..08efa31 --- /dev/null +++ b/TASKS/task-33-bandwidth-step-down.md @@ -0,0 +1,21 @@ +# TASK 33 — Bandwidth auto step-down (queued 2026-09-01; v1 per the complete-v1 ruling) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** when the upload can't hold the tier, drop it automatically instead of drowning viewers in +freeze frames. The `variable` reusable stream (shipped) makes this zero-API; the governor that decides +WHEN is what TASKS.md:25 always claimed existed and was never built. + +1. ☐ Pure `StepDownPolicy` — sliding window over the encoder's reported dropped-frame rate + (`StreamHealth`, already parsed): sustained overrun → downgrade exactly one 16:9 tier + (1080p60 → 1080p30 → 720p60 → 720p30). Hysteresis cooldown; **one-way within a broadcast** + (no oscillation — a recovered connection doesn't climb back mid-show; the creator may manually + step up). Vertical tier stays a manual choice. +2. ☐ Restart machinery = TASK 32 slice 1 (tier change is an encoder relaunch at new W×H/FPS over the + same pipe/URL; the master canvas is 1920×1080 regardless, so geometry never rewrites). +3. ☐ Surface: the resolution badge + dropdown update to the landed tier; ONE Info toast + ("Quality lowered to hold your stream") — no nagging after that. +4. ☐ Tests: pure-policy units (drop-rate windows, one-way, cooldown) + the ONE integration: fake + health stream with sustained drops → exactly one lower-tier encoder restart. + diff --git a/TASKS/task-34-scheduled-streams.md b/TASKS/task-34-scheduled-streams.md new file mode 100644 index 0000000..2511fbe --- /dev/null +++ b/TASKS/task-34-scheduled-streams.md @@ -0,0 +1,34 @@ +# TASK 34 — Scheduled streams, Text-drawer version (queued 2026-09-01; option (b) with the creator's placement) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Scope ruling (2026-09-01):** this is the whole scheduling story — announce a LIVE show (countdown +watch-page + subscriber notifications), then the creator shows up and pushes. "Going live is our +schedule." Scheduled *playout* of pre-recorded files: discussed and DECLINED by the creator. +Premiere/upload scheduling: out of product (see closed list). + +**Goal:** announce "Friday 8pm" from inside the app — YouTube notifies subscribers and runs the +channel countdown; come Friday, one click goes live into the scheduled broadcast. Scheduling is a +"my channel" concern, so it lives in the always-visible **Text drawer**, not the go-live modal. + +1. ☐ Drawer gains ☑ *Scheduled livestream* + date/time (replacing the read-only "Scheduled Start" + row). Checking + Save **creates** the broadcast: insert `liveBroadcasts` with a future + `scheduledStartTime`, bound to the cached reusable stream — the drawer's Save path grows its first + create operation (today it can only update an existing broadcast). +2. ☐ Invariant: **at most one open scheduled broadcast**; unchecking before airtime → + `liveBroadcasts.delete` (legal in created/ready). +3. ☐ **Adoption:** go-live checks for the open scheduled broadcast and pushes into it instead of + inserting a new one. Mid-stream title/desc edits ride the existing Update path (unchanged). +4. ☐ `LiveBroadcastFormViewModel` owns `IsScheduled` + `ScheduledStart` (nullable), persisted with the + other `Broadcast.*` keys; local time in, RFC3339 offset on the wire — the timezone edge is where the + bugs will live: test it (DST boundary, non-UTC creator). +5. ☐ **Research flags (cite before building — spin guard):** (a) pushing *before* the scheduled start + with `enableAutoStart` — live immediately or held? Decides a "starting early — it goes live NOW" + confirmation. (b) delete semantics for a ready broadcast bound to a reusable stream. +6. ☐ Test (the ONE): schedule → persists + inserts (fake service captures a future scheduledStartTime + + boundStreamId); start → adopts instead of inserting; uncheck → delete called. +- **Test-phase note:** dark by design while the visibility lock holds (Private notifies nobody); the + creator can still exercise insert/adopt/cancel. Full shine arrives with TASK 36's unlock — pleasing + coupling, deliberate. + diff --git a/TASKS/task-35-scene-linked-audio.md b/TASKS/task-35-scene-linked-audio.md new file mode 100644 index 0000000..9c9767c --- /dev/null +++ b/TASKS/task-35-scene-linked-audio.md @@ -0,0 +1,19 @@ +# TASK 35 — Scene-linked audio: "scenes remember the room" (queued 2026-09-01; spec approved by the creator same day) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** the BRB scene mutes the mic; the Chat scene brings it back. The creator stops fiddling with +audio at transitions — the scene carries the room state. OBS needs scripts/plugins; native = the +creator-proof pitch. + +1. ☐ `Scene` gains nullable `SceneMutesMic` / `SceneMutesDesktop` — `null` = don't touch (the default; + scenes are passive until told). Schema column bump + roundtrip. +2. ☐ Applied on **live transitions only** (thumbnail click / hotkey), ONCE at entry — never on + offline staging (no surprise stream-state edits while auditioning). +3. ☐ The creator is boss: a manual mute/unmute mid-scene is never overridden until the next entry. +4. ☐ BRB ships `SceneMutesMic = true`; other scenes default `null` — creator's choice. +5. ☐ Edit UI: two checkboxes in the scene's Properties section (left panel, edit mode) — no new chrome. +6. ☐ Test (the ONE): transition applies scene state; `null` leaves state untouched; manual override + survives until the next entry. + diff --git a/TASKS/task-36-gold-pass.md b/TASKS/task-36-gold-pass.md new file mode 100644 index 0000000..a15fc72 --- /dev/null +++ b/TASKS/task-36-gold-pass.md @@ -0,0 +1,67 @@ +# TASK 36 — Gold pass (queued 2026-09-01; the going-gate, always the last work unit) + +> Catalog: [`TASKS.md`](../TASKS.md) — status and requirements live here. + + +**Goal:** the single deliberate pass that turns the dev product into the shipped product. Nothing here +may be done opportunistically mid-development — each item has a lock/comment pointing at this task. + +1. ☐ **Visibility unlock** — remove the `CreateBroadcast` Private override (TASK 9 item 6); dialog + selection rules. The channel-protection stance ends at GA, by hand, here. +2. ☐ **Branding flash goes live** — end the preview-only pre-GA posture: composite the escalating + obnoxious-promo flash onto output + recordings for free users. Paid removes it; that stays the + ONLY paid delta. (Escalation curve finalized here, first.) +3. ☐ **Screens/layers settings audit** — the retired 2026-08-22 landmine (ai.md): fine-tooth-comb + every screen's Background layer properties, pill persistence, context-menu visibility, (+) items. +4. ☐ **Native-Windows verification suite** — the checklist TASKS/HANDOFF never had: real webcam, + real mp4 decode/loop/pacing (TASK 21 picker must have landed), real recording file opens + + plays (fragmented MP4), a real private go-live end-to-end, GUI/RealApp test classes green on + the desktop (they've only ever run against fakes). +5. ☐ **License-expiry reminders** — T-14 / T-7 / T-1 toasts via the existing notification stack + ("renews — manage here" → `PremiumUrl`); lapsed: flash returns (the product itself is the + notice) + one Info toast; active paid users see ZERO license UI. Research: does Polar's + validate response carry `expires_at` reliably — cite before building. +6. ☐ **Release engineering** (the `Distribution.md` plan, 644 lines, finally tasked): code signing + (HARD blocker — an unsigned installer is a SmartScreen scarewall), installer + Velopack update URL + (TASK 10 item 8), EULA draft/review, THIRD-PARTY-NOTICES license-texts gate (TASK 4 req 9). + Build posture (agreed ceiling): compile flags max out at `HARDENED` (release-only anti-debug/ + tamper checks) + `MOCK_REWARDS` (fake reward payloads for Alerts/dev) — **additive-only**, never + branching normal code paths, both compiled every CI build so they can't rot. Obfuscation / + assembly-splitting stay GA-time decisions, not compile-time ones. + +--- + +## v1 = the finished product (creator ruling, 2026-09-01) + +**There is no v1.x.** v1 ships feature-complete: everything in the queue above lands, or is +explicitly buried below. The parking lot is closed — nothing is "deferred to a later version" +anymore; the only two states are **v1 task** and **out of product, permanently**. The 10% margin of +error is this list, bounded and written down — not vibes. + +**Monetization posture restated (what "final" means):** free gives everything; the only difference +is the branding flash. License activation flips exactly one bit (`IsPremium` → flash off). Nagware = +the escalating obnoxious self-promo flash itself — never modals, never feature-locks. Pre-GA the +flash is preview-only (see ai.md → Monetization; TASK 36 flips it live). + +## Out of product — permanently (the 10% margin, closed list — do not re-litigate) + +| Idea | Verdict & reason | +|------|------------------| +| Stream Deck / Loupedeck integration | The device sends keystrokes — **global hotkeys already ARE the integration** (TASK 20). Redundant, not deferred. | +| Per-source audio sync offset | Global offset shipped (TASK 22); per-source is the OBS mixer rabbit hole. Condemned by the Audio Assumption (one knob). | +| Profiles / presets (multi-config) | Same hydra as multi-layout, already ruled at TASK 30: **that's what OBS is for.** | +| Chroma key / background removal | Already ❌ (TASK 3 item 19). NVIDIA Broadcast does it free; model+GPU+lighting caveats = support hell for a solo dev. Stays buried. | +| Virtual camera output (Zoom/Discord) | Different product, device-driver swamp. Our output is one YouTube stream. | +| Replay buffer (instant replay) | Game-clipping culture, not our persona. **The most valuable corpse here** — if the list is ever reopened at all, this is candidate #1, and it reopens by creator ruling, not by AI suggestion. | +| Multi-destination restreaming | "Casual streamers outgrow LlamaCasty first" — the map already said it; the per-destination audio/monitoring expectations are a different product. | +| Stream clipping | YouTube killed clips; trimming local MP4 is an editor. | +| Advanced broadcast-form tab (latency/DVR/embed/projection/CC/region) | TASK 9 item 7 rescoped: fixed sane defaults, invisible. Every exposed field is a support ticket. | +| In-app bug-report mechanism | Never built, never will be: support = email + GitHub issues. (ai.md's support line corrected to match, 2026-09-01. Second-place corpse if the list ever reopens: with replay buffer.) | +| D3DImage/GPU preview compositor | TASK 3 item 16 superseded: XAML preview + software output compositor are the design; a D3D11 swap remains a seam-respecting possibility, not a feature. | +| Simultaneous record + stream | Creator ruling 2026-09-01: the VOD is the copy; dual encode drags mediocre hardware and degrades both outputs. "We're not them." Two modes, radio pills, one stop. | +| Premiere / video-upload pipeline | Uploading finished videos is Studio's job (quota-brutal, processing-state polling, API premiere support unproven). Watch-parties are fandom behavior, not our persona. "Going live is our schedule." | +| Scheduled playout (pre-recorded file airs as live) | Discussed fully 2026-09-01 and declined by the creator — "let's not." If it ever returns, it returns by ruling, with machine-on caveats and the media-audio slice as its price of admission. | + +*If a future session is tempted by anything on this list, the answer is already written. New ideas +must survive this page before they get a task number.* + diff --git a/schema.md b/schema.md index 3a0469b..a9d9486 100644 --- a/schema.md +++ b/schema.md @@ -19,7 +19,7 @@ that every other memory file follows. Read `ai.md` first; this file explains | `AGENTS.md` | Auto-read on-ramp for AI sessions — points to the map and the working rules | **always read first** | | `README.md` | Human-facing intro: what the app is, how to run it, roadmap | no | | `ai.md` | **AI guide + session handoff** — architecture, patterns, decisions, the cognitive map home | **yes — start here** | -| `TASKS.md` | Task queue + authoritative YouTube API research facts + task statuses | yes — for status | +| `TASKS.md` | **Catalog/index** for the task queue. The actual task files live in [`TASKS/`](TASKS/) (one file per task); `TASKS.md` carries the status table, open items, and research facts pointer. Read this for queue state; open task files for full detail. | yes — for status | | `HANDOFF.md` | Current operational state: what's in flight, landmines, next step, secret/DB/port locations | yes — trust it as current state | | `MyMistakes.md` | Per-task failure log **+ RECIPES REGISTRY** (derived-solution rule): one-off solutions / how-tos get recorded here the same session, then grepp'd before ever re-deriving them (see `AGENTS.md` 🔬) | grep first on any "did this before?" wall | | `schema.md` | This file: the conventions below | when in doubt |