LlamaCasty/ai.md

ytLlive — AI Guide

Memory map entry point. Conventions live in schema.md; task status and YouTube API research in TASKS.md; brand/marketing palette and launch strategy in MARCOM.md (gitignored — read it before any branding or marketing work); directory maps in each folder's index.md. Reading order: this file → TASKS.md → MARCOM.md → <dir>/index.md → source.

Subsystem detail is split out: large blocks of this file now live under docs/ai/ and this file keeps a short invariant stub + link where each was removed. Open docs/ai/index.md to find the right one, or ignore it entirely until your task touches a moved subsystem.

Response style

No default "Plans & Pitfalls" / planning boilerplate. Respond directly and concisely: do the queued work, then report what changed and what's next. Skip feature pitch, step-by-step implementation plans, pros/cons tables, and "potential pitfalls" sections unless the user explicitly asks for a plan first. A short diff-style summary beats a proposal document every time.

Derivative work (spin guard mandated in AGENTS.md)

Nothing we build is novel — streaming/overlay/WebView2 problems were solved by OBS, CEV, and WPF ecosystems long before us. Two consequences:

1. Proactive: new features start with a quick external scan, not head-first design (citation goes in the commit message).
2. Objectively triggered spin guard: a second failed fix for the same symptom means STOP theorizing and research the established answer externally; there is no unique bug in this repo that the wider ecosystem hasn't hit. This project's hard lesson: the web-source bounding saga (2026-08) burned 9 commits rediscovering that a browser source is a fixed canvas — OBS keeps it at a stable page size, crops/hugs content via the box, and clips at the edge. The apparent "gap" that ended the saga was a widget's own CSS glow effect, not a bug.

Derived-solution rule (2026-08-29, driven by the image-shrink incident): when you work out any one-off, reusable solution (recipe / workaround / how-to), write it into MyMistakes.md → Recipes registry that same session, and grep it before ever re-deriving. A solution recorded once ends the loop; an un-recorded one guarantees the user hears it derived again (see AGENTS.md → Working rules → 🔬).

No-Fluff Mode (on demand)

Invoke with "no-fluff mode" (or similar) when you want ruthless review instead of reassurance. In that mode:

• Strip all polite pleasantries, emojis, transitions, and conversational padding.
• Treat the user's input as a draft to be methodically deconstructed or strengthened — argue, correct, and sharpen rather than agree.
• Give unvarnished truth, not reassurance.

This is an occasional, explicitly-invoked mode — never the default. The default response style above stays in effect unless invoked.

Architecture

C# / WPF (.NET 8) following MVVM:

Path	Role
Models/	Plain data types — Scene, Source (incl. ClipShape, IsMirrored, VideoImageSource), QualityOption, StreamConfig, StreamHealth, YouTubeChannel, ChatMessage, Socials (SocialService enum + SocialEntry/SocialsConfig + SocialServiceIcons) — the social bar
ViewModels/	MainViewModel — public partial class, one file per functional area (Scenes, Background, Webcam, Audio, Trax, Socials, Streaming, Chat, Overlays, Account, License, Recording — split complete, see ViewModels/index.md); Chat.cs is a thin delegating facade over Services/ChatOverlayLayer.cs (Commit G, first true decomposition), and now also over Services/AlertOverlayLayer.cs (TASK 43 — the native alert box, same true-component pattern); GoLiveViewModel, ReuseImageViewModel, CameraPickerViewModel, SocialsDialogViewModel
Services/	YouTube OAuth2, stream/broadcast management, live chat polling, LayoutStore (SQLite), SocialValidator (ISocialValidator seam + HttpSocialValidator default), webcam: VideoFrame seam + CameraDeviceInfo/ICameraEnumerator/ICameraFrameSource interfaces + MediaCaptureCameraEnumerator/MediaCaptureFrameSource (WinRT) + CameraManager, screen capture: IFullScreenDetector/Win32FullScreenDetector + IScreenCaptureSource/ScreenCaptureFrameSource (WinRT GraphicsCapture) + ScreenCaptureManager + ScreenCaptureSourceFactory + Direct3D11Helper/CaptureInterop (COM bridges), media source: IMediaFrameSource + MediaVideoSource (spawns ffmpeg rawvideo BGRA decode) + MediaVideoSourceManager (refcount-by-path session owner) + pure RawVideoFrameReader + IDecodeProcess/FfmpegDecodeProcess process seam (binary-stdout mirror of IEncoderProcess; see "Media source"), compositor: SceneCompositor + CompositorOptions + pure StretchMath + StaticPixelCache (see docs/ai/frame-pipeline.md → "Scene compositor"), alerts: ChatOverlayLayer + AlertOverlayLayer/AlertRenderer (the native celebration-zone overlay, TASK 43 — see "Native events & alerts"), audio: IAudioSource seam + WasapiLoopbackAudioSource/WasapiMicAudioSource (NAudio WASAPI) + AudioMixer + pure AudioLevelMeter/WaveToFloat/VoiceFilterChain/LowShelfFilter/HighShelfFilter/NoiseGate/Compressor/AutoDucker/AudioRingBuffer/TinyResampler/AudioSyncDelay + MusicPlayer + ITraxPlayer (TRAX toggle seam — FakePlayer in tests; the dot is green/red/gray and the error scrolls terse in the desktop-audio bar; MusicVolume fixed 0.5) + IAudioPipeWriter/NamedPipeAudioWriter (see docs/ai/output.md → "Live audio capture"), encoder: IFfmpegEncoder/FfmpegEncoder + IEncoderProcess/FfmpegEncoderProcess + IFfmpegLocator/FfmpegLocator + pure FfmpegArgs/FfmpegProgressParser/FfmpegEncoderPicker + the FramePump frame producer (see docs/ai/output.md → "Live encoder" + docs/ai/frame-pipeline.md → "Live frame pipeline"), notifications: INotificationService seam (AppNotificationSeverity Info/Success/Warning/Error) + NotificationService (Notification.Wpf toasts, see "Toast notifications")
Helpers/	ViewModelBase (INotifyPropertyChanged), RelayCommand, ImageCache, AppLog (file logger), FocusPreservingListBox, OAuthCredentials, TokenStore (DPAPI session persistence), BuildStamp (wordmark release counter #N, incremented by GenerateBuildStamp from the local BuildCounter.txt; per-build GUID/time are logged to startup.log), visibility converters
Themes/	Controls.xaml — the single dark-theme source, merged once in App.xaml (see Themes/index.md)
MainWindow.xaml	Dark theme; top bar + health banner; work area has a fixed 360px left column and preview column, top-left readouts/resolution and scene strip above preview, lower left panel + preview; bottom bar is an empty spacer

Environment, LAN, build & tests — moved

Server IPs and SSH, the Windows dotnet.exe build commands, the test suite map and the hermetic real-MainWindow test rule live in docs/ai/environment.md — read it before building, running, or adding a test.

Product, brand, licensing & distribution — moved

The brand/naming split, AI transparency, the GPL licensing prohibitions (do not violate), the free-forever monetization posture, auth/account assumptions, and packaging/distribution live in docs/ai/product.md.

YouTube Live API — moved

Go-live ordering, the broadcast-binding fix, the private-only lock, health parsing, end-stream pre-flight, Test mode and chat-insert shape live in docs/ai/youtube-api.md.

Overlays: social bar, alerts, chat — moved

The social bar, the native alert box (six animations, alert clip, ticker) and the chat box conventions live in docs/ai/overlays.md.

UI surface — moved

The control surface, scene strip, preview rail, hotkeys, broadcast drawer, toasts, transitions, brand logo, launch geometry and the theme/properties conventions live in docs/ai/ui-surface.md.

Capture — backdrop, webcam, effects & the vertical tier — moved

Screen/game/window backdrop capture, the webcam seam and sessions, the MediaPipe background effects and the vertical tier's per-scene camera placement live in docs/ai/capture.md — read it before touching capture, cameras, the Background row, or anything the 9:16 tier re-places.

Invariants that hold even if you never open that file:

• The Background is a locked per-screen layer: exactly one per canonical scene, named "Background", pinned at index 0, never addable/removable/reorderable — Live is DisplayCapture, every other screen is static art. NormalizeBackgrounds heals any layout to that model on load.
• Never feed a capture row to the ActiveBackgroundImage overlay. That layer is fed by static-art rows only; a flavor-blind IsBackground lookup there permanently covers the live desktop/game feed (the 2026-08-23 regression).
• window:/picker: capture keys are session-scoped and stripped on load (a recycled HWND would capture the wrong window), and an explicit window pin beats auto game/desktop detection while that window is alive.
• The WinRT interop paths are settled: read frame bytes with WindowsRuntimeMarshal.TryGetDataUnsafe (the IMemoryBufferByteAccess ComImport throws Invalid cast on every frame under CsWinRT) and build the D3D device via CreateDirect3D11DeviceFromDXGIDevice (the QI-for-IDirect3DDxgiInterfaceAccess trick returns E_NOINTERFACE on current Windows).
• VideoFrame + the capture interfaces are the seam — nothing above the WinRT layer leaks WinRT types; capture is CPU-first (MemoryPreference.Cpu, SharedReadOnly), the reader-subtype ladder tries each candidate under its OWN catch (one rejected format must never abort acquisition), and there is deliberately no fallback camera — the user's pick is never overridden.
• One webcam identity app-wide, placements per scene. IsWebcamAttainable needs a live lock, not just a saved id; the app keeps its own base ref, so removing every placement leaves the session up.
• CameraManager is the ONE background-effects processing point — segment once per raw frame, render per-scene variants keyed by effect+strength, mask smoothed per camera session (0.75 new / 0.25 previous). The raw path stays untouched so a slow model never stalls the webcam, and the bundled model's provenance is fixed (do not swap in PINTO #109).
• The vertical tier is geometry only: BeginVerticalLayout/EndVerticalLayout park and swap the live placement, so the compositor and PreviewPane.xaml never learn about tiers, and the stored X/Y/W/H/ClipShape columns stay the 16:9 layout. The Chat camera is rect-only through the MODEL (VerticalShapeLocked), not merely through a hidden button.

Rendering budget, measurement & the compositor — moved

Frame cadence, per-element composite timing, the scene dump, the render budget and the SceneCompositor design live in docs/ai/frame-pipeline.md — read it before changing the render loop, the compositor, or anything that moves per-frame cost.

Invariants that hold even if you never open that file:

• Every global overlay must be composited on EVERY render path (pump, preview/thumbnails, scene dump — SceneCompositor.Overlay is shared for all three). A feature wired into one path is silently absent from the output; that bug (flash + social bar "preview-only forever", 2026-09-26) is why the rule exists.
• VideoFrame.Epoch is not optional. A producer that recycles buffers MUST bump Epoch per frame, or FramePump's buffer-identity paste cache and the full-render signature serve stale pixels. BrandFlashPresenter reuses one 8MB master buffer and stamps Epoch per read for exactly this reason.
• StripPool owns four dedicated BelowNormal background threads for strip-parallel compositing; strips below 64 rows run inline. Never demote threads inside Parallel.For — it runs on thread-pool threads and starves the scheduler (that mistake once broke audio-mix timing).
• Read render numbers in Release only. Debug pixel paths are several× slower; a Release 1080p composite measures ~1.0–1.1ms against the 16.67ms budget, and only Release takes are evidence.

Output — ffmpeg, encoder, recording & audio — moved

The bundled ffmpeg.exe decision, the encoder subprocess (FfmpegEncoder/IEncoderProcess), local recording, and the whole WASAPI capture → filter → mix → named-pipe audio chain live in docs/ai/output.md — read it before touching ffmpeg args, the encoder seams, REC/ON-AIR, or the audio mix.

Invariants that hold even if you never open that file:

• ffmpeg is BUNDLED, never downloaded or fetched at runtime. IFfmpegLocator checks the bundle first, PATH second, and throws loudly if neither exists; GuardBundledFfmpeg fails the build when the folder is missing. TheLocatorCarriesNoNetworkCapability prevents the old download design (Store policy 10.2.2, and it 404'd in production on 2026-09-01) from coming back.
• Args stay pure (FfmpegArgs.Build) and never use -re — the pump is the pacer; a second clock on the rawvideo demux fights it. Keyframes -g fps×4 with a closed GOP, -bf 0, yuv420p.
• Encoder choice is probed, never libx264 (GPL): NVENC → QSV → AMF → OpenH264. Never add a static or GPL codec path — see the licensing guardrails in docs/ai/product.md.
• REC and ON-AIR are radio-exclusive by creator ruling. The dual-output machinery stays generic, but the UI can never arm both: one ffmpeg, one output, "we're not them".
• A modal's falsy result must never fall through into its side effect. Cancel on the recording save dialog DISCARDS the temp file explicitly (the bug that kept recordings under the default name).
• Torn-down sessions must leave no lit pill: every pump death / go-live prep failure runs the full StopStream teardown, and AudioMixer.StopLive is idempotent for rollbacks that never started.
• Audio stop order: StopStream closes the audio pipe (StopLive → audio EOF) BEFORE stopping the frame pump (video EOF); the reverse order stalls on pipe backpressure.
• Audio is honest by measurement, not by ear: meters show true dBFS (ToDisplay), the mix runs SpeechLeveler + MasterLimiter (−1 dBFS ceiling), and gains are Func seams read live each tick (AudioGainProvider) — a decorative volume control is a bug. Judge loudness with the bundled ffmpeg (ebur128), never the in-app meter alone.

The live frame pipeline — moved

FramePump and the whole path from frame producer to encoder (pacing, pool/paste caches, slice history, teardown, health stats) live in docs/ai/frame-pipeline.md.

Invariants that hold even if you never open that file:

• rawvideo is stamped by ARRIVAL, never by a timestamp. ffmpeg assigns pts from frame order at the declared fps, so supply rate = output speed. The pump duplicates the previous frame on a missed deadline (correct duration) and never resets nextTick to wall-now — rebasing turns a hitch into a silent time-lapse (take-2 lesson, 2026-09-01).
• Submit never blocks. The encoder queue is bounded and a drain task owns the stdin write + flush; on overflow the newest frame is dropped and counted (DroppedFrames) — overload costs content, never a frozen or smeared session.
• Stop order: StopAsync stops the encoder first — flushing the pending queue to EOF so the file finalizes with no accepted frame lost — before awaiting the pump loop; the reverse order deadlocked on pipe backpressure.
• Judge output in Release, by the burned _outputIndex — the 6-digit counter burned into each frame is what lets a recorded file be audited for drops after the fact; stats alone are not proof.

Current limitations / TODOs

• Helpers/OAuthCredentials.cs contains the real ClientId/ClientSecret. Auth is complete and the session persists via Windows DPAPI (Helpers/TokenStore.cs → %APPDATA%\ytLlive\ytLlive.auth, CurrentUser scope), reloaded best-effort at startup with a proactive refresh of a near-expiry access token. Sign-in/Change Account lives inside the Start Stream dialog (two-state flow — no separate Connect button). Sign-out is explicit only, and that means Logout — NOT Change Account: StopStream() does NOT clear the session — TASK 18's 2026-08-29 reversal ("stopping a recording leaves the creator signed in") — the older "graceful End signs out" line here was stale and is corrected against the code (2026-09-01). YouTubeAuthService takes an optional HttpClient + sessionChanged + openBrowser callback (test seams + save hook; services are still constructed in MainViewModel). Change Account must never sign out first (fixed 2026-09-27): it used to await SignOutYouTubeAsync(); await SignInAsync();, and since SignOutYouTubeAsync → TokenStore.Clear() deletes ytLlive.auth, cancelling in the Google chooser left the creator logged out of a working account. It is now just await SignInAsync() — success overwrites the token via the sessionChanged hook, cancel/failure leaves the session untouched. The OAuth flow also releases its loopback callback port on retry: abandoning the browser leaves the attempt holding localhost:8765 for its 5-minute timeout, so a new AuthenticateAsync cancels and awaits the abandoned attempt before binding, serialised by a SemaphoreSlim (_authGate) — without that gate, two overlapping calls both see a free port and race to bind it (the exact failure the takeover prevents)
• Google Cloud project ownership (recorded 2026-09-27, do not re-hunt): OAuth ClientId is 198374416003-s8tm3rb5csh91quvlg6mnec6srmrr6md.apps.googleusercontent.com, so the Cloud project number is 198374416003. That project is owned by the llamachile.shop@gmail.com ("gramps") Google account. ⛔ This is NOT the same account as the YouTube channel a creator signs into the app with — console access and channel identity are separate logins, and assuming they are the same was the trap that stalled the search. Direct link that resolves it in one hop: https://console.cloud.google.com/apis/credentials?project=198374416003. If the picker shows no match, it defaults to the "Recent" tab — switch to "All" before concluding the project is missing.
• Scene/source/asset layout + the social bar persist (SQLite, schema v8); the OAuth session persists (DPAPI); the paid-unlock state persists (LayoutStore Settings table — LicenseKey/LicenseValidatedAt/IsPremium, 14-day offline grace)
• YouTubeStreamService manages the variable reusable stream (shipped 2026-08-16): GetOrCreateReusableStreamAsync lists liveStreams?mine=true and reuses the existing cdn.isReusable stream, creating it only on first use (resolution=variable, frameRate=variable); the stream is cached via LayoutStore (SaveReusableStream/LoadReusableStream, Settings table) and bound at broadcast insert (boundStreamId). Health (shipped 2026-08-16): GetStreamHealthAsync(streamId) polls liveStreams?part=status for healthStatus + configurationIssues[] → StreamHealth; the real API nests them under status.healthStatus = {status, lastUpdateTimeSeconds, configurationIssues[]} — the parse accepts that object shape and the legacy flat-string shape (fix 2026-09-22: the object shape crashed every poll with requires an element of type 'String'); banner decision in pure StreamHealthReporter
• Webcam capture is shipped (milestone 1); the live desktop/game backdrop is shipped (ship task #1); the output compositor (TASK 4 ship step 1) is SHIPPED, the FFmpeg locator (TASK 4 ship step 2) is SHIPPED, the encoder + RTMP push (TASK 4 ship step 3) is SHIPPED, WASAPI audio capture (TASK 4 ship step 4) is SHIPPED, frame-pipeline wiring (TASK 4 ship step 5) is SHIPPED, health stats (TASK 4 ship step 6) is SHIPPED, one-click go-live + private-only enforcement (TASK 4 ship step 7) is SHIPPED — TASK 4 (RTMP Ingest) is fully done; full plan in TASKS.md
• StreamConfig defaults (TargetBitrate=6000, Resolution="1920x1080") are stale — the live dropdown drives StreamHealth.CurrentBitrate/FPS instead
• v1 task queue (2026-08-19): tasks 19-23 added for v1 feature completeness; TASK 3.18 (chat box) shipped; TASK 16 (infinity display) shipped; TASK 10 (Polar billing) shipped — steps 1-7 (Velopack bootstrap, PolarLicenseService, license entry UI, LayoutStore entitlement, BrandFlash toggle, startup re-validation, PremiumUrl wired); Velopack update URL pending:
  • TASK 10: Polar billing — license key entry, offline entitlement, brand flash gating, Velopack auto-updates
  • TASK 19/23 merged: Control Surface UX — director's control room with an explicit EDIT toggle, scene strip above the preview, readouts in the top-left cell, and preview-rail drawers; current behavior is in docs/ai/ui-surface.md
  • TASK 20: Hotkeys (global keyboard shortcuts) — the single biggest UX gap
  • Broadcast metadata pull-out + launch geometry — shipped 2026-08-24 ("Text" tab on the Live screen; see docs/ai/ui-surface.md)
  • TASK 21: Media source (video file playback) — starting soon videos, BRB loops
  • TASK 22: Audio sync offset — lip-sync correction for USB mics/capture cards
  • These were identified through competitive analysis and are table-stakes for any streaming software in 2025-2026. Full details in TASKS.md.