The cadence work ended with a question the log could not answer: the compositor
manages 6.8 composites/s at ~141ms while the 10:03 take did 29-34ms on the "same
scene shape" (full-render split=0 elements=6 dynamic=5). Nothing in the stats line
said WHICH of the six elements cost what, and nothing recorded the scene itself, so
the two takes could only be compared by eye. This closes that gap and nothing else:
no fix is attempted until a real take names the culprit.
- Services/Compositor/CompositeTimings.cs (new): per-window collector.
Add(stage, ticks) accumulates; Describe(renders, minMs = 0.1) prints the averaged
breakdown slowest-first, dropping sub-0.1ms noise.
- SceneCompositor.Render / CompositeLayers / BakeStaticBase / BlitBakedElement /
BlitCachedLayer + SceneGraph.GetBakedBase: trailing optional CompositeTimings?
Every stage is timed (backdrop, base, flash, social, ticker, zoom, scale, overlay,
bake backdrop, bake overlays) and EVERY element gets its own entry keyed by
z-index + name ("#2 Webcam"), with the nested "raster" sub-stage recording the
paste-cache rasterize miss - the split between "re-samples every tick" and "slow
paste" without a second session. Bake-pass elements are prefixed "bake #..." so a
one-off rebake cannot masquerade as a per-composite cost. Callers that pass
nothing (thumbnails, existing tests) are unaffected and pay nothing.
- FramePump: owns the collector, threads it through RenderScene/RenderFull/
GetBakedBase/CompositeLayers, and logs "FramePump parts (N renders/5s): ..." once
per 5s window beside the stats line (averaged per render, so it compares directly
with "avg render"), then resets it. Also dumps the scene - "FramePump scene dump:"
header plus one line per element (z-index, name, kind, source type, visibility,
rect, opacity, clip) - at pump start and on every scene change.
- One Good Dog integration test (ytLive.Tests/CompositeTimingTests.cs): a real
composite of a full-canvas element plus a 64x64 one through the real collector
must rank the full-canvas element first, carry the raster sub-stage and the fixed
stages, and the dump must name both elements with z-index/type/rect and echo the
split point.
Derivative work (AGENTS.md): OBS's answer to "which source is slow" is the Source
Profiler - average execution time of each source's tick and render functions
(https://docs.obsproject.com/reference-libobs-util-source-profiler, util/profiler.h).
Same shape here: one timer per element per composite, reported as an average, plus
the stage split this case needs.
Docs in the same commit: ai.md (new subsection "Per-element composite timing + scene
dump", the closed gap corrected in "Frame cadence"), Services/index.md (new file row
+ FramePump/SceneCompositor rows), TASKS.md (unit marked shipped, next = creator
take), HANDOFF.md (rewritten: unpushed commit, next step, verify.sh whitelist hole).
Also recorded this session (from the creator): the launch command is
`dotnet run -c Release` - Release is the DEFAULT, Debug only when explicitly named,
and perf takes must read Release. ai.md -> Run updated.
Full suite 428 tests, 0 warnings; the only failures are the known audio env flakes
(both pass in isolation). Scope check passed.
- Model: Google's official MediaPipe Selfie Segmenter (landscape, Apache-2.0 per model card),
converted to ONNX by tools/selfie-seg/convert_tflite_to_onnx.py (custom op
Convolution2DTransposeBias -> ConvTranspose) and verified against MediaPipe's own runtime
(verify_against_mediapipe.py: max |diff| 5e-5, 100% binary agreement). Embedded resource, 430 KB.
- Services/BackgroundEffects: IPersonSegmenter seam, OnnxPersonSegmenter (onnxruntime 1.31.0 CPU,
never throws), BackgroundEffectProcessor (None passthrough, Remove = alpha mask). Not wired yet.
- Tests: Remove integration test (fake segmenter) + bundled-model contract test vs MediaPipe
reference mask. Full suite 407/408 — the one failure is the documented interactive-desktop flake
RealMouseDrag_OnTheLayerList_PersistsTheReorder (passed in the run before; code path untouched).
- Docs: ai.md reversal + section + licensing guard, THIRD-PARTY-NOTICES 11-12, TASK 51, indexes,
MyMistakes recipe, HANDOFF.
References: obs-backgroundremoval (approach only — ONNX Runtime on CPU, 2 threads; GPL-3.0, no code
used) https://github.com/royshil/obs-backgroundremoval ; MediaPipe image segmenter
https://ai.google.dev/edge/mediapipe/solutions/vision/image_segmenter
TASK 43's alert box grows a real video celebration. Per-alert IAlertClipDecoder
(ffmpeg bgra + f32le pipes, real-time paced, disposed at drain) plays the shipped
Assets/alert-default.mp4 (stamped into the Asset table at startup) unless the
creator picks their own file — path reference only, never stored in the DB; the
six AlertRenderer animations stay the fallback. ~0.3s fade rides the alpha
envelope on straight-source copies (EOF freeze-frames then fades out); audio
forwards to a new AudioMixer alert ring (8s, 48k stereo) drained at unity — no
duck, creator ruling — scaled by volume × fade. An auto-composed marquee ticker
('Funder — Super Chat · $10.00', 140px/s) scrolls top-of-frame via a
FramePump._alertTicker seam through Render/CompositeLayers, mixed into the cache
signature (dynamic overlay, never baked). New Stream Alerts section in LeftPanel.
Derivative-work references (how OBS/Streamlabs alert boxes do per-alert video):
- https://support.streamlabs.com/hc/en-us/articles/217741147-Setting-Up-Your-Streamlabs-Alerts (custom image/video per alert type + variations)
- https://obsproject.com/kb/stream-tutorial-2-alerts (alert overlay as an on-screen zone)
- https://streamlabs.com/content-hub/widgets/alert-box (per-event alert playback)
Good Dog: AlertLayerVideoTests drives a fake IAlertClipDecoder through the whole
lifecycle in one pass (custom path wins, decoder spawns/disposes, fade envelope
0→127→255, audio volume×fade, ticker scrolls, EOF fade-drain to idle). It caught
the clip branch of Advance not clearing _current before AdvanceToNext — the layer
stayed IsPlaying after drain (MyMistakes post-mortem).
Full vstest 319/319; clean build 0 warnings; scope check green.
Replace the external StreamElements feed with NATIVE YouTube events per the creator's
2026-09-23 question.
Chat parity half:
- YouTubeChatService decodes ALL SIX liveChat/messages event types into
ChatMessage.Kind (superChat/superSticker/newSponsor/membershipGifting/
giftMembershipReceived/memberMilestoneChat) — the four formerly-empty overlay
rows are real now — and re-arms its one-shot poll on the server's
pollingIntervalMillis (streamList connection semantics; clamp 1000-6000ms,
maxResults=2000). ParsePage internal static seam + ChatPage record for
deterministic tests; optional HttpClient ctor seam kept; InjectSimulatedMessage
(TASK 41) preserved.
Alerts half (creator rulings: one celebration zone, six UNIQUE animations, no
menus/polls, sub mention = chat row only, NO viewer count):
- New SourceType.AlertBox 'Stream Alerts' (one per layout, CanAddAlerts gate
mirroring chat; idle = transparent).
- AlertRenderer: six distinct branded animations (SuperChat slide-up/shine/
count-up, SuperSticker scale-pop, NewMember drop-in/flash, MemberGift slide-left/
chip-fan, GiftReceived confetti, MemberMilestone rise/growth-bar), every card
drawing the 'made with LlamaCasty!' brand line.
- AlertOverlayLayer: true component (Commit-G pattern) — queue (cap 10, drop tail
never stall), 33ms UI ticker, cache-first RenderFrame + UpdatePreview,
Advance(double) as the deterministic test clock; ChatEventKind.None rows never
enqueue.
- Wired: resolver RenderAlertBox, _alertLayer ctor + dispose, LoadLayout previews,
Add menu item (Controls/LeftPanel.xaml), TestSessionViewModel sims tagged
(member->NewMember, superchat->SuperChat).
Good Dog ONE integration test: AlertLayerTests (RealApp STA, real WPF raster) —
ParsePage classifies all six kinds + cadence fields; None rows enqueue nothing;
six events play pairwise-distinct moving frames then drain to null.
Gate: clean build 0 warnings; full suite 316/317, the one failure
(LayerReorderPersistenceTests.RealMouseDrag) repros on the clean tree — the
known environmental class (real-mouse-drag no-ops with a game/fullscreen window
focused).
References (OBS/overlay ecosystem):
- streamList semantics: https://developers.google.com/youtube/v3/live/docs/liveChatMessages/streamList
- OBS alert-box pattern (designated celebration zone, idle transparent): creator-chosen model
startup.log 2026-09-22 (user runs of the new build): ALL THREE session stops
logged 'Broadcast transition(complete) failed (403) invalidTransition → End
close-out: enableAutoStop will finish'. The old ai.md assumption ('invalidTransition
when autoStop already fired') was wrong — the blind complete POST races
YouTube/autoStop marking the broadcast complete (enableAutoStop=true is always
set, tests included), so a blind POST earned only the 403 + log noise. Per the
transition errors table, invalidTransition is a current-status problem and
complete is not gated on streamStatus (only testing/live are):
https://developers.google.com/youtube/v3/live/docs/liveBroadcasts/transition
Fix: EndBroadcastAsync pre-flights liveBroadcasts.list -> status.lifeCycleStatus
and skips the POST when the broadcast is already complete/revoked; an inconclusive
pre-check still posts (old behavior) rather than silently stranding a live
broadcast; still never throws (enableAutoStop finishes every skip).
Good Dog: EndBroadcast test rewritten to the one contract — live -> GET pre-check
+ POST transition (URL shape asserted); already complete -> zero transition POSTs;
inconclusive pre-check + 403 POST -> error string, never a throw. Full suite
315/315 (one-run audio/mouse timing flakes cleared), build 0 warnings.
Recipe recorded in MyMistakes.md; ai.md / Services index / TASK-9 note corrected.
startup.log 2026-09-22 (via the user's run of the new build): every session
start logged 'Stream health poll failed: The requested operation requires an
element of type String, but the target element has type Object'.
Root cause: GetStreamHealthAsync did status.healthStatus.GetString(), but the
real API nests it as status.healthStatus = {status, lastUpdateTimeSeconds,
configurationIssues[]} — an OBJECT, per the liveStreams reference
https://developers.google.com/youtube/v3/live/docs/liveStreams (checked the
docs before fixing, per derivative-work rule). Our flat-string fixture was the
wrong assumption since TASK 9; the report-by-exception health banner had been
silently dead.
Fix: read the nested healthStatus.status + nested configurationIssues[], tolerating
the legacy flat-string shape so nothing else breaks. Tests flipped to the real
object shape (the Good Dog for this change). Full suite 315/315, build 0 warnings.
Also recorded in MyMistakes.md (API-shape recipe) and ai.md/Services index. End-of-session
transition(complete) 403 invalidTransition (enableAutoStop fallback) observed on all three
2026-09-22 stops — separate racy-design issue, carried, not in this change.
The ~30Hz CapturePreviewAsync PNG poll capped real cadence at ~20Hz
(35–165ms full-HD encode+decode), so a 60fps widget still juddered at ~1/6
speed. Replaced polling with frame-driven capture of the composition
controller's root visual — the mechanism WebView2CompositionControl and
Flutter's webview_windows use (graphics_context.cc captures the root
surface_ visual via CreateGraphicsCaptureItemFromVisual; reference:
github.com/microsoft/Windows.UI.Composition.WinUI / flutter-internal
webview_windows). Frames now arrive at the renderer's own pace; capture
memory is epoch'd ring reuse + one crop-sized shared WriteableBitmap.
New Services/WebCaptureFrameSource.cs owns GraphicsCaptureItem + free-
threaded Direct3D11CaptureFramePool + session (Straight alpha readback,
per-frame FindContentBounds → CropBounds). WebView2Manager reworked around
per-session composition controllers + one UI-thread Compositor created via
the CoreMessaging CreateDispatcherQueueController P/Invoke (the 19041
projection lacks CreateOnCurrentThread); internal seam ctor
(Dispatcher, Func<string,IScreenCaptureSource>?) for hermetic tests.
CaptureScheduler.cs deleted; the three SetCaptureInterval cadence hooks
removed; InitWebView2() moved from MainWindow ctor to Loaded (a parent
HWND must exist for the composition controller); the hidden WebViewHostPanel
overlay deleted. TransparentBackgroundScript unchanged.
Tests: WebView2ManagerTests reworked — 4 control-size + scheduler tests
dropped, FindContentBounds tests moved to WebCaptureFrameSource, ONE
integration test (Frames_PublishCroppedPreview_And_CoalesceToLatest_CarryingCropBounds)
drives the seam with a FakeWebSource + background-STA DispatcherPump.
Suite 293/293, 0 warnings.
NOTE: composition path NOT yet verified on a device — the take is the
next step. Web work committed locally only (no push per standing rule).
Docs same-commit: ai.md Slice 14 + supersede marker on Slice 11, HANDOFF,
MyMistakes (CoreMessaging DQ + namespace-landmine recipe), TASK 17,
Controls/ViewModels/Services indexes.
IFfmpegLocator seam that resolves ffmpeg.exe (PATH -> cache -> pinned download),
plus the licensing compliance that makes a paid GA product defensible.
NOTE FOR USERS: this change shows NO difference in the app's behavior except a
new top-bar "About" button (opens THIRD-PARTY-NOTICES.txt). It is scaffolding
for the encoder/streaming work.
- Services/Encoder/: IFfmpegLocator + FfmpegLocator. Pin is BtbN lgpl-shared
autobuild-2026-08-09-13-03 (NOT gyan.dev/GPL or static: LGPLv2.1 §6 static
relink material avoided by dynamic linking); extracts ffmpeg.exe + libav*.dll
via a staging dir so a crash never leaves a partial cache
- THIRD-PARTY-NOTICES.txt: LGPL/BSD/MIT notices + source offer, copied to the
build output, surfaced by the About button; v1 gate = bundle full license
texts (TASK 4 requirement 9)
- ai.md "Licensing - do not violate" guardrails (never GPL/nonfree/static/latest-
tag, never link FFmpeg in, never drop notices); TASKS.md + Services/index.md
updated
- Tests: FfmpegLocatorTests - hermetic decision-ladder integration test + shared-
build DLL extraction + edge cases; docs updated (78 tests passing, 0 warnings)
Software compositor that renders a scene into the encoder's master VideoFrame,
mirroring the XAML preview minus editing chrome: backdrop -> background ->
elements (UniformToFill cover-crop, round clip, mirror, opacity, border) ->
branding flash.
NOTE FOR USERS: this change shows NO difference in the app's UI — it is pure
backend scaffolding laying the groundwork for live video capture/streaming.
The preview you see is unchanged.
- Services/Compositor/: SceneCompositor (Render(scene, frameFor resolver,
flashFrame, CompositorOptions)), CompositorOptions (source rect + output
size; 16:9 full master, vertical 607x1080 -> 1080x1920), StretchMath (pure
UniformToFill + bilinear), StaticPixelCache (asset bytes -> BGRA8 frame)
- Frame sources injected via Func<SceneElement, VideoFrame?> resolver, so the
compositor is pure, WPF-free, and hermetic to test (D3D11 upgrade behind the
same seam later)
- SceneElement.TryGetBorderColor public (shared hex parse), stale
MainViewModel comment fixed, pre-existing CS1998 in YouTubeAuthServiceTests
cleaned up
- Tests: SceneCompositorTests integration (full scene + vertical tier + flash)
+ StretchMath units, docs updated (72 tests passing, 0 warnings)
- Meter is a READ-ONLY realtime level display: fill = Math.Min(1, AudioLevel * MicVolume) (AudioLevel = live input from the future mixer, 0 with no input; MicVolume is gain on ambient noise). While the slider is dragged the bar previews the slider position (PreviewMouseLeftButtonDown/Up + LostMouseCapture -> SetVolumeAdjusting); on release it returns to the live level (bounces to 0 with no input). Clicking the meter does nothing.
- MicVolume drives MicMuted (read-only, muted <=> volume 0): sliding to 0 flips the speaker to the red slashed mute icon, sliding up from 0 clears it; speaker button runs volume to 0 or restores the prior level (_volumeBeforeMute, default 0.8) and flashes the meter to the restored position for ~300ms (BeginVolumeFlash/EndVolumeFlash DispatcherTimer, cancelled if the slider is grabbed).
- MIC label opens MicPickerDialog (WinRT audio-capture device enumeration, mirrors camera picker, no new packages); the chosen voice source name shows left-justified INSIDE the meter bar (fill at 75% opacity so text + ruler markings show through).
- New files: Services/IMicrophoneEnumerator, MicrophoneDeviceInfo, WinRtMicrophoneEnumerator; ViewModels/MicPickerViewModel; MicPickerDialog.
- Docs updated (ai.md, TASKS.md, Services/index.md, ViewModels/index.md); build 0 warnings, 65 tests passing
gramps