gramps
80038ff152
Routing and ruling both good: 'procs in the chat windows but nothing in the alert box'
with zero exceptions — every failure stage was silent by design (Debug.WriteLine-only
preview catch, bare catch{} in RunAsync firing Completed regardless). A decode run that
starts but yields neither a frame nor an audio chunk left _clip != null with
_latestClipFrame == null, so RenderClipFrame returned null forever: the box stayed
transparent for the whole alert and the six-animation fallback (which only ran on
setup-time throws) never fired.
Fix: AlertOverlayLayer.Advance gives a started clip a NoFrameFallbackSeconds (1.0s)
grace — produce no frame AND no audio inside it and the clip is torn down, _elapsed
resets, and the SAME alert continues as the animation branch (never blank, never drained
early). Diagnostics stop the lying silence: BeginClip logs box id/path/size + setup
failures; AlertClipDecoder.RunAsync logs frame/chunk end-state and the exception it used
to swallow.
Good Dog test: SilentDecoder_FallsBackToTheAnimationAfterTheNoFrameGrace (transparent
inside grace, disposed past it, animation renders, drains to idle).
Reference: alert playback must degrade to its fallback on dead-air, never vanish —
same contract as every player/booth failure budget (e.g. OBS source fallbacks).
https://obsproject.com/forum/threads/source-visibility-fallback.187383/
186 lines
13 KiB
Markdown
186 lines
13 KiB
Markdown
# TASK 47 — Alert box video (TASK 43 follow-up): creator-replaceable alert clip + read-time fade + auto ticker
|
||
|
||
> Catalog: [`TASKS.md`](../TASKS.md). Status: ✅ **SHIPPED 2026-09-26** — the TASK 43
|
||
> alert box grows a real **video** celebration: a built-in mp4 loops on every alert unless
|
||
> the creator picks their own clip (referenced from disk, never stored in the DB), the
|
||
> clip fades in/out in ~0.3s (alpha envelope on the video + volume on the audio, mixed
|
||
> into the live stream at unity — no duck), and an auto-composed **message ticker**
|
||
> ("Funder — Super Chat · $10.00") scrolls across the very top of the frame, toggleable.
|
||
> The six TASK 43 animations remain the intrinsic fallback when no asset is playable.
|
||
|
||
## Provenance (recorded per the feature-provenance rule)
|
||
|
||
- **2026-09-24, the creator asked (right after TASK 43 shipped)** — "make the alerts
|
||
animate like a video" / "I want to play my own video for alerts." The TASK 43 unit
|
||
closed with this as the explicit next step.
|
||
- **Lineage:** Research (websearch: how do OBS/StreamElements/Streamlabs alert boxes show
|
||
per-alert videos?) settled the shape, cited in the commit message:
|
||
- Alert videos play **per-alert** (a fresh decoder per event), never a resident media
|
||
player — the OBS "media source reset on activation" model, not a playlist.
|
||
- The clip **must not block**: decode is a background pipe (ffmpeg child, rawvideo
|
||
frames + f32le audio), paced real-time, so alerts never stall the 60fps frame pump.
|
||
- A **freeze-frame at clip EOF fades out** rather than hard-dropping (Streamlabs'
|
||
"end by fade" option) — the tail of a long clip dissolves without a new spawn.
|
||
- The custom file is served by **path, not bytes** (OBS media-source semantics; a
|
||
read-at-play clip lets the creator re-edit the file without a re-import). Decided
|
||
with the creator: "never store the video in the layout." The built-in default is the
|
||
one BLOB, stamped into the internal `Asset` table at startup.
|
||
- **Creator rulings (asked once, held):** custom video + six-animation fallback
|
||
(not either/or); ticker on top of the canvas (very top, full width); **no ducking** —
|
||
"don't lower my game audio for a tip" (the TASK 43 planner's auto-duck idea was
|
||
rejected on the floor); a per-alert **Volume** slider instead.
|
||
|
||
## Product model (as shipped)
|
||
|
||
- **Five new `Source` props** (schema v10, additive guarded ALTERs): `AlertVideoPath`
|
||
(custom file), `AlertUseDefaultVideo` (bool, default true — the built-in clip),
|
||
`AlertShowTicker` (bool, default true), `AlertVideoVolume` (0–1, default 1).
|
||
`AlertVideoAssetId` (internal reference to the stamped default asset row).
|
||
- **Built-in clip:** `Assets/alert-default.mp4` (~4.9MB, branded 2.5s loop, included as a
|
||
csproj Resource). At startup `MainViewModel.StampDefaultAlertVideo()` upserts its bytes
|
||
into the `Asset` table and records the row id in the settings key
|
||
`AlertDefaultVideoAssetId` (`AlertDefaultVideoKey`). The **prune** step exempts that
|
||
key via a UNION so the default never gets cleaned.
|
||
- **Per-alert resolver** (`ResolveAlertClipPath` in `MainViewModel.Chat.cs`): custom path
|
||
wins ONLY when `AlertUseDefaultVideo == false` AND the file exists; otherwise the
|
||
thumbnailed built-in is materialized once to `%TEMP%\ytLive-alert-{id}.mp4`; failure →
|
||
the six `AlertRenderer` animations play (fallback, not an error).
|
||
- **`Services/AlertClipDecoder.cs`** — `IAlertClipDecoder` (Start/Stop/Dispose +
|
||
`FrameAvailable(Action<VideoFrame>)` / `AudioReady(Action<AudioSample>)` /
|
||
`Completed(Action)`) + `AlertClipDecoder` impl: ffmpeg `-loglevel error` child
|
||
emitting `rawvideo bgra` (`-vf scale=W:H`) on one pipe and `f32le -ar 48000 -ac 2` on
|
||
another; `RawVideoFrameReader` (the TASK 21 media-source reader) on the video pipe,
|
||
a **carry-buffer** loop on the audio pipe (float-boundary straddles never drop a
|
||
sample — the 3-byte PCM16↔4-byte-f32 mismatch that broke the first draft), both
|
||
real-time paced. Per-play lifetime: the layer creates one per alert and disposes it at
|
||
drain. Prefer `Start`-synced `TrackFrameRate` — the frame pipe throttles to vfr
|
||
timestamps; audio paces itself. ffmpeg bgra is opaque (alpha=255) — the layer owns all
|
||
alpha.
|
||
- **`Services/AlertOverlayLayer.cs`** — clip branch beside the animation path:
|
||
`Enqueue` → `BeginClip` (factory + path + box dims seam), `Advance` advances `_elapsed`
|
||
(fade-in frame copies scale alpha 0→255 straight-source over the box), events forward
|
||
to the audio sink scaled by `volume × fade`, EOF → **freeze-frame** + fade-out over the
|
||
same `FadeDurationSeconds = 0.30` envelope → drain (StopClip + dispose, idle null
|
||
again). Ticker: `AlertTickerFrame` composes "Author — Kind · amount" from the live
|
||
message and scrolls it.
|
||
- **`Services/Compositor/AlertTickerRenderer.cs`** — marquee strip: 1920×48 transparent
|
||
BGRA, text-pill rasterized once per text on the UI thread (cached, unpremultiplied),
|
||
then pure byte-math per call (scroll `pos = elapsed × 140px/s % cycle`, pill drawn
|
||
straight-alpha over transparent with a repeat-gap copy), so the pump can call it from
|
||
its own thread every tick with no locking.
|
||
- **Ticker threading** — dynamic overlay like the social bar, but **never baked**
|
||
(the social bar IS baked into `BakeStaticBase`; the ticker is a per-frame marquee):
|
||
`SceneCompositor.Render` + `CompositeLayers` take a trailing `tickerFrame` param
|
||
(blitted at 0,0 after the social bar), threaded by `FramePump._alertTicker`
|
||
(`Func<VideoFrame?>` seam, ctor param) through `RenderScene`/`RenderFull`/
|
||
`BuildFullRenderSignature` (mixed into the cache signature so a ticker change
|
||
invalidates — ticker is distinct from static-cache-eligible content).
|
||
- **Audio path** — `AudioMixer.EnqueueAlertAudio` uses a dedicated ring
|
||
(`AlertBufferSeconds = 8` at 48k stereo): resample/upmix to stereo 48k, drained
|
||
pre-limiter in `FillAndMix` and added at **unity** (never ducked); `StartLive` clears
|
||
it. The layer applies `volume × fade` per sample before forwarding.
|
||
- **UI — Stream Alerts section** (LeftPanel, visible only on an `AlertBox`):
|
||
"Built-in video" pill; custom path box + Browse…; "Reset to built-in"; "Message
|
||
ticker" pill; "Volume" slider with % readout. Wiring: `BrowseAlertVideoCommand` +
|
||
`ResetAlertVideoCommand` (CanExecute gates on `Source { Type: AlertBox }`); browse =
|
||
OpenFileDialog (mp4/mov/webm), result writes `AlertVideoPath` + switches the pill off.
|
||
|
||
## Test (Good Dog — ONE integration test per change)
|
||
|
||
`ytLive.Tests/AlertLayerVideoTests.cs` (RealApp STA host, real WPF raster) —
|
||
`AlertVideo_PlaysCustomClip_FadesReadTime_TickerScrolls_ForwardsScaledAudio_DrainsIdle`
|
||
drives a **fake `IAlertClipDecoder`** (no ffmpeg in tests) through the whole lifecycle
|
||
in one pass: the custom path wins per-source (resolution seam asserts box W/H on the
|
||
factory + same source to the resolver); the decoder spawns on play (StartCount 1) and
|
||
dies at drain (DisposeCount 1); fade-in rides the 0.3s envelope (alpha 0 → 127 at 0.15s
|
||
→ 255, straight-alpha copy); pre-fade audio is dropped while ramp audio forwards at
|
||
`volume × fade` (0.8 × 0.5 → [0.2, −0.1]) and full-fade at volume only; the ticker
|
||
renders non-null with pixels and **scrolls** between two pump ticks while idle returns
|
||
null; EOF freeze → fade-out 127 → drain to `IsPlaying false` + null frame + disposed
|
||
decoder.
|
||
|
||
> Catches a REAL bug the animation path never had: the clip branch of `Advance` failed
|
||
> to clear `_current` before `AdvanceToNext`, so after the fade-out the layer stayed
|
||
> `IsPlaying` and re-rendered the finished alert via the animation fallback. The clean
|
||
> frame literally forced the `Assert.False(layer.IsPlaying)` to see it.
|
||
|
||
## Files (declared scope)
|
||
|
||
- `Models/Source.cs` (five alert props + `IsAlertBox`), `Services/LayoutStore.Migrations.cs`
|
||
(guarded ALTERs), `Services/LayoutStore.Save.cs` (INSERT + prune UNION),
|
||
`Services/LayoutStore.Load.cs` (reader indices),
|
||
`Services/LayoutStore.Settings.cs` (`AlertDefaultVideoKey` + asset stamps)
|
||
- `Assets/alert-default.mp4` + `ytLive.csproj` (Resource include)
|
||
- **NEW** `Services/AlertClipDecoder.cs`, **NEW** `Services/Compositor/AlertTickerRenderer.cs`
|
||
- `Services/AlertOverlayLayer.cs` (clip branch + fade + ticker + sink), `Services/Audio/AudioMixer.cs`
|
||
(alert ring + `EnqueueAlertAudio` + unity drain), `Services/Compositor/SceneCompositor.cs` +
|
||
`Services/Encoder/FramePump.cs` (ticker seam)
|
||
- `ViewModels/MainViewModel.cs` + `ViewModels/MainViewModel.Chat.cs` (seams, stamp,
|
||
resolver, materialize, browse/reset), `Controls/LeftPanel.xaml` (Stream Alerts section)
|
||
- **NEW** `ytLive.Tests/AlertLayerVideoTests.cs`
|
||
- Docs: `ai.md`, `TASKS.md`, this file, `Services/index.md`, `HANDOFF.md`, `MyMistakes.md`
|
||
|
||
## Gate
|
||
|
||
Clean build 0 warnings (Windows dotnet host); alert tests green (`AlertLayer*
|
||
FullyQualifiedName` filter = 2/2). Full suite 318/319 — the one failure
|
||
`LayerReorderPersistenceTests.RealMouseDrag…` is the HANDOFF-documented environmental
|
||
class (physical-mouse drag no-ops when the desktop/session isn't interactive; it passed
|
||
in isolation on the third re-run of this unit; the two sibling reorder tests — identical
|
||
persistence path — pass). `scripts/scope-check.sh` green.
|
||
|
||
## Post-ship crash fix (2026-09-26)
|
||
|
||
First live run found a crash the sims never could: test-tab alerts work (button commands run on
|
||
the UI thread — startup.log shows the alert clip's audio in the live mix, `peakMix 0.375 → 0.891`
|
||
with mic and loop at 0.000), but the first **real** message through the chat poller killed the app
|
||
at `11:22:01.301`:
|
||
|
||
System.ArgumentException: Must create DependencySource on same Thread as the DependencyObject.
|
||
at ...MS.Internal.Data.DataBindEngine.ProcessCrossThreadRequests()
|
||
|
||
`AlertOverlayLayer.OnMessageReceived` lacked the UI-thread marshal `ChatOverlayLayer.OnMessageReceived`
|
||
has: a polled message ran `RefreshAlertPreviews` → `UpdatePreview` on the MTA poller thread and
|
||
stamped `alertBox.VideoImageSource` (INPC-raised and WPF-bound) with a `WriteableBitmap` created
|
||
there — the binding engine's cross-thread re-bind crashed. Fix: marshal the whole ingest
|
||
(`Dispatcher.Invoke`, `Application.Current`-null guard) so enqueue, ticker, and preview writes stay
|
||
on the UI thread in lockstep. Good Dog test
|
||
`OnMessageReceived_FromPollerThread_MarshalsPreviewWritesToTheUiThread` (red on the old seam: the
|
||
WriteableBitmap's `Dispatcher` was the poller's) — committed with the fix.
|
||
|
||
## Silent-decoder fallback + decode diagnostics (2026-09-26)
|
||
|
||
Creator repro after the crash fix: On-Air → test → remove and re-add the Stream Alerts
|
||
layer → TEST tab event → procs in the chat windows but NOTHING in the alert box.
|
||
Startup.log 11:44 (the re-add) shows `Test chat simulation 3 injected` but `peakMix 0.105`
|
||
(no alert audio), vs 11:21's 0.375–0.891 (alert audio in the mix) — the re-added box's
|
||
alert never reached the mix. No exception anywhere: every failure stage in the clip path
|
||
was silent by design. Hunted facts: `LiveScene` is never assigned (the encoder always
|
||
renders `StagedScene`, MainViewModel.cs:323) so the re-added box IS in the render graph;
|
||
the bake is invalidated per-element on add; the box's DB row (`c85b46ac-…`, X=602 Y=26
|
||
W=680 H=200) is healthy; ffmpeg tools incl. the pinned binary decode the default clip to
|
||
`scale=680:200` rawvideo correctly to a file. The silent void: a decoder whose `Start()`
|
||
succeeds but whose task yields no frames AND no audio leaves `_clip != null` with
|
||
`_latestClipFrame == null` → `RenderClipFrame` returns null forever → permanently blank
|
||
box, no animation fallback (the fallback only ran on setup-time throws) and no log trail.
|
||
|
||
Fix (commit `…)`: a no-frame grace in `AlertOverlayLayer.Advance`
|
||
(`NoFrameFallbackSeconds = 1.0`) — a started clip that has produced neither a frame nor
|
||
an audio chunk inside the grace is torn down (`StopClip`), `_elapsed` resets, and the SAME
|
||
alert continues as the six-animation render (never a blank box, never drained early).
|
||
Diagnostics stop the lying silence: `BeginClip` logs the box id + resolved path + size and
|
||
the setup failure that used to only `Debug.WriteLine`; `AlertClipDecoder.RunAsync` now logs
|
||
frame/chunk end-state and the exception it used to swallow. Good Dog test
|
||
`SilentDecoder_FallsBackToTheAnimationAfterTheNoFrameGrace` pins the whole contract —
|
||
transparent inside the grace, decoder disposed past it, animation rendering, drains to
|
||
idle. This lands the corpus at 321, with the one known-env flake (RealMouseDrag reorder,
|
||
passes with no fullscreen windows up) — unrelated to this change. Unit was the direct
|
||
lineage of bug surface: **a decoder booth production failure surfaces as dead air.**
|
||
|
||
## Open follow-ups (NOT this unit)
|
||
|
||
- TASK 3 item 20 (RewardEvent SQLite persistence half) and item 16 (Text source) remain
|
||
queued, unchanged.
|
||
- The alert-store folder (`Apps_Commands`-style per-clip assets) stays out of scope —
|
||
the current model is read-at-play from a path, per the creator's "never store the
|
||
video" ruling. |