LlamaCasty/TASKS/task-32-stream-resilience.md

TASK 32 — Stream resilience (queued 2026-09-01 — merges the orphaned TASK 2 resume clause into the pills/signs architecture)

Catalog: TASKS.md — status and requirements live here.

Goal: a dropped stream heals itself when physically possible, tells the truth when it isn't, and gets the creator back on air in one click. The ON-AIR sign + primary button in the top bar are the surface — they already mean "reality" (pills = intent, lights = reality, TASK 18/30); reality just isn't two-state. Resumption of a completed broadcast is explicitly out of scope (impossible via the API — creator ruling 2026-09-01): no lying in the copy.

Slices (one integration test each, in order):

1. ✅ Blip retry (shipped 2026-11-05) — app-side encoder restart: push dies while live (ProcessFailed / stdin backpressure death) → bounded-backoff relaunch (2s/5s/10s, N attempts) against the cached constant reusable-stream ingest URL. Success = push running again while the broadcast is still live; viewers never know. Fixes the map's stale "ffmpeg does reconnect" claim (corrected 2026-09-01: reconnect flags are input-side only). Test: fake process fails once then starts → pump recovers without surfacing to the VM. Shipped: Services/StreamRetryPolicy.cs (immediate, +1 s, +2 s, +3 s — minimum waits, because YouTube's grace is short), Services/StreamRecovery.cs (the loop, with the audio pipe recreated per attempt because its single-instance server is unusable after the old client died), and FramePump.EncoderFailed separating a dead push (recoverable) from a fatal pump fault. Each episode reports into the left panel's alert stack; a reconnect that lands on a complete broadcast is reported as such instead of showing a live sign over a dead stream. Tests: StreamRecoveryTests.

2. 🔶 Poll-driven sign — NO invented countdown (spec revised 2026-11-05, creator challenge: "I really don't think that's going to be a constant — it's going to depend on so many variables, some that are dark to us"). The original plan was a measured grace constant and a countdown to YouTube's cutoff. Dropped: YouTube's cutoff is server-side and depends on things we cannot observe (ingest node health, prior stream quality, region, load), so any pinned number is a guess wearing a timer. What ships instead:

   • Amber RECONNECTING in the top bar, with measured facts only: elapsed time and attempt count ("RECONNECTING · 0:12 · attempt 2 of 4"). No countdown.
   • The status poll speeds up while recovering (30 s → ~4 s) and drives the truth. Signals used, cheapest first: liveBroadcasts.list status.lifeCycleStatus (authoritative complete), liveStreams.list status.streamStatus/healthStatus (is the push being received), and the chatEndedEvent YouTube already puts in the chat feed we poll — free, and previously skipped.
   • At complete: red OFF AIR, stop retrying, and (slice 3) offer Back on air. Test: a fake status provider drives the sign through live → reconnecting → off air; a chatEndedEvent triggers an immediate status check rather than a decision of its own.

3. ☐ One-click Back on air — the relabeled button fires the existing go-live path with saved Broadcast.* + cached stream, dialog skipped: new broadcast, new VOD (accepted cost, stated in-product). Test: seeded saved form + fake stream service → click creates + pushes, no dialog.

4. ✅ Crash-safe recording (shipped 2026-11-05) — fragmented MP4 on the record block (-movflags frag_keyframe+empty_moov+default_base_moof): plain MP4's trailing moov atom means a power loss kills the WHOLE file. Rename-on-stop unchanged. Closes the TASK 18 "MKV as an option" design-note remnant (fragmented MP4 chosen over MKV: single container, YouTube-upload-native — verify player/ editor tolerance, cite the research). Test: Args_RecordBlock_IsFragmentedMp4_SoACrashKeepsTheFile — the flags are on the record block and absent from the streaming block. Verified by construction only: the file still needs one real interrupted-take check on Windows (kill ffmpeg mid-recording, play the result).

5. ✅ Pre-flight (shipped 2026-11-05) — before broadcast insert, one TCP probe to the ingest host:1935 (~2s timeout). Unreachable → Error toast, no insert, no ghost broadcast ("check VPN/network"). The "forgot the VPN" story, solved as prevention instead of recovery. Test: fake probe failure aborts go-live before any API call. Shipped: Services/IngestProbe.cs (IIngestProbe + TcpIngestProbe, 2 s timeout, rtmp→1935 / rtmps→443) and MainViewModel.IngestPreflightAsync, called BEFORE the first API call whenever a cached ingest URL exists (first run probes after resolving the stream, still before any broadcast is created). Tests: IngestPreflightTests — unreachable aborts, reachable continues, and the probe is exercised against real sockets (a listening port answers true, the same port closed answers false quickly).

Design decisions

• The sign is the status, the button is the action — no new chrome, no log lines, no spinners.
• Honest windows: countdown = observed grace; at zero we say OFF AIR, we don't pretend to reconnect.
• Why us > incumbents: constant reusable stream + persisted form make retry/restart cheap; OBS shows a reconnecting log line and Streamlabs silently dies.