UNIT 0: make a dropped webcam frame measurable, not just inferable

The creator is DHH and lipreads their own recordings, so a webcam that
judders is an accessibility defect, not a polish item. Take
ty-20261005-1001-821-0001.mp4 holds 25.6 distinct webcam images/s against
the camera's 30fps with 281 gaps of 50-100ms - roughly one hitch in five -
in a file that muxes flawlessly. Nothing in the app could see that: rawvideo
carries no per-frame timestamps (ffmpeg stamps by ARRIVAL at the declared
60fps), and the stats line divided every average by the number of frames
SUBMITTED, which deadline padding always fills to the full target. So the
pump reported "300/300 frames, avg render 12.4ms" while producing 60
distinct composites.

Helpers/FrameCadence.cs is one pure counter fed from three points:
MediaCaptureFrameSource logs the device's own SystemRelativeTime stamps
against our arrival times (TryAcquireLatestFrame keeps only the LATEST
frame, so falling behind is silent by design), and FramePump logs the
distinct webcam Epoch values that reached a composite. Read together they
name the culprit - device/USB, our reader, or our compositor - instead of
guessing. Render/resolve/wait now divide by the real render count, and
"frames per 5s from R renders" is the duplicate-on-lag ratio stated
honestly.

An out-of-order timestamp counts as a frame but produces no gap and does not
become the next interval's reference, so one bad stamp cannot invent a
multi-second hitch twice.

Also corrects a stale ai.md line (the "webcam renders at ~18fps" figure
came from a different session and counted every newly rendered layer) and
re-records the background-removal ruling as permanently out. Records the
agreed-but-unbuilt Units 1-2, the no-fallback-camera ruling, and the
SetFormatAsync dead end found by probing the 19041 projection.

Suite 402/402, 0 warnings. The one red, LayerReorderPersistenceTests'
RealMouseDrag, fails identically on a stashed clean tree - it needs an
interactive desktop session this shell lacks.
This commit is contained in:
2026-10-05 11:10:22 -07:00
parent b2f219739d
commit 55ca1029a3
9 changed files with 532 additions and 30 deletions
+94 -20
View File
@@ -1,13 +1,79 @@
# HANDOFF — current state
**Branch:** `main` (pre-1.0, no feature branches — creator ruling 2026-08-24).
**Pushed 2026-10-04:** `origin/main` = `048cab7`. **NOT PUSHED since — local `main` is 5 commits
ahead and the tree is clean.** The stale-push-state failure `MyMistakes.md` warns about has now bitten
**Pushed 2026-10-04:** `origin/main` = `048cab7`. **NOT PUSHED since — local `main` is ahead and the
tree is clean.** The stale-push-state failure `MyMistakes.md` warns about has now bitten
twice, so the rule stands: **check `git rev-list --count origin/main..main` before trusting any line
here about push state.**
Unpushed: `5ad0106` broadcast diagnostic → `74da99c` + `0fb2f9c` privacy/Test rulings →
`8fbc370` diagnostic parse fix → `07a088a` **the explicit-bind fix** → `509ef36` window placement.
`8fbc370` diagnostic parse fix → `07a088a` **the explicit-bind fix** → `509ef36` window placement →
`b2f2197` handoff → `UNIT 0` frame-cadence measurement.
## 🚧 IN FLIGHT 2026-10-05 — UNIT 0 shipped: a dropped webcam frame is finally measurable
**The measurement the map never had.** Take `ty-20261005-1001-821-0001.mp4` (85.6s, 1920×1080,
H.264 NVENC, 5124 frames @60) holds **25.6 distinct webcam images/s** in its lower-right region
against the camera's 30fps: 1,153 normal 33ms intervals, **281 gaps of 50–100ms**, 96 bunched 17ms
gaps. One hitch in five — the judder a DHH creator lipreads against, invisible in a file that muxes
flawlessly. Measured with `signalstats`/YDIF over `crop=480:270:1400:780`; the bundled ffmpeg has no
`mpdecimate`. ⛔ The old "webcam renders at ~18fps" claim was wrong (different session, counted all
newly rendered layers) — corrected in `ai.md`.
**Shipped:** `Helpers/FrameCadence.cs` (pure counter: frames, gaps, hitches ≥50ms, longest/avg), fed
from three points — `MediaCaptureFrameSource` logs `Camera cadence WxH: arrivals … | camera …`
(our delivery vs the device's own `SystemRelativeTime`), and `FramePump` logs distinct webcam
`Epoch`s that reached a composite. Full detail in `ai.md` → "Frame cadence".
**Also fixed, and it was a real measurement bug:** every pump average was divided by the number of
frames **submitted**, which deadline padding always fills to the full 60fps target — so a pump fed
one webcam frame per 200ms logged "300/300 frames, avg render 12.4ms". Render/resolve/wait now divide
by `statRenders` and the line reads `300/300 frames per 5s from 60 renders`. That pair IS the
duplicate-on-lag ratio. Test: `FramePumpTests.Stats_ReportRealRendersAndDistinctWebcamFrames_…`.
**⏭ NEXT STEP — needs the creator, one short recording (60s is plenty):** record with the webcam
visible and paste nothing — I read `%APPDATA%\ytLlive\startup.log` for the `Camera cadence` and
`FramePump stats` lines. Reading them together names the culprit: camera steady + arrivals ragged =
we drop it (reader/compositor); ragged in both = device/USB/driver; pump `webcam` below `arrivals`
= the compositor missed frames the reader already had. **Then Unit 1** (exclusive lock + 720p, see
below) is judged against those numbers.
**Agreed but NOT yet built (decided 2026-10-05, so a later session does not re-derive it):**
- **Unit 1 — exclusive camera lock + 720p on the camera the creator picked.** Open with
`MediaCaptureSharingMode.ExclusiveControl`; if init is refused, reopen the **SAME camera** in
`SharedReadOnly`. Target MJPG ≥30fps up to 1280×720, ranked by frame rate then size (this also fixes
the existing candidate-ordering bug). **The creator REVERSED the ban on taking an exclusive camera
lock** — that ban was never written into the map, which is why the reversal must be recorded this
time. Accepted trade-off: an app needing camera *control* that opens it after us will fail.
- ✅ **API probe DONE 2026-10-05 against the actual 19041.57 projection** (a throwaway
compile-probe file, since deleted — do not re-derive, and note the build cost of finding out):
`MediaCaptureSharingMode.ExclusiveControl` **is** projected, so the old comment in
`MediaCaptureFrameSource.cs` ("Exclusive is not projected by the 19041 SDK") is wrong and must be
corrected. `Exclusive` and `SharedReadOnly` are the **only two** `MediaCaptureSharingMode` values.
`MediaFrameSource.SupportedFormats` and `CurrentFormat` exist.
⛔ **`MediaFrameSource.SetFormatAsync(MediaFrameFormat)` is a DEAD END here** — it takes a
`MediaFrameFormat`, and that type has **no public constructor and no static factory** in the
projection (it is only obtainable from a live frame's `VideoMediaFrame.Format`). The two-argument
`(subtype, properties)` overload in Microsoft's docs does not exist here. So the format change must
go through the `VideoDeviceController.SetMediaStreamPropertiesAsync` path the code ALREADY has —
it is only refused because `SharingMode` is `SharedReadOnly`. Under an exclusive lock the existing
negotiation should simply start working, which is a much smaller change than planned.
- **Remaining unknown (needs a real device, not research):** whether `SetMediaStreamPropertiesAsync`
actually succeeds under `ExclusiveControl` on this C920. The `MediaFrameSource.SupportedFormats`
route is the fallback probe if it refuses — enumerate, and log what the device offers before
assuming anything.
- **Unit 2 — blinking "SD" badge in the preview only**, when a better format was available and we did
not get it. Top-right corner of the webcam, or the 1:30 point ON THE RIM for a round clip (a round
webcam has no corner); outside the mirror transform. ~1Hz opacity blink (under the 3/sec
photosensitivity limit), tooltip on hover only, **must never appear in the recording or stream**
(the compositor draws from camera frames, not the WPF tree). Style lives in `Themes/Controls.xaml`
— a DataTrigger storyboard cannot be stopped inside a DataTemplate (`PreviewPane.xaml:168-171`).
- ⛔ **NO fallback-camera setting, ever** — creator ruling: *"no fallback - that's bypassing the
user's explicit choice."* Picking a vcam is a user action in the camera picker. Corollary, also
recorded in `ai.md`: **Windows Studio Effects is NOT a vcam** — it runs inside the physical camera's
driver and never appears as a selectable device. Only Broadcast / XSplit VCam install one.
- ⛔ **Background removal: permanently out.** Creator confirmed the existing note: *"no - not doing
it. the note is correct and accurate."* `ai.md` said "milestone 2", which was stale — now corrected.
## 🚨 FIXED 2026-10-04 — broadcasts never went live because they were never BOUND
@@ -281,11 +347,12 @@ Detail and the open 11.12 re-check: [`TASKS/task-50-publish-recording.md`](TASKS
## Landmines
- **Frame pump jitter, unquantified (2026-10-04).** The same test stream logged **86 stalls
per 5s** (`FramePump stall: iteration 40–50ms` against a 17ms budget; 142 such lines in an
8-second window) while `FramePump stats` still read 301/300 frames, 0 dropped. Nominal 60fps
with zero drops *and* repeated 3× overruns is a contradiction worth one measurement, not a
guess. Do not "fix" it from the log alone.
- **Frame pump jitter — counters now honest, cause NOT yet named (2026-10-05).** The same test
stream logged **86 stalls per 5s** (`FramePump stall: iteration 40–50ms` against a 17ms budget)
while `FramePump stats` read 301/300 frames, 0 dropped. UNIT 0 removed the reason those two
numbers could disagree (the averages were diluted by padded submits) and added the counters that
separate reader from compositor — **but no one has read a post-UNIT-0 log yet.** A creator
recording is required first. Do not "fix" the pump before that reading.
- **Stale fact inside `TASKS.md` (left alone, flagged here per the Scope Lock).** The "1.0 gates"
section asserts "**TASK 36 is now shipped**", but the catalog row 36 reads **☐ Queued** and
`task-36-gold-pass.md` still shows items 1–6 unchecked. The line most likely means *item 2*
@@ -293,9 +360,10 @@ Detail and the open 11.12 re-check: [`TASKS/task-50-publish-recording.md`](TASKS
flagging rather than fixing because it is not this unit's job.
- **A stale `ytLive.exe` (PID 2544) locked `bin/.../ytLive.exe`** and broke `dotnet run` with
MSB3027. Killed. **If the build fails to copy `ytLive.exe`, check for a running instance first.**
- **`LayerReorderPersistenceTests.RealMouseDrag_OnTheLayerList_PersistsTheReorder` is flaky by
environment** (needs an interactive desktop session; 2 fail / 1 pass in isolation). **Run the
suite with ytLive CLOSED.** Not a regression.
- **`LayerReorderPersistenceTests.RealMouseDrag_OnTheLayerList_PersistsTheReorder` needs an
interactive desktop session.** As of 2026-10-05 it fails deterministically here (`ImgC` vs
`ImgA`), verified identical on a clean tree. **Run the suite with ytLive CLOSED** and expect this
one to fail in this shell. Not a regression — don't diagnose it from a WSL-shell failure.
- **`RealAppHost.RunAsync` exists for a reason** — a frame-pump test MUST `await` inside it. A
blocking wait occupies the one shared STA thread and **hangs the whole suite with no output.**
Always background a `vstest` run and poll the log; a foreground piped `vstest` returns nothing
@@ -308,11 +376,17 @@ Detail and the open 11.12 re-check: [`TASKS/task-50-publish-recording.md`](TASKS
## Test state
**395/395, 0 warnings, re-verified 2026-10-04 on a clean test-project build** (381 + 3 diagnostic
`FirstItem` cases + 1 window fact + 3 placement theory cases + 1 bind test, less 1 removed).
Build the **test** project, not just the app project, or VSTest runs a stale DLL.
**402/402, 0 warnings, verified 2026-10-05** (395 + 6 `FrameCadenceTests` + 1 pump-stats integration
test). Build the **test** project, not just the app project, or VSTest runs a stale DLL.
**Launch exe for the creator's test run: `bin/Debug/net8.0-windows10.0.19041.0/ytLive.exe`.**
**Confirmed environmental flake, NOT a regression (re-checked 2026-10-05, and it now fails
deterministically rather than 2-of-3):** `LayerReorderPersistenceTests.RealMouseDrag_OnTheLayerList_
PersistsTheReorder` fails with `Expected "ImgC", Actual "ImgA"` at `RealAppHost.Run` line 68. **It
fails identically on a stashed clean tree**, so it has nothing to do with UNIT 0 — it needs an
interactive desktop session this shell does not have (the mouse drag never lands). Do not chase it;
it is listed under Landmines as the known interactive-session test.
**⚠️ New flake, do not chase it as a regression:** on the *first* full-suite run,
`AudioPipelineTests.MasterLimiter_CapsTheLiveMix_OnThePipe` failed with `Expected 0.891,
Actual 0` — a **zero**, not a slightly-off value, which is the signature of the pipe not being
@@ -335,12 +409,12 @@ whole-suite scheduling artifact of the same family as the `RealAppHost` STA land
1. **~PUSH, then the creator's 60-second Test** — see the FIXED box above. Packaging is NOT next;
the creator's ruling 2026-10-04: still in active development, so ship-time work waits and queued
features come first.
2. **Frame-pump jitter (queued, unquantified):** ~86 stalls per 5 s in one log window, 142 in an
8-second window, while stats reported 301/300 frames and **zero** drops. Measure before touching
anything — the counter and the missing-frame count currently disagree, and "fixing" the pump
without knowing which number is wrong risks making it worse. **Creator to-do, not yet queued:
long-run resource/consumption/recycling benchmarking** (how long the app stays healthy under full
burden) — record only, do not start it.
2. **Webcam cadence → Unit 1 (exclusive lock + 720p).** UNIT 0 is shipped and the counters are in
`startup.log`; the reading needs one 60-second creator recording (details + the exact
read-the-log-together table in the IN FLIGHT box above). Then Unit 1: exclusive camera lock +
720p, and Unit 2: the blinking SD badge. **Creator to-do, not yet queued: long-run
resource/consumption/recycling benchmarking** (how long the app stays healthy under full burden) —
record only, do not start it.
2. **The packaging project + `Package.appxmanifest`** (TASK 48 items 2–3) — the next code unit.
Full trust, `webcam`/`microphone`, English only; Velopack deleted. Add the
`DefaultRecordFolder()` comment in the same unit, and the WebView2 framework dependency +