UNIT 0: make a dropped webcam frame measurable, not just inferable

The creator is DHH and lipreads their own recordings, so a webcam that
judders is an accessibility defect, not a polish item. Take
ty-20261005-1001-821-0001.mp4 holds 25.6 distinct webcam images/s against
the camera's 30fps with 281 gaps of 50-100ms - roughly one hitch in five -
in a file that muxes flawlessly. Nothing in the app could see that: rawvideo
carries no per-frame timestamps (ffmpeg stamps by ARRIVAL at the declared
60fps), and the stats line divided every average by the number of frames
SUBMITTED, which deadline padding always fills to the full target. So the
pump reported "300/300 frames, avg render 12.4ms" while producing 60
distinct composites.

Helpers/FrameCadence.cs is one pure counter fed from three points:
MediaCaptureFrameSource logs the device's own SystemRelativeTime stamps
against our arrival times (TryAcquireLatestFrame keeps only the LATEST
frame, so falling behind is silent by design), and FramePump logs the
distinct webcam Epoch values that reached a composite. Read together they
name the culprit - device/USB, our reader, or our compositor - instead of
guessing. Render/resolve/wait now divide by the real render count, and
"frames per 5s from R renders" is the duplicate-on-lag ratio stated
honestly.

An out-of-order timestamp counts as a frame but produces no gap and does not
become the next interval's reference, so one bad stamp cannot invent a
multi-second hitch twice.

Also corrects a stale ai.md line (the "webcam renders at ~18fps" figure
came from a different session and counted every newly rendered layer) and
re-records the background-removal ruling as permanently out. Records the
agreed-but-unbuilt Units 1-2, the no-fallback-camera ruling, and the
SetFormatAsync dead end found by probing the 19041 projection.

Suite 402/402, 0 warnings. The one red, LayerReorderPersistenceTests'
RealMouseDrag, fails identically on a stashed clean tree - it needs an
interactive desktop session this shell lacks.
This commit is contained in:
2026-10-05 11:10:22 -07:00
parent b2f219739d
commit 55ca1029a3
9 changed files with 532 additions and 30 deletions
+56 -3
View File
@@ -483,6 +483,11 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr
fails with "camera in use" (or, if init slips through, the device exposes no preview source). Fix:
quit Broadcast while streaming, or pick its virtual "NVIDIA Broadcast" device from the picker and the
app captures the processed feed. This is a real-device finding (Logitech `VID_046D&PID_082D`).
Correction (2026-10-05): **Windows Studio Effects is NOT a second option of this kind** — it runs
inside the physical camera's own driver, so it never appears as a selectable device. Only tools
that install a virtual camera (NVIDIA Broadcast, XSplit VCam) do. "Pick the vcam" is a user action in
the camera picker, never an automatic substitution: creator ruling 2026-10-05 — the app must not
bypass the user's explicit camera choice, so there is deliberately **no fallback-camera setting**.
- **TFM:** `net8.0-windows10.0.19041.0` (app + tests) pulls the WinRT projection from the SDK reference
packs — no NuGet package, no capability manifest (unpackaged desktop app works; the Windows privacy
camera toggle still applies). `EnableWindowsTargeting` keeps WSL builds working.
@@ -597,8 +602,56 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr
- **GPU posture:** webcam frames are CPU (GPU-agnostic; WPF hardware-presents the preview anyway). Hardware
encoders (NVENC/AMF/QSV) matter for the encoder task, not capture. D3DImage GPU compositing is deferred
to the encoder task.
- **Background removal = milestone 2** — ONNX Runtime + DirectML (CPU fallback), MediaPipe Selfie
Segmentation (Apache-2.0), wired into the same `VideoFrame` seam. Not part of milestone 1.
- **Background removal = PERMANENTLY OUT** (creator ruling 2026-10-05, confirmed: "no - not doing
it. the note is correct and accurate"). The line here used to read "milestone 2", which was stale —
it was never a plan, only a leftover from TASK 3's original scoping. The rationale that stands:
a vcam app (NVIDIA Broadcast, XSplit VCam) does it free, and an in-app model means a runtime, a
GPU/driver matrix, and per-scene lighting tuning — support hell for a solo dev
(`TASKS/task-36-gold-pass.md`, `TASKS/task-03-capture-pipeline.md` item 19). Don't reopen it.
### Frame cadence — how a dropped frame became measurable (UNIT 0, 2026-10-05)
**The problem this exists to solve.** The recorded file cannot distinguish a dropped camera frame
from a padded repeat. `-f rawvideo` carries no per-frame timestamps — ffmpeg assigns pts from frame
order at the DECLARED fps — so a webcam delivering 25 of its 30 frames still muxes a full-length,
correctly-timed 60fps file. Measured on take `ty-20261005-1001-821-0001.mp4` (85.6s, 1080p60): the
lower-right webcam region held 25.6 distinct images/s against the camera's 30fps, with 1,153 normal
33ms intervals, 281 gaps of 50–100ms, and 96 bunched 17ms gaps — roughly one hitch in five, which is
what makes a face judder while the file looks flawless. (The "webcam renders at ~18fps" figure that
used to sit in this map was WRONG: it came from a different session and counted every newly rendered
layer, not webcam arrivals. Corrected 2026-10-05.)
**`Helpers/FrameCadence.cs`** — one pure counter, three feeds. `Add(timestamp)` on any shared
monotonic clock; `Summarize()` → frames, rate (gaps ÷ span, NOT frames ÷ span — N frames span N-1
periods), gaps, hitches (default threshold 50ms ≈ 1.5× a 30fps period), longest and average gap;
`Describe(label)` gives every producer the same one-line shape so two lines can be read against each
other. An out-of-order stamp counts as a frame but produces no gap AND does not become the reference
for the next interval — otherwise one bad stamp invents a multi-second "hitch" twice.
**Three measurement points, and what each separates:**
| Line | Fed by | Answers |
|---|---|---|
| `Camera cadence WxH: arrivals … \| camera …` | `MediaCaptureFrameSource` | device/USB/driver cadence (`MediaFrame.SystemRelativeTime`, QPC domain, nullable) vs what the reader handed us. `TryAcquireLatestFrame` keeps only the LATEST frame, so falling behind is silent by design — this is where that shows |
| `webcam …` on the pump's stats line | `FramePump.TimedResolver` | distinct webcam `VideoFrame.Epoch` values that reached a composite — what actually got into the output |
| `frames per 5s from R renders` | `FramePump` | submits vs REAL composites — the duplicate-on-lag ratio, the only honest reading of how much of the stream is a repeat |
**Why the stats-line denominators changed (the actual bug).** Every average was divided by
`statFrames`, the number of frames SUBMITTED — which deadline padding always fills to the full 60fps
target. A pump fed one webcam frame per 200ms logged "300/300 frames per 5s, avg render 12.4ms"
while producing 60 distinct composites: the render average was diluted ~5× and the loss was
invisible. Render/resolve/wait are now divided by `statRenders`; submit stays per-frame (it really
happens per submit). Regression test:
`FramePumpTests.Stats_ReportRealRendersAndDistinctWebcamFrames_NotJustPaddedSubmits`.
**Reading the three together:** camera 30/s steady + arrivals 25/s = WE dropped it (reader or
compositor). Ragged in both = the device, the USB link, or the driver. Pump `webcam` below
`arrivals` = the compositor missed frames the reader had already delivered.
**Still unmeasured (next unit):** the camera is stuck at the C920 default 640×480 YUY2 @30 because
all 185 `SetMediaStreamPropertiesAsync` attempts are refused under `SharedReadOnly`, so 720p MJPG is
on the table and the compositor's headroom at ~3× the bytes is unknown. Measure first, then change
the acquisition.
### Scene compositor (TASK 4 ship step 1 — shipped 2026-08-10, plan in TASKS.md)
@@ -943,7 +996,7 @@ scene each tick, resolves every element to its latest frame, composites it into
and paces frames into the encoder at the tier's FPS. **rawvideo is stamped by ARRIVAL:** ffmpeg assigns
pts from frame order at the declared fps — supply rate = output speed. A starved producer (pump < fps)
ships a time-lapse, truncated file with NO error (take-2 lesson, 2026-09-01); the pump therefore logs
`FramePump stats: n/target frames per 5s, avg render Xms (resolve R), avg submit Yms` so a slow stage names itself — and since 2026-09-04 so does the BUILD: every build gets a GUID stamped
`FramePump stats: n/target frames per 5s from R renders, avg render Xms (resolve R2), avg submit Yms` so a slow stage names itself (the `frames … from R renders` pair and the render-count denominators arrived with UNIT 0 — see "Frame cadence" below; before that every average was divided by the SUBMITTED count, which deadline padding always fills to the full target, so a pump fed one webcam frame per 200ms still logged "300/300 frames" and its render average was diluted ~an order of magnitude) — and since 2026-09-04 so does the BUILD: every build gets a GUID stamped
into `Helpers/BuildStamp` (csproj `GenerateBuildStamp` target, fresh per compile — no lying incremental build), logged at startup
("Build xxxxxxxx (compiled ...)"). **Take 6 (stamp-less, ambiguous): render 35-41ms with slice 3 built or not unknown — attribution is why the stamp exists; takes 7+ are
readable.** If the breakdown says resolve dominates, the suspects are pre-cached chat/config thrash or the web frame; if blit dominates, the general-path element sizes. **Pattern — everything is a constructor-injected