UNIT 0: make a dropped webcam frame measurable, not just inferable
The creator is DHH and lipreads their own recordings, so a webcam that judders is an accessibility defect, not a polish item. Take ty-20261005-1001-821-0001.mp4 holds 25.6 distinct webcam images/s against the camera's 30fps with 281 gaps of 50-100ms - roughly one hitch in five - in a file that muxes flawlessly. Nothing in the app could see that: rawvideo carries no per-frame timestamps (ffmpeg stamps by ARRIVAL at the declared 60fps), and the stats line divided every average by the number of frames SUBMITTED, which deadline padding always fills to the full target. So the pump reported "300/300 frames, avg render 12.4ms" while producing 60 distinct composites. Helpers/FrameCadence.cs is one pure counter fed from three points: MediaCaptureFrameSource logs the device's own SystemRelativeTime stamps against our arrival times (TryAcquireLatestFrame keeps only the LATEST frame, so falling behind is silent by design), and FramePump logs the distinct webcam Epoch values that reached a composite. Read together they name the culprit - device/USB, our reader, or our compositor - instead of guessing. Render/resolve/wait now divide by the real render count, and "frames per 5s from R renders" is the duplicate-on-lag ratio stated honestly. An out-of-order timestamp counts as a frame but produces no gap and does not become the next interval's reference, so one bad stamp cannot invent a multi-second hitch twice. Also corrects a stale ai.md line (the "webcam renders at ~18fps" figure came from a different session and counted every newly rendered layer) and re-records the background-removal ruling as permanently out. Records the agreed-but-unbuilt Units 1-2, the no-fallback-camera ruling, and the SetFormatAsync dead end found by probing the 19041 projection. Suite 402/402, 0 warnings. The one red, LayerReorderPersistenceTests' RealMouseDrag, fails identically on a stashed clean tree - it needs an interactive desktop session this shell lacks.
This commit is contained in:
@@ -483,6 +483,11 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr
|
||||
fails with "camera in use" (or, if init slips through, the device exposes no preview source). Fix:
|
||||
quit Broadcast while streaming, or pick its virtual "NVIDIA Broadcast" device from the picker and the
|
||||
app captures the processed feed. This is a real-device finding (Logitech `VID_046D&PID_082D`).
|
||||
Correction (2026-10-05): **Windows Studio Effects is NOT a second option of this kind** — it runs
|
||||
inside the physical camera's own driver, so it never appears as a selectable device. Only tools
|
||||
that install a virtual camera (NVIDIA Broadcast, XSplit VCam) do. "Pick the vcam" is a user action in
|
||||
the camera picker, never an automatic substitution: creator ruling 2026-10-05 — the app must not
|
||||
bypass the user's explicit camera choice, so there is deliberately **no fallback-camera setting**.
|
||||
- **TFM:** `net8.0-windows10.0.19041.0` (app + tests) pulls the WinRT projection from the SDK reference
|
||||
packs — no NuGet package, no capability manifest (unpackaged desktop app works; the Windows privacy
|
||||
camera toggle still applies). `EnableWindowsTargeting` keeps WSL builds working.
|
||||
@@ -597,8 +602,56 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr
|
||||
- **GPU posture:** webcam frames are CPU (GPU-agnostic; WPF hardware-presents the preview anyway). Hardware
|
||||
encoders (NVENC/AMF/QSV) matter for the encoder task, not capture. D3DImage GPU compositing is deferred
|
||||
to the encoder task.
|
||||
- **Background removal = milestone 2** — ONNX Runtime + DirectML (CPU fallback), MediaPipe Selfie
|
||||
Segmentation (Apache-2.0), wired into the same `VideoFrame` seam. Not part of milestone 1.
|
||||
- **Background removal = PERMANENTLY OUT** (creator ruling 2026-10-05, confirmed: "no - not doing
|
||||
it. the note is correct and accurate"). The line here used to read "milestone 2", which was stale —
|
||||
it was never a plan, only a leftover from TASK 3's original scoping. The rationale that stands:
|
||||
a vcam app (NVIDIA Broadcast, XSplit VCam) does it free, and an in-app model means a runtime, a
|
||||
GPU/driver matrix, and per-scene lighting tuning — support hell for a solo dev
|
||||
(`TASKS/task-36-gold-pass.md`, `TASKS/task-03-capture-pipeline.md` item 19). Don't reopen it.
|
||||
|
||||
### Frame cadence — how a dropped frame became measurable (UNIT 0, 2026-10-05)
|
||||
|
||||
**The problem this exists to solve.** The recorded file cannot distinguish a dropped camera frame
|
||||
from a padded repeat. `-f rawvideo` carries no per-frame timestamps — ffmpeg assigns pts from frame
|
||||
order at the DECLARED fps — so a webcam delivering 25 of its 30 frames still muxes a full-length,
|
||||
correctly-timed 60fps file. Measured on take `ty-20261005-1001-821-0001.mp4` (85.6s, 1080p60): the
|
||||
lower-right webcam region held 25.6 distinct images/s against the camera's 30fps, with 1,153 normal
|
||||
33ms intervals, 281 gaps of 50–100ms, and 96 bunched 17ms gaps — roughly one hitch in five, which is
|
||||
what makes a face judder while the file looks flawless. (The "webcam renders at ~18fps" figure that
|
||||
used to sit in this map was WRONG: it came from a different session and counted every newly rendered
|
||||
layer, not webcam arrivals. Corrected 2026-10-05.)
|
||||
|
||||
**`Helpers/FrameCadence.cs`** — one pure counter, three feeds. `Add(timestamp)` on any shared
|
||||
monotonic clock; `Summarize()` → frames, rate (gaps ÷ span, NOT frames ÷ span — N frames span N-1
|
||||
periods), gaps, hitches (default threshold 50ms ≈ 1.5× a 30fps period), longest and average gap;
|
||||
`Describe(label)` gives every producer the same one-line shape so two lines can be read against each
|
||||
other. An out-of-order stamp counts as a frame but produces no gap AND does not become the reference
|
||||
for the next interval — otherwise one bad stamp invents a multi-second "hitch" twice.
|
||||
|
||||
**Three measurement points, and what each separates:**
|
||||
|
||||
| Line | Fed by | Answers |
|
||||
|---|---|---|
|
||||
| `Camera cadence WxH: arrivals … \| camera …` | `MediaCaptureFrameSource` | device/USB/driver cadence (`MediaFrame.SystemRelativeTime`, QPC domain, nullable) vs what the reader handed us. `TryAcquireLatestFrame` keeps only the LATEST frame, so falling behind is silent by design — this is where that shows |
|
||||
| `webcam …` on the pump's stats line | `FramePump.TimedResolver` | distinct webcam `VideoFrame.Epoch` values that reached a composite — what actually got into the output |
|
||||
| `frames per 5s from R renders` | `FramePump` | submits vs REAL composites — the duplicate-on-lag ratio, the only honest reading of how much of the stream is a repeat |
|
||||
|
||||
**Why the stats-line denominators changed (the actual bug).** Every average was divided by
|
||||
`statFrames`, the number of frames SUBMITTED — which deadline padding always fills to the full 60fps
|
||||
target. A pump fed one webcam frame per 200ms logged "300/300 frames per 5s, avg render 12.4ms"
|
||||
while producing 60 distinct composites: the render average was diluted ~5× and the loss was
|
||||
invisible. Render/resolve/wait are now divided by `statRenders`; submit stays per-frame (it really
|
||||
happens per submit). Regression test:
|
||||
`FramePumpTests.Stats_ReportRealRendersAndDistinctWebcamFrames_NotJustPaddedSubmits`.
|
||||
|
||||
**Reading the three together:** camera 30/s steady + arrivals 25/s = WE dropped it (reader or
|
||||
compositor). Ragged in both = the device, the USB link, or the driver. Pump `webcam` below
|
||||
`arrivals` = the compositor missed frames the reader had already delivered.
|
||||
|
||||
**Still unmeasured (next unit):** the camera is stuck at the C920 default 640×480 YUY2 @30 because
|
||||
all 185 `SetMediaStreamPropertiesAsync` attempts are refused under `SharedReadOnly`, so 720p MJPG is
|
||||
on the table and the compositor's headroom at ~3× the bytes is unknown. Measure first, then change
|
||||
the acquisition.
|
||||
|
||||
### Scene compositor (TASK 4 ship step 1 — shipped 2026-08-10, plan in TASKS.md)
|
||||
|
||||
@@ -943,7 +996,7 @@ scene each tick, resolves every element to its latest frame, composites it into
|
||||
and paces frames into the encoder at the tier's FPS. **rawvideo is stamped by ARRIVAL:** ffmpeg assigns
|
||||
pts from frame order at the declared fps — supply rate = output speed. A starved producer (pump < fps)
|
||||
ships a time-lapse, truncated file with NO error (take-2 lesson, 2026-09-01); the pump therefore logs
|
||||
`FramePump stats: n/target frames per 5s, avg render Xms (resolve R), avg submit Yms` so a slow stage names itself — and since 2026-09-04 so does the BUILD: every build gets a GUID stamped
|
||||
`FramePump stats: n/target frames per 5s from R renders, avg render Xms (resolve R2), avg submit Yms` so a slow stage names itself (the `frames … from R renders` pair and the render-count denominators arrived with UNIT 0 — see "Frame cadence" below; before that every average was divided by the SUBMITTED count, which deadline padding always fills to the full target, so a pump fed one webcam frame per 200ms still logged "300/300 frames" and its render average was diluted ~an order of magnitude) — and since 2026-09-04 so does the BUILD: every build gets a GUID stamped
|
||||
into `Helpers/BuildStamp` (csproj `GenerateBuildStamp` target, fresh per compile — no lying incremental build), logged at startup
|
||||
("Build xxxxxxxx (compiled ...)"). **Take 6 (stamp-less, ambiguous): render 35-41ms with slice 3 built or not unknown — attribution is why the stamp exists; takes 7+ are
|
||||
readable.** If the breakdown says resolve dominates, the suspects are pre-cached chat/config thrash or the web frame; if blit dominates, the general-path element sizes. **Pattern — everything is a constructor-injected
|
||||
|
||||
Reference in New Issue
Block a user