From 9d04aea55bd42367acc068024ec2028dd581801c Mon Sep 17 00:00:00 2001 From: gramps Date: Mon, 5 Oct 2026 12:06:28 -0700 Subject: [PATCH] UNIT 0 reading: the camera was never the problem, so Unit 1 is the wrong lever Take ty-20261005-1201-556-0001.mp4 (75.3s) plus its log settle it. The C920 delivers 30.0/s with ZERO gaps over 50ms and its own device stamps agree - camera, USB and driver are clean. The pump managed 34 composites per 5s at ~141ms each, and the file holds 6.8 distinct webcam images per second at 88.7% duplicates with no arrival hitch whatsoever. The judder is us holding one face for 140ms and padding the rest of a 60fps container, which is precisely what a lipreader experiences as "cutting out". So 720p is a SHARPNESS win, not a smoothness win: it triples the bytes the compositor stretches per render on a path already 4.4x over budget. The exclusive camera lock stays agreed and stays queued, but it moves after the compositor work or it makes the judder measurably worse. The number still unexplained: the 10:01 take hit 25.6 images/s and this one 6.8, with the same scene shape (full-render split=0 elements=6 dynamic=5) and the same 640x480 camera - composites went 29-34ms to 143-155ms between sessions and nothing logged says why. There is no per-element composite breakdown and no scene dump at pump start, so the next unit is measurement, not a fix. Recorded the pump-counter overcount (13/s vs 6.8/s: one 141ms composite lets ~4 camera frames arrive, and the counter stamps the first epoch the resolver sees, not the frame actually drawn) so it is not re-derived as a bug later. --- HANDOFF.md | 40 ++++++++++++++++++++++++++++++++++++++++ TASKS.md | 23 ++++++++++++++++------- ai.md | 30 ++++++++++++++++++++++++++---- 3 files changed, 82 insertions(+), 11 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index 4c7dde0..9c150d2 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -10,6 +10,46 @@ Unpushed: `5ad0106` broadcast diagnostic → `74da99c` + `0fb2f9c` privacy/Test `8fbc370` diagnostic parse fix → `07a088a` **the explicit-bind fix** → `509ef36` window placement → `b2f2197` handoff → `UNIT 0` frame-cadence measurement. +## 🚨 ANSWERED 2026-10-05 — the judder is the COMPOSITOR, not the camera. Unit 1 is the wrong next move. + +Take `ty-20261005-1201-556-0001.mp4` (75.3s, 1920×1080, 60fps) + `startup.log`, read together: + +| Counter | Value | Reading | +|---|---|---| +| `Camera cadence … arrivals` | **30.0/s, 0 gaps ≥50ms**, avg 33.4ms | the camera is **perfect**. Not the culprit. | +| `Camera cadence … camera` (device stamps) | 30.0/s, identical | the USB link and driver are clean too | +| `FramePump … from N renders` | **34 renders per 5s = 6.8/s**, avg render **141ms**, worst 294ms | one full composite takes longer than four camera frames | +| `webcam …` on the pump's line | 13.0/s, 34 gaps ≥50ms | only 13 of the 30 delivered frames ever reached a composite | +| the recording itself | **6.8 distinct webcam images/s, 88.7% duplicates, ZERO arrival hitches** | the judder is *only* us repeating a frame — nothing arrives late | + +**⇒ The camera delivers all 30 of its frames, evenly. The compositor can only draw ~7 composites a +second, so the 60fps container is 89% repeats of a face held for 140ms.** That is exactly what a +lipreader experiences as "cutting out". + +**⛔ Do NOT do Unit 1 (720p / exclusive lock) as the judder fix — it is the wrong lever.** 720p would +triple the bytes the compositor stretches per render on a path that is already 4.4× over its budget. +The exclusive lock is still worth having **for sharpness** (the C920 is pinned at 640×480 YUY2 and +the Chat scene upscales that ~2.1×), but it must be scheduled as a quality win, not a smoothness win, +and it should land **after** the compositor work — otherwise it makes the judder measurably worse. + +**The number that needs explaining:** the 10:01 take hit **25.6** distinct images/s; this one hit +**6.8**. Same scene shape both times (`full-render split=0 elements=6 dynamic=5`), same camera, same +640×480. Composites went **29–34ms at 10:03 → 143–155ms at 12:01**, a 4.5× regression between +sessions with nothing in the log explaining it. `resolve` is 0.1ms, so the cost is pure pixel work, +not frame lookup. **That regression is the next thing to find, and the log cannot say what changed** — +no scene dump is written at stream start. ⛔ Do not guess at it. + +**Cheapest next step, and it is a measurement not a fix:** log the per-element composite breakdown +(size + ms for each of the 6 elements) plus the scene contents at pump start. One run names which +element eats 140ms and whether the 10:03→12:01 regression was a scene change or something else. +Everything else in the compositor is guesswork until that exists. + +**Also worth knowing (a counter subtlety, so nobody re-derives it):** the pump's `webcam` line reads +13/s while the file shows 6.8/s. Not a discrepancy — the counter stamps a frame when the RESOLVER +first sees it, and one composite takes 141ms, so ~4 camera frames arrive during it: the signature +pass records one epoch and the actual draw uses a later one. **The file measurement is the truth; +the pump counter overcounts by design** (it is a pipeline-health signal, not a frame-delivery metric). + ## 🚧 IN FLIGHT 2026-10-05 — UNIT 0 shipped: a dropped webcam frame is finally measurable **The measurement the map never had.** Take `ty-20261005-1001-821-0001.mp4` (85.6s, 1920×1080, diff --git a/TASKS.md b/TASKS.md index 4a5c4e2..2537a38 100644 --- a/TASKS.md +++ b/TASKS.md @@ -160,13 +160,22 @@ second encoder path needing a "redirect". Record+simulcast is one ffmpeg with tw webcam judders, ~1 hitch in 5, invisible in the output file). **UNIT 0 shipped** (counters + `Helpers/FrameCadence.cs`; see `ai.md` → "Frame cadence"). **Needs a creator recording before the next unit.** Remaining units, all agreed 2026-10-05, one Good Dog test each: - - **Unit 1:** `MediaCaptureSharingMode.ExclusiveControl` + MJPG ≥30fps up to 1280×720 via the - existing `SetMediaStreamPropertiesAsync` path (refused only because `SharedReadOnly`); same-camera - `SharedReadOnly` retry. **The creator reversed the ban on taking an exclusive camera lock.** - API probe already done — see HANDOFF.md, incl. the `SetFormatAsync` dead end. - - **Unit 2:** blinking amber **"SD" badge**, preview only, never in the recording/stream, when a - better format was available and not obtained. Top-right corner, or the 1:30 rim point when the - clip is round. + - **⛔ UNIT 1 IS NOT NEXT — re-scoped 2026-10-05.** The first real reading exonerated the camera: + it delivers 30.0/s with zero hitches; the compositor manages **6.8 composites/s at ~141ms each**, + so the file is **88.7% duplicate frames** and that is the judder. 720p triples the bytes stretched + per render on an already-over-budget path. Keep `ExclusiveControl` + 720p as a **sharpness** win + and schedule it AFTER the compositor work. + - **⏭ ACTUAL NEXT: per-element composite timing + a scene dump at pump start.** One run names which + of the 6 elements eats 140ms and explains the 10:03→12:01 regression (29–34ms → 143–155ms, same + scene shape). Measurement only — no fix until it names the element. + - **Unit 2 (later, unchanged):** blinking amber **"SD" badge**, preview only, never in the + recording/stream, when a better format was available and not obtained. Top-right corner, or the + 1:30 rim point when the clip is round. + - **Deferred to after Unit 1:** `MediaCaptureSharingMode.ExclusiveControl` + MJPG ≥30fps up to + 1280×720 via the existing `SetMediaStreamPropertiesAsync` path (refused only because + `SharedReadOnly`); same-camera `SharedReadOnly` retry. **The creator reversed the ban on taking an + exclusive camera lock.** API probe already done — see HANDOFF.md, incl. the `SetFormatAsync` + dead end. - ⛔ **No fallback-camera feature, ever** — *"that's bypassing the user's explicit choice."* - ⛔ **No background removal** — re-affirmed out 2026-10-05. - **TASK 40 — App Settings round** — plan saved 2026-09-22 (`TASKS/task-40-app-settings-round.md`), diff --git a/ai.md b/ai.md index eeffc7e..d79fcf0 100644 --- a/ai.md +++ b/ai.md @@ -648,10 +648,32 @@ happens per submit). Regression test: compositor). Ragged in both = the device, the USB link, or the driver. Pump `webcam` below `arrivals` = the compositor missed frames the reader had already delivered. -**Still unmeasured (next unit):** the camera is stuck at the C920 default 640×480 YUY2 @30 because -all 185 `SetMediaStreamPropertiesAsync` attempts are refused under `SharedReadOnly`, so 720p MJPG is -on the table and the compositor's headroom at ~3× the bytes is unknown. Measure first, then change -the acquisition. +**✅ WHAT THE FIRST REAL READING SAID (2026-10-05) — the camera was never the problem.** +`ty-20261005-1201-556-0001.mp4` + its log: the C920 delivers **30.0/s with ZERO gaps ≥50ms**, and the +device's own stamps are identical, so the camera, USB link, and driver are all clean. The pump did +**34 composites per 5s (6.8/s) at ~141ms each** (worst 294ms), and the file holds **6.8 distinct +webcam images/s at 88.7% duplicates with no arrival hitches at all**. The judder is entirely us +holding one face for 140ms and padding the rest of a 60fps container. `resolve` is 0.1ms, so it is +pure pixel work in the composite, not frame lookup. Two consequences, both load-bearing: + +- **720p is a sharpness win, NOT a smoothness win** — it triples the bytes stretched per render on a + path already 4.4× over budget. Schedule the exclusive lock AFTER the compositor work or it makes + the judder measurably worse. +- **The 10:01 take hit 25.6 images/s and the 12:01 take 6.8, same scene shape + (`full-render split=0 elements=6 dynamic=5`), same 640×480 camera.** Composites went 29–34ms → + 143–155ms between sessions with nothing logged explaining it. **There is no per-element composite + breakdown and no scene dump at pump start**, so the log cannot name the culprit — that gap is the + next thing to close, and guessing at it is explicitly out of bounds. + +**Counter subtlety, so it is not re-derived:** the pump's `webcam` line read 13/s while the file held +6.8/s. Not a bug — the counter stamps a frame when the resolver FIRST sees it, and one composite takes +141ms, so ~4 camera frames arrive during it (signature pass stamps one epoch, the actual draw uses a +later one). The file is the truth; the pump counter is a pipeline-health signal and overcounts by +design. + +**Still unmeasured:** the camera is stuck at the C920 default 640×480 YUY2 @30 because all 185 +`SetMediaStreamPropertiesAsync` attempts are refused under `SharedReadOnly`, so 720p MJPG is available +— but see above, it is a quality win only. ### Scene compositor (TASK 4 ship step 1 — shipped 2026-08-10, plan in TASKS.md)