diff --git a/HANDOFF.md b/HANDOFF.md index 72150ab..4c7dde0 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,13 +1,79 @@ # HANDOFF — current state **Branch:** `main` (pre-1.0, no feature branches — creator ruling 2026-08-24). -**Pushed 2026-10-04:** `origin/main` = `048cab7`. **NOT PUSHED since — local `main` is 5 commits -ahead and the tree is clean.** The stale-push-state failure `MyMistakes.md` warns about has now bitten +**Pushed 2026-10-04:** `origin/main` = `048cab7`. **NOT PUSHED since — local `main` is ahead and the +tree is clean.** The stale-push-state failure `MyMistakes.md` warns about has now bitten twice, so the rule stands: **check `git rev-list --count origin/main..main` before trusting any line here about push state.** Unpushed: `5ad0106` broadcast diagnostic → `74da99c` + `0fb2f9c` privacy/Test rulings → -`8fbc370` diagnostic parse fix → `07a088a` **the explicit-bind fix** → `509ef36` window placement. +`8fbc370` diagnostic parse fix → `07a088a` **the explicit-bind fix** → `509ef36` window placement → +`b2f2197` handoff → `UNIT 0` frame-cadence measurement. + +## 🚧 IN FLIGHT 2026-10-05 — UNIT 0 shipped: a dropped webcam frame is finally measurable + +**The measurement the map never had.** Take `ty-20261005-1001-821-0001.mp4` (85.6s, 1920×1080, +H.264 NVENC, 5124 frames @60) holds **25.6 distinct webcam images/s** in its lower-right region +against the camera's 30fps: 1,153 normal 33ms intervals, **281 gaps of 50–100ms**, 96 bunched 17ms +gaps. One hitch in five — the judder a DHH creator lipreads against, invisible in a file that muxes +flawlessly. Measured with `signalstats`/YDIF over `crop=480:270:1400:780`; the bundled ffmpeg has no +`mpdecimate`. ⛔ The old "webcam renders at ~18fps" claim was wrong (different session, counted all +newly rendered layers) — corrected in `ai.md`. + +**Shipped:** `Helpers/FrameCadence.cs` (pure counter: frames, gaps, hitches ≥50ms, longest/avg), fed +from three points — `MediaCaptureFrameSource` logs `Camera cadence WxH: arrivals … | camera …` +(our delivery vs the device's own `SystemRelativeTime`), and `FramePump` logs distinct webcam +`Epoch`s that reached a composite. Full detail in `ai.md` → "Frame cadence". + +**Also fixed, and it was a real measurement bug:** every pump average was divided by the number of +frames **submitted**, which deadline padding always fills to the full 60fps target — so a pump fed +one webcam frame per 200ms logged "300/300 frames, avg render 12.4ms". Render/resolve/wait now divide +by `statRenders` and the line reads `300/300 frames per 5s from 60 renders`. That pair IS the +duplicate-on-lag ratio. Test: `FramePumpTests.Stats_ReportRealRendersAndDistinctWebcamFrames_…`. + +**⏭ NEXT STEP — needs the creator, one short recording (60s is plenty):** record with the webcam +visible and paste nothing — I read `%APPDATA%\ytLlive\startup.log` for the `Camera cadence` and +`FramePump stats` lines. Reading them together names the culprit: camera steady + arrivals ragged = +we drop it (reader/compositor); ragged in both = device/USB/driver; pump `webcam` below `arrivals` += the compositor missed frames the reader already had. **Then Unit 1** (exclusive lock + 720p, see +below) is judged against those numbers. + +**Agreed but NOT yet built (decided 2026-10-05, so a later session does not re-derive it):** +- **Unit 1 — exclusive camera lock + 720p on the camera the creator picked.** Open with + `MediaCaptureSharingMode.ExclusiveControl`; if init is refused, reopen the **SAME camera** in + `SharedReadOnly`. Target MJPG ≥30fps up to 1280×720, ranked by frame rate then size (this also fixes + the existing candidate-ordering bug). **The creator REVERSED the ban on taking an exclusive camera + lock** — that ban was never written into the map, which is why the reversal must be recorded this + time. Accepted trade-off: an app needing camera *control* that opens it after us will fail. + - ✅ **API probe DONE 2026-10-05 against the actual 19041.57 projection** (a throwaway + compile-probe file, since deleted — do not re-derive, and note the build cost of finding out): + `MediaCaptureSharingMode.ExclusiveControl` **is** projected, so the old comment in + `MediaCaptureFrameSource.cs` ("Exclusive is not projected by the 19041 SDK") is wrong and must be + corrected. `Exclusive` and `SharedReadOnly` are the **only two** `MediaCaptureSharingMode` values. + `MediaFrameSource.SupportedFormats` and `CurrentFormat` exist. + ⛔ **`MediaFrameSource.SetFormatAsync(MediaFrameFormat)` is a DEAD END here** — it takes a + `MediaFrameFormat`, and that type has **no public constructor and no static factory** in the + projection (it is only obtainable from a live frame's `VideoMediaFrame.Format`). The two-argument + `(subtype, properties)` overload in Microsoft's docs does not exist here. So the format change must + go through the `VideoDeviceController.SetMediaStreamPropertiesAsync` path the code ALREADY has — + it is only refused because `SharingMode` is `SharedReadOnly`. Under an exclusive lock the existing + negotiation should simply start working, which is a much smaller change than planned. + - **Remaining unknown (needs a real device, not research):** whether `SetMediaStreamPropertiesAsync` + actually succeeds under `ExclusiveControl` on this C920. The `MediaFrameSource.SupportedFormats` + route is the fallback probe if it refuses — enumerate, and log what the device offers before + assuming anything. +- **Unit 2 — blinking "SD" badge in the preview only**, when a better format was available and we did + not get it. Top-right corner of the webcam, or the 1:30 point ON THE RIM for a round clip (a round + webcam has no corner); outside the mirror transform. ~1Hz opacity blink (under the 3/sec + photosensitivity limit), tooltip on hover only, **must never appear in the recording or stream** + (the compositor draws from camera frames, not the WPF tree). Style lives in `Themes/Controls.xaml` + — a DataTrigger storyboard cannot be stopped inside a DataTemplate (`PreviewPane.xaml:168-171`). +- ⛔ **NO fallback-camera setting, ever** — creator ruling: *"no fallback - that's bypassing the + user's explicit choice."* Picking a vcam is a user action in the camera picker. Corollary, also + recorded in `ai.md`: **Windows Studio Effects is NOT a vcam** — it runs inside the physical camera's + driver and never appears as a selectable device. Only Broadcast / XSplit VCam install one. +- ⛔ **Background removal: permanently out.** Creator confirmed the existing note: *"no - not doing + it. the note is correct and accurate."* `ai.md` said "milestone 2", which was stale — now corrected. ## 🚨 FIXED 2026-10-04 — broadcasts never went live because they were never BOUND @@ -281,11 +347,12 @@ Detail and the open 11.12 re-check: [`TASKS/task-50-publish-recording.md`](TASKS ## Landmines -- **Frame pump jitter, unquantified (2026-10-04).** The same test stream logged **86 stalls - per 5s** (`FramePump stall: iteration 40–50ms` against a 17ms budget; 142 such lines in an - 8-second window) while `FramePump stats` still read 301/300 frames, 0 dropped. Nominal 60fps - with zero drops *and* repeated 3× overruns is a contradiction worth one measurement, not a - guess. Do not "fix" it from the log alone. +- **Frame pump jitter — counters now honest, cause NOT yet named (2026-10-05).** The same test + stream logged **86 stalls per 5s** (`FramePump stall: iteration 40–50ms` against a 17ms budget) + while `FramePump stats` read 301/300 frames, 0 dropped. UNIT 0 removed the reason those two + numbers could disagree (the averages were diluted by padded submits) and added the counters that + separate reader from compositor — **but no one has read a post-UNIT-0 log yet.** A creator + recording is required first. Do not "fix" the pump before that reading. - **Stale fact inside `TASKS.md` (left alone, flagged here per the Scope Lock).** The "1.0 gates" section asserts "**TASK 36 is now shipped**", but the catalog row 36 reads **☐ Queued** and `task-36-gold-pass.md` still shows items 1–6 unchecked. The line most likely means *item 2* @@ -293,9 +360,10 @@ Detail and the open 11.12 re-check: [`TASKS/task-50-publish-recording.md`](TASKS flagging rather than fixing because it is not this unit's job. - **A stale `ytLive.exe` (PID 2544) locked `bin/.../ytLive.exe`** and broke `dotnet run` with MSB3027. Killed. **If the build fails to copy `ytLive.exe`, check for a running instance first.** -- **`LayerReorderPersistenceTests.RealMouseDrag_OnTheLayerList_PersistsTheReorder` is flaky by - environment** (needs an interactive desktop session; 2 fail / 1 pass in isolation). **Run the - suite with ytLive CLOSED.** Not a regression. +- **`LayerReorderPersistenceTests.RealMouseDrag_OnTheLayerList_PersistsTheReorder` needs an + interactive desktop session.** As of 2026-10-05 it fails deterministically here (`ImgC` vs + `ImgA`), verified identical on a clean tree. **Run the suite with ytLive CLOSED** and expect this + one to fail in this shell. Not a regression — don't diagnose it from a WSL-shell failure. - **`RealAppHost.RunAsync` exists for a reason** — a frame-pump test MUST `await` inside it. A blocking wait occupies the one shared STA thread and **hangs the whole suite with no output.** Always background a `vstest` run and poll the log; a foreground piped `vstest` returns nothing @@ -308,11 +376,17 @@ Detail and the open 11.12 re-check: [`TASKS/task-50-publish-recording.md`](TASKS ## Test state -**395/395, 0 warnings, re-verified 2026-10-04 on a clean test-project build** (381 + 3 diagnostic -`FirstItem` cases + 1 window fact + 3 placement theory cases + 1 bind test, less 1 removed). -Build the **test** project, not just the app project, or VSTest runs a stale DLL. +**402/402, 0 warnings, verified 2026-10-05** (395 + 6 `FrameCadenceTests` + 1 pump-stats integration +test). Build the **test** project, not just the app project, or VSTest runs a stale DLL. **Launch exe for the creator's test run: `bin/Debug/net8.0-windows10.0.19041.0/ytLive.exe`.** +**Confirmed environmental flake, NOT a regression (re-checked 2026-10-05, and it now fails +deterministically rather than 2-of-3):** `LayerReorderPersistenceTests.RealMouseDrag_OnTheLayerList_ +PersistsTheReorder` fails with `Expected "ImgC", Actual "ImgA"` at `RealAppHost.Run` line 68. **It +fails identically on a stashed clean tree**, so it has nothing to do with UNIT 0 — it needs an +interactive desktop session this shell does not have (the mouse drag never lands). Do not chase it; +it is listed under Landmines as the known interactive-session test. + **⚠️ New flake, do not chase it as a regression:** on the *first* full-suite run, `AudioPipelineTests.MasterLimiter_CapsTheLiveMix_OnThePipe` failed with `Expected 0.891, Actual 0` — a **zero**, not a slightly-off value, which is the signature of the pipe not being @@ -335,12 +409,12 @@ whole-suite scheduling artifact of the same family as the `RealAppHost` STA land 1. **~PUSH, then the creator's 60-second Test** — see the FIXED box above. Packaging is NOT next; the creator's ruling 2026-10-04: still in active development, so ship-time work waits and queued features come first. -2. **Frame-pump jitter (queued, unquantified):** ~86 stalls per 5 s in one log window, 142 in an - 8-second window, while stats reported 301/300 frames and **zero** drops. Measure before touching - anything — the counter and the missing-frame count currently disagree, and "fixing" the pump - without knowing which number is wrong risks making it worse. **Creator to-do, not yet queued: - long-run resource/consumption/recycling benchmarking** (how long the app stays healthy under full - burden) — record only, do not start it. +2. **Webcam cadence → Unit 1 (exclusive lock + 720p).** UNIT 0 is shipped and the counters are in + `startup.log`; the reading needs one 60-second creator recording (details + the exact + read-the-log-together table in the IN FLIGHT box above). Then Unit 1: exclusive camera lock + + 720p, and Unit 2: the blinking SD badge. **Creator to-do, not yet queued: long-run + resource/consumption/recycling benchmarking** (how long the app stays healthy under full burden) — + record only, do not start it. 2. **The packaging project + `Package.appxmanifest`** (TASK 48 items 2–3) — the next code unit. Full trust, `webcam`/`microphone`, English only; Velopack deleted. Add the `DefaultRecordFolder()` comment in the same unit, and the WebView2 framework dependency + diff --git a/Helpers/FrameCadence.cs b/Helpers/FrameCadence.cs new file mode 100644 index 0000000..85ac51f --- /dev/null +++ b/Helpers/FrameCadence.cs @@ -0,0 +1,123 @@ +using System; +using System.Diagnostics; + +namespace ytLive.Helpers; + +/// +/// Arrival cadence for one window of a timestamped frame stream: how many frames +/// arrived, how far apart they were, and how many gaps were long enough to read +/// as a hitch. Fed any shared monotonic clock ( +/// values, or a device's system-relative stamps on the same clock) and read through +/// . +/// +/// Why it exists: a 60fps container makes a missing camera frame indistinguishable +/// from a padded repeat. rawvideo carries no per-frame timestamps — ffmpeg stamps +/// frames by ARRIVAL at the declared rate — so the OBS "duplicate on lag" loop +/// (FramePump) emits 300 authored frames per 5s whether the renderer managed 300 +/// composites or 84. The recorded file therefore cannot name the stage that lost +/// the frames; only these counters can. The same counter run twice tells the two +/// apart: the camera's OWN timestamps vs our ARRIVAL timestamps. Steady camera +/// stamps with ragged arrivals means WE dropped it (compositor/reader); ragged in +/// both means the camera, USB link, or driver did. +/// +public sealed class FrameCadence +{ + /// Default hitch threshold in ms. 50ms is ~1.5x a 30fps period — a + /// missed 30fps frame arrives as ~66ms — so this counts dropped frames without + /// flagging ordinary jitter (a 33ms frame can legitimately measure 35ms). + public const double DefaultHitchThresholdMs = 50; + + private readonly double _hitchThresholdMs; + private long _firstTicks; + private long _previousTicks; + private double _gapTicksTotal; + private int _frames; + private int _gaps; + private int _hitches; + private double _longestGapMs; + + public FrameCadence(double hitchThresholdMs = DefaultHitchThresholdMs) + { + if (hitchThresholdMs <= 0) + throw new ArgumentOutOfRangeException(nameof(hitchThresholdMs), hitchThresholdMs, + "Hitch threshold must be positive."); + _hitchThresholdMs = hitchThresholdMs; + } + + public int Frames => _frames; + public int Gaps => _gaps; + public int Hitches => _hitches; + public double HitchThresholdMs => _hitchThresholdMs; + + /// Records one frame at on the shared + /// monotonic clock. Monotonicity is asserted, not trusted: an out-of-order stamp + /// (a device whose clock domain differs from ours, a wrapped counter, a dropped + /// timestamp) is counted as a frame but produces no gap AND does not become the + /// reference for the next one — otherwise a single bad stamp would poison both + /// its own interval and the one after it, inventing a multi-second "hitch". + public void Add(long timestampTicks) + { + if (_frames == 0) + { + _firstTicks = timestampTicks; + _previousTicks = timestampTicks; + _frames++; + return; + } + + var delta = timestampTicks - _previousTicks; + _frames++; + if (delta <= 0) return; + var gapMs = delta * 1000d / Stopwatch.Frequency; + _gaps++; + _gapTicksTotal += gapMs; + if (gapMs > _longestGapMs) _longestGapMs = gapMs; + if (gapMs >= _hitchThresholdMs) _hitches++; + _previousTicks = timestampTicks; + } + + public FrameCadenceSummary Summarize() + { + var spanMs = _frames >= 2 + ? (_previousTicks - _firstTicks) * 1000d / Stopwatch.Frequency + : 0d; + // Rate is gaps-per-span, not frames-per-span: N frames span N-1 periods, and + // counting the extra interval would report one extra frame per window. + var rate = _gaps > 0 && spanMs > 0 ? _gaps / (spanMs / 1000d) : 0d; + return new FrameCadenceSummary( + _frames, rate, spanMs, _gaps, _hitches, _longestGapMs, + _gaps > 0 ? _gapTicksTotal / _gaps : 0d, _hitchThresholdMs); + } + + /// Starts a fresh window, keeping the configured threshold. + public void Reset() + { + _firstTicks = 0; + _previousTicks = 0; + _gapTicksTotal = 0; + _frames = 0; + _gaps = 0; + _hitches = 0; + _longestGapMs = 0; + } +} + +/// One measured cadence window — see . +public readonly record struct FrameCadenceSummary( + int Frames, + double RatePerSecond, + double SpanMs, + int Gaps, + int Hitches, + double LongestGapMs, + double AverageGapMs, + double HitchThresholdMs) +{ + /// One-line log form. Identical in shape from every producer (camera + /// source, frame pump) so two lines can be read against each other: same window + /// length, same fields, same order. + public string Describe(string label) => Frames == 0 + ? $"{label} no frames" + : $"{label} {Frames}f/{SpanMs / 1000d:F1}s {RatePerSecond:F1}/s, {Gaps} gaps, " + + $"{Hitches} >= {HitchThresholdMs:F0}ms, avg {AverageGapMs:F1}ms, longest {LongestGapMs:F1}ms"; +} \ No newline at end of file diff --git a/Helpers/index.md b/Helpers/index.md index 0de6cb2..3050aca 100644 --- a/Helpers/index.md +++ b/Helpers/index.md @@ -14,6 +14,7 @@ Cross-cutting utilities. See [`schema.md`](../schema.md) for the memory-map conv | `InverseBoolToVisibilityConverter.cs` / `NotNullToVisibilityConverter.cs` | XAML value converters for visibility bindings | | `AllTrueToVisibilityConverter.cs` | Multi-value AND-gate → Visible; gates the layer-row capture menu items on row-IsBackground + staged-scene-is-Live (TASK 25) | | `CameraConflictProbe.cs` | Best-effort diagnostic: when a camera won't start, enumerates running processes known to hold cameras (OBS, Zoom, Teams, NVIDIA Broadcast, browsers, etc.) — Windows doesn't expose "which process has this device" via any public API, so this is a suspect list, not a verdict | +| `FrameCadence.cs` | Pure arrival-cadence counter (`FrameCadence` + `FrameCadenceSummary.Describe`): frames, gaps, hitches, longest/avg gap for a window of a timestamped stream. Fed by `MediaCaptureFrameSource` (camera stamps vs our arrivals) and `FramePump` (distinct webcam Epochs that reached the output) — UNIT 0, see ai.md "Frame cadence" | Related: [`Themes/Controls.xaml`](../Themes/Controls.xaml) styles the lists this class backs; [`Models/index.md`](../Models/index.md) and diff --git a/Services/Encoder/FramePump.cs b/Services/Encoder/FramePump.cs index 6bc3573..0bc8e84 100644 --- a/Services/Encoder/FramePump.cs +++ b/Services/Encoder/FramePump.cs @@ -338,6 +338,27 @@ public sealed class FramePump : IDisposable int stalls = 0; long lastCacheHits = 0, lastCacheRenders = 0; var statsNext = DateTime.UtcNow + TimeSpan.FromSeconds(5); + + // Real-render accounting (UNIT 0, 2026-10-05): every average above was + // divided by statFrames — the number of frames SUBMITTED, which deadline + // padding always fills to the full target even when the renderer only + // produced a fifth of them. A 60fps container therefore reported "300/300 + // frames, avg render 12ms" while the source actually fed it 60 distinct + // images. Render cost is now divided by the render count, and the render + // count is reported beside the submit count — the two numbers side by side + // ARE the duplicate-on-lag ratio, which is the only honest reading of how + // much of the stream is a repeat. + int statRenders = 0; + + // Webcam cadence (UNIT 0, 2026-10-05): the pump sees each element's resolved + // frame, and VideoFrame.Epoch is the webcam ring's monotonic generation — so a + // NEW epoch means the webcam actually delivered something new between two of + // our ticks. This is the counterpart to MediaCaptureFrameSource's arrivals + // line: it measures what reached the OUTPUT, not what the camera produced. If + // the camera reports 30/s and this reports 25/s, the loss is between them. + var webcamCadence = new Helpers.FrameCadence(); + var seenWebcamEpochs = new HashSet(); + void ReportStats() { if (DateTime.UtcNow < statsNext) return; @@ -360,19 +381,23 @@ public sealed class FramePump : IDisposable } _log?.Invoke(statFrames == 0 ? "FramePump stats: NO frames produced in 5s (loop stalled?)" + (detail.Length > 0 ? " | " + detail : "") - : $"FramePump stats: {statFrames}/{target:F0} frames per 5s, " + - $"avg render {renderTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / statFrames:F1}ms " + - $"(resolve {resolveTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / statFrames:F1}), " + + : $"FramePump stats: {statFrames}/{target:F0} frames per 5s from {statRenders} renders, " + + $"avg render {renderTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / Math.Max(1, statRenders):F1}ms " + + $"(resolve {resolveTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / Math.Max(1, statRenders):F1}), " + $"avg submit {submitTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / statFrames:F1}ms, " - + $"avg wait {waitTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / statFrames:F1}ms, " + + $"avg wait {waitTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / Math.Max(1, statRenders):F1}ms, " + $"worst render {worstRender / (double)System.Diagnostics.Stopwatch.Frequency * 1000:F1}ms, " + $"worst submit {worstSubmit / (double)System.Diagnostics.Stopwatch.Frequency * 1000:F1}ms, " - + $"dropped {dropped}, stalls {stalls}" + cacheDetail + (detail.Length > 0 ? " | " + detail : "")); + + $"dropped {dropped}, stalls {stalls}{cacheDetail}, {webcamCadence.Summarize().Describe("webcam")}" + + (detail.Length > 0 ? " | " + detail : "")); worstRender = 0; worstSubmit = 0; renderTicks = submitTicks = resolveTicks = waitTicks = 0; statFrames = 0; + statRenders = 0; stalls = 0; + webcamCadence.Reset(); + seenWebcamEpochs.Clear(); statsNext = DateTime.UtcNow + TimeSpan.FromSeconds(5); } @@ -387,6 +412,8 @@ public sealed class FramePump : IDisposable var frame = _frameResolver(element); resolveSw.Stop(); resolveTicks += resolveSw.ElapsedTicks; + if (element.IsWebcam && frame != null && seenWebcamEpochs.Add(frame.Epoch)) + webcamCadence.Add(System.Diagnostics.Stopwatch.GetTimestamp()); return frame; } @@ -444,6 +471,7 @@ public sealed class FramePump : IDisposable renderSw.Stop(); renderTicks += renderSw.ElapsedTicks; if (renderSw.ElapsedTicks > worstRender) worstRender = renderSw.ElapsedTicks; + statRenders++; IFfmpegEncoder? encoder; lock (_gate) encoder = _encoder; diff --git a/Services/MediaCaptureFrameSource.cs b/Services/MediaCaptureFrameSource.cs index 16c1d66..5aa3685 100644 --- a/Services/MediaCaptureFrameSource.cs +++ b/Services/MediaCaptureFrameSource.cs @@ -50,7 +50,9 @@ public sealed class MediaCaptureFrameSource : ICameraFrameSource // Fallback ladder: prefer the camera's VideoPreview stream, but if the // reader refuses to start there (NoVideoFrameAvailable), retry against // its VideoRecord stream — some devices only deliver through it. - // (MediaCaptureSharingMode.Exclusive is not projected by the 19041 SDK.) + // (The exclusive-lock upgrade is a separate, later unit — creator + // ruling 2026-10-05; see HANDOFF.md. It is NOT blocked by an SDK gap: + // `MediaCaptureSharingMode.ExclusiveControl` IS projected in 19041.) if (!await TryStartAsync(preferVideoRecord: false)) { if (!await TryStartAsync(preferVideoRecord: true)) @@ -246,6 +248,7 @@ public sealed class MediaCaptureFrameSource : ICameraFrameSource _frameReader = reader; _lastError = null; _isFailed = false; + StartCadenceWindow(); return true; } @@ -287,6 +290,30 @@ public sealed class MediaCaptureFrameSource : ICameraFrameSource } } + private const int CadenceWindowSeconds = 5; + + private void StartCadenceWindow() + { + _arrivalCadence.Reset(); + _cameraCadence.Reset(); + _cadenceNextReportTicks = System.Diagnostics.Stopwatch.GetTimestamp() + + System.Diagnostics.Stopwatch.Frequency * CadenceWindowSeconds; + } + + /// Logs one cadence window and opens the next. The size rides along + /// because the whole question started as "the webcam is soft": a future format + /// change must be readable in the SAME line that proves the frame rate. + private void ReportCadence(int width, int height) + { + _cadenceNextReportTicks = System.Diagnostics.Stopwatch.GetTimestamp() + + System.Diagnostics.Stopwatch.Frequency * CadenceWindowSeconds; + var arrivals = _arrivalCadence.Summarize(); + var camera = _cameraCadence.Summarize(); + _arrivalCadence.Reset(); + _cameraCadence.Reset(); + AppLog.Write($"Camera cadence {width}x{height}: {arrivals.Describe("arrivals")} | {camera.Describe("camera")}"); + } + private void UnsubscribeCaptureEvents(MediaCapture capture) { capture.Failed -= OnCaptureFailed; @@ -349,12 +376,35 @@ public sealed class MediaCaptureFrameSource : ICameraFrameSource return fresh; } + // Arrival vs production cadence (UNIT 0, 2026-10-05): the recorded file cannot + // tell a dropped camera frame from a padded repeat — rawvideo stamps frames by + // ARRIVAL at the declared 60fps, so a webcam delivering 25 of its 30 frames still + // muxes a full-length file. Two counters, one reader, settle it: the camera's OWN + // SystemRelativeTime stamps (device/USB/driver cadence) against the moment the + // reader handed each frame to us (our cadence, including coalescing drops — the + // reader deliberately keeps only the LATEST frame, so falling behind is silent). + // Steady camera + ragged arrivals = we are behind. Ragged in both = the device. + private readonly FrameCadence _arrivalCadence = new(); + private readonly FrameCadence _cameraCadence = new(); + private long _cadenceNextReportTicks; + private void OnFrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args) { + var arrivedTicks = System.Diagnostics.Stopwatch.GetTimestamp(); using var frame = sender.TryAcquireLatestFrame(); var videoFrame = frame?.VideoMediaFrame?.SoftwareBitmap; if (videoFrame == null) return; + // SystemRelativeTime is QPC-domain, the same clock Stopwatch reads here, so the + // camera's own frame spacing is measurable without a second conversion. It is + // NULLABLE in the projection (null = the device supplied no stamp), so the + // production side of the comparison is then simply missing from the line — + // arrivals alone still measure what reached us. + _arrivalCadence.Add(arrivedTicks); + if (frame!.SystemRelativeTime is { } cameraStamp) + _cameraCadence.Add((long)(cameraStamp.TotalSeconds * System.Diagnostics.Stopwatch.Frequency)); + if (arrivedTicks >= _cadenceNextReportTicks) ReportCadence(videoFrame.PixelWidth, videoFrame.PixelHeight); + var bitmap = videoFrame.BitmapPixelFormat == BitmapPixelFormat.Bgra8 ? videoFrame : SoftwareBitmap.Convert(videoFrame, BitmapPixelFormat.Bgra8); diff --git a/TASKS.md b/TASKS.md index 7a9a88f..4a5c4e2 100644 --- a/TASKS.md +++ b/TASKS.md @@ -156,6 +156,19 @@ second encoder path needing a "redirect". Record+simulcast is one ffmpeg with tw adopts a solo camera as default. TASK 26's app-wide gate superseded by creator directive. The App Settings webcam selector is now **TASK 40 Unit A** (plan saved 2026-09-22). Remaining next slice (user-queued): static (+) catalog rows (Background, YouTubeEvent, quoted labels) with reason-greying. +- **🔶 Webcam cadence → resolution — IN PROGRESS, creator-queued 2026-10-05** (DHH/lipreading: the + webcam judders, ~1 hitch in 5, invisible in the output file). **UNIT 0 shipped** (counters + + `Helpers/FrameCadence.cs`; see `ai.md` → "Frame cadence"). **Needs a creator recording before the + next unit.** Remaining units, all agreed 2026-10-05, one Good Dog test each: + - **Unit 1:** `MediaCaptureSharingMode.ExclusiveControl` + MJPG ≥30fps up to 1280×720 via the + existing `SetMediaStreamPropertiesAsync` path (refused only because `SharedReadOnly`); same-camera + `SharedReadOnly` retry. **The creator reversed the ban on taking an exclusive camera lock.** + API probe already done — see HANDOFF.md, incl. the `SetFormatAsync` dead end. + - **Unit 2:** blinking amber **"SD" badge**, preview only, never in the recording/stream, when a + better format was available and not obtained. Top-right corner, or the 1:30 rim point when the + clip is round. + - ⛔ **No fallback-camera feature, ever** — *"that's bypassing the user's explicit choice."* + - ⛔ **No background removal** — re-affirmed out 2026-10-05. - **TASK 40 — App Settings round** — plan saved 2026-09-22 (`TASKS/task-40-app-settings-round.md`), four units, one Good Dog test each: **A** active web camera section under Recordings (Change… sub-menu = live enumeration incl. vcams; Tune… config dialog via `ICameraControlProbe`, live-only @@ -357,7 +370,9 @@ See [`TASKS/research-youtube-api.md`](TASKS/research-youtube-api.md) — authori These were explicitly ruled out: - **D3D11 swap compositor** (superseded by software compositor; TASK 3 item 16) -- **Background removal** (ONNX/DirectML; TASK 3 item 19) +- **Background removal** (ONNX/DirectML; TASK 3 item 19) — ⛔ **re-affirmed out 2026-10-05**: *"no - not + doing it. the note is correct and accurate."* A vcam app does it free; an in-app model is a runtime, + a GPU/driver matrix, and per-scene lighting tuning. **Never reopen without a new ruling.** - **OBS-style source expansion** (game capture, browser source, media playlist, VLC, color-key, MIDI — the minimal source set is deliberate; creators who need more have graduated to OBS) - **Multi-layout** (auto-save, single layout; multi-layout = OBS territory) - **Cross-process stream resume** (API makes it impossible; creator ruling 2026-09-01) diff --git a/ai.md b/ai.md index 29604f8..eeffc7e 100644 --- a/ai.md +++ b/ai.md @@ -483,6 +483,11 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr fails with "camera in use" (or, if init slips through, the device exposes no preview source). Fix: quit Broadcast while streaming, or pick its virtual "NVIDIA Broadcast" device from the picker and the app captures the processed feed. This is a real-device finding (Logitech `VID_046D&PID_082D`). + Correction (2026-10-05): **Windows Studio Effects is NOT a second option of this kind** — it runs + inside the physical camera's own driver, so it never appears as a selectable device. Only tools + that install a virtual camera (NVIDIA Broadcast, XSplit VCam) do. "Pick the vcam" is a user action in + the camera picker, never an automatic substitution: creator ruling 2026-10-05 — the app must not + bypass the user's explicit camera choice, so there is deliberately **no fallback-camera setting**. - **TFM:** `net8.0-windows10.0.19041.0` (app + tests) pulls the WinRT projection from the SDK reference packs — no NuGet package, no capability manifest (unpackaged desktop app works; the Windows privacy camera toggle still applies). `EnableWindowsTargeting` keeps WSL builds working. @@ -597,8 +602,56 @@ This replaces the old five-seeder cluster (`Seed{Starting,Brb,Ending,Chat}Backgr - **GPU posture:** webcam frames are CPU (GPU-agnostic; WPF hardware-presents the preview anyway). Hardware encoders (NVENC/AMF/QSV) matter for the encoder task, not capture. D3DImage GPU compositing is deferred to the encoder task. -- **Background removal = milestone 2** — ONNX Runtime + DirectML (CPU fallback), MediaPipe Selfie - Segmentation (Apache-2.0), wired into the same `VideoFrame` seam. Not part of milestone 1. +- **Background removal = PERMANENTLY OUT** (creator ruling 2026-10-05, confirmed: "no - not doing + it. the note is correct and accurate"). The line here used to read "milestone 2", which was stale — + it was never a plan, only a leftover from TASK 3's original scoping. The rationale that stands: + a vcam app (NVIDIA Broadcast, XSplit VCam) does it free, and an in-app model means a runtime, a + GPU/driver matrix, and per-scene lighting tuning — support hell for a solo dev + (`TASKS/task-36-gold-pass.md`, `TASKS/task-03-capture-pipeline.md` item 19). Don't reopen it. + +### Frame cadence — how a dropped frame became measurable (UNIT 0, 2026-10-05) + +**The problem this exists to solve.** The recorded file cannot distinguish a dropped camera frame +from a padded repeat. `-f rawvideo` carries no per-frame timestamps — ffmpeg assigns pts from frame +order at the DECLARED fps — so a webcam delivering 25 of its 30 frames still muxes a full-length, +correctly-timed 60fps file. Measured on take `ty-20261005-1001-821-0001.mp4` (85.6s, 1080p60): the +lower-right webcam region held 25.6 distinct images/s against the camera's 30fps, with 1,153 normal +33ms intervals, 281 gaps of 50–100ms, and 96 bunched 17ms gaps — roughly one hitch in five, which is +what makes a face judder while the file looks flawless. (The "webcam renders at ~18fps" figure that +used to sit in this map was WRONG: it came from a different session and counted every newly rendered +layer, not webcam arrivals. Corrected 2026-10-05.) + +**`Helpers/FrameCadence.cs`** — one pure counter, three feeds. `Add(timestamp)` on any shared +monotonic clock; `Summarize()` → frames, rate (gaps ÷ span, NOT frames ÷ span — N frames span N-1 +periods), gaps, hitches (default threshold 50ms ≈ 1.5× a 30fps period), longest and average gap; +`Describe(label)` gives every producer the same one-line shape so two lines can be read against each +other. An out-of-order stamp counts as a frame but produces no gap AND does not become the reference +for the next interval — otherwise one bad stamp invents a multi-second "hitch" twice. + +**Three measurement points, and what each separates:** + +| Line | Fed by | Answers | +|---|---|---| +| `Camera cadence WxH: arrivals … \| camera …` | `MediaCaptureFrameSource` | device/USB/driver cadence (`MediaFrame.SystemRelativeTime`, QPC domain, nullable) vs what the reader handed us. `TryAcquireLatestFrame` keeps only the LATEST frame, so falling behind is silent by design — this is where that shows | +| `webcam …` on the pump's stats line | `FramePump.TimedResolver` | distinct webcam `VideoFrame.Epoch` values that reached a composite — what actually got into the output | +| `frames per 5s from R renders` | `FramePump` | submits vs REAL composites — the duplicate-on-lag ratio, the only honest reading of how much of the stream is a repeat | + +**Why the stats-line denominators changed (the actual bug).** Every average was divided by +`statFrames`, the number of frames SUBMITTED — which deadline padding always fills to the full 60fps +target. A pump fed one webcam frame per 200ms logged "300/300 frames per 5s, avg render 12.4ms" +while producing 60 distinct composites: the render average was diluted ~5× and the loss was +invisible. Render/resolve/wait are now divided by `statRenders`; submit stays per-frame (it really +happens per submit). Regression test: +`FramePumpTests.Stats_ReportRealRendersAndDistinctWebcamFrames_NotJustPaddedSubmits`. + +**Reading the three together:** camera 30/s steady + arrivals 25/s = WE dropped it (reader or +compositor). Ragged in both = the device, the USB link, or the driver. Pump `webcam` below +`arrivals` = the compositor missed frames the reader had already delivered. + +**Still unmeasured (next unit):** the camera is stuck at the C920 default 640×480 YUY2 @30 because +all 185 `SetMediaStreamPropertiesAsync` attempts are refused under `SharedReadOnly`, so 720p MJPG is +on the table and the compositor's headroom at ~3× the bytes is unknown. Measure first, then change +the acquisition. ### Scene compositor (TASK 4 ship step 1 — shipped 2026-08-10, plan in TASKS.md) @@ -943,7 +996,7 @@ scene each tick, resolves every element to its latest frame, composites it into and paces frames into the encoder at the tier's FPS. **rawvideo is stamped by ARRIVAL:** ffmpeg assigns pts from frame order at the declared fps — supply rate = output speed. A starved producer (pump < fps) ships a time-lapse, truncated file with NO error (take-2 lesson, 2026-09-01); the pump therefore logs -`FramePump stats: n/target frames per 5s, avg render Xms (resolve R), avg submit Yms` so a slow stage names itself — and since 2026-09-04 so does the BUILD: every build gets a GUID stamped +`FramePump stats: n/target frames per 5s from R renders, avg render Xms (resolve R2), avg submit Yms` so a slow stage names itself (the `frames … from R renders` pair and the render-count denominators arrived with UNIT 0 — see "Frame cadence" below; before that every average was divided by the SUBMITTED count, which deadline padding always fills to the full target, so a pump fed one webcam frame per 200ms still logged "300/300 frames" and its render average was diluted ~an order of magnitude) — and since 2026-09-04 so does the BUILD: every build gets a GUID stamped into `Helpers/BuildStamp` (csproj `GenerateBuildStamp` target, fresh per compile — no lying incremental build), logged at startup ("Build xxxxxxxx (compiled ...)"). **Take 6 (stamp-less, ambiguous): render 35-41ms with slice 3 built or not unknown — attribution is why the stamp exists; takes 7+ are readable.** If the breakdown says resolve dominates, the suspects are pre-cached chat/config thrash or the web frame; if blit dominates, the general-path element sizes. **Pattern — everything is a constructor-injected diff --git a/ytLive.Tests/FrameCadenceTests.cs b/ytLive.Tests/FrameCadenceTests.cs new file mode 100644 index 0000000..7569917 --- /dev/null +++ b/ytLive.Tests/FrameCadenceTests.cs @@ -0,0 +1,111 @@ +using System; +using System.Diagnostics; +using Xunit; +using ytLive.Helpers; + +namespace ytLive.Tests; + +/// +/// UNIT 0 (2026-10-05): the camera and the pump could both drop frames invisibly — +/// a 60fps rawvideo container stamps frames by ARRIVAL, so a missing camera frame +/// and a padded repeat look identical in the recorded file, and the old stats line +/// counted SUBMITTED frames (always the full padded target) over an unknown number +/// of renders. These tests pin the one shared counter that makes the two measurable: +/// gaps, hitches, and the longest stall in a window of a timestamped stream. +/// +public sealed class FrameCadenceTests +{ + private static long Ms(double ms) => (long)(ms * Stopwatch.Frequency / 1000d); + + private static long AtMs(double ms) => Ms(ms); + + [Fact] + public void SteadyStream_ReportsNominalRateAndNoHitches() + { + var cadence = new FrameCadence(); + + // 30 frames at a true 33.3ms period: 29 gaps, none over the 50ms threshold. + for (var i = 0; i < 30; i++) cadence.Add(AtMs(i * (1000d / 30d))); + + var s = cadence.Summarize(); + Assert.Equal(30, s.Frames); + Assert.Equal(29, s.Gaps); + Assert.Equal(0, s.Hitches); + Assert.Equal(30d, Math.Round(s.RatePerSecond, 0)); + Assert.Equal(33.3, Math.Round(s.AverageGapMs, 1)); + Assert.InRange(s.LongestGapMs, 33.0, 33.5); + } + + [Fact] + public void DroppedFrames_CountAsHitchesAndLongestGap() + { + var cadence = new FrameCadence(); + + // 30fps with two dropped frames (66.6ms gaps) and one double-frame (0.1ms): + // the pattern the 2026-10-05 recording showed in the webcam corner. + foreach (var ms in new[] + { + 0.0, 33.3, 33.4, 66.7, 100.0, 133.3, 199.9, 233.2, 266.5, 333.1, 366.4, + }) + cadence.Add(AtMs(ms)); + + var s = cadence.Summarize(); + Assert.Equal(11, s.Frames); + Assert.Equal(10, s.Gaps); // the bunched pair still counts as an interval + Assert.Equal(2, s.Hitches); // the two dropped-frame gaps at 30fps + Assert.Equal(66.6, Math.Round(s.LongestGapMs, 1)); + } + + [Fact] + public void OutOfOrderStamp_CountsAsFrameWithoutPoisoningGaps() + { + var cadence = new FrameCadence(); + + cadence.Add(AtMs(0)); + cadence.Add(AtMs(33.3)); + cadence.Add(AtMs(10.0)); // out of order — a stamp from another clock domain + cadence.Add(AtMs(66.6)); + + var s = cadence.Summarize(); + Assert.Equal(4, s.Frames); + Assert.Equal(2, s.Gaps); // the two valid intervals only + Assert.Equal(0, s.Hitches); + Assert.Equal(33.3, Math.Round(s.LongestGapMs, 1)); + } + + [Fact] + public void Reset_StartsFreshWindowAndKeepsThreshold() + { + var cadence = new FrameCadence(hitchThresholdMs: 20); + + cadence.Add(AtMs(0)); + cadence.Add(AtMs(100)); + Assert.Equal(1, cadence.Hitches); + + cadence.Reset(); + + Assert.Equal(0, cadence.Frames); + Assert.Equal(0, cadence.Gaps); + Assert.Equal(20, cadence.Summarize().HitchThresholdMs); + Assert.Contains("no frames", cadence.Summarize().Describe("cam")); + } + + [Fact] + public void Describe_NamesTheWindowRateHitchesAndLongestGap() + { + var cadence = new FrameCadence(); + + foreach (var ms in new[] { 0.0, 16.7, 33.4, 50.1, 120.0 }) cadence.Add(AtMs(ms)); + + Assert.Equal( + "webcam 5f/0.1s 33.3/s, 4 gaps, 1 >= 50ms, avg 30.0ms, longest 69.9ms", + cadence.Summarize().Describe("webcam")); + } + + [Fact] + public void ThresholdsMustBePositive() + { + Assert.Throws(() => new FrameCadence(0)); + Assert.Throws(() => new FrameCadence(-1)); + } +} \ No newline at end of file diff --git a/ytLive.Tests/FramePumpTests.cs b/ytLive.Tests/FramePumpTests.cs index 8736cc0..96db11b 100644 --- a/ytLive.Tests/FramePumpTests.cs +++ b/ytLive.Tests/FramePumpTests.cs @@ -1,5 +1,6 @@ using System.Net; using System.Text; +using System.Text.RegularExpressions; using Xunit; using ytLive.Models; using ytLive.Services; @@ -573,6 +574,52 @@ public class FramePumpTests $"resolver ran on the starting thread ({startThread}) — the loop is back on the caller context")); } + /// The ONE integration test for UNIT 0's stats correction (2026-10-05). + /// The old line divided every average by the number of frames SUBMITTED — deadline + /// padding always fills that to the full 60fps target, so a pump fed one webcam + /// frame per 200ms still logged "300/300 frames per 5s" and the "avg render" was + /// diluted by an order of magnitude. It also never reported the webcam at all, so + /// a camera delivering 25 of its 30 frames was indistinguishable from one + /// delivering 30. Here a webcam advancing a new Epoch every ~200ms must show up + /// as FEWER distinct frames than submits, with the render count beside them. + [Fact] + public async Task Stats_ReportRealRendersAndDistinctWebcamFrames_NotJustPaddedSubmits() + { + var webcamPixels = SceneCompositorTests.Solid(64, 48, 0, 255, 0).BgraPixels; + var scene = new Scene { Name = "Live" }; + scene.Elements.Add(new WebcamSceneConfig { Name = "Webcam", IsVisible = true }); + var encoder = new FakeEncoder(); + var log = new List(); + var epoch = 0L; + using var pump = NewPump(encoder, scene: () => scene, log: log, resolve: e => + { + if (!e.IsWebcam) return null; + // One NEW webcam frame every ~200ms — slower than the 60fps container, + // exactly the case the old stats line hid completely. + Thread.Sleep(200); + epoch++; + return new VideoFrame(64, 48, webcamPixels) { Epoch = epoch }; + }); + + await pump.StartAsync(); + var deadline = DateTime.UtcNow.AddSeconds(8); + while (DateTime.UtcNow < deadline && !log.Any(l => l.Contains("webcam "))) + await Task.Delay(50); + await pump.StopAsync(); + + var stats = log.FirstOrDefault(l => l.StartsWith("FramePump stats:") && l.Contains("webcam ")); + Assert.True(stats != null, $"no cadence-bearing stats line in: {string.Join(" | ", log)}"); + + var submits = int.Parse(Regex.Match(stats!, @"stats: (\d+)/").Groups[1].Value); + var renders = int.Parse(Regex.Match(stats!, @"from (\d+) renders").Groups[1].Value); + var webcamFrames = int.Parse(Regex.Match(stats!, @"webcam (\d+)f/").Groups[1].Value); + + Assert.True(submits > 0, stats); + Assert.True(renders > 0 && renders <= submits, $"renders {renders} vs submits {submits}: {stats}"); + Assert.True(webcamFrames > 0 && webcamFrames < submits, + $"webcam frames {webcamFrames} should be far fewer than the {submits} padded submits: {stats}"); + } + private sealed class InlineSyncContext : SynchronizationContext { public override void Post(SendOrPostCallback d, object? state) => d(state);