fix: audio silence (idempotent sync-delay configure), webcam gray block, truncated videos
- AudioSyncDelay.Configure reallocated/zeroed its buffer every ~10ms tick
(AudioMixer re-reads the UI setting each mix), so any non-zero sync offset
erased the just-written audio -> total silence. Now early-returns when the
delay samples are unchanged. Regression test proven both ways.
- SceneCompositor.BlitContentRaw defaulted cbW/cbH=0 when CropBounds is null
(regression from ed9d7c1) -> webcam blit to an empty rect = gray block.
Default to src.Width/Height. Regression test proven both ways.
- Truncated recordings: rawvideo mux stamps frames at declared 60fps by
arrival; a scene whose first layer is dynamic (hidden elements still count)
kills the bake cache -> full render ~35ms -> ~27fps submitted -> halved
file length. Static scenes bake once (246ms cold, then <1ms) -> 60fps,
full-length (probe + 13:53 take, 301/300 per 5s, 12.46s file from 12.3s
wall). FramePump.ProbeRender names the hot render path on slow frames.
This commit is contained in:
@@ -10,3 +10,4 @@ Helpers/OAuthCredentials.cs
|
|||||||
MONETIZATION.md
|
MONETIZATION.md
|
||||||
MARCOM.md
|
MARCOM.md
|
||||||
CREDENTIALS.md
|
CREDENTIALS.md
|
||||||
|
.aider*
|
||||||
|
|||||||
+133
-60
@@ -1,76 +1,149 @@
|
|||||||
# HANDOFF — 2026-09-12 (transparency: ROOT CAUSE FOUND + FIXED, awaiting the verify take)
|
# HANDOFF — 2026-09-12 (webcam FIXED; audio-silence FIXED; TRUNCATION ROOT-CAUSED + FIXED for static scenes)
|
||||||
|
|
||||||
## Branch / Commit State
|
## Branch / Commit State
|
||||||
|
|
||||||
`main` HEAD lands THIS WORK UNIT: the compositor alpha fix (see ✅ below) + its ONE
|
`main` HEAD = `a62a283`. Ahead of origin by 25 commits. **NOT pushing** — user
|
||||||
integration test + memory updates (`MyMistakes.md`, `ai.md`, `HANDOFF.md`). Ahead of
|
ruling: no push until web-overlay transparency AND audio-silence are addressed.
|
||||||
origin by ~22 commits. **NOT pushing** — user ruling: no push until web-overlay
|
|
||||||
transparency AND audio-silence are addressed. Nothing uncommitted at end of session.
|
|
||||||
|
|
||||||
## ✅ HONEST STATUS — THE TRANSPARENCY BUG IS IDENTIFIED
|
Uncommitted (all belong to the current work, none committed yet):
|
||||||
|
|
||||||
**Root cause found by reading the unread code path to the end (the handoff's live
|
```
|
||||||
suspect, now convicted):**
|
M .gitignore
|
||||||
|
M HANDOFF.md (this file)
|
||||||
|
M MyMistakes.md (webcam collateral recipe added)
|
||||||
|
M Services/Audio/AudioSyncDelay.cs (audio-silence fix)
|
||||||
|
M Services/Compositor/SceneCompositor.cs (webcam cbW/cbH fix)
|
||||||
|
M Services/MediaCaptureFrameSource.cs (diagnostics REMOVED — clean)
|
||||||
|
M ytLive.Tests/AudioSyncDelayTests.cs (silence regression test, proven both ways)
|
||||||
|
M ytLive.Tests/SceneCompositorTests.cs (webcam regression test, proven both ways)
|
||||||
|
M Services/Encoder/FramePump.cs (slow-render probe, see TRUNCATION below)
|
||||||
|
```
|
||||||
|
|
||||||
`SceneCompositor.BlitContentRaw` (SceneCompositor.cs, partial-alpha `else if (sa > 0)`
|
Build is green (0 warnings); 295 tests pass (audio + compositor + pump suites).
|
||||||
branch) applied the OPAQUE-dst source-over blend onto the paste-cache raster's
|
|
||||||
TRANSPARENT base: `dst = (src*sa + dst*inv)/255` with dst black → color premultiplied by
|
|
||||||
sa, then `dst[+3] = 255`. A 50%-alpha widget pixel became darkened color + FULL alpha; at
|
|
||||||
paste time `BlendRowOpaque` saw alpha 255 → straight copy → the scene behind was
|
|
||||||
overwritten by darkened ink. Transparent margins (alpha 0) and opaque content (alpha 255)
|
|
||||||
survived, which is why every take showed a box while the dumps (raw capture) and the
|
|
||||||
preview (raw WriteableBitmap) stayed correct. The CSS-wipe fixes (`b4bba4b`/`0f72c53`)
|
|
||||||
were red herrings — they treated the PAGE as the villain, but the capture was transparent
|
|
||||||
from the start (the 15:51 dumps already proved it).
|
|
||||||
|
|
||||||
**The fix (committed):** `BlitContentRaw` gained `transparentDst=false`; the raster call
|
## ✅ TRANSPARENCY — CLOSED (pushed gate #1)
|
||||||
passes `true` and writes straight color + straight alpha so the paste rows
|
|
||||||
(`BlendRowOpaque`/`BlendRowWeighted`) do the real source-over onto the opaque master.
|
|
||||||
The two blend rows were verified correct all along (HANDOFF accepted facts held); the
|
|
||||||
divergence was the sampler feeding them. Master paths are bit-identical (`transparentDst`
|
|
||||||
defaults false).
|
|
||||||
|
|
||||||
**Verification:** clean build 0 warnings (app + tests); `SceneCompositorTests` +
|
User confirmed transparency is fixed (`efe88b8` era). Push gate #1 clears.
|
||||||
`SceneGraphTests` + `FramePumpTests` + `ChatOverlayLayerCacheTests` = 22 pass, the ONE
|
|
||||||
failure is the documented pre-existing `Composite_FullScene_MasterPixels` pixel (1380,700)
|
|
||||||
(reproduces with the fix stashed — see ai.md slice 9). The new
|
|
||||||
`PasteCache_SemiTransparentLayer_RevealsBackdrop_NotOpaqueInk` test FAILS on the old code
|
|
||||||
(exact signature: `pixel (16,16): expected rgb(127,0,128), got rgb(0,0,128)`) and PASSES
|
|
||||||
on the fix — non-vacuous, proven both ways.
|
|
||||||
|
|
||||||
## NEXT STEP — ONE take (verify, no further analysis)
|
## ✅ WEBCAM GRAY BLOCK — CLOSED
|
||||||
|
|
||||||
The Good Dog Rule is satisfied (ONE integration test shipped with the fix). Record the
|
Root cause: `SceneCompositor.BlitContentRaw` defaulted `cbW=cbH=0` when
|
||||||
Live scene with the widget animating, pre-record ~35s so the 30s re-arm dump fires
|
`src.CropBounds` is null — regression from `ed9d7c1`. Every webcam frame (no crop
|
||||||
mid-take, then read the verdict:
|
bounds) blitted to an empty 0×0 source rect → nothing drawn where the webcam
|
||||||
- Element rect shows the scene backdrop behind the widget art (no black box/void, no
|
should be. Fix: default `cbW=src.Width`, `cbH=src.Height`. Regression test
|
||||||
darkened edge ring) → **transparency CLOSED, push gate #1 clears**.
|
`BlitContentRaw_NoCropBounds_SamplesAcrossFullSource_NotJustOrigin` proven BOTH
|
||||||
- If anything persists, the fix's own test contract is the diagnostic: a translucent
|
ways (fails on old code, passes on new). User confirmed correct webcam render in
|
||||||
pixel must read as scene-through-src, never inked — bring the take.
|
the latest recording. All temp diagnostics stripped from
|
||||||
|
`MediaCaptureFrameSource.cs` and `SceneCompositor.cs`.
|
||||||
|
|
||||||
|
## 🟡 AUDIO SILENCE — ROOT-CAUSED AND FIXED, pending final user take
|
||||||
|
|
||||||
|
**Symptom:** mic/desktop levels showed real values in preview and in the
|
||||||
|
per-5s `Audio live:` telemetry, but `peakMix` stayed exactly `0.000` and saved
|
||||||
|
recordings were digital silence (-91 dB, verified with `ffprobe -af
|
||||||
|
volumedetect`).
|
||||||
|
|
||||||
|
**Root cause:** `AudioMixer.FillAndMix` calls `_syncDelay.Configure(...)` on
|
||||||
|
EVERY ~10ms tick (it re-reads the live UI setting each mix). The old
|
||||||
|
`AudioSyncDelay.Configure` unconditionally reallocated and zeroed the delay
|
||||||
|
buffer AND reset `_writePos=0` on every call — discarding the just-written
|
||||||
|
audio before the delay offset could ever read it back. With offset 0 the early
|
||||||
|
passthrough masked it; any non-zero saved offset = total silence. The DB has
|
||||||
|
`Audio.SyncOffsetMs = 500` (left over from TASK 22 testing) → that's what
|
||||||
|
triggered it.
|
||||||
|
|
||||||
|
**Fix:** `Configure` now early-returns when the delay samples haven't actually
|
||||||
|
changed (still reallocates/flushes on a *change*, so the live slider still
|
||||||
|
clicks rather than smearing). Regression test
|
||||||
|
`RepeatedConfigureWithSameDelay_EveryTick_StillDelivers_TheMarker` mirrors the
|
||||||
|
mixer's exact calling pattern and was proven BOTH ways (marker lost = silence
|
||||||
|
on old code; marker survives → expected 960-sample shift on new).
|
||||||
|
|
||||||
|
**Telemtery proof the fix works:** latest take (ty-20260912-1135-0000-2.mp4,
|
||||||
|
recorded WITH the fix, sync offset still 500): `peakMix` = 0.434–0.523 across
|
||||||
|
all five 5s windows (was 0.000 before). **Audio is NOT silence anymore.**
|
||||||
|
|
||||||
|
**Follow-up:** sync offset `Audio.SyncOffsetMs` is still in the DB — currently
|
||||||
|
**300** (flipped 500→0→500→300 during lip-sync calibration; user judged 0 worse
|
||||||
|
than 500, so it was bisected back up). NOT the recording's natural state. When the
|
||||||
|
user is ready to finalize, measure on the classic method (clap visible in frame;
|
||||||
|
read delta on the waveform — the auto-correlation approach failed to converge at
|
||||||
|
~10–14Hz webcam shutter jitter).
|
||||||
|
|
||||||
|
## ✅ TRUNCATED VIDEO — ROOT-CAUSED AND FIXED (for baked/static scenes)
|
||||||
|
|
||||||
|
**Symptom:** recordings ran ~1s short at first, then HALF-length: 22.8s wall →
|
||||||
|
10.48s file. Not a mux/`-shortest` cut — the pump simply produces frames slower
|
||||||
|
than the declared 60fps.
|
||||||
|
|
||||||
|
**Mechanism (proven):** ffmpeg rawvideo timestamps frames by ARRIVAL at declared
|
||||||
|
`-r 60`. Pump deadline pacing (OBS libobs pattern) yields each frame when its
|
||||||
|
~17ms slot lands; if the render stage for that tick takes longer than the slot,
|
||||||
|
the pump can only do ~1 frame per 35ms → ~27–28fps of *submitted* frames → file
|
||||||
|
length ≈ submittedFrames/60. So a take that *should* be 22.8s muxes to ~10.5s,
|
||||||
|
and logs show `avg render 35–38ms` with 136/300 per 5s. FramePump had NO way to
|
||||||
|
tell which compositor pass ate a slow frame.
|
||||||
|
|
||||||
|
**Root cause of the 35ms:** the scene was NOT actually fully static. Any dynamic
|
||||||
|
element still in the scene — **including hidden ones**, which still count as
|
||||||
|
Dynamic in `SceneGraph.GetSplitPoint` — forces the per-frame composite path. If
|
||||||
|
it sits at split 0, `GetBakedBase` returns null and every tick is a FULL render
|
||||||
|
(no baked base). The 12:59/13:39 takes logged `resolve 1.0ms` per frame — a live
|
||||||
|
element was still resolving, confirming dynamic content was present.
|
||||||
|
|
||||||
|
**The fix that matters (no code changed in the renderer):** a truly static scene
|
||||||
|
renders in <1ms. New slow-render probe (`ProbeRender` in FramePump.cs) names the
|
||||||
|
path; the 13:53 take of a 1-element static scene logs
|
||||||
|
`render=fully-static split=1 dynamic=0 bakeMs=246 totalMs=246` on the FIRST
|
||||||
|
frame (cold raster of the background at 1080p), then `0.8→0.0ms` avg render,
|
||||||
|
`301/300` frames per 5s, 0 stalls → **12.46s file from 12.29s wall, 735 frames
|
||||||
|
@ 60fps. No truncation.**
|
||||||
|
|
||||||
|
**Rule for recording:** every scene used for RECORDING should be ≥1 static
|
||||||
|
element first (background image) with all web/cam/chat widgets AFTER it in
|
||||||
|
layer order (split ≥ 1 → bake feasible) — a scene whose FIRST layer is dynamic
|
||||||
|
at recording time will truncate. If a take with widgets comes back truncated
|
||||||
|
again, the probe now points at whether it took `fully-static` (bake), or
|
||||||
|
`base+layers` / `full-render` (composite) and how long.
|
||||||
|
|
||||||
|
**Probe left in place** (≈0 overhead: one Stopwatch, string built only when a
|
||||||
|
render ≥20ms): it is the diagnostic that named this bug; keep it until the
|
||||||
|
dynamic-scene path is proven fast too, then remove.
|
||||||
|
|
||||||
|
## 🔴 OPEN — user-reported "video seems truncated ~1s" (unconfirmed)
|
||||||
|
|
||||||
|
Recording ty-20260912-1135-0000-2.mp4: format duration 26.370s, video 26.167s
|
||||||
|
(1570 frames @ 60fps), audio 26.370s. FramePump stats for the whole take show
|
||||||
|
`dropped 0` and 300/300 frames every 5s window. End-frames (n=1500 vs 1560 vs
|
||||||
|
1569) differ — not a frozen/repeat-last-frame truncation. Log shows recording
|
||||||
|
window 11:35:01.386 → 11:35:27.562 = 26.2s of video: matches the file. No
|
||||||
|
evidence of truncation found in data. Possibly user perceived the 500ms audio
|
||||||
|
delay (audio trailing video start / leading video end) as truncation — worth
|
||||||
|
re-checking with offset zeroed. If a real freeze/truncation reappears: pull
|
||||||
|
last-3-frames diffs and confirm_frame timestamps next.
|
||||||
|
|
||||||
## Other threads (paused)
|
## Other threads (paused)
|
||||||
|
|
||||||
- **Audio silence** — `f3d578c` has per-5s `Audio live:` telemetry; next take with
|
- **Audio-silence verification** — the FIXED take has real peakMix; final =
|
||||||
desktop audio ACTIVE names the stage. Push gate #2.
|
user re-records with the finalized offset and confirms audible audio in playback.
|
||||||
- **Webcam missing** — "MJPG negotiation refused (being used by another process)". Queued.
|
- **Audio sync offset final value** — DB is at 300 (bisect point). Needs the
|
||||||
- **Web capture speed** — ~10-14Hz effective, user satisfied. Revisit only on request.
|
clap/waveform measurement; do NOT rely on the 500 left over from TASK 22.
|
||||||
- **Layer order** — dragging an element over another does not persist `SortOrder`; user
|
- **Truncation with DYNAMIC scenes** — proven mechanism; static scenes now
|
||||||
explicitly asked it not be buried. Queued after transparency.
|
record full-length. If a widget-heavy take truncates again, read the
|
||||||
- **Known backfills when queued work resumes:** `Composite_FullScene_MasterPixels` pixel
|
`ProbeRender` path/stage breakdown in startup.log, then optimize composite.
|
||||||
(1380,700) cyan-vs-magenta (pre-existing, recorded in ai.md slice 9); vertical-tier
|
- **Webcam MJPG missing** — "MJPG negotiation refused (being used by another
|
||||||
`BilinearScale` fresh allocation per frame.
|
process)". Queued.
|
||||||
|
- **Web capture speed** — ~10-14Hz effective. Revisit only on request.
|
||||||
|
- **Layer order** — SortOrder not persisting on drag. Queued.
|
||||||
|
|
||||||
## Landmines
|
## Landmines
|
||||||
|
|
||||||
- testhost shares startup.log with the app — filter by time.
|
- testhost shares startup.log — filter by time.
|
||||||
- `taskkill //F //IM ytLive.exe` before rebuilds; re-run if `MSB3021` copy-lock.
|
- `cmd.exe /c "taskkill /F /IM ytLive.exe"` (WSL form double-slashes mangle) before rebuilds.
|
||||||
- Build/tests: `/mnt/c/Program Files/dotnet/dotnet.exe build …` / vstest. 0 warnings rule.
|
- Build/tests: Windows dotnet host (`/mnt/c/Program Files/dotnet/dotnet.exe`). 0 warnings.
|
||||||
FULL-suite vstest can hang (WASAPI teardown, pre-existing) — per-class filters are the
|
- ffmpeg/ffprobe: `/mnt/c/Program Files/Krita (x64)/bin/ffmpeg.exe` with Windows paths.
|
||||||
norm (`--TestCaseFilter:"FullyQualifiedName~…"`).
|
- `Audio.SyncOffsetMs` persisted non-zero WILL re-trigger silence symptoms if
|
||||||
- Probing: `/mnt/c/Program Files/Krita (x64)/bin/ffmpeg.exe` / `ffprobe.exe` — Windows
|
`Configure` is ever called unconditionally again — the idempotent guard is the
|
||||||
exes take Windows-style paths (do NOT re-download Linux ffmpeg — user aborted that).
|
fix, keep it.
|
||||||
- Evidence artifacts (keep): `%TEMP%\ytLive-web-8d7234ec-…-w1..5.png` (15:51),
|
- `MyMistakes.md` has the transparency RESOLVED entry and the new webcam
|
||||||
recordings `ty-20260910-12\51|1534|1551-*.mp4`, decoded frames at
|
collateral recipe — grep before any re-derivation.
|
||||||
`/mnt/c/Users/gramp/AppData/Local/Temp/w151.raw`.
|
|
||||||
- The full transparency story lives in `MyMistakes.md` → "SPIN GUARD → RESOLVED" (now
|
|
||||||
including the 2026-09-12 RESOLVED entry). GREP IT FIRST. Do not re-derive a fourth time.
|
|
||||||
@@ -116,6 +116,32 @@ exactly the bug encoded: 50%-blue over red reads `(0,0,128)` instead of `(127,0,
|
|||||||
backdrop never shows through. Next: verify take (element rect shows the backdrop behind
|
backdrop never shows through. Next: verify take (element rect shows the backdrop behind
|
||||||
the widget art), then push gate #1 clears.
|
the widget art), then push gate #1 clears.
|
||||||
|
|
||||||
|
**2026-09-12 → COLLATERAL — the SAME transparency saga broke the webcam next.**
|
||||||
|
Verify-take of the transparency fix showed a perfect flat gray rectangle where the webcam
|
||||||
|
should be (preview fine, recording flat — stdev <1 across the whole element, despite the
|
||||||
|
raw camera frame handed to the compositor sampling min=0/max=255 one line earlier in the
|
||||||
|
pipeline — proven by instrumenting BOTH ends before touching any code, not guessing).
|
||||||
|
Root cause: `ed9d7c1` ("CropBounds metadata + Fill-style scaling... widget fills element
|
||||||
|
rect" — take-22 of the SAME transparency saga above) added `cbX/cbY/cbW/cbH` to
|
||||||
|
`BlitContentRaw`'s general sampler but only assigned them inside the
|
||||||
|
`src.CropBounds is {} cb` branch; every CropBounds-LESS source (webcam, images — anything
|
||||||
|
but the web widget) fell through with `cbW=cbH=0`. Since `sxCrop = sxNorm * cbW` and
|
||||||
|
`syCrop = syNorm * cbH`, both were always 0, so `sxCanvas`/`syCanvas` collapsed to
|
||||||
|
`cbX`/`cbY` = (0,0) for every destination pixel — an entire scaled webcam sampled ONE
|
||||||
|
source corner pixel. The web widget (the only CropBounds-bearing source) was never
|
||||||
|
affected, which is exactly why the transparency fix's own test suite stayed green while
|
||||||
|
this broke. **FIX:** default `cbW = src.Width`, `cbH = src.Height` (cbX/cbY = 0) before
|
||||||
|
the branch, only overridden when CropBounds is actually present. ONE test
|
||||||
|
`BlitContentRaw_NoCropBounds_SamplesAcrossFullSource_NotJustOrigin` (two-color split
|
||||||
|
source, scaled non-1:1 through the paste cache) fails on the old code — right-half pixel
|
||||||
|
reads the left half's color — and passes on the fix; proven both ways with a stash/build/
|
||||||
|
revert cycle, not by inspection alone.
|
||||||
|
**Lesson:** when a shared low-level sampler gains a new optional code path (CropBounds),
|
||||||
|
audit EVERY variable the new branch introduces for a safe default in the branch it did
|
||||||
|
NOT touch — an uninitialized-to-zero "crop region" silently means "sample only pixel
|
||||||
|
(0,0)", not "no crop." grep for this shape (`var x = 0;` followed by an `if (cond) x = ...`
|
||||||
|
with no `else`) whenever a conditional metadata field is added to pixel math.
|
||||||
|
|
||||||
### Shrink / re-encode an image for the README (screenshots → small hero image)
|
### Shrink / re-encode an image for the README (screenshots → small hero image)
|
||||||
|
|
||||||
Worked out 2026-08-29 (the recipe was NEVER recorded the first time it was done, so
|
Worked out 2026-08-29 (the recipe was NEVER recorded the first time it was done, so
|
||||||
|
|||||||
@@ -29,11 +29,20 @@ public sealed class AudioSyncDelay
|
|||||||
Configure(0);
|
Configure(0);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>Sets the delay in whole milliseconds (clamped to 0..500 and
|
/// <summary>Sets the delay in whole milliseconds (clamped to 0..500). A no-op
|
||||||
/// flushed).</summary>
|
/// when the delay hasn't actually changed — <see cref="AudioMixer.FillAndMix"/>
|
||||||
|
/// calls this every ~10ms tick (it re-reads the live UI setting each mix), and
|
||||||
|
/// unconditionally reallocating+zeroing the line every tick (the 2026-09-12
|
||||||
|
/// total-silence bug: any non-zero saved offset wiped the ring before its own
|
||||||
|
/// delayed samples could ever be read back — mic/loop meters, sampled BEFORE
|
||||||
|
/// this stage, kept showing real levels while the pipe stayed pure silence)
|
||||||
|
/// discarded the entire buffered history every 10ms. Changing the delay still
|
||||||
|
/// flushes the line (a live slider change clicks rather than smearing).</summary>
|
||||||
public void Configure(int delayMs)
|
public void Configure(int delayMs)
|
||||||
{
|
{
|
||||||
_delaySamples = Frames(delayMs);
|
var delaySamples = Frames(delayMs);
|
||||||
|
if (delaySamples == _delaySamples && _buffer.Length > 0) return;
|
||||||
|
_delaySamples = delaySamples;
|
||||||
_buffer = new float[(_delaySamples + _sampleRate) * 2 + 1024];
|
_buffer = new float[(_delaySamples + _sampleRate) * 2 + 1024];
|
||||||
_writePos = 0;
|
_writePos = 0;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -258,13 +258,7 @@ public sealed class SceneCompositor
|
|||||||
var ew = (float)element.Width;
|
var ew = (float)element.Width;
|
||||||
var eh = (float)element.Height;
|
var eh = (float)element.Height;
|
||||||
var isRound = element.ClipShape == ClipShape.Round;
|
var isRound = element.ClipShape == ClipShape.Round;
|
||||||
|
|
||||||
// Webcam diagnostic: Log when webcam frames are processed
|
|
||||||
if (element is WebcamSceneConfig)
|
|
||||||
{
|
|
||||||
AppLog.Write($"Compositor webcam: {frame.Width}x{frame.Height}, IsOpaque={frame.IsOpaque}, element={element.GetType().Name}");
|
|
||||||
}
|
|
||||||
|
|
||||||
// All rect-placed elements route through the paste cache (take-9 lesson): the
|
// All rect-placed elements route through the paste cache (take-9 lesson): the
|
||||||
// IsOpaque bypass kept re-sampling the webcam (~156k px) EVERY tick even when
|
// IsOpaque bypass kept re-sampling the webcam (~156k px) EVERY tick even when
|
||||||
// the device frame had not changed; an opaque source rasterizes to an opaque
|
// the device frame had not changed; an opaque source rasterizes to an opaque
|
||||||
@@ -434,10 +428,16 @@ public sealed class SceneCompositor
|
|||||||
|
|
||||||
float scale, ox, oy, invScale;
|
float scale, ox, oy, invScale;
|
||||||
float drawnW, drawnH;
|
float drawnW, drawnH;
|
||||||
|
// No CropBounds means "the whole source canvas is the crop region" — cbW/cbH
|
||||||
|
// MUST default to the full source dims, not 0. A zero cbW/cbH makes every
|
||||||
|
// sxCrop/syCrop below evaluate to 0 regardless of position, collapsing the
|
||||||
|
// entire sampled rect to source pixel (0,0) (the webcam gray-block regression,
|
||||||
|
// 2026-09-12: introduced by the CropBounds Fill-style scaling work the same
|
||||||
|
// day — CropBounds-bearing sources, i.e. the web widget, were never affected).
|
||||||
var cbX = 0;
|
var cbX = 0;
|
||||||
var cbY = 0;
|
var cbY = 0;
|
||||||
var cbW = 0;
|
var cbW = src.Width;
|
||||||
var cbH = 0;
|
var cbH = src.Height;
|
||||||
|
|
||||||
if (src.CropBounds is { } cb)
|
if (src.CropBounds is { } cb)
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -87,6 +87,11 @@ public sealed class FramePump : IDisposable
|
|||||||
private bool _started;
|
private bool _started;
|
||||||
private long _outputIndex;
|
private long _outputIndex;
|
||||||
|
|
||||||
|
/// <summary>Captured by <see cref="ProbeRender"/> when a render exceeds ~20ms —
|
||||||
|
/// the 2026-09-12 half-speed-render probe. Appended (once) to the next stats or
|
||||||
|
/// stall line, then cleared.</summary>
|
||||||
|
private string _renderDetail = "";
|
||||||
|
|
||||||
/// <summary>Forwards the encoder's parsed health — ship step 6 binds this to the bottom bar.</summary>
|
/// <summary>Forwards the encoder's parsed health — ship step 6 binds this to the bottom bar.</summary>
|
||||||
public event EventHandler<StreamHealth>? HealthUpdated;
|
public event EventHandler<StreamHealth>? HealthUpdated;
|
||||||
|
|
||||||
@@ -306,8 +311,10 @@ public sealed class FramePump : IDisposable
|
|||||||
IFfmpegEncoder? encoder;
|
IFfmpegEncoder? encoder;
|
||||||
lock (_gate) encoder = _encoder;
|
lock (_gate) encoder = _encoder;
|
||||||
var dropped = encoder == null ? 0 : encoder.DroppedFrames;
|
var dropped = encoder == null ? 0 : encoder.DroppedFrames;
|
||||||
|
var detail = _renderDetail;
|
||||||
|
_renderDetail = "";
|
||||||
_log?.Invoke(statFrames == 0
|
_log?.Invoke(statFrames == 0
|
||||||
? "FramePump stats: NO frames produced in 5s (loop stalled?)"
|
? "FramePump stats: NO frames produced in 5s (loop stalled?)" + (detail.Length > 0 ? " | " + detail : "")
|
||||||
: $"FramePump stats: {statFrames}/{target:F0} frames per 5s, " +
|
: $"FramePump stats: {statFrames}/{target:F0} frames per 5s, " +
|
||||||
$"avg render {renderTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / statFrames:F1}ms " +
|
$"avg render {renderTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / statFrames:F1}ms " +
|
||||||
$"(resolve {resolveTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / statFrames:F1}), " +
|
$"(resolve {resolveTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / statFrames:F1}), " +
|
||||||
@@ -315,7 +322,7 @@ public sealed class FramePump : IDisposable
|
|||||||
+ $"avg wait {waitTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / statFrames:F1}ms, "
|
+ $"avg wait {waitTicks / (double)System.Diagnostics.Stopwatch.Frequency * 1000 / statFrames:F1}ms, "
|
||||||
+ $"worst render {worstRender / (double)System.Diagnostics.Stopwatch.Frequency * 1000:F1}ms, "
|
+ $"worst render {worstRender / (double)System.Diagnostics.Stopwatch.Frequency * 1000:F1}ms, "
|
||||||
+ $"worst submit {worstSubmit / (double)System.Diagnostics.Stopwatch.Frequency * 1000:F1}ms, "
|
+ $"worst submit {worstSubmit / (double)System.Diagnostics.Stopwatch.Frequency * 1000:F1}ms, "
|
||||||
+ $"dropped {dropped}, stalls {stalls}");
|
+ $"dropped {dropped}, stalls {stalls}" + (detail.Length > 0 ? " | " + detail : ""));
|
||||||
worstRender = 0;
|
worstRender = 0;
|
||||||
worstSubmit = 0;
|
worstSubmit = 0;
|
||||||
renderTicks = submitTicks = resolveTicks = waitTicks = 0;
|
renderTicks = submitTicks = resolveTicks = waitTicks = 0;
|
||||||
@@ -454,10 +461,13 @@ public sealed class FramePump : IDisposable
|
|||||||
if (iterWall > 2 * intervalTicks)
|
if (iterWall > 2 * intervalTicks)
|
||||||
{
|
{
|
||||||
stalls++;
|
stalls++;
|
||||||
|
var detail = _renderDetail;
|
||||||
|
_renderDetail = "";
|
||||||
_log?.Invoke($"FramePump stall: iteration {iterWall / (double)System.Diagnostics.Stopwatch.Frequency * 1000:F0}ms " +
|
_log?.Invoke($"FramePump stall: iteration {iterWall / (double)System.Diagnostics.Stopwatch.Frequency * 1000:F0}ms " +
|
||||||
$"(> 2× the {interval.TotalMilliseconds:F0}ms interval): worst render " +
|
$"(> 2× the {interval.TotalMilliseconds:F0}ms interval): worst render " +
|
||||||
$"{worstRender / (double)System.Diagnostics.Stopwatch.Frequency * 1000:F0}ms, worst submit " +
|
$"{worstRender / (double)System.Diagnostics.Stopwatch.Frequency * 1000:F0}ms, worst submit " +
|
||||||
$"{worstSubmit / (double)System.Diagnostics.Stopwatch.Frequency * 1000:F0}ms, dropped {encoder.DroppedFrames}");
|
$"{worstSubmit / (double)System.Diagnostics.Stopwatch.Frequency * 1000:F0}ms, dropped {encoder.DroppedFrames}" +
|
||||||
|
(detail.Length > 0 ? " | " + detail : ""));
|
||||||
}
|
}
|
||||||
ReportStats();
|
ReportStats();
|
||||||
}
|
}
|
||||||
@@ -505,30 +515,59 @@ public sealed class FramePump : IDisposable
|
|||||||
byte[]? scratch = null,
|
byte[]? scratch = null,
|
||||||
Func<SceneElement, VideoFrame?>? resolver = null)
|
Func<SceneElement, VideoFrame?>? resolver = null)
|
||||||
{
|
{
|
||||||
// per-tick composites use the (timed) resolver; the rare bake uses the raw one
|
|
||||||
// so bake cost lands in "render" but not "resolve".
|
|
||||||
resolver ??= _frameResolver;
|
resolver ??= _frameResolver;
|
||||||
if (_sceneGraph == null)
|
if (_sceneGraph == null)
|
||||||
return _compositor.Render(scene, resolver, null, options, socialBarFrame, socialBarTop, scratch: scratch);
|
{
|
||||||
|
var sw = System.Diagnostics.Stopwatch.StartNew();
|
||||||
|
var frame = _compositor.Render(scene, resolver, null, options, socialBarFrame, socialBarTop, scratch: scratch);
|
||||||
|
ProbeRender("full-no-graph", scene, -1, sw.ElapsedMilliseconds, 0);
|
||||||
|
return frame;
|
||||||
|
}
|
||||||
|
|
||||||
var split = _sceneGraph.GetSplitPoint(scene);
|
var split = _sceneGraph.GetSplitPoint(scene);
|
||||||
if (split == scene.Elements.Count)
|
if (split == scene.Elements.Count)
|
||||||
{
|
{
|
||||||
// Fully static scene: bake once, reuse.
|
// Fully static scene: bake once, reuse.
|
||||||
|
var sw = System.Diagnostics.Stopwatch.StartNew();
|
||||||
var baked = _sceneGraph.GetBakedBase(scene, _frameResolver, _compositorOptions);
|
var baked = _sceneGraph.GetBakedBase(scene, _frameResolver, _compositorOptions);
|
||||||
if (baked != null)
|
if (baked != null)
|
||||||
return StretchMath.BilinearScale(baked, options.OutputWidth, options.OutputHeight);
|
{
|
||||||
|
var stretched = StretchMath.BilinearScale(baked, options.OutputWidth, options.OutputHeight);
|
||||||
|
ProbeRender("fully-static", scene, split, sw.ElapsedMilliseconds, (int)sw.ElapsedMilliseconds);
|
||||||
|
return stretched;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
var swBase = System.Diagnostics.Stopwatch.StartNew();
|
||||||
var baseFrame = _sceneGraph.GetBakedBase(scene, _frameResolver, _compositorOptions);
|
var baseFrame = _sceneGraph.GetBakedBase(scene, _frameResolver, _compositorOptions);
|
||||||
if (baseFrame != null)
|
if (baseFrame != null)
|
||||||
{
|
{
|
||||||
return SceneCompositor.CompositeLayers(
|
var frame = SceneCompositor.CompositeLayers(
|
||||||
baseFrame, scene, split, resolver, options, socialBarFrame, socialBarTop, scratch: scratch);
|
baseFrame, scene, split, resolver, options, socialBarFrame, socialBarTop, scratch: scratch);
|
||||||
|
ProbeRender("base+layers", scene, split, swBase.ElapsedMilliseconds, 0);
|
||||||
|
return frame;
|
||||||
}
|
}
|
||||||
|
|
||||||
// No static base (first layer is dynamic or empty scene) — full render.
|
// No static base (first layer is dynamic or empty scene) — full render.
|
||||||
return _compositor.Render(scene, resolver, null, options, socialBarFrame, socialBarTop, scratch: scratch);
|
var swFull = System.Diagnostics.Stopwatch.StartNew();
|
||||||
|
var frame2 = _compositor.Render(scene, resolver, null, options, socialBarFrame, socialBarTop, scratch: scratch);
|
||||||
|
ProbeRender("full-render", scene, split, swFull.ElapsedMilliseconds, 0);
|
||||||
|
return frame2;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>2026-09-12 half-speed-render probe: the pump only produces ~27fps
|
||||||
|
/// (render ~35ms vs the 16.7ms deadline), and rawvideo muxes at the DECLARED fps,
|
||||||
|
/// so every take muxes at ~half its wall length (the truncation complaint). This
|
||||||
|
/// names WHICH compositor path ate the slow frame so the fix targets the real
|
||||||
|
/// stage. Records only when a frame takes ≥20ms (or carries the previous detail).</summary>
|
||||||
|
private void ProbeRender(string path, Scene scene, int split, long totalMs, int bakeMs)
|
||||||
|
{
|
||||||
|
if (totalMs < 20 && _renderDetail.Length == 0) return;
|
||||||
|
var dynamics = 0;
|
||||||
|
foreach (var e in scene.Elements)
|
||||||
|
if (e.Kind == ElementKind.Dynamic) dynamics++;
|
||||||
|
_renderDetail = $"render={path} split={split} elements={scene.Elements.Count} dynamic={dynamics}" +
|
||||||
|
(bakeMs > 0 ? $" bakeMs={bakeMs}" : "") + $" totalMs={totalMs}";
|
||||||
}
|
}
|
||||||
|
|
||||||
// Dot-matrix digits (5×7, one row per raster line, '1' = lit) burned into the
|
// Dot-matrix digits (5×7, one row per raster line, '1' = lit) burned into the
|
||||||
|
|||||||
@@ -267,9 +267,6 @@ public sealed class MediaCaptureFrameSource : ICameraFrameSource
|
|||||||
private int _ringNext;
|
private int _ringNext;
|
||||||
private long _epoch;
|
private long _epoch;
|
||||||
|
|
||||||
// Alpha probe diagnostics (one-shot, throttled)
|
|
||||||
private DateTime _lastAlphaProbe = DateTime.MinValue;
|
|
||||||
|
|
||||||
private byte[] RentRingBuffer(int size)
|
private byte[] RentRingBuffer(int size)
|
||||||
{
|
{
|
||||||
for (var tries = 0; tries < _frameRing.Length; tries++)
|
for (var tries = 0; tries < _frameRing.Length; tries++)
|
||||||
@@ -308,25 +305,11 @@ public sealed class MediaCaptureFrameSource : ICameraFrameSource
|
|||||||
var pixels = RentRingBuffer((int)capacity);
|
var pixels = RentRingBuffer((int)capacity);
|
||||||
Marshal.Copy(pixelsPtr, pixels, 0, (int)capacity);
|
Marshal.Copy(pixelsPtr, pixels, 0, (int)capacity);
|
||||||
|
|
||||||
// Alpha probe diagnostics (throttled to once per 5s)
|
|
||||||
var now = DateTime.UtcNow;
|
|
||||||
if ((now - _lastAlphaProbe).TotalSeconds >= 5.0)
|
|
||||||
{
|
|
||||||
_lastAlphaProbe = now;
|
|
||||||
ProbeAlphaStats(pixels, bitmap.PixelWidth, bitmap.PixelHeight);
|
|
||||||
}
|
|
||||||
|
|
||||||
// MediaCapture video carries no alpha plane — the OS fills 255.
|
// MediaCapture video carries no alpha plane — the OS fills 255.
|
||||||
var videoFrame = new VideoFrame(bitmap.PixelWidth, bitmap.PixelHeight, pixels)
|
var outFrame = new VideoFrame(bitmap.PixelWidth, bitmap.PixelHeight, pixels)
|
||||||
{ IsOpaque = true, Epoch = ++_epoch };
|
{ IsOpaque = true, Epoch = ++_epoch };
|
||||||
|
|
||||||
// Diagnostic: Log frame properties for compositor debugging
|
FrameAvailable?.Invoke(outFrame);
|
||||||
if ((now - _lastAlphaProbe).TotalSeconds >= 5.0)
|
|
||||||
{
|
|
||||||
AppLog.Write($"Webcam frame emit: {videoFrame.Width}x{videoFrame.Height}, IsOpaque={videoFrame.IsOpaque}, Epoch={videoFrame.Epoch}");
|
|
||||||
}
|
|
||||||
|
|
||||||
FrameAvailable?.Invoke(videoFrame);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
@@ -339,37 +322,4 @@ public sealed class MediaCaptureFrameSource : ICameraFrameSource
|
|||||||
bitmap.Dispose();
|
bitmap.Dispose();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private void ProbeAlphaStats(byte[] pixels, int width, int height)
|
|
||||||
{
|
|
||||||
try
|
|
||||||
{
|
|
||||||
// Sample 1000 alpha bytes (every 4th byte in BGRA format)
|
|
||||||
var sampleCount = Math.Min(1000, pixels.Length / 4);
|
|
||||||
var alphaMin = 255;
|
|
||||||
var alphaMax = 0;
|
|
||||||
var alphaSum = 0;
|
|
||||||
|
|
||||||
for (var i = 0; i < sampleCount; i++)
|
|
||||||
{
|
|
||||||
var alphaIndex = (i * 4) + 3; // Alpha is 4th byte in BGRA
|
|
||||||
if (alphaIndex < pixels.Length)
|
|
||||||
{
|
|
||||||
var alpha = pixels[alphaIndex];
|
|
||||||
alphaMin = Math.Min(alphaMin, alpha);
|
|
||||||
alphaMax = Math.Max(alphaMax, alpha);
|
|
||||||
alphaSum += alpha;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
var alphaMean = sampleCount > 0 ? alphaSum / (double)sampleCount : 0;
|
|
||||||
AppLog.Write($"Webcam alpha probe: {width}x{height}, epoch {_epoch}, " +
|
|
||||||
$"alpha[min={alphaMin}, max={alphaMax}, mean={alphaMean:F1}], " +
|
|
||||||
$"sampled {sampleCount} pixels");
|
|
||||||
}
|
|
||||||
catch (Exception ex)
|
|
||||||
{
|
|
||||||
AppLog.Write($"Webcam alpha probe failed: {ex.Message}");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -47,6 +47,44 @@ public class AudioSyncDelayTests
|
|||||||
Assert.Equal(960, MarkerShift(delay, 480 * 2, configureMs: 10));
|
Assert.Equal(960, MarkerShift(delay, 480 * 2, configureMs: 10));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>2026-09-12 total-silence regression: <see cref="AudioMixer.FillAndMix"/>
|
||||||
|
/// calls <c>Configure(_syncOffsetMs?.Invoke() ?? 0)</c> on EVERY ~10ms tick (it
|
||||||
|
/// re-reads the live UI setting each mix, not just on change) — the old
|
||||||
|
/// `Configure` unconditionally reallocated and zeroed the delay line and reset
|
||||||
|
/// `_writePos` every single call, discarding the buffered marker before the
|
||||||
|
/// delay offset could ever read it back. Any persisted non-zero
|
||||||
|
/// `Audio.SyncOffsetMs` (this repo's default is 0, but TASK 22 testing/a user
|
||||||
|
/// slider left 500 saved) turned the WHOLE recording into digital silence:
|
||||||
|
/// mic/loop level meters (sampled upstream of this stage) stayed honest while
|
||||||
|
/// the delay line fed the encoder pipe pure zeros forever. This test
|
||||||
|
/// reconfigures with the SAME delay before every Process call — exactly the
|
||||||
|
/// mixer's calling pattern — and the marker must still survive.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void RepeatedConfigureWithSameDelay_EveryTick_StillDelivers_TheMarker()
|
||||||
|
{
|
||||||
|
const int chunkLen = 960; // 10ms of 48kHz interleaved stereo
|
||||||
|
var delay = new AudioSyncDelay(48000);
|
||||||
|
var chunk = new float[chunkLen];
|
||||||
|
var output = new float[chunkLen];
|
||||||
|
chunk[0] = 1f; // sole non-zero: left of frame 0
|
||||||
|
|
||||||
|
var foundAt = -1;
|
||||||
|
var total = 0;
|
||||||
|
for (var c = 0; c < 6 && foundAt < 0; c++)
|
||||||
|
{
|
||||||
|
delay.Configure(10); // the mixer calls this every tick, not just on change
|
||||||
|
delay.Process(chunk, output);
|
||||||
|
for (var i = 0; i < chunkLen; i++)
|
||||||
|
{
|
||||||
|
if (output[i] != 0f) { foundAt = total + i; break; }
|
||||||
|
}
|
||||||
|
total += chunkLen;
|
||||||
|
chunk[0] = 0f; // only the first tick carries the marker
|
||||||
|
}
|
||||||
|
|
||||||
|
Assert.Equal(960, foundAt); // same 10ms-delay shift as the single-Configure test
|
||||||
|
}
|
||||||
|
|
||||||
private static int MarkerShift(AudioSyncDelay delay, int chunkLen, int configureMs = -1)
|
private static int MarkerShift(AudioSyncDelay delay, int chunkLen, int configureMs = -1)
|
||||||
{
|
{
|
||||||
if (configureMs >= 0)
|
if (configureMs >= 0)
|
||||||
|
|||||||
@@ -303,6 +303,46 @@ public class SceneCompositorTests
|
|||||||
Assert.Equal(build.BgraPixels, hit.BgraPixels);
|
Assert.Equal(build.BgraPixels, hit.BgraPixels);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>2026-09-12 regression — the webcam gray-block bug: the CropBounds
|
||||||
|
/// Fill-scaling work (ed9d7c1, same day as the transparency fix) added
|
||||||
|
/// cbX/cbY/cbW/cbH but only assigned them inside the CropBounds-present branch;
|
||||||
|
/// every CropBounds-less source (webcam, images — anything but the web widget)
|
||||||
|
/// fell through with cbW=cbH=0, so sxCrop/syCrop (= norm * cbW/cbH) were always
|
||||||
|
/// 0 and EVERY destination pixel sampled source (0,0) — a whole scaled webcam
|
||||||
|
/// collapsed to one corner pixel's color. All rect-placed elements route through
|
||||||
|
/// this exact sampler via the paste cache (take-9), so this is not webcam-specific;
|
||||||
|
/// a two-color source proves distinct destination regions read distinct source
|
||||||
|
/// regions instead of a single flat pixel.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void BlitContentRaw_NoCropBounds_SamplesAcrossFullSource_NotJustOrigin()
|
||||||
|
{
|
||||||
|
var source = Split(64, 32, 255, 0, 0, 0, 0, 255); // left=red, right=blue, no CropBounds
|
||||||
|
var background = new Source { Type = SourceType.DisplayCapture, IsBackground = true, CaptureKey = "monitor:0" };
|
||||||
|
var webcam = new WebcamSceneConfig { X = 0, Y = 0, Width = 40, Height = 20 }; // scaled (not 1:1)
|
||||||
|
var scene = new Scene { Name = "Live" };
|
||||||
|
scene.Elements.Add(background);
|
||||||
|
scene.Elements.Add(webcam);
|
||||||
|
|
||||||
|
VideoFrame? FrameFor(SceneElement e) => e switch
|
||||||
|
{
|
||||||
|
Source { IsBackground: true } => Solid(128, 64, 0, 0, 0),
|
||||||
|
WebcamSceneConfig => source,
|
||||||
|
_ => null,
|
||||||
|
};
|
||||||
|
|
||||||
|
var options = new CompositorOptions
|
||||||
|
{
|
||||||
|
SourceRectX = 0, SourceRectY = 0, SourceRectWidth = 128, SourceRectHeight = 64,
|
||||||
|
OutputWidth = 128, OutputHeight = 64,
|
||||||
|
};
|
||||||
|
|
||||||
|
var compositor = new SceneCompositor();
|
||||||
|
var output = compositor.Render(scene, FrameFor, null, options);
|
||||||
|
|
||||||
|
AssertColor(output, 5, 10, 255, 0, 0); // near the left edge of the webcam rect: source's red half
|
||||||
|
AssertColor(output, 35, 10, 0, 0, 255); // near the right edge: source's blue half — fails (reads red) on the bug
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>take-16 regression gate — GC churn convicted mechanically: a ring-style
|
/// <summary>take-16 regression gate — GC churn convicted mechanically: a ring-style
|
||||||
/// producer hands out a small set of rotating arrays with a NEW EPOCH every frame
|
/// producer hands out a small set of rotating arrays with a NEW EPOCH every frame
|
||||||
/// (exactly what the capture/camera rings do). The compositor must re-rasterize INTO
|
/// (exactly what the capture/camera rings do). The compositor must re-rasterize INTO
|
||||||
|
|||||||
Reference in New Issue
Block a user