Skip to content

Latest commit

 

History

History
133 lines (120 loc) · 127 KB

progress-log.md

File metadata and controls

133 lines (120 loc) · 127 KB

진행 기록 — 전문

날짜별 작업과 결과. 요약은 ../README.md 에 있고, 여기가 원본이다. 각 행의 Notes 열에 그날 무엇을 어떻게 확인했는지, 무엇이 나중에 뒤집혔는지가 적혀 있다.

Progress Log

Date Task Result Notes
2026-07-06 ROS 2 Lyrical native source build (Mac) Done GUI-related packages excluded; core succeeded
2026-07-06 Gazebo Jetty apt availability check (Docker) Failed Not in OSRF apt at the time; source build attempted, OOM
2026-07-07 PR #44 multibeam sonar without Nvidia (Jazzy+Harmonic, Mac Metal) Done Confirmed WGPU backend works without CUDA
2026-07-07 ros_gz bridge build for Lyrical+Jetty (Mac) Done 35 packages
2026-07-08 DAVE core packages ported to Lyrical+Jetty (Mac) Done 10 packages; CMake + SphericalCoords.cc changes only
2026-07-08 Gazebo Jetty source build retry (Docker) Done OOM root-caused to colcon+make parallelism multiplying
2026-07-11 DAVE + PR #44 sonar demo actually run in Docker Done Discovered apt already vendors Gazebo Jetty — source build unnecessary
2026-07-13 Mac vs Docker frame timing measured Done ~86–96 ms (Mac, Metal) vs ~273–438 ms (Docker, llvmpipe)
2026-07-13 Confirmed Ubuntu 26.04 is the official target OS for ROS 2 Lyrical Done docs.ros.org: "Deb packages ... available for Ubuntu Resolute (26.04)"; 24.04 has zero ros-lyrical-* packages
2026-07-13 REXROV vehicle launch verified Done Real odometry/imu/magnetometer/camera data via ros_gz bridge
2026-07-13 Real GUI confirmed visually via RDP (Docker) Done See Known Issues for the OGRE2 caveat and the xrdp setup gotcha
2026-07-14 Full world list catalogued (18 worlds) Done Only 6 previously known; 3 are manipulation scenarios, 2 require a separate sonar-demo branch
2026-07-14 DVL / underwater camera / ocean current / sea pressure sensors verified; USBL attempted PASS 4 / PARTIAL 1 DVL/camera/current/pressure launch and publish correctly; USBL server/plugin publishes but GUI client crashes — see Verified demos
2026-07-14 BlueROV2 + ArduSub SITL built and launched Done Required fixing several chained build/environment issues end to end (non-root sudo user, a hardcoded python-argparse reference, PEP 668's externally-managed-environment pip block, two Python stdlib modules removed from waf itself — imp and pipes — and a waf tool-loader search-path gap), not all of them specifically Python-3.14-caused — see Known Issues and notes/ardusub-sitl-setup.md for the full walkthrough
2026-07-14 git diff --stat compared Mac vs Docker Done Identical: 8 files, +172/−147
2026-07-15 Root-caused Docker RDP desktop crash Done XFCE + xrdp works; in the tested Ubuntu 26.04 image, the installed GNOME 50 session requires Wayland while xorgxrdp produces an X11 session, so GNOME could not be used as the RDP desktop here. docker/ Dockerfile updated to install XFCE explicitly
2026-07-15 Clean (--no-cache) Docker build + RDP re-verified Done lyrical-sim:jetty-rdp built clean, RDP login reached a usable XFCE desktop, prompt docker@lyrical_docker:~$ confirmed
2026-07-17 Docker CA-certificate bootstrap fix + clean (--no-cache) rebuild (Docker) Done arm64v8/ubuntu:26.04 ships without ca-certificates, breaking HTTPS apt; fixed via digest-pinned curlimages/curl CA bundle copy plus Acquire::https::CaInfo on both apt-get update and apt-get install. Clean build succeeded end-to-end (image lyrical-sim:jetty-rdp-pr1-ca-fix), RDP login reached XFCE, ROS_DISTRO=lyrical and dave_demos/multibeam_sonar_system package presence confirmed — scoped to build + login + package-presence only, not a re-run of world/vehicle/performance rows; see docker/README.md Known limitations
2026-07-18 Ocean Current service names verified against a running container (Docker) Done Launched dave_robot.launch.py world_name:=ocean_current_plugin on lyrical-sim:jetty-rdp-pr1-ca-fix and ran ros2 service list | grep -i current; real services are all under /hydrodynamics/ (set_current_velocity, set_current_horz_angle, set_current_vert_angle, set_stratified_current_*, matching get_*/*_model variants, plus standard underwater_current_ros_plugin parameter services) — see Next steps for the full confirmed list
2026-07-18 Tested narrowing --privileged on docker run (Docker) Done Ran lyrical-sim:jetty-rdp-pr1-ca-fix with no --privileged and no extra --cap-add; RDP login still reached a working XFCE desktop (screenshot confirmed). xorgxrdp needs no host GPU device access in this image (llvmpipe software rendering). Documented docker run example updated to drop --privileged
2026-07-18 rosdep || true masking check + mavros source-build + clean rebuild (Docker) Done Removing || true surfaced a real failure: ros-lyrical-mavros not on apt for lyrical yet. Built mavros from source instead (official ros2-branch procedure); hit and fixed two real build bugs along the way — an rm -rf /var/lib/apt/lists/* wiping the apt index before rosdep needed it (self-inflicted, libasio-dev was never actually missing), and an OOM kill of cc1plus from unconstrained make -j$(nproc) on mavros's plugins (fixed by scoping MAKEFLAGS=-j1 to that one colcon build). Clean --no-cache rebuild completed end-to-end: 51m 22s, 21.9GB image, 62.66MiB idle container RAM; mavros/mavros_extras/mavros_msgs/mavros_examples confirmed via ros2 pkg list. Also recovered a critical host-disk-full (133Mi free) + hung Docker Desktop daemon along the way — resolved by force-killing Docker, deleting and letting it recreate Docker.raw, and raising the Docker Desktop memory limit to 12GB. Destructive recovery step, not a routine one: deleting Docker.raw wipes all local Docker images, containers, volumes, and build cache, not just the stuck state — only reach for this if Docker Desktop is genuinely hung and a normal restart doesn't clear it
2026-07-20 RAM/CPU measured under an active demo workload (Docker) Done Launched the representative smoke test (REXROV world + multibeam sonar, blueview_p900) headless in the background (docker exec -d) on lyrical-sim:jetty-rdp-pr1-ca-fix and sampled docker stats 5× over ~20s once the sonar plugin was loaded: steady ~1008MiB (8.44% of the 11.67GiB container limit), ~1.2–1.5% CPU (of the container's allocated multi-core quota). Cross-checked against ps aux inside the container: gz-sim-main alone was using ~836MB RSS and ~15.7% of one core, parameter_bridge ~90MB — consistent with the docker stats total. A "Stack trace (most recent call last)" line appeared in the log right after the sonar plugin finished loading, but gz-sim-main was still alive and actively consuming CPU 7+ minutes later, so this did not crash the process — noted as a new non-fatal observation in docker/README.md Known limitations. This is idle (62.66MiB) vs. loaded (~1008MiB) — roughly a 16× increase, dominated by the Gazebo server process itself, not ROS/mavros overhead
2026-07-20 Added and fully validated Lyrical/Jetty support for dockwater (Docker) Done, end-to-end Confirmed upstream IOES-Lab/dockwater has no lyrical/ folder (only up to jazzy/). Drafted notes/dockwater-lyrical-draft.Dockerfile modeled on dockwater's own jazzy/Dockerfile; fixed (a) Acquire::https::CaInfo needing to be set on every apt call not just update, (b) dev-tools block (has curl) needing to run before the ROS apt-source block, (c) ros-lyrical-navigation2/ros-lyrical-nav2-bringup/ros-lyrical-plotjuggler-ros not yet on apt for lyrical (removed; confirmed ros-lyrical-mavros-msgs is available, unlike full ros-lyrical-mavros). Forked IOES-Lab/dockwater to yeseorizi/dockwater, added the file as lyrical/Dockerfile, and validated the full dockwater toolchain: ./build.bash lyrical succeeded (reused the standalone build's cache), then pip install rocker (into a venv) + ./run.bash -i dockwater:lyrical (internal-GPU rocker extensions: devices/git/name/volume/x11) launched a real interactive container. Verified inside: lsb_release -a → Ubuntu 26.04 Resolute, $ROS_DISTRO → lyrical, which ros2 → /opt/ros/lyrical/bin/ros2, gz sim --versions → 10.4.0 (Jetty). This is the first time DAVE's dockwater/rocker workflow (the same one this Wiki's Docker Installation Manual documents for Jazzy/Harmonic) has been shown to work for Lyrical/Jetty. Also tried -r (RDP) — connection failed; root-caused to the -r rocker extension only mapping a port, not installing an RDP server, and lyrical/Dockerfile (like dockwater's own jazzy/Dockerfile) never installs xrdp. See Next steps
2026-07-20 Added xrdp/XFCE to lyrical/Dockerfile and root-caused + fixed the resulting RDP black-screen bug (Docker) Done, end-to-end Added the same xrdp/dbus-x11/XFCE install block validated in docker/lyrical.arm64v8.dockerfile to notes/dockwater-lyrical-draft.Dockerfile, pushed to yeseorizi/dockwater's lyrical/Dockerfile. First rebuild+run.bash -r reached a live xrdp login (auth succeeded, Xorg + full XFCE session confirmed running via docker exec — xfwm4/xfce4-panel/xfdesktop all alive) but the RDP client showed a solid black screen. Root-caused to a confirmed upstream xrdp 0.10.x bug (neutrinolabs/xrdp#3118): forcing max_bpp=16 in xrdp.ini (copied over from docker/lyrical.arm64v8.dockerfile) breaks GFX-pipeline negotiation with modern clients (macOS "Windows App" among them) — server and session stay fully alive, screen data just never reaches the client. Removed the max_bpp=16 sed line (left at xrdp's default of 32); rebuilt and re-tested — XFCE desktop now renders correctly over RDP (localhost:3389, user/pass docker/docker). Note: run.bash -r -p <port> still ignores the custom port argument and always maps host 3389:3389 regardless of -p — connect on 3389, not the requested port
2026-07-22 Defined validation_matrix.csv + test_worlds.sh (August prerequisite) Done Enumerated all 18 world files directly from the real naitikpahwa18/dave checkout (dave_ws_lyrical/src/dave/models/dave_worlds/worlds/, not GitHub's API, which was unreachable this session) — count matches the Progress Log's 2026-07-14 "18 worlds" figure exactly. Built notes/validation_matrix.csv (18 rows: world file, category, inferred launch file, status, evidence date, notes), carrying over every already-confirmed result (multibeam sonar, REXROV/dave_ocean_waves, ocean current, USBL) with their exact documented launch commands as citations. Flagged one open discrepancy: dave_ocean_waves_sonar(_integrated).world files physically exist in this wgpu_integration checkout despite the Verified-demos table saying they need the separate sonar-demo branch — not yet resolved. Built notes/test_worlds.sh, a resumable smoke-test runner (launch each world headless, timeout-and-classify PASS/CRASH/EXITED, append to results/test_worlds_results.csv); manipulation and sonar-demo-branch worlds are skipped by default (--include-unknown to force). Not yet executed — needs to be run inside the Docker container or Mac native env by the user, since this sandbox has no Gazebo/ROS install
2026-07-22 USBL crash — source-level root-cause investigation (initial pass) Superseded same day Ruled out a USBL-specific GUI bug (no GUI plugin exists in the USBL source at all) and proposed an OGRE2/default-GUI-config hypothesis. Superseded a few hours later by a real crash log — see the next row. Kept in notes/usbl-gui-crash-investigation.md for the record
2026-07-22 Ran test_worlds.sh for the first time (6/18 worlds; 10 skipped by design, 2 out of scope) — USBL root cause CONFIRMED Done First real execution of the August smoke-test script, on lyrical-sim:jetty-rdp-theme. Results: camera_tutorial, dave_multibeam_sonar, dave_ocean_waves, dvl_world all PASS (32s alive, no crash). usbl_tutorial — real crash log finally captured, and it overturns the earlier same-day hypothesis entirely: this is not a GUI crash, it's the Gazebo server process (gz sim ... -s) aborting (SIGABRT, exit 134) on a libstdc++ std::normal_distribution assertion failure (_M_stddev > 0 failed). Root-caused to UsblTransponder.cc:263, which constructs the noise distribution directly from the world file's <sigma> value with no validation; usbl_tutorial.world sets <sigma>0.0</sigma> on both UsblTransponder instances, and the plugin's own safe default (1.0) never applies because the world file explicitly overrides it. ocean_current_plugin exited early on a robot_config.py file-not-found — likely a script/image-specific launch-arg gap, not a world bug; does not overturn the existing 2026-07-18 PASS (verified on a different image). Full logs in notes/results/, raw run transcript in notes/test_worlds-run.log
2026-07-22 USBL crash — fix applied and verified live Done Patched <sigma>0.0</sigma> → <sigma>0.0001</sigma> on both UsblTransponder instances in usbl_tutorial.world (patches/usbl_sigma_fix.diff, no C++ rebuild needed). Verified by docker cp-ing the patched world file over the built-in copy in the running lyrical-theme-test container and re-running the exact launch command under a 15s timeout: process survived the full window with no assertion/abort (previously died within ~3s). Not yet persisted — the Docker image build (docker/lyrical.arm64v8.dockerfile) still bakes in the unpatched world file; this fix only lives in the running container and the Mac native workspace so far. Not yet reported upstream
2026-07-22 Root-caused dave_robot.launch.py's robot_config.py gap, then swept 7 more worlds (11/18 → all PASS) Done Investigated why ocean_current_plugin and 6 other worlds exited early on [Errno 2] No such file or directory: .../dave_robot_models/config/robot_config.py: dave_robot.launch.py always requires namespace to match one of dave_robot_models/config/{rexrov,bluerov2,bluerov2_heavy,bluerov2_heavy_multibeam_sonar,glider_slocum}/robot_config.py — there is no generic/environment-only default, confirmed by reading upload_robot.launch.py's PathJoinSubstitution logic and the actual config directory contents. Not a DAVE bug, a test_worlds.sh gap (missing namespace:=rexrov, the combination README already used successfully for dave_ocean_waves). Fixed the script's defaults and re-ran the 7 affected worlds (dave_Santorini, dave_graded_seabed, dave_integrated, dave_ocean_models, dave_ocean_waves_mossy_ground, dave_ocean_waves_transient_current, ocean_current_plugin) — all 7 PASS (32–33s alive, no crash). Also re-ran new_dvl.world (previously skipped as "unknown args"): args were fine, but it exposed a real, distinct bug — Unable to find uri[.../North-East-Down frame] (404), because the world file's Fuel model URI uses hyphens where the working dave_ocean_waves.world uses spaces for the identical model name. Not yet fixed. Combined running total: 12/18 worlds confirmed PASS, 1 fixed (USBL), 1 new bug found (new_dvl), 3 manipulation worlds still out of scope, 2 worlds still pending the sonar-demo-branch discrepancy. See notes/validation_matrix.csv, notes/results/, notes/test_worlds-run2.log, notes/test_worlds-run3.log
2026-07-22 new_dvl.world bug fixed and verified live Done Patched the broken Fuel model URI (North-East-Down frame → North East Down frame, hyphens→spaces, matching the identical model's working URI in dave_ocean_waves.world) — patches/new_dvl_uri_fix.diff. Verified by docker cp-ing the patched world file into the running lyrical-theme-test container and re-running under a 15s timeout: no more Unable to find uri/404, process survived the full window (only two harmless SDF warnings about undefined latitude_deg/longitude_deg elements). Not yet baked into the Docker image build. Running total: 13/18 worlds confirmed PASS, 2 crashes root-caused and fixed same day (USBL, new_dvl)
2026-07-22 Found and fixed a process-cleanup bug in test_worlds.sh/benchmark_worlds.sh; ran quantitative benchmark (RTF + CPU/RAM) Done Discovered while starting the benchmark: test_worlds.sh's cleanup only sent SIGTERM to the top-level ros2 launch PID, which does not reliably kill its gz-sim-main grandchild — after 3 test_worlds.sh runs, 14 orphaned gz-sim-main processes were still alive in the container, one having grown to 3.2GB RSS / 108% CPU. This silently inflated the first benchmark attempt's memory readings to 8-10GB (vs. the real ~200MB-1GB range). Fixed both scripts to pkill -9 by world-specific pattern match unconditionally, confirmed clean (ps aux empty, docker stats back to a 1022MiB/3.55% CPU idle baseline) after the fix. Separately, also caught and fixed a methodology bug: Gazebo's real_time_factor field in /stats is itself a windowed/smoothing average that needs time to converge — sampling it immediately after a short (8s) settle period captured it still ramping from ~0, biasing the average badly low (initial, since-discarded numbers: 0.09-0.13 RTF for all 3 worlds, implausibly close together). Fixed by extending the settle window to 25s and sampling over a real 6s wall-clock window (300-2562 messages per world) with the first half of samples discarded as residual transient. Final results (Docker/llvmpipe, lyrical-theme-test, per-world gz-sim-main RSS): dave_ocean_waves RTF 0.660 / 990.5MiB; dave_multibeam_sonar RTF 0.533 / 618.5MiB; usbl_tutorial (lightweight, no vehicle) RTF 0.935 / 186.3MiB. All well below RTF 1.0 (real-time), consistent with llvmpipe CPU software rendering being the bottleneck, and orders qualitatively as expected (heaviest-compute world = lowest RTF). Mac (Metal) not yet benchmarked with the same script for a same-methodology comparison — see Next steps. Raw data in notes/bench_results/, script in notes/benchmark_worlds.sh
2026-07-22 Started long-duration stability test In progress Launched notes/stability_test.sh detached (nohup/disown, survives terminal closing) on lyrical-theme-test, world dave_ocean_waves (REXROV), planned 4h duration, RSS/CPU sampled every 2min, crash/monotonic-leak detection built in. Started 06:51 UTC, expected completion ~10:51 UTC. First sample confirmed healthy: 2min elapsed, alive, 80MiB RSS, 28% CPU. Results land in notes/stability_results/dave_ocean_waves-stability.csv and -summary.txt once complete — not yet reviewed
2026-07-22 Baked the USBL and new_dvl world-file fixes into the Docker image build Done Both fixes were previously verified only live in a running container (see the two rows above); added a late-layer sed fix to docker/lyrical.arm64v8.dockerfile (same cache-preserving pattern as the max_bpp xrdp fix) that patches the already-installed usbl_tutorial.world and new_dvl.world directly, with a build-time grep -q assertion so the build fails loudly if either sed doesn't match. Rebuilt (docker build, no --no-cache): 16.1s total, all 42 prior layers cache-hit, confirming the fix didn't invalidate any expensive upstream layer. Verified end-to-end on the freshly built image (lyrical-sim:jetty-rdp-worldfix, real docker run + docker exec): both worlds ran the full 15s test window (timeout exit code 124 = killed by timeout while still alive, not a crash) with no assertion/abort or 404/URI error. Test image and container removed after verification
2026-07-22 Resolved the sonar-demo-branch discrepancy (now PASS) and root-caused the 3 manipulation worlds' untestability Done Sonar worlds: read dave_ocean_waves_sonar.world/dave_ocean_waves_sonar_integrated.world in full — both only reference multibeam_sonar_system (the same plugin dave_multibeam_sonar.world already uses, confirmed present and built in this naitikpahwa18/dave checkout) plus ordinary remote Fuel includes; nothing sonar-demo-branch-specific found anywhere. The 2026-07-14 "needs IOES-Lab/dave sonar-demo branch" claim appears to have been simply incorrect for this checkout — smoke-tested both with the same args as dave_multibeam_sonar.world: both PASS, 31-32s alive, no crash. Manipulation worlds: read all 3 (dave_bimanual_example, dave_electrical_mating, dave_plug_and_socket) — none contain a robot arm/manipulator/gripper at all, just static prop models (poles, vases, plugs, panels); no manipulator vehicle exists anywhere in this checkout's dave_robot_models either. The real blocker is dave_world.launch.py (the only launch file that can load them, no vehicle spawn needed): it declares no gui/headless arguments at all and unconditionally spawns a real Qt GUI client, which aborts in this headless container (qt.qpa.xcb: could not connect to display, SIGABRT) — confirmed identically in all 3 worlds' logs. Also caught and fixed a test_worlds.sh classifier bug this surfaced: the "still alive at timeout" check alone doesn't prove nothing crashed — dave_plug_and_socket.world was misclassified PASS despite its log containing the identical abort trace found in the other two (correctly caught as CRASH only by timing luck); fixed by always grep-ing the log for crash signatures regardless of alive/dead status, and adding qt.qpa/could not connect to display to the recognized signature list. Running total: 15/18 worlds now resolved (13 PASS/fixed + 2 newly-confirmed PASS), 3 correctly reclassified as "not testable with the current launch file" rather than falsely PASS/CRASH/NOT TESTED
2026-07-23 Reviewed the long-duration stability test — CRASHED, initially looked like a real DAVE bug Done Reviewed dave_ocean_waves-stability.csv/-summary.txt (now archived at notes/stability_results_4h_original_CONTAMINATED/) from the 2026-07-22 06:51 UTC run (planned 4h/240min): RSS jumped 83→606MiB over elapsed minutes 34-44, then RSS stayed pinned at exactly 606MiB while CPU climbed monotonically 42%→93%, before the process was killed by an external SIGKILL (exit 137) at elapsed 266min. Sampling itself also has a large unexplained gap (151min→266min, ~115min with no samples logged despite a 2min interval) — plausibly the host Mac suspending/sleeping for a stretch and Docker Desktop's VM pausing along with it, which would also explain the process dying right around wake-up. The launch log's only anomaly: [SimulationRunner.cc:274] Another world of the same name is running, logged at the very start. Looked like a genuine DAVE/Gazebo-Jetty long-duration bug at first, but see the next row — a second, unrelated cause was found and the crash did not reproduce once fixed
2026-07-23 Root-caused the "crash" to leftover test-script processes, not a DAVE bug — fixed cleanup in all 4 scripts Done Ran a 1h clean re-run to isolate the cause: it died within 1 minute via external SIGKILL, no internal warning at all. docker stats at that point showed lyrical-theme-test at 204% CPU / 6.07GiB RAM, plus a forgotten ram-load-test container (from the 2026-07-20 RAM/CPU load-test work) still running at 102% CPU. Inside lyrical-theme-test, ps aux found several parameter_bridge/static_transform_publisher processes from hours-earlier dave_multibeam_sonar runs still alive (one at 418+ CPU-minutes) — these survive because test_worlds.sh/benchmark_worlds.sh's cleanup only pattern-matches world_name:=/worlds/*.world in the command line, which the gz-sim-main process has but sibling processes like parameter_bridge (command line only lists topic names) never do. Killed the stragglers and the extra container (CPU dropped 204%→1.82%), then re-ran the 1h test again: survived the full duration cleanly, zero "Another world" collisions, exactly one gz-sim-main PID throughout. Conclusion: resource starvation from this repo's own test scripts leaving processes running for hours across earlier, unrelated test sessions (confirmed directly for the 1-minute-crash repro) is at minimum a major contributor, plausibly combined with the host Mac sleeping mid-run for the original 4h test specifically (see previous row) — not a DAVE/Gazebo-Jetty defect either way. Fixed all 4 scripts (test_worlds.sh, benchmark_worlds.sh, benchmark_worlds_mac.sh, stability_test.sh) to also kill -KILL the whole process group of the backgrounded ros2 launch job, not just pattern-match individual processes. Open observation (not alarming): the clean re-run's RSS/CPU settled higher (~1660MiB/122%) than the very first clean baseline sample taken earlier in the day (80MiB/28%) — plausibly just run-to-run startup variance, but worth a full 4h re-run in a verified-clean environment before calling this fully closed
2026-07-23 Ran the bug-fixed Mac benchmark end-to-end; reproduced and characterized a real dave_multibeam_sonar performance/stability issue Done, with an open finding First run of the bug-fixed benchmark_worlds_mac.sh (namespaced /world/.../stats topic preference + process-group kill, see the 2026-07-22/23 rows above) produced 0 RTF samples for dave_ocean_waves/dave_multibeam_sonar despite finding the right topic — traced to the default 25s settle being too short for these two heavier worlds on Mac (their launch logs show bridge creation / GPU sonar-backend compilation still in progress at the 30s mark). Raising settle to 60s fixed dave_ocean_waves (RTF 0.667, 3 samples) but not dave_multibeam_sonar (still 0 samples, RSS essentially flat 231.7→229.7MiB despite the extra 30s wait). Manually investigated: gz-sim-main for this world was alive and burning 104%→181% CPU over 15s with zero new /stats messages and zero new log lines past the "sonar backend selected" line — a high-CPU stall, livelock-like behavior, not a hang or crash (no thread dump was taken, so "livelock" describes the symptom, not a confirmed mechanism). Tried compute_backend:=cpu instead of wgpu to rule out a Metal-specific cause — same stall, ruling that out. A longer manual sample (20s window, ~130s into the run) did catch exactly one message: real_time_factor: 0.0123 (sim_time 3.665s / real_time 103.3s) — confirming the world can progress, just catastrophically slowly (~80× slower than real time), consistent with the sonar plugin's own log showing a heavy compute load for this world's config (513 beams × 301 rays × 4 freq, vs. a 1×1×4 warm-up call logged at only 10-11ms). Re-ran the full official benchmark with SETTLE_SEC=90 SAMPLE_WINDOW_SEC=60 for a consistent, comparable methodology across all 3 worlds. Final Mac (Metal, Apple M2) results: dave_ocean_waves RTF 0.423 (48 samples) / 852.2MiB; dave_multibeam_sonar RTF 0.015 (214 samples, this run — highly unreliable, see next row) / 435.0MiB; usbl_tutorial RTF 0.998 (272 samples) / 152.2MiB. Raw data: notes/bench_results/SUPERSEDED_2026-07-23_PRE_CLEANUP_FIX/benchmark_results_mac_2026-07-23.csv
2026-07-23 Re-ran the Docker benchmark with the same fixed methodology; confirmed dave_multibeam_sonar is broken on both platforms, not a Mac-only quirk Done, with an open finding The existing Docker RTF numbers (0.660/0.533/0.935, logged 2026-07-22) were measured before the /stats-topic-collision fix, so per that row's own caveat they needed re-verification. Copied the fixed benchmark_worlds.sh into lyrical-theme-test and re-ran with the same SETTLE_SEC=90 SAMPLE_WINDOW_SEC=60 used for the final Mac run. Results: dave_ocean_waves RTF 0.277 (69 samples) / 1026.4MiB gz-sim RSS (5305MiB container); usbl_tutorial RTF 1.000 (298 samples) / 185.6MiB (4681MiB container) — both plausible, both lower than the pre-fix Docker numbers, consistent with those older numbers having been biased by the same stale-topic risk already flagged (unconfirmed which way). dave_multibeam_sonar — 0 samples again, even at 90s+60s (798.2MiB RSS, 5191MiB container, alive but no stats). Extended to a dedicated 90s-settle + 180s-window manual run targeting only this world: still 0 samples after 4.5 minutes total, and ps aux showed gz-sim-main at only 0.9% CPU — i.e. this world doesn't show the high-CPU stall seen on Mac, it shows a near-idle stall instead (no progress, low CPU, consistent with a possible deadlock — no thread dump/lock analysis was done, so this is a symptom description, not a confirmed mechanism). Conclusion: dave_multibeam_sonar (the blueview_p900 513-beam/301-ray config) is unreliable on both Mac native and Docker/Linux under ROS 2 Lyrical + Gazebo Jetty — it sometimes crawls forward at a catastrophic RTF (~0.012-0.015, see previous row) and sometimes shows a simulation-progress stall with no progress at all, with different platform-specific symptoms (Mac: high-CPU stall, livelock-like; Docker: near-idle stall, possible deadlock). This does not look like a benchmarking artifact — it reproduced across two platforms, two compute backends (wgpu/cpu), and multiple fresh launches. Treat the 0.015/0.533 RTF figures for this world as not representative, not as a valid performance number; the real finding is that this world/sensor config needs its own investigation (likely in the sonar ray-compute plugin itself under Gazebo Jetty, not yet done — no source-level root cause identified) before it can be benchmarked meaningfully. Not yet reported upstream. Raw data: notes/bench_results/SUPERSEDED_2026-07-23_PRE_CLEANUP_FIX/benchmark_results_docker_2026-07-23.csv; superseded 2026-07-22 numbers moved to notes/bench_results/SUPERSEDED_2026-07-22/. See Next steps
2026-07-23 Documentation audit — resolved contradictions between the latest findings and how they were phrased in README.md/docker/README.md Done A labmate/professor-facing review flagged 8 places where the previous day's wording overclaimed relative to what was actually confirmed, likely to read as self-contradictory (e.g. "15/18 PASS" next to a same-day dave_multibeam_sonar livelock/deadlock finding). Fixed: (1) reclassified world results into SMOKE PASS/FUNCTIONAL PASS/PARTIAL/NOT AUTOMATED (defined near the top of this README) instead of a flat PASS/FAIL, moving dave_multibeam_sonar from PASS to PARTIAL and the 3 manipulation worlds from "NOT TESTABLE" to "NOT AUTOMATED" (a current headless-launch-path limitation, not a claim they're impossible to test); (2) narrowed the PR #44 WGPU sonar claim to what's actually confirmed (build, GPU device selection, real PointCloud2 output) separate from the still-open simulation-progress stability question; (3) reworded the USBL fix as a world-file workaround (changes "zero noise" to "near-zero noise") with the correct plugin-level fix (sigma == 0 → use the mean, sigma < 0 → reject) still open, not applied; (4) flagged a new, previously undocumented issue in the SphericalCoords.cc migration patch — a failed std::optional conversion silently falls back to (0,0,0) and still returns success, indistinguishable from a real near-origin result; (5) split the "BlueROV2 + ArduSub SITL PASS" row into what's actually validated (ArduSub/mavros build and launch) versus what isn't (BlueROV2 vehicle/sensor behavior inside a DAVE world — only REXROV has been used as the smoke-test vehicle); (6) refreshed the top-of-file status block's stale RTF numbers (0.53-0.94, "Mac not yet measured") to the final 2026-07-23 same-methodology numbers; (7) confirmed the stability-test wording already correctly said "1h clean PASS, 4h re-verification open" rather than claiming the 4h test itself completed — no change needed there; (8) rewrote docker/README.md's status block and results table, which still said USBL PARTIAL/all-worlds-NOT-TESTED/RTF-0.53-0.94/stability-NOT-DONE, to match the corrected main README. Also updated notes/validation_matrix.csv's dave_multibeam_sonar row (PASS → PARTIAL) and the 3 manipulation worlds' status label (NOT TESTABLE → NOT AUTOMATED) to match. macOS reproduction-command completeness (missing wgpu_vendor/multibeam_sonar/multibeam_sonar_system packages, venv, and underlay-sourcing steps) and the stale main branch (23 commits behind docs/review-fixes) were also flagged — see the rows below and Next steps
2026-07-23 Fixed the macOS reproduction section and added a main-branch banner Done Recovered the real missing steps from this session's own .zsh_history (grepped for colcon build/venv/ros_gz_ws/CMAKE_OSX) rather than guessing: a Python venv (~/ros2_lyrical/.venv) is activated before every build/run; ros_gz is built from source into a separate ~/ros_gz_ws_lyrical workspace (confirmed its actual package list via ls src/ — the standard ros2/ros_gz packages plus Gazebo Jetty's gz-*-vendor packages, ~35 total, matching the 2026-07-07 Progress Log figure) using -DCMAKE_OSX_SYSROOT pointed at the real Xcode SDK, not just Command Line Tools; and DAVE's own build was missing the WGPU sonar packages (wgpu_vendor, multibeam_sonar, multibeam_sonar_system) entirely, meaning a fresh checkout following the old snippet couldn't have run the multibeam-sonar command the same snippet ended with. Updated the Reproduction section accordingly; flagged as still-open that ros_gz_ws_lyrical's exact package versions/tags were never captured as a reusable .repos manifest. Also added a short banner to the top of the main branch's README.md (separately, main is 23 commits behind docs/review-fixes — decided against merging without review, added a pointer instead) — see main's README
2026-07-23 Fifth documentation/code audit — fixed a broken CSV row, real process-group cleanup, a stability-test leak-detection gap, test_worlds.sh classifier bugs, and remaining reproduction/traceability gaps Done, two items still open A deeper pass (beyond wording) checked CSV parsing, actual script behavior, and the reproduction steps end to end. Found and fixed: (1) validation_matrix.csv's USBL row had an unquoted comma in the status field, shifting every column after it when parsed by real CSV tooling — quoted the field. (2) All 4 scripts' kill -KILL -- "-$pid" cleanup was not actually reliable — a background job's PID is not a process-group leader without job control, confirmed via a minimal repro ("group -<pid> DOES NOT exist"); fixed by launching via setsid (Docker/Linux scripts) or bash's set -m (the Mac script, since setsid isn't standard there), added a script-level trap ... EXIT INT TERM HUP, routed the early-return path in both benchmark scripts through the same cleanup (previously skipped it), and added a post-cleanup pgrep check that prints a warning if anything survived. (3) stability_test.sh's header comment claimed monotonic-RSS-growth detection that was never actually implemented (only first_rss/max_rss/liveness were tracked) — implemented a real (if coarse) heuristic: first-half vs. second-half average RSS comparison, flagging WARNING (not CRASH) above a threshold; sanity-checked against the existing clean 1h run's CSV (correctly reports OK, -6% growth) and a synthetic leak CSV (correctly reports WARNING). (4) test_worlds.sh's WORLDS array had every one of the 18 entries marked known=1, silently making --include-unknown a no-op and running the 3 manipulation worlds by default every time (their expected GUI abort was then recorded as a surprise CRASH) — set known=0 for just those 3. (5) test_worlds.sh treated a bare "stack trace (most recent call last)" log line as an automatic CRASH signature even when the process was confirmed still alive — but docker/README.md documents a real case of that same line appearing with gz-sim-main staying alive and working 7+ minutes later; split into a "hard" signature list (still CRASH) vs. stack-trace-alone-while-alive (downgraded to a new REVIEW status). (6) Softened remaining overclaims from the contaminated dave_ocean_waves_sonar dual-process sample (README Verified demos + Next steps, and validation_matrix.csv's "attributable to CPU contention" causal claim) to "confirms at least one instance progressed, cannot establish a clean per-instance RTF or rule out the same problem." (7) The claim that PointCloud2 output was reconfirmed "again during the 2026-07-20 RAM/CPU load test" wasn't actually traceable to any recorded topic-echo evidence from that day (only liveness/CPU/RAM/a non-fatal stack trace were recorded) — removed that specific reconfirmation claim from README (both the Purpose paragraph and Verified-demos table) and validation_matrix.csv, keeping only the two evidenced confirmations (2026-07-07 Mac Jazzy/Harmonic, 2026-07-11 Docker Lyrical/Jetty). (8) Minor consistency: docker/README.md's USBL row still said plain "PASS" instead of the corrected "FUNCTIONAL PASS (combined evidence, workaround)" label; benchmark_worlds.sh's early-return CSV write was one column short of its own 8-column header (fixed, plus corrected the header comment's claim that container_cpu_pct is actually measured — it's a never-populated placeholder, always N/A). Two items intentionally left open, not fabricated: the exact macOS ROS 2 Lyrical underlay path needed before building ros_gz from source (flagged inline in the Reproduction section, needs a fresh .zsh_history check to confirm rather than guess), and a local, uncommitted ros_gz diff (ros_gz_bridge/ros_gz_sim CMakeLists.txt, adds find_package(tinyxml2 REQUIRED) + a CLI11 include-dir shim) that the .repos manifest doesn't capture — flagged as a known gap pending the actual diff from the real workspace, rather than reconstructing it from a partial description
2026-07-23 Branch synchronization and closed the two remaining reproduction gaps from the row above Done main and docs/review-fixes were made identical at the time (both at commit 83ff9b6; HEAD has since moved forward through further audit-fix commits — see later rows for the current state) — main was no longer 23 commits behind, and the temporary "see docs/review-fixes for the latest" banner was removed from main's README since it was no longer accurate. The two items left open above are now resolved: confirmed the real macOS ROS 2 Lyrical underlay path via .zsh_history (~/ros2_lyrical/install/setup.zsh, sourced before building ros_gz) and captured the actual local ros_gz CMakeLists.txt modifications via git diff on the real workspace as patches/ros_gz_lyrical_jetty_mac.diff, added to the Reproduction section's vcs import step. Note: this row's own date, and dates on several nearby audit-fix rows, were originally miswritten as 2026-07-24 (a date that hadn't occurred yet) — corrected to 2026-07-23 after being caught in a later review; see git history for the exact commit timestamps if precise ordering matters
2026-07-23 Sixth documentation/code audit — signal-handling, cross-platform, and factual-accuracy fixes Done Found and fixed, beyond wording: (1) a wrong future date (2026-07-24, which hadn't occurred yet) had propagated into ~20 places across README.md and 4 scripts from the fifth-round fixes — corrected to 2026-07-23 (confirmed via git log timestamps) everywhere except the explanatory sentence describing the mistake itself. (2) All 4 scripts' trap cleanup_current EXIT INT TERM HUP ran cleanup on a signal but did not actually terminate the script — confirmed via a live repro (kill -TERM on a running instance kept going and exited 0). Split into a plain trap cleanup_current EXIT plus trap 'exit N' INT/TERM/HUP (128+signum codes); re-tested the same repro and confirmed immediate termination with the correct exit code (143 for TERM) and cleanup still running exactly once. (3) test_worlds.sh documented Mac support but unconditionally called setsid (not a standard macOS command) with only a warning, so worlds would fail to even launch on Mac, not just fail to clean up — added a real fallback (bash set -m, same approach as benchmark_worlds_mac.sh) so Mac support is now actually implemented, not just documented. (4) test_worlds.sh's crash-signature list didn't include the USBL server's actual confirmed abort pattern (std::normal_distribution assertion, SIGABRT/exit 134) — a regression of the sigma workaround would have been misclassified EXITED instead of CRASH; added assertion.*failed|aborted|sigabrt|exit code 134 (pipes escaped here since this is a Markdown table cell). (5) The Jetty migration patch left a pre-existing message(STATUS "Compiling against Gazebo Harmonic") unchanged in 3 of the CMakeLists.txt files it touches, so a Jetty build's own log output says "Harmonic" — cosmetic only (the actual find_package versions right above it were correctly migrated), but confusing; fixed the text to "Gazebo Jetty" in the 3 files confirmed present in the patch's diff context (dave_gz_model_plugins, dave_gz_world_plugins, dave_ros_gz_plugins) and flagged dave_gz_sensor_plugins as unconfirmed (the message wasn't inside this patch's existing diff hunks, so it's invisible to a limited-context diff even if present in the real file). (6) docker/lyrical.arm64v8.dockerfile's rosdep install ... || true was tolerating any rosdep failure, not just the one documented, known-missing ros-lyrical-mavros apt package — narrowed to --skip-keys mavros (no blanket || true), not yet re-verified with an actual rebuild. (7) notes/cmake-migration-patterns.md conflated two different Boost facts ("Boost 1.89+ made boost_system header-only") — corrected: Boost.System has been header-only since 1.69; what actually broke here is that 1.89 removed a compatibility stub library, which is a different, later change. (8) The SphericalCoords.cc silent-failure Known Issue only documented TransformFromSphericalCoord — confirmed via the actual patch diff that TransformToSphericalCoord has the identical pattern (each of latitude_deg/longitude_deg/altitude independently falls back to 0.0 on a failed optional, still returns success) — expanded the bullet to cover both directions. (9) Added notes/evidence/multibeam_pointcloud2_evidence.md: the "513 beams × 301 rays" PointCloud2 claim had no small evidence artifact in the repo backing the exact figure — documented what this repo's own logs do confirm (bridge creation + at least one message passing) versus what they don't (the exact dimensions), rather than either fabricating a fake topic-echo capture or silently leaving the gap unacknowledged. (10) Minor: test_worlds.sh's header comment claimed PASS/CRASH/TIMEOUT/SKIPPED status values when the real ones are PASS/CRASH/REVIEW/EXITED/SKIPPED; validation_matrix.csv's manipulation-world classifier-bug note was rewritten to make clear it describes the 2026-07-22 version of the script, already fixed, not current behavior; softened "reproducible Docker image" to "repeatable Docker build procedure" given apt-get install has no version pins and both Firefox (firefox-latest-ssl) and QGroundControl (DailyBuild AppImage URL) are floating dependencies; flagged (not fixed — needs a real .zsh_history check) that the macOS Reproduction section may be missing a CMAKE_PREFIX_PATH=/opt/homebrew:...-style variable and a psutil pip install. All 4 scripts re-passed bash -n and the CSV re-validated as 19 well-formed rows after every change in this round

| 2026-07-27 | Retrieved and reviewed the 2026-07-23 clean 4h stability re-run — found it also predates the real fix, like the benchmark finding above | Preliminary, clean rerun still needed | The 4h re-run mentioned as "in progress" in the sixth-round row above had actually finished (container lyrical-theme-test still alive 6 days later, world dave_ocean_waves), with a summary reporting SURVIVED full planned duration. Two problems found on review: (1) the summary is missing the Memory-growth heuristic line entirely, and grep-ing the actual script baked into the container (/home/docker/stability_test.sh) for leak_line/Memory-growth/the real terminating signal traps found none of them — confirming this run used the pre-fix script (it started 05:35 UTC on 2026-07-23, before the fc48555 fix commit at 07:12 UTC the same day). (2) the raw CSV has a 114-minute sampling gap (elapsed 194min→308min, timestamps 08:50:15Z→10:44:18Z, versus a steady 2min interval everywhere else) — the same signature previously attributed to the host Mac sleeping mid-run for the original crashed attempt, meaning only ~194 of the planned 240 minutes were actually continuously monitored. Retrieved via docker cp and saved as notes/stability_results_4h_rerun_2026-07-23_PRE_LEAKFIX_SCRIPT/ — kept for the record, not cited as a completed clean verification. See Next steps for what a valid rerun still needs (current script version, no multi-hour monitoring gap) | | 2026-07-27 | Re-ran the Mac/Docker RTF benchmark clean, using the current post-fc48555 scripts | Done | Copied the current notes/benchmark_worlds.sh into lyrical-theme-test and the current notes/benchmark_worlds_mac.sh into the Mac workspace, then ran both with SETTLE_SEC=90 SAMPLE_WINDOW_SEC=60 (same methodology as 2026-07-23). Results: dave_ocean_waves RTF 0.170 (Docker, 78 samples, 932.8MiB gz-sim RSS) / 0.376 (Mac, 46 samples, 397.5MiB); usbl_tutorial RTF 0.687 (Docker, 292 samples, 182.7MiB) / 0.646 (Mac, 216 samples, 92.3MiB); dave_multibeam_sonar — Docker: 0 samples (N/A, unreliable as always), Mac: 82 samples but RTF 0.107 (still far below 1.0, excluded as unreliable per the existing finding). These clean numbers differ meaningfully from the superseded 2026-07-23 PRELIMINARY run (e.g. Docker usbl_tutorial 1.000 → 0.687, dave_ocean_waves 0.277 → 0.170), confirming the earlier caution was warranted. Raw data: notes/bench_results/benchmark_results_docker_2026-07-27.csv, notes/bench_results/benchmark_results_mac_2026-07-27.csv; superseded 2026-07-23 CSVs moved to notes/bench_results/SUPERSEDED_2026-07-23_PRE_CLEANUP_FIX/. The 4h stability re-verification remains open — see the row above and Next steps | | 2026-07-27 | First BlueROV2 spawn test inside a DAVE world (dave_ocean_waves) | PARTIAL | ros2 launch dave_demos dave_robot.launch.py world_name:=dave_ocean_waves ... namespace:=bluerov2 on lyrical-theme-test: entity spawn confirmed (Entity creation successful), real ros_gz bridges created for /model/bluerov2/{odometry,odometry_with_covariance,pose,imu,magnetometer} — same pattern as the already-confirmed REXROV bridges. Blocked before MAVROS came up: ros2 pkg list on this container has no mavros* packages at all (confirmed via direct grep) — lyrical-theme-test runs a different image (lyrical-sim:jetty-rdp-theme) than the one the from-source mavros build was verified on (lyrical-sim:jetty-rdp-pr1-ca-fix), so this is an untested-image gap, not a regression. Also observed in the same run, not yet investigated: (1) Failed to load system plugin [libArduPilotPlugin.so]: Could not find shared library — ArduSub still started via its SITL socket interface regardless, impact unclear; (2) a real script bug in dave_robot_models/keyboard_publisher.py — its /dev/tty-missing exception handler itself crashes with AttributeError: 'RcutilsLogger' object has no attribute 'warn' (rclpy renamed .warn() to .warning()), only reachable via the already-abnormal no-TTY path so low priority. See Verified demos and Next steps | | 2026-07-27 | First Slocum Glider spawn test inside a DAVE world (dave_ocean_waves) | Done | namespace:=glider_slocum on lyrical-theme-test: entity spawn confirmed, and a full bidirectional bridge configuration was created — battery/battery/state, joint/propeller_joint/{cmd_thrust,ang_vel,enable_deadband}, navsat, odometry, odometry_with_covariance, pose, imu. Clarified 2026-08-27: this was launch-log evidence that endpoints were created, not message or command coverage; the integrated enable_deadband ROS→Gazebo direction later remained unobserved. No mavros dependency (Glider isn't ArduSub-based), so it doesn't hit the same-day BlueROV2 blocker. A SimulationRunner.cc:274: Another world of the same name is running warning appeared — traced to this test coincidentally running against the same world_name:=dave_ocean_waves as an unrelated, already-in-progress stability test, confirmed via ps aux to be a world-name collision, not a leftover/orphaned process (only one gz-sim-main was running throughout, correctly attributable to the stability test). See Verified demos and Next steps | | 2026-07-27 | Fixed dave_world.launch.py's missing headless mode, unblocking the manipulation worlds | Done, 1/3 confirmed | Added gui/headless DeclareLaunchArguments to examples/dave_demos/launch/dave_world.launch.py, mirroring the existing dave_sensor.launch.py convention: condition=IfCondition(gui) on the gz_sim IncludeLaunchDescription, and -s appended to gz_args when headless:=true. Edited directly on the Mac native dave_ws_lyrical checkout (--symlink-install, so live immediately, no rebuild); syntax-checked (python3 -m py_compile) and diffed (git diff, 21 insertions, 0 deletions) — saved as patches/dave_world_launch_headless_fix.diff. Verified live twice: timeout 20 ros2 launch dave_demos dave_world.launch.py world_name:=dave_bimanual_example gui:=true headless:=true survived the full 20s both times (exit code 124 = killed by timeout, not a crash), no qt.qpa.xcb/SIGABRT. dave_electrical_mating/dave_plug_and_socket share the identical launch file and previously-identical crash trace, so the fix is expected to apply the same way, but neither has been independently re-tested yet. This fix has not been applied inside the Docker container or reported upstream — Mac-only so far. See Known issues, Verified demos, and Next steps | | 2026-07-27 | Re-tested dave_electrical_mating and dave_plug_and_socket with the dave_world.launch.py headless fix — all 3 manipulation worlds now confirmed | Done | timeout 20 ros2 launch dave_demos dave_world.launch.py world_name:=<world> gui:=true headless:=true, run twice each on both remaining worlds: all 4 runs survived the full 20s, exit code 124 (killed by timeout, not a crash), no qt.qpa.xcb/SIGABRT in any run. All 3 manipulation worlds are now SMOKE PASS, closing out the last NOT AUTOMATED category in the 18-world matrix (now 15/18 PASS-level, 3/18 PARTIAL, 0/18 NOT AUTOMATED). Fix remains Mac-only — not applied inside Docker or reported upstream. See Known issues, Verified demos, and notes/validation_matrix.csv | | 2026-07-29 | Live-debugged and closed usbl_tutorial's remaining FUNCTIONAL PASS gap — found and worked around a second real bug (paused:=false), after ruling out several red herrings | Done — upgraded PARTIAL → FUNCTIONAL PASS | Set out to close the "no single run confirms both real topic data and no-abort" gap via an extended live session (lyrical-theme-test). Tooling detour: ros2 topic echo fails immediately with RuntimeError: !rclpy.ok() or RCLError: failed to initialize wait set for any topic on this container's stack (ROS 2 Lyrical + Python 3.14 + rclpy 10.0.10 + ros2cli 0.40.7, built 2026-06-06), while ros2 topic pub/list/info work normally — no matching upstream report found (ros2/rclpy#1221, ros2/ros2cli#519, neither matches); worked around with a standalone rclpy Python subscriber script instead of the CLI verb throughout. Confirmed red herring: UsblTransponder's own m_interrogationMode (separate from UsblTransceiver's, which does default to "common") is never given a default in Configure() — only ever set via interrogationModeCallback — so usbl_tutorial.world needs an external publish to interrogation_mode before the ping chain does anything; a real, previously-undocumented precondition, worked around by publishing "common" to it. Real root cause of the actual blocker: even with that precondition satisfied and common_interrogation_ping confirmed matched (ros2 topic info -v), zero transponder_location messages arrived and the live gzserver terminal's own scrollback showed nothing appended since startup — meaning cisRosCallback itself was never invoked despite DDS reporting a matched subscription. Traced to: dave_sensor.launch.py only appends gz sim's -r (run-on-start) flag when given paused:=false explicitly (confirmed by reading the launch file directly); without it the world loads paused, and both UsblTransponder::PostUpdate/UsblTransceiver::PostUpdate gate their rclcpp::spin_some() call behind if (!_info.paused) — so ROS2 subscription callbacks structurally never fire, independent of and invisible to DDS-level topic matching. Also matches the earlier (2026-07-22) investigation's own finding that usbl_tutorial.world has no gz-sim-user-commands-system plugin, which is why a live unpause via the /world/.../control service also timed out on the already-running instance — had to kill and relaunch with the arg. Verified: killed the paused instance, relaunched with paused:=false, and in that one continuous session confirmed both the no-abort state (sigma fix still holding) and real dave_interfaces/msg/Location data streaming from both transponders after publishing to interrogation_mode/common_interrogation_ping — closing the combined-evidence gap. usbl_tutorial upgraded to FUNCTIONAL PASS; notes/validation_matrix.csv and the Verified demos table updated to match. Plugin-level sigma fix and an upstream report (now covering both bugs) still not filed. Full session detail in notes/usbl-gui-crash-investigation.md | | 2026-07-29 | Re-investigated dave_multibeam_sonar's simulation-progress stall in Docker | Refined finding, not yet root-caused | Set out to catch the world in an actual stalled state via gdb. First two attach attempts targeted the wrong process — pgrep -f 'gz-sim-main' has no world-name filter, and a concurrently-running unrelated gz-sim-main (the same-day 4h dave_ocean_waves stability test) was alive at the same time; caught before being reported as a finding, confirmed the correct PID via ps -eo pid,ppid,pcpu,etime,lstart,cmd. With the correct process, read the sonar's async computeThread_ (MultibeamSonarSensor.cc) and ruled it out as the blocker — it's genuinely asynchronous and not awaited by Update()/PostUpdate(). Sampled the real /world/oceans_waves/stats topic (the world's actual internal name — differs from the dave_multibeam_sonar.world filename, which is why an earlier query against /world/dave_multibeam_sonar/stats found nothing) over a ~34-minute window: iterations was continuously increasing across two 20s-apart sample pairs, and two gdb all-thread backtraces 30s apart showed the main thread's program counter had moved — no hard stall in this run. Instead: sustained crawl at real_time_factor ~0.0008-0.0077 (mostly ~0.0018), lower than the previously-documented 0.012-0.015, at CPU ~31-32% (not the near-idle ~1% previously documented for Docker). Does not contradict the 2026-07-23 near-idle/zero-message finding (that observation was also real) — reads as a second, different-looking failure mode on the same world, possibly an RTF so low it's indistinguishable from a stall over a short sample. Leading remaining suspect: the synchronous GPU ray-cast call (Render(), upstream in gz-rendering) under Docker's llvmpipe software rasterizer — not yet directly confirmed. README.md/docker/README.md/notes/validation_matrix.csv updated with this finding | | 2026-07-29 | Completed a genuinely clean 4h stability re-run, closing the long-open gap | Done | Ran dave_ocean_waves for the full planned 4h (04:39:49Z–08:40:07Z) using the current post-fc48555 script version, monitored the whole time. Verified after completion: 121-line CSV (120 samples + header), a clean unbroken 2-minute cadence from elapsed=2min to elapsed=240min with no gaps anywhere in the file — unlike the two prior 4h attempts (2026-07-22 original: contaminated by leftover processes, crashed; 2026-07-23 rerun: pre-fix script version plus a 114-minute monitoring blind spot). Outcome: SURVIVED full planned duration, RSS 932MiB → 1253MiB (growth 133.4MiB, 12%, Memory-growth heuristic: OK). This is the first 4h run that is simultaneously current-script, uncontaminated, and gap-free — closes the "genuinely clean 4h re-run still needed" item open since 2026-07-27. Raw data copied out of the container and committed 2026-07-31 to notes/stability_results_4h_clean_2026-07-29/ — re-verified on the committed copy: 120 samples, elapsed_min 2→240 with every interval exactly 2 minutes (no gaps), alive=1 on every sample, RSS 932→1253 MiB | | 2026-07-29 | Re-investigated dave_ocean_waves_sonar_integrated's simulation-progress in Docker with the same corrected method | Superseded a stale figure | Fresh launch, correct PID confirmed via ps (no contamination, unlike the risk in earlier ad hoc samples), correct topic name (/world/oceans_waves_sonar_integrated/stats). Sampled 4 times over ~4.5 minutes: iterations increased continuously at every check, never stalled, at a delta-based real_time_factor consistently in the ~0.1-0.4 range — roughly 10× better than the 2026-07-23 figure of ~0.03 that's been the documented number since then. CPU climbed steadily and unexplainedly over the window (32%→47%→69%), not yet plateaued or investigated further. This measurement ran concurrently with two other gz-sim-main instances competing for CPU on the same container (the same-day stability test and the dave_multibeam_sonar re-check above) — an atypical, non-isolated condition that, if anything, should have hurt this result rather than helped it, which strengthens confidence in the improved number rather than casting doubt on it. Status held at PARTIAL — the climbing CPU is unexplained and a clean isolated re-benchmark hasn't been done — but the old RTF ~0.03 figure should no longer be cited as current. Notably, this world shares dave_multibeam_sonar's exact plugin (multibeam_sonar_system) yet performs roughly 50-200× better by RTF; worth comparing their beam/ray configs. README.md/notes/validation_matrix.csv updated | | 2026-07-31 | Established the first no-sonar control baseline for dave_multibeam_sonar on Mac, re-measured the world with the sonar enabled, and identified a sensor-initialisation delay that invalidates several earlier short-window measurements | Refined finding — Mac figure not reproduced | Control (new): ran the same world through dave_world.launch.py with no sonar sensor attached — RTF 0.9996 (544 /world/default/stats samples, 60s window). Until now every dave_multibeam_sonar figure had been quoted without a same-world control, so the sonar's actual share of the cost was never isolated. With sonar (Mac, Metal, default 10m range): RTF 0.2223 — endpoint-delta over a 119.2s window, 814 samples, single confirmed instance (iterations 37,688 → 64,188, monotonic). That is a ~4.5× slowdown against the control, not the stall or livelock characterised on 2026-07-23, and 15-18× higher than the 0.012-0.015 Mac figure documented then — that figure was not reproduced today. Method fault found (applies retroactively): the multibeam sonar sensor does not finish initialising until ~163s after launch on this machine — SONAR PLUGIN LOADED / # of Beams = 513 / Persistent GPU buffers allocated for 513×301×399 all appear only then. Before that point the compute backend has merely warmed up (1×1×4 buffers) and the world runs sonar-free at RTF ≈ 1.0. Any measurement taken on a settle shorter than ~180s therefore reports a sonar-less world while appearing valid — six such readings were produced and discarded today before the cause was found. Earlier short-sample figures on this world should be re-checked against this. Not yet re-measured on Docker, so the 2026-07-29 Docker figure (~0.0018) still stands and remains ~123× worse than today's Mac number — consistent with the llvmpipe software-rasteriser suspicion, but still not directly confirmed. Probe and sweep scripts added under notes/experiments/; they poll the launch log for the 513×301×399 line instead of using a fixed sleep, and abort a measurement outright if a stale gz-sim is still alive |

| 2026-08-03 | Mapped dave_multibeam_sonar's startup phase continuously on Mac and reproduced the 2026-07-23 stall symptom | Root-caused — the 2026-07-23 stall was the startup window | Sampled /world/default/stats in back-to-back 25s windows from launch (t=0) to t=360s in a single confirmed instance, recording for each window whether the sonar had finished initialising. The stall reproduces, and it is the startup window: for ~80s (t≈54–106s) /stats emits fewer than two messages per 25s window, against 69–85 per window (~3.3 Hz) once healthy — the "zero new /stats messages" symptom recorded on 2026-07-23, seen directly. One window at t=27s reads RTF 0.0014, the same order as that day's 0.012–0.015, though that value rests on two messages 0.711s apart and is indicative only, not a measured rate. From t=160s the sonar is live and RTF is flat over eight consecutive windows (mean 0.2060, min 0.1866, max 0.2219), matching the four 2026-07-31 repeats. That day's 25s and 90s settles both fall inside the startup window, which explains both the "0 RTF samples" result and the crawl figure. This closes the question of what the 2026-07-23 Mac characterisation was measuring — it was not a livelock. Remaining gaps recorded rather than glossed: the curve is a single run; it does not fully agree with 2026-07-31, where the pre-sonar window read RTF ~1.0 six times at 90–150s while here t=133s already reads 0.1900, so sonar init time varies by ~20–30s between runs and "pre-sonar means fast" is not dependable — only post-init-confirmed measurements should be quoted; the probe cannot separate "publisher silent" from "sim not stepping"; the steady-state 0.19–0.22 is still a ~4.5x cost against the 0.9996 no-sonar control and that cost is unexplained; and Docker has not been re-measured, so its ~0.0018 figure and the llvmpipe hypothesis are untouched. Raw data and write-up: notes/results/multibeam_startup_2026-08-03/; reproduce with bash notes/experiments/go.sh 6 | | 2026-08-03 | Attempted the Docker re-measurement; found the world no longer starts, and found the 2026-07-29 Docker RTF figure was read from the wrong topic | Figure withdrawn — no valid Docker measurement exists | Set out to re-take the Docker RTF with the corrected settle. The world does not run at all in lyrical-theme-test today, on either render engine. As shipped (<render_engine>ogre2 — note the ogre2→ogre patch documented in this README is not applied to the world files in this container) the sonar plugin loads correctly (# of Beams = 513, (301, 1), 399) and then segfaults: Ogre2Scene::PreRender → Ogre2GpuRays::PreRender → CreateGpuRaysTextures → CreateSampleTexture → __memcpy_generic, null address, exit 139, reproduced 2/2. Patched to ogre it dies earlier instead — OgreRenderEngine::LoadPlugins → RenderSystem_GL.so.1.9.0 dllStartPlugin → installPlugin → __cxa_call_terminate → abort, exit 134, never reaching the sensor. Setting DISPLAY=:10 against the container's live xrdp X socket changed nothing (same abort; XAUTHORITY was not set, so this is weak evidence rather than a clean refutation). Note this is not the Failed to load plugin [ogre2] failure recorded elsewhere in this README — ogre2 loads and runs as far as PreRender here. Separately and more consequentially: the 2026-07-29 entry above sampled /world/oceans_waves/stats and asserted that was dave_multibeam_sonar's internal world name. It is not — checked in the world files on both Mac and in the container, dave_multibeam_sonar.world declares <world name="default">. oceans_waves is the internal name of dave_ocean_waves.world and dave_ocean_waves_sonar.world, and that same 2026-07-29 entry records a 4-hour dave_ocean_waves stability run executing concurrently in the same container. The wrong-PID problem was corrected that day; the topic name was not. The ~0.0018 figure and the "Docker is ~123x worse than Mac" comparison built on it are therefore withdrawn. The gdb program-counter evidence from that day concerns the right process and still stands; the RTF numbers do not. Also found: dave_ocean_waves.world and dave_ocean_waves_sonar.world both declare <world name="oceans_waves">, so running them together makes /world/oceans_waves/stats unattributable. Full write-up and stack traces: notes/results/docker_multibeam_crash_2026-08-03/ | | 2026-08-03 | Measured dave_ocean_waves_sonar_integrated on Mac for the first time with the corrected method | Two prior claims refuted | 15-minute isolated run, sonar-init-aware sampling, resolved stats topic. RTF 0.2241 (334 msgs, sim +10.989s / real +49.029s, iterations 183,449→194,438 monotonic). (1) "The empty scene is slower than the heightmap scene" is refuted on Mac: this world and dave_multibeam_sonar measure the same — 0.2241 vs 0.19–0.22 — despite 14 includes plus a Sand Heightmap versus 5 includes and no ground. That claim compared two Docker figures (0.0018 / 0.03), and the 0.0018 was withdrawn the same day as a wrong-world measurement. Scene content does not explain the sonar cost. This kills the premise of exp3_heightmap.sh and the original motivation for exp1_range.sh. (2) The "CPU climbs 32%→47%→69%" finding is not reproduced: after the sonar goes live, 26 consecutive samples over 12.5 minutes oscillate 150.6–201.1% with no upward trend (first half 181.0, second half 175.6), and RSS is flat at 279–336 MiB. There is a step at sonar init (107%→178%) and a ramp before it; the earlier observation ran alongside other gz-sim instances and did not separate the startup window. This removes the stated basis for this world's PARTIAL status — not reclassified yet, since this is a single run. What remains: the ~4.5x sonar cost against the 0.9996 no-sonar control is real, reproducible and unexplained, and now looks fixed per frame rather than scene-dependent, since two very different scenes pay it identically. That yields a testable prediction — range (exp1) should barely matter, ray count (exp1b, 513×301 rays and a 153,600-point FillPointCloudMsg loop on the render-thread callback) should matter a lot. Data: notes/results/integrated_isolated_2026-08-03/ | | 2026-08-03 | Ray-traversal hypothesis tested by sweeping sonar max range on Mac | Refuted — range has no effect | <range><max> 10 → 3 → 1 m, everything else identical, each a fresh launch with cleanup verified and the sonar confirmed live before a 60s window. RTF 0.2186 / 0.2507 / 0.1933 — not monotonic: if range drove the cost, 1 m would be fastest; it is the slowest. The 1.30x spread is barely above the known fixed-setting noise floor (1.19x across 4 repeats on 2026-07-31 and 1.19x across 8 consecutive windows the same day) and has no direction. Combined with the same-day integrated-world result, every axis varied on Mac leaves the number unmoved — scene content, world, and now range all give 0.19–0.25, while sonar present vs absent gives 0.9996 → ~0.22. The ~4.5x cost therefore looks fixed per frame, incurred by having the sensor at all rather than by how much scene it examines. Caveats recorded rather than glossed: n=1 per range, so a small effect could hide in the noise (a large one is excluded, and the non-monotonicity argues against any); and only <max> was changed — <maxDistance> (also 10 in this model) and <raySkips> were not, so if the implementation keys its work loop off maxDistance this test would show nothing either way. That has not been checked in the source yet. Remaining untested axis is ray/point count (513×301 rays, 153,600-point FillPointCloudMsg loop on the render-thread callback) — exp1b, whose sonar-wait pattern had 513 hardcoded and would have silently skipped the two reduced-beam conditions; fixed. Data: notes/results/range_sweep_2026-08-03/ | | 2026-08-05 | Identified what the sonar's ~4.5x cost actually scales with, and replaced the settle criterion that had been invalidating measurements | Root cause found — ~90% of the cost is proportional to ray count | Swept <beams>/<rays>: 153,600 → 0.2180, 30,720 → 0.5017, 7,680 → 0.6632, 960 → 0.6835 (control, no sonar: 0.9996). Working in cost (1/RTF) and subtracting the control, a line fitted through only the two largest points gives overhead = 0.345 + 2.111e-5 × N and then predicts the 7,680-ray point at 0.5066 against a measured 0.5075 — 0.2% error on a point not used in the fit. At the default 153,600 rays that is 10% fixed, 90% ray-proportional. This is the axis that moves the number; scene content, world and range were all flat. Two things it does not settle: extrapolating to zero rays still leaves an RTF ceiling of 0.744, so about a quarter of the slowdown is fixed and unexplained; and linear-in-N is equally consistent with the Render() ray-cast and with FillPointCloudMsg — raySkips (currently 10) separates compute rays from emitted points and is the untested discriminator. n=1 per condition; the 960-ray point sits 0.098 above the line. Method — this was the third attempt and the first valid one. The first two were voided by using log lines as the ready signal: the hardcoded ...for 513 silently skipped reduced-beam conditions; generalising it matched a dummy 1×1×4 allocation and passed at ~20s; [sonar_wgpu] GPU # also passed too early. Sampling iterations directly settled it — the simulator does essentially nothing for the first ~60–75s (8 → 11 iterations between t=30s and t=60s), then steps steadily at ~9,000 per 30s. No log line marks that transition, and over 2026-08-03 their timing drifted from ~163s to ~20s while the frozen phase stayed the same length, which is exactly why they looked valid and then stopped working. common.sh now waits on iterations growth (≥1000 per 15s, twice), giving 47–50s of usable data in a 60s window against 1–12s before. exp1 (range) and exp4 (integrated) were measured under the old criterion and should be re-run. Data: notes/results/raycount_2026-08-05/ | | 2026-08-05 | Re-ran exp1 (range) and exp4 (integrated) under the stepping criterion | One conclusion corrected, two confirmed | Both had been measured on 2026-08-03 with the log-line settle that was later shown to fire while the simulator is idle. Correction — range has a small real effect, not none. New: 10 m → 0.2100, 3 m → 0.2156, 1 m → 0.2344, i.e. monotonic and in the expected direction, where 2026-08-03 gave a non-monotonic result recorded as "no measurable effect, the spread is noise". In overhead terms (1/RTF − 1/0.9996) a 10x range reduction removes 13%; for scale, a 5x ray reduction removes 72%. So range matters roughly an order of magnitude less than ray count and is not the explanation for the ~4.5x slowdown — but "irrelevant" was too strong and is withdrawn. (The earlier caveat stands: only <max> was changed, not <maxDistance> or <raySkips>.) Confirmed — scene content is irrelevant: integrated measures 0.2197 (317 msgs, iterations 173,615→183,879 monotonic) against 0.2241 on 2026-08-03 and 0.19–0.22 for the empty world, despite 14 includes plus a heightmap versus 5 and no ground. Confirmed — no CPU climb and no leak: from the point stepping genuinely starts (t ≥ 150s, n=26 over 12.5 min) CPU averages 170.7% with no trend (first half 172.6, second half 168.9) and RSS falls 405 → 281 MB. Bonus: this run shows the log/stepping gap in a second independent signal — sonar_up flipped at t=60s but CPU held ~107% until t=120s and only jumped to 192.9% at t=150s. Data: notes/results/remeasure_2026-08-05/ | | 2026-08-05 | Separated the sonar compute stage from the raycast using raySkips | Compute is not the bottleneck — ~8% of the overhead | exp1b showed ~90% of the cost scales with ray count, but it varied <beams>/<rays>, which changes the raycast resolution, the compute workload and the emitted point count together. <spec><raySkips> subsamples at the compute stage only, leaving the raycast at 513×301. Effective ray counts confirmed from the plugin's own log (301 / 31 / 11 / 4 / 2), so the parameter demonstrably took effect. The response is strongly asymmetric. Increasing compute ~10x (raySkips=1, 301 rays) raises overhead 3.785 → 6.559, +73% — the positive control passes. Reducing compute 15.5x (31 → 2 rays) changes nothing: overhead spans 3.635–4.649, a 1.28x spread against a 1.19x fixed-setting noise floor, with no trend. A line through the two endpoints gives overhead = 3.466 + 0.01028 × compute_rays, so at the default 31 rays compute contributes 0.319 of 3.785 — 8% and 92% is independent of it. Combined with exp1b: cutting beams×rays 5x (raycast + compute + points) gave −72%; cutting compute alone 15x gave nothing. The large gain therefore came from the GPU raycast and/or the PointCloud2 fill, not the signal-processing path (backscatter/matmul/FFT) — which had been the leading suspect and is real work (~93.7 ms per GPU frame) but does not govern RTF here. Still open: raycast vs point fill are coupled by <beams>/<rays> in the SDF and no knob separates them, so the next step is profiling or reading MultibeamSonarSensor.cc, not another sweep. Also unexplained: the 4- and 2-ray points measure ~1.1 above the fitted line, higher than the default configuration — likely noise at n=1, but not accounted for. Data: notes/results/rayskips_2026-08-05/ | | 2026-08-05 | Found the workspace builds with no optimisation, and rebuilt the sonar packages Release | Two thirds of the sonar cost was build configuration | CMAKE_BUILD_TYPE and CMAKE_CXX_FLAGS are both empty in every one of the 14 DAVE packages' CMakeCache.txt, and no package CMakeLists.txt sets them — so the compile line carries no -O flag. Rebuilding multibeam_sonar + multibeam_sonar_system with -DCMAKE_BUILD_TYPE=Release (verified: grep -o '\-O[0-3s]*' compile_commands.json → 4 -O3) and re-running the ray sweep unchanged: at the default 153,600 rays RTF goes 0.2180 → 0.4380, a 2.01x speed-up, removing 64% of the overhead over the no-sonar control. This is the discriminator the SDF could not provide: gz-rendering, which owns the GPU raycast, was not rebuilt, so the gain is plugin-side by construction — and the marginal cost per ray fell 6.1x (2.111e-05 → 3.459e-06). The ray-proportional cost is therefore in plugin code, consistent with FillPointCloudMsg; ComputeSonarImage also got faster but exp7 already showed it is async and off the critical path. The relationship also changed shape: as-shipped it was strikingly linear (a two-point fit predicted a held-out third to 0.2%), under -O3 it is sub-linear, marginal cost/ray falling ~10x from the small end to the large end — plausibly vectorisation and per-frame amortisation, but that is an interpretation of four single measurements, not established. Consequences: the README's reproduction section produces an unoptimised build, so anyone following it measures what we measured all day; the 0.19–0.22 figures recorded 2026-07-31 and 2026-08-05 for this world describe the unoptimised build and are superseded for Release (0.438). Relative conclusions from earlier experiments are unaffected — each compared within one build. A real cost remains: 0.438 against 0.9996 is still ~2.3x, so build configuration explained roughly two thirds of the gap, not all of it. Caveat: the as-shipped build was never directly confirmed to be -O0 — inferred from the empty cache entries, with no compile_commands.json to read. What is measured is "Release is 2x faster", not "the old build was -O0". Data: notes/results/release_rebuild_2026-08-05/, code reading: notes/results/sonar_code_reading_2026-08-05/ | | 2026-08-05 | Profiled the running process instead of inferring from RTF deltas | The leading hypothesis was wrong | sample(1) on the live gz-sim, 10s at 1ms, no rebuild. FillPointCloudMsg accounts for 1.0% of CPU — the redundant loop-invariant trig identified by the code read is real, but sits inside a function that costs 1%, so the planned hoist would not have moved RTF. Profiling before patching is what caught this; the reverse order would have cost a code change, a rebuild and a measurement to learn the same thing. It also retires the release_rebuild note's attribution of the 6.1x per-ray drop to that function. What appeared instead: 49.3% of busy CPU in spin-waits — 21.8% Intel TBB work-stealing across seven workers on an 8-core machine, 16.1% Fast DDS shared-memory transport, 4.3% OGRE barriers — with the main thread only 5.4% busy. Method corrections made the same day: the first summary used grep -c, which counts distinct call paths rather than samples; inclusive time initially included blocked time (gz-sim read 124% of busy); idle threads must be excluded or the totals measure sleep. All replaced by notes/experiments/analyze_profile.py. Data: notes/results/profile_2026-08-05/ | | 2026-08-06 | Attributed the TBB pool, by controlled comparison | It is the sonar's, via OpenCV | Nothing in DAVE, gz-sim, gz-rendering or DART links TBB; libopencv_core does, and the sonar draws its display image with OpenCV. A world without the sonar has zero TBB workers (grep -c stealing_loop_backoff → 0) and 8.6% spin against ~48%, at roughly 1/15th the CPU. So the seven spinning workers are this plugin's. This made cv::setNumThreads(1) a candidate with profiler evidence behind it, unlike the trig hoist — though it does not show the spinning costs RTF, which needed the before/after below | | 2026-08-06 | Root-caused the intermittent spawn hang that had been blocking measurement | Fast DDS shared-memory transport, not DAVE or Gazebo | ros_gz_sim create had been hanging with no output, leaving a world running at RTF ~1.0 with no sensor in it — a failure that looks like success. sample-ing the hung process (the one diagnostic not yet tried) put 4218 of 4218 samples on one stack: rclcpp::Node::make_shared → rmw_create_node → rmw_fastrtps_cpp::create_subscription → DataReaderImpl::enable() → RTPSDomain::createRTPSReader. It blocks constructing its own ROS node and never reaches the model SDF or any Gazebo code. FASTDDS_BUILTIN_TRANSPORTS=UDPv4 avoids it: 5/5 spawned with shared memory disabled, 1/9 with it enabled. Within the previous day's thread sweep the correlation was exact 4/4 — the two conditions that happened to set the variable, for an unrelated reason, were precisely the two that spawned; that was in the data for hours before it was read that way. Now the default in common.sh (SHM=1 reverts). Still open: why it is intermittent, and whether stale segments from pkill -9 are the trigger. Data: notes/results/spawn_hang_2026-08-05/ROOT_CAUSE.md | | 2026-08-06 | Re-established both baselines after the DDS switch and the settle fix | Sonar cost is 1.90x, not 4.5x | Every earlier RTF figure went out of scope for comparison: the DDS transport changed, and the sonar-ready check was fixed (the old one matched the plugin's dummy 1 beams warm-up frame and passed at 0-20s instead of waiting for the real sensor at ~110s). Re-measured with repeats: no-sonar control 0.9974 (n=3, spread 0.2%), with sonar 0.5243 (n=3, spread 7.9%). The denominator barely moved (0.9996 → 0.9974), so essentially all of the change since 2026-07-31 is in the numerator. The control is far more reproducible than the sonar — 0.2% against 7.9% — so the run-to-run variation lives in the sonar path, not the measurement method, and tightening the protocol further will not shrink it. The 5% band quoted from an n=2 estimate was optimistic; 8% is the working threshold. Data: notes/results/baseline_udp_2026-08-06/ | | 2026-08-06 | Tried three interventions; the one that backfired showed why | Frequency is the knob, not speed | cv::setNumThreads(1): 0.5243 → 0.5054, inside the band, no effect. Skipping the OpenCV visualisation: 40% worse (0.3012), reproducibly across three runs, with spread rising 5% → 21%. The plugin's frame counter explains it — over comparable runs it reached GPU #550 as shipped against #1300 with the image skipped, 2.4x more sonar frames. The compute thread runs as fast as it can and takes whatever that requires, so cheaper frames just means more of them; exp7 found the same axis from the other side when raySkips=1 cost +73%. So <update_rate> was tried, and it works: 30 Hz (shipped) 0.5243 → 10 Hz 0.5984 (n=3) → 2 Hz 0.8140 (n=2), removing 26% and 75% of the sonar's overhead, monotonic, both clearing the 8% band, with no code change and no rebuild. The shipped 30 Hz is unreachable — the sensor achieves ~2.8 frames/s — and above the physical device (~15 Hz). This also shows profiler share is not RTF cost: TBB was 21.8% of busy CPU and both attempts to reduce it failed, one of them backwards. Data: notes/results/updaterate_2026-08-06/, draft: notes/updaterate-issue-draft.md | | 2026-08-06 | Fixed six measurement-tool defects, two of which had already corrupted results | All of them silently produced wrong numbers | (1) The sonar-ready check matched the dummy 1 beams frame — the second time a string pattern lost to that line; it now parses the reported beam count and requires >1. (2) wait_until_stepping called gz topic -e -n 1 with no timeout, so when /stats went quiet during sonar init the enclosing loop never advanced and its own MAX never fired — indistinguishable on screen from a hang. (3) The first fix used a 10s timeout, too short once the sonar is up, so it reported "no stats" forever; it now uses 40s and computes steps/sec from wall clock rather than assuming a fixed interval. (4) assert_model_spawned treated "could not ask" as "not there" — gz model --list times out on /world/<name>/state even on an empty world before any spawn. (5) cleanup sent SIGTERM and returned without checking; when a process survived, the next repeat either aborted or shared the machine with it. That cost a 40-minute run — 4 of 6 repeats lost, and the two that completed came in at roughly half the values measured cleanly an hour earlier. It now escalates TERM → verify → KILL → verify and fails loudly. (6) rtf_probe.sh accepted impossible values: one repeat reported real +947s over a 60s window; left in, it would have moved a 2 Hz mean from 0.814 to 0.556 and the spread from 6.2% to 143.9%, reversing the conclusion. It now rejects real_delta > 1.5x window and sim > real |

| 2026-08-07 | Reclassified the two PARTIAL rows in the validation matrix, whose stated grounds no longer held | 17/18 PASS-level, 1 PARTIAL | Both rows still carried reasons that later work had refuted, so the matrix was asserting problems no longer supported by evidence. dave_ocean_waves_sonar_integrated: PARTIAL → SMOKE PASS. Its two grounds were an RTF ~0.03 degradation and a 32→47→69% CPU climb. The first came from a 20 s Docker sample; Mac measurements under the corrected method give 0.2241 and 0.2197, the same as the empty sonar world, so it is the sonar's cost and not specific to this world. The second is not reproduced in two independent isolated runs — CPU flat over 12.5 minutes (halves 172.6 vs 168.9) and RSS falls 405 → 281 MiB; the original was taken with two other gz-sim instances on the same container and without separating the startup window. Classified SMOKE PASS rather than FUNCTIONAL PASS because this world's sonar output topics have never been echoed — only /world/.../stats was read — matching dave_ocean_waves_sonar.world, same family. dave_multibeam_sonar: PARTIAL retained, basis replaced. The recorded reason — "simulation-progress stall / crawl" — is no longer true: the stall was the startup window, and the crawl figures came either from inside it or, for Docker, from the wrong world and were withdrawn. On Mac the cost is now characterised (1.90x, and 0.8140 at <update_rate> 2 Hz) — a configurable sensor cost, not a defect. It stays PARTIAL for a different and currently-true reason: the world does not start in Docker at all, segfaulting in Ogre2GpuRays::CreateSampleTexture() as shipped and aborting in RenderSystem_GL when patched to ogre, so no valid Docker measurement exists or can be taken until that is resolved |

| 2026-08-07 | Closed the vehicle axis of the validation matrix — the world axis had been complete for weeks, the vehicle axis never was | No vehicle's IMU had ever reached ROS | REXROV was the only vehicle ever used as a smoke-test subject; BlueROV2 had its ArduSub/mavros build confirmed but no in-world behaviour checked, and BlueROV2 Heavy had never been used. Checking sensor output rather than liveness found a defect present the whole time: four base models declare imu_sensor with no <topic> — and a fifth, bluerov2_heavy_multibeam_sonar, was source-inspection-only at the time (found 2026-08-21 and then runtime-tested on 2026-08-27, so it remains outside the original before/after), so Gazebo publishes on its default path while every robot_config.py bridges /model/<ns>/imu — the ROS topic exists and is permanently silent. Measured before and after on the running simulation: all four silent before, all four publishing after one line per model (patches/vehicle_imu_topic_fix.diff). The control is inside the same files — rexrov's magnetometer and bluerov2's camera declare <topic> and work. After the fix rexrov is 7/7 and glider_slocum 6/6 FUNCTIONAL in the state/sensor scope; the extractor explicitly removed all joint/... paths, while bluerov2 and bluerov2_heavy are 4/5, the gap being a magnetometer topic their configs bridge for a sensor the models never declare. A smoke test cannot see any of this — the vehicle spawns, physics steps, odometry/pose publish, and the process stays up. Two errors in the test script, both the same shape: a hard-coded four-topic list flagged glider_slocum (which bridges navsat, not a magnetometer), and the regex that replaced it truncated at /, turning camera/image and battery/battery/state into camera and battery and flagging rexrov and glider_slocum too. Both invented a missing topic and then reported it; removing one hard-coded assumption introduced another. glider_slocum came in at 6/6 state/sensor FUNCTIONAL on a re-run after one transient NOT_STEPPING; this did not include its three joint/actuator paths. Data: notes/results/vehicles_2026-08-07/, draft: notes/vehicle-imu-topic-issue-draft.md |

| 2026-08-07 | Ran a stock gpu_lidar with no DAVE code, to answer the question blocking the Docker crash report | The two Docker failures have different causes and only one is DAVE's | The Docker draft had carried a "Before filing" block since 2026-08-03: a maintainer would ask whether a plain GpuRays sensor fails the same way, and the report should not go out until that was answered. gpu_lidar_probe.world contains no DAVE code — base systems, a ground plane, a box, and one gpu_lidar at the sonar's own 513 × 301. On ogre2, in the same container where the sonar segfaults, it initialises and publishes 3/3 (and 3/3 on the Mac control). The first version of that probe was not a matched comparison — it copied the sonar's ray count but used arbitrary angles, where the sonar specifies ±1.13447 horizontal (130°, past the 120° point where Ogre2GpuRays switches to a cubemap path) and ±0.10472 vertical (a field of view 2.5× narrower). Re-run with every ray-geometry parameter matched — 513 × 301, both angle ranges, range 0.1–10.0 — the stock sensor still runs. So the crash is not explained by ray count, horizontal FOV, vertical FOV or range; what remains different is the construction path (type="custom" calling Scene()->CreateGpuRays() directly, SetClamp(false), odd-forced ray counts), the presence of a camera and depth camera on the same model, and scene content — none of which was tested. On ogre, the stock sensor fails too, with Couldn't open X display in GLXGLSupport::getGLDisplay: OGRE1's GL render system needs an X display and a docker exec session has none. That half is not DAVE's, and it invalidates the documented ogre2 → ogre workaround for any headless world with a rendering sensor, not just this one. Liveness alone would have proved nothing — a world whose sensor was never created also stays alive, which is exactly what happened on 2026-08-06 — so the probe checks that /gpu_lidar actually publishes and records the ogre rows as ALIVE_NO_DATA rather than passes. Honest limit: the stock sensor is built by gz-sensors from <sensor type="gpu_lidar"> while the sonar is type="custom" and constructs GpuRays itself; both reach Ogre2GpuRays but not by the same path, so this narrows the suspect without identifying the call. Data: notes/results/gpu_lidar_probe_2026-08-07/, draft updated: notes/docker-sonar-crash-issue-draft.md |

| 2026-08-07 | Gave the Docker container a properly authorised X display and re-ran the sonar world | It runs. World-level matrix: 18/18 PASS-level, 0 PARTIAL | The gpu_lidar probe above showed OGRE1 needs an X display; the container has one (Xorg :10, owned by user docker, .Xauthority mode 600). Run as -u docker with DISPLAY=:10 and XAUTHORITY=/home/docker/.Xauthority, with the world patched ogre2 → ogre: no crash across 300 s, sensor initialised (513 beams / 301 rays / 399 time data), iterations reached 72,107, and /sensor/multibeam_sonar/point_cloud published — the first direct confirmation of this sensor's PointCloud2 output under Lyrical+Jetty, closing the traceability gap flagged 2026-07-23. This supersedes the 2026-08-03 finding that the world "does not start on either render engine": the 2026-08-03 attempt set DISPLAY but not XAUTHORITY and ran as root, so the connection was refused. That entry's own caveat — that XAUTHORITY was untested and the evidence was therefore weak — turned out to be the entire answer. dave_multibeam_sonar reclassified PARTIAL → FUNCTIONAL PASS, leaving the matrix with no PARTIAL rows. Two things stay open and moved to Known issues rather than holding the row back: as shipped (ogre2) it still segfaults in Docker, and under this X configuration WGPU cannot obtain a GPU adapter (tries vulkan, then all, panics) and falls back to the CPU compute backend — so the confirmed output is not from the WGPU path PR #44 is about. n=1 for the published-data result; message content was not inspected; no RTF was measured in this configuration. Data: notes/results/docker_sonar_x_display_2026-08-07/ |

| 2026-08-11 | Recorded a Mac PointCloud2 echo in the session notes; the raw echo artifact was not retained | WGPU path confirmed; echo artifact gap remains | Since 2026-07-23 the Purpose section had recorded that Mac under Lyrical+Jetty had build, adapter-selection and /stats confirmation but no direct PointCloud2 topic-echo — a gap in the headline claim of the whole repository. The contemporaneous session note records /sensor/multibeam_sonar/point_cloud with: frame_id: blueview_p900::blueview_p900_base_link::multibeam_sonar, seq: 1291, FLOAT32 x/y/z fields, while the plugin logged GPU #200 | 93.9 ms | 513 beams × 31 rays × 399 freq. This is the WGPU GPU path, unlike the Docker confirmation on 2026-08-07, which came from the CPU fallback backend — so the platform where PR #44's actual backend is exercised now has direct sensor-output evidence. Session-note limit: the recorded echo was truncated at 20 lines, so the header and field definitions were read but width/height/point_step were not captured, and message values were not validated against ground truth — this confirms structured output with the expected frame and fields, not correctness. Evidence limit: the committed Mac log contains the launch, Metal adapter and plugin output but not the ros2 topic echo body. |

| 2026-08-21 | Checked the SeaPressure plugin against a running simulation, had the first write-up's evidence rejected in review, and re-ran it properly | Two runtime confirmations: a 1000x unit error, and <noise_sigma> ignored | The wiki correction on 2026-08-20 had recorded, from reading sea_pressure_sensor.cc, that the plugin computes in kPa and assigns straight into sensor_msgs/msg/FluidPressure, whose definition reads # Absolute pressure reading in Pascals. Reading is not seeing, so REXROV was launched at z:=0 in dave_ocean_waves: fluid_pressure: 101.32505915145917 where the message contract implies 101325. The factor of 1000 is measured. headless:=true turned out to be load-bearing rather than incidental — the default GUI launch exits -6 here, so a reproduction without it is not the procedure that produced the number. The first write-up's second claim was rejected in review, and correctly. variance: 9.0 had been presented as independent confirmation that noiseSigma sits at its compiled-in default, and therefore that Configure() ignores <noise_sigma>. But rexrov/model.sdf:896 also specifies 3.0 — the SDF value and the default are the same number, so the observation is equally consistent with the tag being honoured. It discriminated nothing. That is the fifth recorded instance of accepting a signal that resembles the target, and the first caught before publication. The experiment that does discriminate was then run: the tag was set to 0.123, the model relaunched, and variance came back 9.0 — against 0.015129 if the tag were applied — about 595×, nearly three orders of magnitude. <noise_sigma> is ignored at runtime, and the claim now rests on a measurement rather than on the absence of a parsing branch. It was right all along; the evidence for it was not, and that distinction is the finding. ros2 topic type was captured in the same session (sensor_msgs/msg/FluidPressure), closing a smaller version of the same error — the first note had shown that output as recorded when it never was. Model restored to 3.0, backup removed, both verified. Two claims deliberately left source-only: saturation is parsed and never read, and the Gaussian noise is commented out above a += 0.0; neither was tested by setting a value and watching nothing happen, and both drafts say so. The variance semantics were also walked back in the other direction — FluidPressure.msg reserves 0 for variance unknown, so "stop publishing a non-zero variance" was bad advice, and the reports now describe what is observable and note the intent is undocumented rather than asserting a filter will misbehave. Cross-checked upstream: all four findings are present on IOES-Lab/dave's ros2 default branch at cc98a539, read directly, so neither report needs a fork-specific hedge. Unsent upstream total now eight; issue 7 goes first and alone. Data: notes/results/seapressure_unit_2026-08-21/ | | 2026-08-25 | Multibeam Wiki 명령과 예제를 직접 재검증 | 출력 PASS, 문서 예제 2건 PARTIAL/FAIL | Demo wrapper·원본 dave_sensor.launch.py·Local Search는 Mac Metal WGPU에서 PointCloud2 513×301과 image/raw sonar 513×399를 발행했다. 별도 overlay의 사용자 정의 센서/world도 65×61/65×319 출력을 냈다. 기본 RViz는 프로세스와 구독은 살아 있지만 macOS에서 visible=true, windows=0; BlueROV2 명령은 차량만 spawn하고 선택 모델에 multibeam sensor가 없어 소나 예제로는 실패했다. Data: results/multibeam_direct_validation_2026-08-25/ | | 2026-08-25 | CPU와 WGPU를 같은 3.99 m 평면 표적에서 수치 비교 | CPU PASS / WGPU FAIL (한 장면) | 두 backend의 PointCloud centre는 동일하게 3.990244 m였다. CPU raw sonar는 5/5 프레임에서 3.988294 m가 최댓값이고 기대 bin rank 1이었다. WGPU Metal은 5/5 프레임에서 6.396–6.446 m가 최대였고 기대 bin rank는 118–316이었다. 측정한 중심 거리만으로 불일치 위치를 확정할 수는 없으며 원인은 미확정이고, 일반 음향 정확도 검증으로 확대하지 않는다. | | 2026-08-25 | Docker RDP 강제 CPU 및 Apple M2 CUDA 부재 경로 확인 | Docker GUI/raw 출력 PASS; CUDA 실패 처리 FAIL | Docker xrdp에서 Gazebo GUI와 강제 CPU raw sonar 513×399를 직접 확인했다. 같은 commit·Release·CPU backend의 전체 발행 주기는 Mac 약 2.71 Hz, Docker 약 0.398 Hz였지만 renderer/VM 조건이 달라 순수 CPU kernel 비교는 아니다. Apple M2에서 명시적 CUDA 요청은 clean rejection 대신 Gazebo -6 종료와 launch/bridge 잔류를 만들었다. 별도로 debug:=true verbosity_level:=4는 -r-v 4를 생성해 exit 109였다. | | 2026-08-25 | Ocean Current Wiki 절차와 전역 Hydrodynamics 경로를 직접 재검증 | FUNCTIONAL PASS (scoped) | Mac Lyrical+Jetty에서 /hydrodynamics/ 사용자 서비스 12개의 타입과 호출을 확인했다. Constant/Stratified/Gauss-Markov 정상 입력은 ROS·Gazebo 출력에 반영됐고, invalid layer/model 입력은 success: false, 원래 값은 복원됐다. 동일한 REXROV spawn과 8.98 s 구간에서 zero current Δx −0.01683 m, +1.5 m/s X current Δx +9.03405 m로 paired difference +9.05087 m였다. 이는 전역 /ocean_current 경로의 물리 효과이며, 주석 처리된 OceanCurrentModelPlugin의 차량별·깊이별 경로는 미검증이다. 첫 launch는 GZ_SIM_SYSTEM_PLUGIN_PATH에 lib/<package>/가 들어 있고 dylib는 lib/에 있어 실패했으며, 실제 lib 경로를 추가해 실행했다(그 항목의 출처는 미확정 — known-issues.md). Data: results/ocean_current_direct_validation_2026-08-25/ | | 2026-08-25 | OceanCurrentModelPlugin 차량별 경로를 직접 검증 | FUNCTIONAL PASS (scoped) | Hydrodynamics <namespace>와 플러그인 블록을 복제한 테스트 자산에서만 활성화했고 DAVE checkout은 수정하지 않았다. 전역 constant current를 끈 상태에서 차량별 토픽 /model/<name>/ocean_current가 전역 /ocean_current와 분리돼 나타났다. 층 중점에 놓은 정지 프로브 2개가 5 m (0.005, 0.040, 0.0), 15 m (0.010, 0.025, 0.0) m/s를 오차 0으로 읽었다 — 이 중 셋은 데이터베이스에 없는 값이라 조회나 층 index 오류로는 나올 수 없고, 15 m의 north 성분만 양쪽 층이 모두 0.010이라 판별력이 없다. 이어서 +1.5 m/s X 해류가 REXROV를 zero 대비 +11.564 m 움직였다(조건당 n=1, 창 8.98/8.99 s). 깊이에 따른 힘 차이는 여전히 미검증 — 보간은 정지 프로브로 쟀고 운동 시험은 12개 층을 같은 값으로 뒀다. Data: results/ocean_current_model_plugin_validation_2026-08-25/ | | 2026-08-26 | Underwater Camera Wiki Quickstart과 파라미터를 Mac·Docker에서 직접 검증 | 출력 FUNCTIONAL PASS / 파라미터 의미론 PARTIAL | 두 플랫폼 모두 /underwater_camera/simulated_image를 sensor_msgs/msg/Image 320×240 bgr8 step 960, 230400 바이트로 발행했다. 정적 장면 하나(최근접 3.975 m)에서 세 조건의 중심 BGR은 [55, 78, 109](no effect)·[48, 68, 95](defaults)·[83, 103, 74](murky)였고, 조건·플랫폼당 5프레임이 모두 동일했으며 보존한 대표 .npy 3개는 Mac과 Docker가 바이트 단위로 같았다. 기본 1/30 감쇠와 background 0은 [48.175, 68.320, 95.473]을 예측해 런타임과 일치했다. 다만 attenuationR/backgroundR가 Blue에, attenuationB/backgroundB가 Red에 적용된다 — 입력이 OpenCV BGR로 변환되는데 SDF 값은 태그 순서로 배열에 저장되고 루프가 그 index를 cv::Vec3b에 그대로 쓴다. murky 조건이 이를 판별한다: 직접 인덱싱 [83, 103, 74] 대 의미상 매핑 [50, 103, 85], 관측은 전자. green은 양쪽 모두 103이라 판별력이 없다. 일반 수중 광학 정확성은 검증하지 않았다 — 장면 1개, 원본 색 1개, murky 태그 6개를 동시에 바꿨다. UnderwaterCamera.cc 자체는 6aef91c에서 미수정이지만 checkout 전체가 pristine인 것은 아니다. Data: results/underwater_camera_direct_validation_2026-08-26/ | | 2026-08-26 | DVL Wiki Quickstart과 bottom/velocity/water-mass 경로를 Mac·Docker에서 직접 검증 | Docker 기능 PASS / 전체 PARTIAL | Docker에서 정확한 Wiki Quickstart가 DAVE DVL을 8 Hz로 발행했고, 평면 bottom range 20/20이 해석적 beam envelope 안에 들었으며, 1 m/s 플랫폼 이동이 ENU→FSD 예상 축으로 복원됐다. 공식 ros_gz bridge도 populated marine_acoustic_msgs/msg/Dvl을 발행했다. 배포 water-mass 태그는 environmental-variable 이름이 아니라 0.이라 20/20 unspecified/no-lock이었지만, 이름과 동적 환경 데이터를 준 통제는 20/20 water-mass/4-lock과 기대 속도를 냈다. 전체 PARTIAL: Mac의 Wiki GUI·headless DAVE·공식 ros_gz demo·ogre 통제는 모두 Gazebo Sensors render thread에서 종료했고, DAVE custom bridge는 Gazebo에 있는 frame ID를 ROS에서 비운다. 일반 DVL 정확성은 검증하지 않았다. Data: results/dvl_direct_validation_2026-08-26/ | | 2026-08-26 | SeaPressure를 Mac·Docker 10조건 행렬로 직접 재검증 | PARTIAL — 발행·일부 설정 PASS, 수치·설정 의미론 FAIL | 두 플랫폼의 압력·variance·depth-topic 존재 여부가 10/10 조건에서 일치했다. standard_pressure·kPa_per_meter·topic·estimate_depth_on은 예상대로 동작했다. 반면 ROS Pascal 필드에 surface 101.325, |z|=10 m에서 199.3888이 들어갔고, saturation=50은 clamp하지 않았으며, noise_sigma=0.123은 variance 9.0을 바꾸지 않았고, update_rate=2도 약 0.001 s 주기를 바꾸지 않았다. z=+10과 z=-10이 같은 pressure/depth를 내 abs(z) 경로를 확인했고, 모든 retained ROS pressure frame의 header.frame_id는 비어 있었다. 첫 full-matrix 시도는 태그 중복과 실패한 cleanup으로 조건 격리를 깨뜨려 제외했고, 한 서버·고유 namespace·단일 태그 runner로 다시 실행했다. Data: results/seapressure_full_validation_2026-08-26/ | | 2026-08-26 | Spherical Coordinates Wiki 예제와 네 서비스를 Mac·Docker에서 직접 재검증 | 유한 왕복 PASS / 전체 PARTIAL | 실제 설치 라이브러리를 찾을 수 있는 조건에서 get/set/restore와 유한점 3개 왕복을 양쪽 플랫폼에서 실행했다. 최대 축 오차는 9.71e-10 m, 플랫폼 간 최대 수치 차이는 1.01e-9였다. 현재 dave_bimanual_example 원점은 Wiki와 다르고, Wiki가 (100,200,3)의 결과로 적은 값은 역변환하면 약 (-100,-200,3)이 된다. NaN은 NaN으로 반환되고 latitude 100°/longitude 200°도 success=true로 수락됐다. Mac 기본 생성 경로에서는 네 서비스가 빠졌지만 실제 lib/ 추가 후 나타났고, Docker 기본 환경에서는 install/lib가 LD_LIBRARY_PATH에 있어 수동 Gazebo 경로 추가 없이 나타났다. 첫 GUI 시도는 제한된 실행 하네스가 소켓·화면을 막아 판정에서 제외했다. Data: results/spherical_coordinates_direct_validation_2026-08-26/ | | 2026-08-27 | USBL Wiki Quickstart와 common/individual 경로를 Mac·Docker에서 직접 재검증 | FUNCTIONAL PASS WITH REQUIRED WORKAROUNDS | 양쪽 플랫폼에서 common mode가 transponder 1·2의 spherical/Cartesian 데이터를 각각 6개씩 냈고, individual channel 1/2는 선택한 ID만 각각 3개씩 냈다. positive-sigma retained run의 정적 좌표 최대 축 오차는 0.000258 m. Paused control은 endpoint가 모두 보여도 출력 0이었고, literal sigma=0은 Mac/libc++에서는 finite output이나 Docker/libstdc++에서는 첫 ping에 exit 134로 abort했다. 기존 Wiki dave_sensor.launch.py namespace:=usbl 명령은 world-embedded plugin 때문에 출력은 나지만 없는 description/usbl/model.sdf spawn 오류를 함께 내므로, 양쪽에서 검증한 dave_world.launch.py world_name:=usbl_tutorial로 교체했다. 일반 음향/travel-time 정확성은 미검증. Data: results/usbl_direct_validation_2026-08-27/ | | 2026-08-27 | Re-ran Dave ROV Models exact and isolated paths on Mac and Docker | REXROV 7/7 scoped PASS; Blue/Heavy/fifth variants PARTIAL; standalone inputs PASS | Patched Mac dave_ocean_waves produced all seven configured REXROV messages with content. Stock empty.sdf spawned REXROV but Docker returned 4/7 because the world lacks a Sensors system and the image lacks the IMU patch. The formerly source-only bluerov2_heavy_multibeam_sonar variant was launched on both platforms: odometry published, but IMU, magnetometer and sonar PointCloud2 were silent for 120 s. Docker keyboard-TTY and WebSocket Joy controls produced discriminating messages. The exact integrated BlueROV2 launch reached entity, WebSocket, Firefox, QGC and ArduSub startup, then failed because mavros_msgs/mavros are absent; no vehicle-control loop was claimed. Evidence: results/rov_direct_validation_2026-08-27/ | | 2026-08-27 | Audited all 18 Dave World Models files and reran the Wiki Quickstart on Mac and Docker | Quickstart liveness PASS / naming audit FAIL (3 duplicate groups) | The current matrix remains 5 FUNCTIONAL PASS + 1 FUNCTIONAL PASS WITH REQUIRED WORKAROUNDS + 11 SMOKE PASS + 1 PARTIAL. The full source/install inventory found only 14 internal names: 7 files share oceans_waves, default, or dvl_world. Mac and Docker both produced advancing /world/oceans_waves/stats; the retained single samples are liveness evidence, not a benchmark. All 18 files contain remote Fuel URIs (128 includes total). Only dave_ocean_waves was rerun that day; no SMOKE row was upgraded. Data: results/world_models_audit_2026-08-27/ | | 2026-08-27 | Re-ran both Dave Glider Models Wiki examples and separated state/sensor coverage from actuator coverage | State/sensor FUNCTIONAL PASS with local IMU patch; actuator PARTIAL | Docker RDP launched both empty.sdf and dave_ocean_waves and listed all nine configured bridge topics. The as-shipped empty-world example published 5/6 state/sensor messages with only IMU silent; the local one-line IMU <topic> overlay published 6/6 in the ocean world. The earlier 2026-08-07 6/6 helper ended with grep -v '^joint', so its six never included cmd_thrust, ang_vel or enable_deadband; wording that called it a thruster-inclusive pass is withdrawn. A direct ROS cmd_thrust=25.0 command appeared on Gazebo as data: 25 and changed ang_vel from -0 to 55776.343531645027, establishing command coupling but not actuator accuracy. Integrated Gazebo→ROS enable_deadband=true passed while ROS→Gazebo timed out; an isolated Bool bridge passed both directions, leaving the integrated path unresolved. Fresh Mac attempts for both examples were NOT_STEPPING and are retained as inconclusive, not as a regression verdict. Data: results/glider_direct_validation_2026-08-27/ | | 2026-08-27 | Directly validated the Object Models Wiki launch, generic Fuel include, source extension and missing-descriptor path | Spawn paths FUNCTIONAL PASS / reporting and reproducibility PARTIAL | The only packaged descriptor, mossy_cinder_block, passed the exact Wiki command on Mac and Docker. The generic Teledyne URL downloaded, validated and spawned in minimal worlds on both platforms. A copied dave_object_models source package with description/codex_versioned_block/ then built, installed and spawned on both; its ten resolved cache files had identical hashes. However, direct probes on both platforms warned that requested version /1 is ignored because only latest tip is supported, so this is not immutable pinning. A missing descriptor still produced server errors and no entity while the client exited 0 and the handler printed success. Data: results/object_models_direct_validation_2026-08-27/ | | 2026-08-28 | 전체 미검증 범위의 실행 가능성·플랫폼 경계 재감사 | 실행 가능한 항목 직접 시험 / 나머지 BLOCKED 명시 | Mac·Docker에서 가능한 잔여 항목을 실행하고, NVIDIA CUDA·Windows/WSL·MAVROS/QGC 통합·실물 USB/gamepad/해양·Fuel account upload는 prerequisite 부재로 BLOCKED 처리했다. Data: results/full_gap_validation_2026-08-27/00_environment_boundaries/ | | 2026-08-28 | Spherical Coordinates를 독립 WGS-84 oracle로 검증 | 유효 입력 PASS / API·설치 PARTIAL | 4지역 13점의 최대 고도 오차 5.16e-7 m, inverse ENU axis 1.80e-7 m. NaN·범위 밖 입력과 plugin path는 남음. | | 2026-08-28 | 배포 DVL descriptor 8개 전부 Docker 실행 | 4/8 output, 4/8 initialization FAIL | 네 descriptor의 100 m water-mass far boundary가 최대 range 75/81/66/90 m를 넘어 초기화 실패. | | 2026-08-28 | Underwater Camera 태그 6개 개별·거리 4단계 검증 | range trend PASS / R·B 의미론 FAIL / fresh Mac 재현 실패 | Docker 12조건은 태그 index와 1·2·4·6 m 감소를 확인. fresh Mac은 image topic 미발행이라 이전 성공과 분리 보존. | | 2026-08-28 | SeaPressure paused·±1000 m·10,000 frame 통제 | 실행 안정성 PASS / 계약 결함 유지 | paused 5초 0프레임, surface 10,000개 1 kHz monotonic, ±1000 m 모두 9907.705. | | 2026-08-28 | Glider integrated deadband 반복 통제 | 50/50 PASS; one-shot 비대칭 철회 | ROS true 50개가 Gazebo true 50개로 관측되고 false 0. Calibration과 Mac/장시간은 별도. | | 2026-08-28 | USBL 이동 표적·travel-time 경계 통제 | moving PASS / delay quantization FAIL | median x 2.999954→8.999973 m. 1539/1541 m latency 0.002803→1.010797초로 integer-second sleep 확인. | | 2026-08-28 | Ocean Current ModelPlugin 두 깊이·두 namespace 물리 통제 | FUNCTIONAL PASS (scoped) | 5 m 0 m/s·Δx 0, 15 m 시작 0.750000 m/s·6초 Δx 2.400436 m. 계수·실해양 정확도는 아님. | | 2026-08-29 | Reproduced and fixed every remaining locally actionable defect in an isolated candidate patch set | 8 patch groups validated; external/scientific boundaries remain open | Mac/Docker controls close WGPU gross planar shift, unavailable-CUDA handling, camera R/B semantics, SeaPressure contract/settings, Spherical validation/status/discovery, DVL local configuration/bridge/8-descriptor failures, USBL zero-noise/paused/fractional delay, plugin hooks, launch/object/build and 18/18 world names. Upstream and installed workspaces remain unchanged. Evidence: results/remaining_defect_fixes_2026-08-29/ | | 2026-08-29 | Post-commit boundary and propagation audit of the eight candidate patches | 1 code boundary and 4 current-document leftovers corrected | The 319-bin WGPU control did not cover the shader's fixed 4096-element workgroup arrays: n_freq > 4096 could enter the exact-DFT branch with out-of-bounds shared-memory indexing. The candidate now rejects oversized spectra before GPU initialisation for the existing C++ CPU fallback; a 4097-bin Rust unit test and cargo check pass. The source guides now describe exact DFT and n_freq buffers, while the current CMake pattern, verified-demo row and two matrix rows distinguish the July installed workaround from the 2026-08-29 candidate launch semantics. Runtime measurements were not rewritten. Evidence: results/remaining_defect_fixes_2026-08-29/post_commit_audit.txt | | 2026-08-29 | Re-ran every locally executable open boundary after the candidate-patch commit | 9th candidate added; 4 stale current verdicts narrowed | Fresh Mac controls reproduced the stock DVL SIGSEGV and RViz windows=0, while Underwater Camera Quickstart passed 3/3 and minimal Fast DDS create passed SHM 5/5 plus UDPv4 5/5. Docker DAVE sonar published PointCloud on an isolated OGRE2 run, so the historical crash is no longer a current always-fails verdict. The fifth ROV sonar was root-caused to dave_ocean_waves.world omitting MultibeamSonarSystem; the ninth candidate produced 513×301 PointCloud2 on Apple-M2 WGPU. Sourcing the existing Docker MAVROS overlay starts ArduSub/MAVROS, but missing libArduPilotPlugin.so, disconnected MAVROS and repeated QGC SIGSEGV keep the integrated loop PARTIAL. All 9 patches reconstruct exactly and pass static checks. Evidence: results/open_gap_revalidation_2026-08-29/ | | 2026-08-29 | Closed the locally executable BlueROV2 external-stack gap; current image finished just after midnight 2026-08-30 | Three Docker control loops FUNCTIONAL PASS / current recipe cache-assisted build PASS | Built pinned official ArduPilot/ardupilot_gazebo for arm64/Jetty. JSON input exposed an ArduSub logger SIGFPE; explicit --speedup 1 removed it. Earlier baseline/Heavy controls and QGC opt-out passed. The full current Dockerfile then completed cache-assisted in 44.86 min (23.9GB, image af9586fa8045), and that exact image passed one baseline, Heavy and Heavy-multibeam headless loop: each MAVROS 4/4, MANUAL force-arm, six-second control, disarm, no FPE/plugin-missing error. Exact-image xrdp service and QGC opt-out/offscreen 20-second survival also passed. At this stage fresh --no-cache, rendered RDP/QGC vehicle connection, combined sonar-candidate image, hardware GPU, Windows/WSL, Fuel upload and physical HIL were outside scope; the following 2026-08-30 row directly tests and fails the combined candidate path. Evidence: results/external_stack_validation_2026-08-29/ | | 2026-08-30 | Combined the fifth-ROV sonar-world candidate with Heavy-multibeam Docker control | FAIL TO REACH FUNCTIONAL CHECK / test gap closed | Live-applied the ninth candidate to the exact-image source tree and rebuilt dave_worlds. WGPU selected llvmpipe; the first 1×1×4 probe took 60053 ms and the sonar reported 513×301×399, then Gazebo began a stack trace. ArduSub printed 105 no-JSON warnings, the MAVROS state probe stayed empty, and neither PointCloud2 nor arm/control/disarm was reached. Manual cleanup means no final exit code or root cause is claimed. The separate exact-image control PASS and separate Mac sonar PASS remain scoped. Evidence: results/external_stack_validation_2026-08-29/dockerfile/combined_sonar_control/ | | 2026-08-30 | Re-ran the remaining locally executable DVL, RViz, Fast DDS, camera, RDP/QGC and combined-backend boundaries | Current bounds established / unresolved causes retained | LLDB placed the Mac DVL crash at the Gazebo Sensors null-scene initialization path and a hidden-camera control restored four-beam output; the distributed stock path remains PARTIAL. RViz controls narrowed its invisible main window to the RViz/OGRE Cocoa external-NSView integration without finding a fix. Fast DDS dirty/clean/SIGKILL/UDP probes passed 18/18, so stale state existed but did not establish the historical hang cause. The exact camera Quickstart passed default 3/3 and UDPv4 3/3 within measured 90–110 s startup latency. The earlier exact cache image passed a real Windows App RDP/QGC replay. Finally, the combined Heavy-multibeam rerun preserved software-WGPU exit 139 while forced CPU published 513×301 PointCloud2 and completed MANUAL arm/control/disarm, superseding the preceding global-failure wording with a backend-dependent verdict. Evidence: results/final_gap_validation_2026-08-30/ | | 2026-08-30 | Rebuilt the current Docker recipe fresh and replayed the rendered integration | Fresh --no-cache build PASS / RDP+Gazebo+QGC rendered PASS | After official BuildKit cache pruning, the current Dockerfile completed from scratch in 4015.02 s (66.917 min), return code 0, producing arm64 image 53744d17f09d (6,260,137,751 bytes). Package, artifact and four pinned-source checks passed. A real FreeRDP/xrdp login started Xorg :10 and XFCE; the captured framebuffer shows Gazebo and QGC Ready/Manual while MAVROS reports connected: true, MANUAL. This fresh client is FreeRDP, distinct from the earlier exact-cache Windows App replay. Evidence: results/final_gap_validation_2026-08-30/docker_no_cache_build/ | | 2026-08-30 | Built and stress-tested a Mac Gazebo Sensors candidate for the stock DVL crash | Isolated candidate PASS / distributed path PARTIAL | Stock control under the same exact-tag harness exits 139 in WaitForInit(). Predicate-only v1 was rejected at 9/10 after the same crash. Candidate v2 also gates render-thread initialization until Apple main-thread RenderUtil::Init() completes: the unmodified official DVL world passes 20/20 with four locks and clean exit; standard camera and no-render regressions pass 3/3 each; the official ROS bridge reports four valid beams. The patch is not installed or upstreamed, and the Homebrew ABI compatibility harness is documented separately. Evidence: results/dvl_macos_force_update_candidate_2026-08-30/ | | 2026-08-30 | Reproduced the Heavy-multibeam software-WGPU crash minimally and validated a deferred-backend candidate | Isolated candidate PASS / distributed path PARTIAL | A minimal Docker baseline without BlueROV2 or MAVROS selects llvmpipe and exits 139 in the OGRE2 sample-texture/null-memcpy path. A render-callback-thread prototype still crashed and was rejected. The retained candidate constructs/probes the backend on the sonar compute thread after the first populated GpuRays frame and disables unused ROS parameter endpoints. Fresh Docker runs pass WGPU 2/2, auto 1/1 and CPU 1/1 with 513×301 PointCloud2 plus 513×399 raw sonar. One combined Heavy+ArduSub+MAVROS run connects, arms, publishes 100 controls, moves X +0.191291 m and disarms; Mac Metal reaches GPU frame 100 without a renderer crash. The candidate is not upstream/installed; hardware GPU, new Mac payload capture, shutdown-only bridge signal and general accuracy remain open. Evidence: results/multibeam_llvmpipe_deferred_backend_candidate_2026-08-30/ | | 2026-08-30 | Extended the deferred-backend candidate with repeated startup, Heavy control, valid shutdown and a bounded soak | Candidate runtime FUNCTIONAL PASS / teardown and distribution PARTIAL | Fresh Docker llvmpipe WGPU cold starts passed 20/20 with 513×301 PointCloud2 and 513×399 raw sonar. Heavy+ArduSub+MAVROS passed 3/3, with X displacement 0.624–0.687 m (median 0.667 m). A 1,861 s soak retained point/raw payloads at both ends, advanced 3,850 WGPU frames, progressed world iterations in four windows and showed no runtime stack trace; bounded fitted memory slopes are reported but not called a leak. Process-group SIGINT exited launch cleanly 10/10, but parameter_bridge reported shutdown-only exit -11 in 7/10. The earlier background-PID shutdown harness is retained as invalid. Evidence: results/multibeam_deferred_backend_extended_validation_2026-08-30/ | | 2026-08-30 | Separated renderer geometry, distributed WGPU raw range bins, an exact-N discriminator, integrated payload and BlueROV sensor contracts | PointCloud matrix PASS / distributed raw WGPU FAIL / exact-N scoped PASS / image sensor coverage PARTIAL | Six Docker scenes at 2/4/7 m and plane/sphere/cylinder geometry produced exactly equal finite CPU/WGPU PointCloud XYZ/intensity. Distributed WGPU raw peaks were shifted +0.584 m, about +1.14–1.17 m and +1.980 m; a correctly rebuilt/relinked exact-N candidate put 18/18 peaks within 0.0736 m (median absolute error 0.0134 m) but needed 88.2–115.3 s for a fresh llvmpipe probe. An integrated world captured point/raw 3/3 with progressing stats. A separate exact-image BlueROV audit preserved camera/odometry and default Gazebo IMU output but found no bridged IMU/magnetometer and no fifth-variant sonar payload, preventing a false 5/5 sensor claim. Evidence: results/multibeam_backend_equivalence_matrix_2026-08-30/ · results/integrated_sonar_payload_validation_2026-08-30/ · results/bluerov_sensor_contract_validation_2026-08-30/ |

| 2026-08-31 | Fast DDS 160-create stress와 focused bridge teardown matrix | Stress PASS / historical cause open; sonar-active teardown FAIL | Fresh Docker에서 dirty default, 20 SIGKILL 뒤, official SHM clean 뒤, UDPv4를 각각 40회씩 총 160/160 성공했다. Bridge는 no-publisher 40/40·stock active 30/30 clean이지만 DAVE-sonar active one-way 9/10, PointCloud-only 2/10 SIGSEGV라 generic 7/10 문구를 철회하고 endpoint-dependent defect로 좁혔다. Evidence: results/fastdds_create_stress_2026-08-31/ · results/parameter_bridge_shutdown_validation_2026-08-31/ | | 2026-08-31 | Sensor long/multi, USBL namespace, Underwater Camera six-tag matrix | Bounded synthetic PASS | SeaPressure 7개/약 200 sim s/2,000 noise frames, DVL 8개/160 messages, USBL 2 transceivers+4 transponders A/B isolation/concurrent, Camera 8개/여섯 tag analytic 0 LSB를 직접 확인했다. Mission endurance와 물리 정확도는 별도다. Evidence: results/sensor_long_multi_validation_2026-08-31/ · results/usbl_multi_namespace_validation_2026-08-31/ · results/underwater_camera_channel_isolation_validation_2026-08-31/ | | 2026-08-31 | Fixed-seed sonar determinism/cache와 clean Docker world replays | Determinism scoped PASS / world output or smoke scoped PASS | Exact-N fixed seed/frame은 fresh container 3회·raw/point 각 9 frame이 byte-identical이었다. Empty-cache probe는 94.192–101 s였다. Clean ocean-waves sonar는 point/raw 3/3과 stats progress를 냈고 manipulation 세 world는 Docker에서 모두 stats 5개, iteration 증가, rc0이었다. Evidence: results/multibeam_seed_determinism_validation_2026-08-31/ · results/ocean_waves_sonar_clean_validation_2026-08-31/ · results/manipulation_docker_replay_2026-08-31/ | | 2026-08-31 | Ocean Current Gauss–Markov와 documented per-model tidal path 직접 검증 | FUNCTIONAL PASS (bounded synthetic controls) | 400 global samples에서 세 축이 모두 변했고, tide metadata가 ModelPlugin에 전달됐다. No-tide model은 200 messages fixed, tide model은 200/200 distinct이고 X range 0.507469 m/s였다. Global topic이 tide로 바뀌어야 한다는 첫 판정은 잘못된 oracle이라 철회했다. Evidence: results/ocean_current_tidal_noise_validation_2026-08-31/ | | 2026-08-31 | Root-caused, installed and submitted the shutdown-only parameter_bridge owner-cycle fix | Actual-workspace candidate PASS / separate component teardown open / upstream review pending | Exact ros_gz 3.0.9 source showed a strong owner/handle cycle. The WeakPtr candidate preserves DAVE payloads and passes bridge-first 20/20, process-group 10/10 and direct/stock controls. Normal Lyrical, official-image ARM64 Humble/Jazzy/Kilted and emulated Jazzy amd64 matrices pass; the ordinary isolated clone passes 24/24 directions and 4/4 service factories. The exact four source files were then applied to the actual user ros_gz workspace while preserving two unrelated CMake edits. It builds, instantiates all 73 generated mapping pairs in both directions, passes payload assertions 73/73 each way, ordered bridge rc0, repeat payload 5/5 and lifecycle 11/11. Issue #951 and signed PR #952 are open; DCO passes and GitHub reports it mergeable. Separate ordered bridge_node assertions pass 3/3 topics + 1/1 service before helper rc134 and component SIGINT exit139, so that macOS teardown remains outside PR #952. Evidence: results/parameter_bridge_cycle_fix_validation_2026-08-31/ |