rustybeds/wiki/12-architecture-deltas.md

# Architecture Deltas — Recent Hardening Phase

This document catalogs all architectural and design changes made in the recent hardening phase.

## Runtime Model: Daemon-Like Resident Process

**Status**: Completed in `src/main.rs`

**Change**: Converted from startup-IPL-then-exit to a resident, coordinated-shutdown runtime.

**Details**:
- IPL loads config, validates services, initializes broker pools, then enters an event loop.
- Loop waits for either:
  - Global shutdown signal (broadcast from dispatcher when AMQP `shutdown` command received).
  - User interrupt (Ctrl+C).
- On signal, loop cleanly shuts down Tokio tasks and exits with status code 0.

**Why**: Aligns with operational daemon expectations (systemd, orchestrators). Ensures graceful lifecycle rather than abrupt termination. Supports hot-reload/redeployment workflows.

---

## Broker Dispatch: Unified Consumer with Shutdown Semantics

**Status**: Completed in `src/brokers/dispatcher.rs` and `src/brokers/mod.rs`

**Change**: Integrated shutdown command handling into the unified dispatcher consumer.

**Details**:
- Dispatcher pool now receives a global `shutdown_tx` channel at spawn time.
- Each dispatcher consumer listens for AMQP `shutdown` operation.
- On `shutdown`: acknowledge the message, broadcast shutdown signal to all peers, and exit cleanly.
- All dispatchers also listen on the global shutdown channel and exit if signaled externally.

**Why**: Enables coordinated, multi-node shutdown without forceful process kill. Aligns with AMQP message semantics (shutdown is a standard operation, not a runtime hack).

---

## Logger: Explicit IPL Persistence to MongoDB

**Status**: Completed in `src/main.rs` and `src/brokers/logger_store.rs`

**Change**: IPL startup/failure events now explicitly persisted to `msLogs` collection with structured context.

**Details**:
- Root GUID generated at IPL start; all startup events tagged with this root ID.
- Structured log entries include:
  - `root_event_id`: chains all startup events to a single root.
  - `timestamp`: human-readable ISO 8601 format.
  - `node_id`: configured node name/role.
  - `event_type`: IPL phase (e.g., "ipl_start", "service_validated", "broker_pool_spawned", "ipl_complete").
  - `message`: human-readable summary.
  - `metadata`: optional structured context (validation results, latency, etc.).
- If IPL fails, best-effort logging of failure event to Mongo before process exit.
- After IPL success, showcase log-level examples (INFO, WARN, ERROR) for visibility.

**Why**: Startup is traditionally hardest to debug (logs often lost). Persistent, queryable startup context enables post-mortem analysis of deployment/initialization issues. Root GUID enables chain-crawl diagnostics across distributed startup events.

---

## Developer Diagnostics: Root GUID Lineage and Chain Tracing

**Status**: Completed in `src/brokers/logger_store.rs` and `src/bin/log_dumper.rs`

**Change**: Added root GUID-based event chain tracing and query layer.

**Details**:
- `logger_store::fetch_chain(root_event_id, limit)`: retrieve all events tagged with a root ID, sorted by timestamp.
- `logger_store::fetch_root_record(root_event_id)`: retrieve the initiating root event.
- `log_dumper` web UI exposes:
  - Root GUID input field to query and visualize entire event chain.
  - Single-record view at `/record?root_event_id=...` to inspect individual startup context.
  - Arrow-trigger UX for expanding compact row summaries without constant page reload.

**Why**: Enables developers to rapidly correlate events across a single startup sequence or transaction. Reduces manual log sifting. Scales from single node to multi-node deployments.

---

## Configuration: Trace-On and Logger Admin Controls

**Status**: Completed in `src/config/structs.rs` and `config/env_dev.toml`

**Change**: Added two new config namespaces for developer and administrative control.

**Details**:

### `[runtime.trace_on]`
- Boolean flag (default: false in production, true in `env_dev.toml`).
- When true, logs method entry/exit at TRACE level for all broker consumers and core trait implementations.
- Enables dev to narrow causality in complex message flows without instrumenting code.

### `[logger_admin]`
- `purge_on_ipl` (boolean, default: false): on successful IPL, automatically purge named collections before startup logging begins.
- `purge_collections` (array of strings): list of collection names to purge (e.g., `["msLogs", "msErrors"]`).
- Enables clean dev iteration: each `cargo run` in dev automatically resets logger state.

**Why**: Reduces friction in dev loops. Trace-on avoids printf debugging. Purge-on-IPL ensures each test iteration starts fresh without manual `mongo` CLI cleanup.

---

## Observability Utility: Modern Logger Reader (log_dumper)

**Status**: Completed in `src/bin/log_dumper.rs`

**Change**: Built a modern Rust equivalent to legacy PHP `utilities/dumper.php` for browsing `msLogs`.

**Details**:
- **Web UI** (Axum):
  - Dashboard route `/` with seed-write action, quick filter by level/node, root GUID chain input.
  - Compact row layout: timestamp | level | node | message snippet | arrow (expand).
  - Single-record view `/record?root_event_id=...` showing full event context.
  - Arrow-trigger expansion shows full message without full-page refresh.

- **Features**:
  - Human-readable timestamps (ISO 8601 formatted).
  - Seed-write to create test events and validate logger pipeline.
  - Root chain traversal via GUID input.
  - Dev-centric UX: minimal clicks, maximum information density.

**Why**: Centralizes all observability into a single web interface. Replaces CLI-based manual querying. Makes startup diagnostics visible to entire team without MongoDB knowledge.

---

## Operational Safety: Dev-Only Purge Controls

**Status**: Completed in `src/main.rs` and config system

**Change**: Added dev-only purge logic to reset logger collections on IPL in non-production environments.

**Details**:
- IPL checks `config.logger_admin.purge_on_ipl` flag.
- If true and node is not production, purges collections listed in `config.logger_admin.purge_collections` before logging startup events.
- Prevents accidental production data loss (flag only honored in non-prod node roles).
- `env_dev.toml` enables this by default for frictionless dev iteration.

**Why**: Closes dev/prod gap. Enables safe, repeatable testing without manual intervention. Prevents stale logger state from polluting diagnostics.

---

## Commit Summary

This hardening phase encompasses:

1. **Runtime lifecycle**: Daemon model, coordinated shutdown, graceful exit.
2. **Broker semantics**: Shutdown operation integration, channel-based signaling.
3. **Logging infrastructure**: Persistent IPL events, root GUID lineage, structured context.
4. **Developer experience**: Trace control, purge controls, web-based observability.
5. **Configuration**: New `trace_on` and `logger_admin` namespaces.
6. **Tooling**: Modern Rust observability utility replacing legacy PHP dumper.

**Files Changed**:
- `src/main.rs`: resident runtime loop, IPL logging, shutdown coordination, trace control.
- `src/brokers/dispatcher.rs`: shutdown operation handling, global shutdown listening.
- `src/brokers/mod.rs`: dispatcher pool accepts shutdown channels.
- `src/brokers/logger_store.rs`: root GUID chain fetch operations, structured logging helpers.
- `src/config/structs.rs`: `trace_on`, `logger_admin` config types.
- `src/bin/log_dumper.rs`: new modern observability utility (Axum web UI).
- `config/env_dev.toml`: dev overrides enabling trace/purge controls.
- `Cargo.toml` / `Cargo.lock`: added `axum`, `chrono`, `uuid` dependencies.
- Wiki updates: `Home.md`, `04-ipl.md`, `06-queue-topology.md`, `10-modernization-roadmap.md`, new `11-beds-architecture-visual-brief.md`.

**Next Phase**: Autoscaling heuristics, metric collection, and cross-node coordinator election (deferred).