feat: resident runtime, shutdown command, observatory, and IPL logging hardening
- keep BEDS resident after IPL and coordinate clean shutdown - propagate AMQP shutdown command across dispatcher pool - add structured IPL milestone/event-chain logging with root GUID context - add optional trace_on config for verbose method-entry diagnostics - add dev purge-on-IPL controls for admin/logger collections - add log level showcase events after IPL node-green - add Mongo logger store helpers for chain/root lookup and purge - add/modernize BEDS Observatory log_dumper utility UI and root record view - refresh source headers and wiki docs for current architecture/runtime - add architecture visual brief for leadership/image-generation workflows
This commit is contained in:
166
wiki/11-beds-architecture-visual-brief.md
Normal file
166
wiki/11-beds-architecture-visual-brief.md
Normal file
@@ -0,0 +1,166 @@
|
||||
# BEDS Architecture Visual Brief
|
||||
|
||||
Audience: Senior managers, directors, and architects
|
||||
Purpose: Explain what BEDS is, why AMQP is central, and how the software stack delivers operational impact.
|
||||
|
||||
## Executive Summary
|
||||
|
||||
BEDS (Back End Data System) is a data-service framework where AMQP (RabbitMQ) is the backbone for all data movement.
|
||||
|
||||
Instead of app code talking directly to databases, every request goes through AMQP first. This provides:
|
||||
|
||||
- Strong control over in-transit data paths
|
||||
- Centralized routing and policy enforcement
|
||||
- Better observability and auditability
|
||||
- Scale-out through queue consumers (dispatcher pools)
|
||||
- Isolation between transport concerns and domain/business logic
|
||||
|
||||
Business impact:
|
||||
|
||||
- Faster onboarding of new data domains
|
||||
- More consistent operational controls
|
||||
- Higher confidence in reliability and replayability
|
||||
- Lower risk from tightly coupled app-to-database patterns
|
||||
|
||||
## One-Sentence Product Positioning
|
||||
|
||||
BEDS is an AMQP-first data orchestration platform that turns data access into a governed, scalable event pipeline instead of direct database calls.
|
||||
|
||||
## Architecture in Plain Language
|
||||
|
||||
When a client asks to read or write data:
|
||||
|
||||
1. The request is published into RabbitMQ.
|
||||
2. A dispatcher pool consumes messages from the queue.
|
||||
3. The dispatcher validates the envelope and selects the target template/class.
|
||||
4. The class chain executes business logic and schema-specific behavior.
|
||||
5. The storage adapters execute persistence in MongoDB or SQL systems.
|
||||
6. A response is published back (if the request expects one).
|
||||
|
||||
The key design point: AMQP is the stable transport spine; domain classes can evolve without changing transport scaffolding.
|
||||
|
||||
## Software Stack (Top to Bottom)
|
||||
|
||||
### Layer 1: Experience and Integration
|
||||
|
||||
- API clients
|
||||
- Internal tools (for example: log reader UI)
|
||||
- Service integrations
|
||||
|
||||
### Layer 2: Transport and Delivery (AMQP)
|
||||
|
||||
- RabbitMQ exchange and queue topology
|
||||
- Message envelopes (versioned request/response contracts)
|
||||
- Acknowledgment and negative acknowledgment behavior
|
||||
- Retry and dead-letter patterns
|
||||
|
||||
### Layer 3: Dispatch and Domain Runtime
|
||||
|
||||
- Dispatcher pool (consumer workers)
|
||||
- Template registry lookup
|
||||
- Class instantiation (domain object selected by template)
|
||||
- Class -> Schema -> Base I/O execution path
|
||||
|
||||
### Layer 4: Data Adapters and Persistence
|
||||
|
||||
- MongoDB access (REC/document-oriented workloads)
|
||||
- Relational DB access (REL/SQL workloads)
|
||||
- Logger event storage (msLogs)
|
||||
|
||||
### Layer 5: Operations and Governance
|
||||
|
||||
- Structured logging and diagnostics
|
||||
- Environment-configured runtime behavior
|
||||
- Health checks and startup validation (IPL)
|
||||
- Future auto-scaling supervisor concept for dispatcher pools
|
||||
|
||||
## Why AMQP Scaffolding Matters
|
||||
|
||||
AMQP is not just transport in this architecture. It is the control plane.
|
||||
|
||||
Benefits in management terms:
|
||||
|
||||
- Policy centralization: transport rules are applied consistently
|
||||
- Observability: queue behavior, consumer behavior, and message flow are inspectable
|
||||
- Safety: retries and DLQs protect against transient failures
|
||||
- Flexibility: adding new domains does not require redesigning transport
|
||||
- Scale: increase dispatcher workers to handle backlog without app rewrites
|
||||
|
||||
## Current Node and Flow Model
|
||||
|
||||
- Single binary deployment with role/config-driven runtime behavior
|
||||
- Unified dispatcher model consuming queue messages
|
||||
- Template-driven class resolution to mask schema details from transport observers
|
||||
|
||||
In effect, transport sees standard envelopes; schema details are revealed only at class execution time.
|
||||
|
||||
## Reference Data Flow (Narrative)
|
||||
|
||||
Client request -> RabbitMQ message -> Dispatcher consumes -> Template/class resolution -> Domain logic -> Mongo/SQL adapter -> Reply message
|
||||
|
||||
This pattern supports both:
|
||||
|
||||
- Request/response operations
|
||||
- Fire-and-forget telemetry (for example: logger semantics)
|
||||
|
||||
## Leadership-Level Benefits
|
||||
|
||||
- Architectural consistency across teams and services
|
||||
- Reduced change risk due to clear layer boundaries
|
||||
- Operational leverage through queue-based scaling
|
||||
- Better compliance posture from centralized message handling
|
||||
- Strong foundation for future AI-assisted data object generation
|
||||
|
||||
## Visual Blueprint (for Diagram or Image Generation)
|
||||
|
||||
Use this structure when creating architecture visuals:
|
||||
|
||||
- Left side: Client/API and internal tools
|
||||
- Center spine: RabbitMQ (highlighted as "AMQP Control Plane")
|
||||
- Right side: Dispatcher pool and class runtime
|
||||
- Bottom: MongoDB + SQL data stores
|
||||
- Overlay badges: Auditability, Retry/DLQ, Scale-out Consumers, Config-Driven Nodes
|
||||
|
||||
Color and style suggestions:
|
||||
|
||||
- Transport spine in teal/blue
|
||||
- Domain runtime in neutral steel/graphite
|
||||
- Data stores in earthy accents
|
||||
- Use arrows to emphasize directional flow and feedback loop for responses
|
||||
|
||||
## Prompt Pack for AI Image Generation
|
||||
|
||||
### Prompt A: Executive Architecture Poster
|
||||
|
||||
Create a modern enterprise architecture poster titled "BEDS: AMQP-First Data Orchestration". Show a left-to-right pipeline: clients and tools -> RabbitMQ AMQP control plane -> dispatcher worker pool -> class/schema execution -> MongoDB and SQL storage. Include callout badges: auditability, delivery guarantees, retry/DLQ safety, scalable consumers, config-driven deployment. Style should be clean, boardroom-ready, and easy to understand for non-engineering leadership.
|
||||
|
||||
### Prompt B: Technical Leadership Slide Graphic
|
||||
|
||||
Generate a diagram for a CTO slide that explains a message-driven data platform. Center RabbitMQ as the control plane. Show message envelopes entering from APIs, dispatcher consumers scaling horizontally, template-driven class resolution, and adapters writing to document and relational stores. Emphasize "no direct app-to-database coupling". Use polished enterprise iconography and concise labels.
|
||||
|
||||
### Prompt C: Impact-Focused Infographic
|
||||
|
||||
Produce an infographic with three bands: "How it works", "Why it matters", "Business impact". In "How it works", illustrate AMQP-first request flow through dispatchers into data stores. In "Why it matters", list governance, observability, reliability, and scaling. In "Business impact", highlight faster delivery, lower operational risk, and better control of data in transit.
|
||||
|
||||
## Diagram Source (Mermaid)
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[Clients and Internal Tools] --> B[RabbitMQ AMQP Control Plane]
|
||||
B --> C[Dispatcher Pool]
|
||||
C --> D[Template Registry and Class Resolution]
|
||||
D --> E[Class -> Schema -> Base IO]
|
||||
E --> F[(MongoDB REC)]
|
||||
E --> G[(SQL REL)]
|
||||
E --> H[(msLogs Logger Store)]
|
||||
E --> I[Response Envelope]
|
||||
I --> A
|
||||
|
||||
B -.-> J[Retry and DLQ]
|
||||
C -.-> K[Horizontal Scale-Out]
|
||||
B -.-> L[Audit and Transport Controls]
|
||||
```
|
||||
|
||||
## Suggested Talking Track (30-45 seconds)
|
||||
|
||||
"BEDS moves data access from direct database calls into an AMQP-governed pipeline. RabbitMQ becomes the transport control plane, and dispatcher workers handle execution through template-driven class resolution. This gives us stronger observability, safer failure handling, and easier horizontal scaling. Operationally, we gain consistency and control; strategically, we gain a platform that is easier to extend and govern."
|
||||
Reference in New Issue
Block a user