rustybeds/README.md

# BEDS — Back End Data System

A Rust rewrite of a battle-tested PHP microservice backend framework with 952 days of continuous production uptime, 40,000+ transactions per second on a single node, and ~200ms round-trip latency from Baja California to West Virginia across dozens of concurrent database fanout calls.

This is not a greenfield project. The architecture is proven. The Rust rewrite exists to go further.

---

## Why Rust

| Problem in PHP | Solution in Rust |
|---|---|
| Memory leaks required planned broker death via SIGCHLD | Tokio tasks don't leak — broker pool is permanent |
| Runtime dependency on PHP interpreter | Single compiled binary, zero runtime deps |
| Source code exposed on deployment | Compiled — IP protected |
| Throughput ceiling on single node | 5–10x improvement expected |
| No AI layer | Phase 2: AI-driven database object generation |

---

## Prerequisites

These services must be installed and running before BEDS will start:

| Dependency | Purpose | Notes |
|---|---|---|
| [Rust (stable)](https://rustup.rs) | Build toolchain | `rustup update stable` |
| [RabbitMQ](https://www.rabbitmq.com/docs/install-debian) | AMQP broker | Management plugin recommended: `rabbitmq-plugins enable rabbitmq_management` |
| [MongoDB](https://www.mongodb.com/docs/manual/installation/) | Logger store (`msLogs`) | Tested against 6.x / 7.x |
| [MariaDB](https://mariadb.org/download/) | Primary relational store | Tested against 10.x / 11.x |
| [Apache + PHP](https://httpd.apache.org) | Observer tool (`log_dumper`) | See below |

### Observer Tool — Apache + PHP

The log observer (`utilities/log_dumper/`) is a standalone PHP/Apache application, intentionally **independent of the BEDS binary**. When BEDS is crashing, you still need a working observer. A Rust binary compiled from the same project is useless in that scenario.

```bash
# Debian/Ubuntu
apt install apache2 php libapache2-mod-php php-mongodb
systemctl enable apache2
```

Once installed, copy or symlink `utilities/log_dumper/` into your Apache docroot:

```bash
ln -s /path/to/rustybeds/utilities/log_dumper /var/www/html/beds-logs
```

The observer requires only a running MongoDB instance — no BEDS process needed.

---

## Architecture

BEDS is **AMQP-first**. No component in the application layer ever touches a database directly. Every data operation flows through a message broker. This is not a constraint — it is the product.

```
Client Request
      │
      ▼
  RabbitMQ / AMQP
      │
      ▼
  Broker Pool  (Tokio async tasks — one per broker type)
      │
      ▼
  Factory  (template name → adapter dispatch)
      │
      ▼
  NamasteCore Trait  (unified CRUD interface)
      │
      ▼
  Database Adapter  (MySQL · MongoDB)
      │
      ▼
  DBA-owned Schema  (views · stored procedures · functions)
```

**The application layer never writes SQL. Ever.**

---

## Core Principles

1. **AMQP-first** — all data access flows through the broker layer, no exceptions
2. **Database agnostic** — MySQL/MariaDB and MongoDB behind a unified trait; no DB-specific logic leaks upward
3. **DBA-owned schema** — all data access goes through named database objects; the application calls template names, not queries
4. **Template-driven CRUD** — each data domain is a struct implementing `NamasteCore`; adding a domain means adding one file
5. **Config-driven nodes** — all nodes run the same binary; role is determined entirely by startup config

---

## Service Nodes

| Node | Role | Brokers |
|---|---|---|
| `appServer` | Primary application | rBroker, wBroker, mBroker |
| `admin` | Administration & observability | adminBrokerIn/Out, adminLogsBroker, adminSyslogBroker, adminGraphBroker |
| `segundo` | Warehouse / cool storage | whBroker, cBroker |
| `tercero` | User & session management | uBroker, sBroker |

Every node runs the same binary. Configuration determines what it does.

---

## Scaling Model

- **Horizontal** — increase broker instance count in config when a node has headroom
- **Vertical** — add nodes to the broker pool when a node saturates
- **Hot-swap** — nodes can be replaced without touching application code; the broker pool absorbs the transition

---

## Project Structure

```
rustybeds/
├── src/
│   ├── bin/
│   │   └── log_dumper.rs       # (deprecated — replaced by utilities/log_dumper/)
│   ├── brokers/
│   │   ├── mod.rs              # Pool manager — spawn_dispatcher_pool()
│   │   ├── dispatcher.rs       # Unified AMQP consumer task (replaces r/w_broker)
│   │   ├── logger_store.rs     # MongoDB logger persistence + chain fetch
│   │   ├── payload.rs          # BrokerPayload — AMQP message envelope struct
│   │   └── error.rs            # BrokerError type
│   ├── config/
│   │   ├── mod.rs              # Loader — load() and load_from() for testability
│   │   └── structs.rs          # Typed config structs (serde Deserialize)
│   ├── core/
│   │   ├── mod.rs              # NamasteCore trait — unified CRUD interface
│   │   └── dispatch.rs         # Dispatch boundary traits: DomainClass, SchemaLayer, BaseIoAdapter
│   ├── services/
│   │   ├── mod.rs              # Groups external service transport modules
│   │   ├── amqp/
│   │   │   ├── mod.rs          # validate() — TCP reachability pre-flight
│   │   │   ├── connection.rs   # AmqpConnection — auth + exchange declare
│   │   │   └── error.rs        # AmqpError type
│   │   ├── mongo/
│   │   │   └── mod.rs          # validate_all() — TCP reachability
│   │   └── mariadb/
│   │       └── mod.rs          # validate_all() — master/secondary pattern
│   ├── template_registry/
│   │   └── mod.rs              # REC template registry — load, validate, runtime snapshot
│   ├── lib.rs                  # Public API surface for integration test harness
│   ├── logging.rs              # tracing + journald + console mirror init
│   └── main.rs                 # IPL sequence + resident runtime loop + coordinated shutdown
├── config/
│   ├── beds.toml               # Base config — checked in, no credentials
│   ├── env_dev.toml            # Dev overrides — gitignored
│   ├── env_qa.toml             # QA overrides — gitignored
│   └── env_prod.toml           # Prod overrides — gitignored
├── templates/
│   ├── example_rec.toml        # Canonical self-documenting REC template
│   └── mst_logger_rec.toml     # Logger collection template (msLogs)
├── utilities/
│   └── log_dumper/             # Standalone PHP/Apache observer (independent of BEDS)
├── tests/
│   ├── broker_pool_test.rs     # Dispatcher pool integration tests
│   ├── common/mod.rs           # Shared test helpers — load_test_config()
│   └── fixtures/
│       └── beds_test.toml      # Canonical test config fixture
└── Cargo.toml
```

---

## Configuration

Layered TOML system. `beds.toml` holds production-safe defaults and is checked into version control. An env override file (`env_{BEDS_ENV}.toml`) is layered on top and is never committed. Set `BEDS_ENV` to `dev`, `qa`, or `prod` — defaults to `dev`.

```toml
# beds.toml — base
[broker_services.app_server]
host = "prod-broker.internal"
port = 5672

# env_dev.toml — local override (gitignored)
[broker_services.app_server]
host = "localhost"
pass = "your-dev-password"
```

The `config` crate deep-merges these at startup. Only keys present in the env file are overridden — everything else inherits from the base.

---

## Status

| Component | Status |
|---|---|
| Config loading (layered TOML + env select) | Done |
| Structured logging (journald + console mirror) | Done |
| IPL sequence with env-aware error handling | Done |
| RabbitMQ reachability validation | Done |
| RabbitMQ authentication + exchange declaration | Done |
| Unit test scaffolding + config fixture pattern | Done |
| MongoDB reachability validation | Done |
| MariaDB reachability validation | Done |
| Unified dispatcher pool (replaces r/w broker split) | Done |
| BrokerPayload — AMQP message envelope struct | Done |
| REC template registry (load, validate, runtime snapshot) | Done |
| Retry / DLQ queue topology | Done |
| Resident runtime loop + coordinated shutdown command | Done |
| MongoDB logger store — IPL persistence + chain fetch | Done |
| Dispatch boundary traits (DomainClass, SchemaLayer, BaseIoAdapter) | Done |
| Observer tool (PHP/Apache log_dumper) | Next |
| RegistryDispatchResolver — class instantiation | Next |
| Database adapters (MariaDB, MongoDB) | Planned |
| Broker task supervision / respawn | Planned |
| Config schema validation at startup | Planned |
| AI database object generation | Phase 2 |

---

## Developer Wiki

Full framework documentation lives in [`wiki/`](wiki/Home.md):

- [Origin Story](wiki/01-origin-story.md) — Where BEDS came from and why it was built the way it was
- [Architecture Overview](wiki/02-architecture.md) — Full system design and core principles
- [The Four Nodes](wiki/03-nodes.md) — appServer, admin, segundo, tercero
- [IPL — Initial Program Load](wiki/04-ipl.md) — Bootstrap sequence, step by step
- [Configuration System](wiki/05-configuration.md) — Layered TOML, env files, topology options
- [Queue Topology](wiki/06-queue-topology.md) — AMQP exchanges, queues, routing keys
- [Design Notes & Discussions](wiki/07-notes.md) — Spew/no-nazi-twitter concept, open questions, architecture discussions in progress
- [Template System](wiki/08-template-system.md) — REC and REL templates, TLA convention
- [Event Lineage](wiki/09-event-lineage.md) — Compound event IDs, parent/child tracking
- [Glossary](wiki/glossary.md) — Terms and abbreviations

---

## Performance Baseline

The PHP predecessor achieved:
- **40,000+ tp/s** on a single node
- **~200ms round-trip** Baja California → West Virginia with dozens of concurrent DB fanout calls per transaction
- **952 days** continuous production uptime without error

The Rust rewrite must meet or exceed all three numbers. Benchmarks run before any architectural change touching the broker pool or factory dispatch path.

---

## Author

gramps@llamachile.shop