docs: document broker-mediated cluster architecture, coordinator vs worker node types
Developer-Architecture.md (§6): - Broker-mediated design model as preferred architecture - Coordinator vs Worker node type table with full service requirements - Service distribution ASCII diagram - Workers connect as AMQP clients only (no local broker needed) - Contrasted with service-mesh alternative docker.md (§9): - New Worker Node Deployment Model section - Worker requirements: llama-server binary + node_agent.py + aio-pika - Explicit table of what workers do NOT run - Architecture note: broker-mediated vs service-mesh - Ref: AMQP-0-9-1 client-server protocol since 2006
This commit is contained in:
gramps
@@ -31,6 +31,8 @@
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
> **This compose stack defines the coordinator.** A coordinator runs jC, the broker, and optional infrastructure services. Workers (headless inference nodes) do not use Docker — they install just llama-server + a Python node agent. See §12 for the worker deployment model.
|
||||
|
||||
### Service roles
|
||||
|
||||
| Service | Image | Role |
|
||||
@@ -644,7 +646,74 @@ If the setup wizard fails mid-way, a partial rollback is better than leaving det
|
||||
| **LLM model download** | HuggingFace CLI integration in setup.sh, or manual download. Manual is simpler. | Low |
|
||||
| **Dockerfile optimization** | Pin pip hashes, use `--no-cache-dir`, consider `slim` vs `alpine`. Alpine has musl compatibility issues with psutil. Stay with slim. | Medium |
|
||||
|
||||
## 9. Checklist (pre-v1.0 gate)
|
||||
## 9. Worker Node Deployment Model
|
||||
|
||||
The Docker stack above defines the **coordinator** only. Workers (headless inference nodes) have a radically lighter footprint.
|
||||
|
||||
### 9.1 What a worker runs
|
||||
|
||||
```
|
||||
Worker machine (e.g. jarvis, Corsair)
|
||||
┌────────────────────────────────────┐
|
||||
│ llama-server │
|
||||
│ (single binary, no build needed) │
|
||||
│ │
|
||||
│ node_agent.py │
|
||||
│ (Python script, aio-pika client) │
|
||||
│ ─ connects to coordinator's RMQ │
|
||||
│ ─ publishes heartbeat + reg │
|
||||
│ ─ consumes model_swap commands │
|
||||
│ │
|
||||
│ ROCm or CUDA runtime (if GPU) │
|
||||
└────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 9.2 What a worker does NOT run
|
||||
|
||||
| Service | Reason |
|
||||
|---------|--------|
|
||||
| RabbitMQ server | Connects as AMQP *client* only (aio-pika) |
|
||||
| FastAPI / uvicorn / jC | No HTTP API, no UI, no database |
|
||||
| SQLite | No persistent state of its own |
|
||||
| SearXNG | No web search needs |
|
||||
| Qdrant | No local vector store |
|
||||
| Ollama | Uses coordinator's embedding endpoint |
|
||||
| Docker | Everything runs as bare binaries |
|
||||
| Python venv with full jC deps | Only needs `aio-pika` + `httpx` |
|
||||
|
||||
### 9.3 Worker setup
|
||||
|
||||
```bash
|
||||
# Install llama-server binary
|
||||
wget https://github.com/ggml-org/llama.cpp/releases/.../llama-server
|
||||
chmod +x llama-server
|
||||
|
||||
# Install node agent deps
|
||||
pip install aio-pika httpx
|
||||
|
||||
# Create node agent script (from repo: tools/node_agent.py)
|
||||
# Configure COORDINATOR_AMQP_URL in environment
|
||||
```
|
||||
|
||||
### 9.4 Multiple workers
|
||||
|
||||
Each worker registers independently with the coordinator's RabbitMQ. The coordinator tracks all registered workers via `CLUSTER_NODES` and routes inference requests to the best-matching node based on classification and availability.
|
||||
|
||||
### 9.5 RabbitMQ and workers — architecture note
|
||||
|
||||
Workers connect to RabbitMQ as **standard AMQP TCP clients** — no broker software required. The AMQP-0-9-1 protocol has always been client-server (since 2006), and libraries like `aio-pika`, `pika`, `amqplib`, `php-amqplib`, etc. connect over a single persistent socket. This is distinct from a service-mesh design where every node runs the same software stack and role is determined by config.
|
||||
|
||||
```
|
||||
Broker-mediated model (this project):
|
||||
Coordinator runs RabbitMQ broker ←── Workers connect as AMQP clients
|
||||
|
||||
Service-mesh model (alternative):
|
||||
Every node runs RabbitMQ broker ←── Nodes cluster together, all autonomous
|
||||
```
|
||||
|
||||
The broker-mediated model is the preferred architecture for this project because workers are intentionally heterogeneous (different GPUs, different models, ARM vs x86) and should not be burdened with infrastructure services.
|
||||
|
||||
## 10. Checklist (pre-v1.0 gate)
|
||||
|
||||
- [ ] `Dockerfile` written and builds clean
|
||||
- [ ] `docker-compose.yml` boots all containers
|
||||
@@ -663,7 +732,7 @@ If the setup wizard fails mid-way, a partial rollback is better than leaving det
|
||||
|
||||
---
|
||||
|
||||
## 10. Files to create for B3
|
||||
## 11. Files to create for B3
|
||||
|
||||
```
|
||||
docker.md ← this file (planning doc)
|
||||
|
||||
@@ -174,9 +174,67 @@ Eviction module at `eviction.py` (re-exported through `rag.py` for backward comp
|
||||
|
||||
`POST /api/rag/flush` (admin required) — deletes all non-pinned vectors. Returns `{deleted_count, collection, status}`.
|
||||
|
||||
## 6. AMQP Cluster Architecture (WIP)
|
||||
## 6. Cluster Architecture
|
||||
|
||||
RabbitMQ on ultron with dedicated `jarvischat` vhost:
|
||||
### 6.1 Design Model: Broker-Mediated
|
||||
|
||||
JarvisChat uses a **broker-mediated** cluster design. This is the preferred architecture and is reflected in all implementation decisions below.
|
||||
|
||||
**How it works:**
|
||||
- A single RabbitMQ broker (or clustered set of brokers) acts as the central nervous system
|
||||
- **Coordinator nodes** run the FastAPI app, host the HTTP API/UI, and publish commands to the broker
|
||||
- **Worker nodes** connect as AMQP *clients only* — they consume commands and publish status events, but run no broker software themselves
|
||||
- Communication is asynchronous and persistent: each node opens a TCP connection on startup and keeps it alive. The AMQP-0-9-1 heartbeat detects silent failures within ~60s.
|
||||
|
||||
**Why broker-mediated:**
|
||||
- Workers are heterogeneous (different GPUs, different models, ARM vs x86) — no assumption of uniform software
|
||||
- Workers are lightweight — a Raspberry Pi with a USB AI accelerator can participate without running a broker
|
||||
- The coordinator delegates work via messages, not by SSH'ing into workers or requiring shared filesystems
|
||||
- Failure is isolated: a crashed worker drops off the heartbeat list; the coordinator reassigns its work
|
||||
|
||||
**What it is NOT:**
|
||||
- Not a service mesh — workers do not run identical software stacks
|
||||
- Not autonomous failover — if the coordinator dies, a replacement must be manually promoted (or pre-configured as a secondary coordinator). Workers cannot self-promote to coordinator because they lack the required services (FastAPI, SQLite, DB schema, SearXNG, Qdrant, etc.)
|
||||
- Not a peer-to-peer cluster — all orchestration flows through the coordinator
|
||||
|
||||
### 6.2 Node Types
|
||||
|
||||
Every physical machine in the cluster is classified by which services it runs. Two node types are defined:
|
||||
|
||||
| Aspect | Coordinator | Worker |
|
||||
|--------|------------|--------|
|
||||
| **Role** | Serves HTTP API/UI, orchestrates inference, owns cluster state | Runs inference models on behalf of the coordinator |
|
||||
| **Python** | Required — runs FastAPI app | Required — runs node agent (aio-pika consumer) |
|
||||
| **RabbitMQ server** | Required — hosts the broker | Not required — connects as AMQP client only |
|
||||
| **RabbitMQ client (aio-pika)** | Required — publishes commands, consumes events | Required — consumes commands, publishes events |
|
||||
| **FastAPI / uvicorn** | Required | Not needed |
|
||||
| **SQLite** | Required — owns jarvischat.db | Not needed |
|
||||
| **Qdrant** | Optional (recommended) — vector DB for RAG | Not needed |
|
||||
| **SearXNG** | Optional — web search | Not needed |
|
||||
| **llama-server** | Optional — can share its own GPU for inference | Required — this is why the worker exists |
|
||||
| **Ollama** | Optional — embeddings for RAG | Not needed |
|
||||
| **rocm-smi / nvidia-smi** | Optional — hardware stats | Optional — node agent reports this at registration |
|
||||
|
||||
### 6.3 Service Distribution Summary
|
||||
|
||||
```
|
||||
Coordinator Worker(s)
|
||||
┌────────────────────┐ ┌─────────────────────────┐
|
||||
│ jarvisChat │ │ llama-server │
|
||||
│ (FastAPI + SQLite)│ │ (inference) │
|
||||
│ RabbitMQ server │◄──AMQP───────│ aio-pika (agent) │
|
||||
│ SearXNG (opt) │ persistent │ ROCm / CUDA (if GPU) │
|
||||
│ Qdrant (opt) │ TCP │ │
|
||||
│ Ollama (opt) │ conn │ No broker │
|
||||
│ llama-server(opt) │ │ No jC │
|
||||
└────────────────────┘ │ No DB │
|
||||
│ No search/vector │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
### 6.4 RabbitMQ Topology
|
||||
|
||||
Every RabbitMQ server belongs to a cluster. Currently only the coordinator runs one; if high availability is needed, additional nodes can join the RMQ cluster without changing the architecture.
|
||||
|
||||
| Exchange | Type | Purpose |
|
||||
|----------|------|---------|
|
||||
|
||||
Reference in New Issue
Block a user