diff --git a/docker.md b/docker.md index d6f83c7..2f01ea3 100644 --- a/docker.md +++ b/docker.md @@ -31,6 +31,8 @@ └─────────────────────────────────────────────────────────┘ ``` +> **This compose stack defines the coordinator.** A coordinator runs jC, the broker, and optional infrastructure services. Workers (headless inference nodes) do not use Docker — they install just llama-server + a Python node agent. See §12 for the worker deployment model. + ### Service roles | Service | Image | Role | @@ -644,7 +646,74 @@ If the setup wizard fails mid-way, a partial rollback is better than leaving det | **LLM model download** | HuggingFace CLI integration in setup.sh, or manual download. Manual is simpler. | Low | | **Dockerfile optimization** | Pin pip hashes, use `--no-cache-dir`, consider `slim` vs `alpine`. Alpine has musl compatibility issues with psutil. Stay with slim. | Medium | -## 9. Checklist (pre-v1.0 gate) +## 9. Worker Node Deployment Model + +The Docker stack above defines the **coordinator** only. Workers (headless inference nodes) have a radically lighter footprint. + +### 9.1 What a worker runs + +``` +Worker machine (e.g. jarvis, Corsair) +┌────────────────────────────────────┐ +│ llama-server │ +│ (single binary, no build needed) │ +│ │ +│ node_agent.py │ +│ (Python script, aio-pika client) │ +│ ─ connects to coordinator's RMQ │ +│ ─ publishes heartbeat + reg │ +│ ─ consumes model_swap commands │ +│ │ +│ ROCm or CUDA runtime (if GPU) │ +└────────────────────────────────────┘ +``` + +### 9.2 What a worker does NOT run + +| Service | Reason | +|---------|--------| +| RabbitMQ server | Connects as AMQP *client* only (aio-pika) | +| FastAPI / uvicorn / jC | No HTTP API, no UI, no database | +| SQLite | No persistent state of its own | +| SearXNG | No web search needs | +| Qdrant | No local vector store | +| Ollama | Uses coordinator's embedding endpoint | +| Docker | Everything runs as bare binaries | +| Python venv with full jC deps | Only needs `aio-pika` + `httpx` | + +### 9.3 Worker setup + +```bash +# Install llama-server binary +wget https://github.com/ggml-org/llama.cpp/releases/.../llama-server +chmod +x llama-server + +# Install node agent deps +pip install aio-pika httpx + +# Create node agent script (from repo: tools/node_agent.py) +# Configure COORDINATOR_AMQP_URL in environment +``` + +### 9.4 Multiple workers + +Each worker registers independently with the coordinator's RabbitMQ. The coordinator tracks all registered workers via `CLUSTER_NODES` and routes inference requests to the best-matching node based on classification and availability. + +### 9.5 RabbitMQ and workers — architecture note + +Workers connect to RabbitMQ as **standard AMQP TCP clients** — no broker software required. The AMQP-0-9-1 protocol has always been client-server (since 2006), and libraries like `aio-pika`, `pika`, `amqplib`, `php-amqplib`, etc. connect over a single persistent socket. This is distinct from a service-mesh design where every node runs the same software stack and role is determined by config. + +``` +Broker-mediated model (this project): + Coordinator runs RabbitMQ broker ←── Workers connect as AMQP clients + +Service-mesh model (alternative): + Every node runs RabbitMQ broker ←── Nodes cluster together, all autonomous +``` + +The broker-mediated model is the preferred architecture for this project because workers are intentionally heterogeneous (different GPUs, different models, ARM vs x86) and should not be burdened with infrastructure services. + +## 10. Checklist (pre-v1.0 gate) - [ ] `Dockerfile` written and builds clean - [ ] `docker-compose.yml` boots all containers @@ -663,7 +732,7 @@ If the setup wizard fails mid-way, a partial rollback is better than leaving det --- -## 10. Files to create for B3 +## 11. Files to create for B3 ``` docker.md ← this file (planning doc) diff --git a/docs/wiki/Developer-Architecture.md b/docs/wiki/Developer-Architecture.md index 4235155..1614dbd 100644 --- a/docs/wiki/Developer-Architecture.md +++ b/docs/wiki/Developer-Architecture.md @@ -174,9 +174,67 @@ Eviction module at `eviction.py` (re-exported through `rag.py` for backward comp `POST /api/rag/flush` (admin required) — deletes all non-pinned vectors. Returns `{deleted_count, collection, status}`. -## 6. AMQP Cluster Architecture (WIP) +## 6. Cluster Architecture -RabbitMQ on ultron with dedicated `jarvischat` vhost: +### 6.1 Design Model: Broker-Mediated + +JarvisChat uses a **broker-mediated** cluster design. This is the preferred architecture and is reflected in all implementation decisions below. + +**How it works:** +- A single RabbitMQ broker (or clustered set of brokers) acts as the central nervous system +- **Coordinator nodes** run the FastAPI app, host the HTTP API/UI, and publish commands to the broker +- **Worker nodes** connect as AMQP *clients only* — they consume commands and publish status events, but run no broker software themselves +- Communication is asynchronous and persistent: each node opens a TCP connection on startup and keeps it alive. The AMQP-0-9-1 heartbeat detects silent failures within ~60s. + +**Why broker-mediated:** +- Workers are heterogeneous (different GPUs, different models, ARM vs x86) — no assumption of uniform software +- Workers are lightweight — a Raspberry Pi with a USB AI accelerator can participate without running a broker +- The coordinator delegates work via messages, not by SSH'ing into workers or requiring shared filesystems +- Failure is isolated: a crashed worker drops off the heartbeat list; the coordinator reassigns its work + +**What it is NOT:** +- Not a service mesh — workers do not run identical software stacks +- Not autonomous failover — if the coordinator dies, a replacement must be manually promoted (or pre-configured as a secondary coordinator). Workers cannot self-promote to coordinator because they lack the required services (FastAPI, SQLite, DB schema, SearXNG, Qdrant, etc.) +- Not a peer-to-peer cluster — all orchestration flows through the coordinator + +### 6.2 Node Types + +Every physical machine in the cluster is classified by which services it runs. Two node types are defined: + +| Aspect | Coordinator | Worker | +|--------|------------|--------| +| **Role** | Serves HTTP API/UI, orchestrates inference, owns cluster state | Runs inference models on behalf of the coordinator | +| **Python** | Required — runs FastAPI app | Required — runs node agent (aio-pika consumer) | +| **RabbitMQ server** | Required — hosts the broker | Not required — connects as AMQP client only | +| **RabbitMQ client (aio-pika)** | Required — publishes commands, consumes events | Required — consumes commands, publishes events | +| **FastAPI / uvicorn** | Required | Not needed | +| **SQLite** | Required — owns jarvischat.db | Not needed | +| **Qdrant** | Optional (recommended) — vector DB for RAG | Not needed | +| **SearXNG** | Optional — web search | Not needed | +| **llama-server** | Optional — can share its own GPU for inference | Required — this is why the worker exists | +| **Ollama** | Optional — embeddings for RAG | Not needed | +| **rocm-smi / nvidia-smi** | Optional — hardware stats | Optional — node agent reports this at registration | + +### 6.3 Service Distribution Summary + +``` +Coordinator Worker(s) +┌────────────────────┐ ┌─────────────────────────┐ +│ jarvisChat │ │ llama-server │ +│ (FastAPI + SQLite)│ │ (inference) │ +│ RabbitMQ server │◄──AMQP───────│ aio-pika (agent) │ +│ SearXNG (opt) │ persistent │ ROCm / CUDA (if GPU) │ +│ Qdrant (opt) │ TCP │ │ +│ Ollama (opt) │ conn │ No broker │ +│ llama-server(opt) │ │ No jC │ +└────────────────────┘ │ No DB │ + │ No search/vector │ + └─────────────────────────┘ +``` + +### 6.4 RabbitMQ Topology + +Every RabbitMQ server belongs to a cluster. Currently only the coordinator runs one; if high availability is needed, additional nodes can join the RMQ cluster without changing the architecture. | Exchange | Type | Purpose | |----------|------|---------|