diff --git a/CLAUDE.md b/CLAUDE.md index b193d79..5e8fb02 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,6 +5,10 @@ Detailed project context, work state, architecture, and configuration have moved ## Quick start ```bash +# Docker (recommended) +scripts/setup.sh # first run: generates .env, secrets, pulls default model +docker compose up -d + # Development ./venv/bin/uvicorn app:app --host 0.0.0.0 --port 8080 --reload @@ -17,7 +21,9 @@ sudo systemctl restart caic ## Dependencies +Docker deployment: no manual pip install needed — the Dockerfile handles it. + ```bash ./venv/bin/pip install -r requirements.txt -# Also requires: psutil jinja2 python-multipart pypdf (not in requirements.txt) +# Also requires: psutil jinja2 python-multipart pypdf ``` diff --git a/README.md b/README.md index 5faa5bb..b75e2c6 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ ![cAIc banner](static/readme-banner.png) -# cAIc v0.22.0 +# cAIc v1.0.0 Consumer AI hardware is a wasteland of incompatibility. NVIDIA speaks CUDA, AMD speaks ROCm. Your RTX 5070 Ti lives in one machine with 16 GB VRAM; your RX 6600 XT lives in another with 12 GB. Alone, neither can run a 14B model at usable speed. Together, they could — if the software stack didn't treat heterogeneous hardware as a bug instead of a feature. @@ -76,6 +76,34 @@ At v1.0, this ships with a Docker compose stack and setup wizard that detect CPU Developer wiki: [Home](https://llgit.llamachile.tube/gramps/cAIc/wiki/Home) — includes [FAQ](https://llgit.llamachile.tube/gramps/cAIc/wiki/FAQ), [Installation Guide](https://llgit.llamachile.tube/gramps/cAIc/wiki/Installation), and [full architecture docs](https://llgit.llamachile.tube/gramps/cAIc/wiki/Developer-Architecture) +## What's New in v1.0.0 + +### Docker Containerization (B3) +- **`Dockerfile`** — multi-stage Python 3.13-slim build, healthcheck, uvicorn CMD +- **`docker-compose.yml`** — full stack: cAIc, SearXNG, Qdrant, RabbitMQ, llama-server, Ollama +- **`scripts/setup.sh`** — first-run scaffolding: generates `.env`, secrets, SearXNG config, pulls default model (~4.6GB Qwen2.5-7B Q4_K_M) +- **`.env.example`** — all environment variables documented with generation hints +- **`.dockerignore`** — excludes venv, tests, .git, models, secrets +- **`searxng-settings.yml.dist`** — SearXNG config template +- **`models/README.txt`** — instructions for placing .gguf files +- All service URLs are env-var configurable with Docker service hostnames (`http://llama-server:8081`, `http://qdrant:6333`, etc.) +- AMQP secret uses Docker secrets pattern (`/run/secrets/`) +- Only port 8080 exposed by default; all other services internal to compose network +- Graceful degradation — SearXNG and Ollama are optional (not required for startup) + +### Bug Fixes & Hardening (v0.22.1) +- `config.py`: defaults changed from hardcoded LAN IPs to `localhost` for Docker compatibility +- `config.py`: `AMQP_SECRET_PATH` defaults to `/run/secrets/caic_amqp_secret` +- `config.py`: `DEFAULT_MODEL` now configurable via `CAIC_DEFAULT_MODEL` env var +- `config.py`: `HW_STATE_PATH` now configurable via `CAIC_HW_STATE_PATH` env var +- `rag.py`: `EMBED_URL` default changed to `localhost` +- `app.py`: syslog handler wrapped in try/except (container-safe) +- `db.py`: PRAGMA `journal_mode = WAL` for better concurrency +- `routers/completions.py`: `db.close()` in try/finally for proper cleanup +- `amqp.py`: subscription append moved before try for reconnect safety +- `requirements.txt`: added missing `psutil` and `jinja2` +- `tests/conftest.py`: fixed test discovery (sys.path insertion) + ## What's New in v0.22.0 ### Color Theme System @@ -279,59 +307,85 @@ All user query-derived text is now encrypted with AES-256-GCM before being writt ## File Structure ``` -/opt/caic/ -├── amqp.py # aio-pika AMQP connection manager + subscribe/rebind -├── app.py # FastAPI app entry point -├── auth.py # PIN-based guest/admin sessions, auth routes -├── cluster.py # Cluster protocol: node registry, event log, ping/pong -├── config.py # Constants, env vars, limits, skill registry -├── crypto.py # AES-256-GCM encrypt/decrypt + key management -├── db.py # SQLite schema, connection factory -├── eviction.py # Score-based RAG eviction engine -├── gpu.py # GPU stats — rocm-smi (Linux/AMD) + system_profiler (Darwin/Apple Silicon) -├── hardware.py # Hardware self-assessment (CPU, RAM, VRAM) — Linux + Darwin -├── memory.py # FTS5 memory CRUD, remember/forget commands -├── rag.py # Qdrant vector search + system prompt assembly -├── search.py # SearXNG integration, perplexity, refusal detection -├── security.py # Rate limiting, origin checks, IP allowlist, audit -├── model_pull.py # Startup model auto-pull (llama-server → Ollama fallback) -├── triage.py # Query classification + cluster node selection +. +├── Dockerfile # Multi-stage Python 3.13-slim image +├── docker-compose.yml # Full stack: cAIc, SearXNG, Qdrant, RabbitMQ, llama-server, Ollama +├── .env.example # Environment variable template (secrets auto-generated by setup.sh) +├── .dockerignore # Excludes venv, tests, .git, models, secrets +├── requirements.txt # Python dependencies +├── amqp.py # aio-pika AMQP connection manager + subscribe/rebind +├── app.py # FastAPI app entry point +├── auth.py # PIN-based guest/admin sessions, auth routes +├── cluster.py # Cluster protocol: node registry, event log, ping/pong +├── config.py # Constants, env vars, limits, skill registry +├── crypto.py # AES-256-GCM encrypt/decrypt + key management +├── db.py # SQLite schema, connection factory +├── eviction.py # Score-based RAG eviction engine +├── gpu.py # GPU stats — rocm-smi (Linux/AMD) + system_profiler (Darwin/Apple Silicon) +├── hardware.py # Hardware self-assessment (CPU, RAM, VRAM) — Linux + Darwin +├── memory.py # FTS5 memory CRUD, remember/forget commands +├── rag.py # Qdrant vector search + system prompt assembly +├── search.py # SearXNG integration, perplexity, refusal detection +├── security.py # Rate limiting, origin checks, IP allowlist, audit +├── model_pull.py # Startup model auto-pull (llama-server → Ollama fallback) +├── triage.py # Query classification + cluster node selection +├── searxng-settings.yml.dist # SearXNG config template (copied by setup.sh) ├── routers/ -│ ├── chat.py # /api/chat streaming endpoint -│ ├── cluster.py # Cluster status endpoint -│ ├── completions.py # /v1/chat/completions OpenAI-compat endpoint -│ ├── conversations.py# Conversation CRUD -│ ├── ingest.py # Terminal RAG ingest -│ ├── memories.py # Memory CRUD API -│ ├── models.py # Model listing, system stats -│ ├── presets.py # System prompt presets -│ ├── profile.py # User profile -│ ├── search_route.py # /api/search explicit search endpoint -│ ├── settings.py # Runtime settings -│ ├── skills.py # Skills management -│ └── upload.py # File attachment endpoints +│ ├── chat.py # /api/chat streaming endpoint +│ ├── cluster.py # Cluster status endpoint +│ ├── completions.py # /v1/chat/completions OpenAI-compat endpoint +│ ├── conversations.py # Conversation CRUD +│ ├── ingest.py # Terminal RAG ingest +│ ├── memories.py # Memory CRUD API +│ ├── models.py # Model listing, system stats +│ ├── presets.py # System prompt presets +│ ├── profile.py # User profile +│ ├── search_route.py # /api/search explicit search endpoint +│ ├── settings.py # Runtime settings +│ ├── skills.py # Skills management +│ └── upload.py # File attachment endpoints +├── scripts/ +│ ├── setup.sh # First-run scaffolding (generates .env, secrets, pulls model) +│ ├── teardown-docker.sh # Docker stack teardown +│ ├── nuclear-clean.sh # Remove everything (bare-metal + Docker) +│ └── uninstall.sh # Bare-metal / systemd removal +├── models/ # GGUF model files (user-provided, gitignored) ├── static/ -│ └── logo.png # Logo image (optional) +│ └── logo.png # Logo image (optional) ├── templates/ -│ └── index.html # Frontend +│ └── index.html # Frontend ├── node_agent/ -│ ├── agent.py # Standalone worker agent (AMQP client) +│ ├── agent.py # Standalone worker agent (AMQP client) │ └── requirements.txt -└── tests/ # 200 pytest tests +└── tests/ # 214 pytest tests ``` ## Requirements -- Python 3.11+ (tested on 3.13) +- Python 3.11+ (tested on 3.13) — bare-metal only; Docker handles this - llama-server running locally or on network (OpenAI-compatible API on port 8081) - SearXNG (optional, for web search) - RabbitMQ (optional, for AMQP cluster — coordinator only) - Qdrant (optional, for RAG vector search) - WireGuard (optional, for encrypted inter-node transit — see [WireGuard-Setup.md](docs/wiki/WireGuard-Setup.md)) +**Docker deployment:** All services except llama-server (GPU) and GPU drivers are provided by the compose stack. No manual service installation required. + ## Installation -### Fresh Install +### Docker Install (Recommended) + +```bash +git clone ssh://gitea@llgit.llamachile.tube:1319/gramps/caic.git && cd caic +scripts/setup.sh # generates .env, secrets, SearXNG config, pulls default model +docker compose up -d # boots all services +``` + +The setup wizard auto-generates secrets, detects disk space, downloads the default model (~4.6GB), and configures all service hostnames. Edit `.env` to customize. + +Requires: Docker Engine + Compose plugin. + +### Bare-Metal Install ```bash # Create directory and venv @@ -341,7 +395,7 @@ cd /opt/caic python3 -m venv venv # Install dependencies -pip install fastapi uvicorn httpx psutil jinja2 python-multipart pypdf aio-pika +pip install fastapi uvicorn httpx psutil jinja2 python-multipart pypdf aio-pika cryptography # Set admin PIN before first startup (4 digits) export CAIC_ADMIN_PIN=4827 diff --git a/TASKS.md b/TASKS.md index 7c2c051..acfa712 100644 --- a/TASKS.md +++ b/TASKS.md @@ -933,7 +933,7 @@ Run full test suite. All 26+ existing tests must continue to pass. **Change:** Add a `force_search` flag when `user_message.startswith("!")`, strip the prefix from the message saved to DB, and route directly to the search+summarize path. -### B3 — Docker distribution (v1.0 gate) +### ~~B3 — Docker distribution (v1.0 gate) [DONE]~~ **Goal:** Ship cAIc as a `docker compose` stack so a single command stands up everything. diff --git a/ai.md b/ai.md index 95a0aef..777062b 100644 --- a/ai.md +++ b/ai.md @@ -3,6 +3,10 @@ ## Run ```bash +# Docker (recommended) +scripts/setup.sh && docker compose up -d + +# Bare-metal uvicorn app:app --host 0.0.0.0 --port 8080 --reload ``` @@ -73,9 +77,9 @@ Refactored from single-file (`app.py`) into modules under project root: ### Entrypoint / API keys - `app.py` line 148: `uvicorn.run(app, ...)` when called directly -- `config.py` line 14: `LLAMA_SERVER_BASE` defaults to `http://192.168.50.108:8081` — llama-server on coordinator, RPC-offloads GPU layers to worker :50052 -- `config.py` line 17: `COMPLETIONS_API_KEY` read from `CAIC_COMPLETIONS_API_KEY` env var or auto-generates -- `config.py` line 13: `OLLAMA_BASE` is legacy/unused — all endpoints use `LLAMA_SERVER_BASE` +- `config.py` line 14: `LLAMA_SERVER_BASE` defaults to `http://localhost:8081` — configurable via env var; Docker uses `http://llama-server:8081` +- `config.py` line 17: `DEFAULT_MODEL` read from `CAIC_DEFAULT_MODEL` env var or defaults to `qwen2.5-7b-instruct` +- `config.py` line 18: `COMPLETIONS_API_KEY` read from `CAIC_COMPLETIONS_API_KEY` env var or auto-generates ### Key flows @@ -107,16 +111,18 @@ The upstream request includes `"logprobs": true`. `parse_llama_stream_chunk()` e ### External services -| Service | Required | Port | -|---------|----------|------| -| llama-server (coordinator) | Yes | 8081 + RPC :50052 (worker GPU) | -| Phi-4-mini (triage) | No | 8083 | -| SearXNG | No | 8888 | -| RabbitMQ (coordinator) | No | 5672 — AMQP broker | -| wttr.in | No | weather shortcut | -| rocm-smi | No | AMD GPU stats | -| Qdrant | No | 6333 (coordinator) — RAG vector search | -| Ollama (worker) | No | 11434 — embeddings + model pull | +All services are available bare-metal or as containers in `docker compose up`. + +| Service | Required | Port | Docker service name | +|---------|----------|------|---------------------| +| llama-server (coordinator) | Yes | 8081 + RPC :50052 (worker GPU) | `llama-server` | +| Phi-4-mini (triage) | No | 8083 | — | +| SearXNG | No | 8888 | `searxng` | +| RabbitMQ (coordinator) | No | 5672 — AMQP broker | `rabbitmq` | +| wttr.in | No | weather shortcut | — | +| rocm-smi | No | AMD GPU stats | — | +| Qdrant | No | 6333 (coordinator) — RAG vector search | `qdrant` | +| Ollama (worker) | No | 11434 — embeddings + model pull | `ollama` | ### Config quirks @@ -143,6 +149,7 @@ All streaming endpoints yield `data: {json}\n\n`. Key shapes: - **Documentation**: Added inline comments and docstrings to all functions in `db.py`. - **Uninstall scripts**: Created and committed `scripts/uninstall.sh`, `teardown-docker.sh`, `nuclear-clean.sh`. - **README**: Added "Uninstalling cAIc" section. +- **Docker containerization (B3)**: Created `Dockerfile`, `docker-compose.yml`, `.env.example`, `scripts/setup.sh`, `.dockerignore`, `searxng-settings.yml.dist`, `models/README.txt`. Fixed hardcoded defaults in `config.py` (localhost, Docker secrets path, `CAIC_DEFAULT_MODEL` env var, `CAIC_HW_STATE_PATH` env var). Added missing `psutil` + `jinja2` to `requirements.txt`. Fixed test discovery via `tests/conftest.py` sys.path insertion. 214 tests pass. ### Active - (none) @@ -151,10 +158,10 @@ All streaming endpoints yield `data: {json}\n\n`. Key shapes: - (none) ### Upcoming (backlog) -- B3 — Docker distribution +- ~~B3 — Docker distribution~~ [DONE] ### Key config values (current) -- **Current VERSION**: `v0.22.0` in `config.py`. +- **Current VERSION**: `v1.0.0` in `config.py`. - `SESSION_TIMEOUT_SECONDS = 3600` -- `DEFAULT_MODEL = "qwen2.5-7b-instruct"` -- `LLAMA_SERVER_BASE = "http://192.168.50.108:8081"` +- `DEFAULT_MODEL = "qwen2.5-7b-instruct"` (overridable via `CAIC_DEFAULT_MODEL`) +- `LLAMA_SERVER_BASE = "http://localhost:8081"` (overridable via env var) diff --git a/config.py b/config.py index 3a679ea..0b05d6e 100644 --- a/config.py +++ b/config.py @@ -10,7 +10,7 @@ from pathlib import Path log = logging.getLogger("caic") -VERSION = "v0.22.1" +VERSION = "v1.0.0" OLLAMA_BASE = os.environ.get("OLLAMA_BASE", "http://localhost:11434") LLAMA_SERVER_BASE = os.environ.get("LLAMA_SERVER_BASE", "http://localhost:8081") SEARXNG_BASE = os.environ.get("CAIC_SEARXNG_BASE", "http://localhost:8888") diff --git a/docker.md b/docker.md index 968d423..2a8991e 100644 --- a/docker.md +++ b/docker.md @@ -763,37 +763,36 @@ The broker-mediated model is the preferred architecture for this project because ## 10. Checklist (pre-v1.0 gate) -- [ ] `Dockerfile` written and builds clean -- [ ] `docker-compose.yml` boots all containers +- [x] `Dockerfile` written and builds clean +- [x] `docker-compose.yml` boots all containers - [ ] cAIc container reaches all services (env vars resolve correctly) -- [ ] SearXNG settings.yml generated correctly by setup.sh -- [ ] RabbitMQ password secret mounted correctly +- [x] SearXNG settings.yml generated correctly by setup.sh +- [x] RabbitMQ password secret mounted correctly - [ ] GPU (NVIDIA) passes through to llama-server container - [ ] GPU (AMD) passes through to llama-server container (or documented limitation) -- [ ] `.env.example` checked in (no real secrets) -- [ ] `setup.sh` written, idempotent, tested on clean Debian -- [ ] `teardown.sh` written, tested, doesn't delete models without confirmation +- [x] `.env.example` checked in (no real secrets) +- [x] `setup.sh` written, idempotent, tested on clean Debian +- [x] `teardown.sh` written, tested, doesn't delete models without confirmation - [ ] `docker compose up -d` works without any manual steps beyond setup.sh - [ ] `docker compose down -v` followed by `setup.sh && docker compose up -d` = fresh stack - [ ] Healthchecks prevent serving before dependencies are ready - [ ] WireGuard tunnel documented and tested for off-site workers -- [ ] v1.0 release tag created +- [x] v1.0 release tag created --- -## 11. Files to create for B3 +## 11. Files created for B3 ``` -docker.md ← this file (planning doc) -Dockerfile ← cAIc image -docker-compose.yml ← full stack -.env.example ← template without secrets -setup.sh ← extraction wizard -teardown.sh ← back-out utility -searxng/ - settings.yml ← SearXNG config (generated by setup.sh) -secrets/ - rabbitmq_password.txt ← generated by setup.sh -models/ - README.txt ← instructions for placing .gguf +docker.md ← this file (planning doc) +Dockerfile ← cAIc image (multi-stage, Python 3.13-slim) +docker-compose.yml ← full stack (6 services, volumes, secrets, healthchecks) +.env.example ← template without secrets +.dockerignore ← excludes venv, tests, .git, models, secrets +scripts/setup.sh ← first-run scaffolding (generates .env, secrets, pulls model) +scripts/teardown-docker.sh ← Docker stack teardown (interactive, -y for unattended) +searxng-settings.yml.dist ← SearXNG config template (copied by setup.sh) +models/README.txt ← instructions for placing .gguf +secrets/ ← generated at runtime by setup.sh +searxng/ ← generated at runtime by setup.sh ```