From 1f162fe6ebad48f4356329f27c87ab6dd088c3f3 Mon Sep 17 00:00:00 2001 From: gramps Date: Mon, 13 Jul 2026 10:45:50 -0700 Subject: [PATCH] add Installation & Configuration page with full setup guide and troubleshooting --- Home.md | 1 + Installation.md | 422 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 423 insertions(+) create mode 100644 Installation.md diff --git a/Home.md b/Home.md index 5d4a8a4..4f90945 100755 --- a/Home.md +++ b/Home.md @@ -9,6 +9,7 @@ This wiki is the developer-facing architecture and process reference for cAIc. ## Start Here +- [Installation & Configuration](Installation) — setup guide, bare metal, Docker, cluster, troubleshooting - Architecture and components: [Developer Architecture](Developer-Architecture) - Active implementation backlog: [Current WiP](Current-WiP) - [Screenshots](Screenshots) — gallery of the UI in action diff --git a/Installation.md b/Installation.md new file mode 100644 index 0000000..d6079ca --- /dev/null +++ b/Installation.md @@ -0,0 +1,422 @@ +# Installation & Configuration + +## Architecture Overview + +cAIc splits into two machine roles: + +- **Coordinator** — runs the FastAPI app, broker, database, and all CPU-bound services. Does not need a GPU. +- **Workers** — run only `llama-server` for GPU inference. No database, no HTTP API, no orchestration overhead. + +You can start with a single machine acting as both coordinator and worker, then split off workers as your hardware fleet grows. + +## Prerequisites + +| Dependency | Minimum | Notes | +|------------|---------|-------| +| Python | 3.12+ | 3.13 recommended | +| OS | Linux | WSL2 on Windows works. macOS untested but may work with changes. | +| RAM | 8 GB | 16 GB+ recommended for coordinator with RAG | +| Disk | 1 GB | Plus model files (~4–10 GB each) | +| GPU | Optional | Required for usable inference speed | + +### Python packages (coordinator) + +``` +fastapi>=0.115.0 +uvicorn[standard]>=0.32.0 +httpx>=0.27.0 +pypdf>=5.0.0 +python-multipart>=0.0.9 +aio-pika>=9.0.0 +psutil>=5.9.0 +``` + +Install: `pip install -r requirements.txt` + +### Optional external services + +| Service | Port | Purpose | Required? | +|---------|------|---------|-----------| +| **llama-server** | 8081 | LLM inference (OpenAI-compat) | **Yes** | +| **RabbitMQ** | 5672 | AMQP broker for cluster messaging | No (single-node skip) | +| **Qdrant** | 6333 | Vector database for RAG | No | +| **SearXNG** | 8888 | Privacy-respecting web search | No | +| **Phi-4-mini** | 8083 | Query triage classification | No (falls back to keywords) | +| **Ollama** | 11434 | Text embeddings for RAG | No (if RAG disabled) | + +All optional services gracefully degrade when absent. + +## Quick Start (Bare Metal) + +### 1. Clone the repo + +```bash +git clone git@llgit.llamachile.tube:gramps/cAIc.git # SSH +# or +git clone https://llgit.llamachile.tube/gramps/cAIc.git # HTTPS +cd cAIc +``` + +### 2. Install Python dependencies + +```bash +python3 -m venv venv +source venv/bin/activate +pip install -r requirements.txt +pip install psutil # for hardware stats +``` + +### 3. Set up llama-server + +```bash +# Download a llama-server binary +wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server +chmod +x llama-server + +# Place a GGUF model file +mkdir -p models +# Download from HuggingFace, e.g.: +# wget -O models/qwen2.5-7b-instruct-q5_k_m.gguf + +# Run llama-server +./llama-server \ + --host 0.0.0.0 --port 8081 \ + --model models/qwen2.5-7b-instruct-q5_k_m.gguf \ + --ctx-size 4096 \ + --embeddings \ + --logprobs \ + --n-gpu-layers 99 # offload 99 layers to GPU +``` + +Verify it's running: `curl http://localhost:8081/health` + +### 4. Configure cAIc + +Copy and edit configuration via environment variables: + +```bash +export LLAMA_SERVER_BASE=http://localhost:8081 +export CAIC_ADMIN_PIN=1234 # change this! +export CAIC_ALLOW_DEFAULT_PIN=true # set false after first login +``` + +Key environment variables: + +| Variable | Default | Purpose | +|----------|---------|---------| +| `LLAMA_SERVER_BASE` | `http://192.168.50.108:8081` | llama-server URL | +| `OLLAMA_BASE` | `http://localhost:11434` | Embeddings endpoint (legacy) | +| `CAIC_ADMIN_PIN` | *(auto-required)* | 4-digit admin PIN | +| `CAIC_ALLOW_DEFAULT_PIN` | `false` | Allow weak PIN in dev | +| `CAIC_COMPLETIONS_API_KEY` | *(auto-generated)* | Bearer token for `/v1/chat/completions` | +| `CAIC_AMQP_URL` | *(file-based)* | RabbitMQ connection string | +| `CAIC_ALLOWED_CIDRS` | *(LAN defaults)* | IP allowlist CIDRs | +| `CAIC_TRUSTED_ORIGINS` | *(none)* | Additional CORS origins | +| `CAIC_TRUST_X_FORWARDED_FOR` | `false` | Trust reverse proxy IPs | +| `CAIC_TRIAGE_BASE` | `http://127.0.0.1:8083/v1` | Phi-4-mini triage endpoint | +| `QDRANT_URL` | `http://192.168.50.108:6333` | Qdrant vector DB URL | + +### 5. Run cAIc + +```bash +uvicorn app:app --host 0.0.0.0 --port 8080 --reload +``` + +Open `http://localhost:8080` in your browser. Click "Admin Login" and enter your PIN. + +### 6. (Optional) Set up as a systemd service + +``` +[Unit] +Description=cAIc Cluster AI Chat +After=network.target + +[Service] +Type=simple +User=gramps +Group=gramps +WorkingDirectory=/opt/jarvischat +ExecStart=/opt/jarvischat/venv/bin/uvicorn app:app --host 0.0.0.0 --port 8080 +Restart=always +RestartSec=5 +Environment=LLAMA_SERVER_BASE=http://localhost:8081 +Environment=CAIC_ADMIN_PIN=1319 +Environment=CAIC_ALLOW_DEFAULT_PIN=true +Environment=CAIC_COMPLETIONS_API_KEY=caic-sk-... + +[Install] +WantedBy=multi-user.target +``` + +Save to `/etc/systemd/system/caic.service`, then: + +```bash +sudo systemctl daemon-reload +sudo systemctl enable --now caic.service +``` + +## Docker Deployment + +> **Note:** Docker deployment (B3) is planned for v1.0. The compose stack and setup wizard are designed but not yet released. See [`docker.md`](https://llgit.llamachile.tube/gramps/cAIc/blob/main/docker.md) for the architecture planning doc. + +The v1.0 stack will ship: +- `Dockerfile` — multi-stage Python 3.13-slim build +- `docker-compose.yml` — cAIc + SearXNG + Qdrant + RabbitMQ + llama-server + Ollama +- `setup.sh` — interactive wizard that probes hardware and generates `.env` +- `teardown.sh` — clean uninstall preserving user data + +For now, bare-metal is the supported deployment model. + +## Cluster Setup + +### Coordinator with RabbitMQ + +```bash +# Install RabbitMQ +apt install rabbitmq-server +systemctl enable --now rabbitmq-server + +# Create user and vhost +rabbitmqctl add_user caic "$(openssl rand -hex 20)" +rabbitmqctl add_vhost caic +rabbitmqctl set_permissions -p caic caic ".*" ".*" ".*" + +# Save password for cAIc +echo -n "$PASSWORD" > /home/gramps/.caic_amqp_secret +chmod 600 /home/gramps/.caic_amqp_secret +``` + +Start cAIc with `CAIC_AMQP_URL` set (or it reads from the secret file). + +### Adding a Worker Node + +Each worker machine needs only `llama-server` and the node agent. + +```bash +# Install llama-server (download binary or build from source) +wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server +chmod +x llama-server + +# Install node agent deps +pip install aio-pika httpx psutil + +# Configure +mkdir -p /etc/caic +cat > /etc/caic/node-agent.conf << 'EOF' +[agent] +node_name = $(hostname) +node_ip = $(hostname -I | awk '{print $1}') +node_type = worker +capabilities = llm +amqp_url = amqp://caic:PASSWORD@COORDINATOR_IP:5672/caic +llama_port = 8081 +models_dir = /var/lib/caic/models +active_model = qwen2.5-7b-instruct-q5_k_m.gguf +EOF + +# Start node agent (from repo checkout) +python3 /opt/caic/node_agent/agent.py +``` + +Workers can also be set up as systemd services (see `node_agent/agent.py` for unit file template). + +## Configuration Reference + +### Payload & Rate Limits + +Tunable in `config.py` or via environment overrides: + +| Setting | Default | Description | +|---------|---------|-------------| +| `SESSION_TIMEOUT_SECONDS` | 90 | Session idle timeout | +| `MAX_PIN_ATTEMPTS` | 5 | PIN lockout threshold | +| `PIN_LOCKOUT_SECONDS` | 300 | PIN lockout duration | +| `RATE_WINDOW_SECONDS` | 60 | Rate limit window | +| `RL_CHAT_PER_WINDOW` | 24 | Max chat requests per window | +| `RL_SEARCH_PER_WINDOW` | 16 | Max search requests per window | +| `BODY_LIMIT_CHAT_BYTES` | 128 KB | Max chat payload | +| `MAX_UPLOAD_BYTES` | 20 MB | Max file upload | + +### RAG Settings + +| Setting | Default | Description | +|---------|---------|-------------| +| `RAG_MAX_VECTORS` | 50000 | Max vectors before eviction | +| `RAG_EVICTION_HIGH_WATER` | 0.80 | Trigger eviction at 80% | +| `RAG_EVICTION_LOW_WATER` | 0.20 | Stop eviction at 20% | +| `RAG_PINNED_SOURCES` | `upload, profile` | Never evict these sources | +| `RAG_GRACE_HOURS` | 1 | Min age before eviction eligible | + +### Model Configuration + +`DEFAULT_MODEL` in `config.py` sets the model name used for inference. When llama-server loads a model, its visible name in the model list determines how `select_node()` matches it. The triage system maps queries to ideal model families: + +- `code` → models with "coder" or "qwen" in the name +- `general` → models with "mistral" or "llama" in the name + +For cluster mode, each worker advertises its loaded model. The coordinator selects the best-matching worker for each query. + +## Verifying the Installation + +```bash +# Check the app is running +curl http://localhost:8080/ + +# Check health endpoints +curl http://localhost:8080/api/hardware # hardware probe results +curl http://localhost:8080/api/models # loaded models +curl http://localhost:8080/api/cluster # cluster status + +# Get a guest session +curl -X POST http://localhost:8080/api/auth/guest \ + -H "Content-Type: application/json" \ + -d '{}' +``` + +## Security Checklist + +- [ ] Set a strong `CAIC_ADMIN_PIN` (not 1234, not your birthday) +- [ ] Set `CAIC_ALLOW_DEFAULT_PIN=false` after first login +- [ ] Generate a strong `CAIC_COMPLETIONS_API_KEY` +- [ ] Review `CAIC_ALLOWED_CIDRS` — defaults allow all RFC1918 space +- [ ] Set `CAIC_TRUSTED_ORIGINS` if accessing from non-LAN origins +- [ ] Put cAIc behind a reverse proxy (Caddy, nginx) for HTTPS if exposed beyond LAN +- [ ] Change RabbitMQ password from default +- [ ] Enable `CAIC_TRUST_X_FORWARDED_FOR=true` if behind reverse proxy + +## Troubleshooting + +### "Origin check failed" (403) + +cAIc requires either an `Origin` or `Referer` header on all `/api/` requests. Browser requests include these automatically. For `curl`: + +```bash +curl -H "Origin: http://localhost:8080" ... +``` + +If you see this from a legitimate browser client, check your proxy configuration — it may be stripping headers. + +### "Client IP not allowed" (403) + +Your IP is not in the allowed CIDR list. Check `CAIC_ALLOWED_CIDRS`: + +```bash +# Temporarily allow all (dev only) +export CAIC_ALLOWED_CIDRS="0.0.0.0/0,::/0" +``` + +Default allowlist covers `127.0.0.0/8`, `::1/128`, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`. + +### "Rate limit exceeded" (429) + +Wait for the rate window to reset (default 60s) or increase limits: + +```bash +# config.py +RL_CHAT_PER_WINDOW = 60 # was 24 +``` + +### "Authentication required" (401) + +You need a valid session. Get a guest session first: + +```bash +curl -X POST http://localhost:8080/api/auth/guest \ + -H "Content-Type: application/json" \ + -d '{}' \ + -H "Origin: http://localhost:8080" +``` + +Use the returned `session_id` as the `x-session-id` header on subsequent requests. + +### "Admin PIN required for this action" (403) + +Log in as admin first: + +```bash +curl -X POST http://localhost:8080/api/auth/login \ + -H "Content-Type: application/json" \ + -d '{"pin": "1319"}' \ + -H "x-session-id: YOUR_SESSION_ID" +``` + +### LLM responds with "I don't have access to current information" + +This is the auto-search trigger. cAIc detects uncertainty (perplexity > 15.0) or refusal patterns and re-queries with web search results. If web search isn't available (SearXNG not running): + +- Set `search_enabled=false` in settings +- Or start SearXNG: `docker run -d -p 8888:8080 searxng/searxng:latest` + +### RabbitMQ connection refused + +```bash +# Check RabbitMQ is running +systemctl status rabbitmq-server + +# Check the port +ss -tlnp | grep 5672 + +# Verify credentials +rabbitmqctl list_users +rabbitmqctl authenticate_user caic "your-password" +``` + +### Worker not showing in cluster status + +```bash +# On the worker, check the node agent logs +journalctl -u caic-node-agent --no-pager -n 50 + +# Verify the worker can reach RabbitMQ +nc -zv 192.168.50.108 5672 + +# On the coordinator, check cluster events +curl http://localhost:8080/api/cluster +``` + +### Qdrant connection failed + +```bash +# Check Qdrant is running +curl http://localhost:6333/healthz + +# Verify the URL in config +# QDRANT_URL should point to the coordinator's Qdrant instance +``` + +### Database errors + +cAIc creates `caic.db` in the working directory automatically. If you see database errors: + +```bash +# Check file permissions +ls -la caic.db + +# Wipe and restart (data loss!) +rm caic.db +# Restart cAIc — init_db() recreates tables +``` + +### "No module named 'caic'" + +cAIc is not an installable package — it runs directly from the checkout directory. Make sure you're running from the repo root: + +```bash +cd /opt/jarvischat +python3 -m uvicorn app:app --host 0.0.0.0 --port 8080 +``` + +### Logs + +cAIc logs to syslog. Check logs with: + +```bash +journalctl -t caic --no-pager -n 100 +tail -f /var/log/syslog | grep caic +``` + +Incident keys in error responses can be looked up: + +```bash +journalctl -t caic --no-pager | grep "" +```