add Installation & Configuration page with full setup guide and troubleshooting
+1
@@ -9,6 +9,7 @@ This wiki is the developer-facing architecture and process reference for cAIc.
|
||||
|
||||
## Start Here
|
||||
|
||||
- [Installation & Configuration](Installation) — setup guide, bare metal, Docker, cluster, troubleshooting
|
||||
- Architecture and components: [Developer Architecture](Developer-Architecture)
|
||||
- Active implementation backlog: [Current WiP](Current-WiP)
|
||||
- [Screenshots](Screenshots) — gallery of the UI in action
|
||||
|
||||
+422
@@ -0,0 +1,422 @@
|
||||
# Installation & Configuration
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
cAIc splits into two machine roles:
|
||||
|
||||
- **Coordinator** — runs the FastAPI app, broker, database, and all CPU-bound services. Does not need a GPU.
|
||||
- **Workers** — run only `llama-server` for GPU inference. No database, no HTTP API, no orchestration overhead.
|
||||
|
||||
You can start with a single machine acting as both coordinator and worker, then split off workers as your hardware fleet grows.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Dependency | Minimum | Notes |
|
||||
|------------|---------|-------|
|
||||
| Python | 3.12+ | 3.13 recommended |
|
||||
| OS | Linux | WSL2 on Windows works. macOS untested but may work with changes. |
|
||||
| RAM | 8 GB | 16 GB+ recommended for coordinator with RAG |
|
||||
| Disk | 1 GB | Plus model files (~4–10 GB each) |
|
||||
| GPU | Optional | Required for usable inference speed |
|
||||
|
||||
### Python packages (coordinator)
|
||||
|
||||
```
|
||||
fastapi>=0.115.0
|
||||
uvicorn[standard]>=0.32.0
|
||||
httpx>=0.27.0
|
||||
pypdf>=5.0.0
|
||||
python-multipart>=0.0.9
|
||||
aio-pika>=9.0.0
|
||||
psutil>=5.9.0
|
||||
```
|
||||
|
||||
Install: `pip install -r requirements.txt`
|
||||
|
||||
### Optional external services
|
||||
|
||||
| Service | Port | Purpose | Required? |
|
||||
|---------|------|---------|-----------|
|
||||
| **llama-server** | 8081 | LLM inference (OpenAI-compat) | **Yes** |
|
||||
| **RabbitMQ** | 5672 | AMQP broker for cluster messaging | No (single-node skip) |
|
||||
| **Qdrant** | 6333 | Vector database for RAG | No |
|
||||
| **SearXNG** | 8888 | Privacy-respecting web search | No |
|
||||
| **Phi-4-mini** | 8083 | Query triage classification | No (falls back to keywords) |
|
||||
| **Ollama** | 11434 | Text embeddings for RAG | No (if RAG disabled) |
|
||||
|
||||
All optional services gracefully degrade when absent.
|
||||
|
||||
## Quick Start (Bare Metal)
|
||||
|
||||
### 1. Clone the repo
|
||||
|
||||
```bash
|
||||
git clone git@llgit.llamachile.tube:gramps/cAIc.git # SSH
|
||||
# or
|
||||
git clone https://llgit.llamachile.tube/gramps/cAIc.git # HTTPS
|
||||
cd cAIc
|
||||
```
|
||||
|
||||
### 2. Install Python dependencies
|
||||
|
||||
```bash
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
pip install psutil # for hardware stats
|
||||
```
|
||||
|
||||
### 3. Set up llama-server
|
||||
|
||||
```bash
|
||||
# Download a llama-server binary
|
||||
wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server
|
||||
chmod +x llama-server
|
||||
|
||||
# Place a GGUF model file
|
||||
mkdir -p models
|
||||
# Download from HuggingFace, e.g.:
|
||||
# wget -O models/qwen2.5-7b-instruct-q5_k_m.gguf <url>
|
||||
|
||||
# Run llama-server
|
||||
./llama-server \
|
||||
--host 0.0.0.0 --port 8081 \
|
||||
--model models/qwen2.5-7b-instruct-q5_k_m.gguf \
|
||||
--ctx-size 4096 \
|
||||
--embeddings \
|
||||
--logprobs \
|
||||
--n-gpu-layers 99 # offload 99 layers to GPU
|
||||
```
|
||||
|
||||
Verify it's running: `curl http://localhost:8081/health`
|
||||
|
||||
### 4. Configure cAIc
|
||||
|
||||
Copy and edit configuration via environment variables:
|
||||
|
||||
```bash
|
||||
export LLAMA_SERVER_BASE=http://localhost:8081
|
||||
export CAIC_ADMIN_PIN=1234 # change this!
|
||||
export CAIC_ALLOW_DEFAULT_PIN=true # set false after first login
|
||||
```
|
||||
|
||||
Key environment variables:
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `LLAMA_SERVER_BASE` | `http://192.168.50.108:8081` | llama-server URL |
|
||||
| `OLLAMA_BASE` | `http://localhost:11434` | Embeddings endpoint (legacy) |
|
||||
| `CAIC_ADMIN_PIN` | *(auto-required)* | 4-digit admin PIN |
|
||||
| `CAIC_ALLOW_DEFAULT_PIN` | `false` | Allow weak PIN in dev |
|
||||
| `CAIC_COMPLETIONS_API_KEY` | *(auto-generated)* | Bearer token for `/v1/chat/completions` |
|
||||
| `CAIC_AMQP_URL` | *(file-based)* | RabbitMQ connection string |
|
||||
| `CAIC_ALLOWED_CIDRS` | *(LAN defaults)* | IP allowlist CIDRs |
|
||||
| `CAIC_TRUSTED_ORIGINS` | *(none)* | Additional CORS origins |
|
||||
| `CAIC_TRUST_X_FORWARDED_FOR` | `false` | Trust reverse proxy IPs |
|
||||
| `CAIC_TRIAGE_BASE` | `http://127.0.0.1:8083/v1` | Phi-4-mini triage endpoint |
|
||||
| `QDRANT_URL` | `http://192.168.50.108:6333` | Qdrant vector DB URL |
|
||||
|
||||
### 5. Run cAIc
|
||||
|
||||
```bash
|
||||
uvicorn app:app --host 0.0.0.0 --port 8080 --reload
|
||||
```
|
||||
|
||||
Open `http://localhost:8080` in your browser. Click "Admin Login" and enter your PIN.
|
||||
|
||||
### 6. (Optional) Set up as a systemd service
|
||||
|
||||
```
|
||||
[Unit]
|
||||
Description=cAIc Cluster AI Chat
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=gramps
|
||||
Group=gramps
|
||||
WorkingDirectory=/opt/jarvischat
|
||||
ExecStart=/opt/jarvischat/venv/bin/uvicorn app:app --host 0.0.0.0 --port 8080
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
Environment=LLAMA_SERVER_BASE=http://localhost:8081
|
||||
Environment=CAIC_ADMIN_PIN=1319
|
||||
Environment=CAIC_ALLOW_DEFAULT_PIN=true
|
||||
Environment=CAIC_COMPLETIONS_API_KEY=caic-sk-...
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
Save to `/etc/systemd/system/caic.service`, then:
|
||||
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now caic.service
|
||||
```
|
||||
|
||||
## Docker Deployment
|
||||
|
||||
> **Note:** Docker deployment (B3) is planned for v1.0. The compose stack and setup wizard are designed but not yet released. See [`docker.md`](https://llgit.llamachile.tube/gramps/cAIc/blob/main/docker.md) for the architecture planning doc.
|
||||
|
||||
The v1.0 stack will ship:
|
||||
- `Dockerfile` — multi-stage Python 3.13-slim build
|
||||
- `docker-compose.yml` — cAIc + SearXNG + Qdrant + RabbitMQ + llama-server + Ollama
|
||||
- `setup.sh` — interactive wizard that probes hardware and generates `.env`
|
||||
- `teardown.sh` — clean uninstall preserving user data
|
||||
|
||||
For now, bare-metal is the supported deployment model.
|
||||
|
||||
## Cluster Setup
|
||||
|
||||
### Coordinator with RabbitMQ
|
||||
|
||||
```bash
|
||||
# Install RabbitMQ
|
||||
apt install rabbitmq-server
|
||||
systemctl enable --now rabbitmq-server
|
||||
|
||||
# Create user and vhost
|
||||
rabbitmqctl add_user caic "$(openssl rand -hex 20)"
|
||||
rabbitmqctl add_vhost caic
|
||||
rabbitmqctl set_permissions -p caic caic ".*" ".*" ".*"
|
||||
|
||||
# Save password for cAIc
|
||||
echo -n "$PASSWORD" > /home/gramps/.caic_amqp_secret
|
||||
chmod 600 /home/gramps/.caic_amqp_secret
|
||||
```
|
||||
|
||||
Start cAIc with `CAIC_AMQP_URL` set (or it reads from the secret file).
|
||||
|
||||
### Adding a Worker Node
|
||||
|
||||
Each worker machine needs only `llama-server` and the node agent.
|
||||
|
||||
```bash
|
||||
# Install llama-server (download binary or build from source)
|
||||
wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server
|
||||
chmod +x llama-server
|
||||
|
||||
# Install node agent deps
|
||||
pip install aio-pika httpx psutil
|
||||
|
||||
# Configure
|
||||
mkdir -p /etc/caic
|
||||
cat > /etc/caic/node-agent.conf << 'EOF'
|
||||
[agent]
|
||||
node_name = $(hostname)
|
||||
node_ip = $(hostname -I | awk '{print $1}')
|
||||
node_type = worker
|
||||
capabilities = llm
|
||||
amqp_url = amqp://caic:PASSWORD@COORDINATOR_IP:5672/caic
|
||||
llama_port = 8081
|
||||
models_dir = /var/lib/caic/models
|
||||
active_model = qwen2.5-7b-instruct-q5_k_m.gguf
|
||||
EOF
|
||||
|
||||
# Start node agent (from repo checkout)
|
||||
python3 /opt/caic/node_agent/agent.py
|
||||
```
|
||||
|
||||
Workers can also be set up as systemd services (see `node_agent/agent.py` for unit file template).
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
### Payload & Rate Limits
|
||||
|
||||
Tunable in `config.py` or via environment overrides:
|
||||
|
||||
| Setting | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| `SESSION_TIMEOUT_SECONDS` | 90 | Session idle timeout |
|
||||
| `MAX_PIN_ATTEMPTS` | 5 | PIN lockout threshold |
|
||||
| `PIN_LOCKOUT_SECONDS` | 300 | PIN lockout duration |
|
||||
| `RATE_WINDOW_SECONDS` | 60 | Rate limit window |
|
||||
| `RL_CHAT_PER_WINDOW` | 24 | Max chat requests per window |
|
||||
| `RL_SEARCH_PER_WINDOW` | 16 | Max search requests per window |
|
||||
| `BODY_LIMIT_CHAT_BYTES` | 128 KB | Max chat payload |
|
||||
| `MAX_UPLOAD_BYTES` | 20 MB | Max file upload |
|
||||
|
||||
### RAG Settings
|
||||
|
||||
| Setting | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| `RAG_MAX_VECTORS` | 50000 | Max vectors before eviction |
|
||||
| `RAG_EVICTION_HIGH_WATER` | 0.80 | Trigger eviction at 80% |
|
||||
| `RAG_EVICTION_LOW_WATER` | 0.20 | Stop eviction at 20% |
|
||||
| `RAG_PINNED_SOURCES` | `upload, profile` | Never evict these sources |
|
||||
| `RAG_GRACE_HOURS` | 1 | Min age before eviction eligible |
|
||||
|
||||
### Model Configuration
|
||||
|
||||
`DEFAULT_MODEL` in `config.py` sets the model name used for inference. When llama-server loads a model, its visible name in the model list determines how `select_node()` matches it. The triage system maps queries to ideal model families:
|
||||
|
||||
- `code` → models with "coder" or "qwen" in the name
|
||||
- `general` → models with "mistral" or "llama" in the name
|
||||
|
||||
For cluster mode, each worker advertises its loaded model. The coordinator selects the best-matching worker for each query.
|
||||
|
||||
## Verifying the Installation
|
||||
|
||||
```bash
|
||||
# Check the app is running
|
||||
curl http://localhost:8080/
|
||||
|
||||
# Check health endpoints
|
||||
curl http://localhost:8080/api/hardware # hardware probe results
|
||||
curl http://localhost:8080/api/models # loaded models
|
||||
curl http://localhost:8080/api/cluster # cluster status
|
||||
|
||||
# Get a guest session
|
||||
curl -X POST http://localhost:8080/api/auth/guest \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{}'
|
||||
```
|
||||
|
||||
## Security Checklist
|
||||
|
||||
- [ ] Set a strong `CAIC_ADMIN_PIN` (not 1234, not your birthday)
|
||||
- [ ] Set `CAIC_ALLOW_DEFAULT_PIN=false` after first login
|
||||
- [ ] Generate a strong `CAIC_COMPLETIONS_API_KEY`
|
||||
- [ ] Review `CAIC_ALLOWED_CIDRS` — defaults allow all RFC1918 space
|
||||
- [ ] Set `CAIC_TRUSTED_ORIGINS` if accessing from non-LAN origins
|
||||
- [ ] Put cAIc behind a reverse proxy (Caddy, nginx) for HTTPS if exposed beyond LAN
|
||||
- [ ] Change RabbitMQ password from default
|
||||
- [ ] Enable `CAIC_TRUST_X_FORWARDED_FOR=true` if behind reverse proxy
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Origin check failed" (403)
|
||||
|
||||
cAIc requires either an `Origin` or `Referer` header on all `/api/` requests. Browser requests include these automatically. For `curl`:
|
||||
|
||||
```bash
|
||||
curl -H "Origin: http://localhost:8080" ...
|
||||
```
|
||||
|
||||
If you see this from a legitimate browser client, check your proxy configuration — it may be stripping headers.
|
||||
|
||||
### "Client IP not allowed" (403)
|
||||
|
||||
Your IP is not in the allowed CIDR list. Check `CAIC_ALLOWED_CIDRS`:
|
||||
|
||||
```bash
|
||||
# Temporarily allow all (dev only)
|
||||
export CAIC_ALLOWED_CIDRS="0.0.0.0/0,::/0"
|
||||
```
|
||||
|
||||
Default allowlist covers `127.0.0.0/8`, `::1/128`, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`.
|
||||
|
||||
### "Rate limit exceeded" (429)
|
||||
|
||||
Wait for the rate window to reset (default 60s) or increase limits:
|
||||
|
||||
```bash
|
||||
# config.py
|
||||
RL_CHAT_PER_WINDOW = 60 # was 24
|
||||
```
|
||||
|
||||
### "Authentication required" (401)
|
||||
|
||||
You need a valid session. Get a guest session first:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/api/auth/guest \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{}' \
|
||||
-H "Origin: http://localhost:8080"
|
||||
```
|
||||
|
||||
Use the returned `session_id` as the `x-session-id` header on subsequent requests.
|
||||
|
||||
### "Admin PIN required for this action" (403)
|
||||
|
||||
Log in as admin first:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/api/auth/login \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"pin": "1319"}' \
|
||||
-H "x-session-id: YOUR_SESSION_ID"
|
||||
```
|
||||
|
||||
### LLM responds with "I don't have access to current information"
|
||||
|
||||
This is the auto-search trigger. cAIc detects uncertainty (perplexity > 15.0) or refusal patterns and re-queries with web search results. If web search isn't available (SearXNG not running):
|
||||
|
||||
- Set `search_enabled=false` in settings
|
||||
- Or start SearXNG: `docker run -d -p 8888:8080 searxng/searxng:latest`
|
||||
|
||||
### RabbitMQ connection refused
|
||||
|
||||
```bash
|
||||
# Check RabbitMQ is running
|
||||
systemctl status rabbitmq-server
|
||||
|
||||
# Check the port
|
||||
ss -tlnp | grep 5672
|
||||
|
||||
# Verify credentials
|
||||
rabbitmqctl list_users
|
||||
rabbitmqctl authenticate_user caic "your-password"
|
||||
```
|
||||
|
||||
### Worker not showing in cluster status
|
||||
|
||||
```bash
|
||||
# On the worker, check the node agent logs
|
||||
journalctl -u caic-node-agent --no-pager -n 50
|
||||
|
||||
# Verify the worker can reach RabbitMQ
|
||||
nc -zv 192.168.50.108 5672
|
||||
|
||||
# On the coordinator, check cluster events
|
||||
curl http://localhost:8080/api/cluster
|
||||
```
|
||||
|
||||
### Qdrant connection failed
|
||||
|
||||
```bash
|
||||
# Check Qdrant is running
|
||||
curl http://localhost:6333/healthz
|
||||
|
||||
# Verify the URL in config
|
||||
# QDRANT_URL should point to the coordinator's Qdrant instance
|
||||
```
|
||||
|
||||
### Database errors
|
||||
|
||||
cAIc creates `caic.db` in the working directory automatically. If you see database errors:
|
||||
|
||||
```bash
|
||||
# Check file permissions
|
||||
ls -la caic.db
|
||||
|
||||
# Wipe and restart (data loss!)
|
||||
rm caic.db
|
||||
# Restart cAIc — init_db() recreates tables
|
||||
```
|
||||
|
||||
### "No module named 'caic'"
|
||||
|
||||
cAIc is not an installable package — it runs directly from the checkout directory. Make sure you're running from the repo root:
|
||||
|
||||
```bash
|
||||
cd /opt/jarvischat
|
||||
python3 -m uvicorn app:app --host 0.0.0.0 --port 8080
|
||||
```
|
||||
|
||||
### Logs
|
||||
|
||||
cAIc logs to syslog. Check logs with:
|
||||
|
||||
```bash
|
||||
journalctl -t caic --no-pager -n 100
|
||||
tail -f /var/log/syslog | grep caic
|
||||
```
|
||||
|
||||
Incident keys in error responses can be looked up:
|
||||
|
||||
```bash
|
||||
journalctl -t caic --no-pager | grep "<incident-key>"
|
||||
```
|
||||
Reference in New Issue
Block a user