gramps/cAIc
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

add Installation & Configuration page with full setup guide and troubleshooting

Llama Chile Shop
2026-07-13 10:45:50 -07:00
parent 69721bc8e9
commit 1f162fe6eb
2 changed files with 423 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Home.md
+1
@@ -9,6 +9,7 @@ This wiki is the developer-facing architecture and process reference for cAIc.
## Start Here ## Start Here
- [Installation & Configuration](Installation) — setup guide, bare metal, Docker, cluster, troubleshooting
- Architecture and components: [Developer Architecture](Developer-Architecture) - Architecture and components: [Developer Architecture](Developer-Architecture)
- Active implementation backlog: [Current WiP](Current-WiP) - Active implementation backlog: [Current WiP](Current-WiP)
- [Screenshots](Screenshots) — gallery of the UI in action - [Screenshots](Screenshots) — gallery of the UI in action
Installation.md
+422
@@ -0,0 +1,422 @@
# Installation & Configuration
## Architecture Overview
cAIc splits into two machine roles:
- **Coordinator** — runs the FastAPI app, broker, database, and all CPU-bound services. Does not need a GPU.
- **Workers** — run only `llama-server` for GPU inference. No database, no HTTP API, no orchestration overhead.
You can start with a single machine acting as both coordinator and worker, then split off workers as your hardware fleet grows.
## Prerequisites
| Dependency | Minimum | Notes |
|------------|---------|-------|
| Python | 3.12+ | 3.13 recommended |
| OS | Linux | WSL2 on Windows works. macOS untested but may work with changes. |
| RAM | 8 GB | 16 GB+ recommended for coordinator with RAG |
| Disk | 1 GB | Plus model files (~4–10 GB each) |
| GPU | Optional | Required for usable inference speed |
### Python packages (coordinator)
```
fastapi>=0.115.0
uvicorn[standard]>=0.32.0
httpx>=0.27.0
pypdf>=5.0.0
python-multipart>=0.0.9
aio-pika>=9.0.0
psutil>=5.9.0
```
Install: `pip install -r requirements.txt`
### Optional external services
| Service | Port | Purpose | Required? |
|---------|------|---------|-----------|
| **llama-server** | 8081 | LLM inference (OpenAI-compat) | **Yes** |
| **RabbitMQ** | 5672 | AMQP broker for cluster messaging | No (single-node skip) |
| **Qdrant** | 6333 | Vector database for RAG | No |
| **SearXNG** | 8888 | Privacy-respecting web search | No |
| **Phi-4-mini** | 8083 | Query triage classification | No (falls back to keywords) |
| **Ollama** | 11434 | Text embeddings for RAG | No (if RAG disabled) |
All optional services gracefully degrade when absent.
## Quick Start (Bare Metal)
### 1. Clone the repo
```bash
git clone git@llgit.llamachile.tube:gramps/cAIc.git # SSH
# or
git clone https://llgit.llamachile.tube/gramps/cAIc.git # HTTPS
cd cAIc
```
### 2. Install Python dependencies
```bash
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pip install psutil # for hardware stats
```
### 3. Set up llama-server
```bash
# Download a llama-server binary
wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server
chmod +x llama-server
# Place a GGUF model file
mkdir -p models
# Download from HuggingFace, e.g.:
# wget -O models/qwen2.5-7b-instruct-q5_k_m.gguf <url>
# Run llama-server
./llama-server \
--host 0.0.0.0 --port 8081 \
--model models/qwen2.5-7b-instruct-q5_k_m.gguf \
--ctx-size 4096 \
--embeddings \
--logprobs \
--n-gpu-layers 99 # offload 99 layers to GPU
```
Verify it's running: `curl http://localhost:8081/health`
### 4. Configure cAIc
Copy and edit configuration via environment variables:
```bash
export LLAMA_SERVER_BASE=http://localhost:8081
export CAIC_ADMIN_PIN=1234 # change this!
export CAIC_ALLOW_DEFAULT_PIN=true # set false after first login
```
Key environment variables:
| Variable | Default | Purpose |
|----------|---------|---------|
| `LLAMA_SERVER_BASE` | `http://192.168.50.108:8081` | llama-server URL |
| `OLLAMA_BASE` | `http://localhost:11434` | Embeddings endpoint (legacy) |
| `CAIC_ADMIN_PIN` | *(auto-required)* | 4-digit admin PIN |
| `CAIC_ALLOW_DEFAULT_PIN` | `false` | Allow weak PIN in dev |
| `CAIC_COMPLETIONS_API_KEY` | *(auto-generated)* | Bearer token for `/v1/chat/completions` |
| `CAIC_AMQP_URL` | *(file-based)* | RabbitMQ connection string |
| `CAIC_ALLOWED_CIDRS` | *(LAN defaults)* | IP allowlist CIDRs |
| `CAIC_TRUSTED_ORIGINS` | *(none)* | Additional CORS origins |
| `CAIC_TRUST_X_FORWARDED_FOR` | `false` | Trust reverse proxy IPs |
| `CAIC_TRIAGE_BASE` | `http://127.0.0.1:8083/v1` | Phi-4-mini triage endpoint |
| `QDRANT_URL` | `http://192.168.50.108:6333` | Qdrant vector DB URL |
### 5. Run cAIc
```bash
uvicorn app:app --host 0.0.0.0 --port 8080 --reload
```
Open `http://localhost:8080` in your browser. Click "Admin Login" and enter your PIN.
### 6. (Optional) Set up as a systemd service
```
[Unit]
Description=cAIc Cluster AI Chat
After=network.target
[Service]
Type=simple
User=gramps
Group=gramps
WorkingDirectory=/opt/jarvischat
ExecStart=/opt/jarvischat/venv/bin/uvicorn app:app --host 0.0.0.0 --port 8080
Restart=always
RestartSec=5
Environment=LLAMA_SERVER_BASE=http://localhost:8081
Environment=CAIC_ADMIN_PIN=1319
Environment=CAIC_ALLOW_DEFAULT_PIN=true
Environment=CAIC_COMPLETIONS_API_KEY=caic-sk-...
[Install]
WantedBy=multi-user.target
```
Save to `/etc/systemd/system/caic.service`, then:
```bash
sudo systemctl daemon-reload
sudo systemctl enable --now caic.service
```
## Docker Deployment
> **Note:** Docker deployment (B3) is planned for v1.0. The compose stack and setup wizard are designed but not yet released. See [`docker.md`](https://llgit.llamachile.tube/gramps/cAIc/blob/main/docker.md) for the architecture planning doc.
The v1.0 stack will ship:
- `Dockerfile` — multi-stage Python 3.13-slim build
- `docker-compose.yml` — cAIc + SearXNG + Qdrant + RabbitMQ + llama-server + Ollama
- `setup.sh` — interactive wizard that probes hardware and generates `.env`
- `teardown.sh` — clean uninstall preserving user data
For now, bare-metal is the supported deployment model.
## Cluster Setup
### Coordinator with RabbitMQ
```bash
# Install RabbitMQ
apt install rabbitmq-server
systemctl enable --now rabbitmq-server
# Create user and vhost
rabbitmqctl add_user caic "$(openssl rand -hex 20)"
rabbitmqctl add_vhost caic
rabbitmqctl set_permissions -p caic caic ".*" ".*" ".*"
# Save password for cAIc
echo -n "$PASSWORD" > /home/gramps/.caic_amqp_secret
chmod 600 /home/gramps/.caic_amqp_secret
```
Start cAIc with `CAIC_AMQP_URL` set (or it reads from the secret file).
### Adding a Worker Node
Each worker machine needs only `llama-server` and the node agent.
```bash
# Install llama-server (download binary or build from source)
wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server
chmod +x llama-server
# Install node agent deps
pip install aio-pika httpx psutil
# Configure
mkdir -p /etc/caic
cat > /etc/caic/node-agent.conf << 'EOF'
[agent]
node_name = $(hostname)
node_ip = $(hostname -I | awk '{print $1}')
node_type = worker
capabilities = llm
amqp_url = amqp://caic:PASSWORD@COORDINATOR_IP:5672/caic
llama_port = 8081
models_dir = /var/lib/caic/models
active_model = qwen2.5-7b-instruct-q5_k_m.gguf
EOF
# Start node agent (from repo checkout)
python3 /opt/caic/node_agent/agent.py
```
Workers can also be set up as systemd services (see `node_agent/agent.py` for unit file template).
## Configuration Reference
### Payload & Rate Limits
Tunable in `config.py` or via environment overrides:
| Setting | Default | Description |
|---------|---------|-------------|
| `SESSION_TIMEOUT_SECONDS` | 90 | Session idle timeout |
| `MAX_PIN_ATTEMPTS` | 5 | PIN lockout threshold |
| `PIN_LOCKOUT_SECONDS` | 300 | PIN lockout duration |
| `RATE_WINDOW_SECONDS` | 60 | Rate limit window |
| `RL_CHAT_PER_WINDOW` | 24 | Max chat requests per window |
| `RL_SEARCH_PER_WINDOW` | 16 | Max search requests per window |
| `BODY_LIMIT_CHAT_BYTES` | 128 KB | Max chat payload |
| `MAX_UPLOAD_BYTES` | 20 MB | Max file upload |
### RAG Settings
| Setting | Default | Description |
|---------|---------|-------------|
| `RAG_MAX_VECTORS` | 50000 | Max vectors before eviction |
| `RAG_EVICTION_HIGH_WATER` | 0.80 | Trigger eviction at 80% |
| `RAG_EVICTION_LOW_WATER` | 0.20 | Stop eviction at 20% |
| `RAG_PINNED_SOURCES` | `upload, profile` | Never evict these sources |
| `RAG_GRACE_HOURS` | 1 | Min age before eviction eligible |
### Model Configuration
`DEFAULT_MODEL` in `config.py` sets the model name used for inference. When llama-server loads a model, its visible name in the model list determines how `select_node()` matches it. The triage system maps queries to ideal model families:
- `code` → models with "coder" or "qwen" in the name
- `general` → models with "mistral" or "llama" in the name
For cluster mode, each worker advertises its loaded model. The coordinator selects the best-matching worker for each query.
## Verifying the Installation
```bash
# Check the app is running
curl http://localhost:8080/
# Check health endpoints
curl http://localhost:8080/api/hardware # hardware probe results
curl http://localhost:8080/api/models # loaded models
curl http://localhost:8080/api/cluster # cluster status
# Get a guest session
curl -X POST http://localhost:8080/api/auth/guest \
-H "Content-Type: application/json" \
-d '{}'
```
## Security Checklist
- [ ] Set a strong `CAIC_ADMIN_PIN` (not 1234, not your birthday)
- [ ] Set `CAIC_ALLOW_DEFAULT_PIN=false` after first login
- [ ] Generate a strong `CAIC_COMPLETIONS_API_KEY`
- [ ] Review `CAIC_ALLOWED_CIDRS` — defaults allow all RFC1918 space
- [ ] Set `CAIC_TRUSTED_ORIGINS` if accessing from non-LAN origins
- [ ] Put cAIc behind a reverse proxy (Caddy, nginx) for HTTPS if exposed beyond LAN
- [ ] Change RabbitMQ password from default
- [ ] Enable `CAIC_TRUST_X_FORWARDED_FOR=true` if behind reverse proxy
## Troubleshooting
### "Origin check failed" (403)
cAIc requires either an `Origin` or `Referer` header on all `/api/` requests. Browser requests include these automatically. For `curl`:
```bash
curl -H "Origin: http://localhost:8080" ...
```
If you see this from a legitimate browser client, check your proxy configuration — it may be stripping headers.
### "Client IP not allowed" (403)
Your IP is not in the allowed CIDR list. Check `CAIC_ALLOWED_CIDRS`:
```bash
# Temporarily allow all (dev only)
export CAIC_ALLOWED_CIDRS="0.0.0.0/0,::/0"
```
Default allowlist covers `127.0.0.0/8`, `::1/128`, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`.
### "Rate limit exceeded" (429)
Wait for the rate window to reset (default 60s) or increase limits:
```bash
# config.py
RL_CHAT_PER_WINDOW = 60 # was 24
```
### "Authentication required" (401)
You need a valid session. Get a guest session first:
```bash
curl -X POST http://localhost:8080/api/auth/guest \
-H "Content-Type: application/json" \
-d '{}' \
-H "Origin: http://localhost:8080"
```
Use the returned `session_id` as the `x-session-id` header on subsequent requests.
### "Admin PIN required for this action" (403)
Log in as admin first:
```bash
curl -X POST http://localhost:8080/api/auth/login \
-H "Content-Type: application/json" \
-d '{"pin": "1319"}' \
-H "x-session-id: YOUR_SESSION_ID"
```
### LLM responds with "I don't have access to current information"
This is the auto-search trigger. cAIc detects uncertainty (perplexity > 15.0) or refusal patterns and re-queries with web search results. If web search isn't available (SearXNG not running):
- Set `search_enabled=false` in settings
- Or start SearXNG: `docker run -d -p 8888:8080 searxng/searxng:latest`
### RabbitMQ connection refused
```bash
# Check RabbitMQ is running
systemctl status rabbitmq-server
# Check the port
ss -tlnp | grep 5672
# Verify credentials
rabbitmqctl list_users
rabbitmqctl authenticate_user caic "your-password"
```
### Worker not showing in cluster status
```bash
# On the worker, check the node agent logs
journalctl -u caic-node-agent --no-pager -n 50
# Verify the worker can reach RabbitMQ
nc -zv 192.168.50.108 5672
# On the coordinator, check cluster events
curl http://localhost:8080/api/cluster
```
### Qdrant connection failed
```bash
# Check Qdrant is running
curl http://localhost:6333/healthz
# Verify the URL in config
# QDRANT_URL should point to the coordinator's Qdrant instance
```
### Database errors
cAIc creates `caic.db` in the working directory automatically. If you see database errors:
```bash
# Check file permissions
ls -la caic.db
# Wipe and restart (data loss!)
rm caic.db
# Restart cAIc — init_db() recreates tables
```
### "No module named 'caic'"
cAIc is not an installable package — it runs directly from the checkout directory. Make sure you're running from the repo root:
```bash
cd /opt/jarvischat
python3 -m uvicorn app:app --host 0.0.0.0 --port 8080
```
### Logs
cAIc logs to syslog. Check logs with:
```bash
journalctl -t caic --no-pager -n 100
tail -f /var/log/syslog | grep caic
```
Incident keys in error responses can be looked up:
```bash
journalctl -t caic --no-pager | grep "<incident-key>"
```