gramps/cAIc

Fork 0

Table of Contents

Installation & Configuration

Architecture Overview
Prerequisites

Python packages (coordinator)
Optional external services

Quick Start (Bare Metal)

1. Clone the repo
2. Install Python dependencies
3. Set up llama-server
4. Configure cAIc
5. Run cAIc
6. (Optional) Set up as a systemd service

Docker Deployment
Cluster Setup

Coordinator with RabbitMQ
Adding a Worker Node

Configuration Reference

Payload & Rate Limits
RAG Settings
Model Configuration

Verifying the Installation
Security Checklist
Troubleshooting

"Origin check failed" (403)
"Client IP not allowed" (403)
"Rate limit exceeded" (429)
"Authentication required" (401)
"Admin PIN required for this action" (403)
LLM responds with "I don't have access to current information"
RabbitMQ connection refused
Worker not showing in cluster status
Qdrant connection failed
Database errors
"No module named 'caic'"
Logs

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

⇦ Back to Home

Installation & Configuration

Architecture Overview

cAIc splits into two machine roles:

Coordinator — runs the FastAPI app, broker, database, and all CPU-bound services. Does not need a GPU.
Workers — run only llama-server for GPU inference. No database, no HTTP API, no orchestration overhead.

You can start with a single machine acting as both coordinator and worker, then split off workers as your hardware fleet grows.

Prerequisites

Dependency	Minimum	Notes
Python	3.12+	3.13 recommended
OS	Linux	WSL2 on Windows works. macOS untested but may work with changes.
RAM	8 GB	16 GB+ recommended for coordinator with RAG
Disk	1 GB	Plus model files (~4–10 GB each)
GPU	Optional	Required for usable inference speed

Python packages (coordinator)

fastapi>=0.115.0
uvicorn[standard]>=0.32.0
httpx>=0.27.0
pypdf>=5.0.0
python-multipart>=0.0.9
aio-pika>=9.0.0
psutil>=5.9.0

Install: pip install -r requirements.txt

Optional external services

Service	Port	Purpose	Required?
llama-server	8081	LLM inference (OpenAI-compat)	Yes
RabbitMQ	5672	AMQP broker for cluster messaging	No (single-node skip)
Qdrant	6333	Vector database for RAG	No
SearXNG	8888	Privacy-respecting web search	No
Phi-4-mini	8083	Query triage classification	No (falls back to keywords)
Ollama	11434	Text embeddings for RAG	No (if RAG disabled)

All optional services gracefully degrade when absent.

Quick Start (Bare Metal)

1. Clone the repo

git clone git@llgit.llamachile.tube:gramps/cAIc.git    # SSH
# or
git clone https://llgit.llamachile.tube/gramps/cAIc.git  # HTTPS
cd cAIc

2. Install Python dependencies

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pip install psutil  # for hardware stats

3. Set up llama-server

The default model is qwen2.5-7b-instruct (7B parameters, instruction-tuned). You can use any GGUF model — the name just needs to match what select_node() looks for (models with "coder" or "qwen" in the name route to code queries; "mistral" or "llama" route to general queries).

# Download a llama-server binary
wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server
chmod +x llama-server

# Place a GGUF model file
mkdir -p models
# Download from HuggingFace, e.g.:
# wget -O models/qwen2.5-7b-instruct-q5_k_m.gguf \
#   https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF/resolve/main/qwen2.5-7b-instruct-q5_k_m.gguf

# Run llama-server
./llama-server \
  --host 0.0.0.0 --port 8081 \
  --model models/qwen2.5-7b-instruct-q5_k_m.gguf \
  --ctx-size 4096 \
  --embeddings \
  --logprobs \
  --n-gpu-layers 99           # offload 99 layers to GPU

Verify it's running: curl http://localhost:8081/health

Note: Auto-download of the default model on first start is planned but not yet implemented. You must place a GGUF file in the models/ directory manually for now.

4. Configure cAIc

Copy and edit configuration via environment variables:

export LLAMA_SERVER_BASE=http://localhost:8081
export CAIC_ADMIN_PIN=1234           # change this!
export CAIC_ALLOW_DEFAULT_PIN=true   # set false after first login

Key environment variables:

Variable	Default	Purpose
`LLAMA_SERVER_BASE`	`http://192.168.50.108:8081`	llama-server URL
`OLLAMA_BASE`	`http://localhost:11434`	Embeddings endpoint (legacy)
`CAIC_ADMIN_PIN`	(auto-required)	4-digit admin PIN
`CAIC_ALLOW_DEFAULT_PIN`	`false`	Allow weak PIN in dev
`CAIC_COMPLETIONS_API_KEY`	(auto-generated)	Bearer token for `/v1/chat/completions`
`CAIC_AMQP_URL`	(file-based)	RabbitMQ connection string
`CAIC_ALLOWED_CIDRS`	(LAN defaults)	IP allowlist CIDRs
`CAIC_TRUSTED_ORIGINS`	(none)	Additional CORS origins
`CAIC_TRUST_X_FORWARDED_FOR`	`false`	Trust reverse proxy IPs
`CAIC_TRIAGE_BASE`	`http://127.0.0.1:8083/v1`	Phi-4-mini triage endpoint
`QDRANT_URL`	`http://192.168.50.108:6333`	Qdrant vector DB URL

5. Run cAIc

uvicorn app:app --host 0.0.0.0 --port 8080 --reload

Open http://localhost:8080 in your browser. Click "Admin Login" and enter your PIN.

6. (Optional) Set up as a systemd service

[Unit]
Description=cAIc Cluster AI Chat
After=network.target

[Service]
Type=simple
User=gramps
Group=gramps
WorkingDirectory=/opt/jarvischat
ExecStart=/opt/jarvischat/venv/bin/uvicorn app:app --host 0.0.0.0 --port 8080
Restart=always
RestartSec=5
Environment=LLAMA_SERVER_BASE=http://localhost:8081
Environment=CAIC_ADMIN_PIN=1319
Environment=CAIC_ALLOW_DEFAULT_PIN=true
Environment=CAIC_COMPLETIONS_API_KEY=caic-sk-...

[Install]
WantedBy=multi-user.target

Save to /etc/systemd/system/caic.service, then:

sudo systemctl daemon-reload
sudo systemctl enable --now caic.service

Docker Deployment

Note: Docker deployment (B3) is planned for v1.0. The compose stack and setup wizard are designed but not yet released. See docker.md for the architecture planning doc.

The v1.0 stack will ship:

Dockerfile — multi-stage Python 3.13-slim build
docker-compose.yml — cAIc + SearXNG + Qdrant + RabbitMQ + llama-server + Ollama
setup.sh — interactive wizard that probes hardware and generates .env
teardown.sh — clean uninstall preserving user data

For now, bare-metal is the supported deployment model.

Cluster Setup

Coordinator with RabbitMQ

# Install RabbitMQ
apt install rabbitmq-server
systemctl enable --now rabbitmq-server

# Create user and vhost
rabbitmqctl add_user caic "$(openssl rand -hex 20)"
rabbitmqctl add_vhost caic
rabbitmqctl set_permissions -p caic caic ".*" ".*" ".*"

# Save password for cAIc
echo -n "$PASSWORD" > /home/gramps/.caic_amqp_secret
chmod 600 /home/gramps/.caic_amqp_secret

Start cAIc with CAIC_AMQP_URL set (or it reads from the secret file).

Adding a Worker Node

Each worker machine needs only llama-server and the node agent.

# Install llama-server (download binary or build from source)
wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server
chmod +x llama-server

# Install node agent deps
pip install aio-pika httpx psutil

# Configure
mkdir -p /etc/caic
cat > /etc/caic/node-agent.conf << 'EOF'
[agent]
node_name = $(hostname)
node_ip = $(hostname -I | awk '{print $1}')
node_type = worker
capabilities = llm
amqp_url = amqp://caic:PASSWORD@COORDINATOR_IP:5672/caic
llama_port = 8081
models_dir = /var/lib/caic/models
active_model = qwen2.5-7b-instruct-q5_k_m.gguf
EOF

# Start node agent (from repo checkout)
python3 /opt/caic/node_agent/agent.py

Workers can also be set up as systemd services (see node_agent/agent.py for unit file template).

Configuration Reference

Payload & Rate Limits

Tunable in config.py or via environment overrides:

Setting	Default	Description
`SESSION_TIMEOUT_SECONDS`	90	Session idle timeout
`MAX_PIN_ATTEMPTS`	5	PIN lockout threshold
`PIN_LOCKOUT_SECONDS`	300	PIN lockout duration
`RATE_WINDOW_SECONDS`	60	Rate limit window
`RL_CHAT_PER_WINDOW`	24	Max chat requests per window
`RL_SEARCH_PER_WINDOW`	16	Max search requests per window
`BODY_LIMIT_CHAT_BYTES`	128 KB	Max chat payload
`MAX_UPLOAD_BYTES`	20 MB	Max file upload

RAG Settings

Setting	Default	Description
`RAG_MAX_VECTORS`	50000	Max vectors before eviction
`RAG_EVICTION_HIGH_WATER`	0.80	Trigger eviction at 80%
`RAG_EVICTION_LOW_WATER`	0.20	Stop eviction at 20%
`RAG_PINNED_SOURCES`	`upload, profile`	Never evict these sources
`RAG_GRACE_HOURS`	1	Min age before eviction eligible

Model Configuration

DEFAULT_MODEL in config.py (default: "qwen2.5-7b-instruct") sets the model name used for inference. This string must match the model name that llama-server reports in its model list — it's used for routing, not for downloading.

When llama-server loads a model, its visible name determines how select_node() matches it. The triage system maps queries to ideal model families:

code → models with "coder" or "qwen" in the name
general → models with "mistral" or "llama" in the name

For cluster mode, each worker advertises its loaded model. The coordinator selects the best-matching worker for each query.

Auto-download on first start: Not yet implemented (see backlog). For now, download a GGUF manually and place it in models/.

Verifying the Installation

# Check the app is running
curl http://localhost:8080/

# Check health endpoints
curl http://localhost:8080/api/hardware    # hardware probe results
curl http://localhost:8080/api/models      # loaded models
curl http://localhost:8080/api/cluster     # cluster status

# Get a guest session
curl -X POST http://localhost:8080/api/auth/guest \
  -H "Content-Type: application/json" \
  -d '{}'

Security Checklist

Set a strong CAIC_ADMIN_PIN (not 1234, not your birthday)
Set CAIC_ALLOW_DEFAULT_PIN=false after first login
Generate a strong CAIC_COMPLETIONS_API_KEY
Review CAIC_ALLOWED_CIDRS — defaults allow all RFC1918 space
Set CAIC_TRUSTED_ORIGINS if accessing from non-LAN origins
Put cAIc behind a reverse proxy (Caddy, nginx) for HTTPS if exposed beyond LAN
Change RabbitMQ password from default
Enable CAIC_TRUST_X_FORWARDED_FOR=true if behind reverse proxy

Troubleshooting

"Origin check failed" (403)

cAIc requires either an Origin or Referer header on all /api/ requests. Browser requests include these automatically. For curl:

curl -H "Origin: http://localhost:8080" ...

If you see this from a legitimate browser client, check your proxy configuration — it may be stripping headers.

"Client IP not allowed" (403)

Your IP is not in the allowed CIDR list. Check CAIC_ALLOWED_CIDRS:

# Temporarily allow all (dev only)
export CAIC_ALLOWED_CIDRS="0.0.0.0/0,::/0"

Default allowlist covers 127.0.0.0/8, ::1/128, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16.

"Rate limit exceeded" (429)

Wait for the rate window to reset (default 60s) or increase limits:

# config.py
RL_CHAT_PER_WINDOW = 60   # was 24

"Authentication required" (401)

You need a valid session. Get a guest session first:

curl -X POST http://localhost:8080/api/auth/guest \
  -H "Content-Type: application/json" \
  -d '{}' \
  -H "Origin: http://localhost:8080"

Use the returned session_id as the x-session-id header on subsequent requests.

"Admin PIN required for this action" (403)

curl -X POST http://localhost:8080/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"pin": "1319"}' \
  -H "x-session-id: YOUR_SESSION_ID"

LLM responds with "I don't have access to current information"

This is the auto-search trigger. cAIc detects uncertainty (perplexity > 15.0) or refusal patterns and re-queries with web search results. If web search isn't available (SearXNG not running):

Set search_enabled=false in settings
Or start SearXNG: docker run -d -p 8888:8080 searxng/searxng:latest

RabbitMQ connection refused

# Check RabbitMQ is running
systemctl status rabbitmq-server

# Check the port
ss -tlnp | grep 5672

# Verify credentials
rabbitmqctl list_users
rabbitmqctl authenticate_user caic "your-password"

Worker not showing in cluster status

# On the worker, check the node agent logs
journalctl -u caic-node-agent --no-pager -n 50

# Verify the worker can reach RabbitMQ
nc -zv 192.168.50.108 5672

# On the coordinator, check cluster events
curl http://localhost:8080/api/cluster

Qdrant connection failed

# Check Qdrant is running
curl http://localhost:6333/healthz

# Verify the URL in config
# QDRANT_URL should point to the coordinator's Qdrant instance

Database errors

cAIc creates caic.db in the working directory automatically. If you see database errors:

# Check file permissions
ls -la caic.db

# Wipe and restart (data loss!)
rm caic.db
# Restart cAIc — init_db() recreates tables

"No module named 'caic'"

cAIc is not an installable package — it runs directly from the checkout directory. Make sure you're running from the repo root:

cd /opt/jarvischat
python3 -m uvicorn app:app --host 0.0.0.0 --port 8080

Logs

cAIc logs to syslog. Check logs with:

journalctl -t caic --no-pager -n 100
tail -f /var/log/syslog | grep caic

Incident keys in error responses can be looked up:

journalctl -t caic --no-pager | grep "<incident-key>"