Table of Contents
- Frequently Asked Questions
- Why not just use Ollama?
- How is this different from llama.cpp RPC?
- How is this different from vLLM?
- How is this different from exo?
- How is this different from a Kubernetes operator (LLMKube)?
- Does cAIc support Apple Silicon?
- Can I run the coordinator and a worker on the same machine?
- Do I need RabbitMQ for a single-machine setup?
- What models work with cAIc?
- Can I expose cAIc to the internet?
- What happens if a worker goes offline?
- Does cAIc support OpenAI-compatible API?
- How many tests does cAIc have?
Frequently Asked Questions
Why not just use Ollama?
Ollama is a single-machine runner. It does not distribute inference across machines. If your model doesn't fit on one GPU, Ollama can't help — it doesn't support tensor parallelism across hosts, RPC offload, or any form of multi-node clustering. cAIc is for people who have multiple machines with different GPUs and want them to cooperate.
Ollama is also a model runner first and a chat UI second. cAIc is a full web application with conversation history, RAG, memory, user profiles, skills, web search, and cluster orchestration. They solve different problems.
How is this different from llama.cpp RPC?
llama.cpp RPC splits one model across multiple GPUs — GPU 0 runs layers 0–15, GPU 1 runs layers 16–31, etc. This is tensor parallelism and it has a lockstep problem: every forward pass must wait for the slowest GPU to finish its layers. In a heterogeneous cluster (AMD + NVIDIA, different VRAM sizes, different memory bandwidth), the fast GPU spends most of its time waiting on the slow one.
cAIc uses query-routing. Each worker runs a complete model on its own GPU. Queries are classified and routed to the best-matching worker. No layer sharing, no lockstep, no straggler problem. The tradeoff is that no single query can use combined VRAM across GPUs — but your cluster throughput stays high because every GPU runs independently.
| llama.cpp RPC | cAIc | |
|---|---|---|
| Splits one model across GPUs? | Yes (tensor parallel) | No (each worker runs full model) |
| Heterogeneous GPUs? | Slowed by weakest link | Each GPU independent |
| Requires fast fabric? | Yes (NCCL / RDMA) | No (LAN is fine) |
| Single query can use combined VRAM? | Yes | No |
| Cluster throughput capped by slowest GPU? | Yes | No |
How is this different from vLLM?
vLLM is designed for datacenter deployments with homogeneous GPU clusters and high request volume. It assumes identical GPUs, CUDA-only, and fast interconnects. It does not support AMD ROCm in any meaningful way, nor does it handle a two-machine homelab with one NVIDIA and one AMD card.
cAIc targets the opposite use case: a handful of mismatched machines on a LAN, where ease of setup matters more than maximum throughput.
How is this different from exo?
exo does zero-config peer discovery across mixed hardware including Apple Silicon, but its throughput is limited — it uses the same layer-splitting approach that cAIc avoids, and it has no persistent state (no conversation history, no RAG, no user profiles). exo is also experimental in ways that cAIc is not: network errors can cascade into full process restarts, and there's no admin model for access control.
cAIc trades exo's zero-config discovery for a more structured AMQP-mediated registration protocol, but gains persistent storage, access control, RAG, web search integration, and a production-style web UI.
How is this different from a Kubernetes operator (LLMKube)?
LLMKube is a Kubernetes-native operator for heterogeneous GPU fleets. It's the right choice if you already run Kubernetes. But if you're a homelab user with two or three machines, Kubernetes is a massive dependency — you're managing a control plane, etcd, CNI, and all the operational overhead of a container orchestrator just to run inference.
cAIc is a single FastAPI process with a SQLite database and an AMQP connection. No Kubernetes, no Docker required (though Docker compose is planned for v1.0). You can install it on bare Debian in five minutes.
Does cAIc support Apple Silicon?
Not yet, but there's nothing preventing it. llama.cpp has a Metal backend that exposes the same OpenAI-compatible API that cAIc already uses. A worker on an M-series Mac would look identical to the coordinator: it registers via AMQP, responds to pings, and serves inference requests. The changes needed are just platform-detection branches in gpu.py, hardware.py, and node_agent/agent.py — no architectural changes. This is on the backlog as B7.
Can I run the coordinator and a worker on the same machine?
Yes. Nothing prevents colocation. The coordinator uses minimal resources (it does no model inference), so there's no meaningful conflict. Just run llama-server alongside uvicorn.
Do I need RabbitMQ for a single-machine setup?
No. cAIc runs fine without AMQP — the cluster features simply won't activate. All core features (chat, RAG, memory, search, upload) work with just llama-server and optionally Qdrant/SearXNG. RabbitMQ is only needed when you add a second machine as a worker.
What models work with cAIc?
Any GGUF model that llama-server can load. The default is qwen2.5-7b-instruct, but you can swap to any model by changing DEFAULT_MODEL in config.py and pointing llama-server at a different GGUF file. Models with "coder" or "qwen" in the name are preferred for code queries; "mistral" or "llama" are preferred for general chat.
Can I expose cAIc to the internet?
Technically yes, but it's not the target deployment. cAIc is designed for local and trusted-LAN operation. If you must expose it, put a reverse proxy (Caddy, nginx) in front with HTTPS, set strong admin credentials, restrict the IP allowlist, and enable CAIC_TRUST_X_FORWARDED_FOR.
What happens if a worker goes offline?
The coordinator detects failure via ping timeout (5s) and auto-deregisters the node. New queries are routed to remaining workers. If no workers are available, the coordinator returns an error. Workers can rejoin at any time by re-registering — no manual intervention needed.
Does cAIc support OpenAI-compatible API?
Yes. /v1/chat/completions is available for IDE integration (Continue.dev, etc.), authenticated via CAIC_COMPLETIONS_API_KEY. Fill-in-the-middle (FIM) requests are proxied directly without persistence. Chat-style requests are persisted to conversation history.
How many tests does cAIc have?
179 tests across 23 test files, all passing. Every router has a dedicated test file with mocked external services. No live dependencies needed — tests use tmp_path fixtures and monkeypatched httpx.AsyncClient.