docs: mention default model qwen2.5-7b-instruct, add B5 (auto-pull) to backlog

2026-07-13 10:46:54 -07:00
parent 1f162fe6eb
commit 8a885388a5
2 changed files with 13 additions and 3 deletions
+2 -1
@@ -1,6 +1,6 @@
# cAIc Current WiP Backlog # cAIc Current WiP Backlog
Last updated: 2026-07-06 Last updated: 2026-07-13
Owner: Gramps Owner: Gramps
Scope: Active roadmap items and backlog. Scope: Active roadmap items and backlog.
@@ -14,6 +14,7 @@ All 15 tasks are done. Final commit: `f0689ac feat: Roadmap N — AMQP cluster n
- B2 — Bang-prefixed (`!`) search routing - B2 — Bang-prefixed (`!`) search routing
- B3 — Docker distribution (planning doc at `docker.md`) - B3 — Docker distribution (planning doc at `docker.md`)
- **B4 — RAG Corpus Management UI (NEXT)** — browse, search, edit, delete individual RAG entries. Backend endpoints at `routers/rag_admin.py`, frontend panel in `templates/index.html` - **B4 — RAG Corpus Management UI (NEXT)** — browse, search, edit, delete individual RAG entries. Backend endpoints at `routers/rag_admin.py`, frontend panel in `templates/index.html`
- B5 — Default model auto-pull on first start — download the default GGUF model from HuggingFace (or configured mirror) if `models/` is empty when `llama-server` first starts. Configurable via `CAIC_DEFAULT_MODEL_URL` or similar. Needs a startup probe that checks model availability and a download-with-progress endpoint.
- HTTPS / reverse proxy (Caddy) - HTTPS / reverse proxy (Caddy)
- Conversation search/filter and export tooling - Conversation search/filter and export tooling
- Keyboard shortcuts, retry button, source-link polish - Keyboard shortcuts, retry button, source-link polish
+11 -2
@@ -68,6 +68,8 @@ pip install psutil # for hardware stats
### 3. Set up llama-server ### 3. Set up llama-server
The default model is **`qwen2.5-7b-instruct`** (7B parameters, instruction-tuned). You can use any GGUF model — the name just needs to match what `select_node()` looks for (models with "coder" or "qwen" in the name route to code queries; "mistral" or "llama" route to general queries).
```bash ```bash
# Download a llama-server binary # Download a llama-server binary
wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server
@@ -76,7 +78,8 @@ chmod +x llama-server
# Place a GGUF model file # Place a GGUF model file
mkdir -p models mkdir -p models
# Download from HuggingFace, e.g.: # Download from HuggingFace, e.g.:
# wget -O models/qwen2.5-7b-instruct-q5_k_m.gguf <url> # wget -O models/qwen2.5-7b-instruct-q5_k_m.gguf \
# https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF/resolve/main/qwen2.5-7b-instruct-q5_k_m.gguf
# Run llama-server # Run llama-server
./llama-server \ ./llama-server \
@@ -90,6 +93,8 @@ mkdir -p models
Verify it's running: `curl http://localhost:8081/health` Verify it's running: `curl http://localhost:8081/health`
> **Note:** Auto-download of the default model on first start is planned but not yet implemented. You must place a GGUF file in the `models/` directory manually for now.
### 4. Configure cAIc ### 4. Configure cAIc
Copy and edit configuration via environment variables: Copy and edit configuration via environment variables:
@@ -249,13 +254,17 @@ Tunable in `config.py` or via environment overrides:
### Model Configuration ### Model Configuration
`DEFAULT_MODEL` in `config.py` sets the model name used for inference. When llama-server loads a model, its visible name in the model list determines how `select_node()` matches it. The triage system maps queries to ideal model families: `DEFAULT_MODEL` in `config.py` (default: `"qwen2.5-7b-instruct"`) sets the model name used for inference. This string must match the model name that llama-server reports in its model list — it's used for routing, not for downloading.
When llama-server loads a model, its visible name determines how `select_node()` matches it. The triage system maps queries to ideal model families:
- `code` → models with "coder" or "qwen" in the name - `code` → models with "coder" or "qwen" in the name
- `general` → models with "mistral" or "llama" in the name - `general` → models with "mistral" or "llama" in the name
For cluster mode, each worker advertises its loaded model. The coordinator selects the best-matching worker for each query. For cluster mode, each worker advertises its loaded model. The coordinator selects the best-matching worker for each query.
**Auto-download on first start:** Not yet implemented (see backlog). For now, download a GGUF manually and place it in `models/`.
## Verifying the Installation ## Verifying the Installation
```bash ```bash