docs: mention default model qwen2.5-7b-instruct, add B5 (auto-pull) to backlog
+2
-1
@@ -1,6 +1,6 @@
|
|||||||
# cAIc Current WiP Backlog
|
# cAIc Current WiP Backlog
|
||||||
|
|
||||||
Last updated: 2026-07-06
|
Last updated: 2026-07-13
|
||||||
Owner: Gramps
|
Owner: Gramps
|
||||||
Scope: Active roadmap items and backlog.
|
Scope: Active roadmap items and backlog.
|
||||||
|
|
||||||
@@ -14,6 +14,7 @@ All 15 tasks are done. Final commit: `f0689ac feat: Roadmap N — AMQP cluster n
|
|||||||
- B2 — Bang-prefixed (`!`) search routing
|
- B2 — Bang-prefixed (`!`) search routing
|
||||||
- B3 — Docker distribution (planning doc at `docker.md`)
|
- B3 — Docker distribution (planning doc at `docker.md`)
|
||||||
- **B4 — RAG Corpus Management UI (NEXT)** — browse, search, edit, delete individual RAG entries. Backend endpoints at `routers/rag_admin.py`, frontend panel in `templates/index.html`
|
- **B4 — RAG Corpus Management UI (NEXT)** — browse, search, edit, delete individual RAG entries. Backend endpoints at `routers/rag_admin.py`, frontend panel in `templates/index.html`
|
||||||
|
- B5 — Default model auto-pull on first start — download the default GGUF model from HuggingFace (or configured mirror) if `models/` is empty when `llama-server` first starts. Configurable via `CAIC_DEFAULT_MODEL_URL` or similar. Needs a startup probe that checks model availability and a download-with-progress endpoint.
|
||||||
- HTTPS / reverse proxy (Caddy)
|
- HTTPS / reverse proxy (Caddy)
|
||||||
- Conversation search/filter and export tooling
|
- Conversation search/filter and export tooling
|
||||||
- Keyboard shortcuts, retry button, source-link polish
|
- Keyboard shortcuts, retry button, source-link polish
|
||||||
|
|||||||
+11
-2
@@ -68,6 +68,8 @@ pip install psutil # for hardware stats
|
|||||||
|
|
||||||
### 3. Set up llama-server
|
### 3. Set up llama-server
|
||||||
|
|
||||||
|
The default model is **`qwen2.5-7b-instruct`** (7B parameters, instruction-tuned). You can use any GGUF model — the name just needs to match what `select_node()` looks for (models with "coder" or "qwen" in the name route to code queries; "mistral" or "llama" route to general queries).
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Download a llama-server binary
|
# Download a llama-server binary
|
||||||
wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server
|
wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server
|
||||||
@@ -76,7 +78,8 @@ chmod +x llama-server
|
|||||||
# Place a GGUF model file
|
# Place a GGUF model file
|
||||||
mkdir -p models
|
mkdir -p models
|
||||||
# Download from HuggingFace, e.g.:
|
# Download from HuggingFace, e.g.:
|
||||||
# wget -O models/qwen2.5-7b-instruct-q5_k_m.gguf <url>
|
# wget -O models/qwen2.5-7b-instruct-q5_k_m.gguf \
|
||||||
|
# https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF/resolve/main/qwen2.5-7b-instruct-q5_k_m.gguf
|
||||||
|
|
||||||
# Run llama-server
|
# Run llama-server
|
||||||
./llama-server \
|
./llama-server \
|
||||||
@@ -90,6 +93,8 @@ mkdir -p models
|
|||||||
|
|
||||||
Verify it's running: `curl http://localhost:8081/health`
|
Verify it's running: `curl http://localhost:8081/health`
|
||||||
|
|
||||||
|
> **Note:** Auto-download of the default model on first start is planned but not yet implemented. You must place a GGUF file in the `models/` directory manually for now.
|
||||||
|
|
||||||
### 4. Configure cAIc
|
### 4. Configure cAIc
|
||||||
|
|
||||||
Copy and edit configuration via environment variables:
|
Copy and edit configuration via environment variables:
|
||||||
@@ -249,13 +254,17 @@ Tunable in `config.py` or via environment overrides:
|
|||||||
|
|
||||||
### Model Configuration
|
### Model Configuration
|
||||||
|
|
||||||
`DEFAULT_MODEL` in `config.py` sets the model name used for inference. When llama-server loads a model, its visible name in the model list determines how `select_node()` matches it. The triage system maps queries to ideal model families:
|
`DEFAULT_MODEL` in `config.py` (default: `"qwen2.5-7b-instruct"`) sets the model name used for inference. This string must match the model name that llama-server reports in its model list — it's used for routing, not for downloading.
|
||||||
|
|
||||||
|
When llama-server loads a model, its visible name determines how `select_node()` matches it. The triage system maps queries to ideal model families:
|
||||||
|
|
||||||
- `code` → models with "coder" or "qwen" in the name
|
- `code` → models with "coder" or "qwen" in the name
|
||||||
- `general` → models with "mistral" or "llama" in the name
|
- `general` → models with "mistral" or "llama" in the name
|
||||||
|
|
||||||
For cluster mode, each worker advertises its loaded model. The coordinator selects the best-matching worker for each query.
|
For cluster mode, each worker advertises its loaded model. The coordinator selects the best-matching worker for each query.
|
||||||
|
|
||||||
|
**Auto-download on first start:** Not yet implemented (see backlog). For now, download a GGUF manually and place it in `models/`.
|
||||||
|
|
||||||
## Verifying the Installation
|
## Verifying the Installation
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
Reference in New Issue
Block a user