diff --git a/Current-WiP.md b/Current-WiP.md index 2afecdf..4084e3c 100755 --- a/Current-WiP.md +++ b/Current-WiP.md @@ -1,6 +1,6 @@ # cAIc Current WiP Backlog -Last updated: 2026-07-06 +Last updated: 2026-07-13 Owner: Gramps Scope: Active roadmap items and backlog. @@ -14,6 +14,7 @@ All 15 tasks are done. Final commit: `f0689ac feat: Roadmap N — AMQP cluster n - B2 — Bang-prefixed (`!`) search routing - B3 — Docker distribution (planning doc at `docker.md`) - **B4 — RAG Corpus Management UI (NEXT)** — browse, search, edit, delete individual RAG entries. Backend endpoints at `routers/rag_admin.py`, frontend panel in `templates/index.html` +- B5 — Default model auto-pull on first start — download the default GGUF model from HuggingFace (or configured mirror) if `models/` is empty when `llama-server` first starts. Configurable via `CAIC_DEFAULT_MODEL_URL` or similar. Needs a startup probe that checks model availability and a download-with-progress endpoint. - HTTPS / reverse proxy (Caddy) - Conversation search/filter and export tooling - Keyboard shortcuts, retry button, source-link polish diff --git a/Installation.md b/Installation.md index d6079ca..9a27c29 100644 --- a/Installation.md +++ b/Installation.md @@ -68,6 +68,8 @@ pip install psutil # for hardware stats ### 3. Set up llama-server +The default model is **`qwen2.5-7b-instruct`** (7B parameters, instruction-tuned). You can use any GGUF model — the name just needs to match what `select_node()` looks for (models with "coder" or "qwen" in the name route to code queries; "mistral" or "llama" route to general queries). + ```bash # Download a llama-server binary wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server @@ -76,7 +78,8 @@ chmod +x llama-server # Place a GGUF model file mkdir -p models # Download from HuggingFace, e.g.: -# wget -O models/qwen2.5-7b-instruct-q5_k_m.gguf +# wget -O models/qwen2.5-7b-instruct-q5_k_m.gguf \ +# https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF/resolve/main/qwen2.5-7b-instruct-q5_k_m.gguf # Run llama-server ./llama-server \ @@ -90,6 +93,8 @@ mkdir -p models Verify it's running: `curl http://localhost:8081/health` +> **Note:** Auto-download of the default model on first start is planned but not yet implemented. You must place a GGUF file in the `models/` directory manually for now. + ### 4. Configure cAIc Copy and edit configuration via environment variables: @@ -249,13 +254,17 @@ Tunable in `config.py` or via environment overrides: ### Model Configuration -`DEFAULT_MODEL` in `config.py` sets the model name used for inference. When llama-server loads a model, its visible name in the model list determines how `select_node()` matches it. The triage system maps queries to ideal model families: +`DEFAULT_MODEL` in `config.py` (default: `"qwen2.5-7b-instruct"`) sets the model name used for inference. This string must match the model name that llama-server reports in its model list — it's used for routing, not for downloading. + +When llama-server loads a model, its visible name determines how `select_node()` matches it. The triage system maps queries to ideal model families: - `code` → models with "coder" or "qwen" in the name - `general` → models with "mistral" or "llama" in the name For cluster mode, each worker advertises its loaded model. The coordinator selects the best-matching worker for each query. +**Auto-download on first start:** Not yet implemented (see backlog). For now, download a GGUF manually and place it in `models/`. + ## Verifying the Installation ```bash