diff --git a/AGENTS.md b/AGENTS.md index dedb9cb..60c6b2e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,7 +40,7 @@ Every router has a dedicated test file: | `test_error_envelopes.py` | Global exception handler + stream error incidents | | `test_upload.py` | `routers/upload.py` — upload, delete, link, by-conversation, attachment_count integration | -Modules that call `httpx.AsyncClient` (chat, completions, models, search_route, upload, ingest) +Modules that call `httpx.AsyncClient` (chat, completions, models, search_route, upload, ingest, model_pull) are mocked via `monkeypatch.setattr` on `AsyncClient.stream`, `.get`, or `.post`. CPU stats in `models.py` (`api/stats`) use real `psutil`; GPU stats are monkeypatched via `routers.models.get_gpu_stats`. @@ -61,6 +61,7 @@ Refactored from single-file (`app.py`) into modules under project root: | `rag.py` | Qdrant vector search + system prompt assembly + chunk_text() helper | | `eviction.py` | Score-based RAG eviction engine | | `gpu.py` | GPU stats — `rocm-smi` (AMD/Linux) or `system_profiler` (Apple Silicon/macOS) | +| `model_pull.py` | Startup model availability check + Ollama pull API | | `triage.py` | Phi-4-mini-based query classification + cluster node selection | | `cluster.py` | Cluster node registry, event log, coordinator election, ping/pong, model swap handlers | | `amqp.py` | AMQP connection manager — connect, disconnect, publish, subscribe, auto-reconnect | @@ -113,7 +114,7 @@ The upstream request includes `"logprobs": true`. `parse_llama_stream_chunk()` e | wttr.in | No | weather shortcut | | rocm-smi | No | AMD GPU stats | | Qdrant | No | 6333 (coordinator) — RAG vector search | -| Ollama (worker) | No | 11434 — embeddings only | +| Ollama (worker) | No | 11434 — embeddings + model pull | ### Config quirks @@ -136,6 +137,7 @@ All streaming endpoints yield `data: {json}\n\n`. Key shapes: ### Completed this session - **B7 (v0.19.0)** — Apple Silicon worker support. `gpu.py` now detects `sys.platform == "darwin"` and parses `system_profiler SPDisplaysDataType` for GPU model/VRAM instead of `rocm-smi`. `hardware.py` has darwin branch via `_get_vram_darwin()`. `node_agent/agent.py` reports VRAM on macOS via `system_profiler`. 5 new tests cover linux/darwin gpu paths, 3 new hardware tests cover darwin assessment + VRAM parsing. +- **B5 (v0.19.1)** — Default model auto-pull on first start. `model_pull.py` with `ensure_model()` checks llama-server availability, falls back to Ollama pull API. Integrated into `app.py` lifespan. 11 tests cover all paths. - **Scroll-position fighting** — `scrollToTop()` now respects `_userScrolledAway` flag (100px threshold), skips auto-scroll when user is reading older content. `resetScrollLock()` called on new messages. - **401 error cascade** — `SESSION_TIMEOUT_SECONDS` bumped 90→3600 (1 hour). All 10 unprotected `authFetch` calls wrapped in try/catch. - **Token counter** — removed localStorage persistence; resets to 0 on page refresh. @@ -155,14 +157,13 @@ All streaming endpoints yield `data: {json}\n\n`. Key shapes: ### Upcoming (backlog) - B4 — RAG Corpus Management UI -- B5 — default model auto-pull on first start - B6 — waterfall direction toggle - B8 — **Encryption & PHI readiness** — spec out encryption at rest (SQLCipher for caic.db, Qdrant payload encryption) and in-transit (TLS for inference, AMQP, RAG). Per-user auth, audit logging, log sanitizer, data lifecycle. Document the "personal LAN HIPAA gap." - **Preferred approach: Private Chat mode** — a toggle that skips DB persistence, memory/RAG injection, content logging, and **external SearXNG searching** entirely. Zero stored data, zero external queries = zero data to protect. Simpler, more robust, less code to audit than full encryption. Design this as the primary PHI path before reaching for crypto. - **In-transit still needs TLS** — Private Chat eliminates at-rest risk, but AMQP (RabbitMQ) and inference (llama-server) traffic between coordinator and workers is still plaintext on the wire. TLS termination on each node is the lightweight fix (self-signed CA, nginx sidecar or RabbitMQ TLS). ### Key config values (current) -- `VERSION = "v0.19.0"` in `config.py` +- `VERSION = "v0.19.1"` in `config.py` - `SESSION_TIMEOUT_SECONDS = 3600` - `DEFAULT_MODEL = "qwen2.5-7b-instruct"` - `LLAMA_SERVER_BASE = "http://192.168.50.108:8081"` diff --git a/README.md b/README.md index bef0caa..7e3af1a 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ ![cAIc banner](static/readme-banner.png) -# cAIc v0.18.0 +# cAIc v0.19.1 Consumer AI hardware is a wasteland of incompatibility. NVIDIA speaks CUDA, AMD speaks ROCm. Your RTX 5070 Ti lives in one machine with 16 GB VRAM; your RX 6600 XT lives in another with 12 GB. Alone, neither can run a 14B model at usable speed. Together, they could — if the software stack didn't treat heterogeneous hardware as a bug instead of a feature. @@ -42,13 +42,19 @@ At v1.0, this ships with a Docker compose stack and setup wizard that detect CPU Developer wiki: [Home](https://llgit.llamachile.tube/gramps/cAIc/wiki/Home) — includes [FAQ](https://llgit.llamachile.tube/gramps/cAIc/wiki/FAQ), [Installation Guide](https://llgit.llamachile.tube/gramps/cAIc/wiki/Installation), and [full architecture docs](https://llgit.llamachile.tube/gramps/cAIc/wiki/Developer-Architecture) +## What's New in v0.19.1 + +### Default Model Auto-Pull on First Start (B5) +- **`model_pull.py`** — new module that checks if `default_model` is available on llama-server at startup, falls back to Ollama pull API if not found +- **Startup integration** — `app.py` lifespan calls `ensure_model()` after `assess_hardware()`, pulling the missing model via Ollama's streaming pull API +- **Idempotent** — skips pull if model already available on llama-server or Ollama + ## What's New in v0.18.0 ### Wiki — Installation Guide, Screenshots Gallery, Full Documentation - **New Installation & Configuration page** — bare-metal walkthrough, cluster setup, config reference, security checklist, 12 troubleshooting topics. Everything a new user needs to get cAIc running. - **Screenshots gallery** — clickable image gallery on the wiki Screenshots page - **Wiki fully populated** — 5 pages linked from Home, renders at root URL -- **B5 added to backlog** — auto-download of default GGUF model on first start ### UX Polish — Waterfall Layout, Barcode Stripes, Confidence Badges - **Waterfall display** — newest messages at top via `prepend()`, scroll to top diff --git a/app.py b/app.py index 1baa3cd..e57ad32 100644 --- a/app.py +++ b/app.py @@ -16,8 +16,8 @@ from fastapi.templating import Jinja2Templates from amqp import connect as amqp_connect, disconnect as amqp_disconnect from cluster import start_cluster_subscriptions -from config import VERSION, RATE_WINDOW_SECONDS, UPLOAD_DIR, RAG_MAX_VECTORS, RAG_EVICTION_HIGH_WATER, RAG_EVICTION_LOW_WATER, RAG_EVICTION_BATCH -from db import init_db +from config import VERSION, DEFAULT_MODEL, RATE_WINDOW_SECONDS, UPLOAD_DIR, RAG_MAX_VECTORS, RAG_EVICTION_HIGH_WATER, RAG_EVICTION_LOW_WATER, RAG_EVICTION_BATCH +from db import init_db, get_db, get_setting from hardware import assess_hardware from memory import get_memory_count from security import ( @@ -60,6 +60,11 @@ async def lifespan(app: FastAPI): init_db() log.info(f"Memory system: {get_memory_count()} memories loaded") await assess_hardware() + from model_pull import ensure_model + _db = get_db() + _user_model = get_setting(_db, "default_model", DEFAULT_MODEL) + _db.close() + await ensure_model(_user_model) await amqp_connect() await start_cluster_subscriptions() diff --git a/config.py b/config.py index 5d116b1..487d114 100644 --- a/config.py +++ b/config.py @@ -9,7 +9,7 @@ import logging log = logging.getLogger("caic") -VERSION = "v0.19.0" +VERSION = "v0.19.1" OLLAMA_BASE = os.environ.get("OLLAMA_BASE", "http://localhost:11434") LLAMA_SERVER_BASE = os.environ.get("LLAMA_SERVER_BASE", "http://192.168.50.108:8081") SEARXNG_BASE = "http://localhost:8888" diff --git a/model_pull.py b/model_pull.py new file mode 100644 index 0000000..3b45530 --- /dev/null +++ b/model_pull.py @@ -0,0 +1,80 @@ +""" +cAIc — Model pull/download helper. + +Uses Ollama's pull API to download models that aren't available on the +inference server. Runs synchronously during startup. +""" +import asyncio +import json +import logging + +import httpx + +from config import DEFAULT_MODEL, LLAMA_SERVER_BASE, OLLAMA_BASE + +log = logging.getLogger("caic") + + +async def _model_available_on_llama(model: str) -> bool: + try: + async with httpx.AsyncClient(timeout=5) as client: + resp = await client.get(f"{LLAMA_SERVER_BASE}/v1/models") + if resp.status_code == 200: + models = resp.json().get("data", []) + return any(m.get("id") == model for m in models) + except Exception: + pass + return False + + +async def _model_available_on_ollama(model: str) -> bool: + try: + async with httpx.AsyncClient(timeout=5) as client: + resp = await client.post(f"{OLLAMA_BASE}/api/show", json={"name": model}) + return resp.status_code == 200 + except Exception: + pass + return False + + +async def _pull_via_ollama(model: str) -> bool: + try: + async with httpx.AsyncClient(timeout=300) as client: + async with client.stream("POST", f"{OLLAMA_BASE}/api/pull", json={"name": model}) as resp: + if resp.status_code != 200: + log.warning("ollama pull returned %s for %s", resp.status_code, model) + return False + async for line in resp.aiter_lines(): + if line.strip(): + try: + data = json.loads(line) + status = data.get("status", "") + if status: + log.info("ollama pull %s: %s", model, status) + except json.JSONDecodeError: + pass + return True + except httpx.ConnectError: + log.warning("ollama not reachable at %s — cannot pull %s", OLLAMA_BASE, model) + except Exception as e: + log.warning("ollama pull error for %s: %s", model, e) + return False + + +async def ensure_model(model: str = "") -> bool: + """Ensure *model* is available for inference. Pull via Ollama if needed.""" + model = model or DEFAULT_MODEL + if await _model_available_on_llama(model): + log.info("model %s already available on llama-server", model) + return True + log.info("model %s not found on llama-server, checking Ollama", model) + if await _model_available_on_ollama(model): + log.info("model %s found on Ollama (available for embeddings)", model) + return True + log.info("model %s not found on Ollama either — pulling", model) + ok = await _pull_via_ollama(model) + if ok: + log.info("model %s pulled successfully", model) + else: + log.warning("model %s could not be pulled", model) + return ok diff --git a/tests/test_model_pull.py b/tests/test_model_pull.py new file mode 100644 index 0000000..3a80f55 --- /dev/null +++ b/tests/test_model_pull.py @@ -0,0 +1,142 @@ +import asyncio + +import httpx + +from model_pull import ensure_model, _model_available_on_llama, _model_available_on_ollama, _pull_via_ollama + + +class _MockAsyncResponse: + def __init__(self, status_code=200, json_data=None): + self.status_code = status_code + self._json_data = json_data or {} + + def json(self): + return self._json_data + + +class _MockStreamResponse: + def __init__(self, status_code=200, lines=None): + self.status_code = status_code + self._lines = lines or [] + + async def __aenter__(self): + return self + + async def __aexit__(self, *args): + pass + + def aiter_lines(self): + class _AIter: + def __init__(self, lines): + self._lines = iter(lines) + def __aiter__(self): + return self + async def __anext__(self): + try: + return next(self._lines) + except StopIteration: + raise StopAsyncIteration + return _AIter(self._lines) + + +def _mock_models_on_llama(*models): + async def _get(*args, **kwargs): + return _MockAsyncResponse(json_data={ + "data": [{"id": m} for m in models] + }) + return _get + + +async def _mock_ollama_available(*args, **kwargs): + return _MockAsyncResponse(status_code=200, json_data={"name": "qwen2.5:latest"}) + + +async def _mock_ollama_unavailable(*args, **kwargs): + raise httpx.ConnectError("refused") + + +def _mock_ollama_pull_ok(*args, **kwargs): + return _MockStreamResponse(200, [ + '{"status": "pulling manifest"}', + '{"status": "success"}', + ]) + + +def _mock_ollama_pull_fail(*args, **kwargs): + return _MockStreamResponse(500, []) + + +def _mock_ollama_connect_error(*args, **kwargs): + raise httpx.ConnectError("refused") + + +def test_available_on_llama(monkeypatch): + monkeypatch.setattr(httpx.AsyncClient, "get", _mock_models_on_llama("qwen2.5-7b-instruct")) + assert asyncio.run(_model_available_on_llama("qwen2.5-7b-instruct")) is True + assert asyncio.run(_model_available_on_llama("nonexistent-model")) is False + + +def test_available_on_llama_unreachable(monkeypatch): + async def _connect_error(*args, **kwargs): + raise httpx.ConnectError("refused") + monkeypatch.setattr(httpx.AsyncClient, "get", _connect_error) + assert asyncio.run(_model_available_on_llama("qwen2.5-7b-instruct")) is False + + +def test_available_on_ollama(monkeypatch): + monkeypatch.setattr(httpx.AsyncClient, "post", _mock_ollama_available) + assert asyncio.run(_model_available_on_ollama("qwen2.5:latest")) is True + + +def test_available_on_ollama_unreachable(monkeypatch): + monkeypatch.setattr(httpx.AsyncClient, "post", _mock_ollama_unavailable) + assert asyncio.run(_model_available_on_ollama("qwen2.5:latest")) is False + + +def test_pull_via_ollama_success(monkeypatch): + monkeypatch.setattr(httpx.AsyncClient, "stream", _mock_ollama_pull_ok) + assert asyncio.run(_pull_via_ollama("qwen2.5:latest")) is True + + +def test_pull_via_ollama_fail(monkeypatch): + monkeypatch.setattr(httpx.AsyncClient, "stream", _mock_ollama_pull_fail) + assert asyncio.run(_pull_via_ollama("qwen2.5:latest")) is False + + +def test_pull_via_ollama_connect_error(monkeypatch): + monkeypatch.setattr(httpx.AsyncClient, "stream", _mock_ollama_connect_error) + assert asyncio.run(_pull_via_ollama("qwen2.5:latest")) is False + + +def test_ensure_model_already_on_llama(monkeypatch): + monkeypatch.setattr(httpx.AsyncClient, "get", _mock_models_on_llama("qwen2.5-7b-instruct")) + assert asyncio.run(ensure_model("qwen2.5-7b-instruct")) is True + + +def test_ensure_model_not_on_llama_but_on_ollama(monkeypatch): + async def _get(*args, **kwargs): + return _MockAsyncResponse(json_data={"data": []}) + + monkeypatch.setattr(httpx.AsyncClient, "get", _get) + monkeypatch.setattr(httpx.AsyncClient, "post", _mock_ollama_available) + assert asyncio.run(ensure_model("qwen2.5-7b-instruct")) is True + + +def test_ensure_model_needs_pull(monkeypatch): + async def _get(*args, **kwargs): + return _MockAsyncResponse(json_data={"data": []}) + + monkeypatch.setattr(httpx.AsyncClient, "get", _get) + monkeypatch.setattr(httpx.AsyncClient, "post", _mock_ollama_unavailable) + monkeypatch.setattr(httpx.AsyncClient, "stream", _mock_ollama_pull_ok) + assert asyncio.run(ensure_model("qwen2.5-7b-instruct")) is True + + +def test_ensure_model_pull_fails(monkeypatch): + async def _get(*args, **kwargs): + return _MockAsyncResponse(json_data={"data": []}) + + monkeypatch.setattr(httpx.AsyncClient, "get", _get) + monkeypatch.setattr(httpx.AsyncClient, "post", _mock_ollama_unavailable) + monkeypatch.setattr(httpx.AsyncClient, "stream", _mock_ollama_pull_fail) + assert asyncio.run(ensure_model("qwen2.5-7b-instruct")) is False