From df405a156e0cfc1c11b6e571afafe65c61bd3f97 Mon Sep 17 00:00:00 2001
From: gramps <gramps@llgit.llamachile.tube>
Date: Tue, 4 Aug 2026 08:31:47 -0700
Subject: [PATCH] Add TASK 2 (context-aware routing) and TASK 3 (RAM-based
 context store)

---
 TASKS.md | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 74 insertions(+)

diff --git a/TASKS.md b/TASKS.md
index bfcb61a..f9ce3e5 100644
--- a/TASKS.md
+++ b/TASKS.md
@@ -40,3 +40,77 @@ User prompt → cAIc coordinator → AMQP/HTTP → corsair (ComfyUI/API) → PNG
 ### Status: ✅ Backend Complete (ComfyUI install pending on corsair)
 
 ---
+
+## TASK 2 — Context-Aware Cluster Routing
+
+**Goal:** Make chat/completions requests route to the best node based on available context capacity, not just model name matching. Solve the core problem: GPU nodes have limited VRAM context, but requests (especially from IDE integrations) can exceed that.
+
+### Requirements:
+
+1. **Node capacity reporting** — node_agent calculates and reports `effective_context_tokens` on registration based on (available RAM/VRAM − model weight size) → KV cache capacity. Updated on model swaps.
+2. **Coordinator tracks cluster capacity** — `CLUSTER_NODES` stores `effective_context_tokens` per node, updated via registration and `model_ready` events.
+3. **Context budget estimation** — before sending upstream, estimate the token count of the assembled message array (~4 chars/token heuristic, or tiktoken if available). Expose as a helper in `config.py` or `rag.py`.
+4. **Resource-aware `select_node()`** — `triage.py` weighs effective context capacity, current load, and model match when picking a node. Nodes that can't fit the estimated context are deprioritized or skipped.
+5. **Wire triage into chat router** — `routers/chat.py` calls `get_inference_url(user_message)` instead of hardcoding `LLAMA_SERVER_BASE`.
+6. **Wire triage into completions router** — `routers/completions.py` uses the same routing logic for IDE/Continue.dev sessions.
+7. **Graceful degradation** — if no node can fit the estimated context, truncate intelligently (drop oldest RAG chunks, summarize history) before falling back to the coordinator.
+
+### Architecture:
+
+```
+User request → build messages → estimate tokens → triage.select_node(effective_context)
+  → node with enough headroom → stream response
+  → no node fits → truncate context → coordinator fallback
+```
+
+### Tests:
+
+- Mock CLUSTER_NODES with varying effective_context_tokens, verify select_node picks the right one
+- Mock message arrays of different sizes, verify token estimation
+- Verify chat router calls get_inference_url instead of hardcoding
+- Verify fallback truncation when no node fits
+- Verify model swap updates effective_context_tokens
+
+### Status: Not started
+
+---
+
+## TASK 3 — RAM-Based Context Store Node (blue-sky)
+
+**Goal:** Enable a RAM-heavy node (e.g. a workstation with 32GB+ RAM) to join the cAIc cluster as a dedicated context store — holding conversation histories, RAG results, uploaded documents, and memories in RAM for fast retrieval, without running inference.
+
+### Requirements:
+
+1. **New node type: `context_store`** — registers with a `context_store` capability, advertises available RAM and current usage
+2. **Lightweight context service** — HTTP API on the context store node exposing:
+   - `POST /context/{session_id}` — store conversation context
+   - `GET /context/{session_id}` — retrieve full context
+   - `PUT /context/{session_id}/chunks` — update RAG/document chunks
+   - `GET /context/{session_id}/relevant?q=...` — ranked context retrieval
+3. **Coordinator integration** — `build_system_prompt()` pulls from the context store node instead of (or in addition to) SQLite when one is available
+4. **Context store discovery** — node_agent supports `context_store` type in config, coordinator queries available RAM on registration
+5. **Failover** — if context store is unreachable, fall back to local SQLite/Qdrant as today
+
+### Architecture:
+
+```
+Coordinator startup → discover context_store nodes → query available RAM
+Build system prompt → pull relevant context from context_store node → assemble → send to inference node
+```
+
+### Considerations:
+
+- Context store node needs minimal resources — Python + aiohttp + Redis or in-memory dict
+- Latency: LAN round-trip (~1ms) is negligible compared to inference time
+- Persistence: optional — RAM-only is fine if the store can repopulate from SQLite on restart
+- Security: context store holds unencrypted data locally (encryption stays at the coordinator layer)
+
+### Tests:
+
+- Mock context store node registration, verify coordinator discovers it
+- Mock context store HTTP responses, verify build_system_prompt pulls from it
+- Verify failover to local SQLite when context store is unreachable
+
+### Status: Not started (blocked on available hardware — Dell Precision Tower 3420 dead, NUC running Home Assistant OS)
+
+---