B4 RAG Corpus Management UI + AGENTS.md → ai.md rename

- Backend: GET/DELETE/PATCH /api/rag/point/{id}, GET /api/rag/points
  (paginated, semantic search, source filter, sort)
- Frontend: admin-only RAG modal with stats bar, search, paginated table,
  per-row edit (re-embed) and delete, bulk flush with double confirm
- 14 new tests, 214 total passing
- AGENTS.md → ai.md (tool-agnostic), CLAUDE.md now references ai.md
- Bump v0.21.0 → v0.22.0
This commit is contained in:
gramps
2026-07-14 14:57:57 -07:00
parent 990b7c860d
commit 61f5a88673
8 changed files with 728 additions and 86 deletions
+3 -74
View File
@@ -1,8 +1,8 @@
# CLAUDE.md # ai.md — Project Context
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. Detailed project context, work state, architecture, and configuration have moved to [`ai.md`](ai.md). This file is kept for backward compatibility — the canonical reference is `ai.md`.
## Running the App ## Quick start
```bash ```bash
# Development # Development
@@ -21,74 +21,3 @@ sudo systemctl restart caic
./venv/bin/pip install -r requirements.txt ./venv/bin/pip install -r requirements.txt
# Also requires: psutil jinja2 python-multipart pypdf (not in requirements.txt) # Also requires: psutil jinja2 python-multipart pypdf (not in requirements.txt)
``` ```
## Architecture
Modular FastAPI app — `app.py` wires routers, middleware, and lifespan. SQLite database auto-created at `caic.db` on first run. No build step, single `templates/index.html`.
### Request Flow: `/api/chat`
1. User message saved to DB → conversation created if new
2. `process_remember_command()` intercepts "remember that..." / "forget about..." first
3. Optional `upload_context_id` → fetches document text from `upload_context` table, injects `[ATTACHED DOCUMENT]` into system prompt
4. `build_system_prompt()` assembles: profile + FTS5 memory search + Qdrant RAG + preset + skills + uploaded doc
5. Streamed to llama-server (`/v1/chat/completions`, `stream: true`, `logprobs: true`) via SSE
6. **Auto web search trigger**: if perplexity > 15.0 OR response matches `REFUSAL_PATTERNS`, re-queries with SearXNG results
7. Final response saved to DB; SSE `done` event sent with perplexity + tokens/sec
### Request Flow: `/api/search` (explicit search)
Bypasses perplexity/refusal — queries SearXNG directly then asks llama-server to summarize results.
### Request Flow: `/api/upload`
Multipart file upload → PDF/text extraction + chunking → optional Qdrant upsert + SQLite context storage (1hr expiry). Supports `mode=(context|ingest|both)`. Images upload as storage only — model cannot process image content.
### Request Flow: `/api/ingest`
Bearer-token-authenticated terminal RAG hook. Accepts raw text, chunks via `chunk_text()`, embeds via Ollama `/api/embeddings`, upserts to Qdrant.
### Memory System
FTS5 virtual table (`memories`) in SQLite. `search_memories()` uses BM25 ranking. `process_remember_command()` intercepts "remember that..." / "forget about..." before the message reaches the model and returns a confirmation string.
### Key Constants (`config.py`)
- `LLAMA_SERVER_BASE``http://192.168.50.108:8081` (coordinator llama-server, RPC offloads to worker GPU)
- `SEARXNG_BASE``http://localhost:8888`
- `QDRANT_URL``http://192.168.50.108:6333` (Qdrant on coordinator)
- `TRIAGE_BASE``http://127.0.0.1:8083/v1` (Phi-4-mini)
- `AMQP_URL``amqp://caic:{pw}@localhost:5672/caic` (RabbitMQ, pw read from `~/.caic_amqp_secret`)
- `PERPLEXITY_THRESHOLD``15.0`
- `EMBED_URL``http://192.168.50.210:11434/api/embeddings` (Ollama on worker)
- `VERSION` — current version string
### External Services
| Service | Required | Port |
|---------|----------|------|
| **llama-server** (coordinator) | Yes | 8081 + RPC :50052 (worker GPU) |
| **Phi-4-mini** (triage) | No | 8083 |
| **SearXNG** | No | 8888 |
| **RabbitMQ** (coordinator) | No | 5672 — AMQP broker |
| **wttr.in** | No | weather shortcut |
| **rocm-smi** | No | AMD GPU stats |
| **Qdrant** (coordinator) | No | 6333 — RAG vector search |
| **Ollama** (worker) | No | 11434 — embeddings only |
### Database
`get_db()` opens a new connection per request (no pool). `init_db()` runs at startup via FastAPI `lifespan`. Tables: `conversations`, `messages`, `settings`, `profile` (singleton id=1), `memories` (FTS5), `upload_context`. Default settings seeded but never overwritten.
### SSE Protocol
All streaming endpoints yield `data: {json}\n\n`. Key event shapes:
- `{token, conversation_id}` — streaming token
- `{searching: true}` — web search triggered
- `{search_results: N}` — N results retrieved
- `{done: true, perplexity, tokens_per_sec, searched?}` — terminal event
- `{error: "...", error_key: "..."}` — error with incident key
### Deployment
Runs as systemd service under user `caic`, working directory `/opt/caic`. Logs via syslog (`journalctl -u caic`). Version bumps via git tag + commit, deployed via `git pull && systemctl restart caic`.
+11 -1
View File
@@ -1,6 +1,6 @@
![cAIc banner](static/readme-banner.png) ![cAIc banner](static/readme-banner.png)
# cAIc v0.21.0 # cAIc v0.22.0
Consumer AI hardware is a wasteland of incompatibility. NVIDIA speaks CUDA, AMD speaks ROCm. Your RTX 5070 Ti lives in one machine with 16 GB VRAM; your RX 6600 XT lives in another with 12 GB. Alone, neither can run a 14B model at usable speed. Together, they could — if the software stack didn't treat heterogeneous hardware as a bug instead of a feature. Consumer AI hardware is a wasteland of incompatibility. NVIDIA speaks CUDA, AMD speaks ROCm. Your RTX 5070 Ti lives in one machine with 16 GB VRAM; your RX 6600 XT lives in another with 12 GB. Alone, neither can run a 14B model at usable speed. Together, they could — if the software stack didn't treat heterogeneous hardware as a bug instead of a feature.
@@ -55,6 +55,16 @@ Untested: Windows 11 / WSL2 (Debian). The codebase is pure Python with no platfo
Under the hood: FastAPI + SQLite + Jinja2 on Python 3.13. AMQP-mediated cluster coordination with an OpenAI-compatible inference endpoint. Under the hood: FastAPI + SQLite + Jinja2 on Python 3.13. AMQP-mediated cluster coordination with an OpenAI-compatible inference endpoint.
## What's New in v0.22.0
### RAG Corpus Management UI (B4)
- New admin modal (RAG button in drawer header) to browse, search, edit, and delete individual RAG corpus entries.
- **Endpoints**: `GET /api/rag/points` (paginated list with semantic search and source filter), `GET /api/rag/point/{id}` (single point detail), `DELETE /api/rag/point/{id}` (single point deletion), `PATCH /api/rag/point/{id}` (edit text with re-embed).
- **Frontend**: Stats bar (vector count, % full, pinned, avg retrievals, at-risk, eviction rate), semantic search bar, source filter dropdown, paginated results table, per-row edit/delete with confirmation, double-confirm bulk flush.
- All endpoints admin-protected; text decrypted for display, re-encrypted on edit.
- 14 new tests (32 total in test_rag_management.py). 214 tests pass overall.
- Renamed `AGENTS.md``ai.md` for tool-agnostic project context. `CLAUDE.md` now points to `ai.md`.
## What's New in v0.21.0 ## What's New in v0.21.0
### Scrollbar + DOM Fixes ### Scrollbar + DOM Fixes
+3 -7
View File
@@ -26,6 +26,7 @@ Every router has a dedicated test file:
| `test_models_router.py` | `routers/models.py` — models list, ps, show, stats, search/status | | `test_models_router.py` | `routers/models.py` — models list, ps, show, stats, search/status |
| `test_presets.py` | `routers/presets.py` — full CRUD, default preset protection | | `test_presets.py` | `routers/presets.py` — full CRUD, default preset protection |
| `test_profile.py` | `routers/profile.py` — get, update, default, length validation | | `test_profile.py` | `routers/profile.py` — get, update, default, length validation |
| `test_rag_management.py` | `eviction.py` + `routers/rag_admin.py` — eviction engine, stats, flush, browse, search, edit, delete individual points |
| `test_search_route.py` | `routers/search_route.py` — explicit search flow, no results, errors | | `test_search_route.py` | `routers/search_route.py` — explicit search flow, no results, errors |
| `test_search_url_sanitization.py` | `search.py` URL sanitizer | | `test_search_url_sanitization.py` | `search.py` URL sanitizer |
| `test_cluster.py` | `cluster.py` — registration, deregistration, pong, events, coordinator query | | `test_cluster.py` | `cluster.py` — registration, deregistration, pong, events, coordinator query |
@@ -137,7 +138,7 @@ All streaming endpoints yield `data: {json}\n\n`. Key shapes:
## Work State ## Work State
### Completed this session ### Completed this session
- **v0.21.0 Release** — Scrollbar/DOM fixes, perplexity persistence, all service URLs env-overridable, single-node deployment docs, hardware.py Qdrant URL bugfix. See README for full changelog. - **B4 — RAG Corpus Management UI** — Paginated browse, semantic search, source filter, edit text with re-embed, single-point delete, bulk flush. Backend: `GET /api/rag/points`, `GET /api/rag/point/{id}`, `DELETE /api/rag/point/{id}`, `PATCH /api/rag/point/{id}`. Frontend: admin-only modal with stats bar, search bar, paginated table, per-row edit/delete, double-confirm flush.
### Active ### Active
- (none) - (none)
@@ -145,16 +146,11 @@ All streaming endpoints yield `data: {json}\n\n`. Key shapes:
### Blocked ### Blocked
- (none) - (none)
### Blocked
- (none)
### Upcoming (backlog) ### Upcoming (backlog)
- B3 — Docker distribution - B3 — Docker distribution
- B4 — RAG Corpus Management UI
- Private Chat mode was implemented in B8 (v0.19.3). WireGuard in-transit encryption documented in wiki. At-rest encryption implemented in v0.20.0.
### Key config values (current) ### Key config values (current)
- **Current VERSION**: `v0.21.0` in `config.py`. - **Current VERSION**: `v0.22.0` in `config.py`.
- `SESSION_TIMEOUT_SECONDS = 3600` - `SESSION_TIMEOUT_SECONDS = 3600`
- `DEFAULT_MODEL = "qwen2.5-7b-instruct"` - `DEFAULT_MODEL = "qwen2.5-7b-instruct"`
- `LLAMA_SERVER_BASE = "http://192.168.50.108:8081"` - `LLAMA_SERVER_BASE = "http://192.168.50.108:8081"`
+1 -1
View File
@@ -9,7 +9,7 @@ import logging
log = logging.getLogger("caic") log = logging.getLogger("caic")
VERSION = "v0.21.0" VERSION = "v0.22.0"
OLLAMA_BASE = os.environ.get("OLLAMA_BASE", "http://localhost:11434") OLLAMA_BASE = os.environ.get("OLLAMA_BASE", "http://localhost:11434")
LLAMA_SERVER_BASE = os.environ.get("LLAMA_SERVER_BASE", "http://192.168.50.108:8081") LLAMA_SERVER_BASE = os.environ.get("LLAMA_SERVER_BASE", "http://192.168.50.108:8081")
SEARXNG_BASE = os.environ.get("CAIC_SEARXNG_BASE", "http://localhost:8888") SEARXNG_BASE = os.environ.get("CAIC_SEARXNG_BASE", "http://localhost:8888")
+1 -1
View File
@@ -10,11 +10,11 @@ Scope: Active roadmap items and backlog.
- **WireGuard TLS (v0.19.4)** — Self-signed WireGuard mesh encrypts all inter-node traffic (AMQP, inference, RPC). No code changes to cAIc. Documented in wiki/WireGuard-Setup.md + docker.md §5.4. - **WireGuard TLS (v0.19.4)** — Self-signed WireGuard mesh encrypts all inter-node traffic (AMQP, inference, RPC). No code changes to cAIc. Documented in wiki/WireGuard-Setup.md + docker.md §5.4.
- **At-Rest Encryption (v0.20.0)** — AES-256-GCM encrypts all query-derived text at rest. crypto.py with auto-keygen, key stored as `heartbeat_interval_ms` in settings. All 12 storage paths wired (SQLite: messages, conversations, memories, upload_context; Qdrant: RAG chunks, ingest, upload). 200 tests pass. - **At-Rest Encryption (v0.20.0)** — AES-256-GCM encrypts all query-derived text at rest. crypto.py with auto-keygen, key stored as `heartbeat_interval_ms` in settings. All 12 storage paths wired (SQLite: messages, conversations, memories, upload_context; Qdrant: RAG chunks, ingest, upload). 200 tests pass.
- **v0.21.0** — Scrollbar/DOM fixes, perplexity persistence per message, all service URLs env-overridable, single-node deployment docs, DOM pairing bugfix, hardware.py Qdrant URL bugfix. - **v0.21.0** — Scrollbar/DOM fixes, perplexity persistence per message, all service URLs env-overridable, single-node deployment docs, DOM pairing bugfix, hardware.py Qdrant URL bugfix.
- **v0.22.0** — B4 RAG Corpus Management UI: paginated browse, semantic search, source filter, edit with re-embed, single-point delete, bulk flush. `routers/rag_admin.py` expanded with 4 new endpoints; `CLAUDE.md``ai.md` rename.
## Backlog ## Backlog
- B3 — Docker distribution (planning doc at `docker.md`, not yet implemented) - B3 — Docker distribution (planning doc at `docker.md`, not yet implemented)
- **B4 — RAG Corpus Management UI (deferred)** — browse, search, edit, delete individual RAG entries
- HTTPS / reverse proxy (Caddy) - HTTPS / reverse proxy (Caddy)
- Conversation search/filter and export tooling - Conversation search/filter and export tooling
- Keyboard shortcuts, retry button, source-link polish - Keyboard shortcuts, retry button, source-link polish
+193 -2
View File
@@ -1,17 +1,38 @@
"""JarvisChat routers — RAG corpus management admin endpoints.""" """JarvisChat routers — RAG corpus management admin endpoints."""
import logging import logging
from datetime import datetime, timezone
import httpx import httpx
from fastapi import APIRouter, HTTPException, Request from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import JSONResponse from fastapi.responses import JSONResponse
from crypto import decrypt_text, encrypt_text
from eviction import get_rag_operational_stats, EVICTION_LOG from eviction import get_rag_operational_stats, EVICTION_LOG
from rag import QDRANT_URL, RAG_COLLECTION from rag import QDRANT_URL, RAG_COLLECTION, EMBED_URL, EMBED_MODEL
log = logging.getLogger("caic") log = logging.getLogger("caic")
router = APIRouter() router = APIRouter()
def _normalise_date(payload: dict) -> str:
return payload.get("ingest_date") or payload.get("upload_date") or ""
def _decrypt_payload(payload: dict) -> dict:
text = payload.get("text", "")
return {
"text": decrypt_text(text),
"source": payload.get("source"),
"type": payload.get("type"),
"date": _normalise_date(payload),
"retrieval_count": payload.get("retrieval_count", 0),
"topic": payload.get("topic"),
"fact": payload.get("fact"),
"filename": payload.get("filename"),
"last_accessed": payload.get("last_accessed"),
}
@router.get("/api/rag/stats") @router.get("/api/rag/stats")
async def rag_stats(request: Request): async def rag_stats(request: Request):
if getattr(request.state, "session_role", "none") != "admin": if getattr(request.state, "session_role", "none") != "admin":
@@ -21,8 +42,178 @@ async def rag_stats(request: Request):
return stats return stats
@router.get("/api/rag/points")
async def rag_list_points(
request: Request,
limit: int = Query(20, ge=1, le=100),
offset: str = Query(None),
search: str = Query(None),
source: str = Query(None),
sort: str = Query("date"),
order: str = Query("desc"),
):
if getattr(request.state, "session_role", "none") != "admin":
raise HTTPException(status_code=403, detail="Admin PIN required for this action")
async with httpx.AsyncClient() as client:
if search:
embed_resp = await client.post(
f"{EMBED_URL}/api/embeddings",
json={"model": EMBED_MODEL, "prompt": search},
timeout=10.0,
)
if embed_resp.status_code != 200:
return JSONResponse(status_code=502, content={"detail": "Embedding failed"})
vector = embed_resp.json()["embedding"]
search_body = {"vector": vector, "limit": limit, "with_payload": True}
if source:
search_body["filter"] = {"must": [{"match": {"key": "source", "value": source}}]}
sr = await client.post(
f"{QDRANT_URL}/collections/{RAG_COLLECTION}/points/search",
json=search_body,
timeout=10.0,
)
if sr.status_code != 200:
return JSONResponse(status_code=502, content={"detail": "Qdrant search failed"})
results = sr.json().get("result", [])
points = []
for r in results:
p = _decrypt_payload(r.get("payload", {}))
p["id"] = r["id"]
p["score"] = r.get("score")
points.append(p)
return {"points": points, "total": len(points), "next_offset": None}
scroll_body = {
"limit": limit,
"with_payload": True,
"with_vector": False,
}
if offset:
scroll_body["offset"] = offset
if source:
scroll_body["filter"] = {"must": [{"match": {"key": "source", "value": source}}]}
if sort:
direction = "asc" if order == "asc" else "desc"
scroll_body["order_by"] = {"key": sort, "direction": direction}
sr = await client.post(
f"{QDRANT_URL}/collections/{RAG_COLLECTION}/points/scroll",
json=scroll_body,
timeout=10.0,
)
if sr.status_code != 200:
return JSONResponse(status_code=502, content={"detail": "Qdrant scroll failed"})
result = sr.json().get("result", {})
raw_points = result.get("points", [])
next_offset = result.get("next_page_offset")
info_resp = await client.get(
f"{QDRANT_URL}/collections/{RAG_COLLECTION}",
timeout=5.0,
)
total = 0
if info_resp.status_code == 200:
total = info_resp.json().get("result", {}).get("vectors_count", 0)
points = []
for r in raw_points:
p = _decrypt_payload(r.get("payload", {}))
p["id"] = r["id"]
points.append(p)
return {"points": points, "total": total, "next_offset": next_offset}
@router.get("/api/rag/point/{point_id}")
async def rag_get_point(point_id: str, request: Request):
if getattr(request.state, "session_role", "none") != "admin":
raise HTTPException(status_code=403, detail="Admin PIN required for this action")
async with httpx.AsyncClient() as client:
resp = await client.get(
f"{QDRANT_URL}/collections/{RAG_COLLECTION}/points/{point_id}",
timeout=10.0,
)
if resp.status_code != 200:
raise HTTPException(status_code=404, detail="Point not found")
result = resp.json().get("result", {})
if not result:
raise HTTPException(status_code=404, detail="Point not found")
payload = result.get("payload", {})
point = _decrypt_payload(payload)
point["id"] = result["id"]
return point
@router.delete("/api/rag/point/{point_id}")
async def rag_delete_point(point_id: str, request: Request):
if getattr(request.state, "session_role", "none") != "admin":
raise HTTPException(status_code=403, detail="Admin PIN required for this action")
async with httpx.AsyncClient() as client:
resp = await client.post(
f"{QDRANT_URL}/collections/{RAG_COLLECTION}/points/delete",
json={"points": [point_id]},
timeout=10.0,
)
if resp.status_code not in (200, 201) and resp.status_code != 404:
return JSONResponse(status_code=502, content={"detail": "Qdrant delete failed"})
log.info(f"RAG point {point_id} deleted by admin")
return {"status": "deleted", "id": point_id}
@router.patch("/api/rag/point/{point_id}")
async def rag_update_point(point_id: str, request: Request):
if getattr(request.state, "session_role", "none") != "admin":
raise HTTPException(status_code=403, detail="Admin PIN required for this action")
body = await request.json()
new_text = (body.get("text") or "").strip()
if not new_text:
raise HTTPException(status_code=400, detail="text required")
async with httpx.AsyncClient() as client:
get_resp = await client.get(
f"{QDRANT_URL}/collections/{RAG_COLLECTION}/points/{point_id}",
timeout=10.0,
)
if get_resp.status_code != 200:
raise HTTPException(status_code=404, detail="Point not found")
existing = get_resp.json().get("result", {})
if not existing:
raise HTTPException(status_code=404, detail="Point not found")
old_payload = existing.get("payload", {})
embed_resp = await client.post(
f"{EMBED_URL}/api/embeddings",
json={"model": EMBED_MODEL, "prompt": new_text},
timeout=10.0,
)
if embed_resp.status_code != 200:
return JSONResponse(status_code=502, content={"detail": "Embedding failed"})
vector = embed_resp.json()["embedding"]
new_payload = dict(old_payload)
new_payload["text"] = encrypt_text(new_text)
date_field = "ingest_date" if "ingest_date" in old_payload else "upload_date"
new_payload[date_field] = datetime.now(timezone.utc).isoformat()
upsert_resp = await client.put(
f"{QDRANT_URL}/collections/{RAG_COLLECTION}/points?wait=true",
json={"points": [{"id": point_id, "vector": vector, "payload": new_payload}]},
timeout=10.0,
)
if upsert_resp.status_code not in (200, 201):
return JSONResponse(status_code=502, content={"detail": "Qdrant upsert failed"})
log.info(f"RAG point {point_id} updated by admin")
return {"status": "updated", "id": point_id}
@router.post("/api/rag/flush") @router.post("/api/rag/flush")
async def rag_flush(request: Request): async def rag_flush(request: Request):
if getattr(request.state, "session_role", "none") != "admin":
raise HTTPException(status_code=403, detail="Admin PIN required for this action")
try: try:
async with httpx.AsyncClient() as client: async with httpx.AsyncClient() as client:
scroll_resp = await client.post( scroll_resp = await client.post(
+249
View File
@@ -129,6 +129,39 @@ body { font-family: var(--font-body); background: var(--bg-primary); color: var(
.modal-close { background:none; border:none; color:var(--text-muted); font-size:24px; cursor:pointer; } .modal-close { background:none; border:none; color:var(--text-muted); font-size:24px; cursor:pointer; }
.modal-close:hover { color:var(--text-primary); } .modal-close:hover { color:var(--text-primary); }
.modal-body { padding: 20px 24px; } .modal-body { padding: 20px 24px; }
.rag-stats-bar { display:flex; flex-wrap:wrap; gap:12px 24px; padding:10px 14px; background:var(--bg-tertiary); border-radius:var(--radius); margin-bottom:14px; font-size:11px; font-family:var(--font-mono); }
.rag-stat-item { display:flex; gap:4px; align-items:center; }
.rag-stat-label { color:var(--text-muted); text-transform:uppercase; }
.rag-stat-value { color:var(--text-primary); font-weight:600; }
.rag-toolbar { display:flex; align-items:center; justify-content:space-between; gap:10px; margin-bottom:12px; flex-wrap:wrap; }
.rag-toolbar-left { display:flex; gap:8px; align-items:center; flex:1; }
.rag-toolbar-right { display:flex; gap:8px; align-items:center; flex-shrink:0; }
.rag-search-input { flex:1; min-width:160px; background:var(--bg-tertiary); border:1px solid var(--border); color:var(--text-primary); font-family:var(--font-mono); font-size:12px; padding:8px 12px; border-radius:var(--radius); }
.rag-search-input:focus { outline:none; border-color:var(--accent-dim); }
.rag-source-select { background:var(--bg-tertiary); border:1px solid var(--border); color:var(--text-primary); font-family:var(--font-mono); font-size:12px; padding:7px 10px; border-radius:var(--radius); cursor:pointer; }
.rag-source-select:focus { outline:none; border-color:var(--accent-dim); }
.rag-table-wrap { overflow-x:auto; margin-bottom:12px; }
.rag-table { width:100%; border-collapse:collapse; font-size:11px; font-family:var(--font-mono); }
.rag-table th { text-align:left; padding:8px 6px; border-bottom:1px solid var(--border); color:var(--text-secondary); text-transform:uppercase; letter-spacing:0.5px; font-size:10px; white-space:nowrap; }
.rag-table td { padding:8px 6px; border-bottom:1px solid var(--border); color:var(--text-primary); vertical-align:top; }
.rag-table tr:hover td { background:var(--bg-hover); }
.rag-col-id { width:80px; }
.rag-col-text { min-width:200px; }
.rag-col-source { width:100px; }
.rag-col-type { width:80px; }
.rag-col-date { width:140px; }
.rag-col-retr { width:50px; text-align:center; }
.rag-col-actions { width:80px; text-align:center; white-space:nowrap; }
.rag-text-snippet { display:-webkit-box; -webkit-line-clamp:2; -webkit-box-orient:vertical; overflow:hidden; line-height:1.4; word-break:break-word; cursor:pointer; }
.rag-text-snippet.expanded { -webkit-line-clamp:unset; }
.rag-action-btn { background:none; border:1px solid var(--border); border-radius:var(--radius); cursor:pointer; font-size:11px; padding:3px 7px; color:var(--text-muted); transition:all 0.15s; }
.rag-action-btn:hover { border-color:var(--accent-dim); color:var(--accent); }
.rag-action-btn.danger:hover { border-color:var(--danger); color:var(--danger); }
.rag-pagination { display:flex; align-items:center; justify-content:center; gap:12px; padding:8px 0; }
.rag-page-info { font-size:11px; color:var(--text-muted); font-family:var(--font-mono); }
.rag-edit-modal { max-width:700px; }
.rag-edit-textarea { width:100%; min-height:200px; background:var(--bg-tertiary); border:1px solid var(--border); color:var(--text-primary); font-family:var(--font-mono); font-size:12px; padding:12px; border-radius:var(--radius); resize:vertical; line-height:1.6; }
.rag-edit-textarea:focus { outline:none; border-color:var(--accent-dim); }
.modal-section { margin-bottom: 24px; } .modal-section { margin-bottom: 24px; }
.modal-section h3 { font-family:var(--font-mono); font-size:13px; color:var(--text-secondary); text-transform:uppercase; letter-spacing:1px; margin-bottom:8px; } .modal-section h3 { font-family:var(--font-mono); font-size:13px; color:var(--text-secondary); text-transform:uppercase; letter-spacing:1px; margin-bottom:8px; }
.modal-section p.desc { font-size:12px; color:var(--text-muted); margin-bottom:10px; line-height:1.5; } .modal-section p.desc { font-size:12px; color:var(--text-muted); margin-bottom:10px; line-height:1.5; }
@@ -318,6 +351,7 @@ body { font-family: var(--font-body); background: var(--bg-primary); color: var(
<span class="drawer-title">cAIc <span class="version">{{ version }}</span></span> <span class="drawer-title">cAIc <span class="version">{{ version }}</span></span>
<div class="drawer-actions"> <div class="drawer-actions">
<button class="drawer-btn" onclick="newChat()" title="New chat">+</button> <button class="drawer-btn" onclick="newChat()" title="New chat">+</button>
<button class="drawer-btn" onclick="openRagAdmin()" title="RAG corpus management">R</button>
<button class="drawer-btn" onclick="openSettings()" title="Settings">&#9881;</button> <button class="drawer-btn" onclick="openSettings()" title="Settings">&#9881;</button>
<button class="drawer-btn danger" onclick="deleteAllConversations()" title="Delete all">&#128465;</button> <button class="drawer-btn danger" onclick="deleteAllConversations()" title="Delete all">&#128465;</button>
</div> </div>
@@ -332,6 +366,53 @@ body { font-family: var(--font-body); background: var(--bg-primary); color: var(
</div> </div>
</aside> </aside>
<div class="modal-overlay" id="ragAdminModal">
<div class="modal" style="max-width:960px;">
<div class="modal-header">
<h2>RAG Corpus Management</h2>
<button class="modal-close" onclick="closeRagAdmin()">&times;</button>
</div>
<div class="modal-body">
<div class="rag-stats-bar" id="ragStatsBar">Loading stats...</div>
<div class="rag-toolbar">
<div class="rag-toolbar-left">
<input type="text" id="ragSearchInput" class="rag-search-input" placeholder="Semantic search..." onkeydown="if(event.key==='Enter')loadRagPoints()">
<select id="ragSourceFilter" class="rag-source-select" onchange="loadRagPoints()">
<option value="">All sources</option>
</select>
</div>
<div class="rag-toolbar-right">
<button class="btn-small btn-save" onclick="loadRagPoints()" id="ragRefreshBtn">&#8635; Refresh</button>
<button class="btn-small btn-reset" onclick="flushRagCollection()" style="color:var(--danger);border-color:var(--danger);">Flush All</button>
</div>
</div>
<div class="rag-table-wrap">
<table class="rag-table" id="ragTable">
<thead>
<tr>
<th class="rag-col-id">ID</th>
<th class="rag-col-text">Text</th>
<th class="rag-col-source">Source</th>
<th class="rag-col-type">Type</th>
<th class="rag-col-date">Date</th>
<th class="rag-col-retr">Retr</th>
<th class="rag-col-actions">Actions</th>
</tr>
</thead>
<tbody id="ragTableBody">
<tr><td colspan="7" style="text-align:center;color:var(--text-muted);padding:40px;">No points loaded</td></tr>
</tbody>
</table>
</div>
<div class="rag-pagination" id="ragPagination" style="display:none;">
<button class="btn-small" id="ragPrevBtn" onclick="ragPage(-1)">&#9664; Prev</button>
<span class="rag-page-info" id="ragPageInfo">Page 1</span>
<button class="btn-small" id="ragNextBtn" onclick="ragPage(1)">Next &#9654;</button>
</div>
</div>
</div>
</div>
<div class="modal-overlay" id="settingsModal"> <div class="modal-overlay" id="settingsModal">
<div class="modal"> <div class="modal">
<div class="modal-header"> <div class="modal-header">
@@ -1128,6 +1209,174 @@ function openSettings() {
function closeSettings() { document.getElementById('settingsModal').classList.remove('visible'); } function closeSettings() { document.getElementById('settingsModal').classList.remove('visible'); }
document.getElementById('settingsModal').addEventListener('click', e => { if (e.target.id === 'settingsModal') closeSettings(); }); document.getElementById('settingsModal').addEventListener('click', e => { if (e.target.id === 'settingsModal') closeSettings(); });
// ---------- RAG Admin ----------
let _ragPageOffset = null;
let _ragPrevOffsets = [];
let _ragNextOffset = null;
let _ragTotal = 0;
function openRagAdmin() {
if (currentRole !== 'admin') { requireAdminNotice(); return; }
document.getElementById('ragAdminModal').classList.add('visible');
_ragPageOffset = null;
_ragPrevOffsets = [];
_ragNextOffset = null;
_ragTotal = 0;
loadRagStats();
loadRagPoints();
populateRagSourceFilter();
}
function closeRagAdmin() { document.getElementById('ragAdminModal').classList.remove('visible'); }
document.getElementById('ragAdminModal').addEventListener('click', e => { if (e.target.id === 'ragAdminModal') closeRagAdmin(); });
async function loadRagStats() {
try {
const resp = await authFetch('/api/rag/stats');
const data = await resp.json();
const bar = document.getElementById('ragStatsBar');
bar.innerHTML = `
<span class="rag-stat-item"><span class="rag-stat-label">Vectors</span> <span class="rag-stat-value">${data.vector_count}</span></span>
<span class="rag-stat-item"><span class="rag-stat-label">Max</span> <span class="rag-stat-value">${data.max_vectors}</span></span>
<span class="rag-stat-item"><span class="rag-stat-label">Full</span> <span class="rag-stat-value">${data.percent_full}%</span></span>
<span class="rag-stat-item"><span class="rag-stat-label">Pinned</span> <span class="rag-stat-value">${data.pinned_count}</span></span>
<span class="rag-stat-item"><span class="rag-stat-label">Avg Retr</span> <span class="rag-stat-value">${data.avg_retrieval_count}</span></span>
<span class="rag-stat-item"><span class="rag-stat-label">At Risk</span> <span class="rag-stat-value">${data.at_risk_count}</span></span>
<span class="rag-stat-item"><span class="rag-stat-label">Evictions (1m)</span> <span class="rag-stat-value">${data.eviction_counts_last_1m}</span></span>
`;
} catch(e) {
document.getElementById('ragStatsBar').textContent = 'Failed to load stats';
}
}
async function populateRagSourceFilter() {
const sel = document.getElementById('ragSourceFilter');
if (sel.options.length > 1) return;
try {
const resp = await authFetch('/api/rag/points?limit=100');
const data = await resp.json();
const sources = new Set();
(data.points || []).forEach(p => { if (p.source) sources.add(p.source); });
const sorted = Array.from(sources).sort();
sorted.forEach(s => {
const opt = document.createElement('option');
opt.value = s;
opt.textContent = s;
sel.appendChild(opt);
});
} catch(e) {}
}
async function loadRagPoints() {
const table = document.getElementById('ragTableBody');
table.innerHTML = '<tr><td colspan="7" style="text-align:center;color:var(--text-muted);padding:40px;">Loading...</td></tr>';
try {
const search = document.getElementById('ragSearchInput').value.trim();
const source = document.getElementById('ragSourceFilter').value;
let url = `/api/rag/points?limit=20`;
if (search) url += `&search=${encodeURIComponent(search)}`;
if (source) url += `&source=${encodeURIComponent(source)}`;
if (_ragPageOffset !== null && !search) url += `&offset=${encodeURIComponent(_ragPageOffset)}`;
const resp = await authFetch(url);
const data = await resp.json();
_ragTotal = data.total || 0;
_ragNextOffset = data.next_offset || null;
table.innerHTML = '';
if (!data.points || data.points.length === 0) {
table.innerHTML = '<tr><td colspan="7" style="text-align:center;color:var(--text-muted);padding:40px;">No points found</td></tr>';
document.getElementById('ragPagination').style.display = 'none';
return;
}
data.points.forEach(p => {
const tr = document.createElement('tr');
const textSnippet = (p.text || '').substring(0, 200);
tr.innerHTML = `
<td class="rag-col-id" title="${escapeHtml(p.id)}">${escapeHtml((p.id || '').substring(0, 24))}${(p.id || '').length > 24 ? '...' : ''}</td>
<td class="rag-col-text"><div class="rag-text-snippet" onclick="this.classList.toggle('expanded')" title="Click to expand/collapse">${escapeHtml(textSnippet)}</div></td>
<td class="rag-col-source">${escapeHtml(p.source || '')}</td>
<td class="rag-col-type">${escapeHtml(p.type || '')}</td>
<td class="rag-col-date" style="white-space:nowrap;">${escapeHtml((p.date || '').substring(0, 16))}</td>
<td class="rag-col-retr">${p.retrieval_count || 0}</td>
<td class="rag-col-actions">
<button class="rag-action-btn" onclick="editRagPoint('${escapeHtml(p.id)}', this)" title="Edit">&#9998;</button>
<button class="rag-action-btn danger" onclick="deleteRagPoint('${escapeHtml(p.id)}')" title="Delete">&#10005;</button>
</td>
`;
table.appendChild(tr);
});
const pagination = document.getElementById('ragPagination');
if (!search) {
pagination.style.display = 'flex';
document.getElementById('ragPageInfo').textContent = `Page ${_ragPrevOffsets.length + 1} (${_ragTotal} total)`;
document.getElementById('ragPrevBtn').disabled = _ragPrevOffsets.length === 0;
document.getElementById('ragNextBtn').disabled = !_ragNextOffset;
} else {
pagination.style.display = 'flex';
document.getElementById('ragPageInfo').textContent = `${data.points.length} results`;
document.getElementById('ragPrevBtn').disabled = true;
document.getElementById('ragNextBtn').disabled = true;
}
} catch(e) {
table.innerHTML = '<tr><td colspan="7" style="text-align:center;color:var(--danger);padding:40px;">Failed to load points</td></tr>';
}
}
function ragPage(dir) {
if (dir === -1 && _ragPrevOffsets.length > 0) {
_ragPageOffset = _ragPrevOffsets.pop();
loadRagPoints();
} else if (dir === 1 && _ragNextOffset) {
_ragPrevOffsets.push(_ragPageOffset);
_ragPageOffset = _ragNextOffset;
_ragNextOffset = null;
loadRagPoints();
}
}
async function deleteRagPoint(pointId) {
if (!confirm('Delete this RAG point?')) return;
try {
const resp = await authFetch(`/api/rag/point/${encodeURIComponent(pointId)}`, { method: 'DELETE' });
if (!resp.ok) { showToast('Delete failed'); return; }
showToast('Point deleted');
loadRagStats();
loadRagPoints();
} catch(e) { showToast('Delete error'); }
}
async function editRagPoint(pointId, btn) {
const row = btn.closest('tr');
const currentText = row ? row.querySelector('.rag-text-snippet')?.textContent : '';
const newText = prompt('Edit RAG text:', currentText || '');
if (newText === null || newText.trim() === '' || newText === currentText) return;
try {
const resp = await authFetch(`/api/rag/point/${encodeURIComponent(pointId)}`, {
method: 'PATCH',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ text: newText }),
});
if (!resp.ok) { showToast('Update failed'); return; }
showToast('Point updated, re-embedded');
loadRagStats();
loadRagPoints();
} catch(e) { showToast('Update error'); }
}
async function flushRagCollection() {
if (!confirm('PERMANENTLY DELETE ALL RAG VECTORS? This cannot be undone.')) return;
if (!confirm('Are you absolutely sure? All RAG corpus entries will be lost.')) return;
try {
const resp = await authFetch('/api/rag/flush', { method: 'POST' });
const data = await resp.json();
showToast(`Flushed ${data.deleted_count} points`);
loadRagStats();
loadRagPoints();
} catch(e) { showToast('Flush failed'); }
}
async function loadConversations() { async function loadConversations() {
try { try {
const resp = await authFetch('/api/conversations'); const resp = await authFetch('/api/conversations');
+267
View File
@@ -8,6 +8,7 @@ from fastapi.testclient import TestClient
import app import app
import config import config
import crypto
import db import db
import rag import rag
from security import SESSIONS, PIN_ATTEMPTS, RATE_EVENTS from security import SESSIONS, PIN_ATTEMPTS, RATE_EVENTS
@@ -407,3 +408,269 @@ def test_eviction_lock_prevents_concurrent_eviction(monkeypatch):
# First call evicted, second found count already below high water or lock serialized # First call evicted, second found count already below high water or lock serialized
assert r1 >= 0 assert r1 >= 0
assert r2 >= 0 assert r2 >= 0
# ---------- GET /api/rag/points ----------
def test_rag_list_points_empty(tmp_path, monkeypatch):
monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: FakeAsyncClient())
with make_client(tmp_path) as client:
resp = client.get("/api/rag/points", headers=_admin_headers(client))
assert resp.status_code == 200
data = resp.json()
assert "points" in data
assert data["total"] == 123
assert len(data["points"]) == 0
def test_rag_list_points_requires_admin(tmp_path, monkeypatch):
monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: FakeAsyncClient())
with make_client(tmp_path) as client:
resp = client.get("/api/rag/points", headers=_guest_headers(client))
assert resp.status_code == 403
def test_rag_list_points_with_data(tmp_path, monkeypatch):
with make_client(tmp_path) as client:
encrypted = crypto.encrypt_text("Hello world test content")
class DataClient(FakeAsyncClient):
async def post(self, url, **kw):
if "/points/scroll" in url:
return FakeResponse(200, {
"result": {
"points": [{
"id": "test-pt-1",
"payload": {
"text": encrypted,
"source": "terminal",
"ingest_date": "2024-06-15T10:00:00",
"type": "ingest",
"retrieval_count": 3,
},
}],
"next_page_offset": None,
}
})
return FakeResponse(200)
async def get(self, url, **kw):
if "/collections/caic_rag" in url:
return FakeResponse(200, {"result": {"vectors_count": 1}})
return FakeResponse(200)
monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: DataClient())
resp = client.get("/api/rag/points", headers=_admin_headers(client))
assert resp.status_code == 200
data = resp.json()
assert len(data["points"]) == 1
p = data["points"][0]
assert p["id"] == "test-pt-1"
assert p["source"] == "terminal"
assert p["type"] == "ingest"
assert "Hello world" in p["text"]
assert p["retrieval_count"] == 3
def test_rag_list_points_source_filter(tmp_path, monkeypatch):
"""Source filter should be passed as Qdrant must-match."""
class FilterClient(FakeAsyncClient):
async def post(self, url, **kw):
if "/points/scroll" in url:
body = kw.get("json", {})
filt = body.get("filter", {})
must = filt.get("must", [])
# Verify the source filter was passed
assert any(m.get("match", {}).get("value") == "upload" for m in must)
return FakeResponse(200, {"result": {"points": [], "next_page_offset": None}})
return FakeResponse(200)
async def get(self, url, **kw):
return FakeResponse(200, {"result": {"vectors_count": 0}})
monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: FilterClient())
with make_client(tmp_path) as client:
resp = client.get("/api/rag/points?source=upload", headers=_admin_headers(client))
assert resp.status_code == 200
def test_rag_list_points_search(tmp_path, monkeypatch):
"""Semantic search should use Qdrant search endpoint."""
encrypted = crypto.encrypt_text("Semantic match text")
class SearchClient(FakeAsyncClient):
call_log = []
async def post(self, url, **kw):
SearchClient.call_log.append(url)
if "/api/embeddings" in url:
vec = [0.1] * 768
return FakeResponse(200, {"embedding": vec})
if "/points/search" in url:
return FakeResponse(200, {"result": [{
"id": "search-hit-1",
"score": 0.85,
"payload": {
"text": encrypted,
"source": "terminal",
"ingest_date": "2024-06-15T10:00:00",
"type": "ingest",
"retrieval_count": 1,
},
}]})
return FakeResponse(200)
async def get(self, url, **kw):
return FakeResponse(200, {"result": {}})
monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: SearchClient())
with make_client(tmp_path) as client:
resp = client.get("/api/rag/points?search=hello+world", headers=_admin_headers(client))
assert resp.status_code == 200
data = resp.json()
assert len(data["points"]) == 1
assert data["points"][0]["id"] == "search-hit-1"
assert data["points"][0]["score"] == 0.85
# ---------- GET /api/rag/point/{point_id} ----------
def test_rag_get_point_requires_admin(tmp_path, monkeypatch):
monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: FakeAsyncClient())
with make_client(tmp_path) as client:
resp = client.get("/api/rag/point/test-1", headers=_guest_headers(client))
assert resp.status_code == 403
def test_rag_get_point_found(tmp_path, monkeypatch):
with make_client(tmp_path) as client:
encrypted = crypto.encrypt_text("Single point text")
class GetClient(FakeAsyncClient):
async def get(self, url, **kw):
if "/collections/caic_rag/points/test-1" in url:
return FakeResponse(200, {"result": {
"id": "test-1",
"payload": {
"text": encrypted,
"source": "terminal",
"ingest_date": "2024-06-15T10:00:00",
"type": "ingest",
"retrieval_count": 5,
},
}})
return FakeResponse(200)
monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: GetClient())
resp = client.get("/api/rag/point/test-1", headers=_admin_headers(client))
assert resp.status_code == 200
p = resp.json()
assert p["id"] == "test-1"
assert p["source"] == "terminal"
assert "Single point" in p["text"]
def test_rag_get_point_not_found(tmp_path, monkeypatch):
class NotFoundClient(FakeAsyncClient):
async def get(self, url, **kw):
if "/collections/caic_rag/points/" in url:
return FakeResponse(404, {"detail": "Not found"})
return FakeResponse(200)
monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: NotFoundClient())
with make_client(tmp_path) as client:
resp = client.get("/api/rag/point/nonexistent", headers=_admin_headers(client))
assert resp.status_code == 404
# ---------- DELETE /api/rag/point/{point_id} ----------
def test_rag_delete_point_requires_admin(tmp_path, monkeypatch):
monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: FakeAsyncClient())
with make_client(tmp_path) as client:
resp = client.delete("/api/rag/point/test-1", headers=_guest_headers(client))
assert resp.status_code == 403
def test_rag_delete_point_success(tmp_path, monkeypatch):
class DeleteClient(FakeAsyncClient):
async def post(self, url, **kw):
if "/points/delete" in url:
pts = kw.get("json", {}).get("points", [])
assert "test-1" in pts
return FakeResponse(200)
return FakeResponse(200)
monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: DeleteClient())
with make_client(tmp_path) as client:
resp = client.delete("/api/rag/point/test-1", headers=_admin_headers(client))
assert resp.status_code == 200
data = resp.json()
assert data["status"] == "deleted"
assert data["id"] == "test-1"
# ---------- PATCH /api/rag/point/{point_id} ----------
def test_rag_update_point_requires_admin(tmp_path, monkeypatch):
monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: FakeAsyncClient())
with make_client(tmp_path) as client:
resp = client.patch("/api/rag/point/test-1", json={"text": "new"}, headers=_guest_headers(client))
assert resp.status_code == 403
def test_rag_update_point_success(tmp_path, monkeypatch):
encrypted_old = crypto.encrypt_text("old text")
class UpdateClient(FakeAsyncClient):
async def get(self, url, **kw):
if "/collections/caic_rag/points/test-1" in url:
return FakeResponse(200, {"result": {
"id": "test-1",
"payload": {
"text": encrypted_old,
"source": "terminal",
"ingest_date": "2024-06-15T10:00:00",
"type": "ingest",
"retrieval_count": 2,
},
}})
return FakeResponse(200)
async def post(self, url, **kw):
if "/api/embeddings" in url:
return FakeResponse(200, {"embedding": [0.2] * 768})
return FakeResponse(200)
async def put(self, url, **kw):
if "/points?wait=true" in url:
pts = kw.get("json", {}).get("points", [])
assert len(pts) == 1
assert len(pts[0]["vector"]) == 768
payload = pts[0]["payload"]
assert "text" in payload
assert payload["source"] == "terminal"
return FakeResponse(200)
return FakeResponse(200)
monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: UpdateClient())
with make_client(tmp_path) as client:
resp = client.patch("/api/rag/point/test-1", json={"text": "updated text"}, headers=_admin_headers(client))
assert resp.status_code == 200
data = resp.json()
assert data["status"] == "updated"
assert data["id"] == "test-1"
def test_rag_update_point_empty_text(tmp_path, monkeypatch):
monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: FakeAsyncClient())
with make_client(tmp_path) as client:
resp = client.patch("/api/rag/point/test-1", json={"text": ""}, headers=_admin_headers(client))
assert resp.status_code == 400
def test_rag_update_point_not_found(tmp_path, monkeypatch):
class NotFoundClient(FakeAsyncClient):
async def get(self, url, **kw):
if "/collections/caic_rag/points/" in url:
return FakeResponse(404)
return FakeResponse(200)
monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: NotFoundClient())
with make_client(tmp_path) as client:
resp = client.patch("/api/rag/point/nonexistent", json={"text": "new"}, headers=_admin_headers(client))
assert resp.status_code == 404