B4 RAG Corpus Management UI + AGENTS.md → ai.md rename

- Backend: GET/DELETE/PATCH /api/rag/point/{id}, GET /api/rag/points (paginated, semantic search, source filter, sort) - Frontend: admin-only RAG modal with stats bar, search, paginated table, per-row edit (re-embed) and delete, bulk flush with double confirm - 14 new tests, 214 total passing - AGENTS.md → ai.md (tool-agnostic), CLAUDE.md now references ai.md - Bump v0.21.0 → v0.22.0
2026-07-14 14:57:57 -07:00
parent 990b7c860d
commit 61f5a88673
8 changed files with 728 additions and 86 deletions
@@ -1,8 +1,8 @@
-# CLAUDE.md
+# ai.md — Project Context

-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+Detailed project context, work state, architecture, and configuration have moved to [`ai.md`](ai.md). This file is kept for backward compatibility — the canonical reference is `ai.md`.

-## Running the App
+## Quick start

 ```bash
 # Development
@@ -21,74 +21,3 @@ sudo systemctl restart caic
 ./venv/bin/pip install -r requirements.txt
 # Also requires: psutil jinja2 python-multipart pypdf (not in requirements.txt)
 ```
-
-## Architecture
-
-Modular FastAPI app — `app.py` wires routers, middleware, and lifespan. SQLite database auto-created at `caic.db` on first run. No build step, single `templates/index.html`.
-
-### Request Flow: `/api/chat`
-
-1. User message saved to DB → conversation created if new
-2. `process_remember_command()` intercepts "remember that..." / "forget about..." first
-3. Optional `upload_context_id` → fetches document text from `upload_context` table, injects `[ATTACHED DOCUMENT]` into system prompt
-4. `build_system_prompt()` assembles: profile + FTS5 memory search + Qdrant RAG + preset + skills + uploaded doc
-5. Streamed to llama-server (`/v1/chat/completions`, `stream: true`, `logprobs: true`) via SSE
-6. **Auto web search trigger**: if perplexity > 15.0 OR response matches `REFUSAL_PATTERNS`, re-queries with SearXNG results
-7. Final response saved to DB; SSE `done` event sent with perplexity + tokens/sec
-
-### Request Flow: `/api/search` (explicit search)
-
-Bypasses perplexity/refusal — queries SearXNG directly then asks llama-server to summarize results.
-
-### Request Flow: `/api/upload`
-
-Multipart file upload → PDF/text extraction + chunking → optional Qdrant upsert + SQLite context storage (1hr expiry). Supports `mode=(context|ingest|both)`. Images upload as storage only — model cannot process image content.
-
-### Request Flow: `/api/ingest`
-
-Bearer-token-authenticated terminal RAG hook. Accepts raw text, chunks via `chunk_text()`, embeds via Ollama `/api/embeddings`, upserts to Qdrant.
-
-### Memory System
-
-FTS5 virtual table (`memories`) in SQLite. `search_memories()` uses BM25 ranking. `process_remember_command()` intercepts "remember that..." / "forget about..." before the message reaches the model and returns a confirmation string.
-
-### Key Constants (`config.py`)
-
- `LLAMA_SERVER_BASE` — `http://192.168.50.108:8081` (coordinator llama-server, RPC offloads to worker GPU)
- `SEARXNG_BASE` — `http://localhost:8888`
- `QDRANT_URL` — `http://192.168.50.108:6333` (Qdrant on coordinator)
- `TRIAGE_BASE` — `http://127.0.0.1:8083/v1` (Phi-4-mini)
- `AMQP_URL` — `amqp://caic:{pw}@localhost:5672/caic` (RabbitMQ, pw read from `~/.caic_amqp_secret`)
- `PERPLEXITY_THRESHOLD` — `15.0`
- `EMBED_URL` — `http://192.168.50.210:11434/api/embeddings` (Ollama on worker)
- `VERSION` — current version string
-
-### External Services
-
-| Service | Required | Port |
-|---------|----------|------|
-| **llama-server** (coordinator) | Yes | 8081 + RPC :50052 (worker GPU) |
-| **Phi-4-mini** (triage) | No | 8083 |
-| **SearXNG** | No | 8888 |
-| **RabbitMQ** (coordinator) | No | 5672 — AMQP broker |
-| **wttr.in** | No | weather shortcut |
-| **rocm-smi** | No | AMD GPU stats |
-| **Qdrant** (coordinator) | No | 6333 — RAG vector search |
-| **Ollama** (worker) | No | 11434 — embeddings only |
-
-### Database
-
-`get_db()` opens a new connection per request (no pool). `init_db()` runs at startup via FastAPI `lifespan`. Tables: `conversations`, `messages`, `settings`, `profile` (singleton id=1), `memories` (FTS5), `upload_context`. Default settings seeded but never overwritten.
-
-### SSE Protocol
-
-All streaming endpoints yield `data: {json}\n\n`. Key event shapes:
- `{token, conversation_id}` — streaming token
- `{searching: true}` — web search triggered
- `{search_results: N}` — N results retrieved
- `{done: true, perplexity, tokens_per_sec, searched?}` — terminal event
- `{error: "...", error_key: "..."}` — error with incident key
-
-### Deployment
-
-Runs as systemd service under user `caic`, working directory `/opt/caic`. Logs via syslog (`journalctl -u caic`). Version bumps via git tag + commit, deployed via `git pull && systemctl restart caic`.
@@ -1,6 +1,6 @@
 ![cAIc banner](static/readme-banner.png)

-# cAIc v0.21.0
+# cAIc v0.22.0

 Consumer AI hardware is a wasteland of incompatibility. NVIDIA speaks CUDA, AMD speaks ROCm. Your RTX 5070 Ti lives in one machine with 16 GB VRAM; your RX 6600 XT lives in another with 12 GB. Alone, neither can run a 14B model at usable speed. Together, they could — if the software stack didn't treat heterogeneous hardware as a bug instead of a feature.

@@ -55,6 +55,16 @@ Untested: Windows 11 / WSL2 (Debian). The codebase is pure Python with no platfo

 Under the hood: FastAPI + SQLite + Jinja2 on Python 3.13. AMQP-mediated cluster coordination with an OpenAI-compatible inference endpoint.

+## What's New in v0.22.0
+
+### RAG Corpus Management UI (B4)
+- New admin modal (RAG button in drawer header) to browse, search, edit, and delete individual RAG corpus entries.
+- **Endpoints**: `GET /api/rag/points` (paginated list with semantic search and source filter), `GET /api/rag/point/{id}` (single point detail), `DELETE /api/rag/point/{id}` (single point deletion), `PATCH /api/rag/point/{id}` (edit text with re-embed).
+- **Frontend**: Stats bar (vector count, % full, pinned, avg retrievals, at-risk, eviction rate), semantic search bar, source filter dropdown, paginated results table, per-row edit/delete with confirmation, double-confirm bulk flush.
+- All endpoints admin-protected; text decrypted for display, re-encrypted on edit.
+- 14 new tests (32 total in test_rag_management.py). 214 tests pass overall.
+- Renamed `AGENTS.md` → `ai.md` for tool-agnostic project context. `CLAUDE.md` now points to `ai.md`.
+
 ## What's New in v0.21.0

 ### Scrollbar + DOM Fixes
@@ -26,6 +26,7 @@ Every router has a dedicated test file:
 | `test_models_router.py` | `routers/models.py` — models list, ps, show, stats, search/status |
 | `test_presets.py` | `routers/presets.py` — full CRUD, default preset protection |
 | `test_profile.py` | `routers/profile.py` — get, update, default, length validation |
+| `test_rag_management.py` | `eviction.py` + `routers/rag_admin.py` — eviction engine, stats, flush, browse, search, edit, delete individual points |
 | `test_search_route.py` | `routers/search_route.py` — explicit search flow, no results, errors |
 | `test_search_url_sanitization.py` | `search.py` URL sanitizer |
 | `test_cluster.py` | `cluster.py` — registration, deregistration, pong, events, coordinator query |
@@ -137,7 +138,7 @@ All streaming endpoints yield `data: {json}\n\n`. Key shapes:
 ## Work State

 ### Completed this session
- **v0.21.0 Release** — Scrollbar/DOM fixes, perplexity persistence, all service URLs env-overridable, single-node deployment docs, hardware.py Qdrant URL bugfix. See README for full changelog.
+- **B4 — RAG Corpus Management UI** — Paginated browse, semantic search, source filter, edit text with re-embed, single-point delete, bulk flush. Backend: `GET /api/rag/points`, `GET /api/rag/point/{id}`, `DELETE /api/rag/point/{id}`, `PATCH /api/rag/point/{id}`. Frontend: admin-only modal with stats bar, search bar, paginated table, per-row edit/delete, double-confirm flush.

 ### Active
 - (none)
@@ -145,16 +146,11 @@ All streaming endpoints yield `data: {json}\n\n`. Key shapes:
 ### Blocked
 - (none)

-### Blocked
- (none)
-
 ### Upcoming (backlog)
 - B3 — Docker distribution
- B4 — RAG Corpus Management UI
- Private Chat mode was implemented in B8 (v0.19.3). WireGuard in-transit encryption documented in wiki. At-rest encryption implemented in v0.20.0.

 ### Key config values (current)
- **Current VERSION**: `v0.21.0` in `config.py`.
+- **Current VERSION**: `v0.22.0` in `config.py`.
 - `SESSION_TIMEOUT_SECONDS = 3600`
 - `DEFAULT_MODEL = "qwen2.5-7b-instruct"`
 - `LLAMA_SERVER_BASE = "http://192.168.50.108:8081"`
@@ -9,7 +9,7 @@ import logging

 log = logging.getLogger("caic")

-VERSION = "v0.21.0"
+VERSION = "v0.22.0"
 OLLAMA_BASE = os.environ.get("OLLAMA_BASE", "http://localhost:11434")
 LLAMA_SERVER_BASE = os.environ.get("LLAMA_SERVER_BASE", "http://192.168.50.108:8081")
 SEARXNG_BASE = os.environ.get("CAIC_SEARXNG_BASE", "http://localhost:8888")
@@ -10,11 +10,11 @@ Scope: Active roadmap items and backlog.
 - **WireGuard TLS (v0.19.4)** — Self-signed WireGuard mesh encrypts all inter-node traffic (AMQP, inference, RPC). No code changes to cAIc. Documented in wiki/WireGuard-Setup.md + docker.md §5.4.
 - **At-Rest Encryption (v0.20.0)** — AES-256-GCM encrypts all query-derived text at rest. crypto.py with auto-keygen, key stored as `heartbeat_interval_ms` in settings. All 12 storage paths wired (SQLite: messages, conversations, memories, upload_context; Qdrant: RAG chunks, ingest, upload). 200 tests pass.
 - **v0.21.0** — Scrollbar/DOM fixes, perplexity persistence per message, all service URLs env-overridable, single-node deployment docs, DOM pairing bugfix, hardware.py Qdrant URL bugfix.
+- **v0.22.0** — B4 RAG Corpus Management UI: paginated browse, semantic search, source filter, edit with re-embed, single-point delete, bulk flush. `routers/rag_admin.py` expanded with 4 new endpoints; `CLAUDE.md` → `ai.md` rename.

 ## Backlog

 - B3 — Docker distribution (planning doc at `docker.md`, not yet implemented)
- **B4 — RAG Corpus Management UI (deferred)** — browse, search, edit, delete individual RAG entries
 - HTTPS / reverse proxy (Caddy)
 - Conversation search/filter and export tooling
 - Keyboard shortcuts, retry button, source-link polish
@@ -1,17 +1,38 @@
 """JarvisChat routers — RAG corpus management admin endpoints."""
 import logging
+from datetime import datetime, timezone

 import httpx
-from fastapi import APIRouter, HTTPException, Request
+from fastapi import APIRouter, HTTPException, Query, Request
 from fastapi.responses import JSONResponse

+from crypto import decrypt_text, encrypt_text
 from eviction import get_rag_operational_stats, EVICTION_LOG
-from rag import QDRANT_URL, RAG_COLLECTION
+from rag import QDRANT_URL, RAG_COLLECTION, EMBED_URL, EMBED_MODEL

 log = logging.getLogger("caic")
 router = APIRouter()


+def _normalise_date(payload: dict) -> str:
+    return payload.get("ingest_date") or payload.get("upload_date") or ""
+
+
+def _decrypt_payload(payload: dict) -> dict:
+    text = payload.get("text", "")
+    return {
+        "text": decrypt_text(text),
+        "source": payload.get("source"),
+        "type": payload.get("type"),
+        "date": _normalise_date(payload),
+        "retrieval_count": payload.get("retrieval_count", 0),
+        "topic": payload.get("topic"),
+        "fact": payload.get("fact"),
+        "filename": payload.get("filename"),
+        "last_accessed": payload.get("last_accessed"),
+    }
+
+
@router.get("/api/rag/stats")
 async def rag_stats(request: Request):
    if getattr(request.state, "session_role", "none") != "admin":
@@ -21,8 +42,178 @@ async def rag_stats(request: Request):
    return stats


+@router.get("/api/rag/points")
+async def rag_list_points(
+    request: Request,
+    limit: int = Query(20, ge=1, le=100),
+    offset: str = Query(None),
+    search: str = Query(None),
+    source: str = Query(None),
+    sort: str = Query("date"),
+    order: str = Query("desc"),
+):
+    if getattr(request.state, "session_role", "none") != "admin":
+        raise HTTPException(status_code=403, detail="Admin PIN required for this action")
+
+    async with httpx.AsyncClient() as client:
+        if search:
+            embed_resp = await client.post(
+                f"{EMBED_URL}/api/embeddings",
+                json={"model": EMBED_MODEL, "prompt": search},
+                timeout=10.0,
+            )
+            if embed_resp.status_code != 200:
+                return JSONResponse(status_code=502, content={"detail": "Embedding failed"})
+            vector = embed_resp.json()["embedding"]
+            search_body = {"vector": vector, "limit": limit, "with_payload": True}
+            if source:
+                search_body["filter"] = {"must": [{"match": {"key": "source", "value": source}}]}
+            sr = await client.post(
+                f"{QDRANT_URL}/collections/{RAG_COLLECTION}/points/search",
+                json=search_body,
+                timeout=10.0,
+            )
+            if sr.status_code != 200:
+                return JSONResponse(status_code=502, content={"detail": "Qdrant search failed"})
+            results = sr.json().get("result", [])
+            points = []
+            for r in results:
+                p = _decrypt_payload(r.get("payload", {}))
+                p["id"] = r["id"]
+                p["score"] = r.get("score")
+                points.append(p)
+            return {"points": points, "total": len(points), "next_offset": None}
+
+        scroll_body = {
+            "limit": limit,
+            "with_payload": True,
+            "with_vector": False,
+        }
+        if offset:
+            scroll_body["offset"] = offset
+        if source:
+            scroll_body["filter"] = {"must": [{"match": {"key": "source", "value": source}}]}
+        if sort:
+            direction = "asc" if order == "asc" else "desc"
+            scroll_body["order_by"] = {"key": sort, "direction": direction}
+
+        sr = await client.post(
+            f"{QDRANT_URL}/collections/{RAG_COLLECTION}/points/scroll",
+            json=scroll_body,
+            timeout=10.0,
+        )
+        if sr.status_code != 200:
+            return JSONResponse(status_code=502, content={"detail": "Qdrant scroll failed"})
+        result = sr.json().get("result", {})
+        raw_points = result.get("points", [])
+        next_offset = result.get("next_page_offset")
+
+        info_resp = await client.get(
+            f"{QDRANT_URL}/collections/{RAG_COLLECTION}",
+            timeout=5.0,
+        )
+        total = 0
+        if info_resp.status_code == 200:
+            total = info_resp.json().get("result", {}).get("vectors_count", 0)
+
+        points = []
+        for r in raw_points:
+            p = _decrypt_payload(r.get("payload", {}))
+            p["id"] = r["id"]
+            points.append(p)
+        return {"points": points, "total": total, "next_offset": next_offset}
+
+
+@router.get("/api/rag/point/{point_id}")
+async def rag_get_point(point_id: str, request: Request):
+    if getattr(request.state, "session_role", "none") != "admin":
+        raise HTTPException(status_code=403, detail="Admin PIN required for this action")
+
+    async with httpx.AsyncClient() as client:
+        resp = await client.get(
+            f"{QDRANT_URL}/collections/{RAG_COLLECTION}/points/{point_id}",
+            timeout=10.0,
+        )
+        if resp.status_code != 200:
+            raise HTTPException(status_code=404, detail="Point not found")
+        result = resp.json().get("result", {})
+        if not result:
+            raise HTTPException(status_code=404, detail="Point not found")
+        payload = result.get("payload", {})
+        point = _decrypt_payload(payload)
+        point["id"] = result["id"]
+        return point
+
+
+@router.delete("/api/rag/point/{point_id}")
+async def rag_delete_point(point_id: str, request: Request):
+    if getattr(request.state, "session_role", "none") != "admin":
+        raise HTTPException(status_code=403, detail="Admin PIN required for this action")
+
+    async with httpx.AsyncClient() as client:
+        resp = await client.post(
+            f"{QDRANT_URL}/collections/{RAG_COLLECTION}/points/delete",
+            json={"points": [point_id]},
+            timeout=10.0,
+        )
+        if resp.status_code not in (200, 201) and resp.status_code != 404:
+            return JSONResponse(status_code=502, content={"detail": "Qdrant delete failed"})
+        log.info(f"RAG point {point_id} deleted by admin")
+        return {"status": "deleted", "id": point_id}
+
+
+@router.patch("/api/rag/point/{point_id}")
+async def rag_update_point(point_id: str, request: Request):
+    if getattr(request.state, "session_role", "none") != "admin":
+        raise HTTPException(status_code=403, detail="Admin PIN required for this action")
+
+    body = await request.json()
+    new_text = (body.get("text") or "").strip()
+    if not new_text:
+        raise HTTPException(status_code=400, detail="text required")
+
+    async with httpx.AsyncClient() as client:
+        get_resp = await client.get(
+            f"{QDRANT_URL}/collections/{RAG_COLLECTION}/points/{point_id}",
+            timeout=10.0,
+        )
+        if get_resp.status_code != 200:
+            raise HTTPException(status_code=404, detail="Point not found")
+        existing = get_resp.json().get("result", {})
+        if not existing:
+            raise HTTPException(status_code=404, detail="Point not found")
+
+        old_payload = existing.get("payload", {})
+        embed_resp = await client.post(
+            f"{EMBED_URL}/api/embeddings",
+            json={"model": EMBED_MODEL, "prompt": new_text},
+            timeout=10.0,
+        )
+        if embed_resp.status_code != 200:
+            return JSONResponse(status_code=502, content={"detail": "Embedding failed"})
+        vector = embed_resp.json()["embedding"]
+
+        new_payload = dict(old_payload)
+        new_payload["text"] = encrypt_text(new_text)
+        date_field = "ingest_date" if "ingest_date" in old_payload else "upload_date"
+        new_payload[date_field] = datetime.now(timezone.utc).isoformat()
+
+        upsert_resp = await client.put(
+            f"{QDRANT_URL}/collections/{RAG_COLLECTION}/points?wait=true",
+            json={"points": [{"id": point_id, "vector": vector, "payload": new_payload}]},
+            timeout=10.0,
+        )
+        if upsert_resp.status_code not in (200, 201):
+            return JSONResponse(status_code=502, content={"detail": "Qdrant upsert failed"})
+
+        log.info(f"RAG point {point_id} updated by admin")
+        return {"status": "updated", "id": point_id}
+
+
@router.post("/api/rag/flush")
 async def rag_flush(request: Request):
+    if getattr(request.state, "session_role", "none") != "admin":
+        raise HTTPException(status_code=403, detail="Admin PIN required for this action")
    try:
        async with httpx.AsyncClient() as client:
            scroll_resp = await client.post(
@@ -129,6 +129,39 @@ body { font-family: var(--font-body); background: var(--bg-primary); color: var(
 .modal-close { background:none; border:none; color:var(--text-muted); font-size:24px; cursor:pointer; }
 .modal-close:hover { color:var(--text-primary); }
 .modal-body { padding: 20px 24px; }
+.rag-stats-bar { display:flex; flex-wrap:wrap; gap:12px 24px; padding:10px 14px; background:var(--bg-tertiary); border-radius:var(--radius); margin-bottom:14px; font-size:11px; font-family:var(--font-mono); }
+.rag-stat-item { display:flex; gap:4px; align-items:center; }
+.rag-stat-label { color:var(--text-muted); text-transform:uppercase; }
+.rag-stat-value { color:var(--text-primary); font-weight:600; }
+.rag-toolbar { display:flex; align-items:center; justify-content:space-between; gap:10px; margin-bottom:12px; flex-wrap:wrap; }
+.rag-toolbar-left { display:flex; gap:8px; align-items:center; flex:1; }
+.rag-toolbar-right { display:flex; gap:8px; align-items:center; flex-shrink:0; }
+.rag-search-input { flex:1; min-width:160px; background:var(--bg-tertiary); border:1px solid var(--border); color:var(--text-primary); font-family:var(--font-mono); font-size:12px; padding:8px 12px; border-radius:var(--radius); }
+.rag-search-input:focus { outline:none; border-color:var(--accent-dim); }
+.rag-source-select { background:var(--bg-tertiary); border:1px solid var(--border); color:var(--text-primary); font-family:var(--font-mono); font-size:12px; padding:7px 10px; border-radius:var(--radius); cursor:pointer; }
+.rag-source-select:focus { outline:none; border-color:var(--accent-dim); }
+.rag-table-wrap { overflow-x:auto; margin-bottom:12px; }
+.rag-table { width:100%; border-collapse:collapse; font-size:11px; font-family:var(--font-mono); }
+.rag-table th { text-align:left; padding:8px 6px; border-bottom:1px solid var(--border); color:var(--text-secondary); text-transform:uppercase; letter-spacing:0.5px; font-size:10px; white-space:nowrap; }
+.rag-table td { padding:8px 6px; border-bottom:1px solid var(--border); color:var(--text-primary); vertical-align:top; }
+.rag-table tr:hover td { background:var(--bg-hover); }
+.rag-col-id { width:80px; }
+.rag-col-text { min-width:200px; }
+.rag-col-source { width:100px; }
+.rag-col-type { width:80px; }
+.rag-col-date { width:140px; }
+.rag-col-retr { width:50px; text-align:center; }
+.rag-col-actions { width:80px; text-align:center; white-space:nowrap; }
+.rag-text-snippet { display:-webkit-box; -webkit-line-clamp:2; -webkit-box-orient:vertical; overflow:hidden; line-height:1.4; word-break:break-word; cursor:pointer; }
+.rag-text-snippet.expanded { -webkit-line-clamp:unset; }
+.rag-action-btn { background:none; border:1px solid var(--border); border-radius:var(--radius); cursor:pointer; font-size:11px; padding:3px 7px; color:var(--text-muted); transition:all 0.15s; }
+.rag-action-btn:hover { border-color:var(--accent-dim); color:var(--accent); }
+.rag-action-btn.danger:hover { border-color:var(--danger); color:var(--danger); }
+.rag-pagination { display:flex; align-items:center; justify-content:center; gap:12px; padding:8px 0; }
+.rag-page-info { font-size:11px; color:var(--text-muted); font-family:var(--font-mono); }
+.rag-edit-modal { max-width:700px; }
+.rag-edit-textarea { width:100%; min-height:200px; background:var(--bg-tertiary); border:1px solid var(--border); color:var(--text-primary); font-family:var(--font-mono); font-size:12px; padding:12px; border-radius:var(--radius); resize:vertical; line-height:1.6; }
+.rag-edit-textarea:focus { outline:none; border-color:var(--accent-dim); }
 .modal-section { margin-bottom: 24px; }
 .modal-section h3 { font-family:var(--font-mono); font-size:13px; color:var(--text-secondary); text-transform:uppercase; letter-spacing:1px; margin-bottom:8px; }
 .modal-section p.desc { font-size:12px; color:var(--text-muted); margin-bottom:10px; line-height:1.5; }
@@ -318,6 +351,7 @@ body { font-family: var(--font-body); background: var(--bg-primary); color: var(
        <span class="drawer-title">cAIc <span class="version">{{ version }}</span></span>
        <div class="drawer-actions">
            <button class="drawer-btn" onclick="newChat()" title="New chat">+</button>
+            <button class="drawer-btn" onclick="openRagAdmin()" title="RAG corpus management">R</button>
            <button class="drawer-btn" onclick="openSettings()" title="Settings">&#9881;</button>
            <button class="drawer-btn danger" onclick="deleteAllConversations()" title="Delete all">&#128465;</button>
        </div>
@@ -332,6 +366,53 @@ body { font-family: var(--font-body); background: var(--bg-primary); color: var(
    </div>
 </aside>

+<div class="modal-overlay" id="ragAdminModal">
+    <div class="modal" style="max-width:960px;">
+        <div class="modal-header">
+            <h2>RAG Corpus Management</h2>
+            <button class="modal-close" onclick="closeRagAdmin()">&times;</button>
+        </div>
+        <div class="modal-body">
+            <div class="rag-stats-bar" id="ragStatsBar">Loading stats...</div>
+            <div class="rag-toolbar">
+                <div class="rag-toolbar-left">
+                    <input type="text" id="ragSearchInput" class="rag-search-input" placeholder="Semantic search..." onkeydown="if(event.key==='Enter')loadRagPoints()">
+                    <select id="ragSourceFilter" class="rag-source-select" onchange="loadRagPoints()">
+                        <option value="">All sources</option>
+                    </select>
+                </div>
+                <div class="rag-toolbar-right">
+                    <button class="btn-small btn-save" onclick="loadRagPoints()" id="ragRefreshBtn">&#8635; Refresh</button>
+                    <button class="btn-small btn-reset" onclick="flushRagCollection()" style="color:var(--danger);border-color:var(--danger);">Flush All</button>
+                </div>
+            </div>
+            <div class="rag-table-wrap">
+                <table class="rag-table" id="ragTable">
+                    <thead>
+                        <tr>
+                            <th class="rag-col-id">ID</th>
+                            <th class="rag-col-text">Text</th>
+                            <th class="rag-col-source">Source</th>
+                            <th class="rag-col-type">Type</th>
+                            <th class="rag-col-date">Date</th>
+                            <th class="rag-col-retr">Retr</th>
+                            <th class="rag-col-actions">Actions</th>
+                        </tr>
+                    </thead>
+                    <tbody id="ragTableBody">
+                        <tr><td colspan="7" style="text-align:center;color:var(--text-muted);padding:40px;">No points loaded</td></tr>
+                    </tbody>
+                </table>
+            </div>
+            <div class="rag-pagination" id="ragPagination" style="display:none;">
+                <button class="btn-small" id="ragPrevBtn" onclick="ragPage(-1)">&#9664; Prev</button>
+                <span class="rag-page-info" id="ragPageInfo">Page 1</span>
+                <button class="btn-small" id="ragNextBtn" onclick="ragPage(1)">Next &#9654;</button>
+            </div>
+        </div>
+    </div>
+</div>
+
 <div class="modal-overlay" id="settingsModal">
    <div class="modal">
        <div class="modal-header">
@@ -1128,6 +1209,174 @@ function openSettings() {
 function closeSettings() { document.getElementById('settingsModal').classList.remove('visible'); }
 document.getElementById('settingsModal').addEventListener('click', e => { if (e.target.id === 'settingsModal') closeSettings(); });

+// ---------- RAG Admin ----------
+let _ragPageOffset = null;
+let _ragPrevOffsets = [];
+let _ragNextOffset = null;
+let _ragTotal = 0;
+
+function openRagAdmin() {
+    if (currentRole !== 'admin') { requireAdminNotice(); return; }
+    document.getElementById('ragAdminModal').classList.add('visible');
+    _ragPageOffset = null;
+    _ragPrevOffsets = [];
+    _ragNextOffset = null;
+    _ragTotal = 0;
+    loadRagStats();
+    loadRagPoints();
+    populateRagSourceFilter();
+}
+function closeRagAdmin() { document.getElementById('ragAdminModal').classList.remove('visible'); }
+document.getElementById('ragAdminModal').addEventListener('click', e => { if (e.target.id === 'ragAdminModal') closeRagAdmin(); });
+
+async function loadRagStats() {
+    try {
+        const resp = await authFetch('/api/rag/stats');
+        const data = await resp.json();
+        const bar = document.getElementById('ragStatsBar');
+        bar.innerHTML = `
+            <span class="rag-stat-item"><span class="rag-stat-label">Vectors</span> <span class="rag-stat-value">${data.vector_count}</span></span>
+            <span class="rag-stat-item"><span class="rag-stat-label">Max</span> <span class="rag-stat-value">${data.max_vectors}</span></span>
+            <span class="rag-stat-item"><span class="rag-stat-label">Full</span> <span class="rag-stat-value">${data.percent_full}%</span></span>
+            <span class="rag-stat-item"><span class="rag-stat-label">Pinned</span> <span class="rag-stat-value">${data.pinned_count}</span></span>
+            <span class="rag-stat-item"><span class="rag-stat-label">Avg Retr</span> <span class="rag-stat-value">${data.avg_retrieval_count}</span></span>
+            <span class="rag-stat-item"><span class="rag-stat-label">At Risk</span> <span class="rag-stat-value">${data.at_risk_count}</span></span>
+            <span class="rag-stat-item"><span class="rag-stat-label">Evictions (1m)</span> <span class="rag-stat-value">${data.eviction_counts_last_1m}</span></span>
+        `;
+    } catch(e) {
+        document.getElementById('ragStatsBar').textContent = 'Failed to load stats';
+    }
+}
+
+async function populateRagSourceFilter() {
+    const sel = document.getElementById('ragSourceFilter');
+    if (sel.options.length > 1) return;
+    try {
+        const resp = await authFetch('/api/rag/points?limit=100');
+        const data = await resp.json();
+        const sources = new Set();
+        (data.points || []).forEach(p => { if (p.source) sources.add(p.source); });
+        const sorted = Array.from(sources).sort();
+        sorted.forEach(s => {
+            const opt = document.createElement('option');
+            opt.value = s;
+            opt.textContent = s;
+            sel.appendChild(opt);
+        });
+    } catch(e) {}
+}
+
+async function loadRagPoints() {
+    const table = document.getElementById('ragTableBody');
+    table.innerHTML = '<tr><td colspan="7" style="text-align:center;color:var(--text-muted);padding:40px;">Loading...</td></tr>';
+    try {
+        const search = document.getElementById('ragSearchInput').value.trim();
+        const source = document.getElementById('ragSourceFilter').value;
+        let url = `/api/rag/points?limit=20`;
+        if (search) url += `&search=${encodeURIComponent(search)}`;
+        if (source) url += `&source=${encodeURIComponent(source)}`;
+        if (_ragPageOffset !== null && !search) url += `&offset=${encodeURIComponent(_ragPageOffset)}`;
+
+        const resp = await authFetch(url);
+        const data = await resp.json();
+        _ragTotal = data.total || 0;
+        _ragNextOffset = data.next_offset || null;
+
+        table.innerHTML = '';
+        if (!data.points || data.points.length === 0) {
+            table.innerHTML = '<tr><td colspan="7" style="text-align:center;color:var(--text-muted);padding:40px;">No points found</td></tr>';
+            document.getElementById('ragPagination').style.display = 'none';
+            return;
+        }
+
+        data.points.forEach(p => {
+            const tr = document.createElement('tr');
+            const textSnippet = (p.text || '').substring(0, 200);
+            tr.innerHTML = `
+                <td class="rag-col-id" title="${escapeHtml(p.id)}">${escapeHtml((p.id || '').substring(0, 24))}${(p.id || '').length > 24 ? '...' : ''}</td>
+                <td class="rag-col-text"><div class="rag-text-snippet" onclick="this.classList.toggle('expanded')" title="Click to expand/collapse">${escapeHtml(textSnippet)}</div></td>
+                <td class="rag-col-source">${escapeHtml(p.source || '')}</td>
+                <td class="rag-col-type">${escapeHtml(p.type || '')}</td>
+                <td class="rag-col-date" style="white-space:nowrap;">${escapeHtml((p.date || '').substring(0, 16))}</td>
+                <td class="rag-col-retr">${p.retrieval_count || 0}</td>
+                <td class="rag-col-actions">
+                    <button class="rag-action-btn" onclick="editRagPoint('${escapeHtml(p.id)}', this)" title="Edit">&#9998;</button>
+                    <button class="rag-action-btn danger" onclick="deleteRagPoint('${escapeHtml(p.id)}')" title="Delete">&#10005;</button>
+                </td>
+            `;
+            table.appendChild(tr);
+        });
+
+        const pagination = document.getElementById('ragPagination');
+        if (!search) {
+            pagination.style.display = 'flex';
+            document.getElementById('ragPageInfo').textContent = `Page ${_ragPrevOffsets.length + 1} (${_ragTotal} total)`;
+            document.getElementById('ragPrevBtn').disabled = _ragPrevOffsets.length === 0;
+            document.getElementById('ragNextBtn').disabled = !_ragNextOffset;
+        } else {
+            pagination.style.display = 'flex';
+            document.getElementById('ragPageInfo').textContent = `${data.points.length} results`;
+            document.getElementById('ragPrevBtn').disabled = true;
+            document.getElementById('ragNextBtn').disabled = true;
+        }
+    } catch(e) {
+        table.innerHTML = '<tr><td colspan="7" style="text-align:center;color:var(--danger);padding:40px;">Failed to load points</td></tr>';
+    }
+}
+
+function ragPage(dir) {
+    if (dir === -1 && _ragPrevOffsets.length > 0) {
+        _ragPageOffset = _ragPrevOffsets.pop();
+        loadRagPoints();
+    } else if (dir === 1 && _ragNextOffset) {
+        _ragPrevOffsets.push(_ragPageOffset);
+        _ragPageOffset = _ragNextOffset;
+        _ragNextOffset = null;
+        loadRagPoints();
+    }
+}
+
+async function deleteRagPoint(pointId) {
+    if (!confirm('Delete this RAG point?')) return;
+    try {
+        const resp = await authFetch(`/api/rag/point/${encodeURIComponent(pointId)}`, { method: 'DELETE' });
+        if (!resp.ok) { showToast('Delete failed'); return; }
+        showToast('Point deleted');
+        loadRagStats();
+        loadRagPoints();
+    } catch(e) { showToast('Delete error'); }
+}
+
+async function editRagPoint(pointId, btn) {
+    const row = btn.closest('tr');
+    const currentText = row ? row.querySelector('.rag-text-snippet')?.textContent : '';
+    const newText = prompt('Edit RAG text:', currentText || '');
+    if (newText === null || newText.trim() === '' || newText === currentText) return;
+    try {
+        const resp = await authFetch(`/api/rag/point/${encodeURIComponent(pointId)}`, {
+            method: 'PATCH',
+            headers: { 'Content-Type': 'application/json' },
+            body: JSON.stringify({ text: newText }),
+        });
+        if (!resp.ok) { showToast('Update failed'); return; }
+        showToast('Point updated, re-embedded');
+        loadRagStats();
+        loadRagPoints();
+    } catch(e) { showToast('Update error'); }
+}
+
+async function flushRagCollection() {
+    if (!confirm('PERMANENTLY DELETE ALL RAG VECTORS? This cannot be undone.')) return;
+    if (!confirm('Are you absolutely sure? All RAG corpus entries will be lost.')) return;
+    try {
+        const resp = await authFetch('/api/rag/flush', { method: 'POST' });
+        const data = await resp.json();
+        showToast(`Flushed ${data.deleted_count} points`);
+        loadRagStats();
+        loadRagPoints();
+    } catch(e) { showToast('Flush failed'); }
+}
+
 async function loadConversations() {
    try {
        const resp = await authFetch('/api/conversations');
@@ -8,6 +8,7 @@ from fastapi.testclient import TestClient

 import app
 import config
+import crypto
 import db
 import rag
 from security import SESSIONS, PIN_ATTEMPTS, RATE_EVENTS
@@ -407,3 +408,269 @@ def test_eviction_lock_prevents_concurrent_eviction(monkeypatch):
    # First call evicted, second found count already below high water or lock serialized
    assert r1 >= 0
    assert r2 >= 0
+
+
+# ---------- GET /api/rag/points ----------
+
+def test_rag_list_points_empty(tmp_path, monkeypatch):
+    monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: FakeAsyncClient())
+    with make_client(tmp_path) as client:
+        resp = client.get("/api/rag/points", headers=_admin_headers(client))
+        assert resp.status_code == 200
+        data = resp.json()
+        assert "points" in data
+        assert data["total"] == 123
+        assert len(data["points"]) == 0
+
+
+def test_rag_list_points_requires_admin(tmp_path, monkeypatch):
+    monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: FakeAsyncClient())
+    with make_client(tmp_path) as client:
+        resp = client.get("/api/rag/points", headers=_guest_headers(client))
+        assert resp.status_code == 403
+
+
+def test_rag_list_points_with_data(tmp_path, monkeypatch):
+    with make_client(tmp_path) as client:
+        encrypted = crypto.encrypt_text("Hello world test content")
+
+        class DataClient(FakeAsyncClient):
+            async def post(self, url, **kw):
+                if "/points/scroll" in url:
+                    return FakeResponse(200, {
+                        "result": {
+                            "points": [{
+                                "id": "test-pt-1",
+                                "payload": {
+                                    "text": encrypted,
+                                    "source": "terminal",
+                                    "ingest_date": "2024-06-15T10:00:00",
+                                    "type": "ingest",
+                                    "retrieval_count": 3,
+                                },
+                            }],
+                            "next_page_offset": None,
+                        }
+                    })
+                return FakeResponse(200)
+            async def get(self, url, **kw):
+                if "/collections/caic_rag" in url:
+                    return FakeResponse(200, {"result": {"vectors_count": 1}})
+                return FakeResponse(200)
+
+        monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: DataClient())
+        resp = client.get("/api/rag/points", headers=_admin_headers(client))
+        assert resp.status_code == 200
+        data = resp.json()
+        assert len(data["points"]) == 1
+        p = data["points"][0]
+        assert p["id"] == "test-pt-1"
+        assert p["source"] == "terminal"
+        assert p["type"] == "ingest"
+        assert "Hello world" in p["text"]
+        assert p["retrieval_count"] == 3
+
+
+def test_rag_list_points_source_filter(tmp_path, monkeypatch):
+    """Source filter should be passed as Qdrant must-match."""
+    class FilterClient(FakeAsyncClient):
+        async def post(self, url, **kw):
+            if "/points/scroll" in url:
+                body = kw.get("json", {})
+                filt = body.get("filter", {})
+                must = filt.get("must", [])
+                # Verify the source filter was passed
+                assert any(m.get("match", {}).get("value") == "upload" for m in must)
+                return FakeResponse(200, {"result": {"points": [], "next_page_offset": None}})
+            return FakeResponse(200)
+        async def get(self, url, **kw):
+            return FakeResponse(200, {"result": {"vectors_count": 0}})
+
+    monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: FilterClient())
+    with make_client(tmp_path) as client:
+        resp = client.get("/api/rag/points?source=upload", headers=_admin_headers(client))
+        assert resp.status_code == 200
+
+
+def test_rag_list_points_search(tmp_path, monkeypatch):
+    """Semantic search should use Qdrant search endpoint."""
+    encrypted = crypto.encrypt_text("Semantic match text")
+
+    class SearchClient(FakeAsyncClient):
+        call_log = []
+        async def post(self, url, **kw):
+            SearchClient.call_log.append(url)
+            if "/api/embeddings" in url:
+                vec = [0.1] * 768
+                return FakeResponse(200, {"embedding": vec})
+            if "/points/search" in url:
+                return FakeResponse(200, {"result": [{
+                    "id": "search-hit-1",
+                    "score": 0.85,
+                    "payload": {
+                        "text": encrypted,
+                        "source": "terminal",
+                        "ingest_date": "2024-06-15T10:00:00",
+                        "type": "ingest",
+                        "retrieval_count": 1,
+                    },
+                }]})
+            return FakeResponse(200)
+        async def get(self, url, **kw):
+            return FakeResponse(200, {"result": {}})
+
+    monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: SearchClient())
+    with make_client(tmp_path) as client:
+        resp = client.get("/api/rag/points?search=hello+world", headers=_admin_headers(client))
+        assert resp.status_code == 200
+        data = resp.json()
+        assert len(data["points"]) == 1
+        assert data["points"][0]["id"] == "search-hit-1"
+        assert data["points"][0]["score"] == 0.85
+
+
+# ---------- GET /api/rag/point/{point_id} ----------
+
+def test_rag_get_point_requires_admin(tmp_path, monkeypatch):
+    monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: FakeAsyncClient())
+    with make_client(tmp_path) as client:
+        resp = client.get("/api/rag/point/test-1", headers=_guest_headers(client))
+        assert resp.status_code == 403
+
+
+def test_rag_get_point_found(tmp_path, monkeypatch):
+    with make_client(tmp_path) as client:
+        encrypted = crypto.encrypt_text("Single point text")
+
+        class GetClient(FakeAsyncClient):
+            async def get(self, url, **kw):
+                if "/collections/caic_rag/points/test-1" in url:
+                    return FakeResponse(200, {"result": {
+                        "id": "test-1",
+                        "payload": {
+                            "text": encrypted,
+                            "source": "terminal",
+                            "ingest_date": "2024-06-15T10:00:00",
+                            "type": "ingest",
+                            "retrieval_count": 5,
+                        },
+                    }})
+                return FakeResponse(200)
+
+        monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: GetClient())
+        resp = client.get("/api/rag/point/test-1", headers=_admin_headers(client))
+        assert resp.status_code == 200
+        p = resp.json()
+        assert p["id"] == "test-1"
+        assert p["source"] == "terminal"
+        assert "Single point" in p["text"]
+
+
+def test_rag_get_point_not_found(tmp_path, monkeypatch):
+    class NotFoundClient(FakeAsyncClient):
+        async def get(self, url, **kw):
+            if "/collections/caic_rag/points/" in url:
+                return FakeResponse(404, {"detail": "Not found"})
+            return FakeResponse(200)
+
+    monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: NotFoundClient())
+    with make_client(tmp_path) as client:
+        resp = client.get("/api/rag/point/nonexistent", headers=_admin_headers(client))
+        assert resp.status_code == 404
+
+
+# ---------- DELETE /api/rag/point/{point_id} ----------
+
+def test_rag_delete_point_requires_admin(tmp_path, monkeypatch):
+    monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: FakeAsyncClient())
+    with make_client(tmp_path) as client:
+        resp = client.delete("/api/rag/point/test-1", headers=_guest_headers(client))
+        assert resp.status_code == 403
+
+
+def test_rag_delete_point_success(tmp_path, monkeypatch):
+    class DeleteClient(FakeAsyncClient):
+        async def post(self, url, **kw):
+            if "/points/delete" in url:
+                pts = kw.get("json", {}).get("points", [])
+                assert "test-1" in pts
+                return FakeResponse(200)
+            return FakeResponse(200)
+
+    monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: DeleteClient())
+    with make_client(tmp_path) as client:
+        resp = client.delete("/api/rag/point/test-1", headers=_admin_headers(client))
+        assert resp.status_code == 200
+        data = resp.json()
+        assert data["status"] == "deleted"
+        assert data["id"] == "test-1"
+
+
+# ---------- PATCH /api/rag/point/{point_id} ----------
+
+def test_rag_update_point_requires_admin(tmp_path, monkeypatch):
+    monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: FakeAsyncClient())
+    with make_client(tmp_path) as client:
+        resp = client.patch("/api/rag/point/test-1", json={"text": "new"}, headers=_guest_headers(client))
+        assert resp.status_code == 403
+
+
+def test_rag_update_point_success(tmp_path, monkeypatch):
+    encrypted_old = crypto.encrypt_text("old text")
+
+    class UpdateClient(FakeAsyncClient):
+        async def get(self, url, **kw):
+            if "/collections/caic_rag/points/test-1" in url:
+                return FakeResponse(200, {"result": {
+                    "id": "test-1",
+                    "payload": {
+                        "text": encrypted_old,
+                        "source": "terminal",
+                        "ingest_date": "2024-06-15T10:00:00",
+                        "type": "ingest",
+                        "retrieval_count": 2,
+                    },
+                }})
+            return FakeResponse(200)
+        async def post(self, url, **kw):
+            if "/api/embeddings" in url:
+                return FakeResponse(200, {"embedding": [0.2] * 768})
+            return FakeResponse(200)
+        async def put(self, url, **kw):
+            if "/points?wait=true" in url:
+                pts = kw.get("json", {}).get("points", [])
+                assert len(pts) == 1
+                assert len(pts[0]["vector"]) == 768
+                payload = pts[0]["payload"]
+                assert "text" in payload
+                assert payload["source"] == "terminal"
+                return FakeResponse(200)
+            return FakeResponse(200)
+
+    monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: UpdateClient())
+    with make_client(tmp_path) as client:
+        resp = client.patch("/api/rag/point/test-1", json={"text": "updated text"}, headers=_admin_headers(client))
+        assert resp.status_code == 200
+        data = resp.json()
+        assert data["status"] == "updated"
+        assert data["id"] == "test-1"
+
+
+def test_rag_update_point_empty_text(tmp_path, monkeypatch):
+    monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: FakeAsyncClient())
+    with make_client(tmp_path) as client:
+        resp = client.patch("/api/rag/point/test-1", json={"text": ""}, headers=_admin_headers(client))
+        assert resp.status_code == 400
+
+
+def test_rag_update_point_not_found(tmp_path, monkeypatch):
+    class NotFoundClient(FakeAsyncClient):
+        async def get(self, url, **kw):
+            if "/collections/caic_rag/points/" in url:
+                return FakeResponse(404)
+            return FakeResponse(200)
+
+    monkeypatch.setattr(httpx, "AsyncClient", lambda *a, **kw: NotFoundClient())
+    with make_client(tmp_path) as client:
+        resp = client.patch("/api/rag/point/nonexistent", json={"text": "new"}, headers=_admin_headers(client))
+        assert resp.status_code == 404