docs: update AGENTS.md, CLAUDE.md, TASKS.md for Tasks 4-6 completion

AGENTS.md

@@ -12,15 +12,16 @@
 ./venv/bin/python -m pytest tests/ -v
 ```
 
-All tests use `tmp_path` fixtures + monkeypatched `httpx.AsyncClient.stream`. No external services needed. Test factories reset `SESSIONS`, `PIN_ATTEMPTS`, `RATE_EVENTS` globals — be careful not to let test state leak. After the modular refactor, tests import directly from the correct modules (`db`, `security`, `config`, `search`, `rag`, `memory`, `routers.*`) — not from the old monolithic `app` namespace.
+All tests use `tmp_path` fixtures + monkeypatched `httpx.AsyncClient.stream/get/post/put`. No external services needed. Test factories reset `SESSIONS`, `PIN_ATTEMPTS`, `RATE_EVENTS` globals — be careful not to let test state leak. Tests import directly from the correct modules (`db`, `security`, `config`, `search`, `rag`, `memory`, `routers.*`).
 
 Every router has a dedicated test file:
 | File | Covers |
 |------|--------|
 | `test_auth_capabilities.py` | `auth.py` — guest/admin sessions, origin blocking, logout |
-| `test_chat_streaming_and_memory_paths.py` | `routers/chat.py` — streaming, auto-search, remember/forget |
+| `test_chat_streaming_and_memory_paths.py` | `routers/chat.py` — streaming, auto-search, remember/forget, upload context injection |
 | `test_completions.py` | `routers/completions.py` — API key auth, FIM, streaming, blocking, errors |
-| `test_conversations.py` | `routers/conversations.py` — full CRUD, guest admin enforcement |
+| `test_conversations.py` | `routers/conversations.py` — full CRUD, guest admin enforcement, attachment_count |
+| `test_ingest.py` | `routers/ingest.py` — Bearer auth, chunk/embed/upsert, validation |
 | `test_memories.py` | `routers/memories.py` — edit, search, stats endpoints |
 | `test_models_router.py` | `routers/models.py` — models list, ps, show, stats, search/status |
 | `test_presets.py` | `routers/presets.py` — full CRUD, default preset protection |
@@ -32,8 +33,9 @@ Every router has a dedicated test file:
 | `test_ip_allowlist.py` | IP allowlist helper + middleware |
 | `test_rate_and_payload_guardrails.py` | Rate limits + payload size enforcement |
 | `test_error_envelopes.py` | Global exception handler + stream error incidents |
+| `test_upload.py` | `routers/upload.py` — upload, delete, link, by-conversation, attachment_count integration |
 
-Modules that call `httpx.AsyncClient` (chat, completions, models, search_route)
+Modules that call `httpx.AsyncClient` (chat, completions, models, search_route, upload, ingest)
 are mocked via `monkeypatch.setattr` on `AsyncClient.stream`, `.get`, or `.post`.
 CPU stats in `models.py` (`api/stats`) use real `psutil`; GPU stats are
 monkeypatched via `routers.models.get_gpu_stats`.
@@ -45,64 +47,68 @@ Refactored from single-file (`app.py`) into modules under project root:
 | File | Role |
 |------|------|
 | `app.py` | FastAPI app, middleware, router registration |
-| `config.py` | Constants, env vars, rate/payload limits, built-in skills registry |
+| `config.py` | Constants, env vars, rate/payload limits, built-in skills registry, upload limits |
-| `db.py` | SQLite schema, connection factory, settings helpers |
+| `db.py` | SQLite schema, connection factory, settings helpers, upload_context CRUD |
 | `auth.py` | PIN-based guest/admin sessions, auth routes |
 | `security.py` | Rate limiting, origin checks, IP allowlist, audit/incident logging |
 | `memory.py` | FTS5 memory CRUD, remember/forget command parsing |
 | `search.py` | SearXNG integration, perplexity scoring, refusal detection |
-| `rag.py` | Qdrant vector search + system prompt assembly |
+| `rag.py` | Qdrant vector search + system prompt assembly + chunk_text() helper |
 | `gpu.py` | AMD GPU stats via `rocm-smi` |
-| `routers/` | One module per endpoint group (chat, search, skills, completions, etc.) |
+| `routers/` | One module per endpoint group (chat, search, skills, completions, upload, ingest) |
 
 ### Entrypoint / API keys
 
 - `app.py` line 148: `uvicorn.run(app, ...)` when called directly
-- `config.py` line 14: `LLAMA_SERVER_BASE` defaults to `http://192.168.50.108:8081` — llama-server, **not** standard Ollama port, used by all model endpoints
+- `config.py` line 14: `LLAMA_SERVER_BASE` defaults to `http://192.168.50.108:8081` — llama-server on ultron, RPC-offloads GPU layers to jarvis :50052
-- `config.py` line 17: `COMPLETIONS_API_KEY` read from `JARVISCHAT_COMPLETIONS_API_KEY` env var or auto-generates a random key — no longer a missing import
+- `config.py` line 17: `COMPLETIONS_API_KEY` read from `JARVISCHAT_COMPLETIONS_API_KEY` env var or auto-generates
-- `config.py` line 13: `OLLAMA_BASE` is legacy/unused — all endpoints now use `LLAMA_SERVER_BASE`
+- `config.py` line 13: `OLLAMA_BASE` is legacy/unused — all endpoints use `LLAMA_SERVER_BASE`
 
 ### Key flows
 
-1. **`/api/chat`** → `process_remember_command()` intercepts "remember that..." / "forget about..." first → else `build_system_prompt()` (profile + FTS5 memory + Qdrant RAG + preset + skills) → stream from llama-server with `logprobs: true` → if perplexity > 15.0 OR `REFUSAL_PATTERNS` match, re-query with SearXNG results
+1. **`/api/chat`** → `process_remember_command()` intercepts "remember that..." / "forget about..." first → optional `upload_context_id` fetches document text from SQLite → `build_system_prompt()` (profile + FTS5 memory + Qdrant RAG + preset + skills + uploaded doc) → stream from llama-server with `logprobs: true` → if perplexity > 15.0 OR `REFUSAL_PATTERNS` match, re-query with SearXNG results
-2. **`/api/search`** → bypasses perplexity/refusal, queries SearXNG directly → summarizes via llama-server (no raw results leaked in SSE)
+2. **`/api/search`** → bypasses perplexity/refusal, queries SearXNG directly → summarizes via llama-server
 3. **`/v1/chat/completions`** → OpenAI-compatible for Continue.dev/IDE integration; FIM requests proxied without persistence
+4. **`/api/upload`** → multipart file upload, PDF/text extraction, `mode=(context|ingest|both)`, stores SQLite context (1hr expiry) + Qdrant upsert
+5. **`/api/ingest`** → Bearer token auth, programmatic RAG ingest (terminal hook, external tools)
 
 ### Perplexity / auto-search
 
-The upstream request includes `"logprobs": true`. `parse_llama_stream_chunk()` extracts per-token logprobs from each chunk's `choices[0].logprobs.content[].logprob`. The `all_logprobs` list is populated during streaming, so `calculate_perplexity()` and `is_uncertain()` work correctly — auto-search on high perplexity is no longer dead code.
+The upstream request includes `"logprobs": true`. `parse_llama_stream_chunk()` extracts per-token logprobs from each chunk's `choices[0].logprobs.content[].logprob`. The `all_logprobs` list is populated during streaming, so `calculate_perplexity()` and `is_uncertain()` work correctly.
 
 ### Auth / lockdown
 
 - Guest session by default (`POST /api/auth/guest`), admin unlock via 4-digit PIN (`POST /api/auth/login`)
 - Admin required for PUT/DELETE/PATCH + all POST except allowlist (`/api/chat`, `/api/search`, `/api/auth/*`)
+- `/api/ingest` is exempt from session auth — self-authenticates via Bearer token
 - IP allowlist, rate limiting, origin checking, payload size limits — all enforced in `app.py` middleware
-- Origin check applies to **all** `/api/` requests (not just state-changing methods); `origin_allowed()` returns `False` when both `Origin` and `Referer` headers are absent, closing CSRF read gap
+- Origin check applies to **all** `/api/` requests; returns `False` when both `Origin` and `Referer` are absent
 - `JARVISCHAT_ADMIN_PIN` env var required on first boot (or `JARVISCHAT_ALLOW_DEFAULT_PIN=true`)
 
 ### Database
 
 - SQLite at `jarvischat.db`, auto-created by `init_db()` on startup via FastAPI `lifespan`
 - `get_db()` opens new connection per request (no pool). Close after use.
-- FTS5 virtual table `memories` for full-text search with BM25 ranking. FTS5 operator keywords (`AND`, `OR`, `NOT`, `NEAR`) are double-quoted to prevent parse errors.
+- FTS5 virtual table `memories` for full-text search with BM25 ranking.
+- `upload_context` table: auto-expiring document storage for chat context injection.
 
 ### External services
 
 | Service | Required | Port |
 |---------|----------|------|
-| llama-server (OpenAI-compat API) | Yes | 8081 (ultron) or env `LLAMA_SERVER_BASE` |
+| llama-server (ultron) | Yes | 8081 + RPC :50052 (jarvis GPU) |
 | SearXNG | No | 8888 |
-| wttr.in | No | weather shortcut bypasses SearXNG; curl UA for plain-text output |
+| wttr.in | No | weather shortcut |
 | rocm-smi | No | AMD GPU stats |
 | Qdrant | No | 6333 (ultron) — RAG vector search |
 
 ### Config quirks
 
-- Rate limits and payload caps in `config.py` — tweak for testing by monkeypatching module attributes (note: patch `security.RL_*` not `config.RL_*` since `security` imports bindings separately)
+- `BODY_LIMIT_UPLOAD_BYTES` = 20MB for `/api/upload`; other paths use smaller limits
-- `ALLOWED_SETTINGS_KEYS` in `config.py` controls which keys the UI can write via `/api/settings`
+- `SUPPORTED_UPLOAD_TYPES` includes images (png/jpeg/gif/svg/webp) + text + PDF + JSON
-- Settings table seeded with defaults (`profile_enabled`, `search_enabled`, `memory_enabled`, `skills_enabled`, `default_model`) — never overwritten by `init_db()`
+- `UPLOAD_CONTEXT_EXPIRY_HOURS` = 1 hour
-- Profile table uses singleton row `id=1`
+- Rate limits and payload caps in `config.py` — patch `security.RL_*` not `config.RL_*` for tests
-- RAG embedding requests go to `EMBED_URL` at `/api/embeddings` (separate Ollama instance)
+- RAG embedding requests go to `EMBED_URL` at `/api/embeddings` (Ollama on jarvis :11434)
 
 ### SSE Protocol
 

CLAUDE.md

@@ -24,41 +24,56 @@ sudo systemctl restart jarvischat
 
 ## Architecture
 
-Single-file FastAPI backend (`app.py`) + single-template frontend (`templates/index.html`). No build step. SQLite database auto-created at `jarvischat.db` on first run.
+Modular FastAPI app — `app.py` wires routers, middleware, and lifespan. SQLite database auto-created at `jarvischat.db` on first run. No build step, single `templates/index.html`.
 
 ### Request Flow: `/api/chat`
 
 1. User message saved to DB → conversation created if new
-2. `build_system_prompt()` assembles: profile + FTS5 memory search results + preset prompt
+2. `process_remember_command()` intercepts "remember that..." / "forget about..." first
-3. Streamed to Ollama (`/api/chat`, `stream: true`, `logprobs: true`) via SSE
+3. Optional `upload_context_id` → fetches document text from `upload_context` table, injects `[ATTACHED DOCUMENT]` into system prompt
-4. **Auto web search trigger**: if perplexity > 15.0 OR response matches `REFUSAL_PATTERNS`, re-queries Ollama with SearXNG results prepended to system prompt
+4. `build_system_prompt()` assembles: profile + FTS5 memory search + Qdrant RAG + preset + skills + uploaded doc
-5. Final response saved to DB; SSE `done` event sent with perplexity + tokens/sec
+5. Streamed to llama-server (`/v1/chat/completions`, `stream: true`, `logprobs: true`) via SSE
+6. **Auto web search trigger**: if perplexity > 15.0 OR response matches `REFUSAL_PATTERNS`, re-queries with SearXNG results
+7. Final response saved to DB; SSE `done` event sent with perplexity + tokens/sec
 
 ### Request Flow: `/api/search` (explicit search)
 
-Bypasses perplexity/refusal detection entirely — queries SearXNG directly then asks Ollama to summarize with results as system context.
+Bypasses perplexity/refusal — queries SearXNG directly then asks llama-server to summarize results.
+
+### Request Flow: `/api/upload`
+
+Multipart file upload → PDF/text extraction + chunking → optional Qdrant upsert + SQLite context storage (1hr expiry). Supports `mode=(context|ingest|both)`. Images upload as storage only — model cannot process image content.
+
+### Request Flow: `/api/ingest`
+
+Bearer-token-authenticated terminal RAG hook. Accepts raw text, chunks via `chunk_text()`, embeds via Ollama `/api/embeddings`, upserts to Qdrant.
 
 ### Memory System
 
-FTS5 virtual table (`memories`) in SQLite. `search_memories()` uses BM25 ranking. `process_remember_command()` intercepts "remember that..." / "forget about..." before the message reaches Ollama and returns a confirmation string. Topic auto-detection via keyword matching in `detect_topic()`.
+FTS5 virtual table (`memories`) in SQLite. `search_memories()` uses BM25 ranking. `process_remember_command()` intercepts "remember that..." / "forget about..." before the message reaches the model and returns a confirmation string.
 
-### Key Constants (top of `app.py`)
+### Key Constants (`config.py`)
 
-- `OLLAMA_BASE` — `http://localhost:11434`
+- `LLAMA_SERVER_BASE` — `http://192.168.50.108:8081` (ultron llama-server, RPC offloads to jarvis GPU)
 - `SEARXNG_BASE` — `http://localhost:8888`
-- `PERPLEXITY_THRESHOLD` — `15.0` (controls auto-search sensitivity)
+- `PERPLEXITY_THRESHOLD` — `15.0`
-- `DEFAULT_MODEL` — `llama3.1:latest`
+- `EMBED_URL` — `http://192.168.50.210:11434/api/embeddings` (Ollama on jarvis)
+- `VERSION` — current version string
 
 ### External Services
 
-- **Ollama** — required, must be running on port 11434
+| Service | Required | Port |
-- **SearXNG** — optional, port 8888; `GET /api/search/status` probes availability
+|---------|----------|------|
-- **wttr.in** — weather shortcut in `query_searxng()`, bypasses SearXNG for weather queries
+| **llama-server** (ultron) | Yes | 8081 + RPC :50052 (jarvis GPU) |
-- **rocm-smi** — AMD GPU stats via subprocess; gracefully degrades if not available
+| **SearXNG** | No | 8888 |
+| **wttr.in** | No | weather shortcut |
+| **rocm-smi** | No | AMD GPU stats |
+| **Qdrant** (ultron) | No | 6333 — RAG vector search |
+| **Ollama** (jarvis) | No | 11434 — embeddings only |
 
 ### Database
 
-`get_db()` opens a new connection per request (no connection pool). `init_db()` runs at startup via the FastAPI `lifespan` handler. The `profile` table uses a singleton row (`id = 1`). Default settings are seeded but never overwritten by `init_db()`.
+`get_db()` opens a new connection per request (no pool). `init_db()` runs at startup via FastAPI `lifespan`. Tables: `conversations`, `messages`, `settings`, `profile` (singleton id=1), `memories` (FTS5), `upload_context`. Default settings seeded but never overwritten.
 
 ### SSE Protocol
 
@@ -67,8 +82,8 @@ All streaming endpoints yield `data: {json}\n\n`. Key event shapes:
 - `{searching: true}` — web search triggered
 - `{search_results: N}` — N results retrieved
 - `{done: true, perplexity, tokens_per_sec, searched?}` — terminal event
-- `{error: "..."}` — error event
+- `{error: "...", error_key: "..."}` — error with incident key
 
 ### Deployment
 
-Runs as systemd service under user `jarvischat`, working directory `/opt/jarvischat`. Logs via syslog (`journalctl -u jarvischat`).
+Runs as systemd service under user `jarvischat`, working directory `/opt/jarvischat`. Logs via syslog (`journalctl -u jarvischat`). Version bumps via git tag + commit, deployed via `git pull && systemctl restart jarvischat`.

TASKS.md

@@ -45,7 +45,9 @@ No pytest tests required for this task.
 
 ---
 
-## TASK 4 — File/Document Attachment: Backend Ingest Endpoint
+## TASK 4 — File/Document Attachment: Backend Ingest Endpoint [DONE]
+
+**Status: `POST /api/upload` with mode=(context|ingest|both), PDF/text extraction, Qdrant upsert, SQLite context (1hr expiry). Committed `4a891c8` (v1.9.0).**
 
 This task implements the backend half of file/document attachment (TODO #21). The goal is dual-aspect upload: a file can be used as immediate chat context, ingested into the RAG corpus (Qdrant), or both.
 
@@ -89,7 +91,9 @@ Run full test suite after implementation. All 26 existing tests must continue to
 
 ---
 
-## TASK 5 — File/Document Attachment: UI Integration
+## TASK 5 — File/Document Attachment: UI Integration [DONE]
+
+**Status: Paperclip icon, file preview pill, gallery overlay, attachment indicators, DELETE/PATCH link/by-conversation endpoints, chat context injection. Committed `81238c0` (v1.10.0).**
 
 This task implements the frontend half of TODO #21. The UI is a single file at `templates/index.html`.
 
@@ -114,7 +118,9 @@ Run full test suite. All existing tests must continue to pass.
 
 ---
 
-## TASK 6 — Roadmap I: Terminal Command RAG Hook
+## TASK 6 — Roadmap I: Terminal Command RAG Hook [DONE]
+
+**Status: `POST /api/ingest` with Bearer token auth, `chunk_text()` shared helper, `jc-ingest.sh` script. Committed `1ac21ad` (v1.11.0).**
 
 This task implements autonomous RAG ingestion of significant terminal activity (TODO #23).