cAIc/docker.md

Docker Distribution — Architecture & Planning

Part of B3 (v1.0 gate). This document catalogs every service, volume, port, configuration, and decision needed to ship jarvisChat as a docker compose stack. It also defines extraction (setup) and back-out (uninstall) procedures so nothing is lost when reality disagrees with the plan.

1. Stack Overview

┌─────────────────────────────────────────────────────────┐
│                    docker compose stack                  │
│                                                         │
│  ┌────────────┐  ┌──────────┐  ┌────────────────────┐  │
│  │  SearXNG    │  │  Qdrant  │  │  RabbitMQ          │  │
│  │  :8888      │  │  :6333   │  │  :5672 / :15672    │  │
│  └──────┬──────┘  └────┬─────┘  └────────┬───────────┘  │
│         │              │                  │              │
│         ▼              ▼                  ▼              │
│  ┌──────────────────────────────────────────────────┐   │
│  │              jarvisChat (FastAPI)                │   │
│  │              :8080 (HTTP)                        │   │
│  │                                                  │   │
│  │  SQLite  ◄── jarvischat.db           (volume)    │   │
│  │  Uploads ◄── /app/uploads            (volume)    │   │
│  └──────────┬──────────────┬───────────────────────┘   │
│             │              │                            │
│             ▼              ▼                            │
│  ┌──────────────┐  ┌──────────────┐                    │
│  │ llama-server │  │   Ollama     │                    │
│  │ :8081        │  │ :11434       │                    │
│  │ (GPU/RPC)    │  │ (embeddings) │                    │
│  └──────────────┘  └──────────────┘                    │
└─────────────────────────────────────────────────────────┘

This compose stack defines the coordinator. A coordinator runs jC, the broker, and optional infrastructure services. Workers (headless inference nodes) do not use Docker — they install just llama-server + a Python node agent. See §12 for the worker deployment model.

Service roles

Service	Image	Role
jarvisChat	Custom Dockerfile	FastAPI app serving UI + API
SearXNG	searxng/searxng:latest	Privacy-respecting web search
Qdrant	qdrant/qdrant:latest	Vector database for RAG
RabbitMQ	rabbitmq:4-management	Message broker for AMQP cluster
llama-server	ghcr.io/ggml-org/llama.cpp:server	LLM inference (OpenAI-compat API)
Ollama	ollama/ollama:latest	Embeddings for RAG chunk vectors

Non-containerized (host-level)

Component	Reason
AMD GPU driver + ROCm	Kernel access required for GPU compute
llama.cpp RPC workers	Runs on other hosts — not on the Docker host
rocm-smi	Hardware stats — not needed for core function
psutil	Already inside the container via pip

---

2. Service Catalog

2.1 jarvisChat (FastAPI app)

Image: jarvischat:latest (built from Dockerfile)

Ports:

Container	Host	Purpose
8080	8080	HTTP API + UI

Volumes:

Container path	Type	Purpose
/app/jarvischat.db	named volume jarvischat_data	SQLite database
/app/uploads	named volume jarvischat_uploads	Uploaded files
/app/hardware_state.json	(inside volume)	Cached hardware probe

Dependencies: Wait for SearXNG, Qdrant, RabbitMQ, llama-server, Ollama before serving.

Restart: unless-stopped

Healthcheck: curl -f http://localhost:8080/

2.2 SearXNG

Image: searxng/searxng:latest

Ports:

Container	Host	Purpose
8080	8888	Search API

Volumes:

Container path	Type	Purpose
/etc/searxng	named volume searxng_config	settings.yml

Environment:

SEARXNG_BASE_URL=https://localhost:8888

Config override (/etc/searxng/settings.yml):

search:
  safe_search: 0
  autocomplete: ""
server:
  secret_key: ${SEARXNG_SECRET_KEY}
  limiter: false
  image_proxy: false
  method: GET
  port: 8080
  bind_address: "0.0.0.0"

Restart: unless-stopped

2.3 Qdrant

Image: qdrant/qdrant:latest

Ports:

Container	Host	Purpose
6333	6333	HTTP API
6334	—	gRPC (internal only)

Volumes:

Container path	Type	Purpose
/qdrant/storage	named volume qdrant_storage	Vector index data

Environment:

QDRANT__SERVICE__GRPC_PORT=6334

Restart: unless-stopped

2.4 RabbitMQ

Image: rabbitmq:4-management

Ports:

Container	Host	Purpose
5672	5672	AMQP messaging
15672	—	Management UI (internal only)

Volumes:

Container path	Type	Purpose
/var/lib/rabbitmq	named volume rabbitmq_data	Message store

Environment:

RABBITMQ_DEFAULT_USER=jarvischat
RABBITMQ_DEFAULT_PASS_FILE=/run/secrets/rabbitmq_password
RABBITMQ_DEFAULT_VHOST=/

Restart: unless-stopped

2.5 llama-server

Image: ghcr.io/ggml-org/llama.cpp:server

Ports:

Container	Host	Purpose
8081	8081	OpenAI-compat API

Volumes:

Container path	Type	Purpose
/models	bind mount ./models	Model GGUF files

Environment:

LLAMA_ARG_MODEL=/models/<model-file>
LLAMA_ARG_N_GPU_LAYERS=0              # set >0 for GPU offload
LLAMA_ARG_MAIN_GPU=0
LLAMA_ARG_CTX_SIZE=4096
LLAMA_ARG_HOST=0.0.0.0
LLAMA_ARG_PORT=8081
LLAMA_ARG_EMBEDDINGS=1
LLAMA_ARG_LOGPROBS=1
LLAMA_ARG_RPC=                         # optional: comma-separated RPC endpoints

Restart: unless-stopped

Healthcheck: curl -f http://localhost:8081/health

Notes:

• Models directory bind mount — user places .gguf files in ./models/ on the host
• RPC offload to other machines (e.g., 10.0.0.50:50052,10.0.0.51:50052)
• If no GPU, set LLAMA_ARG_N_GPU_LAYERS=0 for CPU-only
• LLAMA_ARG_EMBEDDINGS=1 required for perplexity scoring
• LLAMA_ARG_LOGPROBS=1 required for auto-search trigger

2.6 Ollama

Image: ollama/ollama:latest

Ports:

Container	Host	Purpose
11434	11434	Embeddings API

Volumes:

Container path	Type	Purpose
/root/.ollama	named volume ollama_models	Pulled model blobs

Restart: unless-stopped

Notes:

• Used exclusively for embeddings (/api/embeddings), not inference
• Typically needs a small model like all-minilm:latest or nomic-embed-text:latest
• Consider replacing Ollama with llama-server's built-in embedding if it supports the same model — would remove one container

---

3. Configuration Management

3.1 .env file (generated by setup wizard)

# --- Secrets (auto-generated, change before production) ---
JARVISCHAT_ADMIN_PIN=
JARVISCHAT_COMPLETIONS_API_KEY=
JARVISCHAT_ALLOW_DEFAULT_PIN=false
RABBITMQ_PASSWORD=
SEARXNG_SECRET_KEY=

# --- Host discovery (auto-detected by setup wizard) ---
LLAMA_SERVER_BASE=http://llama-server:8081
OLLAMA_BASE=http://ollama:11434
SEARXNG_BASE=http://searxng:8888
QDRANT_URL=http://qdrant:6333
RABBITMQ_HOST=rabbitmq
RABBITMQ_PORT=5672

# --- Performance tuning (calculated by setup wizard) ---
RAG_MAX_VECTORS=50000
RAG_EVICTION_HIGH_WATER=0.80
RAG_EVICTION_LOW_WATER=0.20
RAG_EVICTION_BATCH=1000

# --- llama-server options ---
LLAMA_MODEL=llama3.1-8b-instruct.Q4_K_M.gguf
LLAMA_N_GPU_LAYERS=0
LLAMA_RPC_ENDPOINTS=
LLAMA_CTX_SIZE=4096

# --- Ollama ---
OLLAMA_EMBED_MODEL=all-minilm:latest

# --- Network ---
JARVISCHAT_ALLOWED_CIDRS=127.0.0.0/8,::1/128,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
JARVISCHAT_TRUSTED_ORIGINS=
JARVISCHAT_TRUST_X_FORWARDED_FOR=false

3.2 Mapping of config.py → .env variable

Every config.py default that references an external service must accept a matching env var at runtime:

config.py constant	.env variable	Service
LLAMA_SERVER_BASE	LLAMA_SERVER_BASE	llama-server
OLLAMA_BASE	OLLAMA_BASE	Ollama
SEARXNG_BASE	SEARXNG_BASE	SearXNG
QDRANT_URL	QDRANT_URL	Qdrant
COMPLETIONS_API_KEY	JARVISCHAT_COMPLETIONS_API_KEY	—
ALLOWED_CIDRS_RAW	JARVISCHAT_ALLOWED_CIDRS	—
TRUST_X_FORWARDED_FOR	JARVISCHAT_TRUST_X_FORWARDED_FOR	—
TRUSTED_ORIGINS	JARVISCHAT_TRUSTED_ORIGINS	—
RAG_MAX_VECTORS	RAG_MAX_VECTORS	— (calc'd from RAM)

3.3 Secrets management

Secret	Generated by	Stored in	Mounted to
JARVISCHAT_ADMIN_PIN	User prompt	.env	jarvisChat container
JARVISCHAT_COMPLETIONS_API_KEY	Auto-generated, shown to user	.env	jarvisChat container
RABBITMQ_PASSWORD	Auto-generated	.env + Docker secret	RabbitMQ container
SEARXNG_SECRET_KEY	Auto-generated	.env	SearXNG container

Docker secrets approach: Use secrets: in compose file for RabbitMQ password (mounted as file) rather than passing via env var, since settings.yml in SearXNG and RabbitMQ config can reference file-based secrets without env-var leakage.

3.4 Dockerfile for jarvisChat

FROM python:3.13-slim-bookworm AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM python:3.13-slim-bookworm
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends \
    curl && \
    rm -rf /var/lib/apt/lists/*
COPY --from=builder /usr/local/lib/python3.13/site-packages /usr/local/lib/python3.13/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
COPY . .

EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=10s --start-period=15s --retries=3 \
    CMD curl -f http://localhost:8080/ || exit 1

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]

Multi-stage rationale: First stage compiles/bundles packages (wheels), final stage is minimal. Devs can skip builder with --target builder for live-reload with volume mount.

---

4. docker-compose.yml structure

services:
  jarvischat:
    build: .
    ports: ["8080:8080"]
    volumes:
      - jarvischat_data:/app/jarvischat.db
      - jarvischat_uploads:/app/uploads
    env_file: .env
    depends_on:
      searxng: { condition: service_started }
      qdrant: { condition: service_started }
      rabbitmq: { condition: service_healthy }
      llama-server: { condition: service_healthy }
      ollama: { condition: service_started }
    restart: unless-stopped

  searxng:
    image: searxng/searxng:latest
    ports: ["8888:8080"]
    volumes:
      - ./searxng/settings.yml:/etc/searxng/settings.yml:ro
      - searxng_config:/etc/searxng
    env_file: .env
    restart: unless-stopped

  qdrant:
    image: qdrant/qdrant:latest
    ports: ["6333:6333"]
    volumes:
      - qdrant_storage:/qdrant/storage
    restart: unless-stopped

  rabbitmq:
    image: rabbitmq:4-management
    ports: ["5672:5672"]
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq
    env_file: .env
    secrets:
      - rabbitmq_password
    healthcheck:
      test: ["CMD", "rabbitmq-diagnostics", "check_port_connectivity"]
      interval: 15s
      timeout: 5s
      retries: 3
    restart: unless-stopped

  llama-server:
    image: ghcr.io/ggml-org/llama.cpp:server
    ports: ["8081:8081"]
    volumes:
      - ./models:/models:ro
    env_file: .env
    command: >
      --model /models/${LLAMA_MODEL}
      --host 0.0.0.0 --port 8081
      --ctx-size ${LLAMA_CTX_SIZE:-4096}
      --n-gpu-layers ${LLAMA_N_GPU_LAYERS:-0}
      --embeddings
      --logprobs
      ${LLAMA_RPC_ENDPOINTS:+--rpc ${LLAMA_RPC_ENDPOINTS}}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8081/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

  ollama:
    image: ollama/ollama:latest
    ports: ["11434:11434"]
    volumes:
      - ollama_models:/root/.ollama
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 30s
      timeout: 10s
      retries: 3
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

volumes:
  jarvischat_data:
  jarvischat_uploads:
  searxng_config:
  qdrant_storage:
  rabbitmq_data:
  ollama_models:

secrets:
  rabbitmq_password:
    file: ./secrets/rabbitmq_password.txt

Notes:

• GPU reservations use resources.reservations.devices — this is compose v3.8+. For AMD GPUs, replace driver: nvidia with driver: amd (experimental Docker support). For hosts without GPU, omit the deploy block entirely.
• The deploy block only applies when deployed as a swarm stack. For docker compose, GPU access may need --gpus all or device_requests in config. Verify compatibility.
• SearXNG config file (settings.yml) is bind-mounted read-only from the host repo clone — the setup wizard should generate this file.

---

5. Networking

5.1 Internal communication (compose network)

From	To	Port	Protocol
jarvisChat	llama-server	8081	HTTP
jarvisChat	Ollama	11434	HTTP
jarvisChat	SearXNG	8080	HTTP
jarvisChat	Qdrant	6333	HTTP
jarvisChat	RabbitMQ	5672	AMQP
RabbitMQ	(cluster peers)	4369	EPMD
RabbitMQ	(cluster peers)	25672	Inter-node

5.2 Exposed ports (host-facing)

Port	Service	Should expose?	Notes
8080	jarvisChat	✅ Required	UI + API
8888	SearXNG	Optional	Only if user wants standalone search
6333	Qdrant	Optional	Only for external tooling
5672	RabbitMQ	Optional	Only for remote AMQP clients
15672	RabbitMQ mgmt	❌ Internal	Healthcheck only
8081	llama-server	Optional	Only for external tooling
11434	Ollama	Optional	Only for external tooling

Design decision: By default, only port 8080 (jarvisChat) is published. All other services remain on the internal compose network. Advanced users can opt-in by uncommenting ports: blocks.

5.3 Reverse proxy consideration

For production, a reverse proxy (Caddy, nginx, Traefik) should sit in front:

# Optional — compose profile: "proxy"
caddy:
  image: caddy:latest
  ports: ["80:80", "443:443"]
  volumes:
    - ./Caddyfile:/etc/caddy/Caddyfile:ro
    - caddy_data:/data

This is out of scope for v1.0 but documented for future.

---

6. Setup Wizard (Extraction)

setup.sh — idempotent, interactive, runs on first boot.

Flow

1. CHECK: Is .env present?
   ├── YES → skip to step 7 (or ask to regenerate)
   └── NO  → continue

2. INTRO: Print banner, explain what's about to happen

3. PROBE: Run hardware assessment
   ├── psutil → RAM total, CPU count
   ├── rocm-smi → VRAM (optional, best-effort)
   └── nvidia-smi → VRAM (optional, best-effort)

4. NETWORK: Ask for
   ├── Hostname / LAN IP for this machine
   ├── Admin PIN (4 digits, or accept auto-generated)
   └── (Optional) RPC endpoints for GPU offload

5. CALCULATE:
   ├── RAG_MAX_VECTORS = max(1000, int(available_ram_gb * 100_000))
   ├── LLAMA_N_GPU_LAYERS = 0 (CPU default; offer GPU detection)
   ├── LLAMA_MODEL = default gguf filename
   └── RABBITMQ_PASSWORD = openssl rand -hex 20

6. GENERATE:
   ├── .env file from template
   ├── ./secrets/rabbitmq_password.txt
   ├── ./searxng/settings.yml (with generated secret_key)
   └── ./models/README.txt (instructions for placing .gguf)

7. VERIFY:
   ├── docker and docker compose plugin installed
   ├── docker compose version >= 2.x
   ├── SUCCESS → "Run: docker compose up -d"
   └── FAILURE  → show diagnostics and links

8. EXTRACT model:
   ├── Prompt for download URL or local path
   ├── Offer to pull from HuggingFace if huggingface-cli available
   └── Guides user to place file in ./models/

What setup.sh creates on disk

./docker-deploy/
├── .env                        # All env vars (SECRET — add to .gitignore)
├── docker-compose.yml          # Compose stack definition
├── Dockerfile                  # jarvisChat image build
├── secrets/
│   └── rabbitmq_password.txt   # RabbitMQ password file
├── searxng/
│   └── settings.yml            # SearXNG config with generated secret_key
├── models/
│   ├── README.txt              # Instructions for model placement
│   └── <model>.gguf            # (user-provided)
└── setup.log                   # Wizard run log

Idempotency

Re-running setup.sh:

• With .env present: ask "Regenerate? This will overwrite existing config."
• Without .env: fresh run
• Never overwrites ./models/*.gguf files
• Never touches running containers — only modifies files on disk

---

7. Back-out Procedure (Uninstall)

teardown.sh — returns the host system to its pre-install state.

What gets removed

Item	Removal method
Docker containers	docker compose down -v
Docker images	docker rmi jarvischat:latest (ask about other images)
Docker volumes	docker volume rm jarvischat_data ... (prompt first)
Network jarvischat_default	Removed with compose
.env file	rm .env
secrets/ directory	rm -rf secrets/
searxng/ directory	rm -rf searxng/
setup.log	rm setup.log
hardware_state.json	rm hardware_state.json

What is preserved (by default)

Item	Reason
./models/*.gguf	User data — prompt for deletion
jarvischat.db (in volume)	Prompt: "Keep database snapshot?"
./uploads/ (in volume)	Prompt: "Keep uploaded files?"
Docker Engine itself	Not installed by this project — leave it

Script flow

1. CHECK: docker compose file exists?
   ├── NO  → warn, continue
   └── YES → docker compose down -v

2. CHECK: .env exists?
   ├── NO  → skip
   └── YES → ask: "Remove .env?" (default no)

3. ASK: "Remove secrets/ and searxng/ directories?" (default no)

4. ASK: "Remove Docker images? (y/N)" (default no)
   ├── Y → docker rmi jarvischat:latest
   ├── Y → docker image ls | grep searxng/qdrant/rabbitmq → prompt per image
   └── N → skip

5. ASK: "Keep database volume snapshot? (Y/n)" (default yes)
   ├── N → docker volume rm jarvischat_data
   └── Y → leave volume (can be reattached later)

6. ASK: "Remove model files from ./models/? (y/N)" (default no)

7. CLEANUP generated artifacts:
   ├── rm -f setup.log
   ├── rm -f hardware_state.json
   └── rm -f docker-compose.yml

8. SUMMARY:
   ├── "Docker stack removed"
   ├── "Persistent data preserved at: <paths>"
   └── "Models kept at: ./models/"

Partial rollback

If the setup wizard fails mid-way, a partial rollback is better than leaving detritus:

Failure point	Clean up
After .env, before compose	rm .env; rm -rf secrets/ searxng/
After compose, before first up	rm docker-compose.yml; rm -rf *
After up but before healthcheck	docker compose down -v; rm -rf ./*

setup.sh should trap EXIT on failure and prompt: "Clean up partial install? [y/N]"

---

8. Open Decisions

Decision	Options	Priority
Ollama vs llama-server embeddings	Both work. Keep both for now — remove Ollama if llama-server handles embeddings. Reduce containers = simpler.	Medium
GPU support in compose	NVIDIA: well-supported. AMD: requires --device=/dev/kfd --device=/dev/dri and ROCm image. Document both.	High
RabbitMQ clustering vs single node	Single node in v1.0. Clustering docs for multi-host later.	Low
SearXNG config management	Bind-mount a generated settings.yml, or let container create default and post-process. Bind-mount is cleaner.	Medium
Reverse proxy	Caddy is simplest for auto-HTTPS. Out of scope for v1.0 but design for it.	Low
Healthcheck strategy	depends_on with condition: service_healthy is the safest approach but increases startup time. Acceptable.	Medium
Database migration	SQLite file in volume — no migration needed for v1.0 format. If schema changes post-v1.0, need a migration container.	Low
Linux vs macOS vs Windows	Linux-primary. macOS may work with changes (no rocm-smi). Windows via WSL2 only.	Low
LLM model download	HuggingFace CLI integration in setup.sh, or manual download. Manual is simpler.	Low
Dockerfile optimization	Pin pip hashes, use --no-cache-dir, consider slim vs alpine. Alpine has musl compatibility issues with psutil. Stay with slim.	Medium

9. Worker Node Deployment Model

The Docker stack above defines the coordinator only. Workers (headless inference nodes) have a radically lighter footprint.

9.1 What a worker runs

Worker machine (e.g. jarvis, Corsair)
┌────────────────────────────────────┐
│  llama-server                      │
│  (single binary, no build needed)  │
│                                    │
│  node_agent.py                     │
│  (Python script, aio-pika client)  │
│    ─ connects to coordinator's RMQ │
│    ─ publishes heartbeat + reg     │
│    ─ consumes model_swap commands  │
│                                    │
│  ROCm or CUDA runtime (if GPU)     │
└────────────────────────────────────┘

9.2 What a worker does NOT run

Service	Reason
RabbitMQ server	Connects as AMQP client only (aio-pika)
FastAPI / uvicorn / jC	No HTTP API, no UI, no database
SQLite	No persistent state of its own
SearXNG	No web search needs
Qdrant	No local vector store
Ollama	Uses coordinator's embedding endpoint
Docker	Everything runs as bare binaries
Python venv with full jC deps	Only needs aio-pika + httpx

9.3 Worker setup

# Install llama-server binary
wget https://github.com/ggml-org/llama.cpp/releases/.../llama-server
chmod +x llama-server

# Install node agent deps
pip install aio-pika httpx

# Create node agent script (from repo: tools/node_agent.py)
# Configure COORDINATOR_AMQP_URL in environment

9.4 Multiple workers

Each worker registers independently with the coordinator's RabbitMQ. The coordinator tracks all registered workers via CLUSTER_NODES and routes inference requests to the best-matching node based on classification and availability.

9.5 RabbitMQ and workers — architecture note

Workers connect to RabbitMQ as standard AMQP TCP clients — no broker software required. The AMQP-0-9-1 protocol has always been client-server (since 2006), and libraries like aio-pika, pika, amqplib, php-amqplib, etc. connect over a single persistent socket. This is distinct from a service-mesh design where every node runs the same software stack and role is determined by config.

Broker-mediated model (this project):
  Coordinator runs  RabbitMQ broker  ←── Workers connect as AMQP clients

Service-mesh model (alternative):
  Every node runs    RabbitMQ broker  ←── Nodes cluster together, all autonomous

The broker-mediated model is the preferred architecture for this project because workers are intentionally heterogeneous (different GPUs, different models, ARM vs x86) and should not be burdened with infrastructure services.

10. Checklist (pre-v1.0 gate)

• Dockerfile written and builds clean
• docker-compose.yml boots all containers
• jarvisChat container reaches all services (env vars resolve correctly)
• SearXNG settings.yml generated correctly by setup.sh
• RabbitMQ password secret mounted correctly
• GPU (NVIDIA) passes through to llama-server container
• GPU (AMD) passes through to llama-server container (or documented limitation)
• .env.example checked in (no real secrets)
• setup.sh written, idempotent, tested on clean Debian
• teardown.sh written, tested, doesn't delete models without confirmation
• docker compose up -d works without any manual steps beyond setup.sh
• docker compose down -v followed by setup.sh && docker compose up -d = fresh stack
• Healthchecks prevent serving before dependencies are ready
• v1.0 release tag created

---

11. Files to create for B3

docker.md             ← this file (planning doc)
Dockerfile            ← jarvisChat image
docker-compose.yml    ← full stack
.env.example          ← template without secrets
setup.sh              ← extraction wizard
teardown.sh           ← back-out utility
searxng/
  settings.yml        ← SearXNG config (generated by setup.sh)
secrets/
  rabbitmq_password.txt  ← generated by setup.sh
models/
  README.txt          ← instructions for placing .gguf