rustybeds/wiki/08-template-system.md

# Template System

## What Is a Template?

A BEDS template is a TOML configuration file that defines a data collection — its schema, indexes, behavioural flags, access controls, and lifecycle policy. The template is the contract between the DBA who owns the data and the framework that serves it.

Templates live in `templates/`. Filename convention: `{wbid}{CollectionName}_{schema}.toml`

```
templates/
├── example_rec.toml       ← canonical self-documenting REC template
└── mst_logger_rec.toml    ← logger collection template
```

## Runtime Template Registry (Implemented)

At IPL, BEDS now loads REC templates from `templates/` into a typed runtime registry and validates them before service connectivity checks.

Current runtime validation enforces:

- Any field listed in `protected_fields` must exist in `[fields]`.
- Any field listed in `index_fields` must exist in `[fields]`.
- Any key in `[single_field_indexes]`, `[unique_indexes]`, and `[ttl_indexes]` must exist in `[fields]`.
- Every field used in `[compound_indexes]` and `partial_indexes` qualifiers must exist in `[fields]`.
- Every field used in `regex_pattern_indexing` and `[cache_map]` must exist in `[fields]`.
- Every `compound_indexes` index name must be present in `index_name_list`.

In `production`, template registry validation failures are fatal during IPL. In non-production environments, failures are warning-only to preserve local POC workflows.

## Two Template Types

| Type | Schema | Database | Use Case |
|---|---|---|---|
| REC | `rec` | MongoDB | Document store — logs, events, user profiles, audit records |
| REL | `rel` | MariaDB | Relational store — anything needing SQL joins or transactions |

## The TLA Convention

Every collection has a **three-letter abbreviation (TLA)** declared in the template as `extension`. The TLA is:

1. Appended to the collection name: `msUsers` → `msUsers_usr`... wait, no. The collection name is `msUsers`. The TLA `_usr` is appended to **every field name** in the collection.
2. Applied to every field: `email_usr`, `status_usr`, `created_usr`

```toml
extension = "_usr"   # the TLA for this collection

[fields]
email_usr  = "string"
status_usr = "string"
created    = "integer"   # system fields like created don't carry the TLA
```

**Why TLA?** In a complex query that joins or aggregates across multiple collections, field names collide. `status` from users and `status` from orders are the same name. `status_usr` and `status_ord` never collide. In log output, `status_usr=active` immediately tells you which collection the field came from without having to trace the query.

This convention was enforced in the PHP Namaste framework from the beginning. BEDS validates TLA compliance at template load time — templates that violate the naming convention are rejected.

## The `wbid` Convention

The `wbid` (white-box identifier, 2 characters) is prepended to every collection name:

```
wbid = "ms"   + CollectionName "Users"   = collection "msUsers"
wbid = "ms"   + CollectionName "Logs"    = collection "msLogs"
```

The `wbid` namespaces your data within a shared MongoDB instance. If you ever run multiple BEDS deployments against the same MongoDB, different `wbid` values keep their collections from colliding.

## REC Template Anatomy

See `templates/example_rec.toml` for the full self-documenting example. Key sections:

### Identity

```toml
version        = 1                  # increment when schema changes
service        = "app_server"       # which node owns this collection
schema         = "rec"              # always "rec" for MongoDB
template_class = "Users"            # human-readable class name
collection     = "msUsers"          # MongoDB collection name
extension      = "_usr"             # TLA — appended to all field names
wh_template    = ""                 # warehousing destination template (empty = none)
```

### Behavioural Flags

```toml
closed_class    = true      # internal only vs. externally accessible
hard_deletes    = false     # soft delete (status=inactive) vs. permanent removal
updates_enabled = true      # can records be updated after insert?
auditing        = "full"    # disabled | destructive | nondestructive | full
journaling      = true      # record journal entries for destructive ops?
record_history  = false     # maintain full version history per record?
record_locking  = false     # optimistic locking for concurrent writes?
query_timers    = true      # record query execution times as metric events?
primary_key     = "token"   # "token" (BEDS GUID) or "mongo_id" (native _id)
tokens          = true      # generate BEDS GUID token for every record?
cache_ttl       = 300       # seconds in in-process cache (0 = disabled)
is_internal     = false     # exclude from public REST API catalog?
```

### Field Types

```toml
[fields]
_id        = "object"    # MongoDB native ID — never returned to clients
db_token   = "string"    # BEDS GUID — the externally-exposed primary key
status     = "string"
created    = "integer"
accessed   = "integer"
email_usr  = "string"
score_usr  = "double"
count_usr  = "integer"
flag_usr   = "boolean"
tags_usr   = "array"
meta_usr   = "object"
```

Valid types: `string`, `integer`, `double`, `boolean`, `object`, `array`, `date`

### The `db_token` Field

Every BEDS record has a `db_token` field — a BEDS-generated GUID that serves as the externally-exposed primary key. The MongoDB native `_id` is never returned to clients.

This is intentional security-through-obscurity. Every script-kiddie who finds an exposed API searches for `id` in the response payload. `db_token` is non-intuitive. It is also structurally meaningful: it is the BEDS-controlled identifier, distinct from the database's internal identifier, which means BEDS controls its format, generation, and uniqueness guarantees.

### Protected Fields

Fields that clients cannot modify:

```toml
protected_fields = ["_id", "db_token", "event_guid", "created", "accessed"]
```

Attempts to update protected fields are silently dropped.

### Index Declarations

BEDS validates all incoming queries against declared indexes at submission time. A query that cannot be satisfied by a declared index is rejected before execution — no full collection scans in production.

```toml
# which fields participate in any index
index_fields = ["db_token", "status", "created", "email_usr"]

# single-field indexes
[single_field_indexes]
db_token  = 1
status    = -1
created   = -1
email_usr = 1

# compound indexes (named — must appear in index_name_list)
index_name_list = ["cIdx1Usr"]

[compound_indexes]
cIdx1Usr = [["status", 1], ["created", -1]]

# unique constraints
[unique_indexes]
db_token  = 1
email_usr = 1

# TTL — automatic record expiry
[ttl_indexes]
# accessed = 86400   # expire records not accessed in 24 hours
```

### Cache Map

The cache map controls field name translation between internal schema names and external client-facing names. Schema column names are never exposed to clients.

```toml
[cache_map]
db_token  = "id"
status    = "status"
created   = "createdDate"
email_usr = "email"
```

A client sees `{ "id": "...", "email": "..." }`. The internal `db_token` and `email_usr` names never appear in API responses.

### Warehousing

```toml
[warehouse]
supported  = true
automated  = true
interval   = "M"    # D=daily, M=monthly, Q=quarterly, Y=yearly
delete     = "H"    # H=hard delete source after warehousing, S=soft delete

[warehouse.qualifier]
created = { operand = "null", operator = "lt", value = "" }   # caller supplies cutoff date
status  = { operand = "null", operator = "eq", value = "active" }
logical_op = "and"
```

## The Logger Template

`templates/mst_logger_rec.toml` is the canonical example of an internal system collection. Key differences from a standard REC template:

- `updates_enabled = false` — log records are immutable
- `auditing = "disabled"` — never audit the logger (infinite recursion)
- `journaling = false` — never journal the logger (same reason)
- `cache_ttl = 0` — no caching — log data is never stale-read
- `is_internal = true` — excluded from the public REST API catalog
- `hard_deletes = true` — log pruning permanently removes records
- TTL index on `created` — automatic log expiry (30 days default)

These choices are documented in the template file with explanations. When in doubt about why a flag is set a certain way on the logger template, the template is the authoritative source.