How LuraDB works

Domains and engines

applies to v0.5.1

A domain is the unit you create, protect and back up. Inside it, each engine keeps the shape of data it is good at: key-value, JSON documents and slim relational tables (LuraSQL). A relational row can point at a key or a document in the same domain.

A domain is a name that each engine knows on its own

Every engine has its own list of domains, with its own endpoint to create one, and a domain exists in an engine only after you created it there. What ties them together is the name: orders in the key-value engine, orders in the JSON engine and orders in the relational engine belong together by that name, and by nothing else.

Engine	Keeps	Create a domain	Typical use
Key-value	Bytes under a key, with TTL, null state and a change stream	`POST /store-api/domains`	Sessions, tokens, flags, counters
JSON	Documents with a version, secondary indexes and search	`POST /store-api/json/domains`	Payloads, catalogs, anything with loose ends
LuraSQL	Tables with typed columns, indexes, views and joins	`POST /store-api/rel/domains`	Fixed fields, relations, reports

The three requests for the same name are three separate calls:

curl -s -X POST http://127.0.0.1:3000/store-api/domains \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"name": "orders"}'
curl -s -X POST http://127.0.0.1:3000/store-api/json/domains \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"name": "orders"}'
curl -s -X POST http://127.0.0.1:3000/store-api/rel/domains \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"name": "orders"}'

{ "name": "orders", "created_at": 1793741760, "require_conditional_writes": false }
{ "name": "orders", "created_at": 1793741760, "state": "active", "require_conditional_writes": false }
{ "name": "orders", "created_at": 1793741760, "state": "active" }

Every start creates a domain called default in each engine if it is missing; for the key-value engine the name is set by domains.default_domain. A name uses letters, digits, _ and -. The JSON and relational engines accept up to 50 characters; for key-value domains the limit is domains.max_name_length, 50 by default. A name that breaks the rule answers 400.

Deleting a domain removes it in that engine only. DELETE /store-api/domains/orders answers 202 Accepted: the domain is gone for clients at once, reads of its keys answer 410 Gone while the data is purged in the background. The JSON and relational domains of the same name stay.

The JSON and the relational engine can be switched off, see json.enabled and rel.enabled; the entries say what clients see then. The key-value engine always runs.

All domains of one name live on the same data shard, in every engine. That is what allows rows to link to keys and documents, and it is why one busy name gives one core the writes, see Multicore.

Key-value keeps bytes under a key

A key-value domain maps a key to a value. The key is valid UTF-8 of at most 256 bytes (domains.max_user_key_length); the value is any bytes, and LuraDB does not interpret them. A key can carry a TTL, after which it disappears, and it can be set to an explicit null state that keeps it listed but empty. Keys are listed and counted by prefix, deleted by prefix in one atomic step, and watched over a server-sent-events stream that delivers each change. Every version of a key has an ETag for conditional writes. The page Key-value goes through each call.

JSON keeps documents with a version

A JSON domain stores documents. A document has a key, either one you choose or a generated UUID, and a version that counts its writes; both come back in the reserved fields _key and _version next to the fields of the document. Search filters on fields that have an index, with equality or ranges, and pages through the result. Documents are loaded in bulk from NDJSON and exported as NDJSON. See JSON documents.

LuraSQL keeps rows in tables

A relational domain holds tables with typed columns (INTEGER, REAL, TEXT, BOOLEAN, TIMESTAMP), primary keys, AUTOINCREMENT, indexes, views and left joins. One request carries one statement, with ? placeholders bound from a params array. Besides /sql, tables have row endpoints and a CSV/TSV import. LuraSQL is deliberately small: see LuraSQL for what it supports and where it stops.

A row can point at a key or a document

A column of type KVREF holds the key of a key-value entry, a column of type JSONREF the key of a JSON document. The target is always in the domain of the same name as the relational domain, in the other engine. A table in the relational domain orders can point at keys of the key-value domain orders and at documents of the JSON domain orders, and at nothing else.

curl -s -X PUT http://127.0.0.1:3000/store-api/kv/orders/keys/label-1001 \
  -H "Authorization: Bearer $KEY" -d 'LISBON-0042'
curl -s -X PUT http://127.0.0.1:3000/store-api/json/orders/documents/o-1001 \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"customer": "Jane Doe", "status": "open", "total": 42.5}'
curl -s -X POST http://127.0.0.1:3000/store-api/rel/orders/sql \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"sql": "CREATE TABLE shipments (id INTEGER PRIMARY KEY AUTOINCREMENT, order_doc JSONREF, label KVREF, carrier TEXT)"}'
curl -s -X POST http://127.0.0.1:3000/store-api/rel/orders/sql \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"sql": "INSERT INTO shipments (order_doc, label, carrier) VALUES (?, ?, ?)", "params": ["o-1001", "label-1001", "DHL"]}'

A plain SELECT returns the keys. With expand, the server looks the targets up and returns them next to the rows:

curl -s -X POST http://127.0.0.1:3000/store-api/rel/orders/sql \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"sql": "SELECT id, order_doc, label, carrier FROM shipments", "expand": ["*"]}'

{
  "columns": [
    { "name": "id", "type": "INTEGER" },
    { "name": "order_doc", "type": "JSONREF" },
    { "name": "label", "type": "KVREF" },
    { "name": "carrier", "type": "TEXT" }
  ],
  "expanded": {
    "label": [{ "encoding": "utf8", "exists": true, "value": "LISBON-0042" }],
    "order_doc": [{ "document": { "customer": "Jane Doe", "status": "open", "total": 42.5 }, "exists": true }]
  },
  "limit_applied": false,
  "row_count": 1,
  "rows": [[1, "o-1001", "label-1001", "DHL"]]
}

A key whose value is not valid UTF-8 comes back base64-encoded, with "encoding": "base64". A key in the null state resolves to exists: true with value: null.

The rules for a link:

A link is checked when it is written. Inserting or updating a row with a target that does not exist is refused with 409 Conflict: cross-engine link target 'o-9999' of column 'order_doc' is missing in json. The check reads the target at that moment; keys and documents can be deleted later, and the link then dangles.
A dangling link resolves to a missing target, not to an error. An expand of a target that is gone returns "exists": false.
When the whole target domain is gone, the cells read NULL. After DELETE /store-api/domains/orders, the same SELECT returns null for label in the rows and in expanded. A background task then sets the cells to NULL in the stored rows for good, up to 100 cells per column every ten seconds. While an engine is switched off, its links read NULL as well, but the stored cells stay and come back with the engine.
A link needs read access to its target. Reading a link cell, and writing one, requires read permission on the key-value or JSON domain it points to. Without it, the cell reads NULL, and a write that sets the link is refused with 403.
Backups carry the links. After a restore they resolve when the target domain is restored too or already exists; otherwise they read as missing and are cleared, as after deleting that domain. Backup and restore has the details.

Rights are granted per engine and domain

A user is a name with an API key and a set of permissions. A permission names an engine (kv, which is the default, json or rel), a domain and a level. The levels build on each other: read allows reading, write also writing, ddl also the DDL statements of LuraSQL: creating, changing and dropping tables, views and indexes. The key of the admin account passes every check.

curl -s -X POST http://127.0.0.1:3000/store-api/auth/users/reporter/permissions \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"domain": "sessions", "access": "read"}'
curl -s -X POST http://127.0.0.1:3000/store-api/auth/users/reporter/permissions \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"domain": "orders", "access": "write", "store_type": "json"}'

With the key of reporter, the key-value domain sessions is readable and nothing else in that engine:

Request	Answer
`GET /store-api/kv/sessions/keys/s-4f2a`	`200`
`PUT /store-api/kv/sessions/keys/s-4f2a`	`403 Forbidden`
`GET /store-api/kv/default/keys/hello`	`403 Forbidden`
`POST /store-api/json/orders/documents`	`201 Created`
`POST /store-api/rel/orders/sql` with a `SELECT`	`403 Forbidden`: no permission on the relational domain
`GET /store-api/domains`	`200`, listing only `sessions`

A permission on the JSON domain orders says nothing about the key-value or relational domain of that name. In /sql, the statement decides the level it needs: a SELECT needs read, an INSERT, UPDATE or DELETE needs write, and CREATE, ALTER or DROP needs ddl. Creating and deleting domains, managing users and the global metrics and configuration endpoints are for the admin account only. Security describes users, key rotation and the other ways to authenticate.