Skip to content
v0.5.1GitHub

Using

JSON documents

applies to v0.5.1

The JSON engine stores documents with a key and a version, finds them through secondary indexes on fields you choose, and loads and exports whole domains as NDJSON. JSON domains are separate from key-value and relational domains, even with the same name. The examples use a domain orders with three indexes, created first so that every document is indexed as it is written; Search needs an index explains them:

Terminal window
curl -s -X POST http://127.0.0.1:3000/store-api/json/domains \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"name": "orders"}'
curl -s -X POST http://127.0.0.1:3000/store-api/json/orders/indexes \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"field": "status", "type": "string"}'
curl -s -X POST http://127.0.0.1:3000/store-api/json/orders/indexes \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"field": "total", "type": "number"}'
curl -s -X POST http://127.0.0.1:3000/store-api/json/orders/indexes \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"field": "shipping.city", "type": "string"}'
{ "name": "orders", "created_at": 1793745837, "state": "active", "require_conditional_writes": false }
{ "field": "status", "type": "string", "created_at": 1793745837 }
{ "field": "total", "type": "number", "created_at": 1793745837 }
{ "field": "shipping.city", "type": "string", "created_at": 1793745837 }

A document is an object with a key and a version

Section titled “A document is an object with a key and a version”

A document is any JSON value you send; in practice an object. LuraDB returns it with two extra fields, _key and _version. _version counts the writes of the document: 1 on creation, plus one on every replacement. The names _key, _version and _content are reserved at the top level of a document; a document that uses them is refused with 400.

POST …/documents creates a document under a generated UUID and answers 201. PUT …/documents/{key} creates or replaces the document with the key you choose: 201 for a new document, 200 for a replacement, which replaces the whole document, not single fields:

Terminal window
curl -s -X POST http://127.0.0.1:3000/store-api/json/orders/documents \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"customer": "Jane Doe", "status": "open", "total": 42.5, "shipping": {"city": "Lisbon"}}'
curl -s -X PUT http://127.0.0.1:3000/store-api/json/orders/documents/o-1001 \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"customer": "Omar Haddad", "status": "open", "total": 19, "tags": ["gift", "express"], "shipping": {"city": "Tunis"}}'
curl -s -X PUT http://127.0.0.1:3000/store-api/json/orders/documents/o-1001 \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"customer": "Omar Haddad", "status": "shipped", "total": 19}'
curl -s -X PUT http://127.0.0.1:3000/store-api/json/orders/documents/o-1002 \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"customer": "Lena Fischer", "status": "shipped", "total": 120, "shipping": {"city": "Lisbon"}}'
{ "_key": "09c3cf09-94a1-46b6-8034-1bf376604a86", "_version": 1, "customer": "Jane Doe", "shipping": { "city": "Lisbon" }, "status": "open", "total": 42.5 }
{ "_key": "o-1001", "_version": 1, "customer": "Omar Haddad", "shipping": { "city": "Tunis" }, "status": "open", "tags": ["gift", "express"], "total": 19 }
{ "_key": "o-1001", "_version": 2, "customer": "Omar Haddad", "status": "shipped", "total": 19 }
{ "_key": "o-1002", "_version": 1, "customer": "Lena Fischer", "shipping": { "city": "Lisbon" }, "status": "shipped", "total": 120 }

The second put replaced the document: tags and shipping are gone. There is no partial update; read the document, change it and put it back, with If-Match if other writers exist. A key is 1 to 235 characters of a-z, A-Z, 0-9, ., _ and - with the default settings, see Limits and defaults. A document is limited to json.lsm.max_value_size bytes. A value that is not an object is stored and comes back wrapped in _content.

GET reads a document and returns its ETag; DELETE removes it and answers 204, or 404 if there is none:

Terminal window
curl -s -X PUT http://127.0.0.1:3000/store-api/json/orders/documents/o-3002 \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '[1, 2, 3]'
curl -si http://127.0.0.1:3000/store-api/json/orders/documents/o-3002 -H "Authorization: Bearer $KEY"
curl -si -X DELETE http://127.0.0.1:3000/store-api/json/orders/documents/o-3002 -H "Authorization: Bearer $KEY"
curl -si -X DELETE http://127.0.0.1:3000/store-api/json/orders/documents/o-3002 -H "Authorization: Bearer $KEY"
{"_content":[1,2,3],"_key":"o-3002","_version":1}


HTTP/1.1 200 OK
content-type: application/json
etag: "d4c84f5274e42a16-1"


{"_content":[1,2,3],"_key":"o-3002","_version":1}


HTTP/1.1 204 No Content


HTTP/1.1 404 Not Found
content-type: text/plain; charset=utf-8


document 'o-3002' not found in domain 'orders'

The ETag works with If-Match and If-None-Match: * on PUT and DELETE, see Consistency and conditional writes. Errors in the body of a request are 400 for a bad key or a reserved field, 415 without Content-Type: application/json, and 413 over the size limit.

Search needs an index

Section titled “Search needs an index”

A search filters on indexed fields only; there is no scan of all documents. An index names a field and the type of its values, string, number or boolean. A nested field is written with dots, shipping.city, in up to 128 characters of letters, digits, ., _ and -. GET …/indexes lists the indexes of a domain:

Terminal window
curl -s http://127.0.0.1:3000/store-api/json/orders/indexes -H "Authorization: Bearer $KEY"
[{ "field": "status", "type": "string", "created_at": 1793745837 }, { "field": "total", "type": "number", "created_at": 1793745837 }, { "field": "shipping.city", "type": "string", "created_at": 1793745837 }]

The index follows every write: replacing a document moves its entries, deleting it removes them. A document whose value in the field is missing, null, an array, an object or of another type than the index has no entry and is never found through that index. An index on tags finds no document whose tags is an array such as ["gift", "express"], as the first version of o-1001 had. A second index on a field answers 409; DELETE …/indexes/{field} removes an index.

A search is a POST with a filter: field names with the value they must have, or with one comparison. Fields in one filter are combined with AND:

Terminal window
curl -s -X POST http://127.0.0.1:3000/store-api/json/orders/search \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"filter": {"status": "shipped"}}'
curl -s -X POST http://127.0.0.1:3000/store-api/json/orders/search \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"filter": {"shipping.city": "Lisbon", "total": {"$lt": 100}}}'
{
  "documents": [
    { "_key": "o-1001", "_version": 2, "customer": "Omar Haddad", "status": "shipped", "total": 19 },
    { "_key": "o-1002", "_version": 1, "customer": "Lena Fischer", "shipping": { "city": "Lisbon" }, "status": "shipped", "total": 120 }
  ],
  "total": 2,
  "offset": 0,
  "limit": 50
}
{
  "documents": [
    { "_key": "09c3cf09-94a1-46b6-8034-1bf376604a86", "_version": 1, "customer": "Jane Doe", "shipping": { "city": "Lisbon" }, "status": "open", "total": 42.5 }
  ],
  "total": 1,
  "offset": 0,
  "limit": 50
}

The comparisons are $eq, $gt, $gte, $lt and $lte, one per field; a plain value means $eq. Numbers compare as numbers and strings byte by byte. The result is ordered by document key. total counts all matches, and limit (50 by default, 1000 at most) and offset page through them, see REST API. All hits, their content and total come from one consistent state, even while writes go on.

A search that cannot run is refused with 400 and says why:

Request Answer
A filter on a field without an index field 'customer' is not indexed in domain 'orders'
{}, no filter at all invalid filter: at least one filter is required
A value of the wrong type, {"total": "high"} invalid filter: filter value for field 'total' does not match index type Number
An unknown operator, {"total": {"$ne": 3}} unknown filter operator '$ne'
Two operators for one field filter for 'total' must contain exactly one operator

An index added later needs a re-index

Section titled “An index added later needs a re-index”

A new index covers the documents written after it. Documents that exist already are not back-filled, so a search on the new field finds nothing until you re-index. POST …/reindex starts a background task for all indexes of the domain, or for one field when you pass it, and answers 202 with a task id. The status shows the progress:

Terminal window
curl -s -X POST http://127.0.0.1:3000/store-api/json/orders/indexes \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"field": "customer", "type": "string"}'
curl -s -X POST http://127.0.0.1:3000/store-api/json/orders/search \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"filter": {"customer": "Jane Doe"}}'
curl -s -X POST http://127.0.0.1:3000/store-api/json/orders/reindex \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"field": "customer"}'
curl -s http://127.0.0.1:3000/store-api/json/orders/reindex/4579b6c1-f54d-4db0-b86e-264bc4c9128e \
  -H "Authorization: Bearer $KEY"
curl -s -X POST http://127.0.0.1:3000/store-api/json/orders/search \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"filter": {"customer": "Jane Doe"}}'
{ "field": "customer", "type": "string", "created_at": 1793745837 }
{ "documents": [], "total": 0, "offset": 0, "limit": 50 }
{ "task_id": "4579b6c1-f54d-4db0-b86e-264bc4c9128e" }
{ "duration_secs": 0, "processed": 3, "state": "completed" }
{ "documents": [{ "_key": "09c3cf09-94a1-46b6-8034-1bf376604a86", "_version": 1, "customer": "Jane Doe", "shipping": { "city": "Lisbon" }, "status": "open", "total": 42.5 }], "total": 1, "offset": 0, "limit": 50 }

The state is running (with processed and total_estimated), completed (with processed and duration_secs) or failed (with error and processed); a status asked right after the start can still say running, so poll until it does not. A second re-index of the same domain while one runs answers 409. Re-indexing only adds missing entries and can be repeated. Send {} or no body to re-index every index of the domain.

Documents are listed and counted

Section titled “Documents are listed and counted”

GET …/documents lists the documents of a domain in key order, 50 per page by default and 1000 at most, without needing an index. With keys_only=true it returns the keys alone, in keys, and documents is empty. …/documents/count counts them; both read all the keys of the domain, so use them on demand.

Terminal window
curl -s "http://127.0.0.1:3000/store-api/json/orders/documents?keys_only=true" -H "Authorization: Bearer $KEY"
curl -s http://127.0.0.1:3000/store-api/json/orders/documents/count -H "Authorization: Bearer $KEY"
{ "documents": [], "keys": ["09c3cf09-94a1-46b6-8034-1bf376604a86", "o-1001", "o-1002"], "total": 3, "offset": 0, "limit": 50 }
{ "count": 3 }

Bulk load and export use NDJSON

Section titled “Bulk load and export use NDJSON”

POST …/bulk takes a body with one JSON object per line. A line may carry _key; without it the document gets a generated key. _version is ignored. Blank lines are skipped. Documents are applied in batches of 100, each batch atomic, and a line that fails is reported and skipped while the rest goes on. The answer counts what happened. With this file, orders.ndjson:

{"_key": "o-2001", "customer": "Chloé Martin", "status": "open", "total": 7.9}
{"_key": "o-2002", "customer": "Noah 🚀", "status": "open", "total": 15}
{"customer": "No Key", "status": "open"}
not json
{"_key": "bad key", "status": "open"}
Terminal window
curl -s -X POST http://127.0.0.1:3000/store-api/json/orders/bulk \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/x-ndjson" \
  --data-binary @orders.ndjson
{
  "imported": 3,
  "failed": 2,
  "errors": [
    { "key": "bad key", "error": "invalid document key: key 'bad key' contains invalid characters (only [a-zA-Z0-9._-] allowed)" },
    { "key": "line 4", "error": "invalid JSON: expected ident at line 1 column 2" }
  ]
}

Three lines are documents, one is not JSON and one has a key with a space; line 4 counts the lines of the file. A bulk load takes no preconditions, is not subject to require_conditional_writes, and is limited to json.bulk_body_limit_bytes bytes, held in memory as a whole. Existing documents with the same key are replaced. The batches already applied stay when a later one fails.

GET …/export streams all documents of the domain as NDJSON, one document per line with _key and _version, without holding them all in memory. The export is not a snapshot: it lists the keys first and reads the documents one after the other. The output is valid input for bulk, so an export is also a copy to another domain:

Terminal window
curl -s http://127.0.0.1:3000/store-api/json/orders/export -H "Authorization: Bearer $KEY"
{"_key":"09c3cf09-94a1-46b6-8034-1bf376604a86","_version":1,"customer":"Jane Doe","shipping":{"city":"Lisbon"},"status":"open","total":42.5}
{"_key":"46525bd2-adc2-4946-8bad-0ff1e33b13c7","_version":1,"customer":"No Key","status":"open"}
{"_key":"o-1001","_version":2,"customer":"Omar Haddad","status":"shipped","total":19}
{"_key":"o-1002","_version":1,"customer":"Lena Fischer","shipping":{"city":"Lisbon"},"status":"shipped","total":120}
{"_key":"o-2001","_version":1,"customer":"Chloé Martin","status":"open","total":7.9}
{"_key":"o-2002","_version":1,"customer":"Noah 🚀","status":"open","total":15}

The switch json.enabled turns the engine off; every JSON endpoint then answers 503 Service Unavailable: JSON engine is disabled (json.enabled = false).

Related settings json.enabled · json.bulk_body_limit_bytes · json.lsm.max_key_length ·

json.lsm.max_value_size