Using

Key-value

applies to v0.5.1

The key-value engine stores values under keys in a domain, optionally with a TTL. Keys can be listed by prefix and counted, deleted in bulk, and watched through a resumable stream over server-sent events. The examples use the domain default, which exists from the first start, and sessions, which you create once:

curl -s -X POST http://127.0.0.1:3000/store-api/domains \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"name": "sessions"}'

{ "name": "sessions", "created_at": 1793745767, "require_conditional_writes": false }

A key holds bytes

The value of a key is a sequence of bytes that LuraDB does not interpret. A key is valid UTF-8 of up to 256 bytes, set by domains.max_user_key_length. In a path it is percent-encoded, so a key with a slash is written user%2F42%2Fprofile.

PUT stores a value and replaces any value the key had. It answers 200 whether the key was new or not, with an empty body. GET returns the bytes, with an ETag that names this version of the key:

curl -si -X PUT http://127.0.0.1:3000/store-api/kv/default/keys/greeting \
  -H "Authorization: Bearer $KEY" -d 'hello'
curl -si http://127.0.0.1:3000/store-api/kv/default/keys/greeting \
  -H "Authorization: Bearer $KEY"

HTTP/1.1 200 OK
content-length: 0

HTTP/1.1 200 OK
content-type: application/octet-stream
etag: "1a1a3944e070000"
content-length: 5

hello

DELETE removes a key and answers 204, also when the key does not exist. A read of a missing key, of an expired key and of a key in a domain that does not exist answers 404, with a line that says which:

curl -si -X DELETE http://127.0.0.1:3000/store-api/kv/default/keys/greeting -H "Authorization: Bearer $KEY"
curl -si http://127.0.0.1:3000/store-api/kv/default/keys/greeting -H "Authorization: Bearer $KEY"
curl -si http://127.0.0.1:3000/store-api/kv/nope/keys/x -H "Authorization: Bearer $KEY"

HTTP/1.1 204 No Content

HTTP/1.1 404 Not Found
content-type: text/plain; charset=utf-8

404 Not Found: key 'greeting' not found

HTTP/1.1 404 Not Found
content-type: text/plain; charset=utf-8

404 Not Found: Domain 'nope' not found

An empty value is a value: GET answers 200 with content-length: 0. A key longer than 256 bytes answers 400 Bad Request: Key length 257 exceeds maximum of 256 bytes. The size of a value is limited by the request body, 2 MiB; the setting lsm.max_value_size does not apply to single puts in this release, see its entry. Writes count against the write budget of the domain (rate_limit.default_write_iops), reads against the read budget, and a request over the budget answers 429.

A write can carry If-Match or If-None-Match: * to make it conditional; the rules are on Consistency and conditional writes.

A key can expire

Add ttl to a put to give the key a time to live in seconds. The read answers with the absolute expiry in the header X-Expires-At, in Unix seconds, and the metadata endpoint answers without reading the value:

curl -s -X PUT "http://127.0.0.1:3000/store-api/kv/sessions/keys/s-4f2a?ttl=3600" \
  -H "Authorization: Bearer $KEY" -d '{"user": "jane"}'
curl -si http://127.0.0.1:3000/store-api/kv/sessions/keys/s-4f2a -H "Authorization: Bearer $KEY"
curl -s http://127.0.0.1:3000/store-api/kv/sessions/keys/s-4f2a/meta -H "Authorization: Bearer $KEY"

HTTP/1.1 200 OK
content-type: application/octet-stream
x-expires-at: 1793749368
etag: "1a1a3944e160000"
content-length: 16

{"user": "jane"}

{ "expires_at": 1793749368, "last_modified_at": 1793745767958 }

expires_at is null for a key without a TTL. last_modified_at is the write time of the newest version in Unix milliseconds, not a creation time: every overwrite advances it.

A key past its time answers 404 at once. A put without ttl writes a key without expiry, also over a key that had one. Whether the key is removed from disk by a sweeper at the next pass, and whether watchers see a delete event for the expiry, depends on ttl_sweeper.enabled; reads never return an expired key either way.

A key can be null

PATCH …/null puts a key into an explicit null state. The key stays registered, appears in listings and counts, and carries no data. It is not the same as an empty value:

curl -si -X PATCH http://127.0.0.1:3000/store-api/kv/sessions/keys/s-9c02/null \
  -H "Authorization: Bearer $KEY"
curl -si http://127.0.0.1:3000/store-api/kv/sessions/keys/s-9c02 -H "Authorization: Bearer $KEY"

HTTP/1.1 200 OK

HTTP/1.1 204 No Content
etag: "1a1a3944e810000"

The read answers 204 with an ETag and no body, where an empty value answers 200. Setting a key to null is a write like any other: it resets a TTL (expires_at is null afterwards) and takes the same preconditions as a put. A relational KVREF column that points at a null key resolves to the value null with exists: true, see Domains and engines.

Keys are listed and counted by prefix

GET …/keys lists the keys of a domain in sorted order, one page at a time. prefix keeps the keys that start with it, contains the keys that contain the text (case-sensitive), and both apply before total, offset and limit. With a few more keys in sessions:

curl -s -X PUT http://127.0.0.1:3000/store-api/kv/sessions/keys/s-9c01 -H "Authorization: Bearer $KEY" -d '{"user": "omar"}'
curl -s -X PUT http://127.0.0.1:3000/store-api/kv/sessions/keys/cart-77 -H "Authorization: Bearer $KEY" -d '3 items'
curl -s -X PUT http://127.0.0.1:3000/store-api/kv/sessions/keys/user%2F42%2Fprofile -H "Authorization: Bearer $KEY" -d 'ok'
curl -s -X PUT http://127.0.0.1:3000/store-api/kv/sessions/keys/caf%C3%A9 -H "Authorization: Bearer $KEY" -d 'clé ✓'
curl -s "http://127.0.0.1:3000/store-api/kv/sessions/keys" -H "Authorization: Bearer $KEY"
curl -s "http://127.0.0.1:3000/store-api/kv/sessions/keys?prefix=s-" -H "Authorization: Bearer $KEY"
curl -s "http://127.0.0.1:3000/store-api/kv/sessions/keys?prefix=s-&limit=1&offset=1" -H "Authorization: Bearer $KEY"
curl -s "http://127.0.0.1:3000/store-api/kv/sessions/keys?contains=9c" -H "Authorization: Bearer $KEY"
curl -s "http://127.0.0.1:3000/store-api/kv/sessions/count?prefix=s-" -H "Authorization: Bearer $KEY"

{ "keys": ["café", "cart-77", "s-4f2a", "s-9c01", "s-9c02", "user/42/profile"], "total": 6, "offset": 0, "limit": 1000 }
{ "keys": ["s-4f2a", "s-9c01", "s-9c02"], "total": 3, "offset": 0, "limit": 1000 }
{ "keys": ["s-9c01"], "total": 3, "offset": 1, "limit": 1 }
{ "keys": ["s-9c01", "s-9c02"], "total": 2, "offset": 0, "limit": 1000 }
{ "count": 3 }

A page holds 1000 keys by default and 10000 at most; a larger limit is reduced to 10000 and the answer says so. An offset past the end answers an empty keys with the real total. Both the listing and the count scan the keys of the domain, so their cost grows with the size of the domain: use them on demand, not in a polling loop.

Keys are deleted by prefix in one step

DELETE …/keys?prefix= removes every key that starts with the prefix, narrowed by contains if you give it. The deletion is atomic: either every selected key is gone, or none is. The answer is the number of keys:

curl -s -X DELETE "http://127.0.0.1:3000/store-api/kv/sessions/keys?prefix=s-9" -H "Authorization: Bearer $KEY"
curl -si -X DELETE "http://127.0.0.1:3000/store-api/kv/sessions/keys?prefix=" -H "Authorization: Bearer $KEY"

{ "deleted": 2 }

HTTP/1.1 400 Bad Request
content-type: text/plain; charset=utf-8

400 Bad Request: prefix must not be empty

The prefix is required and must not be empty; to empty a whole domain, delete the domain. A selection of more than domains.max_bulk_delete_keys keys answers 413 and deletes nothing. A bulk delete is not covered by require_conditional_writes. In this release, a key longer than lsm.max_key_length minus the 19-byte domain prefix can be written, but a bulk delete that selects it fails with 500; the entry of that setting explains the interplay and how to avoid it.

A watch streams every change

GET …/watch opens a stream of server-sent events that stays open until the client leaves. The optional prefix limits it to the keys that start with it. Each change is one event: event is set or delete, data is the key, and id identifies the event. In a second terminal:

curl -sN "http://127.0.0.1:3000/store-api/kv/sessions/watch?prefix=s-" \
  -H "Authorization: Bearer $KEY"

While that runs, a put of s-100, a put of other-1, a put of s-101 and a delete of s-100 produce:

id: w-89a74aafbe480000-21
event: set
data: s-100

id: w-89a74aafbe480000-23
event: set
data: s-101

id: w-89a74aafbe480000-24
event: delete
data: s-100

The put of other-1 is not delivered, and its number 22 is missing from the ids: the numbers count the changes of the whole shard, not of your prefix. An id is w, the epoch of the server process in hexadecimal, and the number of the change. The server also sends keep-alive pings, so a proxy in between does not close an idle stream. A watch needs a read permission on the domain. It exists for the key-value engine only; the lifecycle events of all engines, such as a domain being created, are the admin stream described under Monitoring.

A reconnect continues where it stopped

A client that lost the connection sends the last id it saw as Last-Event-ID; a browser’s EventSource does this by itself. A client that cannot set headers passes last_event_id in the query. The server replays the changes after that id and then continues live:

curl -sN "http://127.0.0.1:3000/store-api/kv/sessions/watch?prefix=s-" \
  -H "Authorization: Bearer $KEY" -H "Last-Event-ID: w-89a74aafbe480000-23"

id: w-89a74aafbe480000-24
event: delete
data: s-100

id: w-89a74aafbe480000-25
event: set
data: s-102

id: w-89a74aafbe480000-26
event: delete
data: s-101

The replay comes from a window of recent changes in memory, lsm.watch_replay_buffer_size events per shard. When the server cannot guarantee a gapless continuation it sends event: reset first, with the reason in data, and an id from which a new reconnect resumes:

id: w-89a74aafbe480000-26
event: reset
data: {"reason":"unknown_id"}

Reason	Meaning
`restart`	The server process restarted since the id was issued.
`window_exceeded`	The id is older than the window, or the window is switched off.
`unknown_id`	The id is not one the server issued.
`lagged`	The client read too slowly and missed events.

After a reset, read the domain again, with a listing or the keys you track, and then keep applying the events. A slow client can only be told about missed events through lagged; the buffer in front of every watcher is lsm.wal_event_channel_capacity events.