How LuraDB works

Resource guard

applies to v0.5.1

The server checks the free space of its data volumes and its own memory use every second. Below the warn thresholds it logs a warning and GET /health reports degraded; below the block thresholds it refuses writes with 507 Insufficient Storage and Retry-After while reads continue, and it resumes on its own once there is room again.

The server watches every volume its data lies on

Once a second the server reads the free space of every volume that one of its data paths lies on: the data files, the write-ahead logs, the log directory and the backup directory. A volume that several paths share is watched once. “Free” is what an unprivileged process can use, without the share reserved for root. A volume that hangs, such as an unreachable network mount, holds up nothing else: it keeps its last reading, and the log shows a warning after five seconds.

The server also reads its own anonymous memory (RssAnon in /proc) and compares it with a budget. The start log reports what is watched:

INFO luradb::control::resources: Watching the volume of /var/lib/luradb/sstables: 1081101176832 bytes, 836397944832 free; warning below 834397000000 bytes free, writes refused below 830397000000, resumed above 832397000000.
INFO luradb::control::resources: Memory budget 12259860480 bytes of 16346480640 (MemTotal); warning from 80 %, writes refused from 95 %, resumed below 90 %.

Setting	Default	Meaning
`resources.disk_free_warn`	`"15% max 10 GB"`	Free space below which the server warns.
`resources.disk_free_block`	`"5% max 2 GB"`	Free space below which the server refuses writes.
`resources.disk_free_resume`	`"10% max 4 GB"`	Free space each volume needs before refused writes resume.
`resources.memory_budget`	`"75%"`	The anonymous memory the server may use: a share of the host’s memory, or of the cgroup limit `memory.max` when that is lower.

A disk threshold is a size, a percentage of the volume, or both, of which the smaller counts. The thresholds are resolved once at start, separately for each volume, and must satisfy block <= resume <= warn with a block threshold of at least 64 MiB, or the start is refused. The memory thresholds are fixed shares of the budget: warning from 80 %, writes refused from 95 %, writes accepted again below 90 %. The budget is at least 256 MiB.

Three states: ok, warning, blocked

State	When	Writes	`GET /health`
`ok`	Every volume is above its warn threshold and memory is below 80 % of the budget.	Accepted.	`"status": "ok"`
`warning`	A volume is below its warn threshold, or memory is at 80 % or more.	Accepted.	`"status": "degraded"`, HTTP status `200`
`blocked`	A volume is below its block threshold, or memory is at 95 % or more.	Refused with `507`.	`"status": "degraded"`, HTTP status `200`

The state is one value for the whole server: the worst of all volumes and of memory. /health carries the parts in its resources field, without paths or numbers:

{
  "status": "degraded",
  "resources": { "disk_state": "warning", "memory_state": "ok", "state": "warning" }
}

GET /store-api/metrics, which needs the admin key, adds the volumes, their paths and the numbers in its own resources block.

Warning: nothing is refused yet

The warning is the early notice. The server logs one line per state change and repeats it every minute while the state lasts:

WARN luradb::control::resources: Resources running short: disk /var/lib/luradb/sstables free 833397870592 of 1081101176832, threshold 834397000000. Nothing is refused yet; writes are refused below the block threshold.

Let monitoring poll /health and alert on degraded; that gives you the time to add space or remove data before writes stop.

Blocked: writes answer 507 and reads go on

Below the block threshold the server logs an error, repeated every minute, and every write answers 507 Insufficient Storage with Retry-After: 5:

curl -si -X PUT http://127.0.0.1:3000/store-api/kv/default/keys/k3 \
  -H "Authorization: Bearer $KEY" -d v3

HTTP/1.1 507 Insufficient Storage
retry-after: 5

507 Insufficient Storage: write blocked: disk space below threshold, retry in 5 s

ERROR luradb::control::resources: Writes refused: disk /var/lib/luradb/sstables free 829397864448 of 1081101176832, threshold 830397000000. Writes, deletes and new backups on a short volume answer 507 until every volume is above its resume threshold; reads, health, backup downloads and deletions and sign-ins go on.

The write is not applied. What is refused and what goes on:

Refused with `507`	Goes on
Writes, deletes and `PATCH …/null` on keys	Reads of keys, documents and rows
Document writes and bulk imports	JSON searches, SQL `SELECT`, listings and counts
SQL `INSERT`, `UPDATE`, `DELETE` and DDL, table imports	`/health` and `GET /store-api/metrics`
Creating domains	Creating users, rotating keys, sign-ins
Restores and the write tools of MCP	Downloading and deleting backups
New backups, unless the backup directory is on another volume above its warn threshold and memory is not short	Compaction and the cleanups of the value log and the write-ahead log, which free space where they can

Background jobs that only write deletion markers, such as the TTL sweeper, wait. Clients of the shared-memory path get the same error. A refused write was not applied; the client can try it again after the Retry-After of 5 seconds.

Writes resume two calm seconds after there is room

Writes are accepted again once every volume has more free space than resources.disk_free_resume and memory is below 90 % of the budget, for two readings in a row:

INFO luradb::control::resources: Writes are accepted again: every volume is above its resume threshold and memory below 90 % of its budget. Still low: disk /var/lib/luradb/sstables free 833397870592 of 1081101176832, threshold 834397000000.

The state then falls to warning, not to ok. /health keeps reporting degraded until every volume is above the warn threshold again, which takes two more readings and ends with INFO … Disk space and memory are back above their warn thresholds. A gap between the three thresholds is what keeps a server that hovers around one value from switching between accepting and refusing writes.

A delete does not free space at once: it writes a marker that compaction removes later. Faster ways out are deleting old backups (DELETE /store-api/backups/{id}), moving files off the volume or growing it.

Memory shortage flushes the MemTables

The memory that counts is the anonymous memory of the server process, which includes MemTables, caches and buffers. At 95 % of resources.memory_budget writes are refused with 507 … write blocked: memory budget exhausted. While memory is short, the server flushes its MemTables to disk and returns free heap to the kernel, at once and then every ten seconds. When the budget is a percentage, it is a share of the host’s memory, or of the memory limit of the server’s cgroup when that is lower; the start log names which one it used (MemTotal or cgroup memory.max). To cap the memory of the service from the outside as well, set MemoryHigh= in a drop-in (sudo systemctl edit luradb).

At start, the server adds up the memory it takes whatever the load, such as block caches, buffer pool and MemTables, see Multicore. When that exceeds half of the budget, it logs a warning that lists the parts.

When the disk fails

A disk that fails is a different matter from a disk that is full. The server stops with its own exit code in two cases:

Exit code 3: written data may not be on disk. The disk reported an error while making data durable (fsync), or writes to a data file (write-ahead log, value log, SSTable) failed for good: at once on a read-only file system, otherwise at the third failure in a row of the same log, or of three flushes in a row, a failed fsync of a new SSTable included. A full disk or an exceeded quota is not counted; the watch refuses writes before that happens. The log shows data sync failed on <path> or data write failed on <path>. Until the stop, /health answers status: "degraded" with the failures in a row in io, and GET /store-api/metrics names the file. systemd starts the service again, and the start recovers every confirmed write from the write-ahead log. A write that failed with an error to the client may be there after the restart. Meanwhile, a failing flush or compaction logs its error once and then every 60th time, and a flush is retried once a second.
Exit code 4: no room to recover. At start, a volume has less free space than the recovery needs (not enough free disk space to recover). systemd does not restart the service (RestartPreventExitStatus=4); free space, then run sudo systemctl start luradb. A start below the block threshold with room for the recovery does not stop: the service starts with writes refused and reads going on.

The unit allows three starts within five minutes, manual ones included; after that systemd refuses further starts until sudo systemctl reset-failed luradb. A container needs a restart policy that fits the codes: Docker cannot leave out an exit code, so restart: on-failure:3 restarts after an exit 3, and after any failure at most three times, while unless-stopped would restart an exit 4 in a loop.

Try it on a test host

To see the three states, set thresholds a few gigabytes below the free space of the volume, then take that space away. This prints the three lines for the [resources] section of luradb.toml:

FREE=$(df --output=avail -B1 /var/lib/luradb | tail -1)
echo "disk_free_warn = \"$(( (FREE - 2000000000) / 1000000 )) MB\""
echo "disk_free_resume = \"$(( (FREE - 4000000000) / 1000000 )) MB\""
echo "disk_free_block = \"$(( (FREE - 6000000000) / 1000000 )) MB\""

disk_free_warn = "834397 MB"
disk_free_resume = "832397 MB"
disk_free_block = "830397 MB"

Restart the service, then take space from the volume in two steps and watch /health and a write after each:

sudo fallocate -l 3GB /var/lib/luradb/fill1    # free space is now below warn: degraded, writes still accepted
sudo fallocate -l 4GB /var/lib/luradb/fill2    # below block: writes answer 507
sudo rm /var/lib/luradb/fill2                  # above resume: writes accepted, still degraded
sudo rm /var/lib/luradb/fill1                  # above warn: ok