Operating

Troubleshooting

applies to v0.5.1

When LuraDB does not start or refuses requests, the cause is usually one of a few conditions: a kernel older than 6.1, a data directory that another LuraDB process uses, a changed shard count, an invalid configuration, a full request queue or a volume that ran short of space. The log shows which: journalctl -u luradb.

Start with the log

A refusal to start prints its message on standard error after Error:, and the process ends with exit code 1. A refusal that happens after the log is set up also appears as an ERROR line in the log. Under systemd, both reach the journal; read the start of the current boot:

journalctl -u luradb -b

To see the message directly, stop the service and run the server by hand as the service user, from the directory the service runs in. The server holds a lock on its data, so it cannot run twice:

sudo systemctl stop luradb
sudo -u luradb sh -c 'cd /var/lib/luradb && exec /usr/bin/luradb --config /etc/luradb/luradb.toml'

For more detail while you search, set log.level to verbose or start with the environment variable RUST_LOG, which overrides the configured levels. See Monitoring.

The server refuses to start

The start checks the configuration first, then the paths, then the data. The messages below are the literal text of v0.5.1.

The kernel is older than 6.1

LuraDB requires Linux kernel 6.1 or newer (io_uring DEFER_TASKRUN)

The server sets up its io_uring rings with flags that the kernel accepts from 6.1 on, and the setup fails on an older one. uname -r shows the version. A container uses the kernel of its host, or of the virtual machine of Docker Desktop. Any other failure of the ring setup ends with failed to set up the engine thread's io_uring ring and the system error. Compatibility lists the requirements.

The configuration file cannot be read

Message	Cause
`Failed to parse /etc/luradb/luradb.toml: TOML parse error at line 1, column 8` followed by the line and `invalid table header`	A syntax error. The message shows the line and column.
`… invalid type: string "abc", expected u16`	A value of the wrong type.
… missing field `api_key`	A required field is missing in an `[[auth.admins]]` block. `[[backup.schedule]]` needs `backup.schedule.keep_last` the same way.
`Error: Permission denied (os error 13)`	The file exists but the service user cannot read it. The message names no path. The packaged file is owned by `root:luradb` with mode `640`.

A --config path that does not exist is not an error. The server logs No config file found at …, using defaults and runs with the built-in defaults: authentication off, loopback only, data in the current directory. A typo in the path therefore shows up as a server without your settings, not as a failed start.

A key that the server does not know is warned about, and the start goes on: unknown config key '…' is ignored. That covers typos and removed keys.

A value is invalid

The start checks the sections one after the other and stops at the first error. Most messages begin with invalid config: , which the tables on this page leave out.

Message	What to do
`auth.enabled = false and server.bind_address = '0.0.0.0' is not loopback — enable auth (auth.enabled = true) or bind to loopback (127.0.0.1 / ::1)`	Turn authentication on, or bind to `127.0.0.1`.
`server.bind_address 'localhost' is not a valid IP address: invalid IP address syntax`	The bind address is an IP address, not a name.
`server.tls_port (3055) must differ from server.port (3055)`	Give the two listeners different ports.
`server.http_enabled and server.tls_enabled are both false — no listener would start`	Keep one listener on.
`mcp.port (3055) must differ from server.port`	The MCP listener needs a port of its own.
`mcp.oauth_enabled = true requires auth.enabled = true`, `mcp.oauth_enabled = true requires mcp.public_url`, `mcp.public_url 'http://db.example.com' must use https:// unless its host is localhost, 127.0.0.1 or ::1`	See MCP server.
`storage.wal_path and storage.sstable_dir resolve to the same location '…'`	Two data paths, or a data path and `backup.dir`, must not resolve to the same place. Symlinks are followed.
`storage.wal_max_size (64 MiB) must be at least 128 MiB`	The cap on each write-ahead log is at least 128 MiB.
`storage.wal_max_size (256) needs a number and a unit (B, kB, KiB, MB, MiB, GB, GiB, TB or TiB), as in "256 MiB"`	Write the unit.
`storage.wal_queue_capacity (0) must be 1 to 65536`
`resources.memory_budget (100 MiB) must be at least 256 MiB`
`resources thresholds on the volume of … (1081101176832 bytes) resolve to disk_free_block 10000000, disk_free_resume 4000000000 and disk_free_warn 10000000000 bytes; they need block <= resume <= warn and block >= 67108864 bytes`	The three disk thresholds, resolved for that volume, are out of order or too small. See Resource guard.
`lsm.max_value_size (5000000000) exceeds the WAL recovery field cap (100663296 bytes) — writes this large could never be recovered`	The size limits of an engine stay below 96 MiB.
`json.lsm.max_value_size (100) must be at least 4096`
`multicore.frontend_workers (999) must be 0 (auto) or 1 to 32 (available cores)`	The same for `multicore.cpu_offload_threads` and `multicore.reader_threads`.
`multicore.shard_queue_capacity (0) must be at least 1`	The same for `multicore.reader_queue_capacity`.
`rel.max_tables_per_domain (0) must be between 1 and 4294967295`
`cors.enabled = true but cors.allowed_origins is empty — no origin would ever be allowed`	See CORS.
`cors.allowed_origins entry 'https://Console.example.com/' is not a valid origin — expected exactly scheme://host[:port] (lowercase, no userinfo/path/query/fragment)`
`log.http_access requires log.path`	The log API reads files, so it needs `log.path`.
`shm.instance_id ('a_b') must be non-empty and contain only [A-Za-z0-9-]`
`shm.registration_socket_path ('…') equals server.unix_socket_path — SHM clients could not register; use a different path`
`Invalid trusted_proxies config: invalid IP address syntax`	An entry of `proxy.trusted_proxies` is neither an address nor a CIDR range.

A backup schedule has its own checks, each naming the schedule:

Message	Cause
`backup.schedule.cron ('0 3 * ') is not a valid cron expression: cron expression '0 3 *' must have exactly 5 fields (minute hour day-of-month month weekday), got 4 (schedule 'nightly')`	Wrong number of fields.
`backup.schedule.cron ('0 3 31 2 *') has no date within 4 years — the schedule would never run (schedule 'nightly')`	A date that does not exist.
`backup.schedule.scope ('everything') is not a valid scope: invalid backup scope 'everything': expected 'all', 'kv', 'json', 'rel', 'kv:<domain>', 'json:<domain>', 'rel:<domain>', or 'domain:<name>' (schedule 'nightly')`	An unknown scope.
`backup.schedule.name ('nightly') must be unique across schedules`	Two schedules with one name.
`backup.schedule.keep_last (0) must be at least 1 (schedule 'nightly')`

TLS files and sockets

Message	Cause
`cannot open server.tls_cert_path '/etc/luradb/tls/server.crt': I/O error: No such file or directory (os error 2)`	The certificate is not there, or the service user cannot read it. The same text exists for `server.tls_key_path`.
`server.tls_cert_path '…' contains no certificate`	The file is not a PEM certificate. The key file has `contains no private key` and `cannot parse server.tls_key_path`.
`unix_socket_path must be absolute: luradb.sock`
`parent directory of unix_socket_path does not exist: /run/luradb-nonexistent`	Under systemd, `/run/luradb` exists only through `RuntimeDirectory=`; outside the unit, create the directory.
`unix_socket_path exists and is not a socket: …`	The server removes a leftover socket of an earlier run, but never another kind of file.

The data directory is not usable

Message	Cause
`creating the data directory /var/lib/luradb/sstables: Permission denied (os error 13)`	The service user cannot create the directory. Check the owner of the parent. Under systemd, a path outside `/var/lib/luradb` is read-only for the service, see Directories and permissions.
`creating the data directory …: File exists (os error 17)`	A file lies where a directory belongs.
`luradb_sys is in use by another LuraDB process`	Another LuraDB process holds the lock on the data directory. The path in the message is `storage.system_dir` as configured. Stop the other process; under systemd, the service itself may already be running.
`Address already in use (os error 98)`	Another process listens on the port. The message does not name the port; `ss -ltnp` shows who holds `server.port`, `server.tls_port` or `mcp.port`.
shard count changed (stored 1, configured 2) — run `luradb --migrate-shards 2`	The data directory was written with another number of shards than `multicore.shards` says. Set it to `0`, or migrate.
`shard migration incomplete — rerun the migration tool`	An interrupted `--migrate-shards`. Run the same command again.
`no data directory at luradb_sys`	`--migrate-shards` found no data directory at the configured system directory: it ran from another directory than the service runs in, so a relative path led nowhere, or without the right `--config`.
`not enough free disk space to recover: <volume> has <n> bytes free, the start needs <m> (the write-ahead logs that recover onto the volume)`	A volume has less free space than the recovery needs. The process exits with code 4 and systemd does not restart it. Free space and start again.

With multicore.shards at 0 and a data directory written for fewer shards than this host would pick, the start does not fail. It warns and goes on with the stored count, naming the command that changes it:

WARN luradb::server::layout: Data shards: 1, as the data directory was written; this host would use 8. Writes stay on 1 core(s) until you stop the server and run: luradb --migrate-shards 8

A log directory that cannot be created

A log.path that the server cannot create is the one case that ends in a panic instead of a message, and with exit code 101:

thread 'main' panicked at …/tracing-appender-0.2.4/src/rolling.rs:154:14:
initializing rolling file appender failed: InitError { context: "failed to create log directory", source: Os { code: 2, kind: NotFound, message: "No such file or directory" } }

Create the directory with the right owner, or point the key at a directory the service user can create.

Exit codes

Code	Meaning
`0`	A stop on `SIGTERM` or `SIGINT`.
`1`	The start was refused with one of the messages above, or the server stopped on an error.
`3`	The disk failed to make data durable, or writes to a data file failed for good. The log shows `data sync failed on <path>` or `data write failed on <path>`. The next start recovers from the write-ahead log.
`4`	At the start, a volume had no room for the recovery.
`101`	A panic, such as the log directory above.
`137`	The process was killed with `SIGKILL`, for instance by the out-of-memory killer. A confirmed write survives this: the next start replays the write-ahead log.

Codes 3 and 4 are explained in Resource guard, and what systemd does with each in systemd and Docker.

503: a queue is full

The server takes only so many requests at once. A request beyond that is refused at once, without being started, and the answer says so and invites a retry:

HTTP/1.1 503 Service Unavailable
retry-after: 1

503 Service Unavailable: the server is overloaded, retry later

There are two queues. multicore.shard_queue_capacity (1024) is the number of requests one data shard takes at once, running or waiting; writes to the domains of a shard go through it. multicore.reader_queue_capacity (4096) is the number of read requests that may wait for a reader thread; running reads do not count. A refused request was not applied, and it is safe to send it again after the second the header names.

A queue that is full for more than a burst means the shard or the readers cannot keep up. In a test with multicore.shard_queue_capacity at 1 and 60 parallel uploads of 300 KB values, 108 of 120 requests were refused. What helps, in this order:

Clients retry. Wait for Retry-After, and add some random jitter, so that the retries do not arrive together.
Spread the writes over domains. All writes to one domain run on one core. Several domains give several shards work at the same time, see Multicore.
Look at the disk. A write completes when it is in the write-ahead log, so slow storage fills the queue. The metrics show the write latency per engine.
Raise the capacity only for a burst you expect. A longer queue holds more requests in memory and makes the tail latency longer.

Not every 503 is a full queue:

Body	Cause
`503 Service Unavailable: the server is overloaded, retry later`	A full queue, with `Retry-After: 1`.
`503 Service Unavailable: the server is shutting down`	The server is stopping.
`503 Service Unavailable: JSON engine is disabled (json.enabled = false)`	The JSON engine is off.
`503 Service Unavailable: backup is disabled (backup.enabled = false)`	Backups are off.
`503 Service Unavailable: log access is disabled (log.http_access = false)`	The log API is off.
`GET /health` answers `503` with the report	A data shard did not answer within 250 ms, see Monitoring.
`GET /store-api/metrics` answers `503`	A data shard is not running.

With rel.enabled off, the relational routes do not exist and answer an empty 404, not a 503.

429: a domain used up its budget

Every key-value and relational domain has a budget of reads and writes per second, rate_limit.default_read_iops (1000) and rate_limit.default_write_iops (500). It works like a bucket that holds as many requests as the limit and fills up again continuously, so a short burst up to the limit passes and a longer one is slowed down. A request that finds the bucket empty is refused with 429 and Retry-After: 1:

HTTP/1.1 429 Too Many Requests
retry-after: 1

429 Too Many Requests: write rate limit exceeded

The key-value engine says read rate limit exceeded or write rate limit exceeded; the relational engine says rate limit exceeded for domain 'orders'. With rate_limit.default_write_iops at 5, eight writes in quick succession answered 200 200 200 200 200 429 200 429: the bucket was empty after five, refilled a little, and ran dry again. rate_limit_rejections in the metrics of the domain counts the refused requests. The limit is per domain and the same for all clients. A request that was refused did nothing; send it again after a second, or raise the limits. The JSON engine does not apply these budgets in v0.5.1: twelve document writes in a second, against rate_limit.default_write_iops at 5, were all accepted.

507: a volume or memory ran short

HTTP/1.1 507 Insufficient Storage
retry-after: 5

507 Insufficient Storage: write blocked: disk space below threshold, retry in 5 s

The server refuses writes while a volume is below its block threshold or the memory of the server is at 95 % of its budget, and it goes on reading. GET /health shows degraded with "disk_state": "blocked", and the log has an ERROR line that names the volume, its free space and the threshold:

ERROR luradb::control::resources: Writes refused: disk /var/lib/luradb/sstables free 836396249088 of 1081101176832, threshold 870000000000. Writes, deletes and new backups on a short volume answer 507 until every volume is above its resume threshold; reads, health, backup downloads and deletions and sign-ins go on.

Free space on the named volume, or move data off it, delete old backups with DELETE /store-api/backups/{id}, or grow the volume; the server resumes on its own. Resource guard describes the thresholds, what is refused and what goes on. If the thresholds fire although the volume has plenty of room, check that resources.disk_free_warn, resources.disk_free_block and resources.disk_free_resume are what you meant: a percentage counts against the size of the volume, and the smaller of size and percentage counts when both are given.

401, 403 and 404 that surprise

Answer	Likely cause
`401` on every request	The key is wrong or has been rotated, the `Authorization: Bearer <key>` header is missing or misspelled, or the admin key in `luradb.toml` changed and the service was not restarted.
`401` for a user who worked yesterday	The key was rotated or the user was deleted.
`403` for a user	Permissions are per engine and domain; `reporter` with `read` on the key-value domain `orders` cannot read the JSON domain `orders`. See Security.
`403` on `/store-api/metrics`, `/store-api/auth/users`, backups	These endpoints are for admin keys only.
`404` with an empty body on `/store-api/rel/…`	The relational engine is switched off, see above.
`410` after `DELETE` of a domain	The domain is gone for clients while the purge runs; creating a domain of that name answers `409` until the purge ends, about ten seconds for a small domain in a test.
`413`	A body over 2 MiB, or over the limit of the endpoint, see Limits and defaults.

Backups and restores that fail

Answer or state	Cause
`409 Conflict: a backup or restore job is already running`	Only one job at a time. Check `GET /store-api/backups` for `running`; a restore with `mode: replace` can take a while.
`503 Service Unavailable: backup is disabled (backup.enabled = false)`	Set `backup.enabled`.
A restore ends as `failed` with `domain_exists: <name>`	A domain of the archive exists, and `fail_if_exists` is the default. Restore under a new name with `into_domain`, or use `mode: replace`.
`400 Bad Request: invalid backup file: checksum mismatch`	The archive was changed or damaged after it was written.
`400 Bad Request: into_domain requires a backup with exactly one domain`	A remap needs a `kv:`, `json:`, `rel:` or `domain:` archive.
`GET /store-api/restores/{id}` answers `404`	The status is kept in memory only and is gone after a restart.
A scheduled backup did not run	Look for `[Backup Scheduler]` lines. A tick is skipped while another job runs or while writes are refused.

Backup and restore has the details.