Operating

Monitoring

applies to v0.5.1

GET /health needs no key and reports ok or degraded, with the state of every data shard, of disk space and of memory. With an admin API key, GET /store-api/metrics adds the numbers behind it, and GET /store-api/events streams lifecycle and DDL changes. The service logs to journald.

/health says whether to look closer

GET /health is for load balancers, container health checks and a first look. It needs no key, names no path and gives no size.

curl -s http://127.0.0.1:3000/health

{
  "domain_count": 1,
  "estimated_memtable_keys": 9,
  "l0_sstable_count": 0,
  "resources": { "disk_state": "ok", "memory_state": "ok", "state": "ok" },
  "shards": [{ "id": 0, "status": "ok" }],
  "status": "ok",
  "uptime_secs": 8,
  "version": "0.5.1",
  "vlog_size_bytes": 0
}

Field	Meaning
`status`	`ok`, or `degraded` while a data shard does not answer, disk space or memory runs short, or writes of a data file fail.
`shards`	One entry per data shard, `ok` once it answered its ping and `timeout` when it did not answer within 250 ms.
`resources`	The state of disk space (the worst of all watched volumes) and of memory: `ok`, `warning` or `blocked`, and `state` for the worse of the two. See Resource guard.
`io`	Present only while writes of a data file fail: `failures`, the most failures in a row of one file. The third failure stops the server with exit code 3.
`uptime_secs`, `version`	Seconds since the start, and the server version.
`domain_count`	Active key-value domains.
`estimated_memtable_keys`, `vlog_size_bytes`, `l0_sstable_count`	Keys in the key-value MemTables, bytes of the key-value value logs, and key-value SSTables on level 0 that wait for compaction, summed over the data shards.

The HTTP status is 200 as long as every shard answered, also with "status": "degraded" for a short disk or memory, because the server still serves reads. When a shard does not answer within 250 ms, the answer is 503 with the full report, and the entry of that shard says "status": "timeout". A health check that treats only 5xx as a failure, such as curl -f, therefore notices a stuck shard but not a volume that is filling up; poll the body and alert on degraded to get the early notice.

The metrics endpoint has the numbers

GET /store-api/metrics needs an admin key. It answers one JSON object of blocks:

curl -s http://127.0.0.1:3000/store-api/metrics -H "Authorization: Bearer $KEY"

{
  "system": { "total_reads": 5, "total_writes": 5, "compaction_runs": 0, "janitor_runs": 0, "memtable_size_bytes": 796 },
  "domains": [
    { "domain": "default", "read_ops": 5, "write_ops": 5, "read_latency_us_p50": 5, "read_latency_us_p99": 5,
      "cache_hit_rate": 1.0, "rate_limit_rejections": 0, "window_secs": 60 }
  ],
  "engines": {
    "kv": { "read_ops": 5, "write_ops": 5, "read_latency_us_p50": 5, "read_latency_us_p95": 5, "read_latency_us_p99": 5,
            "write_latency_us_p50": 5000, "write_latency_us_p95": 5000, "write_latency_us_p99": 5000, "window_secs": 60 },
    "json": { "read_ops": 0, "write_ops": 1, "read_latency_us_p50": 0, "read_latency_us_p95": 0, "read_latency_us_p99": 0,
              "write_latency_us_p50": 5000, "write_latency_us_p95": 5000, "write_latency_us_p99": 5000, "window_secs": 60 },
    "rel": { "read_ops": 0, "write_ops": 0, "read_latency_us_p50": 0, "read_latency_us_p95": 0, "read_latency_us_p99": 0,
             "write_latency_us_p50": 0, "write_latency_us_p95": 0, "write_latency_us_p99": 0, "window_secs": 60 }
  },
  "block_cache": { "hits": 0, "misses": 0, "small_hits": 0, "main_hits": 0, "small_evictions": 0, "main_evictions": 0, "current_bytes": 0 },
  "backup": null,
  "shm": { "snapshot_publishes_total": 0, "snapshot_publish_skips_total": 0, "snapshot_consecutive_skips": 0, "clients_kicked_total": 0 },
  "resources": {
    "disk": [{ "path": "/var/lib/luradb/sstables", "size_bytes": 1081101176832, "free_bytes": 836399935488, "state": "ok" }],
    "memory": { "rss_anon_bytes": 12029952, "budget_bytes": 12259860480, "state": "ok" },
    "state": "ok"
  }
}

Block	What it counts
`system`	`total_reads` and `total_writes` summed over the data shards; `compaction_runs`, `janitor_runs` and `memtable_size_bytes` of the key-value engines.
`domains`	One entry per key-value domain over the last 60 seconds: operations, read latency, `cache_hit_rate`, and `rate_limit_rejections`, the requests that answered `429`. Only key-value domains are listed.
`engines`	`kv`, `json` and `rel`, each with the same nine fields over the window: reads and writes, and the latency percentiles. A disabled engine still has its block, all zero.
`block_cache`	Hits, misses, evictions and size of the key-value block cache.
`backup`	`null` while `backup.enabled` is off. Otherwise `last_success_at`, `last_success_at_by_schedule`, `last_duration_ms`, `last_size_bytes`, `running` (the job in progress, if any) and `failed_total`.
`shm`	Counters of the shared-memory snapshot publisher; all zero while `shm.enabled` is off.
`resources`	The numbers behind `/health`: per watched volume `path`, `size_bytes`, `free_bytes` and `state`, and the server’s anonymous memory against its budget.
`io`	Present only while writes of a data file fail: the `path` of the file with the most failures in a row, and `failures`.

Three things help in reading it:

The window is 60 seconds. The counters are refreshed every second and cover the last 60 full seconds, so read_ops / window_secs is an average rate. total_reads and total_writes in system count since the start.
Latencies are bucket values. Each operation falls into one of seven buckets, below 10 µs, 100 µs, 1 ms, 10 ms, 100 ms, 1 s, and above. A percentile reports the middle of its bucket: 5, 50, 500, 5000, 50000, 500000 or 2000000 microseconds. 5000 means between 1 and 10 ms, and 0 means that no operation fell into the window.
There is no Prometheus endpoint. The answer is JSON; a collector polls it.

While a data shard is not running, the endpoint answers 503. GET /store-api/metrics/domains/{name} returns the entry of one key-value domain, and a user with a permission on that domain may call it; it answers 404 for an unknown domain.

The event stream reports lifecycle changes

GET /store-api/events is a server-sent-events stream of what happens to domains and schemas, across all three engines. It is for tools that follow the structure of the database, such as an admin console or a cache of domain names. It carries no keys or values; for those, a key-value domain has its own watch stream, see Key-value. The stream needs an admin key, and curl -N switches off buffering:

curl -N http://127.0.0.1:3000/store-api/events -H "Authorization: Bearer $KEY"

id: g-bf6b45314834ffff-1
event: domain_created
data: {"engine":"kv","type":"domain_created","domain":"sessions","ts":1793743075}

id: g-bf6b45314834ffff-2
event: domain_created
data: {"engine":"json","type":"domain_created","domain":"orders","ts":1793743075}

id: g-bf6b45314834ffff-3
event: index_created
data: {"engine":"json","type":"index_created","domain":"orders","object":"status","ts":1793743075}

id: g-bf6b45314834ffff-4
event: domain_created
data: {"engine":"rel","type":"domain_created","domain":"orders","ts":1793743075}

id: g-bf6b45314834ffff-5
event: table_created
data: {"engine":"rel","type":"table_created","domain":"orders","object":"customers","ts":1793743075}

id: g-bf6b45314834ffff-6
event: view_created
data: {"engine":"rel","type":"view_created","domain":"orders","object":"lisbon","ts":1793743075}

id: g-bf6b45314834ffff-7
event: domain_deleted
data: {"engine":"kv","type":"domain_deleted","domain":"sessions","ts":1793743075}

These events came from creating the domains sessions (key-value) and orders (JSON and relational), an index on status, a table customers and a view lisbon, and then deleting sessions. The data field is JSON with the engine, the type, the domain, the name of the table, view or index as object where there is one, and the Unix time ts. The event types are:

Engine	Types
All	`domain_created`, `domain_deleted` (the domain is gone for clients), `domain_purged` (the background purge has removed its data).
JSON	`index_created`, `index_dropped`
Relational	`table_created`, `table_altered`, `table_dropped`, `index_created`, `index_dropped`, `view_created`, `view_dropped`

Every event has an id. After a dropped connection, a client sends the last id it saw as the Last-Event-ID header, or as ?last_event_id= where it cannot set headers, and the server replays what the client missed from a ring in memory. When it cannot replay without a gap, it sends event: reset with a reason, and the client should re-read the domain lists before it follows the stream again. The reasons are restart (the server restarted, so the ring is gone), lagged (the client fell more than events.channel_capacity events behind) and unknown_id (the id is not in the ring). The ring holds events.replay_buffer_size events; with 0 every reconnect gets a reset.

Logs go to journald or to files

By default LuraDB writes its log to standard output, which systemd hands to the journal:

journalctl -u luradb
journalctl -u luradb -f

The text format is one line per message, with the time, the level, the module and the message. It carries ANSI colour codes, written to standard output and to files alike. For a log shipper, set log.format to json, which writes one JSON object per line without colour codes:

{"timestamp":"2026-11-03T21:58:18.244543Z","level":"INFO","fields":{"message":"Auth: admin 'admin' reconciled"},"target":"luradb::auth"}

log.level picks the amount: verbose (debug), info and prod, which logs warnings and errors only. The environment variable RUST_LOG overrides the configured levels when it is set. A level per module goes into [log.modules]; a module without a level uses the global one, and the modules are log.modules.auth, log.modules.api, log.modules.engine and log.modules.storage.

With log.path set, the log goes to files in that directory instead of standard output, and the directory is created if it is missing. A path the server cannot create does not stop the start with a message but with a panic, see Troubleshooting. log.rotation names the file:

Rotation	File name
`daily`	`luradb.log.2026-11-03`
`hourly`	`luradb.log.2026-11-03-21`
`none`, or any other value	`luradb.log`

log.retention_days deletes rotated files older than that many days, judged by their modification time, with 0 keeping them forever. The cleanup runs once for every 24 hours the server is up, counted from its start, so a server that is restarted more often than that never cleans up.

Lines worth searching for:

In the log	Means
`Resources running short: disk … Nothing is refused yet`	A volume is below its warn threshold.
`Writes refused: …`	A volume or the memory is below its block threshold; writes answer `507`.
`Writes are accepted again: …`	The shortage is over.
`data sync failed on <path>`, `data write failed on <path>`	The disk failed; the server stops with exit code 3.
`[Backup] job '…' completed`, `[Backup Scheduler] …`	A backup finished, or a scheduled one was skipped.
`unknown config key '…' is ignored`	The configuration names a key that does not exist or was removed.

Reading the log over HTTP

With log.http_access on, an admin reads the log files through the API. It needs log.path, because standard output cannot be read back. GET /store-api/logs/files lists the files, newest first, and GET /store-api/logs returns the end of one:

curl -s http://127.0.0.1:3000/store-api/logs/files -H "Authorization: Bearer $KEY"
curl -s "http://127.0.0.1:3000/store-api/logs?lines=2&q=Shard" -H "Authorization: Bearer $KEY"

{ "files": [{ "file": "luradb.log.2026-11-03", "size": 2324, "modified": 1793743074 }] }

{
  "file": "luradb.log.2026-11-03",
  "format": "text",
  "lines": ["… INFO luradb::server::control: Shard 0: block cache 65536 KiB per engine; MemTable threshold 4194304 bytes per engine; buffer pool 1024 pages; registered I/O buffers off."],
  "truncated": false
}

Parameter	Effect
`lines`	The number of lines from the end of the file: 100 by default, at most 1000. `0` answers `400`.
`q`	Only lines that contain this text, case-sensitive; only matching lines count towards `lines`.
`file`	A name from the file list, which must start with `luradb.log`. A name with a path answers `400`, an unknown file `404`. Without it, the newest file.

The lines come back oldest first. The server reads backwards through at most 4 MiB of the file; when it stops there before it has lines matches, truncated is true. Without log.http_access, both endpoints answer 503 Service Unavailable: log access is disabled (log.http_access = false). With auth.enabled off, anyone who reaches the port can read the log, and the start log warns about it.

What to alert on

Signal	Where	What to do
`"status": "degraded"` and `resources.state`	`/health`	Free space or memory is short, see Resource guard.
HTTP `503` and a shard with `"timeout"`	`/health`	A data shard does not answer within 250 ms. Look in the log of the same minute.
`io.failures` above 0	`/health`, `/store-api/metrics`	Writes of a data file fail. The file is named in the metrics.
`backup.last_success_at` older than the schedule allows, `backup.failed_total` growing	`/store-api/metrics`	Backups do not complete, see Backup and restore.
`rate_limit_rejections` above 0 for a domain	`/store-api/metrics`	The domain uses up its budget, see Troubleshooting.
`l0_sstable_count` keeps growing	`/health`	Level-0 SSTables appear faster than compaction merges them. Compaction starts at `compaction.l0_threshold` files.