How LuraDB works

Write path

applies to v0.5.1

A write is confirmed only once it is in the write-ahead log. From there the data moves into the MemTable and, once the MemTable is full, into SSTables on disk; the log is what recovery replays after a crash.

From request to disk

A write is confirmed once it is in the write-ahead log

The three engines store their data the same way, in LSM trees. What follows describes one write to a key-value domain; JSON documents and relational rows take the same road, each engine with a log, SSTables and a value log of its own.

The shard admits the write. The request has been routed to the shard that owns the domain, see Multicore. While disk space or memory is short, the write is refused here with 507, see Resource guard.
The write waits for the storage thread. One dedicated thread writes the logs of all engines through io_uring. Writes wait for it in a queue of storage.wal_queue_capacity places; when the queue is full, further writes are held back until there is room.
The storage thread appends the record to the log of the engine and syncs it with fdatasync. It batches concurrent writes into one sync per log (group commit), so a busy server does not pay a sync for every write. Only after the sync does the write count as confirmed.
A large value goes to the value log. A value of at least lsm.vlog_inline_threshold bytes (1024 by default) is appended to the value log of the engine; the MemTable keeps a pointer to it. Smaller values stay inline.
The MemTable takes the write. From this moment readers see it, and the client gets its answer.

If a sync fails, the service stops with exit code 3 instead of carrying on with data it may not have on disk, and the next start recovers from the log; see Resource guard.

A full MemTable becomes an SSTable

The MemTable is a sorted structure in memory that holds the newest writes. When its size, keys plus stored values, reaches lsm.memtable_size_threshold (4 MiB by default), it is frozen and a fresh MemTable takes over. The flush loop looks every lsm.flush_check_interval_ms (100 ms) for a frozen MemTable and writes the oldest one to disk as a level-0 SSTable, an immutable sorted file, and then saves the manifest, the list of files that make up the tree.

You can watch this happen. Writing 6000 values of 900 bytes into a fresh server crosses the threshold once, and /health counts the file:

{ "estimated_memtable_keys": 1523, "l0_sstable_count": 1, "vlog_size_bytes": 0 }

l0_sstable_count is the number of level-0 files, summed over the shards; estimated_memtable_keys is what the MemTables still hold. The JSON and relational engines flush at the same size with the same loop; only the key-value engine reads the [lsm] settings, the other two use these values built in.

Compaction merges SSTables into levels

Level-0 files overlap each other, so every read has to look at all of them. Compaction keeps their number down. When compaction.l0_threshold level-0 files exist (4), they are merged with the overlapping files of level 1 into new sorted files. A level from 1 on is limited in size: level 1 to compaction.l1_max_size (100 MiB), every following level to compaction.level_size_ratio times the one before (10). When a level exceeds its limit, its largest file is merged with the overlapping files of the next level. A merge writes its output as a stream, one file at a time, each at most compaction.max_sstable_size (64 MiB); its memory does not grow with the size of its inputs. A merge drops the versions of a key that no reader can still see, and a deletion marker only once no older version of the key exists below it. The loop checks for work every lsm.compaction_check_interval_ms (1 s).

The value log keeps large values and a janitor cleans it

SSTables hold keys and small values. A large value lives in the value log, a file that is only appended to, so that merging SSTables never copies it. An overwrite or a delete leaves the old bytes in the value log as dead space. Every janitor.check_interval_secs (60 s) the janitor estimates the dead share. Once the value log is at least janitor.min_vlog_size_bytes (64 MiB) and at least janitor.dead_bytes_threshold of it is dead (30 %), it copies the live values into a new generation of the file, rebuilds the SSTables that point into the old one and deletes the old generations once no reader uses them. A run waits when the volume lacks the room for the copy.

vlog_size_bytes in /health is the size of the value logs summed over the shards.

The log is trimmed once its contents are in SSTables

A write-ahead log is a directory of segment files of 64 MiB. A new segment starts when the current one is full. Once a second the server drops every segment whose records all shards already have in SSTables. A shard that rarely writes would keep old segments alive; so when the segments of a log reach storage.wal_max_size (256 MiB), the shards that still hold records of the oldest segment only in memory write them out, and the segment is deleted. A segment that has held records for five minutes is closed and dropped the same way. The entry for storage.wal_max_size explains the trade-off between early writes and recovery time.

After a crash, the log is replayed

The next start reads each log and replays the records that are not yet in SSTables, streaming them so that memory stays bounded. Sequence numbers go on where they stopped. A clean shutdown writes the MemTables out first, so little is left to replay; a crash leaves the MemTables’ contents in the log, and nothing confirmed is lost. The more the log holds, the longer the replay takes, which is what storage.wal_max_size limits.

Expired keys leave in two steps

A key with a TTL stays visible for at least its TTL and becomes invisible to reads within a second after that. The TTL sweeper then removes it for good: every ttl_sweeper.interval_secs (60 s) it checks up to ttl_sweeper.batch_size (500) candidates and writes a tombstone and a delete event for each expired one. Compaction drops the tombstone later, by the rule above. With ttl_sweeper.enabled off, expiry is purely lazy: expired keys are hidden on read but stay in the files until compaction meets them.

Readers work on snapshots

A read works on one consistent set of the engine’s files and MemTables, so a flush or a compaction that finishes during the read cannot make it miss a key. Queries that need one moment, such as a JSON search or a SELECT, read in a snapshot of their own. A batch of writes, such as the documents of one bulk-import step, becomes visible all at once.

Local clients can read snapshots from shared memory

With [shm] switched on, the server publishes a snapshot of the key-value data into shared memory for clients on the same host. It builds the snapshot from one consistent point across all data shards, checks every shm.snapshot_interval_ms (100 ms) and rebuilds it only when the data changed. A local client reads it without locks and without a network round trip; values that live in the value log come through a command ring to the server. The segments are two data buffers of shm.data_buffer_size each (256 MiB) and a state header:

luradb_docs.lock
luradb_docs_data_a
luradb_docs_data_b
luradb_docs_state

These are the files in /dev/shm of a server started with shm.enabled and shm.instance_id set to docs. The path where clients register is shm.registration_socket_path. The feature is off by default: members of the luradb group can read all key-value data through it without an API key, and with authentication on only the users listed in auth.trusted_uids can register as clients. Every rebuild briefly holds back the writes of all shards. /store-api/metrics carries the counters in its shm block.

What the data directory looks like

With the packaged configuration, the files of one server are laid out like this under /var/lib/luradb, here for one data shard:

luradb.wal/00000001.wal            key-value write-ahead log: segments of 64 MiB
luradb.wal/00000002.wal            the next segment, prepared in advance
luradb_json.wal/                   the same for the JSON engine
luradb_rel.wal/                    and for the relational engine
luradb_sys/                        system store: users, API key hashes, permissions, shard layout
shard_0/luradb.vlog                value log of the key-value engine
shard_0/luradb_json.vlog           value log of the JSON engine
shard_0/luradb_rel.vlog            value log of the relational engine
shard_0/luradb.db                  page file, empty in this release
sstables/shard_0/MANIFEST          list of the SSTables of the key-value engine
sstables/shard_0/sstable_000000.sst
luradb_json_sstables/shard_0/      SSTables of the JSON engine
luradb_rel_sstables/shard_0/       SSTables of the relational engine

Each log is one directory for all shards. The other files exist once per shard, under shard_<k>/: inside the SSTable directories, and beside the configured path for the value logs and the page file. Never change a file in these directories by hand. The paths are set by storage.wal_path, storage.vlog_path, storage.sstable_dir, storage.system_dir, json.wal_path, json.vlog_path, json.sstable_dir, rel.wal_path, rel.vlog_path and rel.sstable_dir.