Skip to content

Every shard keeps a Write-Ahead Log (WAL): an append-only record of the operations applied to it (document writes and deletes, metadata changes and commits), written alongside those changes before the shard’s files are committed. Each entry is tagged with the revision it produced, so the log is an ordered, replayable history of the shard.

The WAL exists so that a change is never lost between the moment it is accepted and the moment it is fully committed. A commit is comparatively expensive, so Xapiand does not perform one on every write; instead the operation is appended to the WAL and the heavier commit happens in the background (or when you ask for it explicitly). If a node stops uncleanly before that commit, the shard recovers the next time it is opened for writing: it replays the WAL from the last committed revision forward, re-applying exactly the operations that had not yet made it into the index, and commits them. Recovery is lazy: an index replays when its first write opens it, not when the node starts (only the node’s own registry indexes open at startup).

WAL entries are written by a pool of asynchronous writer threads so that logging a change stays off the request’s critical path: a write is acknowledged once its log line is queued, and the writers append and sync the lines (see Visibility and Durability). Two options tune this pool:

  • --writers sets the number of asynchronous WAL writer threads.
  • --wal-writer-cache-size sets the writer’s cache size.

The WAL is enabled by default. Certain paths manage their own durability and write with it disabled: a restore, whether the command-line one (xapiand --restore) or an HTTP RESTORE, which loads its documents and commits once at the end.

The WAL method answers the log of an index as an array of entries, oldest first:

PUT /walapi/
{
"_settings": {
"number_of_shards": 1
}
}
PUT /walapi/doc/1?commit
{
"title": "logged"
}
WAL /walapi/

An entry is one of:

opOther members
REPLACE_DOCUMENTdocid and the document that was written
DELETE_DOCUMENTdocid
SET_METADATAthe key and its data (the index’s schema is one)
COMMITnone

The log carries no spelling or synonym entries: nothing writes them. The query param raw leaves the documents in their serialized form.

WAL reads one shard’s log, so it answers 400 (“This operation can only be executed on a single shard”) for an index of more than one shard, and for a list of indexes (the default is 5 shards: create the index with number_of_shards set to 1, as above).

WAL /walapi-five/

Sources: src/server/search_views.cc:1752-1785 (wal_view), src/database/handler.cc:248-265, src/database/wal.cc:324-400 (the entries); recovery, src/database/shard.cc:453-502 and docs/mechanisms/wal-replay.md.