WAL
Edit pageEvery shard keeps a Write-Ahead Log (WAL): an append-only record of the operations applied to it (document writes and deletes, metadata changes and commits), written alongside those changes before the shard’s files are committed. Each entry is tagged with the revision it produced, so the log is an ordered, replayable history of the shard.
The WAL exists so that a change is never lost between the moment it is accepted and the moment it is fully committed. A commit is comparatively expensive, so Xapiand does not perform one on every write; instead the operation is appended to the WAL and the heavier commit happens in the background (or when you ask for it explicitly). If a node stops uncleanly before that commit, the shard recovers the next time it is opened for writing: it replays the WAL from the last committed revision forward, re-applying exactly the operations that had not yet made it into the index, and commits them. Recovery is lazy: an index replays when its first write opens it, not when the node starts (only the node’s own registry indexes open at startup).
Writing the log
Section titled “Writing the log”WAL entries are written by a pool of asynchronous writer threads so that logging a change stays off the request’s critical path: a write is acknowledged once its log line is queued, and the writers append and sync the lines (see Visibility and Durability). Two options tune this pool:
--writerssets the number of asynchronous WAL writer threads.--wal-writer-cache-sizesets the writer’s cache size.
The WAL is enabled by default. Certain paths manage their own durability and write with
it disabled: a restore, whether the command-line one (xapiand --restore) or an HTTP
RESTORE, which loads its documents and commits
once at the end.
Reading the log
Section titled “Reading the log”The WAL method answers the log of an index as an array of entries, oldest first:
PUT /walapi/
{ "_settings": { "number_of_shards": 1 }}PUT /walapi/doc/1?commit
{ "title": "logged"}WAL /walapi/An entry is one of:
op | Other members |
|---|---|
REPLACE_DOCUMENT | docid and the document that was written |
DELETE_DOCUMENT | docid |
SET_METADATA | the key and its data (the index’s schema is one) |
COMMIT | none |
The log carries no spelling or synonym entries: nothing writes them. The query param
raw leaves the documents in their serialized form.
WAL reads one shard’s log, so it answers 400 (“This operation can only be executed on
a single shard”) for an index of more than one shard, and for a list of indexes
(the default is 5 shards: create the index with number_of_shards set to 1, as above).
WAL /walapi-five/Sources: src/server/search_views.cc:1752-1785 (wal_view), src/database/handler.cc:248-265,
src/database/wal.cc:324-400 (the entries); recovery, src/database/shard.cc:453-502 and
docs/mechanisms/wal-replay.md.