Skip to content

Append Only

Edit source

A blob store that never overwrites a record, so a crash can only ever cost you the last write.

Part 16 of 38ofFamiliars
  1. 1Session Zero
  2. 2Rolling for a Name
  3. 3The Haunted Handler
  4. 4The Sparse Wheel
  5. 5The Fork in the Path
  6. 6The Planet as a Prefix
  7. 7The Long Division
  8. 8The Line You Usually Cancel
  9. 9Sixteen Bytes Too Far
  10. 10Packing a UUID
  11. 11Bytes You Can Type
  12. 12The Shunting Yard
  13. 13Interchangeable Parts
  14. 14One Loop Per Core
  15. 15The Seam
  16. 16Append Only
  17. 17Who Owns the Tree
  18. 18Down the Flume
  19. 19The Syscalls That Lie
  20. 20Asking the Machine
  21. 21The Earth Is Not a Ball
  22. 22Smith and Schmidt
  23. 23Many Ways to Be Similar
  24. 24Built to Disappear
  25. 25Sometimes a Mutex Is Right
  26. 26Work, Not Threads
  27. 27Before the Program Runs
  28. 28No Collisions, Ever
  29. 29The Names Enums Throw Away
  30. 30The Bag of Holding
  31. 31A Map of Blocks
  32. 32Certainly Absent
  33. 33A Tree Made of Types
  34. 34Everything but the Views
  35. 35Who Owns the Wheel
  36. 36A Guest on the Loop
  37. 37Acknowledged Means Durable
  38. 38Sorted as Bytes

Part of Familiars: the small, sharp libraries I carved out of Xapiand. This one is where the data lives: storage.

Every datastore has one moment it is most afraid of, and it is the same moment: the power fails, or the process is killed, in the middle of a write. What happens to your data in that instant is decided long before it, by the shape of the format. A store that overwrites in place, that seeks to an existing record and writes new bytes over it, is exposed at that moment in the worst way, because a half-finished overwrite has destroyed the old record without finishing the new one. The data that was safe a millisecond ago is now torn.

storage is built so that moment cannot cost you anything but the write that was in flight. It is a compressed, append-only, multi-volume blob store: opaque binary records, written into fixed-size blocks inside volume files, each write returning a stable offset you use to read it back later. Large records are transparently compressed, Zstandard by default. And it never overwrites a record. New data only ever gets appended past the end of what is already there.

The append-only discipline is the whole game. Because a record, once written and committed, is never touched again, a crash can only ever damage the region currently being appended: the tail. Everything before it is immutable and therefore intact by construction. Recovery is not a delicate repair of overwritten structures; it is finding where the last good record ended and treating everything after it as the write that did not make it. That is why crash safety here is a property of the format and not of luck. You do not overwrite a record, so a torn write cannot reach into the past.

Xapiand’s data store had lived on this shape for years, and part of pulling it out was hardening it further into a crash-safe v2 volume format: it writes the new layout while still reading old v1 volumes, so existing data comes across unchanged. Each record carries a header and an optional footer checksum, so a torn tail record is not just detectable by position but by a checksum that will not match, and recovery can draw the line exactly. The one thing v2 does rewrite is the volume header in block 0, on every commit, and only after the data it points to is on disk. The bytes that change fit in one 512-byte sector, which the disk writes atomically, so a crash leaves either the old header or the new one.

The store is one header-only template, and the interesting design choice is what it does not decide:

template <typename Header, typename BinHeader, typename Footer,
typename IO = storage::DefaultIO>
class Storage;

Header, BinHeader, and Footer are the on-disk record framing, and you supply them: init and validate hooks let you stamp your own magic numbers, UUIDs, and checksums into the format. IO is the file-operation policy. Both are injection seams, so the engine itself carries nothing application-specific, and two different systems can use the same append-and-offset core with entirely different on-disk framing and durability policies. The append-only spine is fixed; the bytes around each record are yours.

Reach for it when you need to persist a lot of opaque blobs, get a stable handle back for each, and care about surviving a crash without an external write-ahead log doing the safety for you. It pairs naturally with an index that stores those offsets. Skip it if you need in-place mutation of records (append-only means updates are new records, and reclaiming space is a compaction pass, not a free lunch), or if a real embedded database already gives you transactions.

The next familiar is what fills those records: the dynamic document value the whole engine passes around, the one that serializes to a compact wire format and patches like JSON.