Skip to content

Session Zero

Edit source

Sixty small libraries, one search engine, and the campaign to open them.

Part 1 of 38ofFamiliars
  1. 1Session Zero
  2. 2Rolling for a Name
  3. 3The Haunted Handler
  4. 4The Sparse Wheel
  5. 5The Fork in the Path
  6. 6The Planet as a Prefix
  7. 7The Long Division
  8. 8The Line You Usually Cancel
  9. 9Sixteen Bytes Too Far
  10. 10Packing a UUID
  11. 11Bytes You Can Type
  12. 12The Shunting Yard
  13. 13Interchangeable Parts
  14. 14One Loop Per Core
  15. 15The Seam
  16. 16Append Only
  17. 17Who Owns the Tree
  18. 18Down the Flume
  19. 19The Syscalls That Lie
  20. 20Asking the Machine
  21. 21The Earth Is Not a Ball
  22. 22Smith and Schmidt
  23. 23Many Ways to Be Similar
  24. 24Built to Disappear
  25. 25Sometimes a Mutex Is Right
  26. 26Work, Not Threads
  27. 27Before the Program Runs
  28. 28No Collisions, Ever
  29. 29The Names Enums Throw Away
  30. 30The Bag of Holding
  31. 31A Map of Blocks
  32. 32Certainly Absent
  33. 33A Tree Made of Types
  34. 34Everything but the Views
  35. 35Who Owns the Wheel
  36. 36A Guest on the Loop
  37. 37Acknowledged Means Durable
  38. 38Sorted as Bytes

This is the opening of Familiars, a series of self-contained C++ libraries I pulled out of Xapiand. A Search Engine from Scratch tells the story of the engine itself; this series opens the small creatures I found living inside it.

I spent the better part of a decade building a distributed search engine. Xapiand wraps a daemon around Xapian: the cluster, the replication, the storage, the geometry, the network, everything you have to build around a good index to make it a real server. That story is its own post. This is not that story.

When I came back to the engine years later, what I most wanted to keep was the tools. This year, whenever a piece of Xapiand was sharp enough and self-contained enough to live on its own, I carved it out into its own repository, gave it a real README and a test suite, and let it stand up by itself. There are about sixty of them now, scattered across github.com/Kronuz: a lock-free timer, a signal-safe crash tracer, a fantasy-name generator, a triangular mesh for the whole planet, an integer with no ceiling. Small things. Sharp things.

This series opens them, one at a time.

In a game, a familiar is a small companion bound to something larger: not the hero, but the creature at the hero’s shoulder that does one strange thing unreasonably well. That is exactly what these libraries are. Each is a single idea, extracted from a system big enough to have needed it in anger, and each earns a post the same way:

  • The one thing it does that nothing else does. Not “a queue,” but the property that made me reach for this instead of the obvious thing.
  • The constraint that forced its shape. Every one of them exists because two normal answers failed, and the failure is the interesting part.
  • The gnarly bit. The bug, the trap, the oozing thing sewn into the lining. A footgun in a data table. A malloc hiding inside a function that looked safe. The same register spelled nine different ways across three operating systems. This is where the learning is, so it stays in.
  • Where it is actually the right tool, and, just as often, where it is not.

You can read any one of these on its own. None of them needs the search engine, or the one before it, to make sense. They are a party, not a plot.

The campaign runs through the engine in loose chapters, each a cluster of related familiars:

  • Little languages. Small compilers and grammars: the name generator that names every node, a boolean-expression parser, a perfect hash built entirely at compile time.
  • The forbidden layer. The libraries that live where the normal rules stop applying: the crash tracer that runs inside a signal handler, the EINTR-safe POSIX layer, the machine-introspection code that has to be portable across every OS the server runs on.
  • The lock-free jewels. The concurrency crown of the codebase: the timer store whose insert never stalls another thread, the deferred logger it was built for, an atomic shared_ptr, a bounded blocking deque, the worker pool and the coroutines that share the loops.
  • The network path. One request’s whole journey: the thread-per-core reactor, the HTTP framework, the streaming parser, the router that matches a URL to a handler without ever walking the whole tree, and the server they assemble into.
  • Bytes and persistence. How data survives a crash and a wire: a crash-safe append-only store, three interchangeable compression codecs, a MessagePack value type, a condensed sortable UUID, an integer with no ceiling, a number codec whose bytes sort like the numbers, a term dictionary at a few bytes a word, the filter in front of it, and a log and a Raft core that never acknowledge too early.
  • Space and sound. Domain algorithms: the Hierarchical Triangular Mesh that turns the planet into terms, geodetic coordinate math, phonetic matching, string similarity.
  • Compile-time magic. The things that run before main does: compile-time hashing and character classification, reflective enums, a trie made of types.
  • The bag of holding. And a heap of tiny, honest helpers too small for a post of their own, tipped out onto the table together at the end.

A few, and they are the same as the rest of this blog:

  • Real numbers. When a familiar is fast, or small, or exact, the measurement says so, not an adjective. When it is a wash, I say that too.
  • The messy parts stay in. The bug I shipped, the design that failed first, the thing I only caught by staring at it too long. Those are the point.
  • Everything is real and open. Every familiar is a public repository you can clone, read, and use. Most are MIT or public domain. The code in each post is the code that runs.

The party is already assembled. Three of them come first: a jester who names the world, a tracer that works in a haunted room, and the strange lock-free thing I would keep over all the others. The rest are waiting.

Roll for initiative.

Last updated: