Skip to content

The Bag of Holding

Edit source

The small, honest tools too little for a post of their own, and a look back at the whole party.

Part 30 of 38ofFamiliars
  1. 1Session Zero
  2. 2Rolling for a Name
  3. 3The Haunted Handler
  4. 4The Sparse Wheel
  5. 5The Fork in the Path
  6. 6The Planet as a Prefix
  7. 7The Long Division
  8. 8The Line You Usually Cancel
  9. 9Sixteen Bytes Too Far
  10. 10Packing a UUID
  11. 11Bytes You Can Type
  12. 12The Shunting Yard
  13. 13Interchangeable Parts
  14. 14One Loop Per Core
  15. 15The Seam
  16. 16Append Only
  17. 17Who Owns the Tree
  18. 18Down the Flume
  19. 19The Syscalls That Lie
  20. 20Asking the Machine
  21. 21The Earth Is Not a Ball
  22. 22Smith and Schmidt
  23. 23Many Ways to Be Similar
  24. 24Built to Disappear
  25. 25Sometimes a Mutex Is Right
  26. 26Work, Not Threads
  27. 27Before the Program Runs
  28. 28No Collisions, Ever
  29. 29The Names Enums Throw Away
  30. 30The Bag of Holding
  31. 31A Map of Blocks
  32. 32Certainly Absent
  33. 33A Tree Made of Types
  34. 34Everything but the Views
  35. 35Who Owns the Wheel
  36. 36A Guest on the Loop
  37. 37Acknowledged Means Durable
  38. 38Sorted as Bytes

The closing of Familiars: the small, sharp libraries I carved out of Xapiand. This is the last one, and it is really a table full of them.

Every adventuring party accumulates a bag of small things that never get their own scene: the rope, the chalk, the ten-foot pole, each one dull until the exact moment it saves you. A codebase is the same. Most of what falls out of a big system is not a jewel with a story; it is a small, honest tool that does one plain thing correctly so you never have to think about it again. There are a lot of these in Xapiand, too small for a deep-dive apiece, and they deserve better than to be lost, so here they are, tipped out onto the table at once.

Time, which nobody gets right the first time

Section titled “Time, which nobody gets right the first time”
  • times and time-point: two small conversions the standard library leaves to you. times makes the operating system’s timespec (seconds plus nanoseconds) behave like a number: add it, subtract it, compare it, read the wall clock into it, and the nanoseconds always carry into the seconds correctly. time-point packs a std::chrono instant into one integer of nanoseconds and back, so you can store it in a column, a record or an atomic. The wall clock is for stamping when something happened; to measure how long it took, use std::chrono::steady_clock, which never jumps.
  • epoch: time since the epoch, header-only and dependency-free, for when you just want the number.
  • nanosleep: an EINTR-safe sleep, because even sleeping is a syscall a signal can interrupt, and io’s lesson applies here too.
  • datetime: a real date/time parser and formatter, the largest of this group, and honestly a full post’s worth of pain (time zones, formats, the calendar’s cruelty) waiting to be written.
  • split: a tiny string splitter, the function everyone re-writes.
  • strings: a toolkit of std::format-style general string utilities.
  • static-string and stringified: a constexpr compile-time string, and a small string_view wrapper, the leaves a lot of the compile-time chapter grows from.
  • escape and repr: turn an arbitrary byte buffer into a printable, escaped string, one for a log line, one for a single-line debug repr. The tools you reach for at 2 a.m. to see what is actually in a buffer.
  • md5 and sha256: the two message digests, buffer in, fixed-size hash out, for checksums and content addressing (md5 for the non-adversarial cases, sha256 for the rest).
  • math: small integer-math helpers, the safe versions of the arithmetic you keep getting subtly wrong.
  • random: a dependency-free random-number helper, because the standard <random> is thorough and unpleasant to use, and most code just wants a number.
  • strict-stox: strict numeric parsing, the stoi that actually rejects "12abc" instead of quietly returning 12, which the standard one does not.
  • lru-cache: an intrusive LRU cache, a doubly-linked list threaded through the nodes themselves, so eviction is O(1) with no separate bookkeeping structure.
  • bloom-filter: a header-only Bloom filter, the probabilistic set that answers “definitely not present” or “maybe present” in 32 bits per element.
  • iterators and lazy: lazy iteration adaptors and deferred-evaluation helpers, small pieces of the “compute it only if someone asks” habit that runs through the whole engine.
  • errno-names: map an errno to its symbolic name and human description, so a failure logs as EAGAIN and a sentence, not 11.
  • located-exception: an exception base that remembers where it was thrown (function, file and line) and formats its message with std::format. It records that one site, not the stack; the whole stack is the tracer’s job.
  • fs: filesystem helpers, the path and directory operations that sit just above io.
  • allocators: an STL-compatible allocator toolkit, for the places where the default allocator is not the right one.
  • scheduler: the timer-wheel scheduler and per-key debouncer built directly on the sparse wheel, the layer between that raw lock-free structure and the code that arms timers. The logger rode it until it moved to a wheel of its own, and scheduler is now retiring along with stash; Who Owns the Wheel tells why.
  • url-parser: splits a URL path and query string into their parts, the small companion to the router that decides where the path goes.

That is the whole bag, and with it, the whole party. Sixty-odd libraries, from a jester that names your servers to a tracer that works in a haunted room to the ten-foot pole of a strict stoi, all of them pulled out of one distributed search engine and made to stand on their own.

Here is what I actually take from having done it. A big system does not feel, from the inside, like a collection of small ideas. It feels like one large, tangled thing you hold in your head all at once, and are a little afraid to touch. But when you finally pull a piece out, give it its own repository, its own README, its own tests, and make it earn its keep alone, the piece turns out to be smaller and clearer than it ever looked while it was load-bearing. The lock-free wheel was never really about Xapiand. The perfect hash was never really about search. They were general ideas that happened to be born under pressure, and the pressure is exactly what made them sharp. Taking the engine apart was not demolition. It was finding out how many separate, reusable, understandable things I had actually built while I thought I was building one.

That is the honest reason this series exists. Not to show off an engine, that story is its own post, but because the tools were the better souvenir. Every familiar here is real, open, and on github.com/Kronuz; nearly all are MIT; the code in each post is the code that runs. Clone the ones that solve a problem you have. That is what they are for, and it is why I bothered to carve them out at all: a thing you build alone, and understand completely, you can hand to someone else.

The campaign is over. The party is yours now.

Last updated: