Skip to content

Blog

Asking the Machine

Reading a host's own limits, on systems that each hide the answer somewhere different.

A long-running server should know its own limits: how many file descriptors it is burning against the cap, how much RAM the process holds, how much disk is left before writes start failing. None of those numbers has a portable source. The nineteenth familiar is the small, unglamorous library that asks each operating system for them in its own dialect, so the rest of the engine can just ask once.

The Syscalls That Lie

The EINTR retry, the fsync that isn't, and the one call you must never retry.

The POSIX calls at the bottom of every server are not as honest as they look. A read can be cut short by a signal and return having done nothing wrong. An fsync on macOS returns success without actually putting your data on the platter. And retrying a close() that was interrupted can silently destroy a file descriptor another thread just reused. The eighteenth familiar is the thin layer that knows all of this and does the boring, correct thing so the rest of the engine never has to.

Down the Flume

Streaming a whole database over a wire, compressed and checked, without holding it in memory.

Replicating a database means moving a file bigger than memory across a link, and doing it so that it arrives smaller, intact, and without ever loading the whole thing into RAM at either end. The seventeenth familiar is named after the wooden channel that floats logs downstream: you feed a file in one end and it comes out the other, compressed and checked, in bounded memory, at a ratio that turns bandwidth into headroom.

Who Owns the Tree

A JSON-shaped MessagePack value whose copies share nothing.

A search engine moves documents around all day, and in C++ the way you hand one over decides what it costs. The sixteenth familiar is the document value itself, and every document lives in an arena of its own: a reference reaches into it, a move hands it over, and a copy builds a whole new one. When Xapiand needed many threads to share one document, it froze it and shared a pointer instead.