Skip to content

Down the Flume

Edit source

Streaming a whole database over a wire, compressed and checked, without holding it in memory.

Part 18 of 38ofFamiliars
  1. 1Session Zero
  2. 2Rolling for a Name
  3. 3The Haunted Handler
  4. 4The Sparse Wheel
  5. 5The Fork in the Path
  6. 6The Planet as a Prefix
  7. 7The Long Division
  8. 8The Line You Usually Cancel
  9. 9Sixteen Bytes Too Far
  10. 10Packing a UUID
  11. 11Bytes You Can Type
  12. 12The Shunting Yard
  13. 13Interchangeable Parts
  14. 14One Loop Per Core
  15. 15The Seam
  16. 16Append Only
  17. 17Who Owns the Tree
  18. 18Down the Flume
  19. 19The Syscalls That Lie
  20. 20Asking the Machine
  21. 21The Earth Is Not a Ball
  22. 22Smith and Schmidt
  23. 23Many Ways to Be Similar
  24. 24Built to Disappear
  25. 25Sometimes a Mutex Is Right
  26. 26Work, Not Threads
  27. 27Before the Program Runs
  28. 28No Collisions, Ever
  29. 29The Names Enums Throw Away
  30. 30The Bag of Holding
  31. 31A Map of Blocks
  32. 32Certainly Absent
  33. 33A Tree Made of Types
  34. 34Everything but the Views
  35. 35Who Owns the Wheel
  36. 36A Guest on the Loop
  37. 37Acknowledged Means Durable
  38. 38Sorted as Bytes

Part of Familiars: the small, sharp libraries I carved out of Xapiand. This one moves whole files, not buffers: flume.

A flume is the wooden channel that floats logs downstream, and the name is the design. When a node has to send another node a whole database, a replication snapshot, a bulk import, you cannot compress it the way you compress a small buffer, because the file is bigger than the memory you are willing to spend on it. You need to feed it in one end of a channel and have it come out the other, smaller and intact, while only ever holding a block of it at a time. That is flume: carry a file, or any fd-backed byte source, downstream over a channel, compressed, framed, integrity-checked, in bounded memory.

It is the transport-layer counterpart to the compressors buffer codecs from a few posts back. compressors squeezes an in-memory buffer; flume owns the parts a buffer codec does not: reading the file descriptor, framing the stream into blocks on the wire, and checking end-to-end that what arrived is what left. The per-block squeeze it delegates to a codec policy, so the two libraries split the job cleanly, one does the compression math, the other does the plumbing of moving a large thing safely.

For a whole-database transfer, the compression ratio is not a nicety, it is the bandwidth: every byte you do not send is a byte the link does not carry. So flume’s default is tuned for exactly that case. Here are the ratio and the bytes on the wire for each codec, measured on a 32 MiB replication-like corpus in 256 KB blocks:

codecratioon the wire
lz4 (classic)7.90x4.25 MB
zstd L3 (zstd’s default)12.63x2.66 MB
zstd L6 (flume default)15.14x2.22 MB
zstd L915.79x2.13 MB

flume defaults to Zstandard at level 6, above zstd’s own default of 3, because L6 is the knee of the curve for bulk data: about 48% fewer bytes than LZ4 (2.22 MB versus 4.25 MB for the 32 MiB corpus) and 17% fewer than level 3, for four to five times its compression time on one core. Push higher and the ratio barely improves while compression keeps slowing down; the classic LZ4 default was leaving nearly half the bandwidth on the table. And decompression does not slow down at higher levels, since there is less encoded data to read back.

The level is a knob, the codec is the wire

Section titled “The level is a knob, the codec is the wire”

The subtle, correct design decision is which choices are on the wire and which are not. A zstd frame is self-describing: it carries everything a decoder needs, so a block compressed at level 1 and a block compressed at level 19 both decode with the same decoder, and the two ends of the flume do not have to agree on the level at all. That makes the level a free, per-call-site knob: reach for ZstdCodec<1> when you are CPU-starved and the link is fat, ZstdCodec<19> when the link is thin and you have cycles to spare, and the receiver neither knows nor cares. The codec choice, zstd versus LZ4, is the actual wire-format decision, because those are incompatible formats; the level never is. Getting that distinction right is what lets you tune a transfer to its link without a protocol negotiation.

Reach for it when you move large fd-backed things between processes or hosts and want them compressed, framed, and checked without buffering the whole payload: replication, backups, bulk load. Skip it if your payload fits comfortably in memory (just use the buffer codec directly) or if you are already inside a transfer protocol that frames and checksums for you.

The next familiar is the floor all of this stands on, the one nobody thanks: the POSIX layer that survives a signal mid-syscall, and knows which of the calls beneath it are quietly lying to you.