Skip to content

compression

2 posts with the tag “compression”

Down the Flume

Streaming a whole database over a wire, compressed and checked, without holding it in memory.

Replicating a database means moving a file bigger than memory across a link, and doing it so that it arrives smaller, intact, and without ever loading the whole thing into RAM at either end. The seventeenth familiar is named after the wooden channel that floats logs downstream: you feed a file in one end and it comes out the other, compressed and checked, in bounded memory, at a ratio that turns bandwidth into headroom.

Interchangeable Parts

Three compression formats behind one interface, swapped by changing a single word.

Compression is a set of tradeoffs, interop versus speed versus ratio, and the honest answer is 'it depends,' which is useless if switching backends means rewriting your call sites. The twelfth familiar puts three incompatible formats behind one identical shape, so choosing between deflate, lz4, and zstd is a matter of changing a single word in your code, and nothing else.