Skip to content

A Guest on the Loop

Edit source

Coroutines that pause, share a thread, and find their way home.

Part 36 of 38ofFamiliars
  1. 1Session Zero
  2. 2Rolling for a Name
  3. 3The Haunted Handler
  4. 4The Sparse Wheel
  5. 5The Fork in the Path
  6. 6The Planet as a Prefix
  7. 7The Long Division
  8. 8The Line You Usually Cancel
  9. 9Sixteen Bytes Too Far
  10. 10Packing a UUID
  11. 11Bytes You Can Type
  12. 12The Shunting Yard
  13. 13Interchangeable Parts
  14. 14One Loop Per Core
  15. 15The Seam
  16. 16Append Only
  17. 17Who Owns the Tree
  18. 18Down the Flume
  19. 19The Syscalls That Lie
  20. 20Asking the Machine
  21. 21The Earth Is Not a Ball
  22. 22Smith and Schmidt
  23. 23Many Ways to Be Similar
  24. 24Built to Disappear
  25. 25Sometimes a Mutex Is Right
  26. 26Work, Not Threads
  27. 27Before the Program Runs
  28. 28No Collisions, Ever
  29. 29The Names Enums Throw Away
  30. 30The Bag of Holding
  31. 31A Map of Blocks
  32. 32Certainly Absent
  33. 33A Tree Made of Types
  34. 34Everything but the Views
  35. 35Who Owns the Wheel
  36. 36A Guest on the Loop
  37. 37Acknowledged Means Durable
  38. 38Sorted as Bytes

Part of Familiars: the small, sharp libraries I carved out of Xapiand. This one is how code takes turns on a shared thread: task.

Picture one waiter serving a whole restaurant. They take an order, hand it to the kitchen, and instead of standing there until the food is ready, they go take the next table’s order. When a dish comes up, they carry it out and pick up where they left off.

That waiter is an event loop: one thread that serves thousands of network connections by never sitting idle on any one of them. It only works if the code it runs can say “I am waiting now, serve someone else, and come back to me here.” A plain function cannot. Once called, it runs to the end, and if it waits for a socket, the whole thread waits with it.

A coroutine is a function that can. In C++20, writing co_await inside a function means “pause here.” Its local variables are saved in a small block of memory called its frame, the thread goes off to do other work, and later something resumes the frame and the function continues from the next line. A waiter with a notepad.

task is my coroutine type for that, with one narrow job: run on a loop that belongs to somebody else, and be a good guest there.

C++20 gives you the language feature and no standard coroutine type, so every library brings its own. A naive one has three problems.

The first is the stack. When task A awaits task B and B finishes at once, B has to resume A, and the obvious way is to call A from inside B. Do that a million times in a loop and you get a million nested calls and a stack overflow.

The second is the thread. A program often runs one loop per core. If A runs on loop 1 and awaits B, which finishes on loop 2, a naive coroutine resumes A on loop 2, a thread it does not own, touching state loop 1 thinks is its alone.

The third is manners. A coroutine that computes for 50 milliseconds without pausing is a waiter reading a novel at one table while every other table waits.

Asio’s own asio::awaitable handles some of this, but it is bound to Asio, and I wanted one type that scoring, storage and network code could all share.

The fix for the stack is called symmetric transfer. Instead of B calling A, B’s last act is to return A’s handle, and the compiler turns that into a jump: B’s frame is done, so A takes over its spot on the stack. A relay runner hands over the baton; they do not carry the next runner on their back. The tests run a million immediate awaits and a chain 100,000 tasks deep, and the stack never grows.

The gnarly part is that compilers disagree. Clang makes that jump at every optimization level; GCC only in optimized builds without sanitizers, and elsewhere it quietly makes a real call that recurses until the stack runs out. So on GCC, task uses a trampoline by default: a small per-thread queue where a finishing task drops the next handle, and a loop at the bottom of the stack that pops and resumes them one after another. Nothing nests, because every resume starts from the same floor. The transfers note has the details.

For the thread problem, task remembers which loop A was on when it awaited B. If B finishes on that loop, it hands the baton directly; if somewhere else, it posts A back to A’s own loop. That is executor affinity: a task always comes home.

The loop itself is the host’s. To task, a scheduler is anything with a post that queues a resumption and a now that tells the time; the library owns no queue and no threads. task_asio.hh provides one on a standalone Asio executor, and an inline one drives the tests.

For manners, there is the budget. A long loop takes a task::budget with a time slice, say 200 microseconds, and calls co_await slice.checkpoint() every few thousand iterations. While the slice lasts, a checkpoint costs one read of the CPU’s cycle counter and never pauses. Once the slice is spent, it pauses and joins a queue of demoted work that the loop serves one entry at a time between everything else. The long job still finishes; it just stops cutting in line.

The last piece is cancellation: a client disconnects or a deadline passes, and the work on its behalf should stop. A cancel_token carries that request and a reason, and a deadline is a token a timer triggers. Deadlines live on one timer wheel per loop, so setting one allocates nothing.

The point is that a token interrupts a wait already in progress. Before it did, a cancelled socket read ended 489 to 491 milliseconds later, at a fallback close; wired into Asio’s own cancellation, it ends in 0.0 to 0.1 milliseconds, as Asio’s own coroutine does. The runs are in the cancellation status.

The table shows the cost of creating, awaiting and destroying the smallest task, next to asio::awaitable, and the median delay from yielding the loop to running again, at -O3 (from the README). task wins on Clang and loses the first column on GCC:

Machine, compilertaskasio::awaitableYield and resume: taskasio::post
Apple M4 Pro, Apple clang 179.2 ns10.2 ns42 ns42 ns
Intel Core i9-9980HK, Apple clang 1718.1 ns30.4 ns115 ns148 ns
Intel Core i9-9980HK, g++ 1656.4 ns23.9 ns90 ns122 ns

On the Intel with Clang, task is 40 percent cheaper than Asio’s coroutine. On GCC the trampoline costs 56 nanoseconds against Asio’s 24, and forcing the tail call brings it to 34, still behind. Across 20 million tasks it allocates nothing, because frames come from per-thread pools of recycled blocks.

Besides the deep chains, the tests cover affinity across threads, budget demotion, cancellation mid-wait, deadlines on a manual clock, and a stress run of a million tasks with random yields and cancellations. CI runs them with GCC and Clang on Linux (x86-64 and arm64), macOS and FreeBSD, plus address, undefined-behavior and thread sanitizer builds.

task has two neighbors. workpool owns workers, threads for blocking work like disk writes, shared fairly. reactor owns loops, one per core, with accepting and clean shutdown. task owns neither: it is the coroutine that runs on a loop the host owns. Its docs tie it to the pool only at shutdown: the owner first stops every loop and pool a suspended task may be waiting on, which in Xapiand means draining its WorkGroups, before any task frame is destroyed.

http is built on it now: a request handler is a task::task<void>, and each connection a task on its reactor’s loop. Reach for task when coroutines share a loop someone else runs, must give it back on a time budget, and must come home to the thread they started on.

Skip it for the fastest coroutines under GCC, where asio::awaitable wins, or when memory must stay on its thread: a frame freed elsewhere joins that thread’s pool. And an exception escaping fire-and-forget work (a spawn) ends the program.