A small and lean coroutine library for Envoy

[penguingao]

Commit Message: A minimum library to support coroutine in Envoy, integrating with Envoy's dispatcher and siloed architecture.

C++20 brings language / compiler support of coroutine, but it doesn't come with STL support, due to the lack of standard runtime / event loop in the language. There are a few libraries that have been considered before writing this library, but they either bring in heavy dependencies or a foreign event loop. Given that Envoy's code is mostly siloed, there is no requirement for the library to worry about locking. This enables a relatively small and lean library like this PR. For any existing library to work with Envoy, custom integration with Envoy's Dispatcher is required any way, which is of similar effort, but brings in more dependencies into Envoy.

C++26 will bring STL support, so we can get rid of this library when it happens. https://cppreference.com/cpp/execution / std::execution::task

The gist of coroutine is that it allows a function / method to suspend execution onceco_await or co_yield is encountered, and return the thread's control to the outer most caller. The inner most function that's suspended can later be resumed once certain condition is met. This enables sequatial-style code for asynchronous operations, which is generally more readable.

This PR includes the following utilities:

• A Task<> template that is the return type of all Envoy coroutines;
• A LeafAwaitable<> template that allows coroutines to co_await on callbacks;
• A launch() method to allow callback to call into coroutine and receive the result via a done callback;

This allows writing pieces of Envoy code in coroutine when appropriate without a major refactor.

I used AI to generate some of the code, and I fully reviewed and understood the code.

Additional Description: https://gist.github.com/penguingao/21b22bec816be0c833c4d92c866a89cf contains the motivation and some design choices.
Risk Level: low - new unused library
Testing: unit tests
Docs Changes: n/a
Release Notes: n/a
Platform Specific Features: n/a

[jwendell]

We are going to bump LLVM to 22 (from 18), hopefully soon. Is this bump somehow going to affect - positively or negatively - this effort?

[penguingao]

We are going to bump LLVM to 22 (from 18), hopefully soon. Is this bump somehow going to affect - positively or negatively - this effort?

Should be a positive change - LLVM 22 further matures C++20 support, while starting to bring in C++26. This PR is only using C++20 features, which are already well supported in LLVM 18.

[botengyao]

thanks @penguingao! flushing some questions.

also would love get @kyessenov @ravenblackx @ggreenway 's input on this direction.

[penguingao]

Thanks @botengyao for the review!

I'd like to also clarify that this library is to incrementally allow new code to be written in coroutine. It doesn't aim at rewriting core Envoy.

The main reason we introduced a launch() with a done callback is to make it easy to call into a coroutine from normal Envoy methods.

That said, if we do find coroutine being a successful attempt, we can always use it to improve existing things incrementally.

[penguingao]

thanks @penguingao! flushing some questions.

also would love get @kyessenov @ravenblackx @ggreenway 's input on this direction.

Happy to discuss and clarify things.

[penguingao]

/retest

[botengyao]

here is another pass, my take here is we can check in this lib to experiment.

/wait

[penguingao]

I also added a microbenchmark to make sure that coroutine's performance is comparable. It's an echo server that does at most one read and one write per epoll loop. Realistic because we don't want to hold the epoll for a single call stack.

The test send 1MB of data through the echo server, and we vary on reader and writer bottleneck to simulate real world. We try different buffer and batch sizes.

On my dev VM, I ran the benchmark with --benchmark_repetitions=5 --benchmark_report_aggregates_only=true, and in general, coroutine is just slightly faster / uses slightly less CPU.

The table here summarizes the relative CPU usage for sending 1MB over different conditions.

pengg@envoy-dev:~/envoy$ cat /tmp/all | ./test/common/coroutine/aggregate_perf_test_stats.awk 
test_param                   cb_cpu        coro_cpu       co/cb
balanced-1024-1024          4660 µs         2940 µs      63.09%
balanced-4096-1024          2424 µs         2243 µs      92.53%
balanced-32768-1024         2380 µs         2190 µs      92.02%
balanced-65536-1024         2347 µs         2234 µs      95.19%
balanced-1024-4096          3201 µs         3028 µs      94.60%
balanced-4096-4096          2333 µs         2225 µs      95.37%
balanced-32768-4096         2481 µs         2206 µs      88.92%
balanced-65536-4096         3009 µs         2246 µs      74.64%
balanced-1024-32768         3119 µs         3014 µs      96.63%
balanced-4096-32768         2272 µs         2291 µs     100.84%
balanced-32768-32768        2251 µs         2261 µs     100.44%
balanced-65536-32768        2277 µs         2248 µs      98.73%
balanced-1024-65536         3129 µs         3006 µs      96.07%
balanced-4096-65536         2212 µs         2234 µs     100.99%
balanced-32768-65536        2189 µs         2181 µs      99.63%
balanced-65536-65536        2223 µs         2242 µs     100.85%
slow_reader-16384-1024      2264 µs         2206 µs      97.44%
slow_reader-16384-4096      2232 µs         2216 µs      99.28%
slow_reader-16384-32768     2291 µs         2194 µs      95.77%
slow_reader-16384-65536     2176 µs         2191 µs     100.69%
slow_writer-4096-1024       2590 µs         2602 µs     100.46%
slow_writer-32768-1024      2415 µs         2367 µs      98.01%
slow_writer-65536-1024      2468 µs         2394 µs      97.00%
slow_writer-4096-4096       2597 µs         2589 µs      99.69%
slow_writer-32768-4096      2451 µs         2344 µs      95.63%
slow_writer-65536-4096      2422 µs         2411 µs      99.55%
slow_writer-4096-32768      2527 µs         2598 µs     102.81%
slow_writer-32768-32768     2340 µs         2376 µs     101.54%
slow_writer-65536-32768     2360 µs         2402 µs     101.78%
slow_writer-4096-65536      2551 µs         2543 µs      99.69%
slow_writer-32768-65536     2434 µs         2341 µs      96.18%
slow_writer-65536-65536     2446 µs         2387 µs      97.59%
--------
balanced_total             42507 µs        38789 µs      91.25%
slow_reader_total           8963 µs         8807 µs      98.26%
slow_writer_total          29601 µs        29354 µs      99.17%
total                      81071 µs        76950 µs      94.92%
std_dev                     3741 µs         1166 µs      31.17%

[botengyao]

lgtm, thanks!