[Results] Latency and proxy behaviour

[drslebedev]

This proof of concept compares OpenCMW's current indexed long-poll with a minimal
streaming implementation for continuous updates, and tests whether an nginx proxy
can share one server response across many clients (fan-out) to reduce load on service. Both
transports are measured directly, and long-poll is also measured through the nginx proxy.
Everything runs on one machine, so the numbers are software and proxy overhead, not real
network latency - the useful signal is the comparisons (transport vs transport, direct vs
proxy, 1 vs 20 clients).

What was tested

Two transports

• Long-poll - using the Opencmw-cpp.
• Streaming - a small PoC: the server pushes a continuous sequence of framed messages down one long-lived HTTP/2
  response, and the client parses the frames. It is deliberately minimal - just enough to measure streaming latency and
  compare it against long-poll.

Metric - event latency = the time a client received a message ? the timestamp the server stamped when it produced
that message. This is only meaningful because client and server share one machine (one clock).

Matrix - each transport direct, and long-poll also through nginx, with 1 and 20 concurrent clients, plus one run
under 80% CPU load. Every run: 180 s measurement, 10 s warm-up, 100 messages/s, HTTP/2, localhost, zero sequence loss.

Results

For 20-client rows, percentiles combine every recorded event and rate is the average per client. Loss counts observed
sequence gaps.

Test	Clients	p50 ms	p90 ms	p99 ms	Max ms	Rate/client	Loss
Long-poll, direct	1	0.404	0.547	0.737	1.674	98.71/s	0
Long-poll, via nginx	1	0.272	0.364	0.491	1.035	98.88/s	0
Long-poll, direct	20	0.785	1.089	1.256	2.847	98.75/s	0
Long-poll, via nginx	20	237.453	447.868	495.241	501.141	98.92/s	0
Streaming, direct	1	0.149	0.216	0.429	2.442	100.00/s	0
Streaming, direct	20	0.465	0.770	0.945	3.143	100.00/s	0
Long-poll, direct, 80% CPU	1	0.250	1.154	2.834	6.898	96.92/s	0

All tests were completed successfully and reported zero sequence gaps.

Each plot is a latency histogram (blue) with the tail P(latency > x) overlaid in red on a
log axis; the dashed green line marks p99.

Long-poll, 1 client - direct vs through nginx

Both paths stay sub-millisecond. With one client there is no cache-lock wait, and the
proxy adds no meaningful latency in this local test.

Long-poll, 20 clients - direct vs through nginx

Direct stays around 1 ms. Through nginx, p50 is 237 ms and the tail reaches 500 ms
because of the cache lock. The rate stays at 99/s with zero loss.

Streaming, 1 and 20 clients (direct)

Streaming showed the lowest measured latency in this PoC: p50 is 0.15 ms with one
client and 0.47 ms with 20 clients, with zero loss.

Long-poll under 80% CPU load (1 client)

CPU load keeps the median low but increases p99 from 0.74 to 2.83 ms, with a maximum of
6.9 ms.

Current direction

Streaming showed the lowest measured latency in this PoC, but the current indexed
long-poll path already provides sub-millisecond single-client latency and works with nginx
caching and fan-out. An endless stream cannot be shared through the nginx cache, so every
client needs its own connection to the origin.

For now, long-poll is a good practical choice. Its networking can be moved to a worker
thread so that it does not block the main application thread; this is tracked in
opencmw-cpp#405. Streaming remains
an option if future latency or load requirements justify it. The overall investigation is
tracked in opendigitizer#462.

The fan-out test (long-poll)

We tested fan-out with 30 clients requesting the same five message indexes. nginx sent only
5 requests to the OpenCMW server instead of 150: one server response per index was
shared by all clients. This confirms the approach proposed in
opendigitizer#462.

The trade-off is latency. With proxy_cache_lock, waiting clients may receive the response
up to about 500 ms later because of nginx's fixed cache-lock timer. In the 20-client test,
the median was 237 ms and no messages were lost. Reducing proxy_cache_lock_timeout may
lower this delay, but can also cause more requests to reach the server.

This setup requires nginx 1.29.4 or newer for HTTP/2 connections to the OpenCMW server.
The tests used nginx 1.30.3.

Required OpenCMW fix

A continuous dashboard must survive HTTP connection replacement. nginx closes a connection
after a configurable number of requests (1000 by default). Chrome opened a new connection
automatically and continued, but the native OpenCMW client stopped.

This is an OpenCMW client issue and should be fixed there. The benchmark uses a higher
request limit only to allow longer tests. A separate opencmw-cpp issue should be created
to track automatic reconnection.

Test scope

The table contains selected latency results from native clients running on the same
machine. The browser client was also tested directly and through nginx, including
connection replacement, and worked correctly. Other rates and configurations were tested
during the investigation but are omitted here to keep the summary focused.

These benchmark tests could later be considered for inclusion in the opencmw-cpp
repository.