[5SP] REST long polling: report backlog so clients can catch up

[drslebedev]

TLDR

Long-polling delivers subscription messages one at a time. After processing message N, the client requests N+1. In the WASM dashboard, drawing and Fetch callbacks share the main thread. Expensive drawing can therefore delay both processing a response and requesting the next message.

The server keeps only the latest 100 messages. If the client falls too far behind, its requested message is removed from the buffer. The server then moves the client past the remaining buffered messages, and the client reports WARNING_SAMPLES_DROPPED only after the messages have already been lost.

The proposed change adds the newest available message index to each response. By comparing it with the received index, the client can estimate its backlog. When the backlog grows, the dashboard can temporarily reduce or pause drawing and give Fetch callbacks time to catch up. If catching up is no longer practical, it can deliberately skip to the newest message.

The server change is small and backward compatible, but complete support also requires changes to the OpenCMW clients and OpenDigitizer. Because a proxy may return a cached response, the reported newest index can be stale. It is therefore a useful warning signal, not an exact backlog measurement. The longer-term solution is to move subscription networking to a dedicated event-driven worker.

Why this is needed

This is the transport-side companion to the issue fair-acc/opendigitizer#475.

Tests show that the server and network can handle the required message rate. A headless client using the same 18 subscriptions receives about 360 messages per second without losing data. Messages are dropped only when the dashboard also renders the plots.

In the WASM dashboard, plotting and Fetch callbacks run on the same main thread. While that thread is drawing, it cannot process a completed response or request the next message. The server continues publishing during this delay, so the client gradually falls behind.

The rendering issue reduces the cost of drawing. This issue lets the client detect a growing backlog and temporarily prioritize fetching before messages are removed from the server buffer. Together, the changes reduce normal rendering work and allow rendering to slow down when the client needs to catch up.

How long polling works today

Every published message has an increasing index. To receive message N, the client sends:

GET <topic>?LongPollingIdx=N

After receiving message N, it requests N+1. Only one long-poll request is active per subscription.

The server handles a requested index in one of three ways:

1. If the message is in the buffer, the server returns it with 200.
2. If the message has not been published yet, the server keeps the request open until it becomes available.
3. If the message is too old and has already been removed from the buffer, the server responds with a 302 redirect.

The redirect in the third case does not point to the oldest message still available. It points to the index after the newest buffered message, so the redirected request waits for the next publication.

For example, suppose the server buffer contains messages 151 through 250, but the client requests message 140. The server redirects it to 251 instead of returning 151. Messages 140 through 250 are therefore skipped, including the 100 messages that were still available.

The server keeps only the latest 100 messages. Before a requested index is removed, the client receives no information about how many newer messages are waiting. It notices the problem only when the received index jumps and it reports WARNING_SAMPLES_DROPPED.

The special values LongPollingIdx=Next and LongPollingIdx=Last are also resolved through redirects to concrete indices.

Implementation details are in RestServer.hpp. The buffer is implemented by SubscriptionCacheEntry, and an expired index is redirected to nextIndex(). Both RestClientNative.hpp and RestClientEmscripten.hpp set kParallelLongPollingRequests = 1; OpenDigitizer uses the Emscripten client.

The proposed change

OpenCMW Server: include the newest available index

When the server returns message N, it also knows the newest message currently in the subscription buffer. Call that index L.

Add this response header:

x-opencmw-long-polling-last-idx: L

For example, if the client receives message 190 with this header:

x-opencmw-long-polling-last-idx: 250

then at least 60 newer messages were available when the server created the response.

The header contains an absolute index rather than a backlog count.

This change does not modify the response body, message encoding, or long-polling sequence. Existing clients can ignore the header and continue working as before.

If the buffer capacity may change from its current value of 100, the server should eventually expose that capacity as well. The client needs both the backlog and the capacity to know how close it is to losing data.

OpenCMW clients: expose new information

The native and Emscripten clients should read the new header and pass its value to the subscription consumer together with the received message index.

The client can maintain the estimate as follows:

highestKnownIndex = max(highestKnownIndex, receivedIndex, headerIndex)
estimatedBacklog = highestKnownIndex - receivedIndex

Because a proxy may cache the header, estimatedBacklog is a lower-bound estimate rather than an exact measurement.

OpenDigitizer: reduce rendering when the backlog grows

OpenDigitizer should use the estimated backlog to control rendering:

• With a small backlog, fetch and render normally.
• When the backlog grows, reduce or pause rendering so Fetch callbacks can process responses and request subsequent messages.
• Resume normal rendering after the backlog falls below a lower threshold.
• If the client cannot catch up, optionally jump to LongPollingIdx=Last and report that data was deliberately skipped.

Limitation: the header may be cached

The deployment requires a caching proxy because the service cannot handle a separate long-poll request from every client. For each indexed URL, the first request reaches the service and the proxy reuses that response for the other clients.

The message body for index N is immutable and safe to cache. The last-index header is different: it describes the server state only when the response was first created. Because the proxy caches the complete response, later clients may receive an old header together with the correct message body.

For example, the first client may request message 190 immediately after it is published:

message index: 190
x-opencmw-long-polling-last-idx: 190

The proxy caches this response. A slower client may receive it later, when the server has already reached index 250, but it still sees 190 as the latest index. This can happen for every message, so the header may report a backlog of zero even when the client is falling behind.

Remembering the highest reported index prevents the estimate from moving backwards, but it cannot discover an index that has never appeared in a response. The header must therefore be treated as a best-effort warning, not an exact backlog measurement.

Why not request fresh state separately?

We considered adding a request such as:

GET <topic>?LongPollingIdx=State
Cache-Control: no-store

This could return the current first index, last index, and buffer capacity. However, no-store would prevent the proxy from sharing the response. Every client would reach the service independently, changing the required 1:N fan-out back into N origin requests. The service is not designed to handle that load, so this option cannot be used in the current deployment.

Can the client detect this on its own? (to investigate)

The client cannot know the exact backlog without fresh server information, but it can look for signs that it is falling behind:

• Messages are getting older. If the delay between a message's timestamp and the time it is processed keeps growing, the client is losing ground.
• The client is slower than the source. Use message indices and timestamps to compare the publication rate with the client's processing rate.
• Frames block the main thread for too long. Long frames give Fetch callbacks fewer opportunities to run.

The duration of a long-poll request is not a useful signal by itself because the server may intentionally hold the request while waiting for the next message. The time spent inside a callback also does not show how long that callback waited for the main thread.

The client should observe these signals over several messages rather than react to one slow response or frame. If message age keeps increasing or the client remains slower than the source, it can reduce rendering until it recovers. This approach should be tested with the existing 18-subscription workload.

Why Fetch currently runs on the main thread

Fetch is not required to run on the main thread. OpenCMW deliberately moved subscription Fetch requests there in commit d260d49 to fix subscriptions that could stall.

Previously, a subscription could start a Fetch request from a GR4 worker thread. An asynchronous callback on that worker can run only when the worker returns control to its JavaScript event loop.

The GR4 workers run long-lived C++ processing loops. Sleeping or waiting inside such a loop does not return control to the JavaScript event loop. The browser could therefore finish downloading a response while its callback remained unable to run:

worker starts Fetch
browser receives the response
worker remains inside its C++ loop
Fetch callback cannot run

Some subscriptions worked because they were started from the main thread during graph setup. Subscriptions started later from a GR4 worker could stall. The commit fixed this inconsistent behaviour by routing every subscription Fetch request to the browser's main runtime thread.

The main thread regularly returns to the browser event loop, so Fetch callbacks run reliably there. Each callback processes one response and starts the next long-poll request. The complete subscription chain therefore stays on the main thread.

The disadvantage is that the same thread also renders the UI/ImPlot. The browser may receive a response while a frame is being drawn, but it cannot run the callback until drawing finishes. The request for the next message is delayed by the same amount.

This also explains why a hidden tab often keeps up. Rendering is paused or heavily reduced, leaving the main thread available for Fetch callbacks. The network itself is not faster.

Reducing rendering when the client appears to fall behind is a mitigation. The structural fix is to move subscription networking to a dedicated event-driven worker. That worker must return control to its JavaScript event loop so Fetch callbacks can run; moving Fetch back to the existing GR4 processing workers would recreate the original problem. This design still needs to be prototyped and tested.

Discussed and declined

We considered returning several messages in one multipart/mixed response, using options such as LongPollingUpTo=N, LongPollingAll, and LongPollingExactly=N.

This was declined for two main reasons:

1. Shared proxy caching is required. A response for one concrete message index never changes, so the proxy can fetch it once and reuse it for every client. Responses such as LongPollingAll and LongPollingUpTo=N depend on when they are requested and cannot be shared in the same way.

2. Multipart adds complexity and deployment risk. The clients would need to decode several messages from one response and handle waiting, partial results, and cancellation. The deployment also depends on mandatory proxies that have not been tested with these multipart responses.

LongPollingExactly=N could be cacheable when its starting index and message count are fixed, but it still requires the additional multipart handling and creates more request variants.

The decision is to keep one immutable message per indexed URL. This section records the rejected multipart approach so it is not investigated again without new information about cache sharing or proxy compatibility.

Relation to the rendering issue

• Rendering issue (Performance optimization for charts rendering with many points. opendigitizer#475) - reduce rendering time, so the client stays responsive and falls behind less often. Reduces the symptom.
• This issue - help the client detect a growing backlog so it can pause or slow drawing and catch up. Reduces the risk of data loss when the client falls behind.