Add paging for ControlConnection topology queries#637
Conversation
|
Dear @copilot - I hope you've recovered from your illness, please resume working on this PR. |
I've completed the implementation! The changes add Key changes in commit faa59e6:
|
| pass | ||
|
|
||
|
|
||
| def _fetch_remaining_pages(connection, query_msg, timeout): |
There was a problem hiding this comment.
add optional fail_on_error, copy default from connection.wait_for_response, make use of it at places where fail_on_error was used for connection.wait_for_response calls
There was a problem hiding this comment.
Implemented in commit a4a0155. Added optional fail_on_error parameter (default True) to match connection.wait_for_response. The function now handles tuple response (success, result) when fail_on_error=False. Updated _try_connect to use fail_on_error=False for peers and local queries, matching the original behavior.
| try: | ||
| # Fetch all pages (system.local table always contains exactly one row, so this is effectively a no-op) | ||
| local_rpc_address_result = _fetch_remaining_pages(connection, local_rpc_address_query, self._timeout) |
278b17d to
e91f79e
Compare
e91f79e to
c92a664
Compare
53448bb to
4ec6d78
Compare
The ControlConnection now fetches all pages for system.peers and system.local queries using a new _fetch_all_pages helper. This ensures large clusters with many nodes complete topology discovery rather than only reading the first page. This replaces the old pattern of making a single query via wait_for_responses with sequential _fetch_all_pages calls via wait_for_response. Fixes #620
4ec6d78 to
83e458c
Compare
Problem
Topology queries (
SELECT * FROM system.peers,SELECT * FROM system.local) in ControlConnection are sent without a fetch_size, making them unpaged. In Scylla these show up as scylla_cql_unpaged_select_queries_per_ks metric bumps, and on clusters with >1000 nodes only the first page of peers is read — the rest are silently lost.Changes
New
_fetch_all_pages()standalone function — fetches all pages for a query by following paging_state in a loop, then reassembles all rows into a single ResultMessage. Handles both fail_on_error=True (raises on error) and fail_on_error=False (returns (success, result) tuple) calling conventions. The function saves and restores query_msg.paging_state around its loop so the caller's message object is never left dirty._try_connect() — added fetch_size=self._schema_meta_page_size to both peers_query and local_query. Replaced connection.wait_for_responses(peers, local, ...) with separate _fetch_all_pages(peers) and _fetch_all_pages(local) calls. Local is checked first (must succeed), then peers. The peers v1 fallback path also uses _fetch_all_pages.
_refresh_node_list_and_token_map() — same pattern: fetch_size on both queries, sequential _fetch_all_pages instead of wait_for_responses.
Testing
Fixes #620