Skip to content

Ci speed a improvements#11

Draft
davidpcls wants to merge 20 commits into
auth_updates_from_tiledfrom
ci-speed-a-improvements
Draft

Ci speed a improvements#11
davidpcls wants to merge 20 commits into
auth_updates_from_tiledfrom
ci-speed-a-improvements

Conversation

@davidpcls

@davidpcls davidpcls commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Summary

it's been long enough I don't know if this is still useful or not. I think perhaps we need to break this down differently.

  • Reduce test runtime by reusing a module-scoped RE Manager in auth-focused suites (test_access_control.py, test_auth_api.py) where manager state is not intentionally mutated.
  • Enable parallel pytest execution in CI with pytest-xdist and --dist=loadfile.
  • Make test infrastructure xdist-safe by isolating worker resources:
    • dynamic per-worker HTTP server port
    • dynamic per-worker ZMQ control/info ports
    • worker-specific xprocess process name

Why

  • Auth test suites were paying repeated RE Manager startup/teardown costs per test, adding avoidable overhead.
  • Parallel execution previously risked cross-worker collisions on fixed ports and shared process names.
  • This change keeps behavior stable while unlocking meaningful CI speedups.

Implementation Details

  • Added re_manager_module fixture in bluesky_httpserver/tests/conftest.py and switched auth suites to use it.
  • Added queue clear in test_resource_access_01 to preserve deterministic behavior with shared manager lifetime.
  • Added worker-aware helpers in bluesky_httpserver/tests/conftest.py to assign unique ports and env vars per xdist worker.
  • Overrode re_manager fixture in local conftest.py to start ReManager on worker-specific ZMQ ports.
  • Updated xprocess server naming to include worker id to avoid worker lock/name collisions.
  • Added pytest-xdist to requirements-dev.txt.
  • Updated CI pytest options to: --durations=20 -n auto --dist=loadfile.

Validation

  • uv run pre-commit run --all-files
  • uv run pytest bluesky_httpserver/tests/test_auth_api.py -q
  • uv run pytest bluesky_httpserver/tests/test_access_control.py -q
  • uv run pytest -q -n 2 --dist=loadfile bluesky_httpserver/tests/test_auth_api.py bluesky_httpserver/tests/test_core_api_main.py -k "test_http_server_start_01 or test_http_server_ping_handler or test_api_auth_post_apikey_01"

Expected Impact

  • Lower per-run overhead from auth suites due to shared manager startup.
  • Additional CI wall-time reduction from parallelized test execution on GitHub runners.

I didn't run it beforehand. Oops.
These seem to resolve the LDAP server issues locally, pushing the
changes to see if that fixes them remotely as well.
This migrates to pyproject.toml which better supports the pixi and uv
tools. Pip should work just fine as well.

This removes the "python 2" support in setup.py but that was removed a long time ago
already.
There was an unused import.
Use a local re_manager_cmd fixture with worker-scoped ZMQ/Redis settings and route tests to it to prevent cross-worker collisions and startup cascades. Also switch queue reset calls to zmq_secure_request so control requests consistently target the active manager.
@davidpcls
davidpcls marked this pull request as draft July 22, 2026 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant