Add Dask Gateway support to lc run (JupyterHub deployments)#159
Draft
EiffL wants to merge 1 commit into
Draft
Conversation
cluster_for_run gains a fourth branch, selected when a Gateway
environment is ambient (DASK_GATEWAY__ADDRESS, as on a lightcone
JupyterHub pod) or a cluster is named via LIGHTCONE_GATEWAY_CLUSTER.
Gateway scheduler addresses use a gateway:// comm scheme that a bare
distributed.Client cannot dial, so the parent/child rendezvous changes
shape: cluster_for_run now yields the env overlay the child snakemake
needs ({DASK_SCHEDULER_ADDRESS: ...} on existing branches,
{LIGHTCONE_GATEWAY_CLUSTER: name} on the new one), and the executor
rejoins the cluster by name through the authenticated Gateway API —
the same rendezvous-by-ambient-context pattern as the SLURM branch.
Ownership mirrors the address branch: clusters we create are adaptive
(minimum=1, bounded by --jobs) and shut down on exit; clusters named
by the user (e.g. created from the JupyterLab Dask sidebar) are left
running with their scaling untouched. Startup fails fast when gateway
workers don't advertise the cpus/memory/gpus resource contract, which
otherwise hangs every task silently.
Also: a lightcone-hub site entry detected via env markers (pods have
generated hostnames, so detect_current_site now checks declared env
markers before hostname patterns), and a [gateway] optional extra for
the dask-gateway client.
Validated against the lightcone-hub GKE staging deployment; design
notes and the deployment procedure live in
docs/design/jupyterhub-dask-gateway-gcp.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
lightcone-cli | da571bc | Commit Preview URL Branch Preview URL |
Jul 12 2026, 12:21 PM |
✅ Eval Results
Graders✅ spec_valid (1.00) Full output |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
lc runnow works on Dask Gateway deployments (our JupyterHub-on-GKE playground) with zero user configuration: on a hub pod,cd /shared/<project> && lc runrequests an adaptive Gateway cluster, runs the pipeline on the spot pool, and shuts the cluster down afterwards. SettingLIGHTCONE_GATEWAY_CLUSTER=<name>instead attaches to an existing cluster (e.g. one created from the JupyterLab Dask sidebar, dashboard panels docked) and leaves it running on exit.Why the rendezvous changes shape
Gateway scheduler addresses are not dialable: they use a custom
gateway://comm scheme (traefik multiplexing + per-cluster TLS + SNI) that only thedask-gatewayclient library understands — verified on the staging cluster (unknown address scheme 'gateway'from a bareClient). Sincelc runand the executor plugin live in different processes, the fix is the same rendezvous-by-ambient-context pattern the SLURM branch already uses: the parent passes the cluster name through the env, and the executor rejoins by name through the authenticated Gateway API.Concretely,
cluster_for_runnow yields the env overlay the child snakemake needs ({"DASK_SCHEDULER_ADDRESS": ...}on existing branches,{"LIGHTCONE_GATEWAY_CLUSTER": name}on the new one) instead of a bare address string — making the actual parent→child contract explicit.Changes
engine/dask_cluster.py— fourth branch (priority: explicit address → gateway → SLURM → local): creates an adaptive cluster (minimum=1, bounded by--jobs) or attaches by name; owned clusters are shut down on exit, attached ones untouched — mirroring the address-branch convention. Fails fast when workers don't advertise thecpus/memory/gpusresource contract (otherwise every task hangs with no error).snakemake_executor_plugin_dask/executor.py— client resolution extracted to_connect_client(): Gateway name → rejoin via Gateway API; elseClient(DASK_SCHEDULER_ADDRESS)as before.engine/site_registry.py—lightcone-hubsite entry detected via env markers (JUPYTERHUB_USER+DASK_GATEWAY__ADDRESS; pods have generated hostnames), checked ahead of hostname patterns indetect_current_site.pyproject.toml— optional extralightcone-cli[gateway]; mypy override fordask_gateway.docs/design/jupyterhub-dask-gateway-gcp.md.Testing
/sharedRWX + flock, hub-proxied gateway) was validated end to end on thelightcone-hubGKE staging cluster. End-to-endlc runon the hub additionally needs the worker image (lc+snakemakebaked in) — tracked in lightcone-hub.🤖 Generated with Claude Code