Containerized Web-App Deployment

A reusable ParallelWorks workflow + Singularity/Apptainer container recipe for serving any interactive web application from an HPC resource — extracted and generalized from a production app-specific workflow.

You get:

File	What it is
workflow.yaml	The PW workflow: pull container → run server → expose as web endpoint or VNC desktop
app.def	Multi-stage container recipe: build with source, ship only the production bundle
app/	A tiny self-contained example app (plain Node, zero deps) proving the contract

---

The topology

setup ─┬─ get_ports ─┬─ pull_software ─┬─ run_app (job_runner) ── expose_session
       └─────────────┴─ pull_data ─────┘

• setup — create the workdir; kill anything a previous launch left running (pidfile).
• get_ports — pw agent open-port auto-selects a free port (or use a fixed one).
• pull_software — pull the read-only app.sif from a bucket on the head node (worker nodes may lack outbound HTTPS). Cached; re-pulled when the image name input changes or refresh is toggled.
• pull_data (optional) — stage + unpack a data package up front, before any scheduler submission, so unpacking never burns scheduler walltime.
• run_app — generates run_app.sh and launches it via marketplace/job_runner (login node or PBS/Slurm). The script forks the app server — and, for web sessions, the pw endpoints http tunnel — detached with setsid.
• expose_session — waits until the app answers HTTP, then either reports the endpoint URL (web) or creates a VNC desktop and opens Firefox on it (desktop).

The five patterns worth stealing

1. Build in the container — ship no source

app.def is multi-stage: the build stage has the source tree and toolchain and produces dist/; the final stage copies only dist/. The published .sif contains the runnable production bundle — no source code, no node_modules, no compilers. App teams can build "in prod" and distribute only the artifact.

This works because nothing session-specific is baked at build time: the app binds 0.0.0.0:$PORT (env, chosen per launch) and serves at base path / (custom-subdomain endpoints need no path prefix). One image serves every session.

2. Read-only container at runtime

Because the bundle is prebuilt, the cluster never builds anything: no --sandbox expansion, no --fakeroot, no --writable-tmpfs overlay chains. Just singularity run --env PORT=$PORT app.sif. Faster launches, fewer failure modes.

3. Fork under job_runner so services outlive the workflow

Background processes forked from an SSH step are reaped when the workflow completes. Processes forked (setsid + nohup) inside the job_runner-launched script survive. So the server and the web endpoint are forked in run_app.sh, and expose_session only reports — letting the workflow reach Completed while the app keeps serving.

4. Pidfile lifecycle — no dangling processes

Every forked PID is appended to app-custom.pids. The next launch's setup job kills those PIDs (and their process groups) before starting fresh. Launches are idempotent; nothing accumulates.

5. Two exposure modes from one workflow

• Web: pw endpoints http --name X --subdomain X PORT → root URL at a custom subdomain. No agent on the client, no path rewriting.
• Desktop: pw sessions create --type desktop → detect your own newest Xvnc process for the real DISPLAY + -auth cookie (a shared login node runs many users' X servers — grabbing the first one fails with "Authorization required"), wait for X to actually accept connections, then launch Firefox (optional --kiosk for fullscreen).

Quick start

# 1. Build the container (root or --fakeroot needed for build only)
cd generic-interactive-app
apptainer build app.sif app.def          # or singularity build

# 2. Smoke-test locally
apptainer run --env PORT=8080 app.sif &
curl http://localhost:8080/api/health    # {"ok":true,...}

# 3. Publish it
pw buckets cp app.sif pw://<user>/<bucket>/app.sif

# 4. Import workflow.yaml as a PW workflow, pick your bucket in the
#    Container Bucket selector, and Execute.

No local build environment? Build on the resource instead

The workflow can build the image for you on the cluster's head node — publish the recipe instead of the image and toggle Build Container on Resource:

tar -czf app-recipe.tar.gz app.def app/
pw buckets cp app-recipe.tar.gz pw://<user>/<bucket>/app-recipe.tar.gz

The build is cached (rebuilt only when the recipe package name changes or refresh is toggled), and Push Built Image to Bucket uploads the resulting .sif back to the bucket so future launches — or other users — pull the prebuilt image instead of rebuilding. Requires --fakeroot build capability and outbound network on the head node; prefer CI-/locally-built images for production.

E2E test mode (CI)

Set Session Method → E2E test and the workflow becomes a self-contained smoke test: it pulls (or builds) the container, launches the server through the exact same job_runner path as a real session, asserts the app answers HTTP 200 at /, then tears down every forked process and exits. No session, no endpoint, nothing left running — workflow success is the end-to-end pass. Pairs well with workflow-tester-tool for automated regression runs.

Adapting to your app

1. Replace app/ with your source tree and change the %post build command in app.def (npm ci && npm run build, pip install . && pyinstaller, etc.).
2. Keep the runtime contract: final stage serves HTTP on 0.0.0.0:$PORT at /. That's the only thing the workflow assumes about your app.
3. If your client-side code calls a second backend service directly (separate port), route it through the frontend server instead — the browser can only reach the exposed port. (This was a real-world lesson: an SSR server also cannot call the app's own public endpoint URL; keep in-node traffic on localhost.)
4. Optional data? Point Data Bucket at a tar.gz package; leave it empty to skip.

Known caveats

• pw endpoints http requires the platform's minimum pw CLI version on the cluster; older CLIs get evicted with displaced-below-minimum-version. Update pw on the resource if endpoints drop.
• run_app.sh runs under job_runner, which may have a different PATH than SSH steps; the script resolves pw with command -v pw and the expose step warns if the endpoint log never appears.
• Desktop mode assumes the app and the VNC desktop share a node (true for login-node runs; scheduled runs put the app on a compute node — use web mode there, or a browser on the desktop pointed at the compute node's host:port if routable).