diff --git a/.claude/plans/openvpn-google-revocation-HANDOFF.md b/.claude/plans/openvpn-google-revocation-HANDOFF.md new file mode 100644 index 0000000..5aa5894 --- /dev/null +++ b/.claude/plans/openvpn-google-revocation-HANDOFF.md @@ -0,0 +1,319 @@ +# Session handoff: OpenVPN cert revocation for deactivated Google users + +Written before an IDE restart to preserve full context. Companion to the +design doc [`openvpn-google-user-revocation.md`](openvpn-google-user-revocation.md). +Date: 2026-07-19. + +## The goal (recap) + +Periodically check Google Workspace and **revoke the OpenVPN certificate** of +any user who has been suspended/deleted. Closes the gap where a deactivated +user can't mint a new cert (portal blocks login) but their already-downloaded +`.ovpn` keeps working until the CA revokes it. + +## Key facts established (don't re-derive) + +- **Certs are minted by the portal**, not Puppet: + `terraform-aws-openvpn/portal/portal/__init__.py` -> `ensure_certificate()` -> + `easyrsa gen-req/sign-req` with `EASYRSA_REQ_CN = `. + So **CN == Google email**. Confirmed with the user. +- PKI lives on shared **EFS** at `/pki/{issued/.crt, + private/.key}`; `pki/index.txt` is the source of truth (`V`=valid, + `R`=revoked). +- Revocation primitive already used by the monthly CRL cron in + `puppet-code .../openvpn_server/config.pp:178` (`easyrsa revoke` + `gen-crl`, + `EASYRSA_PASSIN=file:/ca_passphrase`). OpenVPN re-reads CRL per + connection, no restart. +- Reconciliation is a pure set-diff: valid CNs in index.txt MINUS active + Google users = certs to revoke. + +## Decisions made + +1. **Auth approach chosen: Workload Identity Federation (WIF), NOT a stored SA + key.** User's words: "SA sounds like fucking 2012." No service-account key + at rest anywhere. +2. The DWD-without-key mechanism is **`iamcredentials.signJwt`** (Google signs + the domain-wide-delegation assertion server-side). Requires the federated + principal to hold BOTH `roles/iam.workloadIdentityUser` AND + `roles/iam.serviceAccountTokenCreator` on the SA. +3. The one step that **cannot** be Terraformed (no resource in + `hashicorp/google`): authorizing the SA's client ID + directory scope in the + Workspace Admin console (Security -> API controls -> Domain-wide delegation). + Reduced to pasting one client ID; emitted as a TF output. +4. Whole feature gated behind `enable_google_directory_revocation` (default + false) so existing module consumers are unaffected. + +## Files I created / changed this session + +### puppet-code (this repo) +- `.claude/plans/openvpn-google-user-revocation.md` — the design doc. +- `.claude/plans/openvpn-google-revocation-HANDOFF.md` — this file. +- **No manifests written yet.** The Puppet side (a + `profile::openvpn_server::google_revocation_sync` class: Python sync script + + daily cron + flock lock) is still TODO. + +### terraform-aws-openvpn (`/Users/aleks/code/terraform/terraform-aws-openvpn`, +symlink of `.../infrahouse/terraform/terraform-aws-openvpn`) +- `terraform.tf` — added `google` provider (`~> 6.0`) to required_providers. +- `google-wif.tf` — **the main deliverable.** Stands up the GCP side: + `google_project_service` (sts/iamcredentials/iam/admin), keyless + `google_service_account.dir_reader`, `google_iam_workload_identity_pool` + + `_provider.aws` (with ARN-normalizing attribute-mapping CEL + attribute + condition locked to the instance role), the two `google_service_account_iam_member` + bindings, the rendered keyless cred-config (`local.wif_credential_config`), + plus feature variables and outputs. + - NOTE: a linter reformatted this file (interpolation -> `format()`/`join()`); + functionally identical, keep it. +- `spikes/dwd-wif/` — copied out of the ephemeral scratchpad so it survives: + `dwd_wif_spike.py` (4-stage end-to-end proof of the AWS->GCP->DWD->Directory + chain), `README.md` (gcloud/console setup + run instructions), `requirements.txt`. + `py_compile` passes; needs `google-auth`+`requests` and a mapped EC2 role to run. + +## Files present that I did NOT create (review on resume) + +Appeared during the session (timestamps 07:50-08:39), likely parallel work: +- `test_data/google_wif/` — a root harness consuming the module with the + feature enabled. **Has a live ~690KB `terraform.tfstate` (modified ~08:35-08:39) + — an apply appears to have actually run against real AWS+GCP.** Check whether + resources are still up and need `terraform destroy`. +- `tests/test_google_wif.py` — pytest integration test asserting the keyless + federation was created (SA has no user-managed keys, pool+provider ACTIVE, + both IAM bindings present). Uses ADC; skips if no ADC/project. +- Other modified files in `git status` I didn't touch: `.bumpversion.cfg`, + `Makefile`, `README.md`, `docs/getting-started.md`, `requirements.txt`, + `test_data/openvpn/.gitignore`, deleted `test_data/openvpn/nul`. + +## Verification status + +- `terraform fmt` — clean on my files. +- `terraform validate` — NOT completed by me (needs `terraform init` against the + private `registry.infrahouse.com`; errored on module-not-installed before + reaching my code). The presence of an applied `test_data/google_wif` state + suggests it did init+apply successfully elsewhere — confirm. + +### 2026-07-19 (later session): WIF chain VERIFIED END-TO-END ✅ + +Ran `sudo /opt/openvpn-wif/verify-wif.sh` on live instance `i-0bb32c31f4bf8d62f` +(t3a.small, us-west-2a, role `openvpn-xCxFB8oEqdFa0e86148bc5db9c6fc9b2b26ad5`) +over SSM. **All four tiers PASS.** The keyless AWS->GCP->DWD->Directory chain +works for real. Do not re-litigate this. + +- Tier 1 instance ARN matches the provider's locked attribute_condition. +- Tier 2 federated token minted from the EC2 role, no key. +- Tier 3 SA impersonation works => both IAM bindings + (`workloadIdentityUser` + `serviceAccountTokenCreator`) are effective. +- Tier 4 Directory API read succeeded as subject `aleks@infrahouse.com`. + +**The manual Workspace DWD authorization is ALREADY DONE** — client id +`110312795661293035653` is authorized for +`.../auth/admin.directory.user.readonly`. This was listed as a pending blocker +above; it is not one. (Next step #3 in the old list is complete.) + +Tier 4 returned `[]` for `isSuspended=true`; a follow-up probe listing all users +returned 3 real accounts (aleks/anton.naumenko/dmytro @infrahouse.com, none +suspended), confirming `[]` was a true negative and not a silent auth failure. +**Consequence for testing the Puppet side: there is currently no suspended user +in the directory to exercise the revocation path.** Either suspend a throwaway +account or unit-test the set-diff against a fixture. + +### Open question #1 — ANSWERED (wiring) + +Resolved *not* via `custom_facts` but via `extra_files`: `asg.tf` now does +`extra_files = concat(var.extra_files, local.wif_extra_files)`, gated so the list +is `[]` when the feature is off. New supporting files (created after the original +handoff was written): `templates/wif.env.tftpl` and `scripts/verify-wif.sh`. +Confirmed working — cloud-init delivered all three files to `/opt/openvpn-wif/` +(`google-wif.json`, `wif.env`, `verify-wif.sh`) on the live instance. + +`wif.env` carries no secret; it exports `GOOGLE_APPLICATION_CREDENTIALS`, +`WIF_SA_EMAIL`, `WIF_SA_CLIENT_ID`, `WIF_EXPECTED_ROLE_ARN`, `WIF_ADMIN_SUBJECT`. +**The Puppet revocation script should source `/opt/openvpn-wif/wif.env`** rather +than re-plumbing any of these through Hiera. + +Python note: the google libs live in the infrahouse-toolkit embedded venv +(`/opt/infrahouse-toolkit/embedded/bin/python3`), NOT system python3 — the distro +`python3-google-auth` is 1.5.x and lacks `external_account`/`impersonated_credentials` +entirely. The revocation script must use the embedded interpreter. + +## Open questions for the user + +1. Wire the cred-config JSON + SA email + admin subject into `asg.tf` + `custom_facts` (gated) so the instance actually receives them? (Deferred — + couples to the not-yet-written Puppet side; mind the 16KB userdata limit, + already gzipped.) +2. Revoke immediately on suspension vs. a grace period (Hiera knob + `profile::openvpn_server::revocation_grace_days`, default 0)? + +## Next steps (resume here) — updated 2026-07-19 post-verification + +Steps 3 (DWD authorization + prove the chain) and the open question #1 wiring are +DONE; see the verification section above. Remaining: + +1. **Write the Puppet `profile::openvpn_server::google_revocation_sync`** (script + + daily cron + flock). This is now the critical path — the auth substrate it + depends on is proven. Source `/opt/openvpn-wif/wif.env`; use the toolkit + embedded python; reuse the `easyrsa revoke` + `gen-crl` pattern already at + `openvpn_server/config.pp:178` (`EASYRSA_PASSIN=file:/ca_passphrase`). +2. Decide open question #2 (immediate revoke vs. `revocation_grace_days`). +3. `terraform init && terraform validate` in terraform-aws-openvpn. +4. Update `openvpn-google-user-revocation.md` to reference the WIF Terraform + instead of the old SA-key approach (the plan still describes Option A). +5. Housekeeping in terraform-aws-openvpn (see below). + +## Puppet side — WRITTEN 2026-07-19 + +Design confirmed with the user: three external dependencies (terraform-aws-openvpn, +infrahouse-toolkit, Google Workspace config). Puppet must not fail before they are +satisfied and must self-activate once they are. Achieved by gating at RUNTIME in a +wrapper script, not at compile time in Puppet — the class only ever writes a file +and a cron entry, so it cannot fail anywhere. + +Files — **in `environments/development/` only**, NOT in global `modules/`: +- `environments/development/modules/profile/manifests/openvpn_server/google_revocation_sync.pp` +- `environments/development/modules/profile/templates/openvpn_server/google-user-sync.sh.erb` +- wired into `environments/development/modules/profile/manifests/openvpn_server.pp` + behind the fact check + +**Promotion path (repo convention, see #278 -> #279 -> #281):** land in +`environments/development` first, then promote to `environments/sandbox`, then to +global `modules/` (= production, all environments). Originally written straight +into global `modules/` by mistake and relocated on 2026-07-19; global +`modules/profile/manifests/openvpn_server.pp` was reverted and is untouched. +Promoting = copying the two files to the next environment's +`modules/profile/{manifests,templates}/openvpn_server/` and adding the same +gated `if $facts.dig('openvpn', 'google_directory_revocation')` block to that +environment's `openvpn_server.pp`. + +### Gating: each dependency at the level it can be observed + +User directive: "why not pass a fact whether enable_google_directory_revocation +true or false? Then in puppet, configuration is trivial without playing with exit +codes?" Correct for dependency 1, and adopted — but it does NOT remove the exit +codes, because the three dependencies are observable at different times: + +| Dependency | Terraform knows? | Gate | +|---|---|---| +| 1. Feature enabled | yes | fact `openvpn.google_directory_revocation` -> Puppet `if` at COMPILE time | +| 2. Toolkit has subcommand | no (own release cycle) | wrapper probes `--help` at RUNTIME | +| 3. Workspace DWD authorized | no (human in a console) | sync exits 78, wrapper maps to "not ready" | + +Dependency 3 can never become a fact: authorizing the client id is a paste into +the Workspace admin console with no `hashicorp/google` resource behind it, so +`enable_google_directory_revocation` can be true for days before it is done. +Terraform side: `asg.tf` custom_facts now emits +`google_directory_revocation : var.enable_google_directory_revocation`. + +### Two bugs found by end-to-end testing (wrapper driving the real toolkit) + +1. **Wrapper did not pass `--config-dir`.** It checked the PKI under Puppet's + `$openvp_config_directory` but invoked the toolkit with no `--config-dir`, so + the toolkit silently used `/etc/openvpn` -- wrapper inspecting one PKI while + the toolkit reconciled another. Fixed: single `CONFIG_DIR` variable in the + template, used for both the index check and the invocation. NOTE the option is + group-level, so it must precede the subcommand: + `ih-openvpn --config-dir X sync-google-users`. +2. **`DefaultCredentialsError` was uncaught** -> traceback -> exit 1 -> the wrapper + would have reported a real failure and mailed cron daily. This is the exact + failure mode the whole design exists to avoid. Hit whenever the credential + config is missing/malformed or IMDS is unreachable. Fixed: translated to + `GoogleNotConfigured` -> 78. Regression test: + `test_default_credentials_error_becomes_not_configured`. + +Both were invisible to unit tests and only surfaced by running the rendered +wrapper against the actual installed CLI. Keep doing that. + +Decisions (user-confirmed): +- Sync logic goes in the **toolkit**, not Puppet: new `ih-openvpn sync-google-users`. + Puppet owns only gate + flock + cron. Keeps the set-diff unit-testable in CI and + reuses existing `cmd_revoke_client` / index.txt parsing. +- **Immediate revocation**, stateless set-diff. No grace period, nothing persisted. +- **NO dry-run / report-only mode.** User directive: "if + `enable_google_directory_revocation` is true, we enforce." A report-only stage was + designed and then explicitly cut. Do not reintroduce it. + - Consequence: no enforcement flag anywhere — no Hiera key, no custom fact, no new + Terraform variable, and `sync-google-users` takes no `--dry-run`/`--enforce`. + - It works because the WIF credential files only reach an instance when + `enable_google_directory_revocation` is true, so their presence (already checked + by the wrapper) *is* the enable signal. A second flag could only disagree with it. + - An intermediate design briefly had Terraform write an `openvpn.google_sync_enforce` + fact; obsolete, dropped with the dry-run. + +Exit-status contract the toolkit subcommand MUST honor (the wrapper depends on it): +- `0` ran fine; `78` (EX_CONFIG) Google side not configured yet -> wrapper treats as + "not ready", logs, exits 0; any other code -> real failure, wrapper exits non-zero + so cron mails. +- The subcommand takes **no mode flags**; invoked bare as `ih-openvpn sync-google-users`. + +Other notes: +- Lock file lives on the **EFS-shared config dir**, not /var/run, so it is + cluster-wide across the ASG (all instances mount the same PKI; concurrent + `easyrsa revoke` would race on index.txt). +- Cron minute defaults to `fqdn_rand(60)` so the ASG does not hit the Directory API + in lockstep. + +Verified (re-run after the dry-run removal): `puppet-lint --fail-on-warnings` clean +on all four module paths; ERB renders; `bash -n` clean; and the full gate matrix was +executed **on the live Ubuntu instance i-0bb32c31f4bf8d62f** (sandboxed copy, never +touching the real PKI). Every "not ready" state exits 0 silently; only a genuine +post-deployment failure exits non-zero. Confirmed against the *currently installed* +toolkit (which lacks the subcommand) -> exits 0 gracefully. + +Note on facts: `custom_facts` from Terraform land in +`/etc/puppetlabs/facter/facts.d/custom.json` and keep their JSON types (e.g. +`"openvpn_port": 1194` is consumed as a Puppet `Integer`). Not needed by this class +as built, but relevant if a fact is ever added. + +## Toolkit side — WRITTEN 2026-07-19 + +Repo `/Users/aleks/code/infrahouse/infrahouse-toolkit`, clean `main`, uncommitted. + +- `infrahouse_toolkit/cli/ih_openvpn/lib.py` (new) — `index_path`, `parse_index`, + `extract_common_name`, `is_user_certificate`, `valid_user_certificates`, + `revoke_client`. +- `infrahouse_toolkit/cli/ih_openvpn/cmd_sync_google_users/__init__.py` (new) — + the subcommand, `GoogleNotConfigured`, `EX_CONFIG = 78`. +- `cmd_revoke` refactored onto `lib.revoke_client`; `cmd_list` onto + `lib.index_path`. +- `--config-dir` added as a GROUP option (alongside `--easyrsa-path`) into + `ctx.obj`, consumed by all three subcommands. `revoke_client` takes it as a + required argument -- deliberately no default, so no caller can silently act on + `/etc/openvpn` while the operator pointed the group elsewhere. +- `requirements.txt`: added `google-auth ~= 2.0`, `google-api-python-client ~= 2.0`. + +**Safety: the server certificate must never be revocable.** The live PKI's only +entry is `V ... /CN=server` -- the OpenVPN server's own cert. A naive +"valid CNs minus active Google users" diff computes `{server}` and revokes it, +killing the VPN, and does so on the FIRST run of a fresh deployment before any +user enrolls. Two independent guards: `is_user_certificate()` requires an `@` in +the CN (portal-minted user CNs are Google emails; `server` has none), and an +empty directory response is refused outright rather than treated as "everyone was +deactivated". Tests: `test_valid_user_certificates_excludes_server`, +`test_never_revokes_the_server_certificate`. + +Verified: pylint 9.95/10 on the module, black + isort clean, full suite +297 passed / 4 skipped, and the rendered Puppet wrapper driving the real +installed CLI exits 78 cleanly with no traceback. + +## Open item for the toolkit work + +`cmd_revoke` uses `EASYRSA_PASSIN=file:/etc/openvpn/ca_passphrase`, but +`templates/openvpn_server/regenerate-crl.sh.erb` documents that `file:` fails with +OpenSSL 3.x in easy-rsa batch mode (OpenVPN/easy-rsa#692) and deliberately uses +`pass:$(cat ...)` instead. A direct openssl probe on the instance (3.0.13) showed +raw `file:` and `pass:` BOTH work, so the failure is in easy-rsa's batch handling, +not the openssl BIO — NOT reproduced and NOT cleared. Check this before relying on +`cmd_revoke_client` in the sync path, or the feature may compute a correct diff and +then silently fail to revoke. + +## Housekeeping / risks to raise with the user + +- **All the terraform work is uncommitted on `main`** — `google-wif.tf`, + `scripts/`, `templates/`, `tests/`, `spikes/`, plus modified `asg.tf`, + `terraform.tf`, `Makefile`, `README.md`, `docs/getting-started.md`, + `requirements.txt`, `.bumpversion.cfg`. Nothing is on a branch. Needs a branch + + PR before it can land. +- **`test_data/google_wif/` has a live applied stack** (~730KB tfstate): full + OpenVPN ASG + NLB + EFS + Route53 + the GCP WIF resources, running in + us-west-2 and billing. Instance `i-0bb32c31f4bf8d62f` launched 2026-07-19 + 20:02 UTC. Ask before destroying — it is the verification rig, and tearing it + down forfeits the ability to test the Puppet side against a real WIF instance. \ No newline at end of file diff --git a/.claude/plans/openvpn-google-user-revocation.md b/.claude/plans/openvpn-google-user-revocation.md new file mode 100644 index 0000000..df53b2d --- /dev/null +++ b/.claude/plans/openvpn-google-user-revocation.md @@ -0,0 +1,122 @@ +# Plan: auto-revoke OpenVPN certs for deactivated Google users + +This is the **Puppet side** of a two-repo feature. The companion Terraform +work (new service-account secret + IAM) lives in +`terraform-aws-openvpn`, and there is one small optional change to the +OpenVPN portal app (also in that repo). + +## Why + +Client certificates are minted by the **OpenVPN portal** (Flask app, +`terraform-aws-openvpn/portal/portal/__init__.py`). On Google OAuth login +the portal calls `ensure_certificate(config_dir, email)` → +`generate_client_key()`, which runs: + +``` +easyrsa gen-req nopass # EASYRSA_REQ_CN = email +easyrsa sign-req client # EASYRSA_PASSIN = file:/ca_passphrase +``` + +So the certificate's **CN is literally the user's Google email**, and the +PKI lives on the shared EFS volume at +`/pki/{issued/.crt, private/.key}`. + +When a user is suspended or deleted in Google Workspace, the portal's +`authorize_email()` stops them from logging in — so they can no longer +*mint* a new cert. But the `.ovpn` profile they already downloaded (cert + +private key) keeps working until the CA **revokes** it. Nothing revokes it +today. This feature closes that window. + +## Identity binding already exists (no mapping layer needed) + +Because CN == email, reconciliation is a pure set-diff: + +``` +{ valid CNs in pki/index.txt } − { currently-active Google users } = certs to revoke +``` + +- **Issued-cert source of truth**: `/pki/index.txt`. + `V` lines = valid, `R` = already revoked; the CN is on each line. +- **Revocation primitive** (already used by the monthly CRL cron in + `profile::openvpn_server::config`, `config.pp:178`): + `easyrsa revoke ` then `easyrsa gen-crl`, signed with + `EASYRSA_PASSIN=file:/ca_passphrase` (the passphrase file is + already present at mode 0400). OpenVPN re-reads the CRL on every new + connection — no service restart needed (README.erb:49). + +The single set-diff catches suspended, deleted, **and** domain-removed +users in one pass. + +## The one net-new credential (Terraform side) + +The portal's existing `google_client` secret is an **OAuth web client** — +only good for interactive user login, not for querying account status. +Determining "is this user suspended?" requires the **Admin SDK Directory +API**, which needs a **service account with domain-wide delegation**, +scope `admin.directory.user.readonly`, impersonating a Workspace admin. + +In `terraform-aws-openvpn`: +- Add an AWS Secrets Manager secret holding the SA JSON — mirror the + existing `google_client` module in `secrets.tf`. +- Grant the instance role `secretsmanager:GetSecretValue` on it, alongside + `ca_key_passphrase_secret` (`asg.tf:32`, `iam.tf`). +- Surface the secret name to the instance as a custom fact (same channel + `openvpn.ca_key_passphrase_secret` uses), so Puppet can read it. + +## Puppet side (this repo) + +New class `profile::openvpn_server::google_revocation_sync`, included from +`profile::openvpn_server` (`openvpn_server.pp:39-41`, next to `nat`, +`auditd`, `cloudwatch_agent`). It deploys: + +1. **A Python sync script** (Python because it needs + `google-api-python-client` + `boto3` — the same stack the portal + already runs). Logic: + - Fetch the SA JSON from Secrets Manager → build a Directory API client + (delegated to an admin user). + - List **active** users across the portal's `allowed_domains` + (`users.list`, paginated, filtering out `suspended`) → build the + allow-set of emails. + - Parse `pki/index.txt` for `V` entries → valid CNs, **excluding** the + `server` cert and the CA CN. + - Diff → for each CN not in the allow-set: `easyrsa revoke `; then + run `easyrsa gen-crl` **once** after the loop. + - Log to syslog (tag `openvpn-google-sync`, auth facility — matching the + CRL cron's convention) and mail `$mailto` a summary of revocations. + +2. **A daily cron**, sibling to the existing monthly + `regenerate_openvpn_crl` (`config.pp:210`), with `MAILTO=${mailto}`. + +## Safety rails (the important part) + +- **Concurrency lock — the real risk.** The portal issues certs (mutating + `index.txt`/`pki`) at the same time this cron revokes. Two concurrent + `easyrsa` processes on the shared EFS PKI can corrupt `index.txt`. Wrap + the cron script — and ideally the portal's `generate_client_key()` — in + `flock` on a shared lockfile under ``. +- **Zero / low-count abort.** If the Directory API returns an empty or + implausibly small active-user list (API hiccup, expired delegation), + abort **without revoking anything**. Optionally cap: refuse a run that + would revoke more than N% of valid certs. +- **Dry-run mode.** A flag that logs intended revocations without touching + the PKI. Run it that way first in dev/sandbox. +- **Idempotent.** Skip CNs already marked `R` in `index.txt`. +- **Scope guard.** Only ever revoke CNs whose domain is in + `allowed_domains`; never touch anything else on the PKI. + +## Open decision + +Revoke **immediately** on suspension, or after a grace period? Immediate is +the secure default; a grace window avoids cutting access during a brief HR +suspension. Cheap to add later as a Hiera-configurable knob +(`profile::openvpn_server::revocation_grace_days`, default 0). + +## Rollout + +1. Terraform: add SA secret + IAM + delegation; publish the secret-name + fact. (`terraform-aws-openvpn`) +2. Puppet: land the profile in `environments/development` first, run with + `--dry-run`, verify the diff against `index.txt` looks right. +3. Add the `flock` lock to the portal's issuance path. +4. Promote development → sandbox → production (the repo's normal per-env + promotion flow). \ No newline at end of file diff --git a/debian/changelog b/debian/changelog index 5a58f2a..251c0e7 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9 @@ +puppet-code (0.1.0-1build315) noble; urgency=medium + + * commit event. see changes history in git log + + -- root Sun, 19 Jul 2026 23:18:49 +0000 + puppet-code (0.1.0-1build314) noble; urgency=medium * commit event. see changes history in git log diff --git a/environments/development/modules/profile/manifests/openvpn_server.pp b/environments/development/modules/profile/manifests/openvpn_server.pp index b81bc76..50b3446 100644 --- a/environments/development/modules/profile/manifests/openvpn_server.pp +++ b/environments/development/modules/profile/manifests/openvpn_server.pp @@ -36,6 +36,18 @@ openvp_config_directory => $openvp_config_directory, } + # Terraform sets this fact from enable_google_directory_revocation, so nodes + # where the feature is off get nothing at all -- no script, no cron. The fact + # is absent on instances provisioned by a module version predating the + # feature, which is the backwards-compatible case. The dependencies Terraform + # cannot see (toolkit version, Workspace delegation) are gated at runtime + # inside the wrapper instead. + if $facts.dig('openvpn', 'google_directory_revocation') { + class { 'profile::openvpn_server::google_revocation_sync': + openvp_config_directory => $openvp_config_directory, + } + } + include 'profile::openvpn_server::nat' include 'profile::openvpn_server::auditd' include 'profile::openvpn_server::cloudwatch_agent' diff --git a/environments/development/modules/profile/manifests/openvpn_server/google_revocation_sync.pp b/environments/development/modules/profile/manifests/openvpn_server/google_revocation_sync.pp new file mode 100644 index 0000000..abcba4e --- /dev/null +++ b/environments/development/modules/profile/manifests/openvpn_server/google_revocation_sync.pp @@ -0,0 +1,101 @@ +# @summary: Revokes OpenVPN certificates of deactivated Google Workspace users. +# +# A user who is suspended or deleted in Google Workspace can no longer mint a +# new certificate (the portal blocks their login), but any .ovpn they already +# downloaded keeps working until the CA revokes it. This class closes that gap +# by scheduling a daily reconciliation: valid CNs in the PKI index minus active +# Google users equals the set to revoke. Certificate CNs are Google email +# addresses, because the portal signs requests with EASYRSA_REQ_CN set to the +# authenticated user's email. +# +# The feature has three dependencies outside this repository, gated at the level +# each one can actually be observed: +# +# 1. terraform-aws-openvpn -- gated at COMPILE time on the +# openvpn.google_directory_revocation fact, which Terraform sets from +# enable_google_directory_revocation. Terraform knows this at provision +# time, so nothing is installed at all on nodes where the feature is off. +# 2. infrahouse-toolkit, which must provide `ih-openvpn $sync_subcommand` -- +# gated at RUNTIME by the wrapper, since the toolkit ships on its own +# release cycle and its version is not known when the catalog compiles. +# 3. Google Workspace, where the service account's client id must be +# authorized for the directory scope -- also gated at RUNTIME, via the +# wrapper's handling of the sync command's exit status. This step is a +# human action in the Workspace admin console with no Terraform resource +# behind it, so no fact can report it: enable_google_directory_revocation +# can be true for days before anyone finishes the console step. +# +# Where the class is declared it only writes a file and a cron entry, so it can +# never fail a Puppet run, and the feature starts working on the next cron tick +# once dependencies 2 and 3 land -- no Puppet run, no Hiera change, no redeploy. +# +# There is no separate report-only mode or enforcement switch: enabling the +# feature in Terraform means it revokes. +# +# @param openvp_config_directory +# OpenVPN configuration directory. Shared across the ASG over EFS, so it also +# hosts the cluster-wide lock. +# @param wif_env_file +# Workload Identity Federation environment file written by Terraform. Holds +# no secret; the credential config it references is keyless and sources an +# AWS identity from IMDS. +# @param sync_subcommand +# The `ih-openvpn` subcommand implementing the sync. The wrapper probes for +# it instead of pinning a toolkit version. +# @param ih_openvpn_path +# The `ih-openvpn` executable. A bare name (the default) is resolved through +# PATH, i.e. the packaged toolkit. Point it at a checkout's virtualenv to test +# an unreleased subcommand against a real instance, e.g. +# /home/ubuntu/code/infrahouse-toolkit/.venv/bin/ih-openvpn. The wrapper also +# honours an $IH_OPENVPN environment variable, which overrides this for a +# single ad-hoc run without a Puppet run. +# @param hour +# Cron hour for the daily run. +# @param minute +# Cron minute. Defaults to a per-node value derived from the hostname so the +# ASG does not query the Directory API in lockstep. +# @param mailto +# Where cron mails failures, including any unmet dependency. +class profile::openvpn_server::google_revocation_sync ( + String $openvp_config_directory, + String $wif_env_file = lookup( + 'profile::openvpn_server::wif_env_file', undef, undef, '/opt/openvpn-wif/wif.env' + ), + String $sync_subcommand = lookup( + 'profile::openvpn_server::google_sync_subcommand', undef, undef, 'sync-google-users' + ), + String $ih_openvpn_path = lookup( + 'profile::openvpn_server::ih_openvpn_path', undef, undef, 'ih-openvpn' + ), + Integer $hour = lookup('profile::openvpn_server::google_sync_hour', undef, undef, 2), + Integer $minute = lookup('profile::openvpn_server::google_sync_minute', undef, undef, fqdn_rand(60, 'openvpn-google-sync')), + $mailto = lookup( + 'profile::cron::mailto', undef, undef, "root@${facts['networking']['hostname']}.${facts['networking']['domain']}" + ), +) { + + $sync_script = "${openvp_config_directory}/google-user-sync.sh" + + # The lock lives on the EFS-backed config directory rather than /var/run so it + # is shared by every instance in the ASG. They all mount the same PKI, and two + # concurrent `easyrsa revoke` runs would race on index.txt. + $lock_file = "${openvp_config_directory}/.google-user-sync.lock" + + file { $sync_script: + ensure => file, + owner => 'root', + group => 'root', + mode => '0700', + content => template('profile/openvpn_server/google-user-sync.sh.erb'), + require => Mount[$openvp_config_directory], + } + + cron { 'openvpn_google_user_sync': + command => $sync_script, + user => 'root', + hour => $hour, + minute => $minute, + environment => ["MAILTO=${mailto}"], + require => File[$sync_script], + } +} \ No newline at end of file diff --git a/environments/development/modules/profile/templates/openvpn_server/google-user-sync.sh.erb b/environments/development/modules/profile/templates/openvpn_server/google-user-sync.sh.erb new file mode 100644 index 0000000..911a6ab --- /dev/null +++ b/environments/development/modules/profile/templates/openvpn_server/google-user-sync.sh.erb @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# Revoke OpenVPN certificates of deactivated Google Workspace users. +# Managed by Puppet - do not edit manually +# +# Puppet installs this script ONLY where the fact +# openvpn.google_directory_revocation is true, i.e. only where Terraform was run +# with enable_google_directory_revocation = true. Its presence on a host +# therefore means an operator asked for this feature. +# +# That is why an unmet dependency is reported as a FAILURE rather than passed +# over quietly. A silent skip would leave a host that was explicitly opted in +# never revoking anything, with nothing anywhere to say so -- the deactivated +# user keeps their VPN access indefinitely and no one finds out. Exiting +# non-zero with output on stderr makes cron mail it. +# +# The dependencies, each checked below: +# 1. terraform-aws-openvpn -> writes <%= @wif_env_file %> +# 2. infrahouse-toolkit -> provides `ih-openvpn <%= @sync_subcommand %>` +# 3. Google Workspace -> domain-wide delegation authorized for the SA +# +# Only genuinely transient conditions stay quiet: another instance in the ASG +# holding the cluster-wide lock, and a PKI that has not been generated yet on a +# host still bootstrapping. Both resolve on their own by the next run. +# +# Silent on success, output only on real problems (suitable for cron). + +set -uo pipefail + +TAG=openvpn-google-sync +WIF_ENV_FILE="<%= @wif_env_file %>" +# Single source of truth for which OpenVPN installation this run acts on: it is +# both checked here and handed to ih-openvpn below, so the two can never +# disagree about which PKI is being reconciled. +CONFIG_DIR="<%= @openvp_config_directory %>" +INDEX_FILE="${CONFIG_DIR}/pki/index.txt" +LOCK_FILE="<%= @lock_file %>" +SYNC_SUBCOMMAND="<%= @sync_subcommand %>" + +# The ih-openvpn executable. Normally the bare name, resolved through PATH to +# the packaged toolkit. $IH_OPENVPN overrides it for a single ad-hoc run, so an +# unreleased subcommand in a checkout's virtualenv can be exercised against a +# real instance without a Puppet run: +# sudo IH_OPENVPN=/home/ubuntu/code/infrahouse-toolkit/.venv/bin/ih-openvpn \ +# <%= @openvp_config_directory %>/google-user-sync.sh +# Set profile::openvpn_server::ih_openvpn_path in Hiera to make it stick. +IH_OPENVPN="${IH_OPENVPN:-<%= @ih_openvpn_path %>}" + +# Exit status contract with `ih-openvpn ${SYNC_SUBCOMMAND}`: +# 0 ran to completion +# 78 EX_CONFIG Google side not usable yet -- most often domain-wide +# delegation not authorized, which is a manual step in the +# Workspace admin console with no Terraform resource behind it. +# Still reported as a failure: the feature is switched on and +# not working, and that manual step is precisely what someone +# needs to be reminded about. +# other a real failure worth surfacing +EX_CONFIG=78 + +log_info() { logger -t "$TAG" -p auth.info "$*"; } +log_err() { logger -t "$TAG" -p auth.err "$*"; } + +# The feature is switched on for this host but cannot run. Log it, put it on +# stderr so cron mails it, and exit non-zero. +misconfigured() { + log_err "misconfigured: $*" + echo "ERROR: openvpn google user sync is enabled on this host but cannot run: $*" >&2 + exit 1 +} + +# Genuinely transient and self-resolving: nothing is wrong, this run simply has +# nothing to do yet. Reserved for lock contention and an uninitialized PKI. +transient() { + log_info "skipping: $* (will retry next run)" + exit 0 +} + +# --- Dependency 1: terraform-aws-openvpn ------------------------------------- +# wif.env carries GOOGLE_APPLICATION_CREDENTIALS, WIF_SA_EMAIL and +# WIF_ADMIN_SUBJECT. It holds no secret: the credential config it points at is a +# keyless external_account that sources an AWS identity from IMDS. +[ -r "$WIF_ENV_FILE" ] \ + || misconfigured "$WIF_ENV_FILE is absent or unreadable -- Terraform enabled the feature but did not deliver the credentials" + +# shellcheck disable=SC1090 +. "$WIF_ENV_FILE" + +for required_var in GOOGLE_APPLICATION_CREDENTIALS WIF_SA_EMAIL WIF_ADMIN_SUBJECT; do + [ -n "${!required_var:-}" ] || misconfigured "$required_var is not set in $WIF_ENV_FILE" +done +[ -r "$GOOGLE_APPLICATION_CREDENTIALS" ] \ + || misconfigured "credential config $GOOGLE_APPLICATION_CREDENTIALS is absent or unreadable" + +# --- Dependency 2: infrahouse-toolkit ---------------------------------------- +# Probe for the subcommand rather than pinning a toolkit version: the check then +# stays correct however the toolkit is versioned or backported. +command -v "$IH_OPENVPN" >/dev/null 2>&1 \ + || misconfigured "$IH_OPENVPN is not installed" +"$IH_OPENVPN" "$SYNC_SUBCOMMAND" --help >/dev/null 2>&1 \ + || misconfigured "$IH_OPENVPN has no '$SYNC_SUBCOMMAND' subcommand -- upgrade infrahouse-toolkit" + +# --- Local preconditions ----------------------------------------------------- +# No PKI yet means this host is still bootstrapping and Puppet has not generated +# it. Transient by definition, and there is nothing to reconcile against. +[ -r "$INDEX_FILE" ] || transient "$INDEX_FILE is absent (PKI not initialized yet)" + +# The lock lives on the shared EFS config directory, not in /var/run, so it is +# cluster-wide. Every instance in the ASG mounts the same PKI, and two +# concurrent `easyrsa revoke` runs would race on index.txt. +exec 9>"$LOCK_FILE" || misconfigured "cannot open lock file $LOCK_FILE" +flock -n 9 || transient "another instance is already running the sync (lock $LOCK_FILE held)" + +# --- Run --------------------------------------------------------------------- +# --config-dir is passed explicitly: it is a group-level option, so it must come +# before the subcommand. Without it the command would default to /etc/openvpn and +# could reconcile a different PKI than the one checked above. +OUTPUT=$("$IH_OPENVPN" --config-dir "$CONFIG_DIR" "$SYNC_SUBCOMMAND" 2>&1) +STATUS=$? + +case "$STATUS" in + 0) + # The subcommand logs each revocation itself; record the run for audit. + log_info "sync completed" + [ -n "$OUTPUT" ] && log_info "$OUTPUT" + ;; + "$EX_CONFIG") + # Dependency 3 outstanding. Everything on this host is in place; what is + # missing is the manual step -- authorizing the service account's client + # id for the directory scope at + # https://admin.google.com/ac/owl/domainwidedelegation + # Reported rather than skipped: until it is done the feature is switched + # on and silently revoking nobody. + log_err "$OUTPUT" + misconfigured "Google Workspace rejected the request; domain-wide delegation is most likely not authorized yet" + ;; + *) + # Every dependency is satisfied, so this is a real regression. + log_err "sync failed (exit $STATUS)" + log_err "$OUTPUT" + echo "ERROR: openvpn google user sync failed (exit $STATUS)" >&2 + echo "$OUTPUT" >&2 + exit "$STATUS" + ;; +esac + +# OpenVPN re-reads the CRL on each new connection attempt, so a revocation takes +# effect without restarting the service.