Skip to content

feat(gcp): bootstrap two data centers with --multi-dc - #631

Open
NJona wants to merge 3 commits into
multi-dc-08-shared-registryfrom
multi-dc-09-multi-dc-bootstrap
Open

feat(gcp): bootstrap two data centers with --multi-dc#631
NJona wants to merge 3 commits into
multi-dc-08-shared-registryfrom
multi-dc-09-multi-dc-bootstrap

Conversation

@NJona

@NJona NJona commented Jul 31, 2026

Copy link
Copy Markdown
Member

Wires up multi-data-center support: --multi-dc bootstraps a second data center in the same project, sharing the VPC, jumpbox and PostgreSQL server but running its own Kubernetes and Ceph cluster. 14 VMs (~100 vCPU) and 6 static IPs, so the region's CPU quota may need raising.

The secondary data center's config and vault are derived from the primary's after its secrets exist, which is why Bootstrap handles it in a loop after the primary's config is written:

  • the config is cloned and its data-center-scoped fields cleared, so the topology, the domains and the shared registry are inherited while the ingress CA and cephadm key are regenerated;
  • the vault is derived through DeriveDataCenterVault, keeping the postgres roles and token keys and dropping the per-cluster secrets;
  • postgres becomes mode: external pointing at the shared server's internal IP — which is what its certificate's only SAN carries — and postgres is appended to operations.skip in the config, so manual re-runs on the jumpbox skip it too.

Review notes

  • A post-generation check fails the bootstrap if secrets that must match across data centers diverged, or if a per-cluster secret was inherited instead of regenerated. Both would otherwise surface long after the fact, against a live shared database.
  • Installs run strictly sequentially in ascending order; the primary's install creates the database, roles and schema the others reuse.
  • validateMultiDC rejects what cannot work: --write-config=false, an explicit --datacenter-id (the IDs are derived and drive the domains), an empty --datacenter-name (the k0s cluster is named after it).
  • --multi-dc with --recover-config is supported: a secondary data center missing from the jumpbox is tolerated, so a second data center can be added to an existing single-DC project.
  • --multi-dc=false stays byte-identical to today's behaviour.

Part of the oms beta bootstrap-gcp --multi-dc stack (10 PRs). Merge in order; each PR is based on its predecessor.

NJona and others added 3 commits July 31, 2026 15:19
Wires up multi-data-center support: --multi-dc bootstraps a second data
center in the same project, sharing the VPC, jumpbox and PostgreSQL
server but running its own Kubernetes and Ceph cluster.

The secondary data center's config and vault are derived from the
primary's after its secrets exist, which is why Bootstrap handles it in
a loop after the primary's config is written:

- the config is cloned and its data-center-scoped fields cleared, so the
  installer's dataCenters topology, the domains and the shared registry
  are inherited while the ingress CA and cephadm key are regenerated;
- the vault is derived through DeriveDataCenterVault, keeping the
  postgres roles and token keys and dropping the per-cluster secrets;
- postgres becomes mode: external pointing at the shared server's
  internal IP, which is what its certificate's only SAN carries, and
  "postgres" is appended to operations.skip in the config so manual
  re-runs on the jumpbox skip it too.

A post-generation check fails the bootstrap if secrets that must match
across data centers diverged, or if a per-cluster secret was inherited
instead of regenerated — both would otherwise surface long after the
fact, against a live shared database.

Installs run strictly sequentially in ascending data center order, since
the primary's install creates the database, roles and schema the others
reuse. validateMultiDC rejects the combinations that cannot work
(--write-config=false, an explicit --datacenter-id, an empty datacenter
name), and the CLI prints one install command per data center in the
required order.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
Signed-off-by: NJona <25478046+NJona@users.noreply.github.com>
Signed-off-by: NJona <25478046+NJona@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant