feat(stacks): karpenter node isolation per organization/stack

[sylr]

Context

Formance runs many customer stacks in a shared Kubernetes cluster. Today there's no way to attribute compute cost to a specific customer/organization/stack — all pods land on shared nodes. This adds the operator's ability to provision dedicated Karpenter node pools per organization (shared <org> pool) or per stack (<org>-<stack> pool), so EC2 instances carry formance.com/{customer,organization,stack} tags for cost allocation and a stack's workloads run only on its dedicated nodes.

What it does

Driven by karpenter.* Settings and the formance.com/organization + formance.com/customer Stack labels:

1. Decides isolation level — karpenter.isolation: organization | stack.
2. Creates an EC2NodeClass by cloning a named cluster-scoped reference (karpenter.ec2-node-class.reference) and injecting spec.tags.
3. Creates a NodePool by cloning a reference (karpenter.node-pool.reference), pointing it at the cloned EC2NodeClass, and injecting node labels + taints.
4. Pins workloads — injects nodeSelector + tolerations into stack Deployments (required, since Karpenter is demand-driven).

Settings

Key	Meaning
karpenter.enabled	master switch (per-stack or *)
karpenter.isolation	organization (default) or stack
karpenter.ec2-node-class.reference	reference EC2NodeClass name to clone
karpenter.node-pool.reference	reference NodePool name to clone
karpenter.api.{ec2-node-class,node-pool}.group-version	GVK override (defaults karpenter.k8s.aws/v1, karpenter.sh/v1)

Design notes

• No new CRD / no second controller — integrated into the existing stacks reconciler (a second For(&Stack{}) would collide on the auto-derived controller name).
• Unstructured, not typed Karpenter deps — avoids the v1beta1→v1 churn; the feature only materializes a configured spec.
• Label-driven cleanup (app.kubernetes.io/managed-by=formance-operator) — removeAllModulesOwnedObjects namespace-filters and can't see cluster-scoped objects, so WithOwn cleanup would silently orphan pools.
• Shared org pools use additive non-controller owner refs (Stack → pool; both cluster-scoped, valid) and survive until the last owning stack is gone. A sweep releases/deletes stale pools on mode change, org relabel, or disable, under optimistic concurrency.
• Scheduling keys are pool-identity only (organization[/stack]) — customer is a tag/node-label for cost attribution but never a taint, since a shared org pool may see differing customer values that would break scheduling.
• Optional CRDs — no-ops when Karpenter isn't installed; a CustomResourceDefinition watch re-enables the feature without an operator restart.

Review

Two independent Codex review passes were incorporated (plan-level and implementation-level): label-driven cleanup for cluster-scoped objects, watch RBAC verb for the CRD watch, mutator idempotency + availability gating, optimistic-concurrency cleanup, and identity-only scheduling keys.

Testing

• go build ./..., go vet, gofmt clean
• Unit tests: resolve_test.go (modes/defaults/pending), karpenter_test.go (clone injection, owner refs, GC on mode change / disable of cluster-scoped objects, org pool surviving a second owner), mutator idempotency test in applications

Out of scope (follow-ups)

• Per-deployment / stack-wide scheduling override keys
• Pinning migration/reindex Jobs (taints only repel, so they still run on shared nodes)
• envtest integration against real Karpenter CRD fixtures

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 28060b85-0434-43ab-9b97-91eabb70925b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/karpenter-node-isolation

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}