Skip to content

OCPBUGS-98492: Fix CAPI MachineSet deletion when MAPI MachineSet has already been deleted - #626

Open
Kaustubh-pande wants to merge 2 commits into
openshift:mainfrom
Kaustubh-pande:OCPBUGS-98492
Open

OCPBUGS-98492: Fix CAPI MachineSet deletion when MAPI MachineSet has already been deleted#626
Kaustubh-pande wants to merge 2 commits into
openshift:mainfrom
Kaustubh-pande:OCPBUGS-98492

Conversation

@Kaustubh-pande

@Kaustubh-pande Kaustubh-pande commented Jul 15, 2026

Copy link
Copy Markdown

Summary

When a CAPI MachineSet is reconciled after its corresponding MAPI MachineSet has already been removed, the existing deletion reconciliation is not executed, leaving the sync.machine.openshift.io/finalizer on the CAPI MachineSet and preventing it from completing deletion.

This change updates the reconciliation flow to invoke the existing CAPI deletion reconciliation when the MAPI MachineSet is absent, allowing the controller to remove the sync finalizer and complete CAPI MachineSet deletion.

This PR also adds a regression integration test covering the scenario where the MAPI MachineSet is absent and the CAPI MachineSet is deleted while still carrying the sync finalizer. The test failed prior to this change and now passes.

Additionally, this PR removes an unreachable capiMachineSet == nil check from Reconcile(), since that case is already handled by the earlier mapiMachineSet == nil && capiMachineSet == nil check.

Testing
✅ Added a regression integration test for OCPBUGS-98492
✅ Verified the regression test fails without the fix and passes with the fix
✅ Ran the full MachineSetSync test suite (123/123 passing)

Fixes: OCPBUGS-98492

Summary by CodeRabbit

  • Bug Fixes
    • Improved synchronization cleanup when the MAPI MachineSet is missing while the CAPI MachineSet is being deleted.
    • Deletion handling now proceeds reliably from the CAPI side, with retry/requeue behavior honored, preventing stale resources from remaining stuck.
  • Tests
    • Added a regression test covering removal of the sync finalizer and full deletion of the CAPI MachineSet when the MAPI MachineSet does not exist.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 15, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@Kaustubh-pande: This pull request references Jira Issue OCPBUGS-98492, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Fixes an issue where a Cluster API (CAPI) MachineSet could become stuck in the Terminating state if its corresponding Machine API (MAPI) MachineSet had already been deleted.

Previously, MachineSetSyncReconciler.Reconcile() returned early when the MAPI MachineSet was not found:

if mapiMachineSet == nil {
   logger.Info("Only CAPI machine set found, nothing to do")
   return ctrl.Result{}, nil
}

As a result, the existing deletion reconciliation logic was never executed, leaving the sync.machine.openshift.io/finalizer on the CAPI MachineSet and preventing it from being deleted.

This change updates the reconcile flow to invoke the existing CAPI deletion reconciliation when the MAPI MachineSet is absent. This allows the controller to remove the sync finalizer and lets the CAPI MachineSet be deleted normally.

Testing

  • Ran make unit
  • Verified existing MachineSetSync tests pass

Fixes: OCPBUGS-98492

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Jul 15, 2026
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@Kaustubh-pande, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 38 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 75004363-eae1-4929-b0a4-1d3813acadf5

📥 Commits

Reviewing files that changed from the base of the PR and between 92f8b2a and 7f7f713.

📒 Files selected for processing (2)
  • pkg/controllers/machinesetsync/machineset_sync_controller.go
  • pkg/controllers/machinesetsync/machineset_sync_controller_test.go

Walkthrough

The MachineSet sync reconciler now handles CAPI MachineSet deletion when the MAPI MachineSet is absent, including error and requeue handling. A regression test verifies that the sync finalizer is removed and the CAPI resource is deleted.

Changes

MachineSet deletion reconciliation

Layer / File(s) Summary
Handle CAPI-only MachineSet deletion
pkg/controllers/machinesetsync/machineset_sync_controller.go, pkg/controllers/machinesetsync/machineset_sync_controller_test.go
The reconciler invokes CAPI-to-MAPI deletion handling with a nil MAPI MachineSet, and the regression test verifies finalizer removal and complete CAPI MachineSet deletion.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: radekmanak, mdbooth

🚥 Pre-merge checks | ✅ 13 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Test Structure And Quality ⚠️ Warning The regression test still uses bare Eventually calls for update/delete and a cache wait, with no explicit timeouts or meaningful failure messages. Add explicit timeouts and descriptive assertion messages to the new cluster-interacting Eventually/Should calls, matching the repo pattern.
Microshift Test Compatibility ⚠️ Warning The new Ginkgo test creates/deletes MAPI MachineSet objects (machine.openshift.io) and has no [Skipped:MicroShift], [apigroup:...] tag, or g.Skip guard. Tag it with [apigroup:machine.openshift.io] or [Skipped:MicroShift], or add an exutil.IsMicroShiftCluster()+g.Skip() guard if it must stay excluded.
✅ Passed checks (13 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The new Ginkgo titles are static, descriptive strings; none include generated names, timestamps, UUIDs, or other run-to-run varying values.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The added Ginkgo test only manipulates MachineSet finalizers and deletion; it has no node-count, scheduling, or HA topology assumptions.
Topology-Aware Scheduling Compatibility ✅ Passed PASS: The change only adjusts MachineSet deletion/finalizer reconciliation and a regression test; it adds no node selectors, affinities, spread constraints, or replica-topology logic.
Ote Binary Stdout Contract ✅ Passed No process-level stdout writes were added; suite BeforeSuite sends klog to GinkgoWriter, and the PR only changes reconcile logic plus Ginkgo test setup.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The new Ginkgo test only creates/deletes MachineSet objects in-cluster; no IPv4 literals, IP parsing, or external network calls were found.
No-Weak-Crypto ✅ Passed Touched files only change deletion reconciliation and a regression test; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons found.
Container-Privileges ✅ Passed PR only changes two Go files in controller/tests; no manifest or pod/container security settings were added or modified.
No-Sensitive-Data-In-Logs ✅ Passed The PR only adds a deletion-reconcile path and generic MachineSet logs; it does not log secrets, tokens, PII, hostnames, or customer data.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: fixing CAPI MachineSet deletion when the corresponding MAPI MachineSet is already gone.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from RadekManak and mdbooth July 15, 2026 10:30
@openshift-ci

openshift-ci Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign racheljpg for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Kaustubh-pande

Copy link
Copy Markdown
Author

/jira refresh

@openshift-ci-robot

Copy link
Copy Markdown

@Kaustubh-pande: This pull request references Jira Issue OCPBUGS-98492, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/controllers/machinesetsync/machineset_sync_controller.go`:
- Around line 175-179: Remove the redundant capiMachineSet nil-check and its
return/logging block from the reconciliation logic, since the preceding
mapiMachineSet and capiMachineSet nil-check already handles that case. Preserve
the remaining machine-set synchronization flow unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 5fb49de9-608a-41c6-8cd6-ee74a5d1067d

📥 Commits

Reviewing files that changed from the base of the PR and between 1fcdc56 and cbc9ab7.

📒 Files selected for processing (1)
  • pkg/controllers/machinesetsync/machineset_sync_controller.go

Comment thread pkg/controllers/machinesetsync/machineset_sync_controller.go Outdated
Comment thread pkg/controllers/machinesetsync/machineset_sync_controller.go Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/controllers/machinesetsync/machineset_sync_controller_test.go`:
- Around line 1177-1194: Add the appropriate timeout and a meaningful failure
message to the `Eventually(k.Get(capiMachineSet), timeout)` assertion in the
“should remove the sync finalizer and allow the CAPI machine set to be deleted”
test, and ensure the nearby `Eventually` calls in this scenario follow the same
timeout-and-message convention without changing their behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: f4868842-9698-4b8a-b832-9a19dc7ec013

📥 Commits

Reviewing files that changed from the base of the PR and between cbc9ab7 and 92f8b2a.

📒 Files selected for processing (2)
  • pkg/controllers/machinesetsync/machineset_sync_controller.go
  • pkg/controllers/machinesetsync/machineset_sync_controller_test.go
💤 Files with no reviewable changes (1)
  • pkg/controllers/machinesetsync/machineset_sync_controller.go

Comment on lines +1177 to +1194
Eventually(k.Update(capiMachineSet, func() {
capiMachineSet.SetFinalizers([]string{machinesync.SyncFinalizer})
})).Should(Succeed())

By("Waiting for the manager's informer cache to observe the finalizer")
eventuallyManagerInformerCache(capiMachineSet).Should(
HaveField("ObjectMeta.Finalizers", ContainElement(machinesync.SyncFinalizer)),
)

By("Deleting the CAPI machine set")
Eventually(kDelete(ctx, capiMachineSet)).Should(Succeed())
})

It("should remove the sync finalizer and allow the CAPI machine set to be deleted", func() {
Eventually(k.Get(capiMachineSet), timeout).Should(
WithTransform(apierrors.IsNotFound, BeTrue()),
)
})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add timeouts and failure messages to Ginkgo assertions.

As per coding guidelines, operations interacting with the cluster must include appropriate timeouts for Eventually, and assertions must include meaningful failure messages.

🛠️ Proposed fix
-				Eventually(k.Update(capiMachineSet, func() {
-					capiMachineSet.SetFinalizers([]string{machinesync.SyncFinalizer})
-				})).Should(Succeed())
+				Eventually(k.Update(capiMachineSet, func() {
+					capiMachineSet.SetFinalizers([]string{machinesync.SyncFinalizer})
+				}), timeout).Should(Succeed(), "should be able to add the sync finalizer to the CAPI machine set")
 
 				By("Waiting for the manager's informer cache to observe the finalizer")
-				eventuallyManagerInformerCache(capiMachineSet).Should(
-					HaveField("ObjectMeta.Finalizers", ContainElement(machinesync.SyncFinalizer)),
-				)
+				eventuallyManagerInformerCache(capiMachineSet).Should(
+					HaveField("ObjectMeta.Finalizers", ContainElement(machinesync.SyncFinalizer)),
+					"manager cache should observe the sync finalizer on the CAPI machine set",
+				)
 
 				By("Deleting the CAPI machine set")
-				Eventually(kDelete(ctx, capiMachineSet)).Should(Succeed())
+				Eventually(kDelete(ctx, capiMachineSet), timeout).Should(Succeed(), "should be able to delete the CAPI machine set")
 			})
 
 			It("should remove the sync finalizer and allow the CAPI machine set to be deleted", func() {
-				Eventually(k.Get(capiMachineSet), timeout).Should(
-					WithTransform(apierrors.IsNotFound, BeTrue()),
-				)
+				Eventually(k.Get(capiMachineSet), timeout).Should(
+					WithTransform(apierrors.IsNotFound, BeTrue()),
+					"eventually the CAPI machine set should be deleted and not found",
+				)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
Eventually(k.Update(capiMachineSet, func() {
capiMachineSet.SetFinalizers([]string{machinesync.SyncFinalizer})
})).Should(Succeed())
By("Waiting for the manager's informer cache to observe the finalizer")
eventuallyManagerInformerCache(capiMachineSet).Should(
HaveField("ObjectMeta.Finalizers", ContainElement(machinesync.SyncFinalizer)),
)
By("Deleting the CAPI machine set")
Eventually(kDelete(ctx, capiMachineSet)).Should(Succeed())
})
It("should remove the sync finalizer and allow the CAPI machine set to be deleted", func() {
Eventually(k.Get(capiMachineSet), timeout).Should(
WithTransform(apierrors.IsNotFound, BeTrue()),
)
})
Eventually(k.Update(capiMachineSet, func() {
capiMachineSet.SetFinalizers([]string{machinesync.SyncFinalizer})
}), timeout).Should(Succeed(), "should be able to add the sync finalizer to the CAPI machine set")
By("Waiting for the manager's informer cache to observe the finalizer")
eventuallyManagerInformerCache(capiMachineSet).Should(
HaveField("ObjectMeta.Finalizers", ContainElement(machinesync.SyncFinalizer)),
"manager cache should observe the sync finalizer on the CAPI machine set",
)
By("Deleting the CAPI machine set")
Eventually(kDelete(ctx, capiMachineSet), timeout).Should(Succeed(), "should be able to delete the CAPI machine set")
})
It("should remove the sync finalizer and allow the CAPI machine set to be deleted", func() {
Eventually(k.Get(capiMachineSet), timeout).Should(
WithTransform(apierrors.IsNotFound, BeTrue()),
"eventually the CAPI machine set should be deleted and not found",
)
})
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/controllers/machinesetsync/machineset_sync_controller_test.go` around
lines 1177 - 1194, Add the appropriate timeout and a meaningful failure message
to the `Eventually(k.Get(capiMachineSet), timeout)` assertion in the “should
remove the sync finalizer and allow the CAPI machine set to be deleted” test,
and ensure the nearby `Eventually` calls in this scenario follow the same
timeout-and-message convention without changing their behavior.

Source: Coding guidelines

@Kaustubh-pande

Copy link
Copy Markdown
Author

/test unit

@Kaustubh-pande

Copy link
Copy Markdown
Author

/pipeline auto

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification

The pipeline-auto label has been added to this PR. Second-stage tests will be triggered automatically when all first-stage tests pass.

@Kaustubh-pande

Copy link
Copy Markdown
Author

/test unit

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-capi-disconnected-techpreview
/test e2e-aws-capi-techpreview
/test e2e-aws-capi-techpreview-post-install
/test e2e-aws-ovn-techpreview
/test e2e-aws-ovn-techpreview-upgrade
/test e2e-azure-capi-techpreview
/test e2e-azure-ovn-techpreview
/test e2e-azure-ovn-techpreview-upgrade
/test e2e-gcp-capi-techpreview
/test e2e-gcp-ovn-techpreview
/test e2e-metal3-capi-techpreview
/test e2e-openstack-capi-techpreview
/test e2e-vsphere-capi-techpreview
/test regression-clusterinfra-aws-ipi-techpreview-capi

@Kaustubh-pande

Copy link
Copy Markdown
Author

/retest-required

@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@Kaustubh-pande: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-techpreview 7f7f713 link true /test e2e-aws-ovn-techpreview
ci/prow/e2e-gcp-ovn-techpreview 7f7f713 link true /test e2e-gcp-ovn-techpreview

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@Kaustubh-pande

Copy link
Copy Markdown
Author

/test e2e-metal3-capi-techpreview

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. pipeline-auto

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants