Non-Collective Root Communicator Managed MPI Buffer#1908
Draft
gberg617 wants to merge 1 commit into
Draft
Conversation
…deleting the buffer before message is finished being sent
bmhan12
reviewed
Jul 9, 2026
bmhan12
left a comment
bmhan12
left a comment
Contributor
There was a problem hiding this comment.
I think this change makes sense, only concern might be if there is significant overhead you've observed with maintaining a buffer of messages.
Comment on lines
+152
to
+153
| void drainCompletedSends(); | ||
| void releasePendingSends(); |
Contributor
There was a problem hiding this comment.
Please add descriptions of what these functions will do.
Comment on lines
+95
to
+108
| drainCompletedSends(); | ||
| if(isPackedMessagesEmpty(packedMessagesToBeSent) == false) | ||
| { | ||
| mpiNonBlockingSendMessages(m_mpiComm, 0, packedMessagesToBeSent); | ||
| const int messageSize = static_cast<int>(std::strlen(packedMessagesToBeSent)); | ||
| PendingSend pendingSend; | ||
| pendingSend.request = MPI_REQUEST_NULL; | ||
| pendingSend.buffer.reset(new char[messageSize + 1]); | ||
| std::memcpy(pendingSend.buffer.get(), packedMessagesToBeSent, messageSize + 1); | ||
|
|
||
| pendingSend.request = | ||
| mpiNonBlockingSendMessagesWithRequest(m_mpiComm, 0, pendingSend.buffer.get()); | ||
| m_pendingSends.push_back(std::move(pendingSend)); | ||
| } | ||
| drainCompletedSends(); |
Contributor
There was a problem hiding this comment.
Are two drain calls necessary here?
| { | ||
| void NonCollectiveRootCommunicator::initialize(MPI_Comm comm, int ranksLimit) | ||
| { | ||
| m_pendingSends.clear(); |
Contributor
There was a problem hiding this comment.
Is this necessary? Shouldn't the pending send buffer start in a valid state?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes a bug for the NonCollectiveRootCommunicator class where messages it sends may be deleted prior to the send being complete. The NonCollectiveRootCommunicator is used to try to send non-blocking point-to-point messages to the root rank when we know that the job will abort due to an error. In Lumberjack, these sorts of lines in
pushMessagesOnce()andpushMessagesFully()may lead to cases where the messages are deleted before the send completes:It's important to keep the MPI sends as non-blocking for this communicator in case the root rank is not able to receive the message. However, we want to prevent cases where the message finishes sending after it's deleted because this will lead to the root rank not receiving a proper message.
The fix being proposed here only applies to the NonCollectiveCommunicator's
push()method, and involves copying the message contents into a buffer owned by the NonCollectiveCommunicator stored in a vector of pending messages. Messages will only be sent by this owned communicator, and will be removed from the list of pending messages once we can confirm that the message sending completed.NOTE: This is still a WIP.
NOTE: PR is Co-authored by Codex