ref(objectstore): Initiate multipart upload#6172
Conversation
|
@sentry review |
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
| use super::outcome::Outcome; | ||
|
|
||
| /// Size of an individual request to objectstore. | ||
| const CHUNK_SIZE: NonZeroUsize = NonZeroUsize::new(5 * 1024 * 1024).unwrap(); |
There was a problem hiding this comment.
Chunking is necessary because Objectstore expects a content length. See code comments further below.
| TraceAttachment(Managed<StoreTraceAttachment>), | ||
| EventAttachment(Managed<StoreAttachment>), | ||
| RawProfile(Managed<StoreRawProfile>), | ||
| Create(Create, Sender<Result<UploadRef, Error>>), |
There was a problem hiding this comment.
There was previously no Create message because a "create" call to the upload endpoint generated an Objectstore key on the Relay side. For multipart uploads, the upload session needs to be created on the Objectstore side.
| pub key: String, | ||
| /// The ID of the multipart upload session (chosen by objectstore). | ||
| /// `None` if the upload is not multipart. | ||
| pub upload_id: Option<UploadId>, |
There was a problem hiding this comment.
For now, upload_id is only none if the location was created by an older relay instance. We might use None as an optimization for small uploads in the future though.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 511c411. Configure here.
| .key(&key) | ||
| .compression(Compression::Zstd) // make explicit because parts need to be manually compressed. | ||
| .send() | ||
| .await?; |
There was a problem hiding this comment.
Create initiate lacks objectstore retries
Medium Severity
handle_create calls initiate_multipart_upload once with no retry loop, while stream uploads use upload() with configurable max_attempts and is_retryable backoff. Transient objectstore 5xx or connection errors on POST create now fail upload session creation immediately instead of being retried like PATCH uploads.
Reviewed by Cursor Bugbot for commit 511c411. Configure here.
There was a problem hiding this comment.
True, but I won't add to the complexity unless we actually need it.
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
2 participants
Use Objectstore's multipart upload feature for large attachment uploads. The purpose of this PR is to validate the robustness of multipart uploads. The next step will be to allow resumable uploads.
Closes INGEST-773.