feat(new source): add Azure Blob Storage source by ldrozdz93 · Pull Request #1 · ldrozdz93/vector

ldrozdz93 · 2026-01-20T10:24:17Z

Summary

This PR adds Azure Blob Storage source. From the user's perspective, it's intended to work in a similar manner to AWS S3 source.

Vector configuration

sources:
  azure_blob:
    type: azure_blob
    connection_string: REDACTED
    container_name: logs
    queue:
      queue_name: eventgrid

sinks:
  console:
    type: console
    inputs:
      - azure_blob
    encoding:
      codec: json

How did you test this PR?

Unit tests (make test, full workspace suite).
Integration tests against Azurite (make test-integration-azure, 70 tests covering the azure sink and source).
Local runs of the CI check suite: make check-clippy, cargo vdev check events, make check-generated-docs, make check-docs (CUE), cargo deny check advisories, dd-rust-license-tool check, markdownlint, prettier.
Manual tests against real Azure infrastructure, described in steps.md.

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Closes: Add Azure Blob Storage source vectordotdev/vector#13882

Notes

Dependency note: the source uses the new official Azure SDK throughout (azure_core 1.0, azure_storage_blob 1.0, and azure_storage_queue 1.0, which went GA in May 2026), sharing the sink's client machinery (connection-string parsing, SharedKey signing policy, credential handling) via the src/azure module. No legacy azure_storage* crates are involved.
Known divergences from S3 parity (intentional): Content-Encoding-based compression detection is not supported (Azure SDK limitation — detection uses Content-Type and file extension instead), and the source additionally handles Microsoft.Storage.BlobRenamed events.
make build-licenses was run to regenerate the license inventory.
historical context: This Azure source was originally implemented for vector 0.38 and run in our product MVP for a few months. It supported just a tiny subset of features, only precisely what we needed. During that time, I've rebased it on the latest vector versions at the time multiple times. Then we've dropped the Azure Blobs approach in our product. I picked this up again a few months ago, added the intended feature parity with AWS S3 source, updated docs, extended tests etc. Basically done all that was required to contribute this to open-source.
This is my first opensource vector contribution. I'd appreciate feedback related to both the code and the process.

Copilot

Pull request overview

This pull request introduces a new azure_blob source for Vector that enables reading logs from Azure Blob Storage via Event Grid notifications delivered through Azure Storage Queues.

Changes:

New Azure Blob Storage source with queue-based event processing
Support for compression (gzip, zstd), multiple codecs, and multiline aggregation
Comprehensive unit and integration tests
Documentation files and configuration examples

Reviewed changes

Copilot reviewed 20 out of 21 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/sources/azure_blob/mod.rs	Main source implementation with streaming and event processing
src/sources/azure_blob/queue.rs	Queue integration, blob retrieval, and Event Grid message processing
src/sources/azure_blob/test.rs	Unit tests for compression detection and blob processing
src/sources/azure_blob/integration_tests.rs	Integration tests covering various scenarios
src/internal_events/azure_queue.rs	Internal event definitions for metrics and logging
website/cue/reference/components/sources/azure_blob.cue	Component documentation and metadata
Cargo.toml	Dependency and feature flag additions
tests/integration/azure/config/*.yaml	Integration test configuration
testing/github-XXXXX/*	Test artifacts and documentation (should be removed)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ldrozdz93 · 2026-02-20T07:38:52Z

@copilot review

Copilot

Pull request overview

Copilot reviewed 258 out of 292 changed files in this pull request and generated 8 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 8 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

anshul-thakkar · 2026-03-11T09:28:08Z

Hi @ldrozdz93, thanks for the work on adding the Azure Blob Storage source.
I wanted to check if there are any updates on this PR or if there’s anything blocking progress toward getting it reviewed/merged?

This feature would be very useful for Azure-based pipelines, so I’m keen to see it move forward. Let me know if there’s anything I can help with from the community side.

…tool 1.0.6 after rebase Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… naming, log levels - Add trailing periods to log messages per instrumentation spec - Rename InvalidRowEventType to InvalidRowEventTypeError (error events must be named *Error) - Downgrade QueueMessageProcessingErrored/Rejected logs from error! to warn!: these track delivery outcomes via dedicated counters, matching S3's warn-level handling of errored/rejected batches; real component errors already emit *Error events with component_errors_total Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…rules Upstream now disallows raw-string metrics::counter! in favor of vector_common's typed counter! macro with CounterName variants. Add the four azure_queue metric names to CounterName and migrate all call sites. Also drop a useless format! in integration tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ests Upstream restructured the azure integration test env (AZURITE_ADDRESS + AZURITE_OAUTH_ADDRESS); the branch's custom AZURE_ADDRESS line was lost in the rebase. Align the source tests with the sink tests' convention instead of reintroducing a third variable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- deny.toml: re-add RUSTSEC-2024-0384 (instant) ignore - reintroduced by the legacy azure_core 0.21 dependency chain; upstream dropped it when nothing pulled instant anymore - azure_blob.cue: drop classes.commonly_used - field removed from the component schema upstream - expect.txt: add deser - pre-existing word flagged by code scanning due to large PR diff - compose.yaml: prettier list indentation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Move connection_string.rs, shared_key_policy.rs, and the auth/client-construction code (AzureAuthentication, build_client, transport assembly) from sinks/azure_common into the shared src/azure module, so the azure_blob source can reuse them. Sinks keep their import paths via re-exports. Also: - Fix SharedKey signing: Content-Length must contribute an empty string when the request body is zero-length (per the Authorize-with-Shared-Key spec and this file's own doc comment). The sink never signs zero-body requests so this was latent; queue operations (create/delete_message) hit it. - Re-gate 'pub mod azure' on the azure component features instead of the 'azure' meta-feature (which only the source enabled, and which will be removed when the legacy SDK is dropped). - Add azure_storage_queue 1.0 dependency and new-SDK deps to sources-azure_blob (transitional, legacy crates removed in a follow-up commit). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…achinery - connection_string: parse QueueEndpoint, add queue_account_endpoint()/queue_url() mirroring the blob logic (explicit endpoint short-circuits dev-storage synthesis) - client: factor resolve_credential()/build_transport() out of build_client and add build_queue_client() for azure_storage_queue 1.0; pin the Azurite-supported service version - shared_key_policy: sign the actual body length when the Content-Length header is absent — generated SDK clients (e.g. queue send_message) rely on the transport to set the header, so signing previously omitted it and the service rejected the signature - add an Azurite SharedKey roundtrip spike test (create/send/receive/delete) covering zero-body signing and query-parameter canonicalization Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ge_blob/queue 1.0 Replace the legacy azure_storage_blobs/azure_storage_queues 0.21 clients with the official azure_storage_blob 1.0 and azure_storage_queue 1.0 SDKs: - queue path: receive_messages with typed options (visibility timeout and batch size as seconds/i32), delete_message(message_id, pop_receipt); ReceivedMessage fields are optional in the new SDK, so extract a MessageHandle up front and fail with the new IncompleteMessage error if the service response lacks identifiers - blob path: get_properties via typed header traits, 404 detection via http_status(), and download() limited to a single in-flight range request so blob contents stream sequentially with bounded memory like the legacy client - client construction goes through the shared src/azure machinery (gains proxy support); make_* helpers and make_blob_with_ack_stream become async - integration tests ported: container/queue create with status-based 409 tolerance, blob upload via RequestContent, queue messages via send_message All 70 azure integration tests pass against Azurite (the moved connection-string unit tests left the integration filter scope and run in the unit suite instead). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Remove azure_storage/azure_storage_blobs/azure_storage_queues 0.21 and the azure_core_for_storage bridge (plus the 'azure' meta-feature and the legacy client builders), now that the source uses the official SDK throughout. This also sheds ~20 transitive crates (instant, http-types, futures-lite 1.x, RustyXML, ...) and the RUSTSEC-2024-0384 deny.toml ignore that existed only for the legacy dependency chain. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

ldrozdz93 requested a review from Copilot January 20, 2026 10:24

github-actions Bot added domain: sources domain: external docs labels Jan 20, 2026

Copilot started reviewing on behalf of ldrozdz93 January 20, 2026 10:24 View session

github-advanced-security AI found potential problems Jan 20, 2026

View reviewed changes

Copilot AI reviewed Jan 20, 2026

View reviewed changes

Comment thread src/internal_events/mod.rs Outdated

Comment thread src/sources/azure_blob/queue.rs Outdated

Comment thread src/sources/azure_blob/mod.rs Outdated

Comment thread PR_CHECKLIST.md Outdated

Comment thread src/internal_events/azure_queue.rs Outdated

Comment thread src/lib.rs Outdated

ldrozdz93 force-pushed the ldrozdz93/azure-blob-storage-source branch 2 times, most recently from e962d8b to 8a72d0d Compare January 21, 2026 16:11

ldrozdz93 force-pushed the ldrozdz93/azure-blob-storage-source branch from 8a72d0d to fa44b4a Compare February 20, 2026 07:35

github-actions Bot added domain: sinks domain: transforms domain: topology domain: codecs domain: core domain: ci domain: vdev domain: releasing labels Feb 20, 2026

ldrozdz93 requested a review from Copilot February 20, 2026 07:44

Copilot AI reviewed Feb 20, 2026

View reviewed changes

ldrozdz93 force-pushed the ldrozdz93/azure-blob-storage-source branch from fa44b4a to 6fd2c39 Compare February 20, 2026 10:28

ldrozdz93 changed the base branch from master to blob-storage-rebase February 20, 2026 10:30

ldrozdz93 changed the base branch from blob-storage-rebase to master February 20, 2026 10:30

ldrozdz93 requested a review from Copilot February 20, 2026 10:31

Copilot started reviewing on behalf of ldrozdz93 February 20, 2026 10:31 View session

Copilot AI reviewed Feb 20, 2026

View reviewed changes

ldrozdz93 force-pushed the ldrozdz93/azure-blob-storage-source branch from 6fd2c39 to 16b3efc Compare March 13, 2026 22:37

github-actions Bot removed the domain: codecs label Mar 13, 2026

ldrozdz93 force-pushed the ldrozdz93/azure-blob-storage-source branch from 16b3efc to 97f2ad9 Compare March 13, 2026 23:23

ldrozdz93 and others added 6 commits June 4, 2026 07:54

chore(azure_blob): regenerate license inventory with dd-rust-license-…

637a04d

…tool 1.0.6 after rebase Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs(azure_blob): regenerate component CUE docs after rebase

09c1722

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs(azure_blob): fix markdownlint violations in manual test docs

f2786ba

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

chore(azure_blob): add azure_blob doc terms to spelling expect list

a1c4d4b

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

ldrozdz93 changed the title ~~feat(azure_blob): add Azure Blob Storage source~~ feat(new source): add Azure Blob Storage source Jun 4, 2026

ldrozdz93 force-pushed the ldrozdz93/azure-blob-storage-source branch from eb3ce96 to e8e4ee5 Compare June 4, 2026 13:29

github-actions Bot added domain: sinks domain: transforms domain: topology domain: core domain: ci domain: vdev domain: releasing domain: rfc labels Jun 4, 2026

github-advanced-security AI found potential problems Jun 4, 2026

View reviewed changes

Comment thread website/cue/reference/components/sources/internal_metrics.cue Fixed

github-actions Bot added the docs review on hold label Jun 5, 2026

ldrozdz93 and others added 4 commits June 6, 2026 08:20

ldrozdz93 requested a review from Copilot June 6, 2026 16:02

Copilot AI reviewed Jun 6, 2026

ldrozdz93 requested a review from Copilot June 6, 2026 22:48

Copilot AI reviewed Jun 6, 2026

ldrozdz93 requested a review from Copilot June 7, 2026 06:05

Copilot AI reviewed Jun 7, 2026

Conversation

ldrozdz93 commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ldrozdz93 commented Feb 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anshul-thakkar commented Mar 11, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

ldrozdz93 commented Jan 20, 2026 •

edited

Loading