Skip to content

feat(stovepipe): add validate controller forwarding commits to batch#255

Open
mnoah1 wants to merge 1 commit into
mainfrom
mnoah1/stovepipe-validate-controller
Open

feat(stovepipe): add validate controller forwarding commits to batch#255
mnoah1 wants to merge 1 commit into
mainfrom
mnoah1/stovepipe-validate-controller

Conversation

@mnoah1

@mnoah1 mnoah1 commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Set up empty validate controller. This consumes events from the existing start controller, than forwards them to batch if valid. Here, we can implement additional validation logic on a given commit before passing it forward to a batch.

Also wires this into the existing controller

Why?

Building out overall workflow outlined in https://github.com/uber/submitqueue/blob/main/doc/rfc/stovepipe/workflow.md

What?

Validate step in the outline above

Test Plan

  • make local-stovepipe-start
  • Insert a message and confirm flow through the steps:
rchestrator-service-1  | 2026-06-16T22:15:23.665Z	DEBUG	consumer/consumer.go:351	processing delivery	{"controller": "start", "topic_key": "start", "message_id": "manual-1781648123538", "partition_key": "github.com/uber/cadence", "attempt": 1}
orchestrator-service-1  | 2026-06-16T22:15:23.665Z	INFO	start_controller	start/start.go:91	received change event	{"uri": "git://github.com/uber/cadence/refs/heads/main/abc123def456", "attempt": 1, "partition_key": "github.com/uber/cadence"}
orchestrator-service-1  | 2026-06-16T22:15:23.665Z	DEBUG	queue_mysql.message_store	mysql/message_store.go:72	inserting messages	{"topic": "validate", "count": 1, "visible_after": 0}
orchestrator-service-1  | 2026-06-16T22:15:23.669Z	DEBUG	queue_mysql.message_store	mysql/message_store.go:125	inserted messages	{"topic": "validate", "count": 1}
orchestrator-service-1  | 2026-06-16T22:15:23.669Z	DEBUG	queue_mysql.publisher	mysql/publisher.go:65	published message	{"topic": "validate", "message_id": "git://github.com/uber/cadence/refs/heads/main/abc123def456"}
orchestrator-service-1  | 2026-06-16T22:15:23.669Z	INFO	start_controller	start/start.go:106	published commit to validate	{"uri": "git://github.com/uber/cadence/refs/heads/main/abc123def456", "topic_key": "validate"}
orchestrator-service-1  | 2026-06-16T22:15:23.674Z	DEBUG	consumer/consumer.go:479	message processed successfully	{"controller": "start", "topic_key": "start", "message_id": "manual-1781648123538", "partition_key": "github.com/uber/cadence", "attempt": 1, "elapsed_ms": 4}
orchestrator-service-1  | 2026-06-16T22:15:23.757Z	DEBUG	queue_mysql.message_store	mysql/message_store.go:217	fetched messages	{"topic": "start", "partition_key": "github.com/uber/cadence", "count": 1}
orchestrator-service-1  | 2026-06-16T22:15:23.767Z	DEBUG	consumer/consumer.go:351	processing delivery	{"controller": "validate", "topic_key": "validate", "message_id": "git://github.com/uber/cadence/refs/heads/main/abc123def456", "partition_key": "github.com/uber/cadence", "attempt": 1}
orchestrator-service-1  | 2026-06-16T22:15:23.767Z	INFO	validate_controller	validate/validate.go:89	received change event	{"uri": "git://github.com/uber/cadence/refs/heads/main/abc123def456", "attempt": 1, "partition_key": "github.com/uber/cadence"}
orchestrator-service-1  | 2026-06-16T22:15:23.767Z	DEBUG	queue_mysql.message_store	mysql/message_store.go:72	inserting messages	{"topic": "batch", "count": 1, "visible_after": 0}
orchestrator-service-1  | 2026-06-16T22:15:23.770Z	DEBUG	queue_mysql.message_store	mysql/message_store.go:125	inserted messages	{"topic": "batch", "count": 1}
orchestrator-service-1  | 2026-06-16T22:15:23.770Z	DEBUG	queue_mysql.publisher	mysql/publisher.go:65	published message	{"topic": "batch", "message_id": "git://github.com/uber/cadence/refs/heads/main/abc123def456"}
orchestrator-service-1  | 2026-06-16T22:15:23.770Z	INFO	validate_controller	validate/validate.go:104	published commit to batch	{"uri": "git://github.com/uber/cadence/refs/heads/main/abc123def456", "topic_key": "batch"}
orchestrator-service-1  | 2026-06-16T22:15:23.774Z	DEBUG	consumer/consumer.go:479	message processed successfully	{"controller": "validate", "topic_key": "validate", "message_id": "git://github.com/uber/cadence/refs/heads/main/abc123def456", "partition_key": "github.com/uber/cadence", "attempt": 1, "elapsed_ms": 3}

Issue

@mnoah1 mnoah1 marked this pull request as ready for review June 16, 2026 22:20
@mnoah1 mnoah1 requested review from a team, behinddwalls and sbalabanov as code owners June 16, 2026 22:20
@mnoah1 mnoah1 force-pushed the mnoah1/stovepipe-validate-controller branch from 4ca3b54 to 5d2e6de Compare June 17, 2026 03:03
@mnoah1 mnoah1 force-pushed the mnoah1/stovepipe-start-wiring branch 2 times, most recently from 07811a3 to fbe881a Compare June 18, 2026 13:28
@behinddwalls behinddwalls force-pushed the mnoah1/stovepipe-start-wiring branch from fbe881a to 57745ce Compare June 18, 2026 15:45
@behinddwalls

Copy link
Copy Markdown
Collaborator

⚠️ Automatic stack rebase failed

This PR could not be automatically rebased after its base PR was merged. The rebase hit conflicts that need manual resolution.

To fix manually:

git fetch origin
git checkout mnoah1/stovepipe-validate-controller
git rebase --onto origin/mnoah1/stovepipe-start-wiring fbe881aa56233c566385b3e9a10ceb177555b5f6 mnoah1/stovepipe-validate-controller
# resolve conflicts, then:
git push --force-with-lease

Then update this PR's base branch:

gh pr edit 255 --base mnoah1/stovepipe-start-wiring

@mnoah1 mnoah1 force-pushed the mnoah1/stovepipe-start-wiring branch from 57745ce to aaf851f Compare June 18, 2026 16:23
@behinddwalls

Copy link
Copy Markdown
Collaborator

⚠️ Automatic stack rebase failed

This PR could not be automatically rebased after its base PR was merged. The rebase hit conflicts that need manual resolution.

To fix manually:

git fetch origin
git checkout mnoah1/stovepipe-validate-controller
git rebase --onto origin/main aaf851f16882bd1876b6f143e9fefb46d56c0949 mnoah1/stovepipe-validate-controller
# resolve conflicts, then:
git push --force-with-lease

Then update this PR's base branch:

gh pr edit 255 --base main

@mnoah1 mnoah1 force-pushed the mnoah1/stovepipe-validate-controller branch from 5d2e6de to 862f0b1 Compare June 18, 2026 18:49
Mirror the start controller's shape for the validate pipeline stage: the
validate controller consumes the validate topic, guards the carried
partition key, and forwards the change event to the batch stage. Adds the
batch topic key, wires the controller into the example orchestrator, and
registers the batch topic in the registry.

Currently a forwarding stub; placeholder comments describe the eventual
commit-metadata resolution per the workflow RFC.
@mnoah1 mnoah1 force-pushed the mnoah1/stovepipe-validate-controller branch from 862f0b1 to d152470 Compare June 18, 2026 18:50
@mnoah1 mnoah1 changed the base branch from mnoah1/stovepipe-start-wiring to main June 18, 2026 20:31
@mnoah1 mnoah1 enabled auto-merge June 18, 2026 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants