Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions doc/rfc/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,7 @@ Design documents and technical proposals, grouped by scope. Shared/cross-cutting
## Stovepipe

- [Stovepipe Workflow](stovepipe/workflow.md) - Post-merge trunk-validation pipeline: ingest trunk push events (webhook + fallback poll), batch since last green, build to validate, record per-commit health, bisect to the offending commit, hand off to a remediation extension

## Runway

- [Runway Workflow](runway/workflow.md) - Landing service: check request mergeability, land scored batches, publish results back to SubmitQueue
73 changes: 73 additions & 0 deletions doc/rfc/runway/workflow.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Runway Workflow

Runway is the landing service: it owns VCS operations — mergeability checking and landing — on behalf of SubmitQueue. The orchestrator subscribes to two inbound topics (`runway-check`, `runway-land`) and publishes results to two outbound topics (`sq-check-result`, `sq-land-result`). It is a consumer-only service with no gateway; work arrives via topic queues and results leave via topic queues.

## Check and land

The two queues operate at different granularities:

- **check** is request-level. A check message carries a request's changes and merge strategy. Runway performs a read-only trial merge and publishes per-change mergeability results back.

- **land** is batch-level. A job message carries the resolved content for each request in the batch (runway has no access to SubmitQueue's request store, so the message is self-contained). Runway pre-validates, lands, and publishes a result with per-item outcomes back.

These are independent input-output flows. Check can run without land ever running, and land does not depend on a prior check.

## Branch serialization

The partition key `repo/target` on both inbound topics serializes all VCS operations for a given branch. The message queue delivers messages with the same partition key to the same consumer in order, so at most one check or land operation is in flight for any given branch at any time.

The outbound topics partition by SubmitQueue queue name, matching SubmitQueue's fan-out model where state updates for the same queue are serialized.

## Workflow

```
┌─────────────────────────────────────────────┐
│ submitqueue orchestrator │
└───────┬───────────────────────┬─────────────┘
│ │
Check (per request) Job (per batch)
│ │
▼ ▼
[runway-check] [runway-land]
│ │
check ctrl land ctrl
(read-only) (pre-validate + push)
│ │
CheckResult Result
│ │
▼ ▼
[sq-check-result] [sq-land-result]
│ │
▼ ▼
┌───────┬───────────────────────┬─────────────┐
│ check-result ctrl land-result ctrl │
│ (update request (update batch state, │
│ mergeability) fan out to conclude) │
│ submitqueue orchestrator │
└─────────────────────────────────────────────┘
```

## Per-controller summary

| Controller | In | Out | One-line role |
|---|---|---|---|
| **check** | Check | CheckResult -> sq-check-result | Check mergeability of a request's changes against the target branch (read-only) |
| **land** | Job | Result -> sq-land-result | Pre-validate, land, and finalize a batch's changes on the target branch |

The check controller always publishes a result — even when all changes are mergeable — so SubmitQueue receives a definitive answer. On infrastructure error it nacks for retry.

The land controller publishes a conflict result (and acks) when pre-validation detects a conflict; SubmitQueue handles rebatching. On infrastructure error it nacks for retry. On success it publishes per-item outcomes (commit SHAs, whether new commits were produced) so SubmitQueue can update its request state.

## Idempotency

Runway has no persistent state — no request store, no job store, no database. Idempotency is achieved through the VCS contract: land detects already-pushed changes (commit SHAs reachable from HEAD) and treats them as already-landed; closing an already-closed PR is a no-op. Check is read-only and naturally idempotent.

## Ownership by service

### Orchestrator

The orchestrator is the only service. It subscribes to two inbound topics (`runway-check`, `runway-land`), performs VCS operations through a pluggable extension, and publishes results to two outbound topics (`sq-check-result`, `sq-land-result`). It owns no persistent data.

### Shared: the messaging queue

Runway communicates with SubmitQueue only through the messaging queue. The inbound topics are owned by runway; the outbound topics are owned by SubmitQueue.
8 changes: 8 additions & 0 deletions runway/core/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
load("@rules_go//go:def.bzl", "go_library")

go_library(
name = "core",
srcs = ["core.go"],
importpath = "github.com/uber/submitqueue/runway/core",
visibility = ["//visibility:public"],
)
20 changes: 20 additions & 0 deletions runway/core/core.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// Copyright (c) 2025 Uber Technologies, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// Package core groups infrastructure shared across Runway's own services —
// the Runway-scoped analogue of the repo-level core/. Cross-domain
// infrastructure lives in the top-level core/; this package is for plumbing
// private to Runway. Subpackages are added here as shared needs emerge,
// mirroring submitqueue/core.
package core
9 changes: 9 additions & 0 deletions runway/core/topickey/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
load("@rules_go//go:def.bzl", "go_library")

go_library(
name = "topickey",
srcs = ["topickey.go"],
importpath = "github.com/uber/submitqueue/runway/core/topickey",
visibility = ["//visibility:public"],
deps = ["//core/consumer"],
)
32 changes: 32 additions & 0 deletions runway/core/topickey/topickey.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
// Copyright (c) 2025 Uber Technologies, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// Package topickey defines Runway pipeline stage identifiers.
package topickey

import "github.com/uber/submitqueue/core/consumer"

// TopicKey is the shared pipeline stage identifier type.
type TopicKey = consumer.TopicKey

const (
// TopicKeyCheck is the inbound topic where mergeability check requests arrive from SubmitQueue.
TopicKeyCheck TopicKey = "check"
// TopicKeyLand is the inbound topic where batch land jobs arrive from SubmitQueue.
TopicKeyLand TopicKey = "land"
// TopicKeyCheckResult is the outbound topic where check results are published back to SubmitQueue.
TopicKeyCheckResult TopicKey = "checkresult"
// TopicKeyLandResult is the outbound topic where land results are published back to SubmitQueue.
TopicKeyLandResult TopicKey = "landresult"
)
31 changes: 31 additions & 0 deletions runway/entity/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
load("@rules_go//go:def.bzl", "go_library", "go_test")

go_library(
name = "entity",
srcs = [
"check.go",
"entity.go",
"job.go",
],
importpath = "github.com/uber/submitqueue/runway/entity",
visibility = ["//visibility:public"],
deps = [
"//entity/change",
"//entity/mergestrategy",
],
)

go_test(
name = "entity_test",
srcs = [
"check_test.go",
"job_test.go",
],
embed = [":entity"],
deps = [
"//entity/change",
"//entity/mergestrategy",
"@com_github_stretchr_testify//assert",
"@com_github_stretchr_testify//require",
],
)
87 changes: 87 additions & 0 deletions runway/entity/check.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
// Copyright (c) 2025 Uber Technologies, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package entity

import (
"encoding/json"

"github.com/uber/submitqueue/entity/change"
"github.com/uber/submitqueue/entity/mergestrategy"
)

// Check is the inbound message on the runway-check topic. SubmitQueue publishes
// one Check per request to determine whether the request's changes can merge
// cleanly against the target branch. The check is read-only — it does not
// mutate the target branch or any external state.
type Check struct {
// Queue is the SubmitQueue queue name.
Queue string `json:"queue"`
// RequestID is the SubmitQueue request ID. Serves as the idempotency key.
RequestID string `json:"request_id"`
// Repo identifies the repository (e.g., "uber/submitqueue").
Repo string `json:"repo"`
// TargetBranch is the destination branch (e.g., "main").
TargetBranch string `json:"target_branch"`
// Changes is the set of code changes to check for mergeability.
Changes []change.Change `json:"changes"`
// Strategy is the landing strategy that would be used to land these changes.
Strategy mergestrategy.MergeStrategy `json:"strategy"`
}

// ToBytes serializes the Check to JSON bytes for queue message payload.
func (c Check) ToBytes() ([]byte, error) {
return json.Marshal(c)
}

// CheckFromBytes deserializes a Check from JSON bytes.
func CheckFromBytes(data []byte) (Check, error) {
var c Check
err := json.Unmarshal(data, &c)
return c, err
}

// MergeabilityResult describes whether a single change can be applied cleanly
// to the target branch.
type MergeabilityResult struct {
// Change is the input change this result corresponds to.
Change change.Change `json:"change"`
// Mergeable is true if the change can be applied cleanly.
Mergeable bool `json:"mergeable"`
// Reason is a human-readable explanation when Mergeable is false; empty when true.
Reason string `json:"reason,omitempty"`
}

// CheckResult is the outbound message published to the sq-check-result topic.
// It carries per-change mergeability detail back to SubmitQueue.
type CheckResult struct {
// Queue is the SubmitQueue queue name (partition key for the outbound topic).
Queue string `json:"queue"`
// RequestID correlates to Check.RequestID.
RequestID string `json:"request_id"`
// Results is one entry per change in the input Check.Changes.
Results []MergeabilityResult `json:"results"`
}

// ToBytes serializes the CheckResult to JSON bytes for queue message payload.
func (r CheckResult) ToBytes() ([]byte, error) {
return json.Marshal(r)
}

// CheckResultFromBytes deserializes a CheckResult from JSON bytes.
func CheckResultFromBytes(data []byte) (CheckResult, error) {
var r CheckResult
err := json.Unmarshal(data, &r)
return r, err
}
Loading