Skip to content

Add README.md documentation for cmd/ tools #5259

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Prucek wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Add README.md documentation for cmd/ tools #5259

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 64 additions & 13 deletions cmd/auto-config-brancher/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,64 @@
# auto-config-brancher

`auto-config-brancher` is a tool that reconciles various parts of CI config
in [openshift/release](https://github.com/openshift/release/) repository.
## What
Orchestrator that runs a sequence of CI configuration tools in order, commits each tool's changes separately, pushes the result, and creates a PR to openshift/release. This is the periodic automation that keeps release branch configs, job definitions, and private org mirrors up to date.

## What it does
## How it works — tool sequence

Executes these tools in order (each one commits its own changes if any):

| Step | Tool | What it does |
|---|---|---|
| 1 | `config-brancher` | Branch ci-operator configs for future releases (`--skip-periodics`) |
| 2 | `ci-operator-config-mirror` | Mirror configs to openshift-priv (`--to-org openshift-priv --only-org openshift`) |
| 3 | `determinize-ci-operator` | Normalize ci-operator config YAML formatting |
| 4 | `ci-operator-prowgen` | Generate Prow jobs from ci-operator configs |
| 5 | `private-prow-configs-mirror` | Mirror Prow configs to private org |
| 6 | `determinize-prow-config` | Normalize Prow config YAML |
| 7 | `sanitize-prow-jobs` | Validate and format Prow job configs |
| 8 | `clusterimageset-updater` | Update Hive ClusterImageSet resources |
| 9 | `promoted-image-governor` | Validate image promotion configs (dry-run) |

### Change detection
1. Records git HEAD SHA before running tools
2. Each tool runs, and if `git status --porcelain` shows changes, stages all and commits with message `"{tool} {args}"`
3. After all tools: compares overall `git diff` from start SHA to current HEAD
4. If no overall diff (changes cancelled out): skips push
5. If changes: pushes to remote branch `auto-config-brancher`

### Authentication
- **Token auth** (`--github-token-path`): pushes via `https://{login}:{token}@github.com/...`
- **GitHub App auth** (no token): uses `GitClientFactory` for authenticated push via app installation

### PR creation
- Title: `"Automate config brancher by auto-config-brancher job at {timestamp}"`
- Labels: `tide/merge-method-merge`, `rehearsals-ack`, `priority/ci-critical`
- If `--self-approve`: adds `approved` and `lgtm` labels
- Body: `/cc @{assign}`
- Updates existing PR if one already exists (matched by title prefix)

## Flags

| Flag | Default | What it controls |
|---|---|---|
| `--target-dir` | — | Root directory of target repository |
| `--github-login` | `openshift-bot` | GitHub username for PR/push |
| `--assign` | `openshift/test-platform` | PR assignee |
| `--self-approve` | false | Auto-add approved+lgtm labels |
| `--current-release` | — | Current OCP release version |
| `--future-release` | — | Target release versions (repeatable) |
| `--git-name` | `""` | Git committer name |
| `--git-email` | `""` | Git committer email |
| `--whitelist-file` | `""` | Path to whitelist file |

## Deployment
Periodic Prow job ([periodic-prow-auto-config-brancher](https://prow.ci.openshift.org/?job=periodic-prow-auto-config-brancher)) that creates [PRs in openshift/release](https://github.com/openshift/release/pulls?q=is%3Apr+%22Automate+config+brancher%22+is%3Aclosed+sort%3Acreated-desc). The PR is configured to be automatically merged without human approval.

---

## Background

### What it does

`auto-config-brancher` runs a sequence of other tools over a working copy of
the [openshift/release](https://github.com/openshift/release/) repository. Each of these tools maintains some subset of
Expand All @@ -15,12 +70,6 @@ be automatically merged (does not need a human approval).

_(subject to bitrot, always consult the code)_

- [ci-operator-yaml-creator](https://github.com/openshift/ci-tools/tree/master/cmd/ci-operator-yaml-creator): ensures
that repositories with in-repo `.ci-operator.yaml` file use `build_root.from_repository: true` in their ci-operator
configs
- [registry-replacer](https://github.com/openshift/ci-tools/tree/master/cmd/registry-replacer): ensures that all builds
specified in ci-operator configs use a local cluster registry (replaces central registry pullspecs with local
ImageStreamTag references)
- [config-brancher](https://github.com/openshift/ci-tools/tree/master/cmd/config-brancher): propagates ci-operator
config changes from `master`/`main` configs to future release branches
- [ci-operator-config-mirror](https://github.com/openshift/ci-tools/tree/master/cmd/ci-operator-config-mirror):
Expand All @@ -31,14 +80,16 @@ _(subject to bitrot, always consult the code)_
configuration from ci-operator configuration
- [private-prow-configs-mirror](https://github.com/openshift/ci-tools/tree/master/cmd/private-prow-configs-mirror):
propagates Prow configuration changes to private forks in `openshift-priv` organization
- [determinize-prow-config](https://github.com/openshift/ci-tools/tree/master/cmd/private-prow-configs-mirror): loads
- [determinize-prow-config](https://github.com/openshift/ci-tools/tree/master/cmd/determinize-prow-config): loads
and saves Prow configuration to fix ordering, formatting, proper sharding etc
- [sanitize-prow-jobs](https://github.com/openshift/ci-tools/tree/master/cmd/sanitize-prow-jobs): loads and saves Prow
job configuration to fix ordering, formatting etc. This tool also assigns jobs to build farm clusters.
- [clusterimageset-updater](https://github.com/openshift/ci-tools/tree/master/cmd/clusterimageset-updater): updates
cluster pool manifests to use the latest stable OCP releases
- [promoted-image-governor](https://github.com/openshift/ci-tools/tree/master/cmd/promoted-image-governor): validates
image promotion configs (dry-run)

## Why it exists
### Why it exists

Over time, we wrote a number of tools that automatically maintain parts of the CI config
in [openshift/release](https://github.com/openshift/release/) so that we do not need to do so as humans. After some
Expand All @@ -47,15 +98,15 @@ started to add new tools as "steps" to the most mature of them (`auto-config-bra
ran [config-brancher](https://github.com/openshift/ci-tools/tree/master/cmd/config-brancher), committed the changes and
submitted a PR).

## How it works
### How it works (detailed)

It iterates over a (hardcoded) sequence of steps that each calls one of the tools that modify some part of the CI
config. After each step, if there are changes in the config, the changes are committed. If there was at least one new
commit, the new series of commits is pushed into a new or existing PR using
the [`bumper` package from test-infra](https://github.com/kubernetes/test-infra/blob/master/prow/cmd/generic-autobumper/bumper/bumper.go)
.

## How is it deployed
### How is it deployed

The periodic
job [periodic-prow-auto-config-brancher](https://prow.ci.openshift.org/?job=periodic-prow-auto-config-brancher) ([definition](https://github.com/openshift/release/blob/55cd2ebb8a00445fb06789433dfe98e2199b9a97/ci-operator/jobs/infra-periodics.yaml#L828-L875))
Expand Down
109 changes: 54 additions & 55 deletions cmd/auto-peribolos-sync/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,57 +1,56 @@
# auto-peribolos-sync

`auto-peribolos-sync` is a like wrapper over the [private-org-peribolos-sync](../private-org-peribolos-sync) tool.

## What it does

`auto-peribolos-sync` maintains the Peribolos configuration stored in a GitHub repository by executing
the [private-org-peribolos-sync](../private-org-peribolos-sync) over a working copy of a repository containing Peribolos
config and filing a PR to that repository if the underlying tool changes the config.

## Why it exists

Together with [private-org-peribolos-sync](../private-org-peribolos-sync) it
manages [Peribolos](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/peribolos) configuration of the
GitHub repositories in
the [openshift-priv](https://docs.ci.openshift.org/architecture/private-repositories/#openshift-priv-organization) organization.

At the time when [private-org-peribolos-sync](../private-org-peribolos-sync) was written, there was no shared bump PR
package available, so we tended to write `config-changing-tool | auto-config-changing-tool` pairs when we needed to
maintain certain configuration automatically in GitHub repositories rather than include the PR filing logic in each
configuration changing binary.

## How it works

`auto-peribolos-sync` runs [private-org-peribolos-sync](../private-org-peribolos-sync) over a working copy of a repository
containing Peribolos config. If the execution results in Peribolos changes, the tool submits or updates a pull request
in that repository that updates the mainline with the changes, using
Prow's [generic-autobumper](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/generic-autobumper) package.

The tool uses a special naming convention to prevent repository name collisions:
- Repositories from the organization specified by `--only-org` keep their original names
- Repositories from organizations specified by `--flatten-org` keep their original names (can be specified multiple times)
- Repositories from the following default organizations always keep their original names for backwards compatibility:
- `openshift`
- `openshift-eng`
- `operator-framework`
- `redhat-cne`
- `openshift-assisted`
- `ViaQ`
- Repositories from other organizations are named as `<org>-<repo>` in the private organization

For example, with `--only-org=openshift --flatten-org=migtools`:
- `openshift/must-gather` becomes `openshift-priv/must-gather` (from --only-org and default)
- `openshift-eng/ocp-build-data` becomes `openshift-priv/ocp-build-data` (from default)
- `migtools/crane` becomes `openshift-priv/crane` (from --flatten-org)
- `redhat-cne/cloud-event-proxy` becomes `openshift-priv/cloud-event-proxy` (from default)
- `custom-org/some-repo` becomes `openshift-priv/custom-org-some-repo` (not in flatten list)

This ensures that repositories from different organizations can coexist in the private organization without naming conflicts.

## How is it deployed

The periodic
job [periodic-auto-private-org-peribolos-sync](https://deck-internal-ci.apps.ci.l2s4.p1.openshiftapps.com/?job=periodic-auto-private-org-peribolos-sync) ([definition](https://github.com/openshift/release/blob/18cc2328d72e34afc97cbb544618600c5c7fb656/ci-operator/jobs/infra-periodics.yaml#L1398-L1449))
uses `auto-peribolos-sync` to
create [PRs in openshift/config](https://github.com/openshift/config/pulls?q=is%3Apr+is%3Aclosed+Automate+peribolos+configuration%22) (
the repository is private).
## What
Automation wrapper that runs `private-org-peribolos-sync`, detects changes, commits them, and creates or updates a pull request on the `openshift/config` repository. This is the outer orchestration layer; the actual Peribolos config generation is delegated to `private-org-peribolos-sync`.

## How it works -- full flow

1. Initialize GitHub client and secrets
2. Shell out to `/usr/bin/private-org-peribolos-sync` with the provided arguments:
- `--destination-org openshift-priv`
- `--peribolos-config {path}`
- `--release-repo-path {path}`
- `--github-token-path {path}`
- Optionally `--whitelist-file`, `--only-org`, `--flatten-org`
3. After the subprocess completes, check for git changes using `bumper.HasChanges()`
4. If no changes, exit cleanly
5. If changes exist:
- Create a commit with title: `"Automate peribolos configuration sync {RFC1123 timestamp}"`
- Push to the `auto-peribolos-sync` branch. With token auth, pushes to the bot's fork (`github.com/{github-login}/config`); with GitHub App auth, pushes directly to the `openshift/config` repo
- Create or update a pull request via `bumper.UpdatePullRequestWithLabels()`:
- Target: `openshift/config` default branch
- Source: `{github-login}:auto-peribolos-sync`
- Description: "Updates the repositories of the openshift-priv organization"
- If `--self-approve` is set, add `approved` and `lgtm` labels to the PR

## Flags

| Flag | Default | What it controls |
|---|---|---|
| `--dry-run` | `true` | When true, uses API tokens but does not create the PR |
| `--self-approve` | `false` | Add `approved` and `lgtm` labels to the PR |
| `--github-login` | `openshift-bot` | GitHub username for push and PR creation |
| `--git-name` | `""` | Git commit author name (must pair with `--git-email`) |
| `--git-email` | `""` | Git commit author email (must pair with `--git-name`) |
| `--peribolos-config` | (required) | Path to the Peribolos config file to update |
| `--release-repo-path` | (required) | Path to openshift/release repository directory |
| `--whitelist-file` | `""` | Path to whitelist file, passed through to `private-org-peribolos-sync` |
| `--only-org` | `""` | Only sync repos from this org, passed through |
| `--flatten-org` | (repeatable) | Additional flatten orgs, passed through |
| GitHub flags | | Standard Prow GitHub options |

## Key files
- `cmd/auto-peribolos-sync/main.go` -- all logic in this single file
- `cmd/private-org-peribolos-sync/main.go` -- the actual config generation tool it shells out to

## Deployment
The periodic job [periodic-auto-private-org-peribolos-sync](https://deck-internal-ci.apps.ci.l2s4.p1.openshiftapps.com/?job=periodic-auto-private-org-peribolos-sync) ([definition](https://github.com/openshift/release/blob/18cc2328d72e34afc97cbb544618600c5c7fb656/ci-operator/jobs/infra-periodics.yaml#L1398-L1449))
uses `auto-peribolos-sync` to create [PRs in openshift/config](https://github.com/openshift/config/pulls?q=is%3Apr+is%3Aclosed+Automate+peribolos+configuration%22) (the repository is private).

The container image (`ci_auto-peribolos-sync_latest`) bundles both `auto-peribolos-sync` and `private-org-peribolos-sync` binaries.

Together with [private-org-peribolos-sync](../private-org-peribolos-sync) it manages [Peribolos](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/peribolos) configuration of the GitHub repositories in the [openshift-priv](https://docs.ci.openshift.org/architecture/private-repositories/#openshift-priv-organization) organization.

## Related
- `cmd/private-org-peribolos-sync` -- the tool this wrapper executes
- The PR targets `openshift/config` which holds the [Peribolos](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/peribolos) config for all OpenShift GitHub orgs
55 changes: 55 additions & 0 deletions cmd/auto-testgrid-generator/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# auto-testgrid-generator

## What
Orchestrator that runs `testgrid-config-generator` to produce updated TestGrid dashboard configurations, then creates or updates a pull request against `kubernetes/test-infra` with the changes. This is the automation that keeps TestGrid dashboards in sync with the current set of OpenShift CI periodic jobs.

## How it works -- full flow

1. **Parse flags**: collects paths to the testgrid config directory, release controller config, Prow jobs directory, allow-list file, and git/PR creation options.

2. **Run testgrid-config-generator**: executes `/usr/bin/testgrid-config-generator` as a subprocess with the following arguments:
- `-testgrid-config {dir}` -- where to write TestGrid YAML
- `-release-config {dir}` -- release controller configuration
- `-prow-jobs-dir {dir}` -- Prow periodic job definitions
- `-allow-list {file}` -- job classification overrides

3. **Create or update PR**: uses `prcreation.PRCreationOptions.UpsertPR()` to:
- Check the git working directory for changes
- Create a branch and commit if changes exist
- Push to the fork and create a PR against `kubernetes/test-infra` (configurable with `--github-org`)
- If a PR with matching title already exists, force-push to update it
- PR title format: `Update OpenShift testgrid definitions by auto-testgrid-generator job at {timestamp}`
- Assigns the PR to `--assign` (default: `openshift/test-platform`)

### PR matching
The tool uses title matching (`matchTitle`) to find existing PRs. If a PR with the prefix "Update OpenShift testgrid definitions by auto-testgrid-generator job" already exists, it updates that PR rather than creating a new one.

## Flags

| Flag | Default | What it controls |
|---|---|---|
| `--testgrid-config` | (none) | Directory where TestGrid output YAML is stored |
| `--release-config` | (none) | Directory of release controller config files |
| `--prow-jobs-dir` | (none) | Directory of Prow job config files |
| `--allow-list` | (none) | File with release-type overrides |
| `--working-dir` | `.` | Git working directory |
| `--github-login` | `openshift-bot` | GitHub username for PR creation |
| `--github-org` | `kubernetes` | GitHub org (override for testing) |
| `--upstream-branch` | `master` | Target branch for the PR |
| `--assign` | `openshift/test-platform` | GitHub user or team to assign the PR to |
| `--git-name` | (from GitAuthorOptions) | Git author name for commits |
| `--git-email` | (from GitAuthorOptions) | Git author email for commits |
| (PRCreationOptions flags) | | GitHub token path, etc. |

## Key files
- `cmd/auto-testgrid-generator/main.go` -- entry point, subprocess execution of testgrid-config-generator, PR creation
- `cmd/testgrid-config-generator/main.go` -- the actual TestGrid config generation logic (invoked as a binary)
- `pkg/github/prcreation/` -- PR creation/update library

## Deployment
Periodic Prow job. The container image includes both the `auto-testgrid-generator` and `testgrid-config-generator` binaries (the latter at `/usr/bin/testgrid-config-generator`). The job runs with a GitHub token for PR creation against `kubernetes/test-infra`.

## Related
- `cmd/testgrid-config-generator` -- the actual generation logic, invoked as a subprocess
- Target repo: `kubernetes/test-infra` (TestGrid configuration lives there)
- TestGrid dashboards: `https://testgrid.k8s.io/redhat-openshift-*`
Loading