From 082af11dfc263c645dcf6cd7611d825285c1099b Mon Sep 17 00:00:00 2001 From: "Mingyu Chen (Rayner)" Date: Wed, 15 Jul 2026 23:19:11 +0800 Subject: [PATCH 1/2] docs: design Doris Operator release tools --- .../specs/2026-07-15-release-tools-design.md | 286 ++++++++++++++++++ 1 file changed, 286 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-15-release-tools-design.md diff --git a/docs/superpowers/specs/2026-07-15-release-tools-design.md b/docs/superpowers/specs/2026-07-15-release-tools-design.md new file mode 100644 index 00000000..f974f497 --- /dev/null +++ b/docs/superpowers/specs/2026-07-15-release-tools-design.md @@ -0,0 +1,286 @@ +# Doris Operator Release Tools Design + +## Context + +The `doris-operator` repository currently relies on manual release steps. This +change adds a small, reusable release toolkit modeled after +`apache/doris/tools/release-tools`, while preserving the existing Doris Operator +release conventions. + +The first target release is `26.0.0` from the Git tag `26.0.0`. + +The design intentionally keeps the current Operator naming scheme: + +- Git tags use the final version only, such as `26.0.0`. +- SVN directories use the final version only. +- Source artifact names do not contain an RC suffix. +- A formal release is packaged and signed again from the Git tag. It does not + promote, compare with, or reuse the artifacts already in dev SVN. + +## Goals + +The toolkit must provide the selected capabilities: + +- A: shared release configuration +- B: environment and prerequisite checks +- C: GPG signing key and Doris `KEYS` management +- D: local and remote Git tag consistency checks +- E: source packaging from a tag +- G: optional signing of prebuilt binary files +- H: upload of a release candidate to dev SVN +- I: generation of the vote email draft +- K: generation of the announcement email draft + +The formal release flow must also package and sign the selected tag again, then +upload the new source artifacts to release SVN before generating the announcement +email. + +## Non-goals + +The toolkit will not: + +- Create, update, or push Git tags. +- Build or push container images. +- Package or publish Helm charts. +- Create GitHub releases or edit GitHub release notes. +- Send public email automatically. +- Count votes or generate the vote result email. +- Validate ASF release policy. +- Reuse, compare, move, or delete the dev SVN artifacts during formal release. +- Upload optional binary files. It only signs and checksums them locally. + +## Chosen Structure + +The repository will add `tools/release-tools` with four numbered entry points and +a shared library: + +```text +tools/release-tools/ +├── 01-check-env.sh +├── 02-package-sign-upload.sh +├── 03-vote-mail.sh +├── 04-release-complete.sh +├── README.md +├── release.env +├── lib/ +│ └── release-common.sh +└── tests/ + ├── run.sh + └── test-*.sh +``` + +The numbered scripts keep the workflow familiar to Doris release managers. The +shared library owns validation, tag checks, packaging, signing, checksums, and +common path handling so the dev and formal release flows cannot drift silently. + +## Configuration + +Every entry point sources `release.env`. The initial file will target `26.0.0` +and remain reusable for later versions. + +Required or derived configuration includes: + +```bash +VERSION="26.0.0" +TAG="${VERSION}" +GIT_REMOTE="upstream-apache" + +PKG_BASE="apache-doris-operator-${VERSION}-src" +ARCHIVE_PREFIX="${PKG_BASE}/" + +DEV_SVN_BASE="https://dist.apache.org/repos/dist/dev/doris/doris-operator" +DEV_SVN_DIR="${DEV_SVN_BASE}/${VERSION}" +RELEASE_SVN_BASE="https://dist.apache.org/repos/dist/release/doris/doris-operator" +RELEASE_SVN_DIR="${RELEASE_SVN_BASE}/${VERSION}" + +KEYS_URL="https://downloads.apache.org/doris/KEYS" +DEV_KEYS_SVN_BASE="https://dist.apache.org/repos/dist/dev/doris" +RELEASE_KEYS_SVN_BASE="https://dist.apache.org/repos/dist/release/doris" +``` + +The file also defines the Apache ID, Apache email, signer display name, optional +signing-key fingerprint, release notes URL, verification guide URL, download URL, +mailing-list addresses, work directory, and optional `BIN_FILES` array. + +The scripts read SVN credentials only from `ASF_USERNAME` and `ASF_PASSWORD` in +the environment. They never store credentials in `release.env` or generated +files. + +## Shared Library + +`lib/release-common.sh` will expose focused functions for these operations: + +- Validate required configuration and derived paths. +- Resolve exactly one usable signing key or honor `SIGNING_KEY`. +- Build SVN authentication argument arrays from environment variables. +- Confirm state-changing operations. +- Verify that the local tag exists. +- Resolve local and remote tags to commit IDs and compare them. +- Create the source archive from the selected tag. +- Create and verify detached ASCII-armored GPG signatures. +- Create and verify SHA-512 checksum files. +- Sign and checksum optional binary files next to their source files. +- Stage and commit one version directory to a configured SVN root. + +The source package command will use `git archive` with the prefix +`apache-doris-operator--src/`. Compression will use deterministic gzip +metadata so repeated packaging of the same Git tree does not add a timestamp to +the gzip header. + +## Script Flows + +### 01-check-env.sh + +This script prepares and validates the local signing environment. + +It will: + +1. Validate `release.env`. +2. Check for `git`, `gpg`, `svn`, `svnmucc`, `sha512sum`, `curl`, and `gzip`. +3. Set `GPG_TTY` so GPG can request a passphrase. +4. Check the recommended SHA-512 GPG settings and offer to append them. +5. Resolve the configured signing key. +6. Offer to import an existing secret key when no usable key exists. +7. Offer to generate an RSA-4096 signing key when requested. +8. Check whether the public key appears in the shared Doris `KEYS` file. +9. Offer to append the public key to both Doris dev and release `KEYS` files. +10. Run a local sign-and-verify test. +11. Report whether SVN credentials are present. + +Every state-changing key operation requires confirmation. + +### 02-package-sign-upload.sh + +This script prepares a dev SVN release candidate for a version whose dev folder +does not already exist. + +It will: + +1. Validate the environment and resolve the signer. +2. Verify that local and remote `TAG` values resolve to the same commit. +3. Create `apache-doris-operator--src.tar.gz` from the tag. +4. Create and verify its `.asc` signature. +5. Create and verify its `.sha512` checksum. +6. Sign and checksum every configured `BIN_FILES` item next to that file. +7. Check whether `DEV_SVN_DIR` already exists. +8. Stop without modifying SVN if the directory exists. +9. Display the target URL and files. +10. Require two confirmations before committing the source archive and sidecars + to dev SVN. + +The initial `26.0.0` flow will normally skip this script because the dev SVN +directory already exists. The script remains available for later releases. + +### 03-vote-mail.sh + +This script generates a vote email that follows the existing Doris Operator +mailing-list format. + +The message will include: + +- The `Apache Doris Operator ` vote subject. +- The GitHub release tag URL. +- The configured release notes URL. +- The dev SVN candidate URL. +- The signing-key fingerprint and Apache email. +- The shared Doris `KEYS` URL. +- The Doris verification guide. +- The standard 72-hour vote choices. +- Optional binary download URLs when `BIN_FILES` is configured. + +The script writes `vote-email.txt` and `vote-email.eml` to `WORK_DIR`, prints the +body for review, and tells the release manager to send it manually. + +### 04-release-complete.sh + +This script implements the user-specified formal release flow. + +It will: + +1. Validate the environment and resolve the signer. +2. Verify that local and remote `TAG` values resolve to the same commit. +3. Package the tag again into a new source tarball. +4. Create and verify a new detached signature. +5. Create and verify a new SHA-512 checksum. +6. Sign and checksum configured binary files again when present. +7. Check whether `RELEASE_SVN_DIR` already exists. +8. Stop without modifying SVN if the release directory exists. +9. Display the release SVN target and staged files. +10. Require two confirmations before committing the new source archive and + sidecars to release SVN. +11. Generate `announce-email.txt` and `announce-email.eml` after a successful + commit. + +The formal release flow does not inspect or modify `DEV_SVN_DIR`. + +`--mail-only` skips packaging and SVN operations and regenerates only the +announcement drafts. + +## Announcement Email + +The announcement draft will include: + +- Subject: `[ANNOUNCE] Apache Doris Operator release`. +- A short description of Doris Operator. +- The configured public download page or GitHub release URL. +- The formal source artifact URL. +- The release notes URL. +- A thank-you and signer name. + +The recipient stays configurable. The script never sends the email. + +## Safety and Error Handling + +All scripts will use `set -euo pipefail` and quote path expansions. + +State-changing operations use these safeguards: + +- No script creates or pushes a Git tag. +- Tag checks compare commit IDs, not tag object IDs. +- SVN credentials remain in environment variables. +- SVN target URLs are printed before checkout or commit. +- Dev and release uploads stop if their version directory already exists. +- SVN commits require two explicit confirmations. +- Generated signatures and checksums are verified before upload. +- A failed command exits with a clear error and leaves existing SVN content + untouched. +- Public emails are drafts only. + +## Testing + +Shell tests will run without modifying external services. Tests will place fake +`git`, `gpg`, `svn`, `svnmucc`, and checksum commands at the front of `PATH` or +use temporary local repositories where practical. + +The suite will cover: + +- Configuration validation and the `26.0.0` defaults. +- Artifact and archive-prefix naming. +- Local and remote tag commit comparison. +- Source archive creation from the configured tag. +- Signature and checksum creation and verification. +- Optional binary signing without binary upload. +- Exact dev and release SVN targets. +- Refusal to overwrite an existing SVN version directory. +- Vote email fields and Operator-specific wording. +- Announcement email fields. +- Formal release packaging that does not read or alter dev SVN. +- `--mail-only` behavior. +- Shell syntax checks for every script. + +`tests/run.sh` will provide one command for the complete release-tool test suite. + +## Acceptance Criteria + +The work is complete when: + +1. All four scripts and `release.env` are documented and executable. +2. `01-check-env.sh` can validate and prepare the selected signing environment. +3. `02-package-sign-upload.sh` can safely prepare a future Operator candidate. +4. `03-vote-mail.sh` creates reviewable Operator vote email drafts. +5. `04-release-complete.sh` can freshly package and sign Tag `26.0.0`, stage the + three source files for the configured release SVN directory, and generate the + announcement drafts. +6. Neither upload path overwrites an existing SVN version directory. +7. No script sends email or stores credentials. +8. The offline test suite passes. From 9914eb6d90201f2732eb52c7fcf7dec1fbb5834e Mon Sep 17 00:00:00 2001 From: "Mingyu Chen (Rayner)" Date: Thu, 16 Jul 2026 11:37:26 +0800 Subject: [PATCH 2/2] feat: add Doris Operator release tools --- .../specs/2026-07-15-release-tools-design.md | 56 ++- tools/release-tools/01-check-env.sh | 91 +++++ tools/release-tools/02-package-sign-upload.sh | 44 ++ tools/release-tools/03-vote-mail.sh | 81 ++++ tools/release-tools/04-release-complete.sh | 113 ++++++ tools/release-tools/README.md | 163 ++++++++ tools/release-tools/lib/release-common.sh | 384 ++++++++++++++++++ tools/release-tools/release.env | 58 +++ tools/release-tools/tests/run.sh | 25 ++ tools/release-tools/tests/test-check-env.sh | 58 +++ tools/release-tools/tests/test-common.sh | 152 +++++++ tools/release-tools/tests/test-config.sh | 51 +++ tools/release-tools/tests/test-mail.sh | 104 +++++ tools/release-tools/tests/test-syntax.sh | 52 +++ tools/release-tools/tests/test-workflows.sh | 163 ++++++++ tools/release-tools/tests/testlib.sh | 53 +++ 16 files changed, 1616 insertions(+), 32 deletions(-) create mode 100755 tools/release-tools/01-check-env.sh create mode 100755 tools/release-tools/02-package-sign-upload.sh create mode 100755 tools/release-tools/03-vote-mail.sh create mode 100755 tools/release-tools/04-release-complete.sh create mode 100644 tools/release-tools/README.md create mode 100644 tools/release-tools/lib/release-common.sh create mode 100644 tools/release-tools/release.env create mode 100755 tools/release-tools/tests/run.sh create mode 100755 tools/release-tools/tests/test-check-env.sh create mode 100755 tools/release-tools/tests/test-common.sh create mode 100755 tools/release-tools/tests/test-config.sh create mode 100755 tools/release-tools/tests/test-mail.sh create mode 100755 tools/release-tools/tests/test-syntax.sh create mode 100755 tools/release-tools/tests/test-workflows.sh create mode 100644 tools/release-tools/tests/testlib.sh diff --git a/docs/superpowers/specs/2026-07-15-release-tools-design.md b/docs/superpowers/specs/2026-07-15-release-tools-design.md index f974f497..ef0da7c8 100644 --- a/docs/superpowers/specs/2026-07-15-release-tools-design.md +++ b/docs/superpowers/specs/2026-07-15-release-tools-design.md @@ -26,7 +26,6 @@ The toolkit must provide the selected capabilities: - C: GPG signing key and Doris `KEYS` management - D: local and remote Git tag consistency checks - E: source packaging from a tag -- G: optional signing of prebuilt binary files - H: upload of a release candidate to dev SVN - I: generation of the vote email draft - K: generation of the announcement email draft @@ -47,7 +46,6 @@ The toolkit will not: - Count votes or generate the vote result email. - Validate ASF release policy. - Reuse, compare, move, or delete the dev SVN artifacts during formal release. -- Upload optional binary files. It only signs and checksums them locally. ## Chosen Structure @@ -98,9 +96,9 @@ DEV_KEYS_SVN_BASE="https://dist.apache.org/repos/dist/dev/doris" RELEASE_KEYS_SVN_BASE="https://dist.apache.org/repos/dist/release/doris" ``` -The file also defines the Apache ID, Apache email, signer display name, optional +The file also defines the Apache ID, Apache email, signer display name, required signing-key fingerprint, release notes URL, verification guide URL, download URL, -mailing-list addresses, work directory, and optional `BIN_FILES` array. +mailing-list addresses, and work directory. The scripts read SVN credentials only from `ASF_USERNAME` and `ASF_PASSWORD` in the environment. They never store credentials in `release.env` or generated @@ -119,7 +117,6 @@ files. - Create the source archive from the selected tag. - Create and verify detached ASCII-armored GPG signatures. - Create and verify SHA-512 checksum files. -- Sign and checksum optional binary files next to their source files. - Stage and commit one version directory to a configured SVN root. The source package command will use `git archive` with the prefix @@ -135,17 +132,17 @@ This script prepares and validates the local signing environment. It will: -1. Validate `release.env`. -2. Check for `git`, `gpg`, `svn`, `svnmucc`, `sha512sum`, `curl`, and `gzip`. -3. Set `GPG_TTY` so GPG can request a passphrase. -4. Check the recommended SHA-512 GPG settings and offer to append them. -5. Resolve the configured signing key. -6. Offer to import an existing secret key when no usable key exists. -7. Offer to generate an RSA-4096 signing key when requested. -8. Check whether the public key appears in the shared Doris `KEYS` file. -9. Offer to append the public key to both Doris dev and release `KEYS` files. -10. Run a local sign-and-verify test. -11. Report whether SVN credentials are present. +1. Require `SIGNING_KEY` in `release.env` and explain how to find the full + fingerprint when it is missing. +2. Validate `release.env`. +3. Check for `git`, `gpg`, `svn`, `svnmucc`, `sha512sum`, `curl`, and `gzip`. +4. Set `GPG_TTY` so GPG can request a passphrase. +5. Check the recommended SHA-512 GPG settings and offer to append them. +6. Resolve the configured signing key. +7. Check whether the public key appears in the shared Doris `KEYS` file. +8. Offer to append the public key to both Doris dev and release `KEYS` files. +9. Run a local sign-and-verify test. +10. Report whether SVN credentials are present. Every state-changing key operation requires confirmation. @@ -161,15 +158,13 @@ It will: 3. Create `apache-doris-operator--src.tar.gz` from the tag. 4. Create and verify its `.asc` signature. 5. Create and verify its `.sha512` checksum. -6. Sign and checksum every configured `BIN_FILES` item next to that file. -7. Check whether `DEV_SVN_DIR` already exists. -8. Stop without modifying SVN if the directory exists. -9. Display the target URL and files. -10. Require two confirmations before committing the source archive and sidecars +6. Check whether `DEV_SVN_DIR` already exists. +7. Stop without modifying SVN if the directory exists. +8. Display the target URL and files. +9. Require two confirmations before committing the source archive and sidecars to dev SVN. -The initial `26.0.0` flow will normally skip this script because the dev SVN -directory already exists. The script remains available for later releases. +The script refuses to overwrite an existing dev SVN version directory. ### 03-vote-mail.sh @@ -186,10 +181,9 @@ The message will include: - The shared Doris `KEYS` URL. - The Doris verification guide. - The standard 72-hour vote choices. -- Optional binary download URLs when `BIN_FILES` is configured. The script writes `vote-email.txt` and `vote-email.eml` to `WORK_DIR`, prints the -body for review, and tells the release manager to send it manually. +subject and body for review, and tells the release manager to send it manually. ### 04-release-complete.sh @@ -202,13 +196,12 @@ It will: 3. Package the tag again into a new source tarball. 4. Create and verify a new detached signature. 5. Create and verify a new SHA-512 checksum. -6. Sign and checksum configured binary files again when present. -7. Check whether `RELEASE_SVN_DIR` already exists. -8. Stop without modifying SVN if the release directory exists. -9. Display the release SVN target and staged files. -10. Require two confirmations before committing the new source archive and +6. Check whether `RELEASE_SVN_DIR` already exists. +7. Stop without modifying SVN if the release directory exists. +8. Display the release SVN target and staged files. +9. Require two confirmations before committing the new source archive and sidecars to release SVN. -11. Generate `announce-email.txt` and `announce-email.eml` after a successful +10. Generate `announce-email.txt` and `announce-email.eml` after a successful commit. The formal release flow does not inspect or modify `DEV_SVN_DIR`. @@ -259,7 +252,6 @@ The suite will cover: - Local and remote tag commit comparison. - Source archive creation from the configured tag. - Signature and checksum creation and verification. -- Optional binary signing without binary upload. - Exact dev and release SVN targets. - Refusal to overwrite an existing SVN version directory. - Vote email fields and Operator-specific wording. diff --git a/tools/release-tools/01-check-env.sh b/tools/release-tools/01-check-env.sh new file mode 100755 index 00000000..0c50b6a6 --- /dev/null +++ b/tools/release-tools/01-check-env.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=release.env +source "${HERE}/release.env" +# shellcheck source=lib/release-common.sh +source "${HERE}/lib/release-common.sh" + +require_configured_signing_key +validate_release_config + +printf '== Apache Doris Operator %s signing environment ==\n' "$VERSION" + +problems=0 +for tool in git gpg svn svnmucc sha512sum curl gzip; do + if command -v "$tool" >/dev/null 2>&1; then + ok "tool available: ${tool}" + else + warn "missing required tool: ${tool}" + problems=$((problems + 1)) + fi +done + +if [[ "$problems" -gt 0 ]]; then + die "${problems} required tool(s) missing" +fi + +export GPG_TTY="$(tty 2>/dev/null || true)" +ok "GPG_TTY=${GPG_TTY:-}" + +gpg_config="$(gpg_config_file)" +if recommended_gpg_settings_present "$gpg_config"; then + ok "GnuPG SHA-512 preferences are configured" +else + warn "recommended SHA-512 settings are missing from ${gpg_config}" + if confirm "Append the recommended SHA-512 settings?"; then + append_recommended_gpg_settings "$gpg_config" + ok "updated ${gpg_config}" + else + problems=$((problems + 1)) + fi +fi + +SIGNER="$(resolve_signing_key)" +ok "signing-key fingerprint: ${SIGNER}" + +if published_key_exists "$SIGNER"; then + ok "signing key is present in ${KEYS_URL}" +else + warn "signing key is not present in ${KEYS_URL}" + if confirm "Append this key to both Doris dev and release KEYS files?"; then + publish_key_to_doris_keys "$SIGNER" + else + problems=$((problems + 1)) + fi +fi + +if test_signing_key "$SIGNER"; then + ok "local sign-and-verify test succeeded" +else + warn "local sign-and-verify test failed" + problems=$((problems + 1)) +fi + +if [[ -n "${ASF_USERNAME:-}" && -n "${ASF_PASSWORD:-}" ]]; then + ok "ASF_USERNAME and ASF_PASSWORD are present in the environment" +else + warn "ASF_USERNAME and ASF_PASSWORD are not both set; SVN publishing will require them" +fi + +if [[ "$problems" -gt 0 ]]; then + die "environment is not ready (${problems} problem(s))" +fi +ok "environment looks READY for ${VERSION}" diff --git a/tools/release-tools/02-package-sign-upload.sh b/tools/release-tools/02-package-sign-upload.sh new file mode 100755 index 00000000..eb26b79b --- /dev/null +++ b/tools/release-tools/02-package-sign-upload.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=release.env +source "${HERE}/release.env" +# shellcheck source=lib/release-common.sh +source "${HERE}/lib/release-common.sh" + +validate_release_config +require_tools git gpg svn sha512sum gzip || die "install the missing release prerequisites" +export GPG_TTY="$(tty 2>/dev/null || true)" + +SIGNER="$(resolve_signing_key)" +ok "signer: ${SIGNER}" +verify_tag_consistency +prepare_source_artifacts "$SIGNER" + +stage_and_commit_version_dir \ + "$DEV_SVN_BASE" \ + "$DEV_SVN_DIR" \ + "dev-svn" \ + "Add Apache Doris Operator ${VERSION} release candidate" \ + "${SOURCE_ARTIFACTS[@]}" + +if [[ "$SVN_COMMITTED" -eq 1 ]]; then + ok "candidate uploaded; next run ./03-vote-mail.sh" +fi diff --git a/tools/release-tools/03-vote-mail.sh b/tools/release-tools/03-vote-mail.sh new file mode 100755 index 00000000..82973f18 --- /dev/null +++ b/tools/release-tools/03-vote-mail.sh @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=release.env +source "${HERE}/release.env" +# shellcheck source=lib/release-common.sh +source "${HERE}/lib/release-common.sh" + +validate_release_config +require_tools gpg || die "GnuPG is required to resolve the signing fingerprint" + +SIGNER="$(resolve_signing_key)" +mkdir -p "$WORK_DIR" +subject="[VOTE] Release Apache Doris Operator ${VERSION}" +body_file="${WORK_DIR}/vote-email.txt" +eml_file="${WORK_DIR}/vote-email.eml" + +BODY="$( + cat < "$body_file" +{ + printf 'To: %s\n' "$VOTE_TO" + printf 'Subject: %s\n' "$subject" + printf 'Content-Type: text/plain; charset=UTF-8\n' + printf '\n%s\n' "$BODY" +} > "$eml_file" + +ok "vote draft: ${body_file}" +ok "mail draft: ${eml_file}" +printf 'Subject: %s\n' "$subject" +printf '%s\n' '----------------------------------------------------------------' +printf '%s\n' "$BODY" +printf '%s\n' '----------------------------------------------------------------' +printf 'Review and send the message manually to %s. No email was sent.\n' "$VOTE_TO" diff --git a/tools/release-tools/04-release-complete.sh b/tools/release-tools/04-release-complete.sh new file mode 100755 index 00000000..92c8c8cf --- /dev/null +++ b/tools/release-tools/04-release-complete.sh @@ -0,0 +1,113 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=release.env +source "${HERE}/release.env" +# shellcheck source=lib/release-common.sh +source "${HERE}/lib/release-common.sh" + +usage() { + printf 'Usage: %s [--mail-only]\n' "$0" + printf ' --mail-only regenerate announcement drafts without packaging or SVN\n' +} + +mail_only=0 +while [[ "$#" -gt 0 ]]; do + case "$1" in + --mail-only) mail_only=1 ;; + -h|--help) usage; exit 0 ;; + *) usage >&2; die "unknown argument: $1" ;; + esac + shift +done + +write_announce_email() { + local subject body_file eml_file body + mkdir -p "$WORK_DIR" + subject="[ANNOUNCE] Apache Doris Operator ${VERSION} release" + body_file="${WORK_DIR}/announce-email.txt" + eml_file="${WORK_DIR}/announce-email.eml" + + body="$( + cat < "$body_file" + { + printf 'To: %s\n' "$ANNOUNCE_TO" + printf 'Subject: %s\n' "$subject" + printf 'Content-Type: text/plain; charset=UTF-8\n' + printf '\n%s\n' "$body" + } > "$eml_file" + + ok "announcement draft: ${body_file}" + ok "mail draft: ${eml_file}" + printf '%s\n' '----------------------------------------------------------------' + printf '%s\n' "$body" + printf '%s\n' '----------------------------------------------------------------' + printf 'Review and send the message manually to %s. No email was sent.\n' "$ANNOUNCE_TO" +} + +if [[ "$mail_only" -eq 1 ]]; then + validate_release_config mail + ok "mail-only mode: skipping tag, package, signing, and SVN operations" + write_announce_email + exit 0 +fi + +validate_release_config +require_tools git gpg svn sha512sum gzip || die "install the missing release prerequisites" +export GPG_TTY="$(tty 2>/dev/null || true)" + +SIGNER="$(resolve_signing_key)" +ok "signer: ${SIGNER}" +verify_tag_consistency +prepare_source_artifacts "$SIGNER" + +stage_and_commit_version_dir \ + "$RELEASE_SVN_BASE" \ + "$RELEASE_SVN_DIR" \ + "release-svn" \ + "Release Apache Doris Operator ${VERSION}" \ + "${SOURCE_ARTIFACTS[@]}" + +if [[ "$SVN_COMMITTED" -eq 1 ]]; then + write_announce_email +fi diff --git a/tools/release-tools/README.md b/tools/release-tools/README.md new file mode 100644 index 00000000..9b64a668 --- /dev/null +++ b/tools/release-tools/README.md @@ -0,0 +1,163 @@ + + +# Doris Operator release tools + +These scripts package, sign, and publish Apache Doris Operator source releases. +They publish source artifacts only and generate vote and announcement email +drafts. They do not create Git tags or send email. + +The checked-in defaults target version and Git tag `26.0.0`. + +## Prerequisites + +Install `git`, `gpg`, `svn`, `svnmucc`, `sha512sum`, `curl`, and `gzip`. The +selected Git tag must already exist locally and on the remote configured by +`GIT_REMOTE`. + +Edit `release.env` before each release. In particular, verify: + +- `VERSION`, `TAG`, `GIT_REMOTE`, and all derived artifact/SVN paths. +- `APACHE_ID`, `APACHE_EMAIL`, and `SIGNER_NAME`. +- `SIGNING_KEY`: required full fingerprint of a locally available secret key. +- release notes, verification, download, and mailing-list URLs. +- `WORK_DIR`, which stores generated artifacts, SVN working copies, and drafts. + +`TAG` must be the final version with no RC suffix. Source artifacts and SVN +version directories also use the final version: + +```text +apache-doris-operator-26.0.0-src.tar.gz +apache-doris-operator-26.0.0-src.tar.gz.asc +apache-doris-operator-26.0.0-src.tar.gz.sha512 +``` + +SVN credentials are never stored in `release.env` or generated mail files. +Export them in the shell that runs a publishing script: + +```bash +export ASF_USERNAME="" +export ASF_PASSWORD="" +``` + +Find the full signing-key fingerprint with: + +```bash +gpg --list-secret-keys --keyid-format=long --with-fingerprint +``` + +Copy that fingerprint into `release.env`; `01-check-env.sh` exits immediately +with this guidance when `SIGNING_KEY` is empty. + +## Workflow + +Run commands from this directory: + +```bash +cd tools/release-tools +``` + +### 1. Check the signing environment + +```bash +./01-check-env.sh +``` + +This requires the signing-key fingerprint configured in `release.env`, checks +the required tools, configures `GPG_TTY`, checks the recommended SHA-512 GPG +preferences, verifies the shared Doris `KEYS` file, and performs a local +sign/verify test. Editing `gpg.conf` and appending a public key to the dev and +release `KEYS` files each require explicit confirmation. + +### 2. Package a release candidate when needed + +```bash +./02-package-sign-upload.sh +``` + +This verifies that the local and remote tags resolve to the same commit, +creates a deterministic source archive with `git archive` and `gzip -n`, signs +it, writes a SHA-512 sidecar, and uploads the three source files to: + +```text +https://dist.apache.org/repos/dist/dev/doris/doris-operator// +``` + +If the configured dev SVN directory already exists, skip this step or select a +new version. The script refuses to overwrite an existing version directory and +asks for confirmation both before staging and before commit. + +### 3. Generate the vote email + +```bash +./03-vote-mail.sh +``` + +This writes `vote-email.txt` and `vote-email.eml` under `WORK_DIR`. +It prints the subject and body, then leaves sending to the release manager. + +### 4. Complete a passed release + +```bash +./04-release-complete.sh +``` + +The formal release is created independently from dev SVN. The script re-checks +the local and remote tag, freshly packages the selected Git tag, signs the new +archive, creates a fresh checksum, and uploads those newly generated source +files to: + +```text +https://dist.apache.org/repos/dist/release/doris/doris-operator// +``` + +It does not inspect, compare, promote, move, or delete anything under dev SVN. +It refuses to overwrite an existing release directory and requires two +confirmations. Only after a successful commit does it create +`announce-email.txt` and `announce-email.eml`. + +To regenerate only the announcement drafts: + +```bash +./04-release-complete.sh --mail-only +``` + +`--mail-only` skips Git, packaging, GPG, checksums, and SVN. + +## Safety boundaries + +- No script creates, updates, or pushes a Git tag. +- Local and remote tags are compared by peeled commit ID. +- Generated signatures and checksums are verified immediately. +- Dev and release uploads stop before checkout if the version directory exists. +- SVN target URLs and staged files are shown before both confirmations. +- SVN uploads contain only the source archive, signature, and checksum. +- Public emails are drafts only. +- No email was sent by any script; the release manager sends drafts manually. +- Formal release packaging is independent from dev SVN. + +## Tests + +The suite uses temporary Git repositories and fake GPG/SVN commands. It does +not modify a real keyring, remote Git repository, SVN repository, or mail +system. + +```bash +./tests/run.sh +``` diff --git a/tools/release-tools/lib/release-common.sh b/tools/release-tools/lib/release-common.sh new file mode 100644 index 00000000..6d7ffab2 --- /dev/null +++ b/tools/release-tools/lib/release-common.sh @@ -0,0 +1,384 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +if [[ -n "${RELEASE_COMMON_SH_LOADED:-}" ]]; then + return 0 +fi +RELEASE_COMMON_SH_LOADED=1 + +ok() { printf '[ OK ] %s\n' "$*"; } +warn() { printf '[WARN] %s\n' "$*" >&2; } +die() { printf '[FAIL] %s\n' "$*" >&2; exit 1; } + +confirm() { + local answer + read -r -p "$1 [y/N] " answer || return 1 + case "$answer" in + y|Y|yes|YES|Yes) return 0 ;; + *) return 1 ;; + esac +} + +require_tools() { + local tool missing=0 + for tool in "$@"; do + if ! command -v "$tool" >/dev/null 2>&1; then + warn "missing required tool: ${tool}" + missing=1 + fi + done + [[ "$missing" -eq 0 ]] +} + +require_config_value() { + local name="$1" + [[ -n "${!name:-}" ]] || die "release.env: ${name} must not be empty" +} + +require_configured_signing_key() { + if [[ -n "${SIGNING_KEY:-}" ]]; then + return 0 + fi + + { + printf '[FAIL] SIGNING_KEY is required in release.env.\n' + printf 'Find available secret signing keys with:\n' + printf ' gpg --list-secret-keys --keyid-format=long --with-fingerprint\n' + printf 'Copy the full fingerprint into release.env, for example:\n' + printf ' SIGNING_KEY=""\n' + } >&2 + return 1 +} + +validate_release_config() { + local mode="${1:-full}" name + local required=( + ROOT VERSION TAG PKG_BASE ARCHIVE_PREFIX + DEV_SVN_BASE DEV_SVN_DIR RELEASE_SVN_BASE RELEASE_SVN_DIR + KEYS_URL DEV_KEYS_SVN_BASE RELEASE_KEYS_SVN_BASE + APACHE_ID APACHE_EMAIL SIGNER_NAME GITHUB_TAG_URL RELEASE_NOTES_URL + VERIFY_GUIDE_URL DOWNLOAD_PAGE_URL VOTE_TO ANNOUNCE_TO WORK_DIR + ) + + if [[ "$mode" != "mail" ]]; then + required+=(REPO_DIR GIT_REMOTE) + fi + + for name in "${required[@]}"; do + require_config_value "$name" + done + + [[ "$TAG" == "$VERSION" ]] || die "release.env: TAG must equal VERSION" + [[ "$PKG_BASE" == "apache-doris-operator-${VERSION}-src" ]] || + die "release.env: PKG_BASE must be apache-doris-operator-${VERSION}-src" + [[ "$ARCHIVE_PREFIX" == "${PKG_BASE}/" ]] || + die "release.env: ARCHIVE_PREFIX must be ${PKG_BASE}/" + [[ "$DEV_SVN_DIR" == "${DEV_SVN_BASE}/${VERSION}" ]] || + die "release.env: DEV_SVN_DIR must be ${DEV_SVN_BASE}/${VERSION}" + [[ "$RELEASE_SVN_DIR" == "${RELEASE_SVN_BASE}/${VERSION}" ]] || + die "release.env: RELEASE_SVN_DIR must be ${RELEASE_SVN_BASE}/${VERSION}" + [[ "$WORK_DIR" == /* ]] || die "release.env: WORK_DIR must be an absolute path" + + if [[ "$mode" != "mail" ]]; then + [[ "$REPO_DIR" == /* ]] || die "release.env: REPO_DIR must be an absolute path" + fi +} + +list_secret_key_fingerprints() { + gpg --batch --with-colons --list-secret-keys 2>/dev/null | + awk -F: '$1 == "sec" { want_fingerprint = 1; next } + want_fingerprint && $1 == "fpr" { print $10; want_fingerprint = 0 }' +} + +signing_key_fingerprint() { + local key="$1" fingerprint + fingerprint="$( + gpg --batch --with-colons --fingerprint --list-secret-keys "$key" 2>/dev/null | + awk -F: '$1 == "sec" { want_fingerprint = 1; next } + want_fingerprint && $1 == "fpr" { print $10; exit }' + )" + [[ -n "$fingerprint" ]] || die "cannot resolve fingerprint for signing key: ${key}" + printf '%s\n' "$fingerprint" +} + +resolve_signing_key() { + local fingerprints count + + if [[ -n "${SIGNING_KEY:-}" ]]; then + gpg --batch --list-secret-keys "$SIGNING_KEY" >/dev/null 2>&1 || + die "configured SIGNING_KEY is not a usable secret key: ${SIGNING_KEY}" + signing_key_fingerprint "$SIGNING_KEY" + return + fi + + fingerprints="$(list_secret_key_fingerprints)" + count="$(printf '%s\n' "$fingerprints" | awk 'NF { count++ } END { print count + 0 }')" + case "$count" in + 1) printf '%s\n' "$fingerprints" ;; + 0) die "no usable secret key found; run ./01-check-env.sh first" ;; + *) die "multiple secret keys found; set SIGNING_KEY in release.env" ;; + esac +} + +verify_tag_consistency() { + local local_commit remote_refs remote_commit + + git -C "$REPO_DIR" show-ref --verify --quiet "refs/tags/${TAG}" || + die "local tag ${TAG} does not exist in ${REPO_DIR}" + local_commit="$(git -C "$REPO_DIR" rev-parse "${TAG}^{commit}")" + + remote_refs="$( + git -C "$REPO_DIR" ls-remote --tags "$GIT_REMOTE" \ + "refs/tags/${TAG}" "refs/tags/${TAG}^{}" + )" || die "failed to query tag ${TAG} from ${GIT_REMOTE}" + remote_commit="$( + printf '%s\n' "$remote_refs" | + awk -v direct="refs/tags/${TAG}" -v peeled="refs/tags/${TAG}^{}" ' + $2 == direct { direct_id = $1 } + $2 == peeled { peeled_id = $1 } + END { if (peeled_id != "") print peeled_id; else print direct_id }' + )" + + [[ -n "$remote_commit" ]] || die "remote tag ${TAG} does not exist on ${GIT_REMOTE}" + [[ "$local_commit" == "$remote_commit" ]] || + die "tag mismatch for ${TAG}: local=${local_commit}, ${GIT_REMOTE}=${remote_commit}" + ok "tag ${TAG} resolves to ${local_commit} locally and on ${GIT_REMOTE}" +} + +SOURCE_ARCHIVE="" +SOURCE_SIGNATURE="" +SOURCE_CHECKSUM="" +SOURCE_ARTIFACTS=() + +create_source_archive() { + local archive temporary + mkdir -p "$WORK_DIR" + archive="${WORK_DIR}/${PKG_BASE}.tar.gz" + temporary="${archive}.tmp.$$" + rm -f "$temporary" "$archive" + + if ! ( + set -o pipefail + git -C "$REPO_DIR" archive --format=tar --prefix="$ARCHIVE_PREFIX" "$TAG" | + gzip -n > "$temporary" + ); then + rm -f "$temporary" + die "failed to create source archive from tag ${TAG}" + fi + + mv "$temporary" "$archive" + SOURCE_ARCHIVE="$archive" + ok "source archive created: ${SOURCE_ARCHIVE}" +} + +sign_and_verify() { + local file="$1" signer="$2" signature="${1}.asc" + rm -f "$signature" + gpg --local-user "$signer" --armor --output "$signature" --detach-sign "$file" + gpg --verify "$signature" "$file" + ok "signature verified: ${signature}" +} + +checksum_and_verify() { + local file="$1" directory basename checksum + directory="$(cd "$(dirname "$file")" && pwd)" + basename="$(basename "$file")" + checksum="${basename}.sha512" + ( + cd "$directory" + rm -f "$checksum" + sha512sum "$basename" > "$checksum" + sha512sum --check "$checksum" + ) + ok "SHA-512 verified: ${file}.sha512" +} + +prepare_source_artifacts() { + local signer="$1" + create_source_archive + sign_and_verify "$SOURCE_ARCHIVE" "$signer" + checksum_and_verify "$SOURCE_ARCHIVE" + SOURCE_SIGNATURE="${SOURCE_ARCHIVE}.asc" + SOURCE_CHECKSUM="${SOURCE_ARCHIVE}.sha512" + SOURCE_ARTIFACTS=("$SOURCE_ARCHIVE" "$SOURCE_SIGNATURE" "$SOURCE_CHECKSUM") +} + +SVN_AUTH_ARGS=() +SVN_COMMITTED=0 + +build_svn_auth_args() { + SVN_AUTH_ARGS=(--non-interactive --no-auth-cache) + [[ -n "${ASF_USERNAME:-}" ]] && SVN_AUTH_ARGS+=(--username "$ASF_USERNAME") + [[ -n "${ASF_PASSWORD:-}" ]] && SVN_AUTH_ARGS+=(--password "$ASF_PASSWORD") + return 0 +} + +svn_url_exists() { + local url="$1" + svn info "${SVN_AUTH_ARGS[@]}" "$url" >/dev/null 2>&1 +} + +stage_and_commit_version_dir() { + local svn_base="$1" svn_dir="$2" stage_name="$3" commit_message="$4" + shift 4 + local relative working_copy file + + [[ "$svn_dir" == "${svn_base}/"* ]] || die "SVN target is not below configured base: ${svn_dir}" + relative="${svn_dir#${svn_base}/}" + [[ -n "$relative" && "$relative" != */* ]] || die "SVN target must be one version directory: ${svn_dir}" + [[ "$#" -gt 0 ]] || die "no files supplied for SVN upload" + + SVN_COMMITTED=0 + build_svn_auth_args + if svn_url_exists "$svn_dir"; then + die "SVN version directory already exists; refusing to overwrite: ${svn_dir}" + fi + + printf 'Target SVN directory: %s/\n' "$svn_dir" + printf 'Files to stage:\n' + for file in "$@"; do + [[ -f "$file" ]] || die "staged file does not exist: ${file}" + printf ' %s\n' "$file" + done + + if ! confirm "Checkout ${svn_base} and stage these files?"; then + warn "stopped before modifying SVN" + return 0 + fi + + mkdir -p "$WORK_DIR" + working_copy="${WORK_DIR}/${stage_name}" + rm -rf "$working_copy" + svn checkout --depth empty "${SVN_AUTH_ARGS[@]}" "$svn_base" "$working_copy" + mkdir -p "${working_copy}/${relative}" + for file in "$@"; do + cp "$file" "${working_copy}/${relative}/" + done + svn add "${working_copy}/${relative}" + svn status "$working_copy" + + printf 'SVN commit target: %s/\n' "$svn_dir" + if ! confirm "FINAL confirmation: commit the staged release files?"; then + warn "staged working copy left at ${working_copy}" + return 0 + fi + + svn commit "${SVN_AUTH_ARGS[@]}" -m "$commit_message" "$working_copy" + SVN_COMMITTED=1 + ok "committed release files: ${svn_dir}/" +} + +gpg_config_file() { + local home="${GNUPGHOME:-}" + if [[ -z "$home" ]]; then + home="$(gpgconf --list-dirs homedir 2>/dev/null || true)" + fi + if [[ -z "$home" ]]; then + [[ -n "${HOME:-}" ]] || die "cannot locate the GnuPG home directory" + home="${HOME}/.gnupg" + fi + printf '%s/gpg.conf\n' "$home" +} + +recommended_gpg_settings_present() { + local config="$1" + grep -Eq '^[[:space:]]*personal-digest-preferences[[:space:]]+SHA512([[:space:]]|$)' "$config" 2>/dev/null && + grep -Eq '^[[:space:]]*cert-digest-algo[[:space:]]+SHA512([[:space:]]|$)' "$config" 2>/dev/null +} + +append_recommended_gpg_settings() { + local config="$1" + mkdir -p "$(dirname "$config")" + { + printf '\n# ASF release signing recommendations\n' + printf 'personal-digest-preferences SHA512\n' + printf 'cert-digest-algo SHA512\n' + printf 'default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed\n' + } >> "$config" +} + +import_secret_key() { + local key_file="$1" + [[ -f "$key_file" ]] || die "secret key file not found: ${key_file}" + gpg --import "$key_file" +} + +generate_signing_key() { + local real_name="$1" email="$2" + [[ "${#real_name}" -ge 5 ]] || die "signing-key real name must contain at least five characters" + [[ "$email" == *@apache.org ]] || die "signing-key email must be an @apache.org address" + gpg --quick-generate-key "${real_name} (CODE SIGNING KEY) <${email}>" rsa4096 sign never +} + +key_in_keys_stream() { + local fingerprint="$1" keyring result=1 + keyring="$(mktemp -d "${TMPDIR:-/tmp}/doris-operator-keys.XXXXXX")" + chmod 700 "$keyring" + if gpg --homedir "$keyring" --batch --import >/dev/null 2>&1 && + gpg --homedir "$keyring" --batch --list-keys "$fingerprint" >/dev/null 2>&1; then + result=0 + fi + rm -rf "$keyring" + return "$result" +} + +published_key_exists() { + local fingerprint="$1" + curl -fsSL "$KEYS_URL" | key_in_keys_stream "$fingerprint" +} + +publish_key_to_doris_keys() { + local fingerprint="$1" spec name base working_copy + build_svn_auth_args + mkdir -p "$WORK_DIR" + + for spec in "dev=${DEV_KEYS_SVN_BASE}" "release=${RELEASE_KEYS_SVN_BASE}"; do + name="${spec%%=*}" + base="${spec#*=}" + working_copy="${WORK_DIR}/keys-${name}" + rm -rf "$working_copy" + svn checkout --depth files "${SVN_AUTH_ARGS[@]}" "$base" "$working_copy" + [[ -f "${working_copy}/KEYS" ]] || die "KEYS not found in ${base}" + + if key_in_keys_stream "$fingerprint" < "${working_copy}/KEYS"; then + ok "signing key already present in ${name} KEYS" + continue + fi + + { + printf '\n' + gpg --list-sigs "$fingerprint" + gpg --armor --export "$fingerprint" + } >> "${working_copy}/KEYS" + svn commit "${SVN_AUTH_ARGS[@]}" -m "Add KEYS entry for ${APACHE_ID}" "${working_copy}/KEYS" + ok "appended signing key to ${base}/KEYS" + done +} + +test_signing_key() { + local signer="$1" temporary signature result=0 + temporary="$(mktemp "${TMPDIR:-/tmp}/doris-operator-sign.XXXXXX")" + signature="${temporary}.asc" + printf 'Apache Doris Operator %s signing test\n' "$TAG" > "$temporary" + if ! gpg --local-user "$signer" --armor --output "$signature" --detach-sign "$temporary" || + ! gpg --verify "$signature" "$temporary"; then + result=1 + fi + rm -f "$temporary" "$signature" + return "$result" +} diff --git a/tools/release-tools/release.env b/tools/release-tools/release.env new file mode 100644 index 00000000..f5c4b538 --- /dev/null +++ b/tools/release-tools/release.env @@ -0,0 +1,58 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Shared configuration for the Apache Doris Operator release helpers. +# Edit this file once for each release. All four numbered scripts source it. + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_DIR="${ROOT}/../.." + +VERSION="26.0.0" +TAG="${VERSION}" +GIT_REMOTE="upstream-apache" + +PKG_BASE="apache-doris-operator-${VERSION}-src" +ARCHIVE_PREFIX="${PKG_BASE}/" + +DEV_SVN_BASE="https://dist.apache.org/repos/dist/dev/doris/doris-operator" +DEV_SVN_DIR="${DEV_SVN_BASE}/${VERSION}" +RELEASE_SVN_BASE="https://dist.apache.org/repos/dist/release/doris/doris-operator" +RELEASE_SVN_DIR="${RELEASE_SVN_BASE}/${VERSION}" + +KEYS_URL="https://downloads.apache.org/doris/KEYS" +DEV_KEYS_SVN_BASE="https://dist.apache.org/repos/dist/dev/doris" +RELEASE_KEYS_SVN_BASE="https://dist.apache.org/repos/dist/release/doris" + +APACHE_ID="morningman" +APACHE_EMAIL="morningman@apache.org" +SIGNER_NAME="Mingyu Chen" +# Required: full fingerprint of the secret key used to sign release artifacts. +SIGNING_KEY="" + +GITHUB_TAG_URL="https://github.com/apache/doris-operator/releases/tag/${TAG}" +RELEASE_NOTES_URL="https://github.com/apache/doris-operator/issues/506" +VERIFY_GUIDE_URL="https://doris.apache.org/community/release-and-verify/release-verify" +DOWNLOAD_PAGE_URL="${GITHUB_TAG_URL}" + +VOTE_TO="dev@doris.apache.org" +ANNOUNCE_TO="announce@apache.org" + +WORK_DIR="${ROOT}/${VERSION}" + +# SVN credentials are read only from the process environment: +# export ASF_USERNAME="" +# export ASF_PASSWORD="" diff --git a/tools/release-tools/tests/run.sh b/tools/release-tools/tests/run.sh new file mode 100755 index 00000000..a6857ef2 --- /dev/null +++ b/tools/release-tools/tests/run.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -euo pipefail +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +for test_script in "${ROOT}"/tests/test-*.sh; do + printf '== %s ==\n' "${test_script##*/}" + "$test_script" +done diff --git a/tools/release-tools/tests/test-check-env.sh b/tools/release-tools/tests/test-check-env.sh new file mode 100755 index 00000000..7f91b5d6 --- /dev/null +++ b/tools/release-tools/tests/test-check-env.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -euo pipefail +# shellcheck source=testlib.sh +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/testlib.sh" +# shellcheck source=../release.env +source "${TOOLS_ROOT}/release.env" +# shellcheck source=../lib/release-common.sh +source "${TOOLS_ROOT}/lib/release-common.sh" + +script="${TOOLS_ROOT}/01-check-env.sh" +common="${TOOLS_ROOT}/lib/release-common.sh" + +for tool in git gpg svn svnmucc sha512sum curl gzip; do + grep -qw "$tool" "$script" || fail "environment check omits required tool: $tool" +done + +assert_file_contains "$script" 'export GPG_TTY=' +assert_file_contains "$script" 'Append the recommended SHA-512 settings?' +assert_file_contains "$script" 'require_configured_signing_key' +assert_file_not_contains "$script" 'Import a secret key into this GnuPG keyring?' +assert_file_not_contains "$script" 'Generate a new RSA-4096 signing key with no expiry?' +assert_file_contains "$script" 'Append this key to both Doris dev and release KEYS files?' +assert_file_contains "$common" '"dev=${DEV_KEYS_SVN_BASE}" "release=${RELEASE_KEYS_SVN_BASE}"' +assert_file_contains "$common" 'test_signing_key()' + +error_file="$(mktemp)" +trap 'rm -f "$error_file"' EXIT +if (SIGNING_KEY=""; require_configured_signing_key) 2>"$error_file"; then + fail "empty SIGNING_KEY was accepted" +fi +assert_file_contains "$error_file" 'SIGNING_KEY is required in release.env' +assert_file_contains "$error_file" 'gpg --list-secret-keys --keyid-format=long --with-fingerprint' +assert_file_contains "$error_file" 'SIGNING_KEY=""' + +(SIGNING_KEY="0123456789ABCDEF"; require_configured_signing_key) + +if grep -Eq 'printf.*ASF_PASSWORD|echo.*ASF_PASSWORD' "$script"; then + fail "environment check may print the SVN password" +fi + +pass diff --git a/tools/release-tools/tests/test-common.sh b/tools/release-tools/tests/test-common.sh new file mode 100755 index 00000000..2766cf0a --- /dev/null +++ b/tools/release-tools/tests/test-common.sh @@ -0,0 +1,152 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -euo pipefail +# shellcheck source=testlib.sh +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/testlib.sh" +# shellcheck source=../release.env +source "${TOOLS_ROOT}/release.env" +# shellcheck source=../lib/release-common.sh +source "${TOOLS_ROOT}/lib/release-common.sh" + +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT + +repo="${tmp}/repo" +remote="${tmp}/remote.git" +git init -q "$repo" +git -C "$repo" config user.name "Release Test" +git -C "$repo" config user.email "release-test@example.com" +printf 'release content\n' > "${repo}/content.txt" +git -C "$repo" add content.txt +git -C "$repo" commit -q -m initial +git -C "$repo" tag 9.9.9 +git init -q --bare "$remote" +git -C "$repo" push -q "$remote" refs/tags/9.9.9 + +VERSION="9.9.9" +TAG="$VERSION" +REPO_DIR="$repo" +GIT_REMOTE="$remote" +PKG_BASE="apache-doris-operator-${VERSION}-src" +ARCHIVE_PREFIX="${PKG_BASE}/" +WORK_DIR="${tmp}/work" + +verify_tag_consistency > "${tmp}/tag-output" +assert_file_contains "${tmp}/tag-output" "tag 9.9.9 resolves" + +create_source_archive +assert_exists "$SOURCE_ARCHIVE" +gzip -dc "$SOURCE_ARCHIVE" | tar -tf - > "${tmp}/tar-list" +if awk -v prefix="${ARCHIVE_PREFIX}" 'index($0, prefix) != 1 { exit 1 }' "${tmp}/tar-list"; then + : +else + fail "archive contains an entry outside ${ARCHIVE_PREFIX}" +fi +first_digest="$(sha512sum "$SOURCE_ARCHIVE" | awk '{print $1}')" +create_source_archive +second_digest="$(sha512sum "$SOURCE_ARCHIVE" | awk '{print $1}')" +assert_eq "$first_digest" "$second_digest" + +fake_bin="${tmp}/fake-bin" +mkdir -p "$fake_bin" +cat > "${fake_bin}/gpg" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +if [[ " $* " == *" --with-colons "* ]]; then + printf 'sec:-:4096:1:TESTKEY:0:0:::::::scESC:::+:::23:\n' + printf 'fpr:::::::::0123456789ABCDEF0123456789ABCDEF01234567:\n' + exit 0 +fi +if [[ " $* " == *" --list-secret-keys "* ]]; then + exit 0 +fi +output="" +while [[ "$#" -gt 0 ]]; do + case "$1" in + --output) output="$2"; shift 2 ;; + --verify) [[ -f "$2" && -f "$3" ]]; exit 0 ;; + *) shift ;; + esac +done +[[ -n "$output" ]] || exit 1 +printf 'fake signature\n' > "$output" +EOF +chmod +x "${fake_bin}/gpg" + +real_path="$PATH" +PATH="${fake_bin}:${PATH}" +SIGNING_KEY="0123456789ABCDEF0123456789ABCDEF01234567" +signer="$(resolve_signing_key)" +assert_eq "$SIGNING_KEY" "$signer" +sign_and_verify "$SOURCE_ARCHIVE" "$signer" +checksum_and_verify "$SOURCE_ARCHIVE" +assert_exists "${SOURCE_ARCHIVE}.asc" +assert_exists "${SOURCE_ARCHIVE}.sha512" +assert_file_contains "${SOURCE_ARCHIVE}.sha512" "$(basename "$SOURCE_ARCHIVE")" + +cat > "${fake_bin}/svn" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +printf '%s\n' "$*" >> "$FAKE_SVN_LOG" +command="$1" +shift +case "$command" in + info) + url="${@: -1}" + [[ -n "${FAKE_EXISTING_URL:-}" && "$url" == "$FAKE_EXISTING_URL" ]] + ;; + checkout) + destination="${@: -1}" + mkdir -p "$destination" + ;; + add|status|commit) exit 0 ;; + *) exit 1 ;; +esac +EOF +chmod +x "${fake_bin}/svn" + +export PATH +export FAKE_SVN_LOG="${tmp}/svn.log" +: > "$FAKE_SVN_LOG" +unset FAKE_EXISTING_URL || true +DEV_SVN_BASE="https://dist.example.test/dev/doris-operator" +DEV_SVN_DIR="${DEV_SVN_BASE}/${VERSION}" +ASF_USERNAME="release-user" +ASF_PASSWORD="release-password" +export ASF_USERNAME ASF_PASSWORD +SOURCE_ARTIFACTS=("$SOURCE_ARCHIVE" "${SOURCE_ARCHIVE}.asc" "${SOURCE_ARCHIVE}.sha512") + +stage_and_commit_version_dir "$DEV_SVN_BASE" "$DEV_SVN_DIR" "dev-svn" "test commit" \ + "${SOURCE_ARTIFACTS[@]}" <<< $'y\ny' +assert_eq "1" "$SVN_COMMITTED" +assert_file_contains "$FAKE_SVN_LOG" "info --non-interactive --no-auth-cache" +assert_file_contains "$FAKE_SVN_LOG" "$DEV_SVN_DIR" +assert_file_contains "$FAKE_SVN_LOG" "commit" + +: > "$FAKE_SVN_LOG" +export FAKE_EXISTING_URL="$DEV_SVN_DIR" +if (stage_and_commit_version_dir "$DEV_SVN_BASE" "$DEV_SVN_DIR" "dev-svn" "test commit" \ + "${SOURCE_ARTIFACTS[@]}" <<< $'y\ny') >/dev/null 2>&1; then + fail "SVN upload accepted an existing version directory" +fi +assert_file_not_contains "$FAKE_SVN_LOG" "checkout" +assert_file_not_contains "$FAKE_SVN_LOG" "commit" + +PATH="$real_path" +pass diff --git a/tools/release-tools/tests/test-config.sh b/tools/release-tools/tests/test-config.sh new file mode 100755 index 00000000..cf00b17f --- /dev/null +++ b/tools/release-tools/tests/test-config.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -euo pipefail +# shellcheck source=testlib.sh +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/testlib.sh" +# shellcheck source=../release.env +source "${TOOLS_ROOT}/release.env" +# shellcheck source=../lib/release-common.sh +source "${TOOLS_ROOT}/lib/release-common.sh" + +assert_eq "26.0.0" "$VERSION" +assert_eq "$VERSION" "$TAG" +assert_eq "upstream-apache" "$GIT_REMOTE" +assert_eq "apache-doris-operator-26.0.0-src" "$PKG_BASE" +assert_eq "${PKG_BASE}/" "$ARCHIVE_PREFIX" +assert_eq "https://dist.apache.org/repos/dist/dev/doris/doris-operator/26.0.0" "$DEV_SVN_DIR" +assert_eq "https://dist.apache.org/repos/dist/release/doris/doris-operator/26.0.0" "$RELEASE_SVN_DIR" +assert_eq "https://downloads.apache.org/doris/KEYS" "$KEYS_URL" +assert_eq "https://dist.apache.org/repos/dist/dev/doris" "$DEV_KEYS_SVN_BASE" +assert_eq "https://dist.apache.org/repos/dist/release/doris" "$RELEASE_KEYS_SVN_BASE" + +validate_release_config + +error_file="$(mktemp)" +trap 'rm -f "$error_file"' EXIT +if (TAG="not-${VERSION}"; validate_release_config) 2>"$error_file"; then + fail "configuration validation accepted a tag that differs from VERSION" +fi +assert_file_contains "$error_file" "TAG must equal VERSION" + +if grep -Eq '^[[:space:]]*ASF_(USERNAME|PASSWORD)=' "${TOOLS_ROOT}/release.env"; then + fail "release.env stores SVN credentials" +fi + +pass diff --git a/tools/release-tools/tests/test-mail.sh b/tools/release-tools/tests/test-mail.sh new file mode 100755 index 00000000..366828ec --- /dev/null +++ b/tools/release-tools/tests/test-mail.sh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -euo pipefail +# shellcheck source=testlib.sh +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/testlib.sh" + +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT +tool_copy="${tmp}/release-tools" +cp -R "$TOOLS_ROOT" "$tool_copy" +mkdir -p "${tmp}/repo" "${tmp}/fake-bin" + +cat > "${tool_copy}/release.env" < "$COMMAND_LOG" +cat > "${tmp}/fake-bin/gpg" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +printf 'gpg %s\n' "$*" >> "$COMMAND_LOG" +if [[ " $* " == *" --with-colons "* ]]; then + printf 'sec:-:4096:1:TESTKEY:0:0:::::::scESC:::+:::23:\n' + printf 'fpr:::::::::0123456789ABCDEF0123456789ABCDEF01234567:\n' +fi +exit 0 +EOF +chmod +x "${tmp}/fake-bin/gpg" + +PATH="${tmp}/fake-bin:${PATH}" "${tool_copy}/03-vote-mail.sh" > "${tmp}/vote-output" +vote_body="${tmp}/work/vote-email.txt" +vote_eml="${tmp}/work/vote-email.eml" +assert_exists "$vote_body" +assert_exists "$vote_eml" +assert_file_contains "$vote_eml" "Subject: [VOTE] Release Apache Doris Operator 9.9.9" +assert_file_contains "$vote_body" "https://github.example.test/apache/doris-operator/releases/tag/9.9.9" +assert_file_contains "$vote_body" "https://dist.example.test/dev/doris/doris-operator/9.9.9/" +assert_file_contains "$vote_body" "0123456789ABCDEF0123456789ABCDEF01234567" +assert_file_contains "$vote_body" "release-manager@apache.org" +assert_file_contains "$vote_body" "The vote will remain open for at least 72 hours." +assert_file_contains "$vote_body" "[ ] +1 Approve the release" +assert_file_contains "${tmp}/vote-output" "Subject: [VOTE] Release Apache Doris Operator 9.9.9" +assert_file_contains "${tmp}/vote-output" "No email was sent." + +: > "$COMMAND_LOG" +PATH="${tmp}/fake-bin:${PATH}" "${tool_copy}/04-release-complete.sh" --mail-only > "${tmp}/announce-output" +announce_body="${tmp}/work/announce-email.txt" +announce_eml="${tmp}/work/announce-email.eml" +assert_exists "$announce_body" +assert_exists "$announce_eml" +assert_file_contains "$announce_eml" "To: announce@example.test" +assert_file_contains "$announce_eml" "Subject: [ANNOUNCE] Apache Doris Operator 9.9.9 release" +assert_file_contains "$announce_body" "automates the deployment and management" +assert_file_contains "$announce_body" "https://dist.example.test/release/doris/doris-operator/9.9.9/apache-doris-operator-9.9.9-src.tar.gz" +assert_file_contains "$announce_body" "Thank you to everyone" +assert_file_contains "${tmp}/announce-output" "mail-only mode: skipping tag, package, signing, and SVN operations" +[[ ! -s "$COMMAND_LOG" ]] || fail "--mail-only invoked an external release command" + +if PATH="${tmp}/fake-bin:${PATH}" "${tool_copy}/04-release-complete.sh" --unknown >/dev/null 2>&1; then + fail "04-release-complete.sh accepted an unknown argument" +fi + +pass diff --git a/tools/release-tools/tests/test-syntax.sh b/tools/release-tools/tests/test-syntax.sh new file mode 100755 index 00000000..34de8c15 --- /dev/null +++ b/tools/release-tools/tests/test-syntax.sh @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -euo pipefail +# shellcheck source=testlib.sh +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/testlib.sh" + +for script in "${TOOLS_ROOT}"/0*.sh "${TOOLS_ROOT}"/tests/run.sh "${TOOLS_ROOT}"/tests/test-*.sh; do + [[ -x "$script" ]] || fail "script is not executable: $script" +done + +bash -n \ + "${TOOLS_ROOT}"/release.env \ + "${TOOLS_ROOT}"/0*.sh \ + "${TOOLS_ROOT}"/lib/*.sh \ + "${TOOLS_ROOT}"/tests/*.sh + +readme="${TOOLS_ROOT}/README.md" +for text in \ + './01-check-env.sh' \ + './02-package-sign-upload.sh' \ + './03-vote-mail.sh' \ + './04-release-complete.sh' \ + '--mail-only' \ + 'refuses to overwrite' \ + 'does not inspect, compare, promote, move, or delete anything under dev SVN' \ + 'prints the subject and body' \ + 'independently from dev SVN' \ + 'freshly packages the selected Git tag' \ + 'No email was sent' \ + './tests/run.sh'; do + assert_file_contains "$readme" "$text" +done + +assert_file_not_contains "$readme" 'initial `26.0.0` dev directory' + +pass diff --git a/tools/release-tools/tests/test-workflows.sh b/tools/release-tools/tests/test-workflows.sh new file mode 100755 index 00000000..34318f53 --- /dev/null +++ b/tools/release-tools/tests/test-workflows.sh @@ -0,0 +1,163 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -euo pipefail +# shellcheck source=testlib.sh +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/testlib.sh" + +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT +tool_copy="${tmp}/release-tools" +cp -R "$TOOLS_ROOT" "$tool_copy" +repo="${tmp}/repo" +remote="${tmp}/remote.git" +fake_bin="${tmp}/fake-bin" +mkdir -p "$fake_bin" + +git init -q "$repo" +git -C "$repo" config user.name "Release Test" +git -C "$repo" config user.email "release-test@example.com" +printf 'source\n' > "${repo}/source.txt" +git -C "$repo" add source.txt +git -C "$repo" commit -q -m initial +git -C "$repo" tag 9.9.9 +git init -q --bare "$remote" +git -C "$repo" push -q "$remote" refs/tags/9.9.9 + +cat > "${tool_copy}/release.env" < "$FAKE_GPG_LOG" +: > "$FAKE_SVN_LOG" + +cat > "${fake_bin}/gpg" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +printf '%s\n' "$*" >> "$FAKE_GPG_LOG" +if [[ " $* " == *" --with-colons "* ]]; then + printf 'sec:-:4096:1:TESTKEY:0:0:::::::scESC:::+:::23:\n' + printf 'fpr:::::::::0123456789ABCDEF0123456789ABCDEF01234567:\n' + exit 0 +fi +if [[ " $* " == *" --list-secret-keys "* ]]; then + exit 0 +fi +output="" +while [[ "$#" -gt 0 ]]; do + case "$1" in + --output) output="$2"; shift 2 ;; + --verify) [[ -f "$2" && -f "$3" ]]; exit 0 ;; + *) shift ;; + esac +done +[[ -n "$output" ]] || exit 1 +printf 'fake signature\n' > "$output" +EOF +chmod +x "${fake_bin}/gpg" + +cat > "${fake_bin}/svn" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +printf '%s\n' "$*" >> "$FAKE_SVN_LOG" +command="$1" +shift +case "$command" in + info) + url="${@: -1}" + [[ -n "${FAKE_EXISTING_URL:-}" && "$url" == "$FAKE_EXISTING_URL" ]] + ;; + checkout) + destination="${@: -1}" + mkdir -p "$destination" + ;; + add|status|commit) exit 0 ;; + *) exit 1 ;; +esac +EOF +chmod +x "${fake_bin}/svn" + +test_path="${fake_bin}:${PATH}" +unset FAKE_EXISTING_URL || true +if ! printf 'y\ny\n' | PATH="$test_path" "${tool_copy}/02-package-sign-upload.sh" > "${tmp}/dev-output" 2>&1; then + cat "${tmp}/dev-output" >&2 + fail "candidate workflow failed" +fi + +source_archive="${tmp}/work/apache-doris-operator-9.9.9-src.tar.gz" +assert_exists "$source_archive" +assert_exists "${source_archive}.asc" +assert_exists "${source_archive}.sha512" +assert_file_contains "$FAKE_SVN_LOG" "https://dist.example.test/dev/doris/doris-operator/9.9.9" +staged_dev="${tmp}/work/dev-svn/9.9.9" +assert_exists "${staged_dev}/apache-doris-operator-9.9.9-src.tar.gz" +assert_exists "${staged_dev}/apache-doris-operator-9.9.9-src.tar.gz.asc" +assert_exists "${staged_dev}/apache-doris-operator-9.9.9-src.tar.gz.sha512" +staged_dev_count="$(find "$staged_dev" -maxdepth 1 -type f | wc -l | tr -d ' ')" +assert_eq "3" "$staged_dev_count" + +: > "$FAKE_SVN_LOG" +: > "$FAKE_GPG_LOG" +if ! printf 'y\ny\n' | PATH="$test_path" "${tool_copy}/04-release-complete.sh" > "${tmp}/release-output" 2>&1; then + cat "${tmp}/release-output" >&2 + fail "formal release workflow failed" +fi +assert_file_contains "$FAKE_SVN_LOG" "https://dist.example.test/release/doris/doris-operator/9.9.9" +assert_file_not_contains "$FAKE_SVN_LOG" "https://dist.example.test/dev/doris/doris-operator" +assert_file_contains "$FAKE_GPG_LOG" "apache-doris-operator-9.9.9-src.tar.gz" +staged_release="${tmp}/work/release-svn/9.9.9" +staged_release_count="$(find "$staged_release" -maxdepth 1 -type f | wc -l | tr -d ' ')" +assert_eq "3" "$staged_release_count" +assert_exists "${tmp}/work/announce-email.txt" +assert_exists "${tmp}/work/announce-email.eml" + +: > "$FAKE_SVN_LOG" +export FAKE_EXISTING_URL="https://dist.example.test/dev/doris/doris-operator/9.9.9" +if printf 'y\ny\n' | PATH="$test_path" "${tool_copy}/02-package-sign-upload.sh" >/dev/null 2>&1; then + fail "candidate workflow overwrote an existing SVN version directory" +fi +assert_file_not_contains "$FAKE_SVN_LOG" "checkout" +assert_file_not_contains "$FAKE_SVN_LOG" "commit" + +pass diff --git a/tools/release-tools/tests/testlib.sh b/tools/release-tools/tests/testlib.sh new file mode 100644 index 00000000..74266a30 --- /dev/null +++ b/tools/release-tools/tests/testlib.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -euo pipefail + +TESTS_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +TOOLS_ROOT="$(cd "${TESTS_DIR}/.." && pwd)" + +fail() { + printf 'FAIL: %s\n' "$*" >&2 + exit 1 +} + +assert_eq() { + [[ "$1" == "$2" ]] || fail "expected '$1', got '$2'" +} + +assert_file_contains() { + grep -Fq -- "$2" "$1" || fail "$1 does not contain: $2" +} + +assert_file_not_contains() { + if grep -Fq -- "$2" "$1"; then + fail "$1 unexpectedly contains: $2" + fi +} + +assert_exists() { + [[ -e "$1" ]] || fail "expected path to exist: $1" +} + +assert_not_exists() { + [[ ! -e "$1" ]] || fail "expected path not to exist: $1" +} + +pass() { + printf 'PASS: %s\n' "${0##*/}" +}