Cherry pick from main to REL_2_STABLE since 0331 for 2.2 - #110
Open
tuhaihe wants to merge 39 commits into
Open
Conversation
Integrate gpbackman (https://github.com/woblerr/gpbackman) — a CLI utility for managing gpbackup backups — into the cloudberry-backup repo. YAML functionality (history-migrate, converters, utils_file) is not ported. Key changes: - Add gpbackman.go entry point with //go:build gpbackman tag - Create gpbackman/cmd/, gpbackman/gpbckpconfig/, gpbackman/textmsg/ - Migrate imports: greenplum-db/* → apache/cloudberry-* - Remove converter layer — use history.BackupConfig directly - Extract 11 BackupConfig methods into standalone functions (helpers.go) - Replace go-pretty/v6 with olekukonko/tablewriter - Add gpbackman to all Makefile targets (build/install/clean/package) - Version via ldflags: -X gpbackman/cmd.version - Rewrite unit tests from stdlib testing to Ginkgo/Gomega CLI subcommands: backup-info, backup-delete, backup-clean, history-clean, report-info.
- Replace searchFilter() with utils.Exists() in gpbckpconfig/helpers.go - Remove searchFilter() from gpbckpconfig/utils.go and its test - Replace getCurrentTimestamp() with history.CurrentTimestamp() in cmd/backup_delete.go - Remove getCurrentTimestamp() from cmd/wrappers.go and its test
Also add vscode to gitignore.
For RC tags like X.Y.Z-incubating-rcN, generate the source tarball filename and top-level directory using VERSION_FILE (without -rcN). $VERSION_FILE is input from the file VERSION. This keeps the voted bits ready for promotion without rebuilding and avoids -rcN showing up in the extracted source directory.
Add LICENSE, NOTICE, and DISCLAIMER files to the package target in Makefile to ensure Apache project compliance requirements are met when distributing binary releases. These files are now copied to the package directory alongside the binaries and install script, and will be included in the final tar.gz distribution.
Update Go to 1.24.
Fix non-constant format string errors for Go 1.24.
Fix patterns applied:
- func(err.Error()) -> func("%s", err.Error())
- func(fmt.Sprintf("...", args)) -> func("...", args)
Update golangci-lint from 1.16.0 to 2.10.1 in Makefile. Create .golangci.yml with v2 configuration format. Remove obsolete gometalinter.config. Linter mapping from gometalinter.config: - golint -> revive (with shadow check enabled) - vet -> govet - varcheck -> unused - unparam and errcheck remain unchanged TODO: consider enabling additional linters in a follow-up PR: dupl, gochecknoinits, gocritic, gocyclo, gosec, ineffassign, misspell, nakedret, prealloc, staticcheck, unconvert. Current goal is to migrate the existing config as-is.
Changes: - Fix variable shadowing and declaring variables before use (backup_helper.go, restore_helper.go, history.go, s3plugin.go, data.go). - Fix in testutils/functions.go where := was shadowing the host parameter, preventing PGHOST env var from being applied. - Remove unused test helper functions batchMapToString and contentMapToString from restore/data_test.go. - Add lint exclusion rules in .golangci.yml for test files: govet shadow, errcheck, and unparam — these are pre-existing issues in tests, not regressions from the v2 migration. Pre-existing issues are not fixed here and will be addressed separately.
The legacy `// +build` form is no longer needed.
The io/ioutil package has been deprecated since Go 1.16. All usages are replaced with their recommended alternatives from os and io packages. Changes: - ioutil.ReadFile -> os.ReadFile - ioutil.WriteFile -> os.WriteFile - ioutil.TempFile -> os.CreateTemp - ioutil.TempDir -> os.MkdirTemp - ioutil.ReadAll -> io.ReadAll - ioutil.ReadDir -> os.ReadDir The os.ReadDir returns []os.DirEntry instead of []os.FileInfo, but all call sites only use .Name(), which is available on both interfaces.
Add smoke tests to verify all cloudberry-backup commands (gpbackup, gprestore, gpbackup_helper, gpbackup_s3_plugin, gpbackman) can be built and executed with --version and --help flags. - build_and_unit_test.yml: Test from GOPATH/bin after build - cloudberry-backup-ci.yml: Test from GPHOME/bin after install This provides fast feedback on basic functionality before running more expensive integration tests.
) Update the installation instructions in README.md to use the specific version tag @2.1.0-incubating instead of @latest. The @latest tag resolves to v1.0.2 due to Go modules semantic versioning rules, which only recognizes version tags with 'v' prefix (v1.0.0, v1.0.1, v1.0.2). The 2.1.0-incubating tag, while functional when specified directly, is not recognized as a semantic version by Go modules. Added a note to clarify why users should use the specific version tag to avoid confusion and ensure they install the correct latest version.
Use base version (without -rcN suffix) for release tarball filename
to align with Apache incubator release conventions.
Changes:
- Modified TAR_NAME to use ${VERSION_FILE} instead of ${TAG}
- Tarball filename now follows pattern:
apache-cloudberry-backup-${VERSION_FILE}-incubating-src.tar.gz
- Example: apache-cloudberry-backup-2.0.0-incubating-src.tar.gz
(instead of apache-cloudberry-backup-2.0.0-incubating-rc1-src.tar.gz)
Benefits:
- Enables direct 'svn mv' to release repository after voting without
renaming artifacts
- Aligns with Apache release best practices where RC identifiers are
used only for Git tags and voting process, not in final artifact names
- Maintains consistency between tarball filename and extracted
directory name
The extracted directory name remains unchanged:
apache-cloudberry-backup-${VERSION_FILE}-incubating/
When restoring with --resize-cluster from a larger to smaller segment count, gprestore expands the oid list into (oid, batch) pairs in oid-major order ([T1B0, T1B1, T2B0, T2B1, ...]) and preloads min(NumConns, len(oidList)) segment pipes before starting the helpers. Coordinator workers take one task per table and iterate batches sequentially inside restoreSingleTableData, so the k-th worker's first COPY targets oidList index k*batches. When batches >= 2 and --jobs > 1 (NumConns > 1), workers beyond NumConns/batches race ahead of helper pipe creation and fail with: ERROR: command error message: cat: .../<pipe>_<oid>_0: No such file or directory and the restore only succeeds with --jobs 1. This matches the known upstream gpbackup issue fixed in 1.30.6. Preload NumConns*batches pipes (clamped to len(oidList)) so every concurrent worker's first batch is covered. The helper then rolls its queue forward one pipe per completed batch, and since a worker must finish all batches of its current table before pulling the next task, helper progress stays ahead of worker demand. For batches == 1 the behavior is unchanged. Fixes apache#92
Use a top-level `unnest(CASE ...) AS privileges` instead of `LEFT JOIN LATERAL`. This removes an extra join and per-array expansion, reducing planner/executor work when scanning large catalogs. Related to apache#88
…d when missing. Two improvements to how gpbackman locates and opens gpbackup_history.db: 1. When --history-db is empty, fall back (in priority order) to $COORDINATOR_DATA_DIRECTORY and $MASTER_DATA_DIRECTORY before the current directory. These variables are exported by the standard Cloudberry/Greenplum environment scripts, so users running gpbackman from a sourced cluster shell no longer need to repeat --history-db on every invocation. 2. OpenHistoryDB now pre-checks the file with os.Stat and opens the SQLite database with the rw URI mode. A missing file produces a friendly error pointing at --history-db and the env vars, instead of silently creating an empty file that later fails with "no such table: backups" when callers issue queries. The previous cwd-only fallback is preserved as a last resort. Tests cover the new env-var resolution order and the safe-open behaviour. Signed-off-by: chenqiang <chenqiang@hashdata.cn>
…ADME.md. Replace the "current directory" wording everywhere it appeared with the new resolution order: $COORDINATOR_DATA_DIRECTORY, $MASTER_DATA_DIRECTORY, then the current directory. Signed-off-by: chenqiang <chenqiang@hashdata.cn>
…y-db opt-in. Address review feedback on the original PR: * The env-var fallback ($COORDINATOR_DATA_DIRECTORY then $MASTER_DATA_DIRECTORY) is now opt-in via a new persistent flag --auto-load-history-db. The default behaviour (current-directory lookup) is unchanged from upstream main, so destructive subcommands (backup-delete, backup-clean, history-clean) cannot silently target the wrong cluster's history DB on a multi-cluster jump host. * Drop the "Greenplum" mention in the comment for getHistoryDBPath to keep wording aligned with the Cloudberry environment scripts. * Trim the verbose --history-db description in gpbackman/README.md and add a separate --auto-load-history-db line. * Refine the not-found error message in OpenHistoryDB to mention the new flag rather than implying that setting env vars alone helps. * Tests updated to cover the opt-in matrix: env vars are ignored when the flag is off, honoured when on, and an explicit --history-db still wins regardless. The OpenHistoryDB safe-open change (os.Stat pre-check + mode=rw URI to avoid silently creating an empty SQLite file) is preserved as-is. Signed-off-by: chenqiang <chenqiang@hashdata.cn>
…istoryDB. Previously a permission-denied or I/O failure on the history DB path returned the bare os.Stat error, which gave no hint about which operation or path failed. Wrap it with fmt.Errorf using %w so errors.Is/As still work while surfacing "stat history db <path>" in the message, matching the friendly tone of the ErrNotExist branch.
Cloudberry switched from MASTER_DATA_DIRECTORY to COORDINATOR_DATA_DIRECTORY
starting around release 1.5.4, and the standard environment scripts no
longer export the legacy variable. Maintaining a fallback for it in
gpbackman just adds surface area (constants, help text, docs, tests) for
a value that should not exist in a current Cloudberry install.
Drop MASTER_DATA_DIRECTORY entirely from the resolution chain, the flag
help, the per-command long help strings, README/COMMANDS docs, and the
corresponding test cases. The flag description is shortened in root.go
to match the concise wording already used in README. The lingering
"Cloudberry/Greenplum" comment in constants.go is updated to "Cloudberry"
in the same pass.
Resolution chain after this change:
explicit --history-db
-> $COORDINATOR_DATA_DIRECTORY (only when --auto-load-history-db is set)
-> current working directory (default)
Addresses review feedback from woblerr, tuhaihe, and MisterRaindrop on
PR apache#97. The pre-existing MASTER_DATA_DIRECTORY reference in
.github/workflows/cloudberry-backup-ci.yml predates this PR and is
unrelated to history-db resolution; it is left for a separate cleanup.
…tabase. (apache#87) * Port gpbackup_exporter to cloudberry-backup. Introduce the Prometheus metrics exporter as a new standalone binary within the repository. - Create gpbackup_exporter.go entry point with CLI flags and collection loop. - Port core exporter logic to the new `exporter/` package. - Adapt type system to use `*history.BackupConfig` directly, aligning with the cloudberry-backup architecture. - Switch receiver methods to standalone gpbckpconfig helpers. - Add Prometheus and Kingpin dependencies to `go.mod`. - Update Makefile to support building, testing, and packaging. - Port unit tests to Ginkgo/Gomega. * Add end-to-end tests for gpbackup_exporter. Add e2e test that runs gpbackup commands and validates that gpbackup_exporter correctly reads history database and exposes metrics. Also remove dead gpbackmanPath assignment from useOldBackupVersion in e2e tests. And suppress noisy test logger output by writing to bytes.Buffer instead of os.Stdout in exporter unit tests. * Add README for gpbackup_exporter. * Add additional tools section to README. * Add smoke tests for gpbackup_exporter in CI. * Fix smoke test for gpbackup_exporter in install target. * Graceful shutdown with exit status 0. SIGINT means "interrupt," signaling a user wants to stop the current operation. SIGTERM means "terminate," requesting a polite program shutdown. So' it's correct to consider as graceful shutdown * Fix port in test. * Prevent panic and make exit. To prevent a panic with an error like "panic: http: invalid pattern", exit in case of an empty endpoint. * Remove unused getDataSuccessStatus and simplify GetGPBackupInfo. Remove unused getDataSuccessStatus. Clarify include/exclude handling: * DB present in both dbInclude and dbExclude - warn and emit gpbackup_exporter_status=0. * DB only in dbExclude - skip, emit no metrics. * dbInclude empty or DB in dbInclude - process and emit full backup metrics. Add unit tests. * Ensure command starts successfully in gpbackup_exporter end-to-end tests. * Update gpbackup_exporter_status metric description. * Fix expected error message in exporter test for missing history DB. This aligns the test with the recent behavior change in commit 1035fb8. `OpenHistoryDB()` now fails fast on missing files during the open stage, rather than failing later during the first SQL query.
Extract install script generation from Makefile to a separate template file for better maintainability and readability. - Add install.sh.template with Apache License header - Simplify Makefile package target using sed for template substitution
Switch the CI Cloudberry checkout from `main` to `REL_2_STABLE branch`. Cloudberry main has upgraded to the PG16 kernel, whose tightened role grant permission checks (GRANT ... GRANTED BY now requires ADMIN OPTION) break multiple backup integration tests. `REL_2_STABLE` still runs PG14 and passes the full test suite without these failures. Add a TODO to revert to main once Cloudberry 3.x stabilizes with PG16. See: apache#103
In our 2.1.0-rc2 review round, Jean Baptiste Onofre suggested we can include the original gpbackup NOTICE file content to the NOTICE. I also check the ASF docs, and find that we should do as the suggestion. You can see more details here: - https://lists.apache.org/thread/s3r9zmkt90cjnx8921mvmojw2zdfgh8k - https://infra.apache.org/licensing-howto.html#mod-notice
* Bump Go from 1.24 to 1.25 * Update go version in README.md * Upgrade go-libs to the latest * Fix unit end_to_end test * Fix deadlock e2e test transaction handling. The test held locks with raw BEGIN/COMMIT SQL on DBConn and then reused the same connection for polling. With pgx v5 this can release the raw transaction lock, so gpbackup was not blocked and the test observed 0 deadlock traps. Use explicit DBConn transactions, separate polling connections, wait for goroutines before closing connections, and wait for the expected 2 blocked AccessExclusiveLock requests. * Fix pgx v5 PgError handling. After moving to pgx v5, database errors are returned as pgx/v5/pgconn.PgError. The old jackc/pgconn type assertions no longer matched, so backup lock handling could treat unexpected errors as lock-not-available, and restore COPY errors could lose their CONTEXT. Use errors.As with pgx/v5 pgconn in backup and restore code, update the restore test, and remove the old pgconn dependency. * Fix lock e2e test transaction handling. The deadlock and signal handler e2e tests still held locks with raw BEGIN/COMMIT SQL on DBConn and reused those connections for pg_locks polling. With pgx v5 this can release the raw transaction lock or leave goroutines polling on closed connections, causing missing blocked-lock counts and DBConn.Get panics. Use explicit transactions, dedicated lock and polling connections, and synchronize goroutines before closing test connections. * Fix gp_segment_configuration lock e2e test. The test intended to verify that gpbackup does not keep a lock on pg_catalog.gp_segment_configuration after passing that metadata query and blocking later on pg_catalog.pg_trigger. However, the old test did not actually exercise that state: it used raw BEGIN/COMMIT SQL through DBConn, committed before gpbackup was started, queried pg_locks in an unsynchronized goroutine, and ignored query errors while scanning SELECT * into an int. Use an explicit DBConn transaction on a dedicated lock connection, start gpbackup while the pg_trigger lock is still held, wait until gpbackup is visibly blocked on pg_trigger, and then synchronously assert that no gp_segment_configuration locks remain. Finally release pg_trigger and verify gpbackup completes successfully. --------- Co-authored-by: woblerr <antkurochkin@gmail.com>
Bump golang.org/x/crypto to v0.52.0. Bump golang.org/x/net to v0.55.0. Bump other dependencies.
) Introduce a manually-triggered GitHub Actions workflow that builds portable tar.gz packages for Apache cloudberry-backup from its official source release tarball, and tests them against Apache Cloudberry built from its own official source release. Key design decisions: - cloudberry-backup binaries are built on Rocky 8 (glibc 2.28) to maximise compatibility across Linux distributions. - Cloudberry is built per-platform from its source release tarball (~8 min), avoiding dependence on pre-built Docker images or external DEB/RPM repositories. - Build-and-test are fused into one job per platform: after Cloudberry compiles, gpdemo spins up directly in the same container, eliminating artifact packaging, transfer, and a separate test job. - Only .sha512 checksums are generated; .asc signatures remain a release manager local step.
tuhaihe
changed the title
tuhaihe
marked this pull request as ready for review
July 29, 2026 12:20
Member
Author
|
Now we're waiting for more commits to be cherry-picked from main to REL_2_STABLE for the cloudberry core. Once done, we can re-run the tests and merge this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes: #ISSUE_Number
Change logs
Contributor's checklist
Here are some reminders before you submit your pull request: