Skip to content

ci: cache the sccache directory across C++ test builds #765

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
abnobdoss wants to merge 1 commit into
base: main
Choose a base branch
from
+127 −6
Open

ci: cache the sccache directory across C++ test builds #765

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion .github/workflows/aws_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,8 @@ jobs:
AWS_DEFAULT_REGION: us-east-1
AWS_ENDPOINT_URL: http://127.0.0.1:9000
AWS_EC2_METADATA_DISABLED: "TRUE"
SCCACHE_DIR: ${{ github.workspace }}/.sccache
SCCACHE_CACHE_SIZE: "2G"
steps:
- name: Checkout iceberg-cpp
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
Expand Down Expand Up @@ -113,11 +115,29 @@ jobs:
if: ${{ matrix.s3 == 'ON' }}
shell: bash
run: bash ci/scripts/start_minio.sh
- name: Restore sccache cache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5

@zhjwpku zhjwpku Jun 20, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the Windows workflow was using mozilla-actions/sccache-action together with SCCACHE_GHA_ENABLED, while this PR changes it to use explicit actions/cache restore/save steps plus mozilla-actions/sccache-action.

I'm not sure which approach is better, but in case you missed it, mozilla-actions/sccache-action README documents the SCCACHE_GHA_ENABLED setup here:
https://github.com/mozilla-actions/sccache-action#cc-code

@abnobdoss abnobdoss Jun 20, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @zhjwpku, thank you for the review!

With SCCACHE_GHA_ENABLED, sccache uploads each compiled object as its own cache entry, which adds up to hundreds per build. On the larger Linux/macOS builds this hits GitHub's upload rate-limit throttle and surfaces as cache write errors. The build still passes when we hit the throttle, but the affected objects silently don't get cached. Pointing sccache at a local directory with a single actions/cache save/restore turns it into one archive per leg, so there's nothing to throttle. In my fork that brought write errors down to zero, and it was a bit faster overall since it uploads once instead of hundreds of small objects that each take a network round trip.

@zhjwpku zhjwpku Jun 20, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, in that case, you may want to make the same change to the Meson Windows build as well. I believe it was missed in this PR, unless I'm overlooking.

@abnobdoss abnobdoss Jun 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call out - I did leave it out of this PR. I was worried there might be too many changes in one PR for review as it already has a sizeable diff on the CI files. If you think it'd be fine here I can add it, or if you prefer I can raise it as a follow-up, wdyt?

with:
path: ${{ github.workspace }}/.sccache
key: sccache-aws-${{ matrix.runs-on }}-bundle${{ matrix.bundle_awssdk }}-s3${{ matrix.s3 }}-sigv4${{ matrix.sigv4 }}-${{ github.run_id }}
restore-keys: |
sccache-aws-${{ matrix.runs-on }}-bundle${{ matrix.bundle_awssdk }}-s3${{ matrix.s3 }}-sigv4${{ matrix.sigv4 }}-
- name: Setup sccache
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Build and test Iceberg
shell: bash
env:
CMAKE_TOOLCHAIN_FILE: ${{ startsWith(matrix.runs-on, 'ubuntu') && matrix.bundle_awssdk == 'OFF' && '/usr/local/share/vcpkg/scripts/buildsystems/vcpkg.cmake' || '' }}
run: ci/scripts/build_iceberg.sh "$(pwd)" OFF OFF ${{ matrix.s3 }} ${{ matrix.sigv4 }} ${{ matrix.bundle_awssdk }}
run: ci/scripts/build_iceberg.sh "$(pwd)" OFF ON ${{ matrix.s3 }} ${{ matrix.sigv4 }} ${{ matrix.bundle_awssdk }}
- name: Show sccache stats
shell: bash
run: sccache --show-stats
- name: Save sccache cache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-aws-${{ matrix.runs-on }}-bundle${{ matrix.bundle_awssdk }}-s3${{ matrix.s3 }}-sigv4${{ matrix.sigv4 }}-${{ github.run_id }}

# Exercise the Meson build with SigV4 enabled (resolves aws-cpp-sdk-core via
# its CMake config, not pkg-config whose Cflags force -std=c++11).
Expand Down
24 changes: 23 additions & 1 deletion .github/workflows/sanitizer_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ jobs:
if: ${{ github.event_name != 'pull_request' || github.event.pull_request.draft == false }}
name: "ASAN and UBSAN Tests"
runs-on: ubuntu-24.04
env:
SCCACHE_DIR: ${{ github.workspace }}/.sccache
SCCACHE_CACHE_SIZE: "2G"
steps:
- name: Checkout iceberg-cpp
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
Expand All @@ -47,14 +50,33 @@ jobs:
- name: Install dependencies
shell: bash
run: sudo apt-get update && sudo apt-get install -y libcurl4-openssl-dev
- name: Restore sccache cache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-sanitizer-ubuntu-${{ github.run_id }}
restore-keys: |
sccache-sanitizer-ubuntu-
- name: Setup sccache
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Configure and Build with ASAN & UBSAN
env:
CC: gcc-14
CXX: g++-14
run: |
mkdir build && cd build
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Debug -DICEBERG_ENABLE_ASAN=ON -DICEBERG_ENABLE_UBSAN=ON
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Debug -DICEBERG_ENABLE_ASAN=ON -DICEBERG_ENABLE_UBSAN=ON \
-DCMAKE_C_COMPILER_LAUNCHER=sccache -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
cmake --build . --verbose
- name: Show sccache stats
shell: bash
run: sccache --show-stats
- name: Save sccache cache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-sanitizer-ubuntu-${{ github.run_id }}
- name: Run Tests
working-directory: build
env:
Expand Down
23 changes: 23 additions & 0 deletions .github/workflows/sql_catalog_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@ jobs:
runs-on: windows-2025
cmake_build_type: Release
cmake_extra_args: -DCMAKE_TOOLCHAIN_FILE=C:/vcpkg/scripts/buildsystems/vcpkg.cmake
env:
SCCACHE_DIR: ${{ github.workspace }}/.sccache
SCCACHE_CACHE_SIZE: "2G"
steps:
- name: Checkout iceberg-cpp
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
Expand All @@ -85,6 +88,15 @@ jobs:
shell: pwsh
run: |
vcpkg install zlib:x64-windows nlohmann-json:x64-windows nanoarrow:x64-windows roaring:x64-windows sqlite3:x64-windows
- name: Restore sccache cache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-sqlcatalog-${{ matrix.runs-on }}-${{ github.run_id }}
restore-keys: |
sccache-sqlcatalog-${{ matrix.runs-on }}-
- name: Setup sccache
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Configure Iceberg
shell: bash
run: |
Expand All @@ -96,10 +108,21 @@ jobs:
-DICEBERG_BUILD_REST=OFF \
-DICEBERG_BUILD_SQL_CATALOG=ON \
-DICEBERG_SQL_SQLITE=ON \
-DCMAKE_C_COMPILER_LAUNCHER=sccache \
-DCMAKE_CXX_COMPILER_LAUNCHER=sccache \
${{ matrix.cmake_extra_args }}
- name: Build SQL catalog tests
shell: bash
run: cmake --build build --target sql_catalog_test
- name: Show sccache stats
shell: bash
run: sccache --show-stats
- name: Save sccache cache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-sqlcatalog-${{ matrix.runs-on }}-${{ github.run_id }}
- name: Run SQL catalog tests
shell: bash
run: ctest --test-dir build -R '^sql_catalog_test$' --output-on-failure -C ${{ matrix.cmake_build_type }}
64 changes: 60 additions & 4 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ jobs:
timeout-minutes: 30
strategy:
fail-fast: false
env:
SCCACHE_DIR: ${{ github.workspace }}/.sccache
SCCACHE_CACHE_SIZE: "2G"
steps:
- name: Checkout iceberg-cpp
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
Expand All @@ -53,12 +56,30 @@ jobs:
- name: Install dependencies
shell: bash
run: sudo apt-get update && sudo apt-get install -y libcurl4-openssl-dev
- name: Restore sccache cache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-test-ubuntu-${{ github.run_id }}
restore-keys: |
sccache-test-ubuntu-
- name: Setup sccache
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Build Iceberg
shell: bash
env:
CC: gcc-14
CXX: g++-14
run: ci/scripts/build_iceberg.sh $(pwd) ON
run: ci/scripts/build_iceberg.sh $(pwd) ON ON
- name: Show sccache stats
shell: bash
run: sccache --show-stats
- name: Save sccache cache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-test-ubuntu-${{ github.run_id }}
- name: Build Example
shell: bash
env:
Expand All @@ -72,14 +93,35 @@ jobs:
timeout-minutes: 30
strategy:
fail-fast: false
env:
SCCACHE_DIR: ${{ github.workspace }}/.sccache
SCCACHE_CACHE_SIZE: "2G"
steps:
- name: Checkout iceberg-cpp
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
persist-credentials: false
- name: Restore sccache cache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-test-macos-${{ github.run_id }}
restore-keys: |
sccache-test-macos-
- name: Setup sccache
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Build Iceberg
shell: bash
run: ci/scripts/build_iceberg.sh $(pwd)
run: ci/scripts/build_iceberg.sh $(pwd) OFF ON
- name: Show sccache stats
shell: bash
run: sccache --show-stats
- name: Save sccache cache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-test-macos-${{ github.run_id }}
- name: Build Example
shell: bash
run: ci/scripts/build_example.sh $(pwd)/example
Expand All @@ -90,6 +132,9 @@ jobs:
timeout-minutes: 60
strategy:
fail-fast: false
env:
SCCACHE_DIR: ${{ github.workspace }}/.sccache
SCCACHE_CACHE_SIZE: "2G"
steps:
- name: Checkout iceberg-cpp
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
Expand All @@ -103,17 +148,28 @@ jobs:
shell: pwsh
run: |
vcpkg install zlib:x64-windows nlohmann-json:x64-windows nanoarrow:x64-windows roaring:x64-windows cpr:x64-windows
- name: Restore sccache cache
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-test-windows-${{ github.run_id }}
restore-keys: |
sccache-test-windows-
- name: Setup sccache
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Build Iceberg
shell: pwsh
env:
SCCACHE_GHA_ENABLED: "true"
run: |
$ErrorActionPreference = "Stop"
bash -lc 'ci/scripts/build_iceberg.sh $(pwd) OFF ON'
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
sccache --show-stats
- name: Save sccache cache
if: github.ref == 'refs/heads/main'
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
key: sccache-test-windows-${{ github.run_id }}
- name: Build Example
shell: pwsh
run: |
Expand Down
Loading