Summary
Running many concurrent CI jobs against a single shared S3 cache prefix, a build occasionally compiles a package from stale source at a commit where that package's source has changed — the cache returns an object whose inputs no longer match. It surfaces as a coverage profile containing block ranges from two different revisions of the same file, and the affected job shows a ~99.9% cache-hit rate, so the stale bytes came from the cache, not a local rebuild. A manual purge of the prefix clears it; it recurs within a pipeline or two.
The failing job is only where it's detected. The same path would serve a stale object to a normal build silently, producing a binary from old source with no error — so we treat this as a correctness issue.
Questions after looking at code:
- The plugin keys purely on the actionID that
cmd/go provides. For a cache shared across toolchains and build variants, should the key be salted with additional build context (toolchain version, cover mode, flags) — or is fully distinguishing builds considered cmd/go's responsibility, and this is really a cmd/go actionID-completeness issue rather than a plugin one?
Get returns the cached object without verifying its content against the requested outputID (the comment defers content-address checking to the toolchain). Would you accept a verify-on-GET (or read-only) mode that treats a hash mismatch as a miss and rebuilds, rather than returning stale bytes? For a shared multi-writer cache that would convert a silent stale-serve into a safe miss.
I can do the fix, if issue is accepted.
Summary
Running many concurrent CI jobs against a single shared S3 cache prefix, a build occasionally compiles a package from stale source at a commit where that package's source has changed — the cache returns an object whose inputs no longer match. It surfaces as a coverage profile containing block ranges from two different revisions of the same file, and the affected job shows a ~99.9% cache-hit rate, so the stale bytes came from the cache, not a local rebuild. A manual purge of the prefix clears it; it recurs within a pipeline or two.
The failing job is only where it's detected. The same path would serve a stale object to a normal build silently, producing a binary from old source with no error — so we treat this as a correctness issue.
Questions after looking at code:
cmd/goprovides. For a cache shared across toolchains and build variants, should the key be salted with additional build context (toolchain version, cover mode, flags) — or is fully distinguishing builds consideredcmd/go's responsibility, and this is really acmd/goactionID-completeness issue rather than a plugin one?Getreturns the cached object without verifying its content against the requestedoutputID(the comment defers content-address checking to the toolchain). Would you accept a verify-on-GET (or read-only) mode that treats a hash mismatch as a miss and rebuilds, rather than returning stale bytes? For a shared multi-writer cache that would convert a silent stale-serve into a safe miss.I can do the fix, if issue is accepted.