Fix PEP-691 fingerprinting against authenticated indexes.#3225
Open
jasonwbarnett wants to merge 2 commits into
Open
Fix PEP-691 fingerprinting against authenticated indexes.#3225jasonwbarnett wants to merge 2 commits into
jasonwbarnett wants to merge 2 commits into
Conversation
Previously, `pex3 lock create` against an index requiring basic auth that serves the PEP-691 JSON simple API (e.g.: AWS CodeArtifact) degraded to downloading every artifact of every locked project to fingerprint it, building sdists along the way, because every request made by the PEP-691 `FingerprintService` failed: 1. The endpoints scraped from the Pip log carry Pip-redacted credentials (`https://user:****@...`), which the stdlib openers cannot even parse as a netloc. 2. The stdlib basic and digest auth handlers raise `ValueError` for `WWW-Authenticate` challenge schemes they do not implement (AWS CodeArtifact challenges with `Bearer`); so known credentials were never retried. Now the scraped endpoints are stripped of redacted credentials, known machine-scoped credentials are sent preemptively via a basic auth header (also covering credentials embedded directly in a URL, which previously failed netloc parsing), and unsupported challenge schemes degrade to their underlying 401 HTTPError instead of an opaque crash. Fixes pex-tool#3224 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ftLY926gkJA6yMbu813cG
jasonwbarnett
force-pushed
the
pep-691-endpoint-auth
branch
from
July 21, 2026 18:03
f2b39e7 to
61816f6
Compare
jasonwbarnett
marked this pull request as ready for review
July 21, 2026 18:05
Amp-Thread-ID: https://ampcode.com/threads/T-019f8594-9574-75e5-aa2a-0cdcc69c2f0a Co-authored-by: Amp <amp@ampcode.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3224.
Root cause
When Pip consumes an authenticated PEP-691 JSON Simple API index, artifact hashes remain available internally in each JSON file entry's
hashesmapping, but they are absent from the artifact URLs written to Pip's log. Pex therefore uses itsFingerprintServiceto re-fetch the logged PEP-691 project endpoints and recover those hashes.That recovery failed against AWS CodeArtifact for two independent reasons:
https://user:****@...).Lockerretained that literal URL, which the stdlib opener could not parse (nonnumeric port: '****@...').URLFetcherwaited for an authentication challenge. CodeArtifact accepts Basic authentication but responds to an unauthenticated request with401and a Bearer challenge; the stdlib Basic/Digest handlers raiseValueErrorfor that unsupported scheme instead of retrying with the known Basic credential.FingerprintServicedeliberately treats metadata-fetch failures as non-fatal and emits a warning. Callers such as Pants invoke Pex with warnings disabled, making this operationally silent. Artifacts left without hashes then enter Pex's fallback path: each retained artifact for each selected locked version is downloaded withpip download --no-deps <artifact-url>and hashed locally. For universal locks this includes the selected artifact and compatible additional artifacts. Besides the large performance regression, directly downloading retained sdists can execute PEP-517 metadata hooks and fail on host build requirements.Fix
This PR makes three coordinated changes:
Lockerstrips Pip-redacted credentials from PEP-691 endpoint URLs before recording them.URLFetcher.get_body_streamstrips credentials embedded in request URLs and sends them via a preemptive Basic authorization header. For clean URLs it looks up matching machine-scoped credentials in itsPasswordDatabaseand sends those preemptively. Machine-lessdefaultnetrc entries are deliberately not sent to arbitrary hosts. When several entries match the same machine, later entries win so explicitly appended repository credentials override earlier netrc credentials.WWW-Authenticateschemes. If no applicable preemptive credential exists, an unsupported challenge now surfaces the underlying 401HTTPErrorrather than an opaqueValueError.Validation
Validated against the AWS CodeArtifact repository that surfaced #3224:
mainat bea9c2e),pex3 lock create --pip-version 24.2 --style universal --no-pypi --index https://aws:<token>@<host>/pypi/<repo>/simple psycopg2-binary==2.9.12exits 1 with apg_configsdist metadata failure after downloading the other 11 retained release artifacts to fingerprint them;#sha256=fragments, including the sdist hash advertised byhashes.sha256;ddtrace.Regression coverage includes:
Fetched pagelog line throughLocker, authenticates the resulting clean endpoint, parses its PEP-691 JSON, and recovers the advertised artifact SHA-256 without an unauthenticated request.Local validation completed successfully:
formatlinttypecheck🤖 Generated with Claude Code and Amp