Skip to content

Fix PEP-691 fingerprinting against authenticated indexes.#3225

Open
jasonwbarnett wants to merge 2 commits into
pex-tool:mainfrom
altana-ai:pep-691-endpoint-auth
Open

Fix PEP-691 fingerprinting against authenticated indexes.#3225
jasonwbarnett wants to merge 2 commits into
pex-tool:mainfrom
altana-ai:pep-691-endpoint-auth

Conversation

@jasonwbarnett

@jasonwbarnett jasonwbarnett commented Jul 21, 2026

Copy link
Copy Markdown

Fixes #3224.

Root cause

When Pip consumes an authenticated PEP-691 JSON Simple API index, artifact hashes remain available internally in each JSON file entry's hashes mapping, but they are absent from the artifact URLs written to Pip's log. Pex therefore uses its FingerprintService to re-fetch the logged PEP-691 project endpoints and recover those hashes.

That recovery failed against AWS CodeArtifact for two independent reasons:

  1. Pip redacts credentials in logged URLs (https://user:****@...). Locker retained that literal URL, which the stdlib opener could not parse (nonnumeric port: '****@...').
  2. Even after removing userinfo and supplying matching machine-scoped credentials, URLFetcher waited for an authentication challenge. CodeArtifact accepts Basic authentication but responds to an unauthenticated request with 401 and a Bearer challenge; the stdlib Basic/Digest handlers raise ValueError for that unsupported scheme instead of retrying with the known Basic credential.

FingerprintService deliberately treats metadata-fetch failures as non-fatal and emits a warning. Callers such as Pants invoke Pex with warnings disabled, making this operationally silent. Artifacts left without hashes then enter Pex's fallback path: each retained artifact for each selected locked version is downloaded with pip download --no-deps <artifact-url> and hashed locally. For universal locks this includes the selected artifact and compatible additional artifacts. Besides the large performance regression, directly downloading retained sdists can execute PEP-517 metadata hooks and fail on host build requirements.

Fix

This PR makes three coordinated changes:

  1. Locker strips Pip-redacted credentials from PEP-691 endpoint URLs before recording them.
  2. URLFetcher.get_body_stream strips credentials embedded in request URLs and sends them via a preemptive Basic authorization header. For clean URLs it looks up matching machine-scoped credentials in its PasswordDatabase and sends those preemptively. Machine-less default netrc entries are deliberately not sent to arbitrary hosts. When several entries match the same machine, later entries win so explicitly appended repository credentials override earlier netrc credentials.
  3. Guarded stdlib Basic/Digest handlers tolerate unsupported WWW-Authenticate schemes. If no applicable preemptive credential exists, an unsupported challenge now surfaces the underlying 401 HTTPError rather than an opaque ValueError.

Validation

Validated against the AWS CodeArtifact repository that surfaced #3224:

  • before (main at bea9c2e), pex3 lock create --pip-version 24.2 --style universal --no-pypi --index https://aws:<token>@<host>/pypi/<repo>/simple psycopg2-binary==2.9.12 exits 1 with a pg_config sdist metadata failure after downloading the other 11 retained release artifacts to fingerprint them;
  • after, the same command exits 0 with zero fallback fingerprint downloads. Its artifact set and SHA-256 values are byte-identical to the lock produced when the same index serves PEP-503 HTML with #sha256= fragments, including the sdist hash advertised by hashes.sha256;
  • the same result was reproduced for other previously failing projects, including ddtrace.

Regression coverage includes:

  • a local server that accepts Basic authentication but gives unauthenticated requests a Bearer-only challenge, matching CodeArtifact;
  • direct URL and machine-scoped preemptive authentication;
  • unsupported-challenge error behavior;
  • explicit credential precedence over an earlier netrc entry;
  • a composed test that feeds a Pip-redacted Fetched page log line through Locker, authenticates the resulting clean endpoint, parses its PEP-691 JSON, and recovers the advertised artifact SHA-256 without an unauthenticated request.

Local validation completed successfully:

  • format
  • lint
  • typecheck
  • 121 focused auth, fetcher, locker, and PEP-691 tests

🤖 Generated with Claude Code and Amp

Previously, `pex3 lock create` against an index requiring basic auth
that serves the PEP-691 JSON simple API (e.g.: AWS CodeArtifact)
degraded to downloading every artifact of every locked project to
fingerprint it, building sdists along the way, because every request
made by the PEP-691 `FingerprintService` failed:

1. The endpoints scraped from the Pip log carry Pip-redacted
   credentials (`https://user:****@...`), which the stdlib openers
   cannot even parse as a netloc.
2. The stdlib basic and digest auth handlers raise `ValueError` for
   `WWW-Authenticate` challenge schemes they do not implement (AWS
   CodeArtifact challenges with `Bearer`); so known credentials were
   never retried.

Now the scraped endpoints are stripped of redacted credentials, known
machine-scoped credentials are sent preemptively via a basic auth
header (also covering credentials embedded directly in a URL, which
previously failed netloc parsing), and unsupported challenge schemes
degrade to their underlying 401 HTTPError instead of an opaque crash.

Fixes pex-tool#3224

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011ftLY926gkJA6yMbu813cG
@jasonwbarnett
jasonwbarnett force-pushed the pep-691-endpoint-auth branch from f2b39e7 to 61816f6 Compare July 21, 2026 18:03
@jasonwbarnett
jasonwbarnett marked this pull request as ready for review July 21, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PEP-691 fingerprinting cannot authenticate to basic-auth indexes (AWS CodeArtifact): lock creation downloads every artifact and builds sdists

1 participant