Handle pagination of the Docker Scout vulnerability API#197
Merged
Conversation
Fetch every vulnerability page instead of only the first one, following the response next-page marker and aggregating advisories before mapping. A hard page cap of 50 guards against infinite loops; when it is hit, a warning (EventId 3419) is logged and the collected partial data is returned as Succeeded. Each page reuses the cached JWT and the existing single 401 refresh-and-retry, which now restarts the whole fetch rather than a single page. Closes #169.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DockerScoutVulnerabilityProvider.FetchVulnerabilitiesAsyncnow walks the full Docker Scout vulnerability result set instead of reading only the first response. It follows a next-page marker on the response body, aggregates the advisories from every page into a single list, and maps them once.ScoutVulnerabilityResponsegains anextproperty ([JsonPropertyName("next")]) holding the absolute URL of the following page, ornullon the last page. A hard cap of 50 pages guards against runaway loops: when it is hit, a new warning (DockerScoutPageCapReached, EventId 3419) is logged and the collected partial data is returned asSucceededso the truncation is visible in the logs rather than silent. Each page reuses the cached JWT and the existing single 401 refresh-and-retry behavior from #164; the retry now restarts the whole fetch instead of an individual page, so at most one refresh happens per fetch.Why
If the endpoint paginates (common for images with hundreds of CVEs), everything after the first page was silently dropped, making the stored finding count wrong without any error. Closes #169.
Linked issues
Closes #169
Review notes
The live response shape could not be verified against the real API in this environment:
https://api.scout.docker.com/v1/repositories/{ns}/{repo}/tags/{tag}/vulnerabilitiesrequires an authenticated Docker Hub JWT (returns HTTP 403 unauthenticated) and no Docker Hub credentials were available here. The pagination field is modeled on Docker's standard DRF-stylenextfull-URL convention (the same shape Docker Hub's own v2 API uses). The change is backward compatible: a response without anextfield yields exactly one page, identical to the previous behavior — all pre-existing tests still pass unchanged. If the live API turns out to expose pagination differently (a different body field name or RFC 5988Linkheaders), only the marker read inFetchVulnerabilitiesAsyncand theNextproperty would need adjusting; the loop, cap, and retry logic stay the same. New tests cover multi-page merge, the page-cap warning path, and a 401 on a later page refreshing the token once and restarting the whole paginated fetch.Follow-up work
Confirm the
nextmarker name against a live many-CVE response and adjust the JSON property if it differs.