Fix Novelight chapter list: recover dropped chapters, refresh stale cache, correct reader labels#73
Merged
Merged
Conversation
Novelight's chapter-pagination endpoint clamps out-of-range pages to the last page's content instead of returning empty, so the paging loop ran to MAX_CHAPTER_PAGES and flooded the host, which then 429-rate-limited the real page fetches. Those failures were silently swallowed, persisting a chapter list with holes (e.g. jumping from ch. 102 to 403). - Terminate paging when a page adds no new chapters (the clamp) or is short, instead of relying on an empty page or the page cap. - Classify fetch results: retry 429/5xx/timeout with backoff, degrade on 403/redirect (premium), and abort rather than persist a middle-page hole. - Lower chapter-page concurrency and make jsonField null-safe. Adds NovelightSourceTest coverage for retry, clamp termination, gap-abort, and gated-degrade.
The chapter list is cached per (novel, source) for 6h, so the Novelight paging fix stayed invisible on already-added entries until the cache expired, and nothing (re-add or delete) invalidated it sooner. Add a version field to the cache entry; entries written before the current version are treated as stale, forcing a one-time refetch that heals every existing install.
Novelight reading URLs are /book/chapter/{id} where the id is not the
chapter number, so the reader's URL-derived label showed the id (e.g.
"Chapter 141313") while the chapter list correctly showed "Chapter 102".
Prefer the chapter list's label when it carries a number that differs from
the current one, a no-op for sources whose URL already encodes the number.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reading "Magic Academy's Genius Blinker" (aka "I Became a Flashing Genius at the Magic Academy") on Novelight was broken in three separate ways. This fixes all of them.
Dropped chapters
Novelight's chapter-pagination endpoint returns the last page's content for any out-of-range page instead of an empty response. The paging loop never saw an empty page, so it kept requesting pages up to the cap (about 188 extra requests per load). That flood tripped Novelight's rate limiter, which started returning 429s, and the failed pages were silently discarded. The result was a chapter list with a hole in the middle: it jumped from chapter 102 to chapter 403.
The loop now stops when a page adds no new chapters or comes back short. Failed pages are retried with backoff on 429/5xx/timeout and treated as gated (skipped) on 403. If a page inside the known range never recovers, the load fails loudly instead of saving a list with holes. Concurrency was also lowered.
Stale cached list
Chapter lists are cached per novel for 6 hours, and nothing invalidated them, so the paging fix stayed invisible on novels that were already in the library. This adds a version field to the cache entry. Entries written before the current version count as stale, which forces a one-time refetch on next open.
Wrong chapter number in the reader
Novelight reading URLs look like
/book/chapter/141313, where that id is not the chapter number. The reader built its label from the URL, so it showed "Chapter 141313" while the chapter list correctly showed "Chapter 102". The reader now takes the label from the parsed chapter list when the list has a number that differs from the URL-derived one. This is a no-op for the other sources, whose URLs already contain the real chapter number.Tests
Added unit tests for each part: retry, termination, gap-abort and gated-degrade in the paging loop; cache version staleness; and the label resolver. The usual check passes locally (unit tests, detekt, and both build flavors).