feat(custom-rows): implement server-side cache and proxy endpoints for custom rows#174
Conversation
Build SuccessfulThe plugin compiled successfully against .NET 8 / Jellyfin 10.10.0.
|
RadicalMuffinMan
left a comment
There was a problem hiding this comment.
Heads up on the Letterboxd part of the PR. Right now the custom rows fetch pulls Letterboxd by crawling their HTML pages. For watchlist, films, and lists it hits letterboxd.com/{user}/watchlist/ etc and regexes the film links out of the markup, then for each film it scrapes letterboxd.com/film/{slug}/ to pull the tmdb id, and that film scraping now runs 10 requests at a time. That is the exact thing we deliberately moved away from when we killed the IMDb GraphQL scraping. It's against Letterboxd's terms.
The good news is we already solved this once and the same fix applies here. Letterboxd should go through MDBList instead of being scraped directly. MDBList ingests external lists, Letterboxd included, and we already have an MDBList path and key in this same controller. So FetchLetterboxd should hand the Letterboxd list URL to MDBList and map the items it returns, and the whole isHtmlCrawled branch, the regex parsing, and the parallel letterboxd.com/film/... slug loop can all come out.
If there is a specific list MDBList can't resolve, the acceptable fallback is the RSS feed, which we still have wired in the default case. RSS is a published feed meant to be consumed, so that one is fine. What we should not do is fetch and parse their HTML pages. Below are the rest of the fixes needed
Co-authored-by: Axl <103554043+RadicalMuffinMan@users.noreply.github.com>
Co-authored-by: Axl <103554043+RadicalMuffinMan@users.noreply.github.com>
Co-authored-by: Axl <103554043+RadicalMuffinMan@users.noreply.github.com>
Co-authored-by: Axl <103554043+RadicalMuffinMan@users.noreply.github.com>
Co-authored-by: Axl <103554043+RadicalMuffinMan@users.noreply.github.com>
Co-authored-by: Axl <103554043+RadicalMuffinMan@users.noreply.github.com>
Co-authored-by: Axl <103554043+RadicalMuffinMan@users.noreply.github.com>
…Ds from RSS namespace - Throw ArgumentException for watchlist, films, and user_list requests due to Letterboxd Terms of Service. - Extract tmdb:movieId directly from the RSS feed, avoiding any film page HTML scraping. - Remove letterboxd_slug_to_tmdb.json persistent cache and helper methods.
|
OK, changes made. One note - Letterboxd only publishes a native RSS feed for a user's profile activity/diary (at https://letterboxd.com/username/rss/). There are no native RSS feeds provided by Letterboxd for watchlists (/username/watchlist/), completed films (/username/films/), or custom lists (/username/list/listname/) - which was why the HTML scrapper was needed. Since that is a ToS violation, supporting watchlists or custom lists directly via the FetchLetterboxd source is no longer possible. As a workaround, user can import their Letterboxd watchlists and custom lists into MDBList and configure them using the client's mdblist custom home row source instead - which would fully complies with Letterboxd's terms while preserving the watchlist and custom list features in the app :) Changes:
|
Summary
Migrate custom home rows fetching, parsing, and caching logic from the client to the server-side Jellyfin plugin. This supports TMDB lists/collections, Letterboxd watchlists/lists/films, and MDBList custom rows, shifting processing and rate-limiting throttling entirely to the server and keeping client implementations clean.
Type of Change
Changes Made
CustomRowCacheServicehandling a thread-safe local JSON cache with a 24-hour TTL mapping custom row configuration keys to resolved items.CustomRowControllerexposing the/Moonfin/CustomRows/ItemsAPI.★★★★½) displayed inside client row cards.