Add lidarr as downloader

[avedor]

Implements an MVP version of a Lidarr downloader using the lidarr-go sdk. Currently, can't seem to automatically download using slskd, but the general workflow is functional. Would probably close #16

[LumePart]

I'll try running it in my test environment when I have the time. the code looks really good at the moment.

It would be nice to get a function to check the download progress of the album, and if it hasn't progressed at all, let's say in 15 minutes, then the album download will stop (and removed from Lidarr). That would allow for the track to be downloaded from another downloader.

[avedor]

sounds good! i've got the actual logic written up. how would you envision something like that running? i've added it as a goroutine for now, but if there's a different architecture you'd prefer, just name it!

[LumePart]

i've added it as a goroutine for now

Yeah, goroutine is what I was thinking too

I looked through and ran the code and I'm a bit confused about findArtist() and addArtistIfNeeded(), previously when I investigated Lidarr, I did a manual flow of API requests to download the album and I only used /api/v1/album endpoint.

This is the request I used for adding and sending the search to indexers:

POST http://URL:PORT/api/v1/album
Header: X-Api-Key: API_KEY
Body:

{
 "foreignAlbumId": "70bdc19f-66fb-4adf-9124-925ace531101",
 "monitored": true,
 "qualityProfileId": 1,
 "rootFolderPath": "/music",
 "monitored": true,
 "artist": {
     "foreignArtistId": "89ea85c2-c90a-46ea-93db-9bed95d5c8f8",
     "qualityProfileId": 1,         // Artist's Quality Profile ID
     "metadataProfileId": 2,        // Artist's Metadata Profile ID
     "monitored": true,
     "rootFolderPath": "/music"
     },
 "addOptions": {
     "searchForNewAlbum": true
     }
}

AlbumID and artistID came from album lookup and the ProfileID's could probably be config parameters if the SDK supports it.
I had trouble implementing the same flow with the SDK, but I can try again tomorrow.

And i fixed a small issue with track.MainArtist in the dev branch, when searching for albums it's best to use that field, track.Artist could contain all the artists on the track (i.e Justice feat. Tame Impala)

[avedor]

Yeah! so the whole flow was a little confusing for me as well and the sdk doesn't seem to map 1:1 onto the api endpoints which is... frustrating.

my understanding is we've got the foreign (read musicbrainz) ids for everything and the internal (lidarr) ids. Lidarr ids for artist/album don't get set if the artist isn't in lidarr already, so findArtist does the lookup using a string query--though could probably be switched to use the artist mbid if we add that as a field in the track struct--then the addArtistIfNeeded is creating the artist in lidarr, if it isn't already there, using defaults pulled from the root settings.

i'll dig into the sdk a little further to see if i can bypass a lot of the lookups and just use the mbids to create the album in lidarr

[avedor]

just for complete context, between these changes, and those in #22, i've now got a new playlist in jellyfin with 33/50 tracks!

as far as the flow using sdk is concerned, i think we're roughly stuck at:

1. look up artist using string term search and return ArtistResource
2. pass that into addArtistIfNeeded which checks to see if a. the artist has a path set and b. has a non-zero timestamp for when the artist was added. if not, it creates a new ArtistResource with some defaults from the root settings and calls CreateArtist to add the artist to lidarr
3. repeat the same basic flow for album and release

[LumePart]

Wouldn't it be more straightforward to do it without the SDK? It would reduce the amount of flows needed and would give us greater control of the request payloads. Especially when we don't need most of the API functionality (most likely 4-5 endpoints for a full workflow).

The flow would look pretty much like this:

1. Query album to get artistID and AlbumID
2. Request album to be downloaded/sent to indexer
3. Periodically check if it's downloading
4. If downloaded, mark track as Present otherwise clean Lidarr of artist/track

Feel free to debate me on this, maybe i just don't get the point of the SDK when Lidarr's API documentation is pretty good.

[avedor]

This is the first time I've used an SDK, so I'm happy to do it either way! I just pushed up a couple squashed commits that should do all of this a little more cleanly (storing the MBIDs) in the track struct for ease. I also updated the goroutine to both clear stale downloads and mark present newly available tracks

Looking forward to your thoughts!

[LumePart]

Made a few changes in the dev branch, main changes include:

• Added a MonitorDownloads([]*models.Track) function to the Downloader interface, Lidarr most likely can make use of this.
• queryTrack() and getTrack() are asynchronous now
• Added a few fields to Track struct, including a field to store the track size (if needed) and an field from ListenBrainz (length (in ms))

I'm also pretty much finished with implementing slskd, you can check how I implemented download monitoring there

Also, the linter fails, but that is expected atm.

[avedor]

i did something that might be a little too crazy with the download monitoring. go doesn't allow for reusability quite as much as my SRE brain would like, but i think this could be a path forward to keep from just copying the same code into lidarr.go

of course happy to find a different solution if this feels too out there!

[LumePart]

Makes perfect sense, I can hopefully test it this week.

I also left a couple of reviews in May, regarding the Lidarr part. I don't believe they have been addressed (or Github is just wonky)

A related question as well:
Is Lidarr fixed now? I heard there were some issues with their metadata servers, but some people say it's working for them 🤷

[tessamerrill]

Is Lidarr fixed now? I heard there were some issues with their metadata servers, but some people say it's working for them 🤷

I'll jump in and say that I've been using a fork that uses a different metadata server (that actually works).

[LumePart]

I think it would be better to move the monitor refactoring into a separate pull request, after Lidarr is fully implemented. This would simplify testing and allow for a cleaner refactor, especially since monitor.go will likely require a shared struct.

Right now, I can't test Lidarr properly because my test environment doesn't like hearring-aid (the MusicBrainz server requirements are a bit too much for it). Since I'm not sure what fields Lidarr needs yet, I can't provide meaningful feedback at the moment.

But according to Lidarr's Issues page, they are pretty close in making it work again.

[avedor]

happy to report that the main lidarr specific flow is working more or less as expected! last thing is to find a way to play nice with the monitor (tracking is at the song level vs the album level) to allow explo to finish adding songs to playlists

[NemesisHubris]

This looks great! Excited to try out when this gets merged!

[LumePart]

Hey - Hey, sorry for the late reply.
I've been testing the PR and it works pretty well!
Got the monitoring working as well by using the track ID instead of the filename (needs more testing though). I will most likely do a PR to your branch once I've got everything sorted.

[LumePart]

I roughly got Lidarr working as expected, at least when pulling songs from ListenBrainz discovery playlists. There's still a bit more work needed to get it working with custom playlist imports.

I see you're in the middle of rebasing. I'll push my changes once you're done and give a more detailed overview of what changed, along with a few Lidarr related issues I ran into.

[avedor]

awesome! yeah i think my last overhaul of this was before the custom playlists, so i'm happy to get that plugged in too!

rebase is set, builds are working again, so i'm looking forward to hearing from you!

[LumePart]

Current state is working for me.

Changes/additions I made

• Made monitoring check TrackID instead of file (For Lidarr TrackID is the album ID)
• Downloaded albums are now detected via Lidarr history. Queue itself was not enough, since it only contains incomplete albums
• Search status is checked for every requested album. Some indexers can be slow, so skipping this could interfere with monitoring. By default, it retries up to 8 times with a 15-second interval before failing (similarly to Slskd)
• Root folders are now searched by name instead of returning the first available one

I'm having trouble deciding whether Explo should move Lidarr downloads itself. I think it would be cleaner if users created a new root folder for Explo and let Lidarr handle moving the downloads into the main library. Root folders also allow custom quality profiles and artist tags, which helps keep the library organized. Adding 50+ new artists can get pretty messy without proper tagging.

One issue I'm still running into is that Lidarr's queue doesn't always detect all downloading albums when running Explo back-to-back. For example:

1. Run Explo, let the downloads populate the downloader, then stop Explo
2. Clear Lidarr artists, queue, and files (including deleting the magnet files from the downloader).
3. Run Explo again.

On the second run, Lidarr's queue might only report around 11 albums downloading even though the downloader has around 30 queued. This makes the queue unreliable. Maybe that's an issue on my end (cache related or something)? After restarting Lidarr, the queue detection is okay again

Has anyone else seen this behavior, or could this be something specific to my setup?