Storing different entities in DB

[diemah77]

@ksassnowski Let's say I would like to fetch and store/update additional information from fussballdaten.de like team line-up, league table. How would I go about this?

I would definitely need different item proccessors for each of this entity. So what is the recommended way to setup the spider to accomplish such a scenario?

[pheeque]

If I understand your question correctly, I think this where the Dispatching requests section on https://roach-php.dev/docs/processing-responses is useful.
You can yield further requests from the regular parse method with their own respective parser methods. Further processing can then be done through the ItemProcessor interface.

[diemah77]

That's right. But all of the yielded items within the dispatched requests will run through all of the defined item processors, right?
What I would like to have is: for each dispatched line-up request just use specific item processors. The only way I see this happening is to provide some kind of flag/identifier within the item data. Then using multiple if/else to check for those flags and apply DB persistence for the flagged entity.

Example: For the first match dispatch an additional request to get the line-ups of both teams. Process both line-ups through specific line-up item processors like ValidateLineUp and SaveLineUpToDB, but not through ValidateMatch and SaveMatchToDB item processors.

[pheeque]

You might be better served by overriding the initialRequests of the spider and defining a parseOverview for the initial stage e.g fetching the list of matches [more details here https://roach-php.dev/docs/spiders].
But it seems like you are trying to make multiple crawls in the subsequent requests. You probably will need multiple spiders with startUrls from previous crawls.

I'll defer to @ksassnowski as to what the best practice is.

[ksassnowski]

Sorry for the long delay.

So there’s not really a "right" answer to this. You could approach this in a couple of different ways.

1. You could set some sort of type key on each item. Your item processors could then check this key and simply pass the item along if they’re not supposed to process it. A little bit clunky, but it would work.
2. If the additional data isn’t intrinsically tied to the data you just scraped, this could also be an entirely different spider that you run separately. Taking the fussballdaten.de example, say you want to know the table standing of each team at the time the match was played. This could be a separate spider that only fetches the table standings and has an item processor that will add that information to the match data your previously scraped.
3. You could also have an item processor, that "collects" match data in an internal array and only passes it to the next processor once all of its data has been collected. In this case, you’d need some kind of identifier for a match so you can relate the different bits of information scraped from different sub pages back to it.

[diemah77]

Is there a way to provide a parameter to the dispatched request yield $this->request('GET', $link->getUrl()); like so yield $this->request('GET', $link->getUrl(), 'parse', $dynamicParameter);?

[ksassnowski]

Check out the 1.1.0 release of Roach. It includes this PR which adds the option to define both custom item classes as well as processors that only handle certain kinds of items. This should make handling different types of items yielded from the same spider much simpler. I haven't updated the docs yet, but the PR has a comprehensive description.