fix(import): RM-17159 AI Importer couchbase sidebar pages duplicated

[xavierandueza]

🚥 Resolves RM-1759

🧰 Changes

This fixes 2 issues:

1. For sites where we had to use an LLM to organize, the LLM would sometimes duplicate a page and we'd end up with a duplicated page in the results
2. Fix: to dedupe after initial scrape from urls, and to dedupe again if LLM used to organize
3. The threshold to use LLMS to organize was too strict - we checked if any section was over 50, but for large docs sites this is common. Now we check that the average is under a threshold of 200 and that we have at minimum 2x categories.

🧬 QA & Testing

Ran on couchbase, and verified its now got good categories - 49x for 3500 pages, with no duplicates.

Unit tests also created.

Also tested on:

• https://docs.ecwid.com/
• https://docs.whereby.com/
• https://www.mongodb.com/docs/ (very large bet good for us to know about - its 19000 docs across 180 categories which it handles fine now)
• https://xata.io/docs/
• https://orm.drizzle.team/docs/overview

All look good with new handling.