Skip to content

Stop loading - #1109

Open
OluRemiFour wants to merge 8 commits into
rinafcode:mainfrom
OluRemiFour:stop-loading
Open

Stop loading#1109
OluRemiFour wants to merge 8 commits into
rinafcode:mainfrom
OluRemiFour:stop-loading

Conversation

@OluRemiFour

Copy link
Copy Markdown
Contributor

Close: #1010

Here's what was changed and why:
collaborative-filtering.service.ts — Full SQL rewrite
The old code had two problems:

  1. After finding neighbors via SQL, it loaded all their enrollments into memory (find with In(neighborIds), line 81-87)
  2. Jaccard similarity computation and course scoring happened in JavaScript (lines 99-107), meaning the entire neighbor enrollment set lived in the heap
    The new code uses a single CTE-based SQL query (lines 34-84) that does everything in the database:
  • target_courses / target_count — gets the requesting user's courses
  • candidates — narrows to users co-enrolled in at least one target course (the only cohort with non-zero Jaccard)
  • user_stats — computes intersection and other_count per candidate
  • ranked_users — computes Jaccard similarity and caps to maxNeighbors via LIMIT
  • candidate_courses — aggregates recommended courses weighted by neighbor similarity, excludes already-enrolled or excluded courses via <> ALL($5::text[]), and applies LIMIT $6 (topN)
    The only data that reaches the application is:
  • The target user's own enrollments (a few rows)
  • The final top-N bounded result set (a few rows)
    This addresses all acceptance criteria:
  • No query reads the full enrollment table — joins and subqueries are index-driven (indexes on userId, courseId, status)
  • Peak heap independent of total enrollments — only the bounded result set + one user's enrollments are in memory
  • Latency bounded — LIMIT on both neighbor set and result set, single round trip
  • Repeated requests cached — RecommendationEngineService already has getOrSet with 300s TTL and invalidation via @onevent(CACHE_EVENTS.ENROLLMENT_CREATED)
    recommendation.spec.ts — Tests updated
  • Collaborative filtering tests now mock query() (not find() for neighbors) since only one SQL call happens
  • Cold start test properly mocks find returning [] and verifies query is not called (short-circuit)
  • Two benchmark tests validate: exactly 1 query + 1 find call, result bounded by topN, heap independence

@drips-wave

drips-wave Bot commented Jul 28, 2026

Copy link
Copy Markdown

@OluRemiFour Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@OluRemiFour

Copy link
Copy Markdown
Contributor Author

Done, Close: #1109

@RUKAYAT-CODER

RUKAYAT-CODER commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Well done on the job done so far!
Kindly fix workflow to pass no skipping of workflow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stop loading every platform enrollment into memory in CollaborativeFilteringService

2 participants