🧠 Redrob Intelligent Candidate Discovery Engine

An Explainable Multi-Stage AI Candidate Ranking System

Ranks 100,000 candidate profiles against any Job Description in under 5 minutes — on CPU — with full explainability.

---

📖 Architectural Deep-Dive Manuals

• 🧭 Lexical & Semantic Retrieval Blueprint (Stages 01–06)
• 📈 Redrob Behavioral Scoring Framework (Stage 08)
• 🤖 Neural Re-ranking & Explainable AI Architecture (Stages 10–11)

---

📌 Problem Statement

Modern recruitment platforms receive hundreds of thousands of applications per job posting. Traditional keyword search fails because it cannot understand semantic relevance, behavioural signals, or contextual fit.

The challenge: design an AI-powered Candidate Discovery Engine that —

• Understands complex, multi-section Job Descriptions
• Ranks candidates at 100K scale without a GPU
• Combines lexical + semantic retrieval strategies
• Applies behavioural intelligence from platform signals
• Produces human-readable explanations for every ranking decision

---

💡 Our Solution — A 11-Stage Progressive Ranking Pipeline

Rather than one monolithic model, we chain eleven independent stages — each adding a unique ranking signal — to progressively refine the candidate pool from 100,000 down to the top-100 submission.

100 000 candidates
        │
        ▼ Stage-00: Dataset Download Guard  ← auto-downloads if missing
        │
        ▼ Stage-01: Tier-A Honeypot Detection (hard-rule filtering)
        │
        ▼ Stage-02: Weighted BM25 Retrieval  → Top 10 000
        │
        ├─▶ Stage-03: Download Embedding Model (once, cached)
        │
        ▼ Stage-04: Positive + Negative Query Embedding
        │
        ├─▶ Stage-05: Candidate Embeddings + FAISS Index (one-time)
        │
        ▼ Stage-06: Semantic FAISS Retrieval
        │
        ▼ Stage-07b: Reciprocal Rank Fusion  → Top 500
        │
        ▼ Stage-08: Redrob Behavioural Signal Layer
        │
        ├─▶ Stage-09: Download Cross Encoder (once, cached)
        │
        ▼ Stage-10: Neural Cross Encoder Re-ranking
        │
        ▼ Stage-11: LLM Reasoning + Submission CSV
        │
        ▼ team_mm.csv  (Top 100, ranked)

---

🏗 Architecture Diagram

flowchart TD
    JD["📄 Job Description"]
    S00["🛡️ Stage-00\nDataset Guard"]
    S01["🔍 Stage-01\nHoneypot Detection"]
    S02["📊 Stage-02\nWeighted BM25"]
    S03["📥 Stage-03\nEmbedding Model"]
    S04["🔁 Stage-04\nQuery Embeddings"]
    S05["🧮 Stage-05\nCandidate Embeddings"]
    S06["⚡ Stage-06\nFAISS Retrieval"]
    S07["⚖️ Stage-07b\nRRF Fusion"]
    S08["📈 Stage-08\nBehavioural Signals"]
    S09["📥 Stage-09\nCross Encoder DL"]
    S10["🤖 Stage-10\nCross Encoder Re-rank"]
    S11["💬 Stage-11\nLLM Reasoning"]
    OUT["🏆 Top-100 CSV"]

    JD --> S00 --> S01 --> S02
    S02 --> S03 --> S04 --> S05 --> S06
    S06 --> S07 --> S08 --> S09 --> S10 --> S11 --> OUT

Loading

---

✨ Key Features

Feature	Detail
🛡️ Honeypot Detection	Hard-rule checks eliminate fabricated / low-quality profiles
📊 Weighted BM25	Alpha/Beta/Gamma keyword weights tuned per JD
➕➖ Pos/Neg Boost	Semantic reward & penalty via dense query embeddings
🧠 Dense Retrieval	bge-base-en-v1.5 + FAISS ANN (768-d, CPU)
⚖️ RRF Fusion	Reciprocal Rank Fusion over BM25 + embedding signals
📈 Behavioural Layer	Availability · Market Interest · Trust — computed from Redrob signals
🤖 Cross Encoder	ms-marco-MiniLM-L12-v2 — profile / experience / skills facets
💬 LLM Reasoning	Qwen2.5-1.5B-Instruct — grounded, hallucination-free explanations
⚡ CPU First	No GPU needed; runs on any modern laptop/server
📦 Modular	Each stage is independently replaceable

---

📂 Repository Structure

TEAM_MM/
│
├── config/
│   └── config.yaml               # All pipeline parameters (BM25, RRF, …)
│
├── data/                         # Challenge bundle (JD, spec, validator …)
│   ├── job_description.docx
│   ├── sample_candidates.json
│   ├── sample_submission.csv
│   ├── validate_submission.py    # Official submission validator
│   └── submission_metadata_template.yaml
│
├── artifacts/                    # Pre-computed / downloaded artefacts  (gitignored)
│   ├── candidates.jsonl          ← auto-downloaded by Stage-00 if missing
│   ├── embeddings/               # candidate_embeddings.npy · faiss.index
│   ├── emb_models/               # bge-base-en-v1.5 (HF download)
│   ├── cross_encoder/            # ms-marco-MiniLM-L12-v2 (HF download)
│   ├── reasoning/                # Qwen2.5-1.5B-Instruct (HF download)
│   ├── emb_query/
│   └── keywords/
│
├── outputs/                      # Per-stage outputs (gitignored)
│   ├── conditions/               # honeypot report
│   ├── bm25/
│   ├── embedding/
│   ├── rrf/
│   ├── redrob_signals/
│   ├── cross_encoder/
│   └── reasoning/
│       ├── team_mm.csv           ← final submission
│       └── reasoning_detail.json
│
├── src/redrob_ranker/
│   ├── pipeline/                 # Stage-XX_*.py orchestrators
│   ├── components/               # honeypot · BM25 · RRF · signals
│   ├── ranking/                  # embedding · cross_encoder
│   ├── features/                 # positive · negative · temporal
│   ├── reasoning/                # LLM generation
│   ├── models/                   # model checkers & downloaders
│   ├── config/                   # ConfigurationManager
│   ├── entity/                   # typed config dataclasses
│   ├── constants/
│   └── utils/
│       ├── download_data.py      ← candidate dataset download utility
│       ├── candidate_loader.py
│       └── …
│
├── docs/
│   └── images/
│
├── main.py                       # Single entry-point — runs all stages
├── requirements.txt
├── pyproject.toml
├── submission_metadata.yaml
└── README.md

---

🚀 Quick Start

1 · Prerequisites

• Python 3.10+
• pip / virtual-env
• Git
• No GPU required — CPU-only pipeline

2 · Clone & Install

git clone https://github.com/artignite3/TEAM_MM.git
cd TEAM_MM

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install -e .            # install redrob_ranker package

3 · Data — candidates.jsonl

The candidate pool (candidates.jsonl, ≈ 465 MB) is not stored in Git (it is too large).
Stage-00 of the pipeline tries to download it automatically when it is missing.

✅ Automatic download (recommended)

python main.py
# Stage-00 will detect the missing file and download it automatically.

If the automated download fails (e.g., due to network or Drive permission restrictions), download manually:

🔗 Manual download

1. Open the shared folder:
   https://drive.google.com/drive/folders/19y2BbhXt5scRlmlgYZFdMDo6_jlpLswk
2. Download candidates.jsonl
3. Place it at:

artifacts/candidates.jsonl

You can also invoke the download utility standalone:

python -m redrob_ranker.utils.download_data

4 · Run the Full Pipeline

python main.py

The pipeline runs all stages sequentially.
Pre-computation notes:

Step	What happens	When
Stage-00	Download candidates.jsonl	Only if missing
Stage-03	Download bge-base-en-v1.5	Only if missing
Stage-05	Build FAISS index from 100K candidates	Only if missing — uncomment in main.py
Stage-09	Download ms-marco-MiniLM-L12-v2	Only if missing
Stage-11	Download Qwen2.5-1.5B-Instruct	Only if missing
All stages	Ranking + reasoning	Every run

5 · Validate Submission

python data/validate_submission.py outputs/reasoning/team_mm.csv

---

🧩 Stage-by-Stage Deep Dive

Stage-00 — Dataset Download Guard

Ensures artifacts/candidates.jsonl exists before any stage runs.
Tries gdown → wget → manual guide in order.

Stage-01 — Honeypot Detection

Applies deterministic hard rules to remove fabricated or low-quality profiles:

• Implausible tenure spans
• Career-start date contradictions
• Skill-count extremes
• Timeline inconsistencies

Output: Clean candidate pool + honeypots.csv

Stage-02 — Weighted BM25 Retrieval

Every job description keyword is assigned to one of three weight classes:

Class	Purpose	Weight
Alpha	Must-have skills	+ 2.0
Beta	Nice-to-have skills	+ 1.0
Gamma	Penalty terms (undesired)	− 0.5

BM25 (k1 = 1.5, b = 0.75) returns the top 10,000 candidates.

Stage-03/04 — Query Embeddings

A positive recruiter query and a negative recruiter query are each embedded with BAAI/bge-base-en-v1.5 (768-d, normalised).

Stage-05 — Candidate Embeddings + FAISS Index

Every candidate's merged profile text is embedded and indexed in a flat FAISS index for ANN retrieval. This step is run once and cached.

Stage-06 — Semantic FAISS Retrieval

Positive query → retrieve high-relevance candidates.
Negative query → identify low-relevance candidates.
Final embedding_score = positive_sim − λ × negative_sim (λ = 0.15).

Stage-07b — Reciprocal Rank Fusion

Three ranked lists (BM25 · positive embedding · negative embedding) are fused via RRF (k = 60) with per-signal weights.
Output: Top 500 candidates.

Stage-08 — Redrob Behavioural Signal Layer

Computes a behaviour multiplier [0.5 → 1.3] from three dimensions:

Dimension	Weight	Signals
🟢 Availability	50%	response rate · recency · notice period · open-to-work
🔵 Market Interest	30%	saves · search appearances · interview completion · offer rate
🟣 Trust	20%	profile completeness · verification · connections

Also emits per-candidate warnings and availability confidence scores.

Stage-10 — Cross Encoder Re-ranking

cross-encoder/ms-marco-MiniLM-L12-v2 jointly encodes the JD and each candidate profile across three facets:

Facet	Weight
Profile summary	30%
Experience	50%
Skills	20%

Stage-11 — LLM Reasoning + Submission

Qwen2.5-1.5B-Instruct generates a grounded, one-paragraph rationale for each top-100 candidate referencing actual resume facts — zero hallucinations.

Final output: outputs/reasoning/team_mm.csv

---

⚙️ Configuration

All tuneable parameters live in config/config.yaml:

bm25_rankings:
  alpha_weight: 2.0      # Must-have keywords
  beta_weight:  1.0      # Nice-to-have keywords
  gamma_weight: -0.5     # Penalty keywords
  top_k:        10000    # BM25 shortlist size

embedding:
  lambda_weight: 0.15    # Negative embedding penalty
  top_k:         10000

rrf_fusion:
  k:             60      # RRF constant
  top_n:         500     # Post-fusion cutoff

redrob_signals:
  weights:
    availability: 0.50
    market:       0.30
    trust:        0.20

---

📊 System Performance

Metric	Value
Candidate Pool	100,000 profiles
Hardware	CPU Only — no GPU
Memory	< 16 GB RAM
Total Pipeline Time	≤ 5 minutes (post pre-computation)
Retrieval Engine	BM25 + FAISS
Fusion Strategy	Reciprocal Rank Fusion
Re-ranker	Neural Cross Encoder
Explainability	✅ Per-candidate LLM rationale
Hallucinations	0 — grounded generation only

---

🛠 Tech Stack

Layer	Technology
Language	Python 3.10
Lexical Retrieval	rank-bm25
Embeddings	sentence-transformers — BAAI/bge-base-en-v1.5
Vector Search	faiss-cpu
Re-ranking	cross-encoder/ms-marco-MiniLM-L12-v2
Reasoning	Qwen2.5-1.5B-Instruct via 🤗 Transformers
Data	pandas, numpy
Config	PyYAML, python-box
Download	gdown, wget (auto-fallback)
Output	CSV + JSON

---

🔮 Future Improvements

• Sparse + Dense (SPLADE / ColBERT) hybrid retrieval
• Learning-to-Rank fine-tuning with recruiter feedback
• Real-time candidate update webhooks
• Multi-language JD & profile support
• Resume parsing pipeline (PDF → structured JSON)
• Skill knowledge-graph integration
• Distributed FAISS for sub-second retrieval at 10M+ scale

---

🌟 Highlights

✔ 11-Stage AI Ranking Pipeline — fully modular
✔ Automated Data Download — Stage-00 fetches the dataset if missing
✔ Hybrid Lexical + Semantic Retrieval — BM25 + FAISS
✔ Behavioural Intelligence — Redrob platform signals
✔ Neural Cross Encoder — deep contextual re-ranking
✔ Explainable AI — LLM-generated, grounded rationales
✔ CPU-Only — no GPU, no cloud dependency
✔ Scalable to 100K+ — FAISS ANN at millisecond speed
✔ Zero Hallucinations — fact-grounded reasoning only