Skip to content

Home

Jump to bottom
Loc Nguyen edited this page Jun 23, 2026 · 3 revisions

📖 Project Wiki: Local LLM & Hybrid RAG System

Welcome to the hoboLocalLLM wiki. This repository contains a fully local, privacy-focused AI chatbot pipeline and an advanced Retrieval-Augmented Generation (RAG) system. No data leaves your machine, no API keys are required, and the entire stack runs offline on your own hardware.

🏗️ System Architecture

The project consists of two primary subsystems working in tandem:

  ┌────────────────────────────────────────────────────────┐
  │                   React/Vite Frontend                  │
  │               (Modern Dark Mode Dashboard)              │
  └──────────────────────────┬─────────────────────────────┘
                             │ (API Requests / SSE Stream)
                             ▼
  ┌────────────────────────────────────────────────────────┐
  │                    FastAPI Backend                     │
  │          (Orchestrates LangChain Pipeline)             │
  └──────────────┬───────────────────────────┬─────────────┘
                 │                           │
                 ▼                           ▼
  ┌──────────────────────────┐   ┌─────────────────────────┐
  │   Local Qdrant Database  │   │      Local LLM Host     │
  │  (Dense + Sparse Index)  │   │  (llama.cpp/llama-server)│
  └──────────────────────────┘   └─────────────────────────┘
  1. LocalLLM Subsystem: Manages the local inference engine (llama.cpp) to host GGUF models on an OpenAI-compatible endpoint.
  2. LocalRAG Subsystem: A LangChain-powered document ingestion and retrieval application featuring Hybrid Search, Contextual Reranking (FlashRank), and a React settings dashboard.

🚀 Key Retrieval Features

To maximize response accuracy and limit hallucinations, the RAG pipeline implements advanced search techniques:

  • Hybrid Search: Combines Dense Vector Search (using SentenceTransformers for conceptual meaning) and Sparse Vector Search (using FastEmbed's SPLADE/BM25 for exact keyword matching). Points are merged using Reciprocal Rank Fusion (RRF).
  • FlashRank Reranking: Base candidates retrieved from the database are passed through a local Cross-Encoder model (ms-marco-MiniLM-L-12-v2) to re-score and sort them, ensuring only the most relevant contexts are fed to the LLM.
  • Negative Control Guardrails: Built-in prompt rules instruct the LLM to refuse queries with a strict "I do not have enough information to answer this question." response if the retrieved context is irrelevant.

🛠️ One-Time Setup

Ensure you have Node.js (v18+) and Miniconda/Anaconda installed.

1. Configure the LLM Environment

Installs llama.cpp and configures the LocalLLM Conda environment:

cd LocalLLM
.\install.ps1

2. Configure the RAG Environment

Installs Python packages (langchain, fastembed, flashrank, fastapi, etc.) and Node modules:

cd LocalRAG
.\install.ps1

🏃 Running the Application

Step 1: Start the Local LLM

  1. Download any .gguf model from HuggingFace (e.g. Qwen2.5, Llama 3.2, Phi-4) and place it in a local directory (e.g., C:\LLMModels\).
  2. Open LocalLLM/startLocalLLM.ps1, update the model path parameter (-m), and run the script: 
    cd LocalLLM
.\startLocalLLM.ps1

This exposes the OpenAI-compatible API on http://localhost:8080.

Step 2: Start the RAG Dashboard

Run the orchestrator script to launch the FastAPI backend (port 8000) and the Vite frontend (port 5173):

cd LocalRAG
.\start_rag.ps1

Your default browser will automatically open to http://localhost:5173.

📊 Offline Model Benchmarking

You can evaluate the latency, generation speed, and response accuracy of different GGUF models on your specific hardware.

To run the offline benchmark suite:

conda activate LocalLLM
cd LocalRAG/backend
python benchmark_models.py

What it evaluates:

  • Time to First Token (TTFT): Measures prompt processing latency.
  • Throughput (Tokens/Sec): Measures text generation speed.
  • Factual Recall: Verifies if the LLM correctly extracts details from ingested texts.
  • Refusal Reliability: Verifies if the model correctly refuses to answer off-context questions instead of hallucinating.

⚙️ RAG Configuration

You can customize parameters directly from the RAG Config tab in the web UI. Changes are written to config.yaml and hot-reloaded instantly:

Parameter Recommended Value Description
Chunk Size 500 - 1000 Target character size of text chunks.
Chunk Overlap 10% of Chunk Size Maintained boundaries between chunks.
Enable Hybrid Search True Combines Dense + Sparse retrievers.
Enable FlashRank Reranker True Re-orders contexts using Cross-Encoder.
Base Retrieve Limit 10 - 15 Chunks retrieved before reranking.
Rerank Limit (Top K) 3 - 5 Final count of chunks passed to the LLM.
Temperature 0.1 Low temperature ensures factual answers.

Clone this wiki locally