Local Web Scraper RAG Dashboard

A local, event-driven Retrieval-Augmented Generation (RAG) web application. The system crawls public web page URLs, parses content, computes vector embeddings, indexes them, and allows users to chat with the crawled information using local LLM inference.

🐙 Repository

• GitHub Repository: https://github.com/Idimma/web-scraper-rag

⚙️ Technical Stack

• Orchestration: LlamaIndex manages Document parsing, sentence split chunking, and search vector indexing pipelines.
• Inference Gateway: Ollama runs tinyllama:latest locally for both computing vector embeddings and running generative completions (100% offline, zero cost).
• Storage Index: LlamaIndex's built-in SimpleVectorStore persists indices as local JSON files, avoiding heavy C++ binary dependencies (like chromadb or onnxruntime) which lack pre-compiled wheels for newer Python environments.
• Web Scraper: Custom BeautifulSoup4 scraper that strips script, style, header, and footer tags to index clean main content.
• Backend Server: FastAPI exposing ingestion (/scrape), query (/query), and dashboard status (/status) endpoints.

🚀 Setup & Execution

1. Prerequisite: Start Ollama

Ensure Ollama is running and has the target model downloaded:

ollama run tinyllama

2. Install Python Dependencies

pip3 install -r requirements.txt

3. Start the Backend Server

python3 app.py

• The FastAPI application will start on http://localhost:8080.
• Open the link in your web browser to access the dashboard.