Self-Hosted AI Chatbot

Your own private ChatGPT-style chatbot that runs on your machine. It can:

• Chat with an AI model (nothing sent to the cloud).
• Read your documents (PDFs, Word, etc.) and answer questions about them.
• Search the web and read pages for you.

Everything is pre-configured in docker-compose.yml. You only need to start it and download one AI model.

Before you start

You need:

• Docker installed (with the docker compose command).
• An NVIDIA graphics card with its drivers + the NVIDIA Container Toolkit. The AI runs on the GPU and won't start without one.

Quick check that the GPU works in Docker:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If that prints your GPU, you're good.

Setup — 3 steps

1. Start everything

First create a .env with a secret key (required — the stack won't start without it):

cp .env.example .env
# then set WEBUI_SECRET_KEY in .env to the output of:  openssl rand -hex 32

Then start:

docker compose up -d

First run pulls the images and, on first boot, open-webui downloads its embedding + reranker models (~2 GB) from the internet — needs connectivity and a few minutes. Check it's running:

docker compose ps

open-webui shows (unhealthy) during that first-boot download — that's normal, not an error. Wait for it to flip to (healthy) before opening the browser. If an image pull dies with short read ... unexpected EOF (flaky network), just re-run docker compose pull — completed layers are cached and it resumes.

2. Download an AI model

docker compose exec ollama ollama pull gemma3:12b

Which model? For chatting with documents, a small model is plenty — the system does the heavy lifting of finding the right pages, the model just reads them. Bigger isn't better here.

Model	Download command	GPU memory needed	Best for
Gemma 3 12B	gemma3:12b	~8 GB	Recommended. Great all-rounder.
Gemma 3 4B	gemma3:4b	~3 GB	Small / older GPUs.
Llama 3.1 8B	llama3.1:8b	~6 GB	Solid alternative.
Qwen3 30B	qwen3:30b	~19 GB	Only if you have a big GPU and want extra reasoning.

Browse more at ollama.com/library. See what you've downloaded with docker compose exec ollama ollama list.

Note: download progress only appears in the terminal running the pull — it does not show up in docker compose logs ollama. To confirm a model arrived, use ollama list.

3. Open it in your browser

Go to http://localhost:3000. The first account you create is the admin. Pick your model from the menu at the top-left, and start chatting.

Done — web search and document reading are already switched on.

How to use it

Search the web

Click the globe icon in the message box, then ask your question. It searches the web and reads the pages for you automatically. No setup, no API keys.

Ask about your PDFs

Quick way (one chat):

1. Click the + in the message box (or drag a PDF in).
2. Wait for it to finish processing.
3. Ask your question. The file only applies to that one chat.

Reusable way (use the same PDFs anytime):

1. Left sidebar → Workspace → Knowledge → +.
2. Give it a name (e.g. "Research Papers") and create it.
3. Click + → Upload and add your PDFs. Add more whenever you like.
4. In any chat, type # then the collection name to attach it, then ask.

Scanned PDFs are OCR'd with Tesseract (forced in docker-compose.yml), so they take a bit longer. Very poor scans may still have minor text errors.

Bulk (no clicking) — a zip or folder of PDFs: there's an optional import service built into the stack. No scripts to install, no API key to make.

1. One-time: copy .env.example to .env and set a secret:

   cp .env.example .env
   # put the output of `openssl rand -hex 32` as WEBUI_SECRET_KEY in .env

   (You already did this if the stack is running — the same .env is used.)
2. Drop your PDFs (or a .zip of them) into the import/ folder.
3. Run the importer:

   docker compose --profile import run --rm pdf-import

It creates a collection named Imported PDFs (change with IMPORT_COLLECTION in .env) and adds every PDF. Attach it in any chat with #Imported PDFs.

Re-run it any time — it's idempotent: already-imported PDFs (matched by filename) are skipped, only new ones are added. Make sure you've created your admin account in the UI first (the importer needs an admin to exist).

While it runs, expect it to look stuck: each PDF is uploaded and fully OCR'd before the next one starts, so large scans take minutes per file. The container also shows (unhealthy) — cosmetic (it reuses the open-webui image, whose health check probes a web port this job never opens). It prints a + filename line per file, a final done: linked N new... line, then exits and removes itself. Watch progress with docker logs -f <container> (name from docker ps). Because matching is by filename, an edited PDF with the same name is skipped — delete it in the UI (Workspace → Knowledge) first, then re-import.

Grounded answers (no guessing) — recommended

By default a model answers from its own knowledge and can make things up. To get a model that answers only from your PDFs, refuses when the answer isn't there, cites sources, and replies in your language, run once after importing:

docker compose --profile setup run --rm setup-grounded

This creates a model called Docs (grounded) bound to your imported PDFs, and sets it as the default. New chats use it automatically — no # needed. Ask something that's not in your documents and it will say it doesn't have that information instead of guessing. (Re-run the command any time to update it.)

This is a one-time step: the model is bound to the collection, not to the files in it. Adding new PDFs later only needs the importer re-run (or an upload via the UI) — no need to run setup-grounded again. Re-run it only to change the system prompt, base model, or collection.

Use it from other apps (API)

Other services (e.g. a chatbot widget on a Joomla site) can chat with your documents through an OpenAI-compatible API. Provision access once:

docker compose --profile setup run --rm setup-api

It prints an API key tied to a locked-down service account (chat-only — the key can't read files or touch settings). Consumers POST /api/chat/completions with "model": "docs-grounded" and get grounded, cited answers. Full guide with Joomla/PHP examples: docs/API.md; runnable demo: examples/consumer-demo/.

⚠️ Two cautions. (1) Keep the API key server-side (your plugin/backend config) — never in browser JavaScript, where anyone can read it from the page source. (2) Port 3000 is plain, unencrypted HTTP on all interfaces of this machine — fine on a trusted LAN, but if a consumer connects over the internet, put a TLS reverse proxy (Caddy/nginx/Traefik) in front instead of forwarding 3000 raw.

Test that everything works (end-to-end)

Run through this once after setup to confirm all three features work.

1. Everything is running

docker compose ps

All six services should show Up.

2. A model is installed

docker compose exec ollama ollama pull gemma3:12b   # downloads it (skip if done)
docker compose exec ollama ollama list              # should list gemma3:12b

3. Chat works — open http://localhost:3000, sign in, pick gemma3:12b top-left, and send "hello". You should get a reply.

4. Web search works — click the globe icon, then ask something current like "What's the latest news about NVIDIA?". The reply should mention websites it read. (If it errors with "does not support tools", see below.)

5. PDF reading works — click +, upload any PDF, then ask a question about its contents. The answer should come from the document.

If all five pass, you're fully set up.

Everyday commands

docker compose up -d       # start
docker compose down        # stop (your data is kept)
docker compose ps          # see what's running
docker compose logs -f open-webui   # watch the app's logs

To update to newer versions:

docker compose pull && docker compose up -d

If something's wrong

• AI won't start / GPU errors — recheck the GPU test command above. The NVIDIA Container Toolkit is usually the missing piece.
• Settings from docker-compose.yml didn't apply — Open WebUI only reads its env settings the very first time it starts with an empty ~/openwebui_data. After that, the settings live in its database and the env is ignored. So editing env vars later does nothing unless you either change the setting in the UI, or start fresh (see below). This affects web search and tool-calling settings.
• Web search finds nothing — open Admin Panel → Settings → Web Search and set: Web Search on, engine duckduckgo, Web Loader Engine safe_web. Save.
• "does not support tools" error — some models (e.g. gemma3) can't use the AI's built-in tool feature (you'll hit this with web search on). Fix: Admin Panel → Settings → Models → ⚙️, set Function Calling to Legacy (Open WebUI then feeds web results in as text instead of asking the model to call a tool). Note: "Default" in that menu is not the safe option here — it behaves like Native. Use Legacy.
• Start completely fresh — stop the stack, delete ~/openwebui_data, then docker compose up -d. It re-reads all settings from docker-compose.yml on that first boot. (This also erases accounts and chats.)

Where your data lives

Everything (models, chats, uploaded files) is saved in these folders in your home directory:

• ~/ollama_data — downloaded models
• ~/openwebui_data — accounts, chats, documents
• ~/pipelines_data — add-ons

Stopping the app keeps them. Deleting these folders erases everything.