Skip to content

development-setup.md

Latest commit

 

History

History
727 lines (521 loc) · 12.9 KB

development-setup.md

File metadata and controls

727 lines (521 loc) · 12.9 KB
Raw

Development Setup Guide

Complete guide to setting up your development environment

📋 Prerequisites

Required Software

  • Python 3.10+ (3.10, 3.11, 3.12 tested)
  • Git (for version control)
  • pip (Python package manager)
  • virtualenv or venv (for isolation)

Optional Software

  • Ollama (for local LLM inference)
  • Docker (for containerized development)
  • VS Code or PyCharm (recommended IDEs)

🚀 Quick Setup (5 Minutes)

Option A: Using Test Script (Recommended)

Best for: Reproducible, isolated testing environment

# Clone the repository
git clone https://github.com/SoftwareDevLabs/unstructuredDataHandler.git
cd unstructuredDataHandler

# Run tests in isolated venv (creates .venv_ci automatically)
./scripts/run-tests.sh

# Or run specific tests
./scripts/run-tests.sh test/unit -k deepagent

This script:

  • Creates .venv_ci virtual environment automatically
  • Installs pinned pytest for reliable test runs
  • Runs tests with proper PYTHONPATH configuration
  • Reuses the same environment on subsequent runs

Option B: Manual Setup

Best for: Active development and debugging

1. Clone Repository

# Clone the repository
git clone https://github.com/SoftwareDevLabs/unstructuredDataHandler.git
cd unstructuredDataHandler

2. Create Virtual Environment

# Create virtual environment
python3 -m venv .venv

# Activate (macOS/Linux)
source .venv/bin/activate

# Activate (Windows)
.venv\Scripts\activate

3. Install Dependencies

# Upgrade pip
pip install --upgrade pip

# Install development dependencies
pip install -r requirements-dev.txt

# Install documentation dependencies (optional)
pip install -r requirements-docs.txt

4. Set Python Path

# macOS/Linux
export PYTHONPATH=.

# Windows
set PYTHONPATH=.

# Or add to .bashrc/.zshrc (macOS/Linux)
echo 'export PYTHONPATH=.' >> ~/.zshrc
source ~/.zshrc

5. Verify Installation

# Run tests
python -m pytest test/ -v

# Check linting
python -m pylint src/ --exit-zero

# Check type checking
python -m mypy src/ --ignore-missing-imports

🔧 Detailed Setup

Python Environment Setup

Option 1: venv (Standard)

# Create environment
python3 -m venv .venv

# Activate
source .venv/bin/activate  # macOS/Linux
.venv\Scripts\activate     # Windows

# Install packages
pip install -r requirements-dev.txt

Option 2: conda (Alternative)

# Create environment
conda create -n unstructuredDataHandler python=3.11

# Activate
conda activate unstructuredDataHandler

# Install packages
pip install -r requirements-dev.txt

Option 3: pyenv (Version Management)

# Install Python 3.11
pyenv install 3.11.5

# Set local version
pyenv local 3.11.5

# Create virtualenv
python -m venv .venv
source .venv/bin/activate

# Install packages
pip install -r requirements-dev.txt

📦 Dependencies

Core Dependencies

# requirements.txt
docling>=2.55.1           # Document parsing
docling-core>=2.48.4      # Docling core library
pydantic>=2.0.0          # Data validation
requests>=2.31.0         # HTTP requests
pyyaml>=6.0             # YAML configuration

Development Dependencies

# requirements-dev.txt
pytest>=8.0.0            # Testing framework
pytest-cov>=4.1.0       # Coverage reporting
pytest-timeout>=2.2.0   # Test timeouts
pylint>=3.0.0           # Linting
mypy>=1.8.0             # Type checking
ruff>=0.1.0             # Fast linting
black>=23.0.0           # Code formatting
isort>=5.12.0           # Import sorting

UI Dependencies

# requirements-streamlit.txt
streamlit>=1.28.0        # Web UI
markdown>=3.5.0         # Markdown rendering
pandas>=2.0.0           # Data manipulation
plotly>=5.17.0          # Visualizations (optional)

Documentation Dependencies

# requirements-docs.txt
mkdocs>=1.5.0           # Documentation site
mkdocs-material>=9.0.0  # Material theme
mkdocstrings>=0.24.0    # API documentation

🤖 LLM Provider Setup

Ollama (Local - Recommended for Development)

1. Install Ollama

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
# Download from https://ollama.com/download

2. Start Ollama Server

# Start server (runs in background)
ollama serve &

# Verify it's running
curl http://localhost:11434/api/tags

3. Pull Models

# Recommended: Balanced model
ollama pull qwen2.5:7b

# Alternative: Faster model
ollama pull qwen2.5:3b

# Alternative: Better quality
ollama pull qwen3:14b

# Alternative: LLaMA
ollama pull llama3.1:8b

4. Configure Environment

# .env file
DEFAULT_LLM_PROVIDER=ollama
DEFAULT_LLM_MODEL=qwen2.5:7b
OLLAMA_BASE_URL=http://localhost:11434

5. Test

# Test Ollama
ollama run qwen2.5:7b "Hello, how are you?"

# Test in Python
python -c "from src.utils.config_loader import create_llm_from_config; llm = create_llm_from_config(); print(llm.generate('Hello!'))"

Cloud Providers (OpenAI, Cerebras, Anthropic)

OpenAI

# Get API key from https://platform.openai.com/api-keys

# Set in .env
OPENAI_API_KEY=sk-...
DEFAULT_LLM_PROVIDER=openai
DEFAULT_LLM_MODEL=gpt-4o-mini

Cerebras

# Get API key from https://cloud.cerebras.ai/

# Set in .env
CEREBRAS_API_KEY=your-key-here
DEFAULT_LLM_PROVIDER=cerebras
DEFAULT_LLM_MODEL=llama3.1-8b

Anthropic

# Get API key from https://console.anthropic.com/

# Set in .env
ANTHROPIC_API_KEY=sk-ant-...
DEFAULT_LLM_PROVIDER=anthropic
DEFAULT_LLM_MODEL=claude-3-5-sonnet-20241022

🔧 IDE Setup

VS Code

1. Install Extensions

code --install-extension ms-python.python
code --install-extension ms-python.vscode-pylance
code --install-extension ms-python.black-formatter
code --install-extension charliermarsh.ruff

2. Configure Settings

Create .vscode/settings.json:

{
  "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
  "python.linting.enabled": true,
  "python.linting.pylintEnabled": true,
  "python.formatting.provider": "black",
  "python.testing.pytestEnabled": true,
  "python.testing.unittestEnabled": false,
  "editor.formatOnSave": true,
  "editor.codeActionsOnSave": {
    "source.organizeImports": true
  }
}

3. Create Launch Configuration

Create .vscode/launch.json:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Python: Current File",
      "type": "python",
      "request": "launch",
      "program": "${file}",
      "console": "integratedTerminal",
      "env": {
        "PYTHONPATH": "${workspaceFolder}"
      }
    },
    {
      "name": "Pytest: Current File",
      "type": "python",
      "request": "launch",
      "module": "pytest",
      "args": ["${file}", "-v"],
      "console": "integratedTerminal",
      "env": {
        "PYTHONPATH": "${workspaceFolder}"
      }
    },
    {
      "name": "Streamlit UI",
      "type": "python",
      "request": "launch",
      "module": "streamlit",
      "args": ["run", "test/debug/streamlit_document_parser.py"],
      "console": "integratedTerminal",
      "env": {
        "PYTHONPATH": "${workspaceFolder}"
      }
    }
  ]
}

PyCharm

1. Configure Python Interpreter

  1. Open Settings → Project → Python Interpreter
  2. Click Add Interpreter → Virtualenv Environment
  3. Select Existing environment
  4. Choose .venv/bin/python

2. Configure PYTHONPATH

  1. Open Run → Edit Configurations
  2. Add Environment variables: PYTHONPATH=.

3. Configure Testing

  1. Open Settings → Tools → Python Integrated Tools
  2. Set Default test runner to pytest

📝 Configuration Files

.env File (Required)

# Copy template
cp .env.example .env

# Edit with your settings
vim .env

Minimal .env:

# LLM Provider
DEFAULT_LLM_PROVIDER=ollama
DEFAULT_LLM_MODEL=qwen2.5:7b
OLLAMA_BASE_URL=http://localhost:11434

# Logging
LOG_LEVEL=INFO
DEBUG=false

# Storage
LOCAL_IMAGE_DIR=./data/outputs/embeddings

Pre-commit Hooks (Optional but Recommended)

1. Install pre-commit

pip install pre-commit

2. Create .pre-commit-config.yaml

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-added-large-files

  - repo: https://github.com/psf/black
    rev: 23.12.1
    hooks:
      - id: black
        language_version: python3

  - repo: https://github.com/pycqa/isort
    rev: 5.13.2
    hooks:
      - id: isort

  - repo: https://github.com/charliermarsh/ruff-pre-commit
    rev: v0.1.9
    hooks:
      - id: ruff

3. Install hooks

pre-commit install

🧪 Testing Setup

Run Tests

# All tests
PYTHONPATH=. python -m pytest test/ -v

# Specific suite
PYTHONPATH=. python -m pytest test/unit/ -v
PYTHONPATH=. python -m pytest test/integration/ -v

# With coverage
PYTHONPATH=. python -m pytest test/ --cov=src/ --cov-report=html

# Open coverage report
open htmlcov/index.html

Test Markers

# Run only unit tests
pytest -m unit

# Skip slow tests
pytest -m "not slow"

# Skip tests requiring Ollama
pytest -m "not requires_ollama"

# Skip tests requiring API keys
pytest -m "not requires_api_key"

🔍 Code Quality

Linting

# Pylint
python -m pylint src/ --exit-zero

# Ruff (faster alternative)
python -m ruff check src/

# Fix automatically
python -m ruff check src/ --fix

Type Checking

# Mypy
python -m mypy src/ --ignore-missing-imports

# Specific module
python -m mypy src/agents/document_agent.py

Code Formatting

# Black
python -m black src/ test/

# Check only (no changes)
python -m black src/ test/ --check

# isort (import sorting)
python -m isort src/ test/

🌿 Branch Strategy

Branch Naming Convention

# Feature branches
dev/<alias>/<feature-name>

# Example
dev/johndoe/add-new-llm-provider

# Bug fix branches
dev/<alias>/fix-<issue-number>

# Example
dev/johndoe/fix-123

Working with Branches

# Create feature branch from dev/main
git checkout dev/main
git pull origin dev/main
git checkout -b dev/yourname/your-feature

# Make changes and commit
git add .
git commit -m "feat: add new feature"

# Push to remote
git push origin dev/yourname/your-feature

# Create pull request on GitHub

Code Submission Process

  1. Develop on feature branch (dev/<alias>/<feature>)
  2. Create PR to dev/main
  3. Get approval and merge (squash recommended)
  4. Cherry-pick to inbox branch
  5. Git2Git auto-creates PR to Overall Tool Repo
  6. Approve and complete the auto-PR

🐛 Troubleshooting

"ModuleNotFoundError"

Solution: Set PYTHONPATH

export PYTHONPATH=.
python -m pytest test/ -v

"ImportError: No module named 'docling'"

Solution: Install dependencies

pip install -r requirements-dev.txt

"Connection refused" (Ollama)

Solution: Start Ollama server

ollama serve &

Tests fail with "API key not found"

Solution: Set API key or skip those tests

# Set API key
export OPENAI_API_KEY="your-key"

# Or skip those tests
pytest -m "not requires_api_key"

Type checking fails

Solution: Add ignore flag or fix types

# Ignore missing imports
mypy src/ --ignore-missing-imports

# Or fix by adding type stubs
pip install types-requests types-pyyaml

📦 Building and Distribution

Building Package

# Install build tools
pip install build twine

# Build distribution
python -m build

# Check distribution
twine check dist/*

Running in Docker

# Build Docker image
docker build -t unstructuredDataHandler .

# Run container
docker run -p 8501:8501 unstructuredDataHandler

🔗 Related Documentation

  • Quick Start: doc/user-guide/quick-start.md
  • Configuration: doc/user-guide/configuration.md
  • Architecture: doc/developer-guide/architecture.md
  • Testing Guide: doc/user-guide/testing.md
  • Contributing: CONTRIBUTING.md

✅ Checklist

Initial Setup

  • Python 3.10+ installed
  • Repository cloned
  • Virtual environment created
  • Dependencies installed
  • PYTHONPATH set
  • Tests passing

LLM Setup

  • Ollama installed (or cloud API key)
  • Model pulled (Ollama) or key set (cloud)
  • .env file configured
  • LLM test successful

IDE Setup

  • IDE configured (VS Code or PyCharm)
  • Extensions/plugins installed
  • Linting enabled
  • Testing framework configured

Quality Tools

  • Pre-commit hooks installed (optional)
  • Linting tools working
  • Type checking working
  • Code formatting working

Last Updated: 2025-01-XX
Version: 2.0