API Reference

Complete API reference for unstructuredDataHandler

📋 Overview

This document provides a comprehensive API reference for all public classes and methods in unstructuredDataHandler. For implementation examples, see the Quick Start Guide.

🤖 Agents API

DocumentAgent

Import: from src.agents.document_agent import DocumentAgent

Constructor

DocumentAgent(config: Optional[Dict[str, Any]] = None)

Parameters:

config (Optional[Dict]): Configuration dictionary (uses defaults if None)

Example:

agent = DocumentAgent()
# or with custom config
agent = DocumentAgent({"provider": "ollama", "model": "qwen2.5:7b"})

extract_requirements()

extract_requirements(
    file_path: Union[str, Path],
    provider: Optional[str] = None,
    model: Optional[str] = None,
    chunk_size: int = 8000,
    overlap: int = 800,
    max_tokens: Optional[int] = None,
    use_llm: bool = True,
    enable_quality_enhancements: bool = True,
    enable_confidence_scoring: bool = True,
    enable_quality_flags: bool = True,
    auto_approve_threshold: float = 0.75
) -> Dict[str, Any]

Parameters:

file_path (str|Path): Path to document (PDF, DOCX, PPTX, HTML, image)
provider (Optional[str]): LLM provider ("ollama", "openai", "cerebras", "anthropic")
model (Optional[str]): Model name (provider-specific)
chunk_size (int): Max characters per chunk (default: 8000)
overlap (int): Overlap between chunks (default: 800)
max_tokens (Optional[int]): Max tokens for LLM response
use_llm (bool): Use LLM for structuring (default: True)
enable_quality_enhancements (bool): Enable 99-100% accuracy mode (default: True)
enable_confidence_scoring (bool): Add confidence scores (default: True)
enable_quality_flags (bool): Detect quality issues (default: True)
auto_approve_threshold (float): Confidence threshold for auto-approve (default: 0.75)

Returns: Dict with structure:

{
    "success": bool,
    "file_path": str,
    "requirements": List[Dict],  # List of requirements
    "sections": List[Dict],      # Document sections (if use_llm=True)
    "quality_metrics": Dict,     # Quality metrics (if quality enhancements enabled)
    "processing_info": Dict,     # Processing metadata
    "error": Optional[str]       # Error message if success=False
}

Example:

result = agent.extract_requirements(
    file_path="requirements.pdf",
    enable_quality_enhancements=True,
    auto_approve_threshold=0.80
)

if result["success"]:
    reqs = result["requirements"]
    metrics = result["quality_metrics"]
    print(f"Extracted {len(reqs)} requirements")
    print(f"Avg confidence: {metrics['average_confidence']:.3f}")

batch_extract_requirements()

batch_extract_requirements(
    file_paths: List[Union[str, Path]],
    **kwargs
) -> Dict[str, Any]

Parameters:

file_paths (List[str|Path]): List of document paths
**kwargs: Same parameters as extract_requirements()

Returns: Dict with structure:

{
    "total_files": int,
    "successful": int,
    "failed": int,
    "results": Dict[str, Dict]  # file_path -> result mapping
}

Example:

results = agent.batch_extract_requirements(
    file_paths=["doc1.pdf", "doc2.pdf", "doc3.pdf"],
    enable_quality_enhancements=True
)

print(f"Processed: {results['successful']}/{results['total_files']}")
for file_path, result in results['results'].items():
    if result['success']:
        print(f"{file_path}: {len(result['requirements'])} requirements")

get_high_confidence_requirements()

get_high_confidence_requirements(
    extraction_result: Dict,
    min_confidence: float = 0.90
) -> List[Dict]

Parameters:

extraction_result (Dict): Result from extract_requirements()
min_confidence (float): Minimum confidence threshold (default: 0.90)

Returns: List of requirements with confidence ≥ min_confidence

Example:

high_conf = agent.get_high_confidence_requirements(
    extraction_result=result,
    min_confidence=0.90
)
print(f"Found {len(high_conf)} high-confidence requirements")

get_requirements_needing_review()

get_requirements_needing_review(
    extraction_result: Dict,
    max_confidence: float = 0.75,
    max_flags: int = 0
) -> List[Dict]

Parameters:

extraction_result (Dict): Result from extract_requirements()
max_confidence (float): Maximum confidence threshold (default: 0.75)
max_flags (int): Maximum acceptable quality flags (default: 0)

Returns: List of requirements needing manual review

Example:

needs_review = agent.get_requirements_needing_review(
    extraction_result=result,
    max_confidence=0.75,
    max_flags=1
)
print(f"{len(needs_review)} requirements need review")

📄 Parsers API

DocumentParser

Import: from src.parsers.document_parser import DocumentParser

Constructor

DocumentParser(
    storage_mode: str = "local",
    local_image_dir: str = "./data/outputs/embeddings",
    minio_config: Optional[Dict] = None
)

Parameters:

storage_mode (str): "local" or "minio" (default: "local")
local_image_dir (str): Local storage directory
minio_config (Optional[Dict]): MinIO configuration

Example:

# Local storage
parser = DocumentParser()

# MinIO storage
parser = DocumentParser(
    storage_mode="minio",
    minio_config={
        "endpoint": "play.min.io:9000",
        "bucket": "my-docs",
        "access_key": "key",
        "secret_key": "secret"
    }
)

get_docling_markdown()

get_docling_markdown(
    file_path: Union[str, Path]
) -> Tuple[str, List[Dict[str, Any]]]

Parameters:

file_path (str|Path): Path to document

Returns: Tuple of (markdown: str, attachments: List[Dict])

Attachment Dict Structure:

{
    "type": "image" | "table",
    "name": str,           # Unique filename
    "path": str,          # Storage path
    "size": int,          # File size in bytes
    "url": Optional[str]  # MinIO URL (if minio mode)
}

Example:

markdown, attachments = parser.get_docling_markdown("document.pdf")

print(f"Markdown length: {len(markdown)} chars")
print(f"Attachments: {len(attachments)}")

for att in attachments:
    print(f"  {att['type']}: {att['name']} ({att['size']} bytes)")

get_docling_raw_markdown()

get_docling_raw_markdown(
    file_path: Union[str, Path]
) -> str

Parameters:

file_path (str|Path): Path to document

Returns: Raw markdown string (no metadata)

Example:

markdown = parser.get_docling_raw_markdown("document.pdf")
print(markdown)

split_markdown_for_llm()

split_markdown_for_llm(
    markdown: str,
    chunk_size: int = 4000,
    chunk_overlap: int = 200,
    respect_headings: bool = True
) -> List[str]

Parameters:

markdown (str): Markdown text to split
chunk_size (int): Max characters per chunk (default: 4000)
chunk_overlap (int): Overlap between chunks (default: 200)
respect_headings (bool): Split at heading boundaries (default: True)

Returns: List of markdown chunks

Example:

chunks = parser.split_markdown_for_llm(
    markdown,
    chunk_size=8000,
    chunk_overlap=800
)

print(f"Created {len(chunks)} chunks")
for i, chunk in enumerate(chunks, 1):
    print(f"Chunk {i}: {len(chunk)} chars")

🧠 LLM API

LLMRouter

Import: from src.llm.router import LLMRouter

Constructor

LLMRouter(
    provider: str,
    model: str,
    config: Optional[Dict] = None
)

Parameters:

provider (str): "ollama", "openai", "cerebras", "anthropic"
model (str): Model name (provider-specific)
config (Optional[Dict]): Additional configuration

Example:

# Ollama
llm = LLMRouter(provider="ollama", model="qwen2.5:7b")

# OpenAI
llm = LLMRouter(provider="openai", model="gpt-4o-mini")

generate()

generate(
    prompt: str,
    system_prompt: Optional[str] = None,
    temperature: float = 0.0,
    max_tokens: Optional[int] = None
) -> str

Parameters:

prompt (str): User prompt
system_prompt (Optional[str]): System prompt
temperature (float): Sampling temperature (0.0-1.0)
max_tokens (Optional[int]): Max tokens in response

Returns: Generated text

Example:

response = llm.generate(
    prompt="Explain quantum computing",
    system_prompt="You are a helpful assistant",
    temperature=0.1
)
print(response)

chat()

chat(
    messages: List[Dict[str, str]],
    temperature: float = 0.0,
    max_tokens: Optional[int] = None
) -> str

Parameters:

messages (List[Dict]): Chat history
temperature (float): Sampling temperature
max_tokens (Optional[int]): Max tokens in response

Message Format:

{
    "role": "system" | "user" | "assistant",
    "content": str
}

Returns: Assistant response

Example:

response = llm.chat([
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "What is Python?"},
    {"role": "assistant", "content": "Python is a programming language."},
    {"role": "user", "content": "What are its main features?"}
])
print(response)

⚙️ Configuration API

ConfigLoader

Import: from src.utils.config_loader import *

load_llm_config()

load_llm_config(
    provider: Optional[str] = None,
    model: Optional[str] = None,
    config_path: str = "config/model_config.yaml"
) -> Dict[str, Any]

Parameters:

provider (Optional[str]): Override provider
model (Optional[str]): Override model
config_path (str): Path to YAML config

Returns: Dict with LLM configuration

Priority: Function params > Env vars > YAML > Defaults

Example:

# Load from config
config = load_llm_config()

# Override with parameters
config = load_llm_config(provider="cerebras", model="llama3.1-8b")

create_llm_from_config()

create_llm_from_config(
    provider: Optional[str] = None,
    model: Optional[str] = None
) -> LLMRouter

Parameters:

provider (Optional[str]): Override provider
model (Optional[str]): Override model

Returns: Configured LLMRouter instance

Example:

# One-line LLM creation
llm = create_llm_from_config()
response = llm.generate("Hello!")

# With custom provider
llm = create_llm_from_config(provider="openai", model="gpt-4o")

📊 Quality Metrics

Confidence Score Structure

{
    "overall": float,        # 0.0-1.0
    "level": str,           # "very_high", "high", "medium", "low", "very_low"
    "factors": List[str]    # Contributing factors
}

Quality Flags

Flag	Description
`vague_text`	Unclear or ambiguous wording
`missing_id`	No requirement ID
`duplicate_id`	ID already used
`incomplete`	Partial requirement
`too_broad`	Requirement too general

Quality Metrics Structure

{
    "average_confidence": float,
    "auto_approve_count": int,
    "needs_review_count": int,
    "total_requirements": int,
    "confidence_distribution": {
        "very_high": int,  # ≥0.90
        "high": int,       # 0.75-0.89
        "medium": int,     # 0.50-0.74
        "low": int,        # 0.25-0.49
        "very_low": int    # <0.25
    },
    "total_quality_flags": int,
    "flag_distribution": Dict[str, int]
}

🔧 Utility Functions

get_api_key()

get_api_key(provider: str) -> Optional[str]

Parameters:

provider (str): "cerebras", "openai", "anthropic", "google"

Returns: API key from environment or None

Example:

key = get_api_key("openai")
if key:
    print("✅ OpenAI key set")
else:
    print("❌ OpenAI key not set")

validate_config()

validate_config(config: Dict) -> bool

Parameters:

config (Dict): Configuration dictionary

Returns: True if valid, False otherwise

Example:

config = load_llm_config()
if validate_config(config):
    print("✅ Configuration valid")
else:
    print("❌ Configuration invalid")

📝 Type Hints

Common Types

from typing import Dict, List, Optional, Union, Any, Tuple
from pathlib import Path

# File path types
FilePath = Union[str, Path]

# Configuration types
Config = Dict[str, Any]

# Result types
ExtractionResult = Dict[str, Any]
Requirement = Dict[str, Any]
Section = Dict[str, Any]
Attachment = Dict[str, Any]

# LLM types
Message = Dict[str, str]
Messages = List[Message]

🚨 Error Handling

Common Exceptions

# Import errors
from src.utils.exceptions import (
    ParsingError,
    LLMError,
    ConfigurationError,
    ValidationError
)

# Example usage
try:
    result = agent.extract_requirements("document.pdf")
except ParsingError as e:
    print(f"Parsing failed: {e}")
except LLMError as e:
    print(f"LLM error: {e}")
except ConfigurationError as e:
    print(f"Configuration error: {e}")

Error Response Structure

{
    "success": False,
    "error": str,              # Error message
    "error_type": str,         # Error type
    "traceback": Optional[str] # Stack trace (debug mode)
}

🔗 Related Documentation

Quick Start: doc/user-guide/quick-start.md
Configuration: doc/user-guide/configuration.md
Architecture: doc/developer-guide/architecture.md
Development Setup: doc/developer-guide/development-setup.md

Last Updated: 2025-01-XX
Version: 2.0

Uh oh!

FilesExpand file tree

api-reference.md

Latest commit

History

api-reference.md

File metadata and controls

API Reference

📋 Overview

🤖 Agents API

DocumentAgent

Constructor

extract_requirements()

batch_extract_requirements()

get_high_confidence_requirements()

get_requirements_needing_review()

📄 Parsers API

DocumentParser

Constructor

get_docling_markdown()

get_docling_raw_markdown()

split_markdown_for_llm()

🧠 LLM API

LLMRouter

Constructor

generate()

chat()

⚙️ Configuration API

ConfigLoader

load_llm_config()

create_llm_from_config()

📊 Quality Metrics

Confidence Score Structure

Quality Flags

Quality Metrics Structure

🔧 Utility Functions

get_api_key()

validate_config()

📝 Type Hints

Common Types

🚨 Error Handling

Common Exceptions

Error Response Structure

🔗 Related Documentation