MLflow GenAI Tutorial Series

Tutorial: Mastering GenAI with MLflow

This tutorial series teaches you, in a step-by-step manner, how to use MLflow's open source platform for building, tracking, tracing, prompt registry and optimization, evaluating, and debugging GenAI applications.

🎬 Series Overview Video

Watch the full overview: Mastering GenAI Development with MLflow

📚 Tutorial Structure

Notebook	Title	Description	Video Tutorial
1.1	Setup and Introduction	Understanding MLflow for GenAI Installation and configuration First tracked run MLflow UI basics	Watch Tutorial
1.2	Experiment Tracking for LLMs	Tracking LLM parameters and metrics Comparing model configurations Cost tracking and optimization Organizing experiments with tags Parent-child runs for workflows	Watch Tutorial
1.3	Introduction to Tracing	Auto-tracing with MLflow Understanding the trace model Manual instrumentation Viewing traces in UI	Watch Tutorial
1.4	Manual Tracing and Advanced Observability	Custom span decorators Tracing complex workflows Debugging with traces Multi-step agentic patterns	Watch Tutorial
1.5	Prompt Management	Creating prompt templates Versioning prompts Registering in the Prompt Registry Searching Prompt Registry Using prompts from the Prompt Registry Linking prompts to experiments	Watch Tutorial
1.6	Framework Integrations	OpenAI direct API integration LangChain chains and workflows LlamaIndex document indexing and RAG Framework comparison matrix Best practices for each framework	Watch Tutorial
1.7	Evaluating Agents	LLM-as-Judge evaluation patterns MLflow built-in scorers (RelevanceToQuery, Correctness, Guidelines) Custom scorers with @scorer decorator DeepEval integration for conversations Session-level multi-turn evaluation	Watch Tutorial
1.8	Prompt Optimization with GEPA	Automatic prompt optimization with GEPA algorithm MLflow Prompt Registry integration Before/after evaluation comparison	Watch tutorial
1.9	Complete RAG Application	Building a full RAG pipeline End-to-end tracing Performance analysis RAG evaluation with RAGAS metrics	Coming Soon...
1.10	Multi-Agent Supervisor Pattern	Supervisor agent that routes queries to specialized subagents Genie agent for structured data queries (SQL over pandas DataFrame) Knowledge Assistant for document retrieval and policy Q&A Full supervisor-subagent trace hierarchy with MLflow 3-layer evaluation: routing accuracy, response quality, orchestration	Coming Soon...
1.11	LangGraph Deep Agents	Deep Agents with built-in planning (write_todos / read_todos) File system context management (read, write, edit files) Sub-agent delegation via task() tool Evaluating Deep Agent outputs with mlflow.genai.evaluate() Auto-tracing with mlflow.langchain.autolog()	Coming Soon...
1.12	CrewAI Multi-Agent Orchestration	Role-based agents with CrewAI (role, goal, backstory) Custom tools: query_disaster_database and search_fema_policies Hierarchical crew with manager agent delegating to specialists Evaluating crew outputs with mlflow.genai.evaluate() Auto-tracing with mlflow.crewai.autolog()	Coming Soon...

🎓 Learning Outcomes

After completing this tutorial, you will be able to:

Core Skills

• ✅ Set up MLflow for GenAI development
• ✅ Track LLM experiments systematically
• ✅ Implement comprehensive tracing
• ✅ Debug GenAI applications effectively
• ✅ Manage prompts with version control
• ✅ Build RAG systems

Advanced Capabilities

• ✅ Cost tracking and optimization
• ✅ Performance analysis and debugging
• ✅ Multi-framework integration
• ✅ Hierarchical trace creation
• ✅ Custom span instrumentation
• ✅ Agent workflow tracing

🚀 Getting Started

This project uses UV for dependency management.

1. Install UV (if not already installed)

curl -LsSf https://astral.sh/uv/install.sh | sh

2. Install Dependencies

uv sync

3. Configure API Keys Create a .env file in the tutorials directory:

OPENAI_API_KEY=your-api-key-here
MLFLOW_TRACKING_URI=http://localhost:5000

4. Start Jupyter

uv run jupyter notebook

5. Start MLflow UI (in a separate terminal)

uv run mlflow ui --port 5000

6. Open Browser Navigate to http://localhost:5000

📋 Prerequisites

• Python 3.10+
• UV package manager
• OpenAI API key (or Databricks Workspace)
• Basic understanding of Python and LLMs

🎯 Learning Objectives

By the end of all tutorials, you will:

• ✅ Understand MLflow's core GenAI components
• ✅ Track and trace LLM experiments systematically
• ✅ Implement comprehensive tracing for observability
• ✅ Debug GenAI applications using trace visualizations and MLflow Assistant
• ✅ Manage prompts with version control and Prompt Registery
• ✅ Evaluate an agent using MLflow predefined judges, custom and integrated judges from DeepEval and RAGAS
• ✅ Build end-to-end RAG applications
• ✅ Build and evaluate multi-agent orchestration systems

📂 Directory Structure

mlflow-genai-tutorial-1/
├── 01_setup_and_introduction.ipynb
├── 02_experiment_tracking.ipynb
├── 03_introduction_to_tracing.ipynb     
├── 04_manual_tracing_advanced.ipynb     
├── 05_prompt_management.ipynb           
├── 06_framework_integrations.ipynb      
├── 07_evaluating_agents.ipynb
├── 08_prompt_optimization.ipynb
├── 09_complete_rag_application.ipynb
├── 10_multi_agent_supervisor.ipynb
├── 11_deep_agents_langgraph.ipynb
├── 12_crewai_multi_agent.ipynb
├── .env                                 (create this yourself and use the template env_template)
└── README.md

🔗 Resources

• MLflow Documentation
• MLflow GenAI Guide
• MLflow GitHub
• MLflow Community

💡 Tips

• Keep the MLflow UI open while working through notebooks
• Experiment with different parameter values
• Compare runs in the UI to understand trade-offs
• Tag runs for easy organization
• Track costs from the beginning

❓ Troubleshooting

Issue: MLflow UI won't start

# Try a different port
uv run mlflow ui --port 5001

Issue: API key not recognized

# Restart Jupyter kernel after adding to .env
# Or set manually:
import os
os.environ["OPENAI_API_KEY"] = "your-key"

Issue: Module not found

# Sync dependencies with UV
uv sync

# Or install specific package
uv add mlflow openai python-dotenv

📝 License

This tutorial series is provided as educational content for learning MLflow's GenAI capabilities.

---

📝 Feedback & Contributions

• Found an issue? Open a GitHub issue
• Have suggestions? Submit a pull request
• Want to share? Tag us on social media
• Questions? Check the MLflow community

---

Authors: Jules (Databricks Developer Relations) + Claude Code Date: Feburary 2026 MLflow Version: 3.9.0+