Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
* @temporalio/sdk
/langgraph_plugin/ @temporalio/sdk @temporalio/ai-sdk
Comment thread
DABH marked this conversation as resolved.

# SDK & Nexus own the README, pyproject.toml, and uv.lock
/README.md @temporalio/sdk @temporalio/nexus
Expand All @@ -16,11 +15,13 @@
# The AI SDK team owns the AI integration samples and their tests. We add
# @temporalio/sdk too, so the SDK team can continue to manage repo-wide concerns.
/google_adk_agents/ @temporalio/sdk @temporalio/ai-sdk
/langfuse_tracing/ @temporalio/sdk @temporalio/ai-sdk
/langgraph_plugin/ @temporalio/sdk @temporalio/ai-sdk
/langsmith_tracing/ @temporalio/sdk @temporalio/ai-sdk
/openai_agents/ @temporalio/sdk @temporalio/ai-sdk
/strands_plugin/ @temporalio/sdk @temporalio/ai-sdk
/tests/google_adk_agents/ @temporalio/sdk @temporalio/ai-sdk
/tests/langfuse_tracing/ @temporalio/sdk @temporalio/ai-sdk
/tests/langgraph_plugin/ @temporalio/sdk @temporalio/ai-sdk
/tests/langsmith_tracing/ @temporalio/sdk @temporalio/ai-sdk
/tests/strands_plugin/ @temporalio/sdk @temporalio/ai-sdk
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@ __pycache__
.mypy_cache/
**/client.key
**/client.pem
.env
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ Some examples require extra dependencies. See each sample's directory for specif
* [hello_standalone_nexus](hello_standalone_nexus) - Use Nexus Operations without using a workflow.
* [hello_standalone_activity](hello_standalone_activity) - Use activities without using a workflow.
* [lambda_worker](lambda_worker) - Run a Temporal Worker inside an AWS Lambda function.
* [langfuse_tracing](langfuse_tracing) - Trace Temporal workflows in Langfuse with the OpenTelemetry plugin and OTLP export.
* [langgraph_plugin](langgraph_plugin) - Run LangGraph workflows as durable Temporal workflows (Graph API and Functional API).
* [langsmith_tracing](langsmith_tracing) - Trace Temporal workflows with LangSmith via the LangSmith plugin.
* [message_passing/introduction](message_passing/introduction/) - Introduction to queries, signals, and updates.
Expand Down
27 changes: 27 additions & 0 deletions langfuse_tracing/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Copy to .env and adjust. Load with: set -a; source langfuse_tracing/.env; set +a

# Langfuse — these defaults match the headless-init values baked into
# langfuse_tracing/langfuse/docker-compose.yml (local demo stack).
LANGFUSE_HOST=http://localhost:3000
LANGFUSE_PUBLIC_KEY=pk-lf-temporal-demo-0000
LANGFUSE_SECRET_KEY=sk-lf-temporal-demo-0000
# Used only for the trace link the starter prints; set to your project ID
# when pointing at your own Langfuse instance or Langfuse Cloud.
LANGFUSE_PROJECT_ID=langfuse-tracing-demo

# LLM — any OpenAI-compatible endpoint works.
OPENAI_API_KEY=sk-...
MODEL_CLASSIFY=gpt-4o-mini
MODEL_DRAFT=gpt-4o-mini
# To use a local OpenAI-compatible gateway (e.g. a LiteLLM proxy) instead:
# OPENAI_BASE_URL=http://localhost:4000/v1
# OPENAI_API_KEY=<gateway key>
# MODEL_CLASSIFY=<gateway model alias>
# MODEL_DRAFT=<gateway model alias>

# LLM span instrumentation flavor: openinference (default) or openai-v2.
# See README for the trade-offs.
LLM_INSTRUMENTATION=openinference
# Required only for LLM_INSTRUMENTATION=openai-v2 content capture:
# OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
# OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only
153 changes: 153 additions & 0 deletions langfuse_tracing/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
# Langfuse Tracing

This sample shows the recommended way to get Temporal workflow traces into
[Langfuse](https://langfuse.com/): Temporal's
[`OpenTelemetryPlugin`](https://python.temporal.io/temporalio.contrib.opentelemetry.html)
plus a standard OTLP/HTTP exporter pointed at Langfuse's native OpenTelemetry
endpoint. No Langfuse SDK or Langfuse-specific plugin is involved, workflow
code stays deterministic and sandboxed, and traces are correctly nested,
correctly typed, and duplicate-free across replay and worker restarts.

Contents:

- **[ticket_triage/](ticket_triage/)** — the recommended pattern: an LLM
ticket-triage workflow (two LLM activities, one plain activity, one human
approval delivered as a workflow update).
- **[verify_trace.py](verify_trace.py)** — checks a trace via the Langfuse
public API: whole-tree equality, observation types, token usage, and
no-duplicates.
- **[langfuse/docker-compose.yml](langfuse/docker-compose.yml)** — pinned
self-hosted Langfuse with org/project/API keys provisioned headlessly.
- **[telemetry.py](telemetry.py)** — the OpenTelemetry wiring (replay-safe
tracer provider, OTLP exporter with Langfuse auth, LLM instrumentation).

## Prerequisites

- Docker (for Langfuse), a local Temporal server
(`temporal server start-dev`), and `uv`.
- An OpenAI-compatible LLM endpoint: either a real `OPENAI_API_KEY`, or any
OpenAI-compatible gateway via `OPENAI_BASE_URL`.

## Run it

```bash
# 1. Start Langfuse (first pull takes a few minutes)
cd langfuse_tracing/langfuse
docker compose up -d
curl -sf http://localhost:3000/api/public/health # repeat until {"status":"OK",...}
# UI: http://localhost:3000 — login demo@temporal.io / langfuse-demo-pw-1

# 2. Install dependencies and set environment (repo root)
cd ../..
uv sync --group langfuse-tracing
cp langfuse_tracing/.env.example langfuse_tracing/.env # edit the LLM settings
set -a; source langfuse_tracing/.env; set +a

# 3. Run the sample (two terminals, same environment)
uv run python -m langfuse_tracing.ticket_triage.worker
uv run python -m langfuse_tracing.ticket_triage.starter

# 4. Verify the trace through the Langfuse API (uses the printed trace ID)
uv run python -m langfuse_tracing.verify_trace --trace-id <printed trace id>
```

The starter prints a direct link to the trace in the Langfuse UI. You should
see one trace shaped like this (types as Langfuse derives them):

```
ticket-triage SPAN (root; session/user/tags)
├─ StartWorkflow:TicketTriageWorkflow SPAN
│ └─ RunWorkflow:TicketTriageWorkflow SPAN
│ ├─ triage SPAN (custom span from workflow code)
│ │ ├─ StartActivity:classify_ticket → RunActivity:classify_ticket
│ │ │ └─ ChatCompletion GENERATION (model, tokens, cost)
│ │ └─ StartActivity:lookup_account → RunActivity:lookup_account
│ └─ StartActivity:draft_reply → RunActivity:draft_reply
│ └─ ChatCompletion GENERATION
└─ StartWorkflowUpdate:approve SPAN
├─ ValidateUpdate:approve SPAN
└─ HandleUpdate:approve SPAN
```

## Prove the replay-safety claims

Durable execution means workflow code re-executes (replays) on worker
restarts and cache evictions. These two experiments show the trace is
unaffected — rerun `verify_trace.py` after each and it still passes with the
identical tree:

```bash
# Replay stress: disable the workflow cache so EVERY workflow task replays
# the workflow from the start of history.
uv run python -m langfuse_tracing.ticket_triage.worker --replay-stress
uv run python -m langfuse_tracing.ticket_triage.starter

# Worker restart mid-workflow: the starter waits 20s before sending the
# approval; kill the worker while the workflow is parked, start a new one,
# and watch the workflow (and its trace) complete cleanly.
uv run python -m langfuse_tracing.ticket_triage.starter --pause-before-approval 20
# ... ctrl+c the worker, then start it again in another terminal
```

## Where spans come from

| Span | Emitted by | Where it runs |
|---|---|---|
| `ticket-triage` (root) + `langfuse.*` trace attributes | starter code | starter |
| `StartWorkflow:*`, `StartWorkflowUpdate:*` | `OpenTelemetryPlugin` | starter (client side) |
| `RunWorkflow:*`, `StartActivity:*`, `ValidateUpdate:*`, `HandleUpdate:*` | `OpenTelemetryPlugin` | worker (workflow) |
| `triage` | plain OpenTelemetry API in workflow code | worker (workflow) |
| `RunActivity:*` | `OpenTelemetryPlugin` | worker (activity) |
| `ChatCompletion` / `chat <model>` GENERATIONs | OpenAI auto-instrumentation | worker (activity) |

## Where tracing works

| Location | Works? | Notes |
|---|---|---|
| Activity bodies | ✅ | Plain OpenTelemetry + any auto-instrumentation, no restrictions. This is where LLM calls (and their GENERATION spans) belong. |
| Workflow bodies | ✅ | Plain OpenTelemetry APIs are replay-safe under the plugin: deterministic span IDs, no re-export on replay. Spans export when they end; the `RunWorkflow` span exports when the run completes. |
| Signal/query/update handlers | ✅ | Handled by the plugin automatically (`HandleUpdate:*` etc.). |
| Client / starter code | ✅ | Standard OpenTelemetry; put Langfuse trace-level attributes on your root span. |

## LLM instrumentation flavors

`LLM_INSTRUMENTATION` selects how OpenAI calls are instrumented (both are
verified against Langfuse by this sample):

| | `openinference` (default) | `openai-v2` |
|---|---|---|
| Package | `openinference-instrumentation-openai` | `opentelemetry-instrumentation-openai-v2` |
| Semantic conventions | OpenInference | OpenTelemetry GenAI (`gen_ai.*`) |
| GENERATION type, model, token usage, cost | ✅ | ✅ |
| Prompt/completion content | ✅ by default | Requires `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` and `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only` |
| GENERATION span name | `ChatCompletion` | `chat <model>` |

## Operational notes

- Langfuse's OTLP endpoint is HTTP-only — this sample uses
`opentelemetry-exporter-otlp-proto-http` (the gRPC exporter will not work).
- Short-lived processes must flush: the starter and worker call
`force_flush()` on exit (see `telemetry.py`).
- One workflow run = one workflow ID = one Langfuse trace/session; never
reuse workflow IDs across runs.
- `OTEL_SDK_DISABLED=true` turns off export without code changes.
- Ingestion is asynchronous; `verify_trace.py` polls until the trace is
stable.

## Tests

`tests/langfuse_tracing/` runs without Langfuse, Docker, or an LLM: mocked
activities, an in-memory span exporter, a worker with the workflow cache
disabled, whole-tree span assertions, and a `Replayer` pass asserting that
replaying the finished workflow's history emits zero new spans.

```bash
uv run --group langfuse-tracing pytest tests/langfuse_tracing -v
```

## Using this outside samples-python

The sample is self-contained: copy the `langfuse_tracing/` directory, change
the absolute imports (`langfuse_tracing.ticket_triage.activities` →
`ticket_triage.activities` or similar), and install the dependencies listed
under `langfuse-tracing` in this repo's `pyproject.toml`.
Empty file added langfuse_tracing/__init__.py
Empty file.
180 changes: 180 additions & 0 deletions langfuse_tracing/langfuse/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
# Self-hosted Langfuse for the langfuse_tracing sample.
#
# Based on the upstream https://github.com/langfuse/langfuse/blob/v3.224.1/docker-compose.yml
# with these changes for a local demo:
# - Langfuse images pinned to a specific release instead of the floating `:3` tag.
# - Only the Langfuse UI/API (port 3000) is published on the host. Postgres, ClickHouse,
# Redis, and MinIO stay on the compose network (upstream binds them to 127.0.0.1, which
# can collide with other local stacks, e.g. another Postgres on 5432).
# - Headless initialization (LANGFUSE_INIT_*) provisions the org, project, API keys, and
# login user on first start, so the sample works without any UI clicking. Idempotent.
# - All secrets below are throwaway demo values. Do not reuse them outside local demos.
#
# Usage:
# docker compose up -d
# curl -sf http://localhost:3000/api/public/health # poll until 200
# UI login: demo@temporal.io / langfuse-demo-pw-1
name: langfuse-demo
services:
langfuse-worker:
image: docker.io/langfuse/langfuse-worker:3.224.1
restart: always
depends_on: &langfuse-depends-on
postgres:
condition: service_healthy
minio:
condition: service_healthy
redis:
condition: service_healthy
clickhouse:
condition: service_healthy
environment: &langfuse-worker-env
NEXTAUTH_URL: ${NEXTAUTH_URL:-http://localhost:3000}
DATABASE_URL: ${DATABASE_URL:-postgresql://postgres:postgres@postgres:5432/postgres}
SALT: ${SALT:-langfuse-demo-salt}
ENCRYPTION_KEY: ${ENCRYPTION_KEY:-0000000000000000000000000000000000000000000000000000000000000000}
TELEMETRY_ENABLED: ${TELEMETRY_ENABLED:-false}
LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES: ${LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES:-false}
CLICKHOUSE_MIGRATION_URL: ${CLICKHOUSE_MIGRATION_URL:-clickhouse://clickhouse:9000}
CLICKHOUSE_URL: ${CLICKHOUSE_URL:-http://clickhouse:8123}
CLICKHOUSE_USER: ${CLICKHOUSE_USER:-clickhouse}
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse}
CLICKHOUSE_CLUSTER_ENABLED: ${CLICKHOUSE_CLUSTER_ENABLED:-false}
LANGFUSE_USE_AZURE_BLOB: ${LANGFUSE_USE_AZURE_BLOB:-false}
LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE: ${LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE:-false}
LANGFUSE_OCI_AUTH_TYPE: ${LANGFUSE_OCI_AUTH_TYPE:-workload_identity}
LANGFUSE_S3_EVENT_UPLOAD_BUCKET: ${LANGFUSE_S3_EVENT_UPLOAD_BUCKET:-langfuse}
LANGFUSE_S3_EVENT_UPLOAD_REGION: ${LANGFUSE_S3_EVENT_UPLOAD_REGION:-auto}
LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID:-minio}
LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY:-miniosecret}
LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: ${LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT:-http://minio:9000}
LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE: ${LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE:-true}
LANGFUSE_S3_EVENT_UPLOAD_PREFIX: ${LANGFUSE_S3_EVENT_UPLOAD_PREFIX:-events/}
LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: ${LANGFUSE_S3_MEDIA_UPLOAD_BUCKET:-langfuse}
LANGFUSE_S3_MEDIA_UPLOAD_REGION: ${LANGFUSE_S3_MEDIA_UPLOAD_REGION:-auto}
LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID:-minio}
LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY:-miniosecret}
LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT: ${LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT:-http://localhost:9090}
LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE: ${LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE:-true}
LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: ${LANGFUSE_S3_MEDIA_UPLOAD_PREFIX:-media/}
LANGFUSE_S3_BATCH_EXPORT_ENABLED: ${LANGFUSE_S3_BATCH_EXPORT_ENABLED:-false}
LANGFUSE_S3_BATCH_EXPORT_BUCKET: ${LANGFUSE_S3_BATCH_EXPORT_BUCKET:-langfuse}
LANGFUSE_S3_BATCH_EXPORT_PREFIX: ${LANGFUSE_S3_BATCH_EXPORT_PREFIX:-exports/}
LANGFUSE_S3_BATCH_EXPORT_REGION: ${LANGFUSE_S3_BATCH_EXPORT_REGION:-auto}
LANGFUSE_S3_BATCH_EXPORT_ENDPOINT: ${LANGFUSE_S3_BATCH_EXPORT_ENDPOINT:-http://minio:9000}
LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT: ${LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT:-http://localhost:9090}
LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID: ${LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID:-minio}
LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY: ${LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY:-miniosecret}
LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE: ${LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE:-true}
LANGFUSE_INGESTION_QUEUE_DELAY_MS: ${LANGFUSE_INGESTION_QUEUE_DELAY_MS:-}
LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS: ${LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS:-}
REDIS_HOST: ${REDIS_HOST:-redis}
REDIS_PORT: ${REDIS_PORT:-6379}
REDIS_AUTH: ${REDIS_AUTH:-myredissecret}
LANGFUSE_BULLMQ_SKIP_REDIS_VERSION_CHECK: ${LANGFUSE_BULLMQ_SKIP_REDIS_VERSION_CHECK:-false}
REDIS_TLS_ENABLED: ${REDIS_TLS_ENABLED:-false}
REDIS_TLS_CA: ${REDIS_TLS_CA:-/certs/ca.crt}
REDIS_TLS_CERT: ${REDIS_TLS_CERT:-/certs/redis.crt}
REDIS_TLS_KEY: ${REDIS_TLS_KEY:-/certs/redis.key}
EMAIL_FROM_ADDRESS: ${EMAIL_FROM_ADDRESS:-}
SMTP_CONNECTION_URL: ${SMTP_CONNECTION_URL:-}

langfuse-web:
image: docker.io/langfuse/langfuse:3.224.1
restart: always
depends_on: *langfuse-depends-on
ports:
- 3000:3000
environment:
<<: *langfuse-worker-env
NEXTAUTH_SECRET: ${NEXTAUTH_SECRET:-langfuse-demo-nextauth-secret}
# Headless initialization: org, project, API keys, and login user are created on
# first startup. These keys must match langfuse_tracing/.env.example.
LANGFUSE_INIT_ORG_ID: ${LANGFUSE_INIT_ORG_ID:-temporal-demo}
LANGFUSE_INIT_ORG_NAME: ${LANGFUSE_INIT_ORG_NAME:-Temporal Demo}
LANGFUSE_INIT_PROJECT_ID: ${LANGFUSE_INIT_PROJECT_ID:-langfuse-tracing-demo}
LANGFUSE_INIT_PROJECT_NAME: ${LANGFUSE_INIT_PROJECT_NAME:-langfuse-tracing-demo}
LANGFUSE_INIT_PROJECT_PUBLIC_KEY: ${LANGFUSE_INIT_PROJECT_PUBLIC_KEY:-pk-lf-temporal-demo-0000}
LANGFUSE_INIT_PROJECT_SECRET_KEY: ${LANGFUSE_INIT_PROJECT_SECRET_KEY:-sk-lf-temporal-demo-0000}
LANGFUSE_INIT_USER_EMAIL: ${LANGFUSE_INIT_USER_EMAIL:-demo@temporal.io}
LANGFUSE_INIT_USER_NAME: ${LANGFUSE_INIT_USER_NAME:-Demo User}
LANGFUSE_INIT_USER_PASSWORD: ${LANGFUSE_INIT_USER_PASSWORD:-langfuse-demo-pw-1}

clickhouse:
image: docker.io/clickhouse/clickhouse-server
restart: always
user: "101:101"
environment:
CLICKHOUSE_DB: default
CLICKHOUSE_USER: ${CLICKHOUSE_USER:-clickhouse}
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse}
volumes:
- langfuse_clickhouse_data:/var/lib/clickhouse
- langfuse_clickhouse_logs:/var/log/clickhouse-server
healthcheck:
test: wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1
interval: 5s
timeout: 5s
retries: 10
start_period: 1s

minio:
image: cgr.dev/chainguard/minio
restart: always
entrypoint: sh
# create the 'langfuse' bucket before starting the service
command: -c 'mkdir -p /data/langfuse && minio server --address ":9000" --console-address ":9001" /data'
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER:-minio}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-miniosecret}
volumes:
- langfuse_minio_data:/data
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 1s
timeout: 5s
retries: 5
start_period: 1s

redis:
image: docker.io/redis:7
restart: always
command: >
--requirepass ${REDIS_AUTH:-myredissecret}
--maxmemory-policy noeviction
volumes:
- langfuse_redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 3s
timeout: 10s
retries: 10

postgres:
image: docker.io/postgres:${POSTGRES_VERSION:-17}
restart: always
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 3s
timeout: 3s
retries: 10
environment:
POSTGRES_USER: ${POSTGRES_USER:-postgres}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-postgres}
POSTGRES_DB: ${POSTGRES_DB:-postgres}
TZ: UTC
PGTZ: UTC
volumes:
- langfuse_postgres_data:/var/lib/postgresql/data

volumes:
langfuse_postgres_data:
driver: local
langfuse_clickhouse_data:
driver: local
langfuse_clickhouse_logs:
driver: local
langfuse_minio_data:
driver: local
langfuse_redis_data:
driver: local
Loading
Loading