feat(agent): export cloud run telemetry over OTLP#3478
Draft
tatoalo wants to merge 11 commits into
Draft
Conversation
…OTLP Adds OtelRunTelemetry, a SessionLogWriter sink that ships an allowlisted metadata subset of the session log (run/turn/tool lifecycle, usage, errors; never message content or tool arguments) to PostHog Logs, and, when a traces URL is configured, builds one APM trace per run (task_run root span, a turn span per prompt, a tool_call:<kind> span per tool call) with trace/span ids stamped on log records so Logs and APM cross-link. Resource attributes carry run_id/task_id/team_id/user_id/distinct_id so cloud runs are filterable per user in the Logs UI. Configured via POSTHOG_AGENT_OTEL_LOGS_URL/_TOKEN (+ optional POSTHOG_AGENT_OTEL_TRACES_URL), deliberately not standard OTEL_* names so OTel SDKs in user code running in the sandbox never auto-export into the telemetry project. Telemetry stays off unless the logs pair is set, flushes on session cleanup/terminal errors, and can never break session log persistence (sink failures are isolated). Replaces the unwired otel-log-writer whose default endpoint (/i/v1/agent-logs) does not exist in the ingest service, and removes the dead AgentConfig.otelTransport field. Generated-By: PostHog Code Task-Id: 0c511836-2180-455a-9b58-45df2a0661ec
Generated-By: PostHog Code Task-Id: 0c511836-2180-455a-9b58-45df2a0661ec
…emetry Addresses three review findings: - The sandbox never emits task_complete for successful runs (the terminal "completed" status is decided by the workflow outside), so the root span was ending with unset status on success. The trace builder now marks the root span OK when a turn ends cleanly with end_turn; a run error still always wins and a turn completion arriving after an error cannot flip the status back. - reportFatalError (uncaught exception / unhandled rejection) marked the run failed via the API but never touched telemetry, leaving hard crashes invisible in the new pipeline. It now mirrors an error record (error_source=agent_server_crash) and shuts telemetry down, ending the root span as errored and flushing before the process dies. - The README "Writing logs" section still described the removed otel-log-writer as the preferred full-transcript path to the nonexistent /i/v1/agent-logs endpoint; rewritten to match the sink-based metadata export. Generated-By: PostHog Code Task-Id: 0c511836-2180-455a-9b58-45df2a0661ec
Generated-By: PostHog Code Task-Id: 0c511836-2180-455a-9b58-45df2a0661ec
…est turn Addresses two external review findings: - _posthog/console records are no longer exported to PostHog Logs. Console lines are free-text agent-server diagnostics that interpolate arbitrary data - the user-message handler logs a 100-char prompt preview and the extension-notification handler stringifies params into that channel - so exporting them verbatim violated the metadata-only allowlist. They remain in the S3 session log and the event-ingest stream. Regression tests assert console entries (including an interpolated prompt preview) never map to an exported record. - The task_run root span status is now resolved at shutdown from the LATEST turn outcome instead of being set sticky-OK on the first clean end_turn. A multi-turn run whose final turn is cancelled/refused no longer reports OK; a run error still always wins. The root span only exports at end, so per-turn status writes were cosmetic anyway. Stacked on posthog-code/agent-run-otel-telemetry (pure fast-forward). Generated-By: PostHog Code Task-Id: 0c511836-2180-455a-9b58-45df2a0661ec
… fixes Generated-By: PostHog Code Task-Id: 0c511836-2180-455a-9b58-45df2a0661ec
Addresses four review findings on the run telemetry export: - _posthog/error records and the root span no longer carry the raw error string: it is free text that can embed prompt or repo content (exception paths, provider errors). Exported signal is now the generic "run error" body plus error_source/stop_reason attributes; the full message stays in the session log and the task run's error_message. - Telemetry flush/shutdown use Promise.allSettled so logs and traces complete independently, and both batch processors cap exports at 5s (exportTimeoutMillis) instead of the SDK's 30s default - a rejecting or hanging traces endpoint can no longer starve log delivery or delay session cleanup. - A run error now closes still-open tool spans as ERROR with tool_status=interrupted, so APM never shows a healthy-looking active tool under a failed run. - OtelTransportConfig/AgentConfig.otelTransport are restored as @deprecated ignored stubs: @posthog/agent is published, so removing exported types is an API break reserved for a major. Generated-By: PostHog Code Task-Id: 0c511836-2180-455a-9b58-45df2a0661ec
The task detail routes redirected away on any 404 from the detail query, even when a cached or list copy of the task exists. Optimistic and cloud-pending tasks are not returnable by the API yet (see the route loader comments), so the unconditional redirect kicked users off tasks they had just created. The detail query still always runs (so a stale cached copy converges on the server's latest run state), but a 404/missing result only triggers the redirect when there is no usable cached task to render. Generated-By: PostHog Code Task-Id: 0c511836-2180-455a-9b58-45df2a0661ec
Local verification of the telemetry branch showed logs and the turn span landing in ClickHouse but never the task_run root span, even after the run reached completed and extra delay. Root cause: the root span only ended in cleanupSession, which is only reachable via SIGTERM or the close command - and sandbox teardown never delivers SIGTERM to agent-server, because it is an exec'd process (docker stop signals only the container's PID 1; Modal terminate is immediate). The root span was still open when the process was killed, so it never exported. The turn span survived only because it ended mid-run and the 2s batch exporter got it out. Fix: end telemetry eagerly at the run's in-process terminal points instead of relying on teardown: - finalizeRunTelemetry: full telemetry shutdown (ends the root span with the resolved status and drains both exporters) when a background run's initial or resume prompt settles - the run is over in-sandbox at that point; the workflow marks the terminal status and destroys the sandbox right after. - signalTaskComplete: upgrade the terminal-failure flush to a full shutdown, after the error mirror is appended, so failed runs export a root span with ERROR status. - cleanupSession remains the path for interactive close. Known limitation, now documented: an interactive session ended by hard teardown (e.g. inactivity timeout) loses the root span; its turn/tool spans and logs still assemble under the same trace id. Tests: order assertion that the error mirror lands before the terminal shutdown, and a parameterized check that finalizeRunTelemetry fires for background runs only. The full agent-server + telemetry suites pass (161). Stacked on posthog-code/agent-run-otel-telemetry (pure fast-forward). Generated-By: PostHog Code Task-Id: 0c511836-2180-455a-9b58-45df2a0661ec
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
This was referenced Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Cloud-task execution details were available only through per-run session logs, which made it difficult to query run health, timing, usage, and failures across tasks and users. The previous OTel path was not wired to a valid ingest endpoint, and terminal cloud-run hydration could also lose persisted state or redirect on transient task-detail failures.
Changes
service.name=posthog-code-agent.task_runroot, per-turn spans, and per-tool-kind spans. Logs carry the active trace and span IDs for correlation.