feat(tasks): mirror scout run logs into posthog logs#71094
Conversation
Scout (and, later, other task-origin) run logs live only as JSONL blobs in object storage, which makes sampling and eyeballing runs painful. Mirror entries persisted via TaskRun.append_log to a PostHog project's Logs product as OTLP/HTTP log records, gated by settings and scoped to signals_scout origin by default. Export runs on a Celery task so it never blocks or breaks the log write.
![hex-security-app[bot]](https://avatars.githubusercontent.com/in/2665233?s=60&v=4){kind=small}
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if response.status_code >= 400: | ||
| logger.warning( | ||
| "task_run.otlp_log_export_rejected", | ||
| run_id=run_id, | ||
| status_code=response.status_code, | ||
| body=response.text[:500], |
There was a problem hiding this comment.
When the endpoint returns 429 or 5xx, requests.post completes normally, so autoretry_for does not run. The task logs the rejection and permanently drops that mirrored batch even though a later attempt could succeed.
Rule Used: When implementing retry logic, include comprehensi... (source)
Learned From
PostHog/posthog#32651
There was a problem hiding this comment.
Obsolete after 8b1b456 — the HTTP export and Celery task were removed entirely; mirroring is now plain structured stdout logging picked up by the existing cluster OTel collector, which handles delivery and retries.
| origin_product: str, | ||
| ) -> dict[str, Any] | None: | ||
| """Build an OTLP `ExportLogsServiceRequest` JSON body from persisted log entries.""" | ||
| records = [_log_record(entry, run_id=run_id) for entry in entries if isinstance(entry, dict)] |
There was a problem hiding this comment.
append_log accepts an unbounded entry list, but this builds one request for the entire list. Even with each body capped at 32,000 characters, enough valid records can exceed the ingestion endpoint's 2 MB limit; the endpoint then rejects the whole batch and none of its records reach Logs.
| try: | ||
| parsed = datetime.fromisoformat(timestamp) | ||
| except ValueError: | ||
| parsed = None | ||
| if parsed is None: | ||
| parsed = timezone.now() |
There was a problem hiding this comment.
When a valid ACP timestamp has no UTC offset, fromisoformat returns a naive datetime and timestamp() interprets it in the worker's local timezone. On a non-UTC worker this silently shifts the OTLP timestamp, so records appear at the wrong time and can be ordered incorrectly.
There was a problem hiding this comment.
Obsolete after 8b1b456 — the mirror no longer sets an OTLP timestamp. Records are stamped by the collector at emission time (entries are mirrored as they arrive), and the original ACP ISO timestamp is preserved verbatim in the entry_timestamp attribute.
| TASK_RUN_LOGS_OTLP_ENDPOINT: str | None = get_from_env("TASK_RUN_LOGS_OTLP_ENDPOINT", None, optional=True) | ||
| TASK_RUN_LOGS_OTLP_TOKEN: str | None = get_from_env("TASK_RUN_LOGS_OTLP_TOKEN", None, optional=True) |
There was a problem hiding this comment.
These new environment values also need entries in the posthog/charts deployment configuration, with the token populated through posthog/secrets. Without that wiring, production workers receive neither complete setting and forwarding remains silently disabled.
Rule Used: When a new secret or environment variable is added... (source)
There was a problem hiding this comment.
Obsolete after 8b1b456 — the endpoint/token settings were removed, so no deployment secret wiring is needed. The one remaining setting (TASK_RUN_LOGS_MIRROR_ORIGIN_PRODUCTS) has a safe default and needs no charts change; delivery reuses the collector pipeline that already ships container stdout.
|
hey @andrewm4894, working on these two atm: PostHog/code#3478 ; #71100, maybe we can unify |
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| origin_product: str, | ||
| ) -> dict[str, Any] | None: | ||
| """Build an OTLP `ExportLogsServiceRequest` JSON body from persisted log entries.""" | ||
| records = [_log_record(entry, run_id=run_id) for entry in entries if isinstance(entry, dict)] |
There was a problem hiding this comment.
High: Cross-team task logs are exported to one project
Every eligible team's raw task entries are converted for export, but the transport authenticates all records with the same global destination-project token. A user with access to that project can therefore read agent messages, analytical tool output, console records, and sandbox stdout/stderr from other enrolled teams; adding the source team_id as an attribute does not preserve tenant isolation. Route exports to a destination owned by the source team, or restrict this feature to an explicit source-team allowlist whose members are authorized for the destination and redact sensitive record types before forwarding.
There was a problem hiding this comment.
Addressed in 8b1b456 — the global destination-token transport is gone. Mirrored lines go through the same per-cluster stdout→OTel-collector pipeline that already ships all application container logs into the region's internal PostHog project (staff-only), so no customer-facing project can read another team's content and nothing crosses regions. Scout content reaching the internal project is the deliberate, kill-switchable dogfooding goal.
PR overviewThis PR adds backend support for mirroring scout task run logs into PostHog-style log ingestion, including conversion/export of run log entries and mirroring stdout records from task runs. The PR currently has open security concerns around tenant isolation and log volume control. The most significant issue is that raw task logs from multiple teams can be exported into a single destination project, allowing users with access to that project to view other teams’ agent messages, tool output, and sandbox logs. There is also an unbounded log amplification path where a task writer can submit many small entries and generate excessive output, potentially tying up workers and collectors. No issues have been fixed or addressed yet, so the current posture remains high risk. Open issues (2)
Fixed/addressed: 0 · PR risk: 8/10 |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: df2e6523b0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| response = requests.post( | ||
| endpoint, | ||
| json=payload, | ||
| headers={"Authorization": f"Bearer {token}"}, | ||
| timeout=OTLP_EXPORT_TIMEOUT_SECONDS, |
There was a problem hiding this comment.
When TASK_RUN_LOGS_OTLP_* is enabled in Cloud, this POST uses one global project token for every allowlisted signals_scout run, so raw task-run log bodies from all source teams land in the same target Logs project rather than the team that owns the run. Those bodies are built from agent/user messages, thoughts, console output, and sandbox output, so this can expose customer-derived scout logs to anyone with access to the dogfooding project. Please restrict forwarding to explicitly internal/test teams or export only scrubbed aggregate metadata with a server-side tenant/region allowlist.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in 8b1b456 — the shared-token OTLP POST was removed. Mirroring now emits stdout lines that the per-cluster collector ships into the region's internal PostHog project, the same place all application container logs already land, so tenant content stays within the region and within staff-only internal observability. Scoped to signals_scout and kill-switchable via TASK_RUN_LOGS_MIRROR_ORIGIN_PRODUCTS.
| forward_task_run_logs_to_posthog_logs.delay( | ||
| entries=entries, | ||
| team_id=self.team_id, |
There was a problem hiding this comment.
When forwarding is enabled, this sends the original entries through Celery before build_otlp_payload truncates record bodies. append_log accepts arbitrary dict payloads and emit_sandbox_output can include large stdout/stderr, so a single large tool/sandbox output or catch-up batch can create a multi-MB Redis/Celery message in the request path and still produce an oversized OTLP request. Please enqueue only compact, already-truncated chunks, or queue a run/S3 reference and let the worker read and size-bound the export.
Useful? React with 👍 / 👎.
| if response.status_code >= 400: | ||
| logger.warning( |
There was a problem hiding this comment.
If the Logs endpoint returns a transient 429/5xx, requests.post does not raise, so this branch only logs the rejection and lets Celery mark the task successful; autoretry_for=(requests.RequestException,) never runs and the mirrored logs are permanently dropped. Please raise or call self.retry for retryable statuses while leaving permanent 4xx responses non-retryable.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Obsolete after 8b1b456 — the requests/Celery transport was removed entirely. Delivery and retry are now the existing OTel collector's responsibility; the app just writes stdout.
| @shared_task( | ||
| ignore_result=True, | ||
| autoretry_for=(requests.RequestException,), | ||
| retry_backoff=True, | ||
| max_retries=3, |
There was a problem hiding this comment.
When forwarding is enabled for active scout runs, every persisted append enqueues an HTTP export task with no queue, rate limit, or coalescing, so Celery uses the default celery queue. Scout runs can emit frequent progress, console, and sandbox-output events, and this new mirror path can therefore compete with unrelated default-queue product work with many low-priority log POSTs. Please batch/coalesce per run or route these exports to a dedicated low-priority queue.
Useful? React with 👍 / 👎.
|
|
||
| try: | ||
| origin_product = self.task.origin_product | ||
| if not otlp_forwarding_enabled(origin_product): |
There was a problem hiding this comment.
When forwarding is configured, this treats origin_product as the only proof that a run is a scout run, but the public task write serializer accepts origin_product and create_task simply persists it when provided. A normal task creator can therefore mark a task as signals_scout and have arbitrary appended logs forwarded through this dogfooding path, even though the change claims to scope export to scout runs. Please gate on a server-owned scout-run relationship/internal flag rather than the user-controllable task origin alone.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fair point, and it still applies to the reworked stdout version — origin_product is user-settable, so a task creator can opt their task's logs into this path. With the transport gone the impact is narrower though: they'd only expose their own task's log entries to PostHog's internal staff project (no cross-tenant read path), and volume is bounded by the 8k body cap plus the 200-entries-per-call budget added in 3139655, with TASK_RUN_LOGS_MIRROR_ORIGIN_PRODUCTS as a kill switch. Gating on the server-owned SignalScoutRun bridge would need a cross-product lookup from tasks on every append and doesn't change that exposure story, so keeping the origin allowlist for this dogfooding v1.
The per-cluster OTel collector already ships all container stdout into the region's internal PostHog project's Logs product, parsing JSON keys into attributes and request_id into a trace id. Emitting structured stdout lines at append_log gets scout run logs into PostHog Logs with no transport, credentials, or Celery hop of its own.
andrewm4894
changed the title
ah sorry - was just poking around - didnt realize you actually had stuff under way - moving this back to draft (was just marking ready to get bot reviews going) |
origin_product is user-settable on task creation, so a hostile append must not be able to flood stdout and the log collector with an unbounded entry list.
|
Thanks — I verified the current diff no longer contains the OTLP transport or destination-project token. The mirror now emits through the existing stdout collector path, so the specific cross-project export finding on this thread is addressed; the remaining internal, region-local observability of scout content is the stated dogfooding design rather than the customer-project disclosure path I reported. |
| origin_product: str, | ||
| ) -> None: | ||
| """Emit one structured stdout log line per persisted entry.""" | ||
| for entry in entries: |
There was a problem hiding this comment.
Medium: Unbounded log amplification
An authenticated user with task:write can append a request containing millions of tiny dictionaries to a team-visible scout run. This loop emits one stdout record per element, allowing a single request to expand into gigabytes of log output and occupy the web worker and collector; enforce a small entry-count or total-output limit before iterating.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8b1b456dae
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| TASK_RUN_LOGS_MIRROR_ORIGIN_PRODUCTS: list[str] = get_list( | ||
| os.getenv("TASK_RUN_LOGS_MIRROR_ORIGIN_PRODUCTS", "signals_scout") | ||
| ) |
There was a problem hiding this comment.
Fresh evidence in this revision is that the old OTLP env gate is gone: with this default, every unset Cloud deployment mirrors signals_scout run entries, and mirror_entries emits the unredacted agent/message/tool/sandbox body to structlog stdout that the docs say is collected into a region-level internal Logs project. Those run logs were previously only in the task's team-scoped object-storage/API path, so scout runs that include project data, MCP query results, or user text can cross tenant boundaries by default; keep this opt-in for explicitly safe teams/destinations or redact bodies before logging.
Useful? React with 👍 / 👎.
| origin_product: str, | ||
| ) -> None: | ||
| """Emit one structured stdout log line per persisted entry.""" | ||
| for entry in entries: |
There was a problem hiding this comment.
When an allowlisted run posts a large catch-up append_log batch, this loop emits one structured stdout line per entry synchronously before append_task_run_log can heartbeat the workflow, while the append-log serializer only rejects empty lists and does not cap entry count. Under log-driver/collector backpressure or a huge malformed batch, the new mirror can tie up the request/worker path and flood stdout even though the docstring calls it fire-and-forget; cap/sample the mirrored entries or move mirroring off this path.
Useful? React with 👍 / 👎.
|
|
||
| object_storage.write(self.log_url, content) | ||
|
|
||
| self._mirror_logs_to_posthog_logs(entries) |
There was a problem hiding this comment.
When sandbox_event_ingest_enabled is true for a run, the workflow skips _relay_sandbox_events and the ingest handler writes accepted agent events directly to TaskRunRedisStream (event_ingest.py), so this append-log hook only mirrors backend-generated calls like progress/console messages and misses the actual session/update/sandbox output stream. In that rollout or state-override context, allowlisted signals_scout runs will show incomplete Logs traces; route the event-ingest writer or another shared stream persistence point through the mirror too.
Useful? React with 👍 / 👎.
| fields: dict[str, Any] = { | ||
| # `request_id` becomes the record's trace id in the collector, grouping the run. | ||
| "request_id": run_id, | ||
| "task_run_id": run_id, |
There was a problem hiding this comment.
When these stdout lines are ingested by the repo collector path, request_id is only parsed as a JSON attribute: otel-collector-config.dev.yaml has the JSON parser but no transform that copies it into the OTLP trace_id, and Logs stores/filtering by trace uses the first-class trace_id column. In local dev, and any deployment using the same collector contract, all mirrored run entries therefore keep the zero/empty trace id and the advertised “one run groups as a trace” workflow does not work; emit real OTLP log records with traceId or add the collector transform with this change.
Useful? React with 👍 / 👎.
2 participants
Problem
Signals scout runs execute on the tasks sandbox infra, and their run logs only exist as JSONL blobs in object storage (
TaskRun.append_log→tasks/logs/team_{team}/task_{task}/run_{run}.jsonl). That makes it hard to sample and eyeball scout runs across projects – you have to find the run, presign the S3 object, and read raw JSONL.We also want to dogfood the PostHog Logs product on our own agent workloads.
Changes
Persisted task-run log entries are now mirrored into the PostHog Logs product, scoped to scout runs for now.
The mechanism is deliberately boring: the per-cluster OTel collector already tails all container stdout and ships JSON log lines into the region's internal PostHog project's Logs, parsing each JSON key into a queryable log attribute,
levelinto severity, andrequest_idinto a trace id. So mirroring needs no transport, credentials, or infra of its own –TaskRun.append_logjust emits one structured stdout line per persisted entry. Locally the dev collector config does the same into your dev logs project.products/tasks/backend/logic/services/run_log_mirror.py– translates each ACP JSONL entry into atask_run_logstructlog line:_posthog/error→ error, console level passthrough)task_run_id/task_id/team_id/origin_product/acp_method/acp_session_updatebecome log attributes, andrequest_id(the run uuid) becomes the record's trace id, so one run groups as one trace and can be pulled up with a singletask_run_idattribute filterTaskRun.append_logcalls the mirror after the S3 write, guarded so any failure is logged and never breaks the run.TASK_RUN_LOGS_MIRROR_ORIGIN_PRODUCTS(inposthog/settings/temporal.py, next to the otherTASKS_*sandbox settings) – which task origins to mirror. Defaults tosignals_scoutonly; widening it later turns the same pipe on for other task origins, empty disables.docs/internal/sandboxes-setup-guide.md.How did you test this code?
Automated tests in
products/tasks/backend/tests/test_run_log_mirror.py(15 cases, all passing locally):append_loggating tests – catch mirroring for non-allowlisted origins (volume blowout), mirroring when disabled, and a mirror failure breaking the S3 log writeI also verified the delivery path empirically: the prod OTel collector daemonset (charts repo,
argocd/otel-collector) ships all container stdout to the internal project's Logs with JSON-key→attribute parsing andrequest_id→trace-id mapping, andtemporal-worker-tasks-agentcontainer logs are already arriving there today. I have not watched a live scout run flow through end to end yet.Automatic notifications
Docs update
Covered in this PR (
docs/internal/sandboxes-setup-guide.md).🤖 Agent context
Autonomy: Human-driven (agent-assisted)
/writing-tests.TaskRun.append_log(the single choke point every persisted entry passes through, covering both Docker and Modal sandbox paths and the web REST push path) and gate by an origin-product allowlist defaulting to scouts, so expansion to other task origins is a config change.