Add Google Cloud OpenTelemetry plugin - #2467
Draft
eamsden wants to merge 3 commits into
Draft
Conversation
lennessyy
added a commit
to temporalio/documentation
that referenced
this pull request
Jul 29, 2026
Every SDK that supports Serverless Workers now has a GCP Cloud Run entry
in its sidebar, so Cloud Run no longer looks Python-only.
New SDK pages (docs/develop/{go,typescript}/workers/serverless-workers/cloud-run.mdx):
- Create a versioned Worker. Cloud Run runs a standard long-lived Worker,
so the only addition is the Worker Deployment Version and a versioning
behavior.
- Configure the Temporal connection through envconfig.
- Keep Activities safe across scale-in with Activity Heartbeats.
- Add observability. The Go page documents the contrib/gcp OpenTelemetry
plugin; the TypeScript page links to general observability, since no
GCP plugin exists for that SDK yet.
Deploy guide: filled the four TODO(lenny) tabs with the Go and TypeScript
Worker setup and Dockerfiles. The Go image builds a static binary into
distroless; the TypeScript image stays on glibc, because the SDK's Rust
core is incompatible with musl.
Snipsync: added sdk-go as an origin. The two Go OpenTelemetry snippets come
from contrib/gcp/e2e/main.go via temporalio/sdk-go#2504, which targets the
plugin branch in temporalio/sdk-go#2467. Both are unmerged, so the source
links 404 until they land, matching how the Python page already references
samples-python#329.
APIs verified against source rather than memory:
- worker.DeploymentOptions exposes UseVersioning, Version, and
DefaultVersioningBehavior; registration panics when a Version is set and
no behavior is given (internal/internal_worker.go).
- setWorkflowOptions takes options first and the Workflow function second
(packages/test/src/deployment-versioning-v1).
- loadClientConnectConfig returns connectionOptions and namespace, which
feed NativeConnection.connect and Worker.create.
Verification: clean yarn build passes with onBrokenLinks and onBrokenAnchors
set to throw. Vale CI-scoped reports no errors or warnings.
lennessyy
added a commit
to temporalio/documentation
that referenced
this pull request
Jul 30, 2026
* add clound run auto scaling
* Fix build errors after merging main
- Fix sidebar doc ID: index -> serverless-workers (matches frontmatter id)
- Fix MDX anchor syntax in autoscaling.mdx: {#anchor} -> {/* #anchor */}
* Restructure encyclopedia to one subpage per compute provider
Instead of a shared autoscaling page split by provider, each compute
provider gets its own page covering autoscaling, versioning, and
lifecycle. The index page keeps section headings but links to the
provider-specific pages for details.
- Create aws-lambda.mdx with Lambda autoscaling + versioning content
- Create cloud-run.mdx as a stub for upcoming Cloud Run content
- Delete autoscaling.mdx (content moved to aws-lambda.mdx)
- Update index.mdx sections to link to provider pages
- Update sidebars
* docs(serverless-workers): add GCP Cloud Run guides; generalize shared pages
- Add Python SDK page for wiring OpenTelemetry on a Cloud Run worker pool
(Snipsync-sourced from the samples-python gcp_open_telemetry sample).
- Add the GCP Cloud Run deploy guide and encyclopedia page (autoscaling with
immediate + periodic re-sizing, scale-in, lifecycle, no_sync_quiet_ms).
- Make the shared Serverless Workers encyclopedia index provider-neutral: nest
Compute providers under the intro, stub Autoscaling/Worker lifecycle to the
per-provider pages, and remove Lambda-only invocation-model wording.
- Move the Lambda-specific Worker lifecycle and long-running-Activity tuning
content onto the AWS Lambda encyclopedia page; repoint inbound anchors.
- Restructure the AWS Lambda deploy pages into an aws-lambda/ subdirectory with
redirects; update sidebars.
- Deploy-guide fixes from an end-to-end dogfooding pass: invoker SA naming,
--namespace on set-current-version/workflow start, note the confirmation
prompt, and complete the logs command.
* docs(serverless-workers): add Go and TypeScript Cloud Run SDK pages
Every SDK that supports Serverless Workers now has a GCP Cloud Run entry
in its sidebar, so Cloud Run no longer looks Python-only.
New SDK pages (docs/develop/{go,typescript}/workers/serverless-workers/cloud-run.mdx):
- Create a versioned Worker. Cloud Run runs a standard long-lived Worker,
so the only addition is the Worker Deployment Version and a versioning
behavior.
- Configure the Temporal connection through envconfig.
- Keep Activities safe across scale-in with Activity Heartbeats.
- Add observability. The Go page documents the contrib/gcp OpenTelemetry
plugin; the TypeScript page links to general observability, since no
GCP plugin exists for that SDK yet.
Deploy guide: filled the four TODO(lenny) tabs with the Go and TypeScript
Worker setup and Dockerfiles. The Go image builds a static binary into
distroless; the TypeScript image stays on glibc, because the SDK's Rust
core is incompatible with musl.
Snipsync: added sdk-go as an origin. The two Go OpenTelemetry snippets come
from contrib/gcp/e2e/main.go via temporalio/sdk-go#2504, which targets the
plugin branch in temporalio/sdk-go#2467. Both are unmerged, so the source
links 404 until they land, matching how the Python page already references
samples-python#329.
APIs verified against source rather than memory:
- worker.DeploymentOptions exposes UseVersioning, Version, and
DefaultVersioningBehavior; registration panics when a Version is set and
no behavior is given (internal/internal_worker.go).
- setWorkflowOptions takes options first and the Workflow function second
(packages/test/src/deployment-versioning-v1).
- loadClientConnectConfig returns connectionOptions and namespace, which
feed NativeConnection.connect and Worker.create.
Verification: clean yarn build passes with onBrokenLinks and onBrokenAnchors
set to throw. Vale CI-scoped reports no errors or warnings.
* docs(serverless-workers): document the runner service account for Cloud Run
The Cloud Run Worker Pool runs as a service account separate from the
invoker, and the invoker needs actAs on it or scaling fails. The guide
referenced it only as a bare <RUNNER_SERVICE_ACCOUNT> placeholder with no
explanation of where the value comes from.
- Step 2 gains a "Choose a runner service account" section: what the
identity is, why it is not the invoker, and the default Compute Engine
service account fallback (which holds roles/editor).
- Step 3 documents runner_service_account_email, added to the Terraform
module today in temporalio/terraform-modules#14. The module grants the
invoker roles/iam.serviceAccountUser on the runner so Cloud Run lets it
attach that identity when scaling the pool.
Terminology follows the Terraform module: "runner service account" and
runner_service_account_email, rather than "runtime".
Also fixes a broken link. https://cloud.google.com/run/docs/worker-pools
404s; the correct target is /run/docs/resource-model#worker-pools. It was
on all three SDK Cloud Run pages. Note that onBrokenLinks only covers
internal links, so external URLs get no build-time checking.
Verification: clean yarn build passes; Vale CI-scoped reports no errors.
* docs(serverless-workers): correct the Cloud Run WCI dynamic config
The scaling algorithm was copied from the AWS Lambda page. Cloud Run needs
rate-based, not no-sync, and the pairing the page documented is rejected by
the server rather than merely suboptimal:
- gcpCloudRunComputeProvider.LaunchStrategy() returns worker-set.
- scalingAlgorithmNoSync.CompatibleLaunchStrategies() returns invoke only.
- scalingAlgorithmRateBased.CompatibleLaunchStrategies() returns worker-set.
A live deployment agrees: describe-version on a working Cloud Run Worker
Deployment Version reports scaler type rate-based.
Also documents the two optional GCP impersonation-chain keys, which have no
AWS equivalent and were therefore missing entirely:
workercontroller.compute_providers.gcp.intermediary_service_accounts and
workercontroller.compute_providers.gcp.first_delegate_as_base.
The compute provider value gcp-cloud-run was already correct, confirmed
against ComputeProviderTypeGCPCloudRun. The AWS Lambda page is also correct
as written and is unchanged; no-sync is right for an invoke provider.
Source: temporalio/temporal-auto-scaled-workers, wci/client/config.go,
wci/workflow/iface/spec.go, and the compute_provider and scaling_algorithm
packages.
Verification: clean yarn build passes; Vale CI-scoped is clean.
* docs(serverless-workers): document off-GCP credentials for self-hosted Cloud Run
The "Outside GCP" guidance named Workload Identity Federation without saying
how the server picks it up, which read as though federation were the only
supported path.
Workload Identity Federation stays the recommendation. Adds that its
credential configuration file is supplied through GOOGLE_APPLICATION_CREDENTIALS,
and that the same variable accepts a service account key file, with Google's
guidance against long-lived keys.
A key file works because the impersonation chain's base credential is plain
ADC: gcp_cloudrun.go calls impersonate.CredentialsTokenSource with no base
option unless first_delegate_as_base is set, so ADC resolves
GOOGLE_APPLICATION_CREDENTIALS normally. Nothing in the provider excludes a
key file.
Resolves the last TODO(lenny) on this page. Self-hosted remains undogfooded.
* docs(serverless-workers): split troubleshooting by compute provider
The troubleshooting page was titled generically but covered only AWS
Lambda. Convert it to a folder so each compute provider gets its own
page, and add the GCP Cloud Run page.
- index.mdx keeps the /troubleshooting/serverless-workers URL and routes
to the provider pages, so no redirect is needed.
- aws-lambda.mdx is the previous content, unchanged apart from
frontmatter and a cross-link.
- cloud-run.mdx is new: triage on the pool's manualInstanceCount,
scalingMode, and lastModifier annotations, then scale-up failures
(validation, current version, Task Queue binding, max_count), Worker
execution failures, and Activities interrupted by scale-in.
Note that Validate Connection on Cloud Run only reads the Worker Pool,
so an invoker with run.workerPools.get but not run.workerPools.update
validates successfully and then fails to scale.
Update the four inbound links, including the deep anchor from the AWS
Lambda deploy guide.
* docs(serverless-workers): give Cloud Run its own release stage
The release badge read from a single `serverlessWorkers` key, so every
Serverless Workers page showed the same stage regardless of compute
provider. Add `serverlessWorkersCloudRun` and point the seven Cloud Run
pages at it, so Cloud Run's stage can move independently of any other
provider.
Rewrite the Cloud Run notice to name the provider instead of the feature
as a whole, and point readers to AWS Lambda as the other supported
provider.
The self-hosted page keeps its shorter notice: Pre-release features are
open to self-hosted users, so the access-request line does not apply.
Lambda pages are untouched here and handled separately.
* docs(serverless-workers): address review feedback on Cloud Run pages
- Name the example Worker Pool per Worker Deployment Version
(my-temporal-worker-pool-build-1) and state the one-pool-per-version
recommendation, since a pool runs a single image and a version pins a
single build.
- Say explicitly that Cloud Run Worker Pools are their own resource type,
distinct from Services and Jobs.
- Add `Versioning` to the Vale heading exceptions. "Worker Versioning" is
the feature name and is already the heading on seven pages on main, so
the rule was contradicting the style guide.
- Lowercase "versioning behavior" in a heading to match how the docs
write it in prose.
* docs(serverless-workers): write the Cloud Run Worker Versioning section
Replaces the "Content coming soon" stub, mirroring the AWS Lambda page.
The point Cloud Run needs and Lambda does not: the compute configuration
names only a project, region, and Worker Pool, with no revision field.
Temporal runs whichever revision the pool currently serves, so the pool
is the unit that maps to a Worker Deployment Version, and deploying a
new image into an existing pool swaps the code under a live version the
same way an unqualified Lambda ARN does.
* docs(serverless-workers): stop explaining Cloud Run by comparison to Lambda
Trim the Cloud Run Worker Versioning section to the action and a short
reason, and drop the Lambda framing there and in four other spots. Each
page should stand on its own.
Remaining Lambda references on Cloud Run pages are navigation links, not
comparisons.
* docs(serverless-workers): drop no_sync_quiet_ms from Cloud Run pages
The scaling and lifecycle parameter is not available in Pre-release, so
remove it from the encyclopedia, deployment, and troubleshooting pages.
Activity Heartbeats remain the documented answer for Activities that can
be interrupted by scale-in.
The encyclopedia Lifecycle section still describes the scale-in waits in
behavioral terms without naming the parameter.
* docs(serverless-workers): address bug bash feedback on Cloud Run
Name the Temporal Cloud API key explicitly so it is not mistaken for a
GCP credential, in the deployment guide, the Go and TypeScript pages, and
the troubleshooting page.
State what the runner service account needs. Cloud Run collects stdout
and stderr through its own infrastructure and the service agent pulls the
image, so the runner has no baseline role. It needs secretAccessor on the
secrets it mounts, logWriter only when the Worker writes through the
Cloud Logging API, and access to whatever else the code calls.
Point --set-env-vars at the environment configuration reference instead
of claiming TEMPORAL_TASK_QUEUE overrides the value set in code. That
variable is not part of environment configuration; the Worker code reads
it directly.
Add Terraform to the prerequisites, and stop attributing the environment
configuration package to a serverless Worker package that Cloud Run does
not use.
* docs(serverless-workers): clarify the two Cloud Run service accounts
Name both accounts in full throughout the deployment guide, and add an
admonition at the end of the IAM step contrasting them: what each one is
for, who creates it, and where its value goes.
The runner section called the same account "runner" in its heading and
"runtime" in its first sentence, which is part of why the two accounts
read as interchangeable. It now says runner service account and keeps
runtime as the descriptive term.
Stop implying the runner service account has to be created. It can be an
account you already have, or the project default when the flag is unset.
Mark the Terraform variables required or optional, add the two that were
missing, and say that impersonator_service_account_emails is specific to
your Temporal Cloud account so it has to come from the UI template.
* docs(serverless-workers): fix TEMPORAL_TASK_QUEUE precedence on Lambda
The table said the environment variable overrides the value set in code.
The precedence runs the other way in both packages: the env var
pre-populates the Task Queue before the configure callback runs, so a
Task Queue set in the callback wins. Neither one set is an error.
Verified in sdk-go contrib/aws/lambdaworker/runworker.go and
sdk-typescript packages/lambda-worker/src/lambda-worker.ts.
* docs(serverless-workers): move the two-account admonition up
Place it directly after the paragraph that introduces the invoker service
account, so the distinction lands where the second account first appears
instead of at the end of the step.
Drop the paragraph's closing sentence about the runner service account,
which the admonition's first bullet now states.
* docs(serverless-workers): name the Access section in the Cloud Run UI path
* docs(serverless-workers): drop the Lambda pointer from Cloud Run notices
Lambda is in Pre-release alongside Cloud Run at launch, so pointing at it
from inside a Pre-release notice implies an availability it does not have.
* docs(serverless-workers): fix newcomer gaps in the deploy docs
The Serverless Workers deploy index described only the Lambda model, so a
reader choosing a provider learned that Temporal invokes the Worker per
Task and that nothing is long-lived, then hit the opposite on the Cloud
Run page. Name the shared flow, then say where the providers diverge.
Document Validate Connection on the Cloud Run guide, which only the
Lambda guide covered, including the caveat that it exercises the read and
not the update permission that scaling needs.
Stop calling a Cloud Run scale-up an invocation.
* docs(serverless-workers): bring Python to parity and fix the Evaluate page
The Python Cloud Run page covered only OpenTelemetry, so a Python reader
got no versioned Worker, connection, or scale-in guidance on their SDK
page. Add those three sections to match Go and TypeScript, using
ClientConfig.load_client_connect_config for the connection.
The Evaluate page described only the Lambda model, in the intro, the
autoscaling and cost sections, the persistent-connection caveat, and every
row of the long-lived comparison. Split each by provider, and list Cloud
Run alongside Lambda under supported providers.
Add the Pre-release notice the page was missing, and replace the Go-only
next step with a table of SDK and provider guides.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
contrib/gcpmodule with an OpenTelemetry client and worker plugin for Google Cloud Run.Why?
Temporal Go workers running on Google Cloud Run need an SDK-supported path for exporting Temporal metrics and traces through the standard local OTLP collector sidecar. Keeping Google resource detection and exporters in the collector avoids Google Cloud client dependencies and direct-export configuration in the worker process. Worker pools are the preferred Cloud Run deployment for continuously polling Temporal workers.
A one-second metric interval can interact poorly with a collector batch processor because the batch may combine multiple cumulative snapshots of each Temporal metric into one Google Monitoring request. The 60-second default prevents routine periodic exports from sharing a five-second batch and matches the upstream OpenTelemetry SDK default. Because a forced shutdown flush can still land in an open collector batch, the recommended Google Managed Prometheus metrics pipeline does not use a batch processor; trace batching remains independent.
This corresponds to temporalio/sdk-java#2955.
Google Cloud / Temporal Cloud E2E
Cloud Run worker-pool and Temporal Cloud E2E verified workflow and activity execution, the resolved service name and OTLP endpoint, the 60-second metrics default, metrics and trace export, GCP resource attribution, and deterministic flush/shutdown lifecycle behavior. The corrected unbatched metrics pipeline completed without duplicate-series, authentication, permission, or collector-configuration errors, and the test worker pool was returned to zero instances.
Checklist
Closes: N/A.
How was this tested:
go run . checkfrominternal/cmd/buildgo run . unit-testfrominternal/cmd/buildgo test -count=1 -race ./...incontrib/gcpgo vet ./...incontrib/gcperrcheck ./...andstaticcheck ./...incontrib/gcpAny docs updates needed?
contrib/gcp/README.mdwith usage, shutdown lifecycle, configuration precedence, collector-sidecar requirements, metric cadence, and resource-detection caveats.contrib/gcp/e2e/README.mdfor the reusable E2E harness and deterministic lifecycle check.Breaking changes?
No.
Server PR
None.