feat(data-warehouse): implement heroku import source#70713
Conversation
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
|
Companion doc PR: PostHog/posthog.com#18461 |
🤖 CI report✅ Bundle size — no changeUncompressed size of every built Total: 64.77 MiB · no change No file changed by more than 1000 B. Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report ✅ Eager graph — within budgetHow much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy
🟢 Largest files eagerly shipped from
|
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 762 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 281.3 KiB | ../node_modules/.pnpm/posthog-js@1.402.3/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 235.5 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 222.9 KiB | ../node_modules/.pnpm/posthog-js@1.402.3/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.8 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 92.7 KiB | ../packages/quill/packages/quill/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Dist folder size — 🔺 +690 B (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1312.48 MiB · 🔺 +690 B (+0.0%)
⚠️ Backend coverage — 98.0% of changed backend lines covered — 6 uncovered
🧪 Backend test coverage
Patch coverage — changed backend lines (products + core): ████████████████████ 98.0% (385 / 391)
| File | Patch | Uncovered changed lines |
|---|---|---|
products/warehouse_sources/backend/temporal/data_imports/sources/heroku/source.py |
94.9% | 75, 79 |
products/warehouse_sources/backend/temporal/data_imports/sources/heroku/heroku.py |
97.3% | 100, 187, 194 |
products/warehouse_sources/backend/temporal/data_imports/sources/heroku/tests/test_heroku_source.py |
98.6% | 115 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29457667443 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
demo |
███████████░░░░░░░░░ 56.2% |
1,497 / 2,663 |
tasks |
█████████████░░░░░░░ 67.4% |
25,560 / 37,895 |
signals |
████████████████░░░░ 79.1% |
19,031 / 24,073 |
data_modeling |
████████████████░░░░ 80.0% |
4,834 / 6,045 |
cdp |
████████████████░░░░ 80.7% |
3,118 / 3,864 |
notebooks |
█████████████████░░░ 84.3% |
6,343 / 7,520 |
agent_platform |
█████████████████░░░ 84.7% |
3,273 / 3,862 |
cohorts |
█████████████████░░░ 86.1% |
4,022 / 4,671 |
actions |
█████████████████░░░ 86.6% |
717 / 828 |
product_tours |
█████████████████░░░ 87.5% |
1,266 / 1,447 |
exports |
██████████████████░░ 88.3% |
6,891 / 7,800 |
conversations |
██████████████████░░ 88.9% |
16,129 / 18,133 |
dashboards |
██████████████████░░ 89.1% |
5,719 / 6,418 |
mcp_analytics |
██████████████████░░ 89.1% |
2,502 / 2,807 |
error_tracking |
██████████████████░░ 89.6% |
9,718 / 10,852 |
alerts |
██████████████████░░ 89.9% |
3,638 / 4,046 |
engineering_analytics |
██████████████████░░ 90.1% |
5,105 / 5,665 |
streamlit_apps |
██████████████████░░ 90.4% |
2,499 / 2,764 |
slack_app |
██████████████████░░ 90.6% |
9,511 / 10,503 |
marketing_analytics |
██████████████████░░ 90.8% |
11,514 / 12,684 |
product_analytics |
██████████████████░░ 91.1% |
5,599 / 6,143 |
data_warehouse |
██████████████████░░ 92.1% |
18,133 / 19,683 |
workflows |
██████████████████░░ 92.4% |
5,148 / 5,574 |
web_analytics |
███████████████████░ 92.7% |
13,624 / 14,691 |
ai_observability |
███████████████████░ 92.8% |
14,868 / 16,019 |
surveys |
███████████████████░ 92.9% |
5,687 / 6,120 |
posthog_ai |
███████████████████░ 93.2% |
1,322 / 1,418 |
approvals |
███████████████████░ 93.3% |
3,395 / 3,640 |
reminders |
███████████████████░ 93.4% |
468 / 501 |
early_access_features |
███████████████████░ 93.8% |
848 / 904 |
endpoints |
███████████████████░ 94.1% |
8,606 / 9,143 |
skills |
███████████████████░ 94.4% |
2,827 / 2,995 |
revenue_analytics |
███████████████████░ 94.5% |
3,598 / 3,809 |
review_hog |
███████████████████░ 94.6% |
6,532 / 6,905 |
logs |
███████████████████░ 95.3% |
9,528 / 9,994 |
experiments |
███████████████████░ 95.6% |
24,171 / 25,288 |
replay_vision |
███████████████████░ 95.7% |
13,354 / 13,952 |
annotations |
███████████████████░ 96.2% |
732 / 761 |
warehouse_sources |
███████████████████░ 96.2% |
230,633 / 239,669 |
feature_flags |
███████████████████░ 96.3% |
16,002 / 16,625 |
user_interviews |
███████████████████░ 96.4% |
2,242 / 2,325 |
data_catalog |
███████████████████░ 97.1% |
2,034 / 2,095 |
customer_analytics |
███████████████████░ 97.2% |
7,480 / 7,698 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 2 · PR risk: 0/10 |
|
Pushed 62ba49e addressing CI and review feedback:
The |
|
👋 Visual changes detected for this PR. Review and approve in PostHog Visual Review If these changes are unexpected, they may be caused by a flaky test or a broken snapshot on master. Don't approve — rerun the job or wait for a fix. |
Implements the Heroku warehouse source: 11 endpoints (apps, addons, builds, collaborators, domains, dynos, formation, invoices, pipelines, releases, teams) over the Platform API v3 with Range-header cursor pagination, per-app fan-out, resumable imports, canonical table descriptions, an icon, and tests. Generated-By: PostHog Code Task-Id: 7fee9ef6-177a-4459-96b3-def090128b08
Nulls capability URLs before rows are yielded (build/release output_stream_url, build source_blob.url, dyno attach_url) — these grant access without Heroku auth and must not land in the warehouse. Also passes response= to HTTPError in the test helper to satisfy the requests stubs. Generated-By: PostHog Code Task-Id: 7fee9ef6-177a-4459-96b3-def090128b08
Several Heroku endpoints return capability URLs carrying secrets (builds' `source_blob.url`, dynos' `rendezvous://.../secret` `attach_url`) that the name-based sample scrubbers can't recognise. `_redact_sensitive_fields` only scrubs the yielded rows, not the raw response the tracked transport would otherwise capture, so disable capture on the shared session. Generated-By: PostHog Code Task-Id: ba6dcc2f-13fe-4967-8c56-2a17a1cf1fa1
Remove unreleasedSource=True so the finished source is visible in the connector catalog, and drop any test asserting the hidden state.
9e4360d to
6580a5b
Compare
2 participants
Problem
Heroku was a scaffolded warehouse source with no sync logic. Heroku account data (deploy history, dyno formations, add-on usage, invoices) is useful warehouse material for engineering and finance analytics, and the Platform API v3 is stable and self-serve (any user can mint a long-lived API key).
Changes
Implements the Heroku source end to end, following the standard
source.py/settings.py/heroku.pysplit:apps,addons,builds,collaborators,domains,dynos,formation,invoices,pipelines,releases,teams. Endpoint catalog is declarative insettings.py.GET /accountprobe. 401/403 are registered as non-retryable errors.Rangerequest header and aNext-Rangeresponse cursor), so the transport uses a small custom paginator instead of a query-param cursor. Requests pinAccept: application/vnd.heroku+json; version=3and sort ascending byidwithmax=1000.ResumableSource): the verbatimNext-Rangecursor plus, for fan-out endpoints, a stable app-ID bookmark are persisted after each yielded page, so Temporal retries resume where they left off instead of restarting.builds,collaborators,domains,dynos,formation, andreleasesiterate the apps list and query each app's child endpoint. Heroku ids are globally unique UUIDs, soidstays a valid table-wide primary key. Apps deleted mid-sync (404) are skipped, and a per-list page cap guards against unbounded scans.make_tracked_session().lists_tables_without_credentials = Trueso the posthog.com doc renders the table catalog, and a Heroku SVG icon.SOURCES.mdupdated (heroku moved from Scaffolded to Implemented).The source stays behind
unreleasedSource=TruewithreleaseStatus=alphafor now.Note
Conservative choices made without live credentials. Endpoint shapes, Range attributes, and the 401 error format were verified against the live API and its published JSON hyper-schema, but paginated
Next-Rangeflows need an authenticated account. Two things follow from that: pagination handling follows the documented contract (cursor resent verbatim, termination whenNext-Rangeis absent) but hasn't run against a real multi-page account, and incremental sync was left off everywhere even though/appsadvertises a Range onupdated_at, since we couldn't smoke-test whether datetime ranges actually filter. Both are noted in code comments and are cheap follow-ups once someone runs a sync with a real account.The user-facing doc is written and consistent (
audit_source_docsshows no heroku findings); it lands separately in the posthog.com repo: PostHog/posthog.com PR to follow (linked in comments).How did you test this code?
hogli test products/warehouse_sources/backend/temporal/data_imports/sources/heroku/tests– 37 tests pass.Rangeheader (page-boundary skips), resume state saved before yield instead of after (data loss on crash), a deleted app failing the whole sync instead of being skipped, credential errors being retried forever, transient 429/5xx not being retried, and unbounded pagination when the cap regresses.supports_incrementalwithout a server-side filter,get_non_retryable_errorskeys drifting from the error stringsrequestsactually produces (verified against the live API), canonical descriptions drifting from the endpoint catalog, andsource_for_pipelineplumbing regressions.python manage.py generate_source_configsregeneratedHerokuSourceConfig; registry-wide suites (sources/tests/, 1551 tests) pass.ruff check/ruff formatclean;hogli ci:preflight --fixreports 0 failures.Automatic notifications
Docs update
Doc added to posthog.com (
contents/docs/cdp/sources/heroku.md) in a companion PR, following the documenting-warehouse-sources template.docsUrlmatches theherokuslug.🤖 Agent context
Autonomy: Fully autonomous
implementing-warehouse-sources,documenting-warehouse-sources,writing-tests.ResumableSourcepull-only rather than addingWebhookSource– Heroku webhooks are per-app (max 10 subscriptions each), so webhook management would need fan-out creation across every app plus drift handling for new apps, which isn't verifiable without credentials; left as a follow-up. Full refresh everywhere since the API has no server-side timestamp filters (Range-on-updated_atexists for/appsbut couldn't be verified).generated_configs.pywas regenerated then trimmed to the heroku hunk only, because the local environment lacks Stripe OAuth settings and the generator emitted an unrelated, environment-dependent Stripe diff.Created with PostHog Code