feat(data-warehouse): implement cody import source#70709
Conversation
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
🤖 CI report✅ Bundle size — no changeUncompressed size of every built Total: 64.77 MiB · no change No file changed by more than 1000 B. Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report ✅ Eager graph — within budgetHow much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy
🟢 Largest files eagerly shipped from
|
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 762 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 281.3 KiB | ../node_modules/.pnpm/posthog-js@1.402.2/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 235.5 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 222.9 KiB | ../node_modules/.pnpm/posthog-js@1.402.2/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.8 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 92.7 KiB | ../packages/quill/packages/quill/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Dist folder size — 🔺 +1.1 KiB (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1312.45 MiB · 🔺 +1.1 KiB (+0.0%)
⚠️ Backend coverage — 99.0% of changed backend lines covered — 3 uncovered
🧪 Backend test coverage
Patch coverage — changed backend lines (products + core): ████████████████████ 99.0% (456 / 459)
| File | Patch | Uncovered changed lines |
|---|---|---|
products/warehouse_sources/backend/temporal/data_imports/sources/cody/cody.py |
97.9% | 157, 215–216 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29452506325 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
demo |
███████████░░░░░░░░░ 56.2% |
1,497 / 2,663 |
tasks |
█████████████░░░░░░░ 67.4% |
25,560 / 37,895 |
signals |
████████████████░░░░ 79.1% |
19,037 / 24,073 |
data_modeling |
████████████████░░░░ 80.0% |
4,834 / 6,045 |
cdp |
████████████████░░░░ 80.7% |
3,118 / 3,864 |
notebooks |
█████████████████░░░ 84.3% |
6,343 / 7,520 |
agent_platform |
█████████████████░░░ 84.7% |
3,273 / 3,862 |
cohorts |
█████████████████░░░ 86.1% |
4,022 / 4,671 |
actions |
█████████████████░░░ 86.6% |
717 / 828 |
product_tours |
█████████████████░░░ 87.5% |
1,266 / 1,447 |
exports |
██████████████████░░ 88.3% |
6,891 / 7,800 |
conversations |
██████████████████░░ 88.9% |
16,129 / 18,133 |
dashboards |
██████████████████░░ 89.1% |
5,719 / 6,418 |
mcp_analytics |
██████████████████░░ 89.1% |
2,502 / 2,807 |
error_tracking |
██████████████████░░ 89.6% |
9,718 / 10,852 |
alerts |
██████████████████░░ 89.9% |
3,638 / 4,046 |
engineering_analytics |
██████████████████░░ 90.1% |
5,105 / 5,665 |
streamlit_apps |
██████████████████░░ 90.4% |
2,499 / 2,764 |
slack_app |
██████████████████░░ 90.6% |
9,511 / 10,503 |
marketing_analytics |
██████████████████░░ 90.8% |
11,514 / 12,684 |
product_analytics |
██████████████████░░ 91.1% |
5,599 / 6,143 |
data_warehouse |
██████████████████░░ 92.1% |
18,133 / 19,683 |
workflows |
██████████████████░░ 92.4% |
5,148 / 5,574 |
web_analytics |
███████████████████░ 92.7% |
13,624 / 14,691 |
ai_observability |
███████████████████░ 92.8% |
14,868 / 16,019 |
surveys |
███████████████████░ 92.9% |
5,660 / 6,094 |
posthog_ai |
███████████████████░ 93.2% |
1,322 / 1,418 |
approvals |
███████████████████░ 93.3% |
3,395 / 3,640 |
reminders |
███████████████████░ 93.4% |
468 / 501 |
early_access_features |
███████████████████░ 93.8% |
848 / 904 |
endpoints |
███████████████████░ 94.1% |
8,606 / 9,143 |
skills |
███████████████████░ 94.4% |
2,827 / 2,995 |
revenue_analytics |
███████████████████░ 94.5% |
3,598 / 3,809 |
review_hog |
███████████████████░ 94.6% |
6,529 / 6,902 |
logs |
███████████████████░ 95.3% |
9,528 / 9,994 |
experiments |
███████████████████░ 95.6% |
24,139 / 25,256 |
replay_vision |
███████████████████░ 95.7% |
13,337 / 13,935 |
annotations |
███████████████████░ 96.2% |
732 / 761 |
warehouse_sources |
███████████████████░ 96.2% |
226,044 / 234,948 |
feature_flags |
███████████████████░ 96.3% |
16,002 / 16,625 |
user_interviews |
███████████████████░ 96.4% |
2,242 / 2,325 |
data_catalog |
███████████████████░ 97.1% |
2,035 / 2,095 |
customer_analytics |
███████████████████░ 97.2% |
7,480 / 7,698 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 2 · PR risk: 0/10 |
|
The checkout and review tools still resolve |
8bdd182 to
fa3b301
Compare
Generated-By: PostHog Code Task-Id: 7fbb7004-2eda-4ea6-a07c-b49b0345cfee
Fetch report CSVs with stream=True and parse them incrementally via a TextIOWrapper over response.raw, so memory stays bounded by the 5000-row chunk size instead of the full report body. Close errored streamed responses before retrying so connections are released. Generated-By: PostHog Code Task-Id: 7fbb7004-2eda-4ea6-a07c-b49b0345cfee
validate_credentials only inspects the probe's status code, so request it with stream=True and close the response without reading the body. Prevents an authenticated user pointed at a large Sourcegraph tenant from making the API worker buffer the full per-user CSV response on repeated credential checks. Generated-By: PostHog Code Task-Id: aa7996fb-7426-40f4-8046-5adae33df079
Remove unreleasedSource=True so the finished source is visible in the connector catalog, and drop any test asserting the hidden state.
fa3b301 to
174ae58
Compare
2 participants
Problem
The
codydata warehouse source exists only as a scaffolded stub (registered withunreleasedSource=True, no fields, no sync logic). Teams using Sourcegraph Cody (Enterprise) can't pull their AI coding assistant usage analytics into PostHog.Changes
Implements the Cody source end to end against the Sourcegraph Analytics API (
analytics.sourcegraph.com, Bearer token auth, CSV report responses):usage_by_user,usage_by_user_month,usage_by_user_day,usage_by_user_day_client_language(the API's default granularity, default-on) from/api/reports/by-user-client-date, pluscredits(default-on) from/api/credits.ResumableSourcewith calendar-monthstartDate/endDatewindows for the two day-grain reports: the date filter is server-side, so a resumed job picks up at the checkpointed window instead of re-downloading history. Aggregate reports and credits are a single all-time request (windowing an aggregate would change its meaning).primary_keys=Noneand no partitioning.make_tracked_session()with token redaction, redirects off, tenacity retries on 429/5xx only.validate_credentialsprobes a one-dayby_userreport;get_non_retryable_errorsstops syncs on 401/403. Instance URLs are normalized (pastedhttps://host/becomes the bare host the API expects).connection_host_fields=["instance_url"]so retargeting the instance requires re-entering the token.lists_tables_without_credentials=Truefor the public docs table list, official Cody SVG icon (from Sourcegraph's Apache-2.0cody-public-snapshotrepo), SOURCES.md moved to Implemented.unreleasedSource=TruewithreleaseStatus=ALPHAper the rollout plan for this source.User-facing doc: PostHog/posthog.com#18455 (
docsUrlmatches thecodyslug).Note
generated_configs.pywas regenerated, but I applied only the Cody hunk: a full regeneration also rewrote unrelated sources (e.g. it turnedStripeAuthMethodConfig's fields required, which would break parsing configs that carry only one auth option's fields). The checked-in file has drifted from whatgenerate_source_configscurrently emits - worth a separate look.How did you test this code?
tests/test_cody.py(transport) andtests/test_cody_source.py(source class): 55 tests covering month-window construction (gaps/overlap/clipping would double-count or drop days), instance URL normalization, CSV header normalization and malformed-row handling, JSON/CSV sniffing for credits, credential validation status mapping (transient errors not conflated with bad credentials), retry behavior, resume-from-checkpoint (earlier windows not re-fetched), checkpoint-after-yield semantics, and clear-state-on-completion. No existing test exercised any of these paths since the source was a stub.test_source_categories.py+test_generated_configs.py+ cody tests, 1605 passed.ruff check --fix,ruff format, andhogli ci:preflight --fix(0 failures).manage.py audit_source_docsagainst a posthog.com checkout with the new doc reports no Cody issues (pre-existing failures for other sources remain).Automatic notifications
Docs update
Doc added in PostHog/posthog.com#18455.
🤖 Agent context
Autonomy: Fully autonomous
/implementing-warehouse-sources,/writing-tests,/documenting-warehouse-sources.ResumableSourcewith calendar-month windows over a single unbounded fetch (bounded responses, graceful-shutdown resume); shipped full refresh everywhere rather than guessing incremental cursor column names from a third-party integration's renamed schema; sniffed the credits content type instead of assuming CSV; kept the coarser report granularities default-off since they're derivable from the detailed report.startDate/endDate(server filter is real but the cursor column name is unverifiable), synthesizing hash primary keys (full refresh doesn't merge, so they'd add nothing), committing the fullgenerate_source_configsoutput (unrelated semantic regressions, see note above).Created with PostHog Code