feat: add code graphs

[m43i]

Summary

Adds backend and API support for code graph ingestion and connector-backed repository graph sync. This PR intentionally excludes the connector UI; the remaining frontend changes are typed API helper functions for the new backend routes.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Refactoring (no functional changes)
• Documentation update
• CI/CD or infrastructure change

Changes Made

• Add code file parsing, repository graph extraction, code metadata, source validity, and relationship-key support.
• Add public repository URL ingestion via POST /graphs/:id/urls, including repository snapshot supersession and code workflow routing.
• Add GitHub/GitLab connector backend package, credential handling, provider repository/branch clients, webhook normalization, and connector tests.
• Add API routes for connector administration, connector installation flow, repository/branch discovery, repository graph creation, repository graph binding lookup, and manual repository sync.
• Add connector webhook ingestion at POST /connectors/webhooks/:provider and durable repository sync workflows for initial, incremental, and full-snapshot syncs.
• Add database schema/migrations for connectors, installations, webhook events, repository graph bindings, external file storage metadata, and source validity.
• Keep typed frontend API helpers in apps/frontend/lib/api/connectors.ts for the follow-up UI work; no connector pages/components are included in this PR.

API Surface

• GET /connectors
• POST /connectors/github/manifest/start
• GET /connectors/github/manifest/callback
• POST /connectors/gitlab
• PATCH /connectors/:id
• GET /connectors/:id/connect
• GET /connectors/github/install/callback
• GET /connectors/:id/installations
• GET /connectors/:id/repositories?installationId=...
• GET /connectors/:id/repositories/:repositoryId/branches?installationId=...
• POST /connectors/:id/repository-graphs
• GET /repository-graph-bindings/:id
• POST /repository-graph-bindings/:id/sync
• POST /connectors/webhooks/:provider
• POST /graphs/:id/urls

Access Model

• System admins create and patch connector definitions, including GitHub app manifests and GitLab connector credentials.
• Users with top-level graph creation access can start organization-scoped connector installation flows and create organization-owned repository graphs.
• Users with team graph creation access can start team-scoped installation flows and create team-owned repository graphs for matching team installations.
• Repository and branch listing requires access to the selected connector installation.
• Binding lookup follows graph view permissions; manual binding sync additionally requires owner management access and an enabled webhook binding.
• Provider webhooks are authenticated by connector secrets and enqueue sync only for matching active bindings.

Related Issues

Closes #

Testing

• I have tested these changes locally
• I have added/updated tests as appropriate
• All existing tests pass

Targeted tests cover graph extraction, connector clients, repository URL imports, file proxying, code manifests, repository sync workflows, migration compatibility, webhooks, API routes, and frontend API helpers.

Checklist

• My code follows the project's coding standards
• I have updated documentation as needed
• I have run make generate if SQL queries were modified (backend)
• I have updated barrel exports if components were added (frontend)

Screenshots (if applicable)

Connector UI is intentionally deferred to a frontend follow-up issue.

[greptile-apps]

Greptile Summary

This PR adds backend and API support for code graph ingestion via repository URLs (POST /graphs/:id/urls) and a full GitHub/GitLab connector system with durable webhook-driven sync workflows. A new packages/connectors package centralises provider clients, credential handling, and webhook normalisation; the packages/graph package gains code-file parsing, manifest extraction, and relationship-key support; and packages/db gains schema/migrations for connectors, installations, bindings, and source validity.

• All 15 previously flagged issues from the earlier review rounds were addressed, including: symlink traversal/broken-symlink handling in loadRepositoryFromUrl, behind/diverged/compare_timeout triggering a full-snapshot fallback instead of erroring, the webhook binding lookup being scoped to the verified connector via a join on connectorInstallationsTable, terminal workflow failure now calling markBindingFailed in the top-level catch, the manual-sync endpoint resetting bindings to "failed" on enqueue errors, and the URL-ingestion path passing retiredFileIds: result.supersededFileIds so superseded sources are expired.
• The deduplication key in POST /graphs/:id/urls now uses ${repository.url}:${checksum} (per-repository), fixing the cross-repository silent file loss from a prior review. The files_graph_active_key_idx index was promoted to UNIQUE (partial WHERE deleted = false) to close the idempotency gap in insertRepositoryFiles.

Confidence Score: 5/5

The change is safe to merge. All 15 issues surfaced in the earlier review rounds were addressed, and no new blocking defects were found.

The connector infrastructure, durable sync workflow, and URL ingestion path are all implemented correctly. The previously identified problems — symlink traversal, force-push/compare_timeout handling, unscoped webhook binding lookups, partial enqueue failures, binding stuck in syncing, and the single-file retry that invalidated unrelated sources — have all been properly resolved. The only remaining observations are a missing functional index on the JSONB supersession query and silent deduplication of same-content files within a repository, neither of which affects correctness or security.

No files require special attention. apps/api/src/lib/graph-route.ts has a minor scalability note (JSONB index) but is otherwise correct.

Important Files Changed

Filename	Overview
apps/api/src/lib/repository-url.ts	New file: public-repository git-clone ingestion. Symlink traversal and broken-symlink crash previously flagged are both fixed (realpath + isPathInsideRoot double-check; try-catch continue on ENOENT). stdout limit throws a RepositoryUrlError. Looks correct.
apps/worker/workflows/sync-repository-graph.ts	Durable repository sync workflow. Top-level try/catch calls markBindingFailed on run.retryTerminal. findReusableProcessRun provides idempotent step recovery. behind/diverged and compare_timeout fall back to full-snapshot sync as expected.
apps/api/src/routes/connector-webhooks.ts	Webhook ingestion endpoint. JSON.parse is now wrapped in parseWebhookPayload. Binding query is scoped to the verified connector via inner join on connectorInstallationsTable. enqueueBindingWorkflows has error recovery that marks already-enqueued bindings as pending and failed bindings as failed.
apps/api/src/routes/connectors.ts	Connector admin and installation routes. assertInstallationBelongsToConnector guards all routes that accept an installationId. Manual sync endpoint resets binding to failed on enqueue error. Installation-connector mismatch issue from prior review is fixed.
apps/api/src/routes/graph.ts	New POST /:id/urls endpoint. Dedup key is per-repository (repository.url + checksum). Passes retiredFileIds: result.supersededFileIds to processFilesSpec so stale sources are expired. Archive-upload retry now passes retiredFileIds: [] to avoid invalidating unrelated sources.
packages/connectors/src/github.ts	GitHub provider client. behind and diverged status now return { isIncremental: false } instead of throwing, triggering full-snapshot fallback in the workflow. Tree truncation is checked and raises a limit error.
packages/connectors/src/gitlab.ts	GitLab provider client. compare_timeout: true now returns { isIncremental: false } so the workflow falls back to a full-snapshot sync rather than marking the binding failed.
apps/api/src/lib/graph-route.ts	commitGraphFileUploads added. Supersession uses JSONB extraction (metadata::jsonb ->> 'repositoryUrl') without an index; could be slow for large graphs. Logic is otherwise correct; superseded file IDs are returned for source invalidation.
migrations/20260615094125_fluffy_gladiator/migration.sql	Promotes files_graph_active_key_idx from a non-unique to a UNIQUE partial index (WHERE deleted = false), closing the insertRepositoryFiles idempotency gap identified in earlier review.
packages/db/src/source-validity.ts	New helpers for source/file predicate building. quoteIdentifier validates against an allowlist regex before injecting into raw SQL, preventing SQL injection through alias strings.
apps/worker/lib/code-repository-finalizer.ts	New file: invalidateSupersededRepositorySources correctly branches on retiredFileIds presence - empty array hits the early-return guard, undefined falls through to the latestFileIds path. Addresses the single-file retry invalidation issue from the prior review.

_{Reviews (18): Last reviewed commit: "fix(worker): scope duplicate webhook eve..." | Re-trigger Greptile}

[bot-kiwi]

👋 GitHub App is working! ✅

/cc @m43i @NHuxoll @bjrump

[m43i]

Will be done in another way later