fix: reliable cli_audit telemetry + docs links → /introduction#461
Merged
Conversation
Separate PR (the 0.0.11 release PR #460 is already merged). Two changes: 1. The `failproofai audit` CLI dropped its telemetry on exit. src/audit/cli.ts emitted cli_audit_started/completed/failed fire-and-forget (void trackHookEvent), then die()->process.exit(1) (failed path) and process.exit(0) (empty-history path) killed the in-flight fetch before it landed — those events never reached PostHog. Now await the two exit-adjacent events before exiting, matching bin/failproofai.mjs's track() helper; cli_audit_started stays fire-and-forget since the multi-second scan keeps the process alive. New __tests__/audit/audit-cli-telemetry.test.ts asserts each path emits its event and that the exit-adjacent events are awaited before process.exit. 2. "Docs" landing links pointed at a bare root that doesn't resolve to a page. Point the failproofai --help LINKS banner, the dev/start launch banner, the dashboard "Reach Us" Documentation entry, and the README docs badge (English + 14 translations) at https://docs.befailproof.ai/introduction. Deep page links (e.g. /built-in-policies) are unchanged. All gates green locally: lint (0 errors), tsc, 1898 unit tests, build, validate:mdx (300 pages), 298 e2e tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TD26MQ8M3gyaTLN2cwufTf
📝 WalkthroughWalkthroughTelemetry is initialized earlier or awaited before exit across audit and auth flows, dashboard reruns, client capture, and postinstall failure handling. Docs links in banners, badges, and locale READMEs now point to the documentation introduction page. ChangesTelemetry Delivery Updates
Documentation Link Updates
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches📝 Generate docstrings
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
__tests__/api/audit-run-route.test.tsESLint skipped: missing config or dependency (missing-dependency). The ESLint configuration references a package that is not available in the sandbox. __tests__/audit/audit-cli-telemetry.test.tsESLint skipped: the ESLint configuration for this file references a package that is not available in the sandbox. __tests__/auth/auth-cli-telemetry.test.tsESLint skipped: the ESLint configuration for this file references a package that is not available in the sandbox.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
…dit run, +) Follow-up to the cli_audit fix, addressing the rest of the telemetry audit: - Auth CLI (src/auth/cli.ts): same fire-and-forget bug as cli_audit — the terminal login/logout/whoami/otp events were dropped when the process exited. Now awaited (mid-flow events stay fire-and-forget behind interactive prompts). - Server-side audit run (POST /api/audit/run): ran runAudit() detached with NO telemetry. Now emits audit_run_started/completed/failed/rejected mirroring the CLI funnel; dashboard now emits the missing audit_rerun_succeeded too. - postinstall: track the build-missing failure (package_install_failed) awaited before exit(1) — previously invisible. - captureClientEvent: add keepalive:true so unload-time events aren't dropped. - login-verify: track validation-400s + add email/source for parity with login-request. - Property gaps: node_version on package_installed, drop duplicate version on first_install, subcommand+exit_code on cli_auth_invoked. - Deliberately left the hook hot-path error events fire-and-forget (awaiting them would add telemetry latency to every tool call). Tests: new __tests__/auth/auth-cli-telemetry.test.ts; extended audit-run-route, client-telemetry, and postinstall tests. All gates green: lint, tsc, 1905 unit, build, validate:mdx (300), 298 e2e. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TD26MQ8M3gyaTLN2cwufTf
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@__tests__/auth/auth-cli-telemetry.test.ts`:
- Around line 1-145: The new auth CLI telemetry test is placed under the wrong
unit-test area relative to the stated repository guideline. Update the placement
of the `auth-cli-telemetry.test.ts` coverage so it follows the expected
unit-test layout used by the test suite (or, if the broader per-area convention
is intended, align the guideline and suite organization consistently); use the
`runAuthCli`-focused test in the auth telemetry file as the location reference
when relocating it.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: bc6e428c-3bba-46f9-9e78-2e938bfc3ba3
📒 Files selected for processing (32)
CHANGELOG.mdREADME.md__tests__/api/audit-run-route.test.ts__tests__/audit/audit-cli-telemetry.test.ts__tests__/auth/auth-cli-telemetry.test.ts__tests__/lib/client-telemetry.test.ts__tests__/scripts/postinstall.test.tsapp/api/audit/run/route.tsapp/api/auth/login-request/route.tsapp/api/auth/login-verify/route.tsapp/audit/_components/audit-dashboard.tsxbin/failproofai.mjscomponents/reach-developers.tsxdocs/i18n/README.ar.mddocs/i18n/README.de.mddocs/i18n/README.es.mddocs/i18n/README.fr.mddocs/i18n/README.he.mddocs/i18n/README.hi.mddocs/i18n/README.it.mddocs/i18n/README.ja.mddocs/i18n/README.ko.mddocs/i18n/README.pt-br.mddocs/i18n/README.ru.mddocs/i18n/README.tr.mddocs/i18n/README.vi.mddocs/i18n/README.zh.mdlib/client-telemetry.tsscripts/launch.tsscripts/postinstall.mjssrc/audit/cli.tssrc/auth/cli.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Separate PR from the merged 0.0.11 release (#460). Started as the
cli_audittelemetry fix + docs links, then expanded to close all the gaps from a full telemetry audit of the repo.Telemetry reliability + coverage
The core bug (delivery on exit). Several CLI events were emitted fire-and-forget (
void trackHookEvent(...)) and then dropped by a followingprocess.exit()/die():failproofai audit(src/audit/cli.ts) —cli_audit_completed(empty-history path) andcli_audit_failedwere lost. Now awaited;cli_audit_startedstays fire-and-forget (the multi-second scan keeps the process alive).failproofai auth(src/auth/cli.ts) — identical bug.audit_cli_auth_login_completed/audit_otp_verified/audit_user_identity_linked/audit_cli_auth_logout_completed/audit_cli_auth_whoaminow awaited; the two mid-flow events stay fire-and-forget behind the interactive prompts.The blind spot (server-side audit run).
POST /api/audit/runranrunAudit()as a detached task with zero telemetry — the dashboard's actual audit work and its failures were invisible (only the client'saudit_rerun_clicked/audit_rerun_failedexisted). Now emitsaudit_run_started,audit_run_completed(duration, events + sessions scanned, findings, hits, persisted),audit_run_failed, andaudit_run_rejected, mirroring the CLI funnel. The dashboard also now emits the previously-missingaudit_rerun_succeeded.Smaller gaps.
package_install_failed, awaited beforeexit(1)); failed installs were invisible.captureClientEvent— addkeepalive: trueso events fired right before a navigation or unload aren't dropped.login-verify— track validation-400s and addemail+sourcefor parity withlogin-request.node_versiononpackage_installed, drop the duplicateversiononfirst_install, addsubcommand+exit_codetocli_auth_invoked.Docs links to /introduction
The "Docs" landing links pointed at a bare root that doesn't resolve to a Mintlify page. Pointed the CLI help banner, the launch banners, the dashboard "Reach Us" Documentation entry, and the README docs badge (English + 14 translations) at
https://docs.befailproof.ai/introduction. Deep page links are unchanged.Tests
New
__tests__/auth/auth-cli-telemetry.test.ts(proves each path emits and awaits its event). Extended theaudit-run-route,client-telemetry, andpostinstalltests. Thecli_auditfix is guarded by__tests__/audit/audit-cli-telemetry.test.ts.Validation (all CI gates green locally)
lint (0 errors) · tsc · 1905 unit tests · build · validate:mdx (300 pages) · 298 e2e
🤖 Generated with Claude Code
Summary by CodeRabbit
Bug Fixes
Documentation