feat(governance): track-event telemetry sink for App Insights customEvents#135
Open
viswa-uipath wants to merge 1 commit into
Open
feat(governance): track-event telemetry sink for App Insights customEvents#135viswa-uipath wants to merge 1 commit into
viswa-uipath wants to merge 1 commit into
Conversation
126d18f to
e5dd14e
Compare
9ba1072 to
5433054
Compare
e5dd14e to
6771d51
Compare
1ee0e52 to
d1a0384
Compare
…vents Adds the track-event telemetry sink: each governance evaluation that the audit pipeline records can also be emitted as an App Insights customEvents row via POST /runtime/log on the platform side. The runtime layer assembles the event payload and hands it off through the GovernanceTrackEventProvider protocol; the actual HTTP / auth / operation_Id correlation lives on the uipath-platform side (PR #1745). What lands here --------------- - New TrackEventsSink that consumes evaluator output and forwards to the platform provider, with the same instance-scoped lifecycle the audit sinks use (bounded pool, atexit hook keyed via WeakSet). - AuditManager wiring so a registered track-event provider receives every audit record, in addition to the console / traces sinks. - Evaluator-side telemetry hooks: per-evaluation event metadata (rule_id, validator, action, severity) is captured at the evaluation seam and forwarded into the sink without leaking evaluator internals. - Small audit-base cleanup: drop the conditional TYPE_CHECKING import block that's no longer needed after the evaluator slice landed. Tests ----- - ``test_track_events_sink`` — sink lifecycle (one atexit, weakref GC, idempotent close), payload assembly, provider-error swallowing, semaphore release on error. - ``test_audit_manager_track_event_wiring`` — manager fans audit records out to the track-event sink alongside other sinks. - ``test_evaluator_telemetry`` — per-evaluation event payload shape + the evaluator → sink boundary. - ``test_governance_metadata`` — rule / validator / severity metadata populated correctly on emitted events. Co-Authored-By: Aditi Kumari <aditi.kumari@uipath.com> Co-Authored-By: Viswanath Lekshmanan <viswanath.lekshmanan@uipath.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
6771d51 to
a987bb1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on top of #133 (
feat/governance-policy-fetch-hoist). Adds a new platform-mandated audit sink that emits governance evaluation telemetry to App InsightscustomEventsvia theUiPathPlatformGovernanceProvider.track_eventcallable shipped in uipath-python PR #1745 (uipath-platform 0.1.74).The runtime stays decoupled from
uipath-platform— the sink takes aCallable[..., None]; the host adaptsprovider.track_eventto it at wiring time. No new dep added touipath-runtime.Event vocabulary
governance.rule.deniedallowgovernance.hook.summaryVolume-controlled: a 50-rule pack on a chatty agent doesn't multiply per-step telemetry by 50. Passed + skipped + matched-allow rules roll into the hook summary; only denials get individual events.
Payload contract
Every event carries:
execution_engineGovernanceRuntimeMetadata(defaultuipath_native_governance_checker, host-overridable for future engines like AGT)agent_typeuipath_coded,uipath_lowcode,servicenow)agent_frameworklangchain)runtime_versionimportlib.metadata.version(\"uipath-runtime\")operation_id(HTTP header)AuditEvent.trace_id→ App Insightsoperation_Idfor cross-event correlationagent_name/hook/timestampPer-rule events add:
pack,clause,rule_name,mode,evaluator_result(DENY/HITL),action_applied(mode-adjusted),duration_ms,mapped_to_uipath,detail.Per-hook summary adds:
mode,passed_count,denied_count,skipped_count,skipped_policy_names,guardrail_dispatched_count(UiPath-mapped guardrails that fell back to/runtime/govern),duration_ms,final_action.Filter rules (
accepts()+emit()defense-in-depth)track_eventfires?mode = DISABLEDmatched = Falsepassed_count)matched = True,action = allowmatched = True, action ∈ {deny, escalate, audit}total_rules = 0ANDskipped_count = 0total > 0ORskipped > 0Wiring
The audit manager auto-registers the sink at construction (mirrors
_register_traces_sinkshape, wrapped in a broadtry/exceptso a misconfigured wiring layer never crashes the agent):```python
manager = AuditManager(
track_event=provider.track_event, # from PR #1745
runtime_metadata=GovernanceRuntimeMetadata(
agent_type="uipath_coded",
agent_framework="langchain",
),
)
```
If the host forgets to wire
track_event, the registration logs a warning and the sink is skipped — telemetry off, runtime continues.Count semantics
matched_rulesdenied_count(new)passed_counttotal_rules - denied_count— includes both unmatched and matched-allow rulesTest plan
uv run ruff check src/uipath/runtime/governance tests— cleanuv run mypy src/uipath/runtime/governance— clean (12 source files)uv run pytest --no-cov— 374 passed, 1 skipped (was 326 → +49 new tests, +1 skipped pre-existing)New test files:
tests/test_governance_metadata.py(5) — defaults, overrides, frozen behavior,PackageNotFoundErrorfallbacktests/test_track_events_sink.py(30) — filter (accepts + emit defense-in-depth) × event types × DISABLED × empty-hook × matched-allow, payload shape, mode handling, operation_id correlationtests/test_audit_manager_track_event_wiring.py(7) — happy path, auto-registration alongside traces sink, missing-callable warning, synthetic registration-failure recovery, opt-outtests/test_evaluator_telemetry.py(7) — per-rule duration, per-hook duration, skipped tracking, denied/passed counts, matched-allow contributes to passed, guardrail dispatched countFiles touched
src/uipath/runtime/governance/_audit/metadata.py(new)GovernanceRuntimeMetadatadataclasssrc/uipath/runtime/governance/_audit/track_events.py(new)TrackEventAuditSinksrc/uipath/runtime/governance/_audit/base.pyemit_*signature extensions (backward-compatible)src/uipath/runtime/governance/native/evaluator.pyNet: 8 files, +1524 / -11.
Out of scope
governance.guardrail.dispatchedevent from the compensator). Today the per-hookguardrail_dispatched_countsupports the mapped-vs-native ratio query.TelemetryProviderprotocol inuipath-core— the callable indirection is sufficient for current needs.🤖 Generated with Claude Code