Skip to content

feat(governance): track-event telemetry sink for App Insights customEvents#135

Open
viswa-uipath wants to merge 1 commit into
feat/governance-guardrail-compensationfrom
feat/traces-metrics
Open

feat(governance): track-event telemetry sink for App Insights customEvents#135
viswa-uipath wants to merge 1 commit into
feat/governance-guardrail-compensationfrom
feat/traces-metrics

Conversation

@viswa-uipath

Copy link
Copy Markdown

Summary

Stacked on top of #133 (feat/governance-policy-fetch-hoist). Adds a new platform-mandated audit sink that emits governance evaluation telemetry to App Insights customEvents via the UiPathPlatformGovernanceProvider.track_event callable shipped in uipath-python PR #1745 (uipath-platform 0.1.74).

The runtime stays decoupled from uipath-platform — the sink takes a Callable[..., None]; the host adapts provider.track_event to it at wiring time. No new dep added to uipath-runtime.

Event vocabulary

Event When Per-hook count
governance.rule.denied A rule matched and its configured action ≠ allow one per matched-restrictive rule
governance.hook.summary Hook end, with at least one rule evaluated or skipped exactly one

Volume-controlled: a 50-rule pack on a chatty agent doesn't multiply per-step telemetry by 50. Passed + skipped + matched-allow rules roll into the hook summary; only denials get individual events.

Payload contract

Every event carries:

Field Source
execution_engine GovernanceRuntimeMetadata (default uipath_native_governance_checker, host-overridable for future engines like AGT)
agent_type host-supplied (e.g. uipath_coded, uipath_lowcode, servicenow)
agent_framework host-supplied (e.g. langchain)
runtime_version auto-resolved via importlib.metadata.version(\"uipath-runtime\")
operation_id (HTTP header) AuditEvent.trace_id → App Insights operation_Id for cross-event correlation
agent_name / hook / timestamp from the audit event

Per-rule events add: pack, clause, rule_name, mode, evaluator_result (DENY/HITL), action_applied (mode-adjusted), duration_ms, mapped_to_uipath, detail.

Per-hook summary adds: mode, passed_count, denied_count, skipped_count, skipped_policy_names, guardrail_dispatched_count (UiPath-mapped guardrails that fell back to /runtime/govern), duration_ms, final_action.

Filter rules (accepts() + emit() defense-in-depth)

Case track_event fires?
Any event with mode = DISABLED
RULE_EVALUATION, matched = False ❌ (rolls into hook summary's passed_count)
RULE_EVALUATION, matched = True, action = allow ❌ (positive informational match, not a denial)
RULE_EVALUATION, matched = True, action ∈ {deny, escalate, audit}
HOOK_END, total_rules = 0 AND skipped_count = 0 ❌ (empty hook, zero operator value)
HOOK_END, total > 0 OR skipped > 0

Wiring

The audit manager auto-registers the sink at construction (mirrors _register_traces_sink shape, wrapped in a broad try/except so a misconfigured wiring layer never crashes the agent):

```python
manager = AuditManager(
track_event=provider.track_event, # from PR #1745
runtime_metadata=GovernanceRuntimeMetadata(
agent_type="uipath_coded",
agent_framework="langchain",
),
)
```

If the host forgets to wire track_event, the registration logs a warning and the sink is skipped — telemetry off, runtime continues.

Count semantics

Field Meaning
matched_rules Raw "any check matched" count — unchanged for backward compat with the legacy traces sink
denied_count (new) Matched AND action ≠ allow — the spec-meaning denial count
passed_count total_rules - denied_count — includes both unmatched and matched-allow rules

Test plan

  • uv run ruff check src/uipath/runtime/governance tests — clean
  • uv run mypy src/uipath/runtime/governance — clean (12 source files)
  • uv run pytest --no-cov374 passed, 1 skipped (was 326 → +49 new tests, +1 skipped pre-existing)

New test files:

  • tests/test_governance_metadata.py (5) — defaults, overrides, frozen behavior, PackageNotFoundError fallback
  • tests/test_track_events_sink.py (30) — filter (accepts + emit defense-in-depth) × event types × DISABLED × empty-hook × matched-allow, payload shape, mode handling, operation_id correlation
  • tests/test_audit_manager_track_event_wiring.py (7) — happy path, auto-registration alongside traces sink, missing-callable warning, synthetic registration-failure recovery, opt-out
  • tests/test_evaluator_telemetry.py (7) — per-rule duration, per-hook duration, skipped tracking, denied/passed counts, matched-allow contributes to passed, guardrail dispatched count

Files touched

File Change
src/uipath/runtime/governance/_audit/metadata.py (new) GovernanceRuntimeMetadata dataclass
src/uipath/runtime/governance/_audit/track_events.py (new) TrackEventAuditSink
src/uipath/runtime/governance/_audit/base.py Auto-register the sink; emit_* signature extensions (backward-compatible)
src/uipath/runtime/governance/native/evaluator.py Per-rule + per-hook timing; skipped-rule tracking; denied/passed split

Net: 8 files, +1524 / -11.

Out of scope

  • Per-rule visibility into UiPath-mapped guardrails (would be a governance.guardrail.dispatched event from the compensator). Today the per-hook guardrail_dispatched_count supports the mapped-vs-native ratio query.
  • A real TelemetryProvider protocol in uipath-core — the callable indirection is sufficient for current needs.

🤖 Generated with Claude Code

@viswa-uipath viswa-uipath requested a review from a team as a code owner June 26, 2026 04:19
@viswa-uipath viswa-uipath changed the base branch from feat/governance-policy-fetch-hoist to feat/governance-guardrail-compensation June 27, 2026 09:48
@viswa-uipath viswa-uipath force-pushed the feat/traces-metrics branch from 126d18f to e5dd14e Compare June 27, 2026 10:21
@viswa-uipath viswa-uipath force-pushed the feat/governance-guardrail-compensation branch 2 times, most recently from 9ba1072 to 5433054 Compare June 27, 2026 10:34
@viswa-uipath viswa-uipath force-pushed the feat/traces-metrics branch from e5dd14e to 6771d51 Compare June 27, 2026 10:38
@viswa-uipath viswa-uipath changed the base branch from feat/governance-guardrail-compensation to feat/governance-audit June 27, 2026 17:00
@viswa-uipath viswa-uipath changed the base branch from feat/governance-audit to feat/governance-guardrail-compensation June 27, 2026 17:04
@viswa-uipath viswa-uipath force-pushed the feat/governance-guardrail-compensation branch 2 times, most recently from 1ee0e52 to d1a0384 Compare June 28, 2026 02:35
…vents

Adds the track-event telemetry sink: each governance evaluation that
the audit pipeline records can also be emitted as an App Insights
customEvents row via POST /runtime/log on the platform side. The
runtime layer assembles the event payload and hands it off through
the GovernanceTrackEventProvider protocol; the actual HTTP / auth /
operation_Id correlation lives on the uipath-platform side (PR #1745).

What lands here
---------------

- New TrackEventsSink that consumes evaluator output and forwards to
  the platform provider, with the same instance-scoped lifecycle the
  audit sinks use (bounded pool, atexit hook keyed via WeakSet).
- AuditManager wiring so a registered track-event provider receives
  every audit record, in addition to the console / traces sinks.
- Evaluator-side telemetry hooks: per-evaluation event metadata
  (rule_id, validator, action, severity) is captured at the
  evaluation seam and forwarded into the sink without leaking
  evaluator internals.
- Small audit-base cleanup: drop the conditional TYPE_CHECKING import
  block that's no longer needed after the evaluator slice landed.

Tests
-----

- ``test_track_events_sink`` — sink lifecycle (one atexit, weakref GC,
  idempotent close), payload assembly, provider-error swallowing,
  semaphore release on error.
- ``test_audit_manager_track_event_wiring`` — manager fans audit
  records out to the track-event sink alongside other sinks.
- ``test_evaluator_telemetry`` — per-evaluation event payload shape +
  the evaluator → sink boundary.
- ``test_governance_metadata`` — rule / validator / severity metadata
  populated correctly on emitted events.

Co-Authored-By: Aditi Kumari <aditi.kumari@uipath.com>
Co-Authored-By: Viswanath Lekshmanan <viswanath.lekshmanan@uipath.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@viswa-uipath viswa-uipath force-pushed the feat/traces-metrics branch from 6771d51 to a987bb1 Compare June 28, 2026 02:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant