[FEATURE]: Add structured control execution records for enforcement observability

[vishu-bh]

Summary

Add a framework-owned, structured execution record for every policy/plugin control evaluated by CPEX. Hosts must be able to derive enforcement telemetry—invocation counts, matched/applied counts, allow/deny outcomes, duration, stable reason/error codes, and per-control history—without parsing logs, exceptions, or plugin-specific metadata.

This feature should produce a language/runtime-neutral result contract. CPEX should collect execution facts centrally inside the executor; host applications remain responsible for mapping those facts into their own OpenTelemetry attributes, metrics, exporters, and dashboards.

This complements #43. Issue #43 discusses a general plugin telemetry protocol and export ownership. This issue is narrower: define and return a trustworthy control-execution ledger as part of pipeline execution, independent of any specific telemetry backend.

Problem

CPEX already exposes useful final state:

• PipelineResult.continue_processing
• a blocking PluginViolation
• non-fatal PluginErrorRecord values
• plugin identity/configuration in PluginRef
• hook, agent, MCP entity, request, trace, and span context in extensions

However, the host cannot reconstruct every control evaluation from the final result. Successful allow results are normally discarded, audit/transform denials may be suppressed by execution-mode semantics, per-plugin duration is not returned, and skipped/timed-out/cancelled controls do not share one result model.

This prevents reliable enforcement analytics such as:

• number of controls invoked for an operation
• number of controls that matched or took action
• final and per-control allow/deny results
• per-control latency
• top/least invoked controls
• control behavior by hook, entity, route, agent, and time range
• drill-down into structured reasons and stable error codes

Asking every plugin to emit custom metadata is not scalable and creates inconsistent semantics, cardinality, security, and performance behavior. Executor-owned facts must come from trusted framework state, not untrusted plugin claims.

User Story 1

• As a: host integrating CPEX
• I want: a structured record for every control evaluated during a hook invocation
• So that: I can emit consistent enforcement telemetry without parsing logs or implementing plugin-specific adapters

Acceptance Criteria

Scenario: Multiple controls allow an operation
  Given multiple controls are selected for a hook
  When every selected control completes and allows processing
  Then PipelineResult contains one ordered execution record per evaluated control
  And each record identifies the hook, control, outcome, status, and duration
  And the final pipeline decision remains allowed

Scenario: A control denies an operation
  Given a blocking control returns a structured violation
  When the executor halts the pipeline
  Then the denying control record contains allowed=false
  And the record contains the framework-attributed control identity
  And the record contains the stable violation code and bounded reason
  And controls not evaluated after the halt are not reported as invoked

Scenario: A non-blocking mode reports a deny
  Given an audit or transform control returns continue_processing=false
  When its execution mode suppresses blocking
  Then the record preserves that the control matched or requested denial
  And the record shows whether enforcement was applied
  And the final pipeline result reflects effective mode semantics

Scenario: A control errors or times out
  Given a control errors or exceeds its timeout
  When on_error policy is applied
  Then the record contains a stable status and error code
  And the final pipeline decision follows the configured on_error policy
  And free-form exception text is not required to classify the outcome

User Story 2

• As a: CPEX plugin author
• I want: framework-provided execution telemetry
• So that: my plugin does not need direct OpenTelemetry dependencies or backend-specific metadata

Acceptance Criteria

Scenario: Plugin returns a normal result
  Given a plugin returns PluginResult::allow, deny, modify, or modify_extensions
  When the executor consumes the result
  Then the framework creates the execution record automatically
  And the plugin does not need to create spans or connect to a collector

Scenario: Plugin metadata is present
  Given a plugin returns optional metadata
  When the result crosses typed, erased, FFI, Python, or Go boundaries
  Then framework execution facts remain separate from plugin-provided metadata
  And plugin metadata cannot overwrite trusted identity, status, duration, or decision fields

Proposed Architecture

Add a neutral execution record to cpex-core, populated by Executor around every plugin invocation.

Illustrative model:

pub struct ControlExecutionRecord {
    pub plugin_id: String,
    pub plugin_name: String,
    pub plugin_kind: String,
    pub hook_name: String,
    pub mode: PluginMode,
    pub status: ControlExecutionStatus,
    pub requested_allow: Option<bool>,
    pub effective_allow: bool,
    pub matched: Option<bool>,
    pub applied: bool,
    pub payload_modified: bool,
    pub extensions_modified: bool,
    pub duration_ns: u64,
    pub reason: Option<String>,
    pub error_code: Option<String>,
    pub config_keys: Vec<String>,
}

Naming is illustrative and should be finalized during design. Important semantic separation:

• requested_allow: what the plugin result requested
• effective_allow: result after execution-mode and on_error policy
• matched: whether the control condition matched, when known
• applied: whether the control changed payload/extensions or effective decision
• status: execution health, separate from allow/deny policy decision

Suggested stable statuses:

completed
skipped
error
timeout
cancelled
disabled

Expose records from PipelineResult, for example:

pub executions: Vec<ControlExecutionRecord>

Records should preserve deterministic execution order where phases are serial. Concurrent results need documented stable ordering, preferably resolved plan/priority order rather than completion order.

Alignment with Existing Architecture

Implementation should preserve current trust boundaries:

• Identity, kind, mode, priority, hook, timeout policy, and configuration keys come from manager-owned PluginRef.trusted_config, never plugin-returned metadata.
• Violation plugin name remains framework-attributed.
• Payload and extension modification acceptance remains governed by execution mode and capability checks.
• on_error behavior remains authoritative for effective outcome.
• Fire-and-forget execution remains represented separately when completion is unavailable at initial PipelineResult return. BackgroundTasks.wait_for_background_tasks() may need a structured completion-record surface.
• Execution records remain data only. CPEX must not require an OpenTelemetry collector or application-specific dashboard.
• Request trace/span context continues through existing extensions; hosts decide how records are attached to spans.

Current Rust result type-erasure should also be reviewed: PluginResult.metadata is not part of ErasedResultFields, while PipelineResult.metadata is initialized as None. This feature should not use arbitrary metadata as the trusted execution ledger, and existing metadata behavior should either be preserved correctly across boundaries or explicitly deprecated/documented.

Aggregates

CPEX may expose convenience methods, but records should remain source of truth:

invocation_count = records with an actual invocation
matched_count = records where matched=true
applied_count = records where applied=true
result_count = number of returned execution records
control_duration = sum of measured control execution durations

Hosts can combine pre- and post-invocation hook results for one operation. CPEX should not flatten dynamic control names into backend-specific attribute keys.

Security and Reliability Requirements

Trusted attribution

• Plugins must not be able to forge plugin_id, plugin_name, plugin_kind, hook, mode, status, duration, or effective decision.
• Framework-owned fields must be populated after plugin return using trusted registry/configuration data.
• Plugin-supplied violation/error data must not override framework attribution.

Sensitive data handling

• Never include configuration values in execution records; expose validated configuration key names only.
• Do not copy violation details, payloads, headers, credentials, extension contents, or stack traces into telemetry records by default.
• Bound and sanitize free-form reason values.
• Prefer stable low-cardinality error codes over exception messages.
• Document which fields may contain tenant/user-provided text.

Cardinality and memory bounds

• Bound records per pipeline invocation.
• Bound all string lengths, configuration key counts, and serialized record size.
• Define truncation/drop behavior and expose a non-sensitive overflow indicator.
• Avoid dynamic attribute-key generation inside CPEX.
• Avoid cloning payloads or large extension data into records.

Timing integrity

• Use monotonic time for durations.
• Define duration unit explicitly; nanoseconds are preferred internally for precision and conversion.
• Prevent integer overflow and use saturating conversion where necessary.
• Duration measurement must include plugin execution but clearly document whether queue/semaphore wait and framework overhead are included.

Failure isolation

• Failure to construct, serialize, or export an execution record must not alter policy outcome.
• Telemetry/recording failures must be observable without recursively generating unbounded records.
• Panic, timeout, cancellation, short-circuit, on_error=ignore, and on_error=disable paths must preserve existing fail-open/fail-closed semantics.

Concurrency

• Concurrent and short-circuited phases must not report controls as completed when tasks were cancelled or never started.
• No shared mutable result object may be written without synchronization.
• Record ordering and duplicate handling must be deterministic.
• Fire-and-forget records must not outlive or mutate an immutable PipelineResult through unsafe shared state.

Language and Boundary Requirements

Expose the same semantic fields through:

• Rust cpex-core
• Python PyO3 binding
• C FFI
• Go binding
• future WASM/plugin-host boundaries where applicable

Serialization must be versioned or backward-compatible. Adding records must not break consumers deserializing older PipelineResult payloads.

Conformance fixtures should prove equivalent outcomes across supported bindings for:

• allow
• deny
• payload modification
• extension modification
• suppressed audit/transform denial
• non-fatal error
• fatal error
• timeout
• disabled plugin
• concurrent deny with cancellation
• fire-and-forget completion/error

Performance Requirements

• Measure one monotonic start/end pair per invoked control.
• Avoid payload serialization and deep copies.
• Avoid per-plugin exporter/network calls.
• Keep record allocation bounded.
• Add benchmarks for long chains and concurrent execution.
• Document overhead target before declaring contract stable.

Out of Scope

• Grafana dashboards or application-specific queries
• vendor-specific names such as controls.*
• flattening control names into dynamic OpenTelemetry attribute keys
• direct plugin connections to OpenTelemetry collectors
• host-specific agent/tool/binding lookup
• changing policy decisions or execution-mode semantics

Compatibility and Delivery

CPEX main is Rust 0.2+; ContextForge currently consumes Python 0.1.1 from the 0.1.x branch. Proposed delivery:

1. Finalize language-neutral record semantics on main.
2. Decide whether a compatible subset should be backported to 0.1.x for current hosts.
3. Expose equivalent result fields through Python/Go/FFI.
4. Let hosts map records into their own OpenTelemetry semantic attributes.

Definition of Done

• Structured execution-record types and stable semantics documented.
• Executor populates trusted records for every relevant execution path.
• PipelineResult exposes records without breaking existing consumers.
• Rust, Python, C FFI, and Go surfaces are aligned.
• Security/cardinality bounds are enforced and tested.
• Concurrency and fire-and-forget behavior is documented and tested.
• Benchmarks demonstrate bounded overhead.
• Host can derive per-control invocation, match, application, decision, error, and duration telemetry without parsing logs or exception strings.

Alternatives Considered

1. Plugin-specific metadata: rejected because every plugin would define different fields and could forge framework facts.
2. Parse logs/exceptions: rejected because text is unstable, incomplete for successful controls, and may leak sensitive data.
3. Direct OpenTelemetry SDK in every plugin: rejected because it couples plugins to infrastructure and duplicates exporter configuration.
4. Only emit framework spans: insufficient as a host-neutral result contract and difficult to preserve consistently across FFI/language boundaries.

Additional Context

• Related telemetry protocol discussion: [FEATURE]: Standardize OpenTelemetry trace and metrics protocol for plugins #43
• CPEX final and error result structures already provide useful foundations: PipelineResult, PluginViolation, and PluginErrorRecord.
• Host applications may project neutral records into OpenTelemetry attributes and use configurable rename/drop policies for downstream consumers.