docs: add proposed server-side policy evaluation tutorial - #338
Draft
pbeckham wants to merge 25 commits into
Draft
docs: add proposed server-side policy evaluation tutorial#338pbeckham wants to merge 25 commits into
pbeckham wants to merge 25 commits into
Conversation
Contributor
|
Claude finished @pbeckham's task in 3m 57s —— View job Review of PR #338 — Evaluate policies server-side tutorial
Scope: 2 doc files ( Diátaxis classification: The page is titled as a tutorial but is closer to a hybrid tutorial + reference. The core loop (Steps 1–4) is tutorial-shaped — it teaches by doing. The later sections (Rego contract, managing, viewing, inspecting) read more like reference/how-to material. For a design preview seeking feedback this is reasonable, and the CriticalNone found. The page is well-structured, all internal links resolve to existing files, the new page is registered in Improvement
Suggestion
Summary
This is a thorough, well-written design preview. The page is long (~1037 lines) but earns its length — each section carries distinct information and the cross-references are precise. The prior review feedback from both claude and AlexKantor87 has been addressed carefully (Rego consistency, missing The main thing to track before this leaves draft is resolving the two TODO comments (OPA version pin and failure exit codes) and noting that the reference schema URLs are aspirational. |
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
… slots Rework the listing/searching/viewing surfaces to lead with the Kosli app experience rather than the CLI: browse policies in the Policies sidebar, view a policy's decisions and decision detail on its detail page, with the CLI kept as a secondary option for scripting and CSV export. Add <Frame> screenshot slots (policy-list, policy-decisions, policy-decision-detail) following the repo placeholder pattern (PR #278): 1x1 transparent PNGs with descriptive alt text and {/* TODO */} markers to replace with real UI mockups before merge. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add a core-flow section heading so the steps aren't nested under Prerequisites, and turn the intro into a linked table of contents - Cross-link the core-flow steps to the validate/manage/view sections - Drop the non-compliance vs missing-evidence split; both surface as violations on the decision - Move "Evaluate a local policy file" out of the main steps into the Validate and test section - Rework "Trace and replicate" into "View and trace the decision", leading with viewing the decision in Kosli - Clarify that the policy identifier is user-provided and immutable, not assigned by Kosli - Recommend keeping Rego tests alongside policies in the source repo Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add --dry-run to the evaluate policy flag table - Reconcile the policy's fields (rego, description, input-context schema, identifier given at publish) with the create policy command - Name the auditor/control-owner and application-team audiences in the view and inspect sections - Strengthen the enforcement pointer (for_control + kosli assert as the PEP) and add a Next steps section Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve the overloaded --evaluation-context flag in the server-side evaluate tutorial: - Rename the create/validate schema selector to --context-schema (trail|trails), so it no longer shares a name with the evaluate-time trail instance flag - Make --flow/--trail the decision destination and add --evaluation-flow for the source, so facts can be evaluated from a different flow than the decision is recorded on, with a worked cross-flow example Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-flow Replace the non-extensible --evaluation-flow flag with typed references on --evaluation-context. In the beta a reference is a trail: trail=<trail_id> (in --flow) or trail=<flow_id>:<trail_id> for a trail in another flow, repeatable. This carries each trail's flow in the reference, keeps --flow/--trail as the decision destination, and leaves room for other context types (snapshots, artifacts) without new flags. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slash matches Kosli's existing hierarchy convention (trail URLs are
{org}/flows/{flow}/trails/{trail}), whereas colon in Kosli denotes
key:value / image:tag pairs. Use trail=<flow_id>/<trail_id>.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kosli create policy already carries a --type enum (currently [env]), so evaluation policies extend it rather than needing a new noun. Align the tutorial with the real command surface: - create policy uses the positional POLICY-NAME POLICY-FILE-PATH form with --type rego (env stays the default for environment policies) - versioning re-runs create policy with --comment, replacing the invented kosli update policy - list policies filters with --type rego - explain inline that create policy is shared with environment policies, selected by --type Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document the contract a server-side policy is written against, so the design preview covers the OPA-facing surface as well as the command surface: entry point, what a policy sees in input, parameters, output, available built-ins, evaluation time, engine version, and evaluation-failure behavior. Publishes the input-context schemas as JSON Schema so authors can type-check with opa check --schema, and expresses the sandbox as an OPA capabilities file so the constraints are checkable locally rather than being a server-side surprise. Replaces time.now_ns with an injected data.kosli.evaluation_time so time windows stay replicable and anchored to the moment a decision is valid as of. Several specifics are proposals for the team to confirm before this goes to customers: the fixed package name, the three docs.kosli.com URLs (not yet published), data.kosli.evaluation_time, and the built-in allow/deny split. Two TODOs remain for the pinned OPA version and the assert exit codes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Close the docs-only gaps found reviewing the tutorial against the user story map and the V1 design decisions: - Record both decision timestamps (requested and recorded), state that a decision is valid as of the requested time, and explain what happens to a fact that arrives between the two. - Add --name to kosli create policy, and note that the positional argument is the immutable identifier for --type rego but the name for --type env. - Say that a server-side decision satisfies a type: decision flow-template slot, linking the two features together. - Resolve the trail vs trails contradiction and add a multi-trail example, including what happens when the context schema and reference count disagree. - Expand what is out of scope with version pinning, replaying a decision against a new policy, advisory outcomes, verbose tracing, and branch or environment filtering, so reviewers can tell deferred from unconsidered. - Separate authoring from execution: the repository is authoritative for authoring, Kosli for what ran. - Note that sampling and CSV export are proposed for a control's decisions too, and are not part of controls today. In the controls tutorial, point the API token prerequisite at the current authentication page rather than the redirect, and align the archive wording with the archive-as-event model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Publishing, updating, and archiving an evaluation policy require org admin, matching controls; evaluating and recording a decision needs only an ordinary API token, so pipelines can invoke a policy without being able to change one. That separation is what makes a server-side decision worth more than a self-reported one. Flag the consequence for the local-policy-file path, which bypasses the admin gate because the caller supplies the Rego: call those decisions out as fit for iteration rather than audit evidence, and ask for feedback on whether the beta should mark or restrict them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Kosli derives the parameters a policy takes from the data.params paths its Rego depends on, rather than asking authors to declare them, and validates supplied parameters before the evaluation is queued — in the CLI so it can be caught locally, and on the server as the real gate. Explain why this matters: a misspelled parameter leaves the alias undefined, so allow falls back to false and the violations rules referencing the same alias do not fire either, producing a non-compliant decision with no violations and a failed pipeline with nothing to fix. Note the limitation that dynamically computed parameter names cannot be derived statically. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…drail Act on three open review questions. Review (question 2): add "Review a policy change" under Manage policies. A policy is the code that decides whether a control passes, so changing it is a change to the control environment. In the beta the review happens in the policy repository — protected branch, CODEOWNERS per policy so the accountable control owner sees changes to their control, publish from CI on merge so the reviewed commit is the published version, and --comment pointing back at the change record. State plainly that Kosli has no in-product approval step and ask whether it needs one. Replicability (question 3): stop claiming a re-run yields the same decision and instead show how to do it. Tabulate what a decision records, including the policy content digest and the engine version, and add "Re-run a decision offline" — export the trail moment, params, injected evaluation time, and policy source, then evaluate with stock OPA. This is the stronger audit story because it does not ask anyone to trust Kosli to check Kosli, and the export doubles as an opa test regression fixture. Cross-flow evidence (question 5): keep the capability, record the source flow and trail on the decision so a reviewer can see a release decision was reached from another flow's evidence, and warn that evidence from a low-assurance flow can otherwise close a control that gates production. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The page had grown to the longest tutorial in the repo, and a reader had to get through a comparison table, prerequisites and four steps before seeing what the feature actually is. Add a "Start here" opener: the whole loop as two commands, a short in-scope and out-of-scope summary so reviewers do not spend feedback on things already deferred, and cards routing the three audiences — governance platform engineer, auditor, application team — to where they should begin. Take the fifteen-row flag table out of the tutorial path. Inline only the four flags that carry the meaning, warn about the destination versus source distinction that the flag names invite people to misread, and collapse the full table into an accordion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replace the three 70-byte placeholder images with rendered mockups, and add a fourth for the Policy tab. Chrome and palette are matched against the real Controls screenshots in images/tutorials/controls-*.png, so the proposed pages sit alongside the shipped ones without looking foreign. Every page carries a "Mockup - proposed design" marker, because they depict a UI that does not exist. Add a "View a policy" section for the Policy tab, which is the default view on a policy: the read-only policy-as-code Kosli actually runs, for the version you choose. A name and description can only claim what a policy checks; the source is what lets a control owner answer whether it meets the control objective, and what lets an auditor check the digest a decision cited. Files in a multi-file bundle are listed with the entrypoint, context schema and parameters read; shipped tests are listed and marked not executed. Note in the ship step that a policy can be a directory of Rego files stored as one versioned bundle with one digest, so a decision still cites one thing. Mockup sources live in mockups/policies-ui/ with regeneration instructions; generated HTML and PNGs are gitignored. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Mintlify auto-applies every .css file it finds in the project — style.css is not referenced from docs.json, it is picked up by convention — so committing mockups/policies-ui/kosli-mock.css injected the mockup chrome into every docs page. It carries a universal margin/padding reset and a fixed-width flex body, which broke the site layout. Move the stylesheet into build.py, which writes it at generation time, and gitignore it alongside the generated HTML and PNGs. No .css is committed outside style.css. Renders are byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two corrections to the offline re-run and the command surface. The export is now kosli get evaluation-context, not kosli get decision. The evaluation context is the entity that actually holds what an evaluation was given, so exporting the decision meant reaching through it to find the inputs. More importantly a context exists from the moment an evaluation is queued, so this also works when there is no decision to look at — a queued evaluation, or one that failed, which is the case you most want to inspect. Decisions now name the context they came from, in the decision detail mockup and in the recorded-on-the-decision table, so the context ID is one hop away. Rename --evaluation-context to --context. The old name described the entity Kosli builds rather than what you pass, and it would have blocked the obvious spelling for re-running a stored context (--context evc_...). --context also survives the scope growing beyond attestations: a reported workload is a state report, not a fact, so "facts" would have been wrong for it, while --context snapshot=... reads naturally. --context-schema needs no change. Note in the flag description that this selects what to evaluate, not a cluster or account — kubectl has trained this audience to read --context the other way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A reviewer read "the first pair says where the decision is written, the second says which facts are read" as describing two pairs of flags, and could only find one. --context is a single flag, so the phrasing was wrong. Name each flag instead of counting them, and lead with the read before the write so the sentence follows the order the command works in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Republishing a policy whose bundle, name, description and context schema are unchanged creates no new version, reports that, and exits 0. Without this the "publish from CI on merge" pattern the page recommends mints byte-identical versions on every merge and the version history stops being useful — and a declarative apply could never be idempotent. Tabulate what does and does not count as a change: source edits version (the digest is over the bytes, so reformatting counts), name/description/schema edits version because a version's description is part of the audit record, and a --comment on its own does not, since a comment describes a change that isn't there. Comparison is against the current version rather than the whole history, so republishing an earlier bundle is a real change and does create a version. That is how a rollback works, and it stays visible in the history. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Addresses review feedback on the proposed server-side policy evaluation tutorial. Reviewer fixes: - Pass --params in the quick-start example; the example policy reads data.params.pr_attestation_name, so as written it would have been rejected by the parameter validation the page documents. - Add a second violations rule covering the missing-attestation case. A missing slot left both allow and the first violations rule undefined, giving a denial with an empty violations list — contradicting the claim in "Inspect a failing evaluation". - Fix the out-of-scope anchor. Mintlify renders the apostrophe into the heading id, so #whats-out-of-scope-for-the-beta did not resolve; drop the apostrophe from the heading and update all four links. - Use := in the default allow assignment, matching opa fmt and the rest of the page. Command surface: - Document --flow and --trail as always required, matching every other Kosli command that writes to a trail. This also removes a circular default, where --context defaulted to the decision trail and --trail defaulted to the evaluation trail. - Name --flow, --trail and --context explicitly in every example rather than relying on the --context default. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The flag names the schema of the data passed to the policy as `input`, so `--input-schema` says what it does. It also stops the publish and evaluate commands sharing a word for two unrelated things: policies are now published with `--input-schema` and evaluated with `--context`, which reviewers had read as one overloaded flag. Renames the flag on `kosli create policy` and `kosli validate policy`, and the concept in prose from "input-context schema" to "input schema". No behavior change: the accepted values are still `trail` and `trails`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Drop the restatement of the async default. It was stated three times within 65 lines: the intro to the command, a paragraph below the flag table, and the --sync row of the flag table itself. The paragraph now only carries what was unique to it — --assert as the pipeline gate, and the pointer to the decisions section. - Replace "here" with "in the example below" where it pointed forward at an example rather than at nearby text, which reviewers read as referring to the paragraph they were in. - Trim the meta description from 163 to 130 characters, under the 160 search engines will show. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A policy cannot open the files attached to an attestation, which left readers asking how a SARIF file or SBOM gets evaluated at all. Documents the pattern: summarize the attachment into JSON at attestation time and pass it as --user-data, which is in the policy input, while keeping the file itself on the same attestation with --attachments. The summary is what the policy reads; the attachment is what makes the summary checkable, since an auditor can confirm it against the file it came from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pbeckham
force-pushed
the
6267-evaluate-policy-tutorial
branch
from
July 29, 2026 09:40
f6a47e4 to
5b300d7
Compare
| version, so a decision states exactly what ran. Author against that version locally and check with | ||
| `opa check --v1-compatible` to avoid a policy that compiles on your machine but not in Kosli. | ||
|
|
||
| {/* TODO: state the pinned OPA version once the evaluation runtime is chosen. */} |
Contributor
There was a problem hiding this comment.
Improvement: Two {/* TODO */} comments remain in the file (here and line 606). They don't render, but they represent open design decisions (pinned OPA version, failure exit codes) that should be tracked. Consider converting them to visible <Warning> callouts so reviewers can comment on them directly, or filing issues before this leaves draft.
Comment on lines
+463
to
+466
| | `--input-schema` | `input` contains | Reference schema | | ||
| |---|---|---| | ||
| | `trail` | `input.trail` — a single trail object | `https://docs.kosli.com/schemas/evaluation-input/trail/v1.json` | | ||
| | `trails` | `input.trails` — an array of trail objects | `https://docs.kosli.com/schemas/evaluation-input/trails/v1.json` | |
Contributor
There was a problem hiding this comment.
Improvement: These reference schema URLs (evaluation-input/trail/v1.json, trails/v1.json) don't exist yet and will 404 if clicked on the preview site. Consider adding "(not yet published)" after each URL to set expectations, since the page is a design preview.
| | `--trail` | The trail the decision is recorded on. **Required** — there is no default, the same as every other Kosli command that writes to a trail. Set it once as `KOSLI_TRAIL` to keep it off each command. | | ||
| | `--policy` | The policy to evaluate — a Kosli policy **identifier** (uses its current version) or a **local** `.rego` file (see [Evaluate a local policy file](#evaluate-a-local-policy-file)). | | ||
| | `--control` | The control identifier the resulting decision is recorded against. | | ||
| | `--context` | One or more **typed references** to what should be evaluated (the **source**), repeatable. This is what Kosli resolves into the [evaluation context](#re-run-a-decision-offline) a decision cites — not a cluster or account selector. In the beta the only reference *type* is a trail: `trail=<trail_id>` (in `--flow`), or `trail=<flow_id>/<trail_id>` to name a trail in another flow. Repeat the flag to evaluate several trails — see [Evaluating multiple trails](#evaluating-multiple-trails). Optional: defaults to the trail named by `--flow` / `--trail`. The typed form leaves room for other context types — snapshots and reported workloads, for example — later, without new flags. | |
Contributor
There was a problem hiding this comment.
Suggestion: This --context flag description is a ~6-line paragraph inside a table cell. On narrow viewports it'll be hard to read. Consider moving the detailed explanation (typed references, repeat semantics, future context types) to prose below the accordion table, keeping the cell to a single sentence like "One or more typed references to what should be evaluated (the source), repeatable. Defaults to the trail named by --flow / --trail."
pbeckham
commented
Jul 29, 2026
| kosli evaluate policy \ | ||
| --flow "$FLOW_NAME" \ | ||
| --trail "$TRAIL_NAME" \ | ||
| --policy ./pr-approved.rego \ |
Contributor
Author
There was a problem hiding this comment.
How would this work if you had a rego policy bundle?
pbeckham
commented
Jul 29, 2026
| Pass thresholds, names, and flags — never secrets or personal data. | ||
| </Warning> | ||
|
|
||
| ### Output |
Contributor
Author
There was a problem hiding this comment.
Now thinking the Result is a better name for the output of an evaluation
pbeckham
commented
Jul 29, 2026
|
|
||
| - the Rego **compiles**; | ||
| - it matches its declared **input schema** (`trail` / `trails`); | ||
| - it satisfies the **output schema** — an `allow` rule is mandatory. |
Contributor
Author
There was a problem hiding this comment.
Potentially result schema would be better term
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Documentation-driven development for the server-side Kosli Evaluate project (kosli-dev/server#6267). Adds a tutorial describing the proposed V1
kosli evaluate policyflow, published so it can be shared internally and with design-partner customers for feedback before the feature is built.This is a design preview — nothing in it is runnable today. The page is marked as such throughout.
Scope (this draft)
Core happy path for a governance platform engineer:
allowmandatory).kosli evaluate policyruns server-side and records a decision against a control; synchronous.attestation_ids, policy version, params, timestamp; includes the T / T+1 / T+2 temporal-correctness case.An explicit out-of-scope-for-V1 section keeps reviewer feedback on target (managing policies, testing/validation, reporting/audit views, unhappy-path inspection, composable policies, non-trail targets, scheduled evals, risk measurement).
Design notes surfaced for feedback
kosli attest decisionandkosli evaluate trailflags.--compliantand evidence (--attachments/--user-data) are intentionally not flags — the server derives compliance and captures the policy/report/violations as output.kosli create policy, which collides with the existing environment-policy command — flagged inline as an open question.inputshape is borrowed from the client-side tutorial and may differ server-side.Draft because
Content describes proposed behavior and will change as the design firms up; opening as draft to gather review before it's merged/published.
🤖 Generated with Claude Code