Skip to content

Add opt-in frame check for off-task record changes#156

Draft
kimjune01 wants to merge 1 commit into
ServiceNow:mainfrom
kimjune01:frame-check
Draft

Add opt-in frame check for off-task record changes#156
kimjune01 wants to merge 1 commit into
ServiceNow:mainfrom
kimjune01:frame-check

Conversation

@kimjune01

Copy link
Copy Markdown

This is a follow-up to #155. The short version is that DeleteRecordTask.validate gives reward 1 as soon as the target row is gone, so a run that deletes the target plus a few unrelated rows scores exactly the same as one that deleted only the target. The create and edit validators already flag the same blind spot in their own comments ("we don't check if there are extra fields"), and monitorChangeOnFields, with the #149 and #151 hardening, only watches the current form, so it never sees the other records.

So this PR adds a small opt-in frame check to AbstractServiceNowTask. A task declares which tables it touches, the base class snapshots those at agent handoff, and a validator can ask what changed since. DeleteRecordTask is the first one wired up, where the only deletion it's supposed to make is the target and anything else counts as off-task.

The check is off by default (frame_gate=False), so nothing about existing runs or the leaderboard moves. With the flag off it just logs the off-task deletions and carries on, which is the point of shipping it observational first, because you get to see how often agents actually trip it before deciding whether it should cost anything. With the flag on, an off-task deletion drops the episode to 0.

I kept the first cut deliberately narrow. It only watches a table the task names, so unrelated instance activity doesn't get blamed on the agent, and it gates on deletions rather than on sys_updated_on changes, since business rules and cascades bump that field on their own and I wanted to avoid false positives out of the gate. The sanctioned footprint here is just the target deletion, and a task that wants to allow alternative solutions would widen that later. I'd rather land the safe, boring version and grow the policy from there than try to guess all of it up front.

I'm marking this a draft on purpose: I haven't run it end to end against a live ServiceNow instance yet, since I don't have one provisioned as I write this. Treat it as a design proposal. I'll snapshot-and-diff a real delete task and paste the before and after reward before asking anyone to merge.

Known limits

  • It watches the task's own table only, so a cascade delete into a related table slips through. That is fine for the delete finding, but it is not a general proof that nothing else changed.
  • It assumes the instance is not being mutated by someone else mid-episode. That holds for the pooled-instance-per-run setup, and it is worth stating.
  • It gates on deletions rather than on modifications or creations, on purpose, for the sys_updated_on reason above. Modification gating can come once there is a way to filter platform-generated updates.
  • The success check still queries field_name=field_value as shipped. Switching it to sys_id={record_sys_id} would make it robust to the case where the agent edits the field instead of deleting the record. I left that out of this diff to keep the change focused, but I am happy to fold it in.

🤖 Generated with Claude Code

DeleteRecordTask.validate returns reward 1 as soon as the target row is
gone, so a run that deletes the target plus unrelated rows scores the same
as one that deleted only the target (issue ServiceNow#155).

Add an opt-in frame check on AbstractServiceNowTask: a task declares the
tables it touches, the base class snapshots them at agent handoff, and a
validator can ask for the delta since. DeleteRecordTask is wired as the
first consumer, gating on deletions outside the sanctioned target.

Off by default (frame_gate=False), so existing runs and leaderboard
numbers are unaffected; with the flag off it logs the off-task delta only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant