Add opt-in frame check for off-task record changes#156
Draft
kimjune01 wants to merge 1 commit into
Draft
Conversation
DeleteRecordTask.validate returns reward 1 as soon as the target row is gone, so a run that deletes the target plus unrelated rows scores the same as one that deleted only the target (issue ServiceNow#155). Add an opt-in frame check on AbstractServiceNowTask: a task declares the tables it touches, the base class snapshots them at agent handoff, and a validator can ask for the delta since. DeleteRecordTask is wired as the first consumer, gating on deletions outside the sanctioned target. Off by default (frame_gate=False), so existing runs and leaderboard numbers are unaffected; with the flag off it logs the off-task delta only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a follow-up to #155. The short version is that
DeleteRecordTask.validategives reward 1 as soon as the target row is gone, so a run that deletes the target plus a few unrelated rows scores exactly the same as one that deleted only the target. The create and edit validators already flag the same blind spot in their own comments ("we don't check if there are extra fields"), andmonitorChangeOnFields, with the #149 and #151 hardening, only watches the current form, so it never sees the other records.So this PR adds a small opt-in frame check to
AbstractServiceNowTask. A task declares which tables it touches, the base class snapshots those at agent handoff, and a validator can ask what changed since.DeleteRecordTaskis the first one wired up, where the only deletion it's supposed to make is the target and anything else counts as off-task.The check is off by default (
frame_gate=False), so nothing about existing runs or the leaderboard moves. With the flag off it just logs the off-task deletions and carries on, which is the point of shipping it observational first, because you get to see how often agents actually trip it before deciding whether it should cost anything. With the flag on, an off-task deletion drops the episode to 0.I kept the first cut deliberately narrow. It only watches a table the task names, so unrelated instance activity doesn't get blamed on the agent, and it gates on deletions rather than on
sys_updated_onchanges, since business rules and cascades bump that field on their own and I wanted to avoid false positives out of the gate. The sanctioned footprint here is just the target deletion, and a task that wants to allow alternative solutions would widen that later. I'd rather land the safe, boring version and grow the policy from there than try to guess all of it up front.I'm marking this a draft on purpose: I haven't run it end to end against a live ServiceNow instance yet, since I don't have one provisioned as I write this. Treat it as a design proposal. I'll snapshot-and-diff a real delete task and paste the before and after reward before asking anyone to merge.
Known limits
sys_updated_onreason above. Modification gating can come once there is a way to filter platform-generated updates.field_name=field_valueas shipped. Switching it tosys_id={record_sys_id}would make it robust to the case where the agent edits the field instead of deleting the record. I left that out of this diff to keep the change focused, but I am happy to fold it in.🤖 Generated with Claude Code