Improve first thought by RedTanny · Pull Request #291 · RHEcosystemAppEng/exploit-iq-agent

RedTanny · 2026-07-13T07:40:58Z

Summary

Refactors L1 intel gathering so the checker extracts CVE understanding and patch intel separately (per-file), then feeds focused context into Source Grep observations. Also includes related checker bugfixes and OSIDB identify improvements from this branch.

Why

The old single-prompt intel extraction mixed CVE narrative with full-patch analysis, which hurt keyword quality on large patches and gave observations little file-specific guidance. Splitting the flow improves search targeting and keeps large patches within the context budget.

High-level feature (intel refactor)

Prompt 1: CVE understanding (no patch required) — vulnerability type, component, ordered grep keywords
Prompt 2/3: Per-file patch analysis — functions/patterns per changed file, with token-budget batching for large patches
Result model: IntelGatheringResult aggregates per-file intel and converts back to VulnerabilityIntel for existing prompts/reports
Focused observations: when Source Grep targets a file, prepend that file’s intel into the observation context
New flow is now the sole intel path (dual-path flag removed)

Testing

Added unit coverage for the refactor and related empty/error tool handling:

test_file_change_summary.py — patch → per-file summaries / token estimates
test_cve_understanding_prompt.py — Prompt 1 formatting and keyword rules
test_patch_analysis_prompt.py — Prompt 2/3 formatting and functions summary helper
test_intel_gathering_result.py — lookup, aggregation, focused-context formatting
test_new_intel_flow.py — file priority + token-budget batch split
test_focused_observations.py — file filter extraction + focused context injection
test_check_empty_output.py — tool-error detection via status/prefix (not mid-content "error:")
tests/test_package_identifier.py — OSIDB affectedness / identify regression coverage

Bug fixes

Missing ADVISORY URL patterns / OSIDB in URL mining (b31b9d9): advisory/OSIDB URLs were missed when mining commit/advisory links for intel.
Weak hints continued into repo search (8ab9421): weak reference hints no longer keep searching repos; flow goes to the agent instead.
Noisy prompt regression (cafce28): removed extra prompt/intel fields that added noise and made the LLM less reliable.
“Patch file:” in report (47f8c01): dropped confusing “Patch file:” labeling from the checker report viewer.
Source Grep tool-call guidance (6c0576d): clarified grep tool instructions/schema so the LLM calls Source Grep correctly.
Identify regression (4132d2b): fixed PackageIdentifier affectedness regression after OSIDB support.

Also in this branch

OSIDB affectedness support in PackageIdentifier (55ac04d)
Stricter tool-error handling in observation/build agents so successful greps containing "error:" in source are not treated as tool failures

Test plan

Unit tests listed above
Local CVE run: Phoenix Initial_Intelligence_Gathering shows file_changes + CVE understanding
File-filtered Source Grep: observation span has focused_context_used=true
Spot-check known CVE verdicts for unexpected flips

vbelouso · 2026-07-13T07:41:19Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Code Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

zvigrinberg

@RedTanny Thank you for the PR.
Please see my comments.

zvigrinberg · 2026-07-13T08:31:11Z

    ReferenceHints,
    AdvisoryContent,
    GitSearchReport,
    CONFIDENCE_THRESHOLD_MIN,


@RedTanny This ( CONFIDENCE_THRESHOLD_MIN) is not being used at all.

@zvigrinberg
fixed
@RedTanny
Issue1 and issue 2 remove unused code

zvigrinberg · 2026-07-13T08:37:38Z

+            prompt = PATCH_ANALYSIS_PROMPT.format(
+                vuln_id=vuln_id,
+                vulnerability_type=result.cve_understanding.vulnerability_type if result.cve_understanding else "",
+                affected_component=result.cve_understanding.affected_component if result.cve_understanding else "",
+                root_cause=result.cve_understanding.root_cause if result.cve_understanding else "",
+                functions_touched_summary=functions_summary,
+                patch_content=patch_content,
+            )


@RedTanny This is not being used at all..

What is being used is at

https://github.com/RHEcosystemAppEng/exploit-iq-agent/pull/291/changes#diff-7e2d863e8e84f6d82687b61d6ae8e02d8abb5b2592f130abcc7df14d362fad07R620-R627

So it's either a duplication with redundant var , or worse even a bug.

@zvigrinberg
fixed
@RedTanny
Issue1 and issue 2 remove unused code

zvigrinberg · 2026-07-13T08:46:58Z

+
+        basename = Path(file_path).name.lower()
+        for fs in self.file_summaries:
+            if basename in fs.file_path.lower():


@RedTanny This is not safe enough, as for example both reasonable.c and sonable.c will match, while it's obvious these are two different files.
another example log.c prelog.c.
Maybe full check after transforming both sides to lowercases?

@zvigrinberg
fixed
@RedTanny
CodeReview 3: do exact compare

zvigrinberg · 2026-07-13T08:50:56Z

+            for fs in files:
+                # Per-file invocation to get structured output per file
+                file_prompt = PATCH_ANALYSIS_PROMPT.format(
+                    vuln_id=vuln_id,
+                    vulnerability_type=result.cve_understanding.vulnerability_type if result.cve_understanding else "",
+                    affected_component=result.cve_understanding.affected_component if result.cve_understanding else "",
+                    root_cause=result.cve_understanding.root_cause if result.cve_understanding else "",
+                    functions_touched_summary=format_functions_touched_summary([fs]),
+                    patch_content=_format_patch_content_for_files(parsed_patch, [fs]),
+                )
+                file_intel: PerFileIntel = await per_file_llm.ainvoke([SystemMessage(content=file_prompt)])
+                file_intel.file_path = fs.file_path  # Ensure path is set
+                results.append(file_intel)
+
+            span.set_output({
+                "files_analyzed": len(results),
+                "files": [r.file_path for r in results],
+            })
+            return results


@RedTanny Why running for each file LLM linearly ?
All files are independent, right?
So better IMO to create tasks for all files, and then run all tasks concurrently using asyncio.gather, it will improve performance and will make it much more rapid.

@zvigrinberg
What's happening:

Lines 608-620: sequential LLM calls per file inside run_patch_analysis()
Lines 630-633: batch1 and batch2 already run in parallel via asyncio.gather
So there's already some parallelism at the batch level. The suggestion is to add a second layer of parallelism inside each batch.

Concerns with blindly parallelizing:

Rate limiting — LLM APIs often have rate limits (requests/min, tokens/min). Firing all file calls simultaneously could trigger throttling or 429 errors.

Intentional backpressure — Sequential calls may be deliberate to avoid overwhelming the LLM service, especially if file count is variable (could be 2 files or 20).

Marginal gain — If typical file count per batch is small (2-4 files), the latency improvement may be minimal vs. added complexity.

because the above i tend to keep it as is

@RedTanny NP...
It's a consideration of favoring LLM Stability and reducing rate limits errors risks over concurrency efficiency

TamarW0 · 2026-07-20T10:36:26Z

/retest

exploit-iq-pac · 2026-07-20T10:36:30Z

Caution

There are some errors in your PipelineRun template.

PipelineRun	Error
vulnerability-analysis-on-pr	`CEL expression evaluation error: expression "event == \"pull_request\" &&\n!body.pull_request.draft &&\n(target_branch == \"main\" \|\| target_branch == \"rh-aiq-main\") &&\n(\"src/\".pathChanged() \|\| \"metrics_lib/\".pathChanged() \|\| \"pyproject.toml\".pathChanged() \|\| \"uv.lock\".pathChanged() \|\| \"Dockerfile\".pathChanged() \|\| \".dockerignore\".pathChanged())\n" failed to evaluate: no such key: pull_request`

TamarW0 · 2026-07-20T10:40:31Z

/test vulnerability-analysis-on-pr

etsien · 2026-07-21T17:31:49Z

Just FYI, with the multi-prompting framework coming, these prompts will be moved to centralized locations, with separate copies per model and if necessary, per environment (I know of Java-specific prompts for certain tool usage chains).

So please be aware of this coming change, and a big rebase/merge.

zvigrinberg · 2026-07-21T18:10:59Z

Just FYI, with the multi-prompting framework coming, these prompts will be moved to centralized locations, with separate copies per model and if necessary, per environment (I know of Java-specific prompts for certain tool usage chains).

So please be aware of this coming change, and a big rebase/merge.

@etsien But the RPM analysis is out of scope for multi repo... ( this could be a future enhancement POST-GA that if the TPA guys wants to prioritize it will happen). So it's expected to be quite smooth rebase.

… to agent

… and llm didn't follow prompt less accurate

Implements issues 01-06 of intel gathering refactor: - FileChangeSummary model and extraction (01) - file_changes span output (02) - CVE_UNDERSTANDING_PROMPT (03) - PATCH_ANALYSIS_PROMPT (04) - IntelGatheringResult object (05) - Dual-path intel gathering with enable_new_intel_flow config (06) Co-authored-by: Cursor <cursoragent@cursor.com>

…bservations

Co-authored-by: Cursor <cursoragent@cursor.com>

zvigrinberg · 2026-07-27T06:30:28Z

+            vulnerable_patterns=all_vulnerable_patterns[:10],
+            fix_patterns=all_fix_patterns[:10],
+            search_keywords=list(dict.fromkeys(all_keywords))[:10],


@RedTanny Why only 10 items?? if there are more it will ignore them

Hi @zvigrinberg
Keeping the top-10 cap intentionally — to_vulnerability_intel() feeds the flat thought prompt, and uncapped pattern/keyword lists blow the context window on large multi-file patches. Per-file focused observations already carry the fuller intel separately.
and also considering that we have in default only 10 iteration of ReAct loop.
this might change in the future on faster LLMs with bigger context window

zvigrinberg · 2026-07-27T06:46:49Z

-5. Do NOT call the same tool with the same input twice - CHECK KNOWLEDGE for prior calls.
-6. When patch is APPLIED: finding vulnerable code = GOOD, not finding fix = GOOD (expected).
-7. If a pattern contains special regex characters, escape them or use literal substrings.
+5. Source Grep uses comma as delimiter. If pattern contains commas (e.g., function calls), use ONLY the function name or target a specific file: 'funcName,file.c'. WRONG: 'foo(a, b)'. RIGHT: 'foo' or 'foo,file.c'.


@RedTanny This rule 5 ( source grep uses comma as delimiter...) appears exactly the same in different 4 prompts, better to refactor/extract it as a constant and then in the prompts, use a placeholder value that will be replaced with this constant in all of them ( change them to f-string, or using str.format() instead to assign the constant value instead of the placeholder, etc) before passing to LLM.

…l LLM calls. Steer forced-finish and empty-grep from FILE_CHANGES/identify signals, keep focused patch context out of NEW OUTPUT, and avoid cartesian comprehension that overflowed context on large single-file diffs. Co-authored-by: Cursor <cursoragent@cursor.com>

zvigrinberg · 2026-07-28T10:53:17Z

+    "Intercepted an error from tool",
+)
+# Source Grep query: pattern[,file_filter] — last comma segment is the file filter.
+GREP_QUERY_FILE_SEPARATOR = ","


@RedTanny Maybe this one and GREP_QUERY_FILE_SEPARATOR of cve_package_code_agent.py could be the same one ( impore it there from here)?

zvigrinberg · 2026-07-28T11:03:01Z

+            # Fallback: compute target_version_in_vulnerable_range from NVD if not set by Identify phase
+            if vulnerability_intel.target_version_in_vulnerable_range is None and fixed_ver:
+                try:
+                    from packaging import version as pkg_version


@RedTanny Are you sure it's the appropriate lib ( packaging.version) to compare the versions? AFAIK it's only tailored to python versions as per PEP-440.

zvigrinberg · 2026-07-28T11:07:35Z

+    normalized_line = _normalize_pattern_text(line)
+    if not normalized_line:
+        return False
+    return pattern in normalized_line or normalized_line in pattern


@RedTanny IMO It's too much permissive, it allows, for instance , to generic pattern like write to pass through if the line contains a much more specific method/function, like write_in_spool_if_ack , but it's clear that there is no affinity at all between these two.

zvigrinberg · 2026-07-28T11:15:39Z

+_GREP_PATTERN_FILE_PAIR_RE = re.compile(
+    r"(?:^|;)\s*([^;]+?)\s*,\s*([^,;]+?\.[A-Za-z0-9*_+-]+)\s*(?=;|$)"


@RedTanny If the C code pattern contains a . literally ( for example, structure' field access) This might be problematic.

RedTanny requested review from TamarW0 and zvigrinberg July 13, 2026 07:40

zvigrinberg added enhancement New feature or request optimization labels Jul 13, 2026

zvigrinberg requested changes Jul 13, 2026

View reviewed changes

RedTanny and others added 12 commits July 26, 2026 07:28

BugFix missing ADVISORY patterns and support for osidb in url minning

f9b2cb9

BugFix when weak hints are found don't continue to repo search but go…

5ff556b

… to agent

Bugfix: reggression bug caused when adding to prompt cause more nosie…

2d7d09f

… and llm didn't follow prompt less accurate

remove Patch file: from report confuses the viewer

b2810a2

make LLM know how to call the grep tool correctly

74e6db7

feat: add OSIDB affectedness support to PackageIdentifier

e5fe33a

fix regression issue identify

a079b22

feat: switch intel gathering to per-file 3-prompt flow with focused o…

d87ba78

…bservations

fix: align OSIDB NOTAFFECTED test with continue-to-verify status

f2a6b86

Co-authored-by: Cursor <cursoragent@cursor.com>

Issue1 and issue 2 remove unused code

b1c8ca6

CodeReview 3: do exact compare

97f5d75

RedTanny force-pushed the improve-first-thought branch from 2601f5a to 97f5d75 Compare July 26, 2026 08:21

fix rebase broke test fix it

a756ee6

zvigrinberg reviewed Jul 27, 2026

View reviewed changes

RedTanny and others added 3 commits July 27, 2026 17:39

CodeReview duplicate grep in prompt

cb48671

fix last issues

8d827a9

zvigrinberg reviewed Jul 28, 2026

View reviewed changes

		_GREP_PATTERN_FILE_PAIR_RE = re.compile(
		r"(?:^\|;)\s([^;]+?)\s,\s([^,;]+?\.[A-Za-z0-9_+-]+)\s*(?=;\|$)"

Uh oh!

Conversation

RedTanny commented Jul 13, 2026

Summary

Why

High-level feature (intel refactor)

Testing

Bug fixes

Also in this branch

Test plan

Uh oh!

vbelouso commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

zvigrinberg left a comment

Choose a reason for hiding this comment

Uh oh!

zvigrinberg Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zvigrinberg Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zvigrinberg Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TamarW0 commented Jul 20, 2026

Uh oh!

exploit-iq-pac Bot commented Jul 20, 2026

Uh oh!

TamarW0 commented Jul 20, 2026

Uh oh!

etsien commented Jul 21, 2026

Uh oh!

zvigrinberg commented Jul 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vbelouso commented Jul 13, 2026 •

edited

Loading

zvigrinberg Jul 13, 2026 •

edited

Loading

zvigrinberg Jul 13, 2026 •

edited

Loading

zvigrinberg Jul 13, 2026 •

edited

Loading