Skip to content

fix(validator): treat "**" as recursive anywhere in a flow pattern#116

Open
jsonITP wants to merge 2 commits into
devicelab-dev:mainfrom
jsonITP:fix/recursive-glob-any-prefix
Open

fix(validator): treat "**" as recursive anywhere in a flow pattern#116
jsonITP wants to merge 2 commits into
devicelab-dev:mainfrom
jsonITP:fix/recursive-glob-any-prefix

Conversation

@jsonITP

@jsonITP jsonITP commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Problem

matchPattern in pkg/validator/validator.go only treats ** as recursive when the pattern is literally ** or starts with **/. Anything else (tests/**/*.yaml, auth/**, a/b/**/c.yaml) falls through to filepath.Glob, which does not understand ** and matches it like a single *. As a result each additional ** matches exactly one more directory level, and flows nested deeper than the number of ** tokens are silently skipped.

Repro against main on a fixture with files at depths 2, 3 and 4 under tests/:

Pattern in config.yaml Depths matched Discovered
tests/**/*.yaml 2 34
tests/**/**/*.yaml 3 +3
tests/**/**/**/*.yaml 4 +16

With this fix, tests/**/*.yaml on its own picks up files at every depth (53 in the same fixture), matching how the same glob behaves in bash globstar, doublestar, minimatch and every other tool authors have muscle-memory for.

Fix

  • matchPattern: route any pattern containing ** through collectRecursive, not just patterns that start with ** or **/.
  • collectRecursive: split the pattern on the first ** into an optional prefix and an optional suffix. The prefix (before **) is joined onto dir to form the walk root. The suffix (after **/) is matched per-segment against the trailing path segments of each file, using filepath.Match on each segment. This makes patterns like a/b/**/c/*.yaml behave as expected.

No new dependencies.

Backward compatibility

Patterns without ** are unchanged (they still go through filepath.Glob). The two existing recursive shapes (** and **/<name>) still resolve to the same set of files. Verified by running the existing pkg/validator test suite, which passes as-is:

  • TestValidate_RecursivePattern (pattern **)
  • TestValidate_RecursivePatternWithSuffix (pattern **/login*.yaml)
  • TestValidate_ConfigYamlFlowPatterns, TestValidate_NestedSubdirPattern, TestValidate_PatternMatchesDirectory (no **, unchanged code path)

Test coverage

Follow-up commit adds five tests plus the two prefix edge cases the reviewer flagged (missing prefix dir, wildcard in prefix). See the follow-up comment for the shape list.

matchPattern only treated "**" as recursive when the pattern was
literally "**" or started with "**/". Patterns with a directory
prefix — e.g. "tests/**/*.yaml" — fell through to filepath.Glob, which
does not understand "**" and matches it like a single "*", so each
extra "**" matched one more directory level instead of any depth.

Route any pattern containing "**" through collectRecursive, and split
the pattern on the first "**" into an optional prefix (walked from)
and an optional suffix (matched per-segment against each file's
trailing path).
@omnarayan

Copy link
Copy Markdown
Contributor

Thanks @jsonITP — great catch, and a really well-written PR. The silent depth-limited matching was a nasty one; flows quietly not running is the worst kind of bug for a test runner, and the repro table made it easy to verify. The zero-directory case (a/**/c.yaml matching a/c.yaml) behaving like globstar is exactly the muscle-memory behavior we want.

I've verified the fix and the existing suite passes. We'll merge this.

Two small edge cases we'll handle in a follow-up (or feel free to push to this branch if you'd like):

  1. A pattern whose prefix directory doesn't exist (e.g. subflows/**/*.yaml with no subflows/) should stay a silent no-match instead of failing the walk.
  2. A wildcard before the ** (e.g. flows-*/**/*.yaml) needs the prefix expanded through filepath.Glob before walking.

And yes — we'd gladly take the extra tests you offered; the two cases above would be great additions to that list.

Follow-up to the review on devicelab-dev#116:

  1. A pattern whose prefix directory doesn't exist (e.g. "subflows/**/*.yaml"
     with no subflows/) now returns an empty match instead of an lstat error
     from filepath.Walk.
  2. A shell wildcard in the prefix (e.g. "flows-*/**/*.yaml") is expanded
     through filepath.Glob before walking, so each matching directory is
     traversed.

Refactors collectRecursive to split into expandPrefixRoots (resolves the
prefix into concrete walk roots) and walkRecursive (per-root file walk).
Results are deduped so an ambiguous glob can't return the same file twice.

Adds five tests covering the shapes discussed on the PR:

  - tests/**/*.yaml at four depths
  - flows/** as a prefix-only pattern
  - a/**/c/*.yaml with a multi-segment suffix
  - subflows/**/*.yaml with a missing prefix dir (silent no-match)
  - flows-*/**/*.yaml with a wildcard prefix
@jsonITP

jsonITP commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review @omnarayan, pushed a follow-up commit that handles both edge cases plus the promised tests:

  • Missing prefix directory (subflows/**/*.yaml with no subflows/): expandPrefixRoots now os.Stats the joined path and returns an empty result set on IsNotExist, so the walk isn't invoked with a bad root.
  • Wildcard in the prefix (flows-*/**/*.yaml): when the prefix contains *, ?, or [, it's expanded via filepath.Glob and each matching directory becomes its own walk root. Results are deduped so overlapping matches can't count the same file twice.

Refactor: split collectRecursive into expandPrefixRoots (resolves the prefix to concrete roots) and walkRecursive (per-root file walk). Same public behavior, easier to reason about.

Added 5 tests in validator_test.go, all passing alongside the existing recursive-pattern tests:

  • TestValidate_RecursivePatternWithPrefix: tests/**/*.yaml at depths 2, 3, 4, and a sibling that must not leak
  • TestValidate_RecursivePatternPrefixOnly: flows/** matches every flow beneath the prefix
  • TestValidate_RecursivePatternMultiSegmentSuffix: a/**/c/*.yaml, verifies both included and excluded shapes
  • TestValidate_RecursivePatternMissingPrefixDir: subflows/**/*.yaml with no subflows/, expects 0 matches and no error
  • TestValidate_RecursivePatternWildcardPrefix: flows-*/**/*.yaml matches flows-alpha/** and flows-beta/**, ignores other-dir/

Ready for another look whenever.

@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 77.19298% with 13 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pkg/validator/validator.go 77.19% 7 Missing and 6 partials ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants