Conversation
…limit The process-files workflow fans out ALL update-descriptions sub-workflows from a single parent step. For large projects with many entities/relationships, this exceeds OpenWorkflow's hardcoded step limit (1000), causing the parent workflow to crash before finalization. This introduces a two-level fan-out pattern: 1. A new process-descriptions-groups workflow spawns multiple update-descriptions sub-workflows internally (bounded by DESCRIPTION_BATCHES_PER_GROUP=50) 2. The process-files workflow now spawns process-descriptions-groups sub-workflows instead of directly spawning all update-descriptions With 50 batches per group, a project with 3511 description batches would spawn 71 groups (~71 steps in parent) instead of 3511 direct calls, staying well under the 1000 step limit. Data is safe during the crash - entities and relationships are fully processed, only the status update is missing.
The group workflow was doing its own internal grouping logic, but each call already receives pre-sliced IDs (one group's worth). It should simply batch them for update-descriptions calls.
Contributor
|
| Filename | Overview |
|---|---|
| apps/worker/workflows/process-descriptions-group-spec.ts | New workflow spec defining the process-descriptions-groups intermediate workflow with retry policy and schema; exports DESCRIPTION_BATCHES_PER_GROUP = 50 constant used by both the parent and group workflow. |
| apps/worker/workflows/process-descriptions-group.ts | New group workflow that re-chunks received entity/relationship IDs and spawns update-descriptions sub-workflows; uses Promise.allSettled and correctly throws on any batch failure. |
| apps/worker/workflows/process-file.ts | Replaces direct update-descriptions fan-out with a two-level fan-out via processDescriptionsGroupsSpec; uses Promise.allSettled with correct failedGroups.length > 0 guard that propagates any partial failure. |
| apps/worker/worker.ts | Adds dynamic import and ow.implementWorkflow registration for processDescriptionsGroups; no issues. |
Reviews (9): Last reviewed commit: "fix: throw on any description group fail..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
added 6 commits
June 23, 2026 15:16
…tency Follow the same pattern as file fan-out: call step.runWorkflow directly at the workflow level instead of wrapping in step.run. This matches the established pattern in the codebase and addresses Greptile P1 feedback.
…group Replace sequential for...of/await loops with Promise.all to avoid multiplying description-phase latency by up to 50x for large projects. Addresses Greptile 3/5 feedback — previously all batches in a group ran serially, making the two-level fan-out counterproductive.
The new processDescriptionsGroups workflow must be registered with OpenWorkflow via ow.implementWorkflow() to execute. Without it, all spawned group tasks queue indefinitely. Addresses Greptile 2/5 finding.
Consistent with the established pattern elsewhere in the codebase. Promise.allSettled waits for all sub-workflows before returning, avoiding premature rejection on a single batch failure.
…Groups Promise.allSettled swallows rejections silently. Add explicit failure detection after allSettled to surface batch failures instead of silently proceeding with incomplete graph data. Consistent with the error-handling pattern used elsewhere in the codebase.
Replace the every(...rejected) guard with some(...rejected) so that if even one group fails, the error propagates and the graph is not marked "ready" with incomplete descriptions. Regressed from the original Promise.all behaviour which would surface any single batch failure upstream.
Contributor
|
Want your agent to iterate on Greptile's feedback? Try greploops. |
m43i
approved these changes
Jun 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix the
process-filesworkflow exceeding OpenWorkflow's hardcoded step limit (1000) on large projects by implementing a two-level fan-out pattern.Type of Change
Changes Made
process-descriptions-groupworkflow that spawns multipleupdate-descriptionssub-workflows internally (bounded to 50 batches per group)process-filesworkflow to spawnprocess-descriptions-groupssub-workflows instead of directly spawning allupdate-descriptionssub-workflowsTechnical Details
The issue occurs because
process-fileswas fanning out ALL child workflows from a single parent step:process-filesub-workflow per fileupdate-descriptionssub-workflow per description batchFor large projects (e.g., 498 files, 68,421 entities + 43,882 relationships → ~3,511
update-descriptionsbatches), this exceeds the 1000 step limit.The two-level fan-out adds an intermediate
process-descriptions-groupsworkflow:process-files) spawns groups (each group = 1 step in parent)Related Issues
Closes #488
Testing
Checklist
make generateif SQL queries were modified (backend)