fix PREP_SUBMISSION and UPDATE_SUBMISSION to stage batch_tsv and sample files as process inputs#361
Open
smanda4 wants to merge 3 commits into
Open
Conversation
…s work
PREP_SUBMISSION and UPDATE_SUBMISSION referenced file paths via a val-typed
meta/samples map (${meta.batch_tsv}, ${sample.fasta}, ${sample.gff},
${sample.fq1}, ${sample.fq2}, ${sample.nanopore}). Nextflow doesn't stage
files referenced from inside a val input, so on cloud executors like
Google Batch the paths resolve to raw workDir URIs
(/tostadas-work/<hash>/...) that don't exist inside the container. Both
modules fail with FileNotFoundError before running the submission scripts.
Fix: lift each per-sample file type out of the meta/samples map into an
explicit path input list. Nextflow stages each list into a per-type
subdirectory in the work dir; the script walks samples in the same order
as the upstream workflow emitted them and pops the next entry from each
file iterator when the sample declares that key. Same fix pattern as
915d2b3 for METADATA_VALIDATION's submission_config.
Also fixed a pre-existing bug where PREP_SUBMISSION checked
sample.get("nnp") but referenced sample.nanopore; the correct key is
"nanopore", so the previous "nnp" check was always false and nanopore
samples were silently omitted from sample_args.
Files:
modules/local/prep_submission/main.nf - tuple input now includes
staged batch_tsv + 5 path lists (fastas, gffs, fq1s, fq2s, nnps);
script uses iterators to zip samples with staged files
modules/local/update_submission/main.nf - tuple input includes
staged batch_tsv; script references \$batch_tsv
subworkflows/local/submission.nf - updated take comment
workflows/genbank.nf - submission_batch_ch emits
9-item tuple (fasta/gff populated; fq1/fq2/nnp empty)
workflows/biosample_and_sra.nf - same shape (fq1/fq2/nnp
populated; fasta/gff empty)
workflows/biosample_update.nf - rebatch_ch tuple now
includes tsv_file
Fixes cloud-executor blocker for genbank, biosample_and_sra, and
biosample_update workflows.
Nextflow binds a `path` input to a single Path object when it receives one file and to a List when it receives multiple. Calling `.iterator()` directly on a Path iterates its name segments instead of the file, so a single-item `fastas` came out as "fastas" (the staging directory name) and submission_prep.py hit `IsADirectoryError: 'gffs'` when it tried to open the value as a file. Wrap each input in an `asList` helper before iterating; empty lists stay empty and are never dereferenced because the sample map doesn't declare the corresponding key.
Reword the fix comments to avoid executor-specific and test-data-specific mentions: - Drop "on Google Batch" phrasing where the point applies to any cloud executor with a URI-shaped workDir - Replace a test-data-specific filename (AF266288_cleaned.fsa) with "the basename" so the example reads generically No functional change; comment-only cleanup.
smanda4
added a commit
to azpathogens/apgap-notebooks
that referenced
this pull request
Jul 8, 2026
The smanda4/tostadas fork was transferred to azpathogens/tostadas so it lives in the org rather than a personal account. GitHub redirects the old URL, but users are better served by the canonical org URL. Also adds a direct link to CDCgov/tostadas#361 (the pending upstream PR) in the intro markdown and parameter cell comment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Please Describe the Bug(s) Fixed and/or the Feature(s) Added:
Fixes
FileNotFoundErrorfailures inPREP_SUBMISSIONandUPDATE_SUBMISSIONon cloud executors (confirmed on Google Batch; applies to any executor with a URI-shaped workDir).Root cause
PREP_SUBMISSIONandUPDATE_SUBMISSIONreference file paths through a val-typed meta/samples map (${meta.batch_tsv},${sample.fasta}, etc.). Because those arevalinputs and notpathinputs, Nextflow doesn't stage the referenced files into the task container. On cloud executors the paths resolve to raw workDir URIs (/tostadas-work/<hash>/...) that don't exist inside the container, and both modules fail before the submission scripts ever run.Example failure on a cloud executor:
(where
/tostadas-work/is the workDir set innextflow.configfor the cloud executor)Fix
Lift each per-sample file type out of the meta/samples map into an explicit
pathinput list. Nextflow then stages each list into a per-type subdirectory in the work dir, and the script walks samples in the same order the upstream workflow emitted them, popping the next entry from each file iterator whenever the sample declares that key. Same pattern used in 915d2b3 forsubmission_configstaging inMETADATA_VALIDATION.Files touched:
modules/local/prep_submission/main.nf— tuple input now includes stagedbatch_tsvplus five path lists (fastas,gffs,fq1s,fq2s,nnps); script uses iterators to zip samples with staged filesmodules/local/update_submission/main.nf— tuple input includes stagedbatch_tsv; script references$batch_tsvsubworkflows/local/submission.nf— updatedtakecommentworkflows/genbank.nf—submission_batch_chnow emits a 9-item tupleworkflows/biosample_and_sra.nf— same shape (fq1/fq2/nnp populated instead of fasta/gff)workflows/biosample_update.nf—rebatch_chtuple includestsv_fileDrive-by fix
While in
PREP_SUBMISSIONI noticed the sample_args builder was checkingsample.get("nnp")but referencingsample.nanopore. Upstream workflows populate the"nanopore"key, so the"nnp"check was always false and nanopore samples were silently omitted fromsample_args. Fixed to check the correct"nanopore"key.Checklist
Go Through Checklist Below and Place A ✔️ (X Inside the Box) if Completed
General Checks
Have you run appropriate tests (unit/integration/end-to-end) to check logic across run environments (Conda/Docker/Singularity on Scicomp/AWS/NF Tower/Local)?
Validated end-to-end on Docker + Google Batch for the
genbankworkflow:AF266288, a public GenBank accession) completed cleanly.sqn,.tbl,.zip, VADR reportsprep_submission.logshowstable2asnran, biomol updated tocRNA, pub block stripped.sqnopens with a properSeq-submit ::= {ASN.1 header and populated contact/affiliation blocksDid not test Conda / Singularity / Scicomp / AWS / NF Tower / Local execution paths, nor the
biosample_and_sraorbiosample_updateworkflows (no biosample test data available in the environment). Those workflows use the same channel-wiring pattern asgenbankand the fix was applied identically, so they should work, but happy to gate on additional testing if the reviewer wants.Have you conducted proper linting procedures?
Have you updated existing documentation (README.md, etc.) or created new ones within docs?
Bug fix, not a new feature. Existing README doesn't need changes. Happy to add a "cloud executor support" note if the reviewer prefers.
CDC Checks
Did you check for sensitive data, and remove any?
Confirmed: no infrastructure identifiers, credentials, or environment-specific paths in the diff or PR body. Comments use generic examples.
If you added or modified HTML, did you check that it was 508 compliant?
N/A — no HTML in this PR.
Are additional approvals needed for this change? If so, please mention them below:
Not that I'm aware of, but please flag if there are.
Are there potential vulnerabilities or licensing issues with any new dependencies introduced? If so, please mention them below:
No new dependencies introduced.