Skip to content

fix PREP_SUBMISSION and UPDATE_SUBMISSION to stage batch_tsv and sample files as process inputs#361

Open
smanda4 wants to merge 3 commits into
CDCgov:feature/measles-vadrfrom
azpathogens:feature/measles-vadr
Open

fix PREP_SUBMISSION and UPDATE_SUBMISSION to stage batch_tsv and sample files as process inputs#361
smanda4 wants to merge 3 commits into
CDCgov:feature/measles-vadrfrom
azpathogens:feature/measles-vadr

Conversation

@smanda4

@smanda4 smanda4 commented Jul 7, 2026

Copy link
Copy Markdown

Description

Please Describe the Bug(s) Fixed and/or the Feature(s) Added:

Fixes FileNotFoundError failures in PREP_SUBMISSION and UPDATE_SUBMISSION on cloud executors (confirmed on Google Batch; applies to any executor with a URI-shaped workDir).

Root cause

PREP_SUBMISSION and UPDATE_SUBMISSION reference file paths through a val-typed meta/samples map (${meta.batch_tsv}, ${sample.fasta}, etc.). Because those are val inputs and not path inputs, Nextflow doesn't stage the referenced files into the task container. On cloud executors the paths resolve to raw workDir URIs (/tostadas-work/<hash>/...) that don't exist inside the container, and both modules fail before the submission scripts ever run.

Example failure on a cloud executor:

submission_prep.py --metadata_file /tostadas-work/<task-hash>/genbank/batch_1.tsv ...
FileNotFoundError: '/tostadas-work/<task-hash>/genbank/batch_1.tsv'

(where /tostadas-work/ is the workDir set in nextflow.config for the cloud executor)

Fix

Lift each per-sample file type out of the meta/samples map into an explicit path input list. Nextflow then stages each list into a per-type subdirectory in the work dir, and the script walks samples in the same order the upstream workflow emitted them, popping the next entry from each file iterator whenever the sample declares that key. Same pattern used in 915d2b3 for submission_config staging in METADATA_VALIDATION.

Files touched:

  • modules/local/prep_submission/main.nf — tuple input now includes staged batch_tsv plus five path lists (fastas, gffs, fq1s, fq2s, nnps); script uses iterators to zip samples with staged files
  • modules/local/update_submission/main.nf — tuple input includes staged batch_tsv; script references $batch_tsv
  • subworkflows/local/submission.nf — updated take comment
  • workflows/genbank.nfsubmission_batch_ch now emits a 9-item tuple
  • workflows/biosample_and_sra.nf — same shape (fq1/fq2/nnp populated instead of fasta/gff)
  • workflows/biosample_update.nfrebatch_ch tuple includes tsv_file

Drive-by fix

While in PREP_SUBMISSION I noticed the sample_args builder was checking sample.get("nnp") but referencing sample.nanopore. Upstream workflows populate the "nanopore" key, so the "nnp" check was always false and nanopore samples were silently omitted from sample_args. Fixed to check the correct "nanopore" key.

Checklist

Go Through Checklist Below and Place A ✔️ (X Inside the Box) if Completed

General Checks

  • Have you run appropriate tests (unit/integration/end-to-end) to check logic across run environments (Conda/Docker/Singularity on Scicomp/AWS/NF Tower/Local)?

    Validated end-to-end on Docker + Google Batch for the genbank workflow:

    • Dry-run on single-sample test data (AF266288, a public GenBank accession) completed cleanly
    • Output artifacts confirmed: .sqn, .tbl, .zip, VADR reports
    • prep_submission.log shows table2asn ran, biomol updated to cRNA, pub block stripped
    • Generated .sqn opens with a proper Seq-submit ::= { ASN.1 header and populated contact/affiliation blocks

    Did not test Conda / Singularity / Scicomp / AWS / NF Tower / Local execution paths, nor the biosample_and_sra or biosample_update workflows (no biosample test data available in the environment). Those workflows use the same channel-wiring pattern as genbank and the fix was applied identically, so they should work, but happy to gate on additional testing if the reviewer wants.

  • Have you conducted proper linting procedures?

    • Block comments added at each change explaining the WHY (why files need to be lifted out of val, why iterator coercion is needed)
    • Variable naming matches existing tostadas conventions
    • Numpy docstrings / class-formatting rules don't apply (Nextflow / Groovy, no Python touched)
  • Have you updated existing documentation (README.md, etc.) or created new ones within docs?

    Bug fix, not a new feature. Existing README doesn't need changes. Happy to add a "cloud executor support" note if the reviewer prefers.

CDC Checks

  • Did you check for sensitive data, and remove any?

    Confirmed: no infrastructure identifiers, credentials, or environment-specific paths in the diff or PR body. Comments use generic examples.

  • If you added or modified HTML, did you check that it was 508 compliant?

    N/A — no HTML in this PR.

Are additional approvals needed for this change? If so, please mention them below:

Not that I'm aware of, but please flag if there are.

Are there potential vulnerabilities or licensing issues with any new dependencies introduced? If so, please mention them below:

No new dependencies introduced.

smanda4 added 3 commits July 7, 2026 10:54
…s work

PREP_SUBMISSION and UPDATE_SUBMISSION referenced file paths via a val-typed
meta/samples map (${meta.batch_tsv}, ${sample.fasta}, ${sample.gff},
${sample.fq1}, ${sample.fq2}, ${sample.nanopore}). Nextflow doesn't stage
files referenced from inside a val input, so on cloud executors like
Google Batch the paths resolve to raw workDir URIs
(/tostadas-work/<hash>/...) that don't exist inside the container. Both
modules fail with FileNotFoundError before running the submission scripts.

Fix: lift each per-sample file type out of the meta/samples map into an
explicit path input list. Nextflow stages each list into a per-type
subdirectory in the work dir; the script walks samples in the same order
as the upstream workflow emitted them and pops the next entry from each
file iterator when the sample declares that key. Same fix pattern as
915d2b3 for METADATA_VALIDATION's submission_config.

Also fixed a pre-existing bug where PREP_SUBMISSION checked
sample.get("nnp") but referenced sample.nanopore; the correct key is
"nanopore", so the previous "nnp" check was always false and nanopore
samples were silently omitted from sample_args.

Files:
  modules/local/prep_submission/main.nf     - tuple input now includes
    staged batch_tsv + 5 path lists (fastas, gffs, fq1s, fq2s, nnps);
    script uses iterators to zip samples with staged files
  modules/local/update_submission/main.nf   - tuple input includes
    staged batch_tsv; script references \$batch_tsv
  subworkflows/local/submission.nf          - updated take comment
  workflows/genbank.nf                      - submission_batch_ch emits
    9-item tuple (fasta/gff populated; fq1/fq2/nnp empty)
  workflows/biosample_and_sra.nf            - same shape (fq1/fq2/nnp
    populated; fasta/gff empty)
  workflows/biosample_update.nf             - rebatch_ch tuple now
    includes tsv_file

Fixes cloud-executor blocker for genbank, biosample_and_sra, and
biosample_update workflows.
Nextflow binds a `path` input to a single Path object when it receives
one file and to a List when it receives multiple. Calling `.iterator()`
directly on a Path iterates its name segments instead of the file, so a
single-item `fastas` came out as "fastas" (the staging directory name)
and submission_prep.py hit `IsADirectoryError: 'gffs'` when it tried to
open the value as a file. Wrap each input in an `asList` helper before
iterating; empty lists stay empty and are never dereferenced because
the sample map doesn't declare the corresponding key.
Reword the fix comments to avoid executor-specific and test-data-specific
mentions:
- Drop "on Google Batch" phrasing where the point applies to any cloud
  executor with a URI-shaped workDir
- Replace a test-data-specific filename (AF266288_cleaned.fsa) with
  "the basename" so the example reads generically

No functional change; comment-only cleanup.
smanda4 added a commit to azpathogens/apgap-notebooks that referenced this pull request Jul 8, 2026
The smanda4/tostadas fork was transferred to azpathogens/tostadas so it
lives in the org rather than a personal account. GitHub redirects the
old URL, but users are better served by the canonical org URL. Also
adds a direct link to CDCgov/tostadas#361 (the pending upstream PR) in
the intro markdown and parameter cell comment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant