Release/4.0

[ArthurDeclercq]

MS²Rescore 4.0.0

This PR prepares the stable MS²Rescore 4.0.0 release. It consolidates the extensive feature-generation, prediction, rescoring, reporting, and configuration refactors developed through the 3.3 alpha series into a single stable release.

The version was increased to 4.0.0 because the release introduces substantial user-visible and breaking changes. Most notably, MS²Rescore now uses ristretto as its integrated rescoring implementation, centralizes spectrum annotation, adopts the latest DeepLC, IM2Deep, MS²PIP, and ms2rescore-rs APIs, and substantially revises its configuration and output files.

Breaking changes

• Percolator and mokapot removed as rescoring engines. Ristretto is now the only rescoring
  engine, and rescoring always runs. The rescoring_engine option has been removed and replaced
  by the rescoring configuration block, with train_fdr and model (svm or lda) options.
  Removes both the external percolator binary dependency and the mokapot dependency.
• DeepLC v4 functional API, IM2Deep v2 functional API (dataset-wide processing,
  per-run linear CCS calibration, IMS fine-tuning folded in), ms2rescore-rs 0.5 API
  (score_ms2_spectra, centralized annotate_spectra() — every PSM's spectrum is now
  annotated once before feature generators run, not per generator).
• ionmob feature generator removed (superseded by IM2Deep v2). maxquant feature generator
  removed (its features folded into the ms2 generator).
• MS2PIPFeatureGenerator/MS2FeatureGenerator: ms2_tolerance, spectrum_path,
  spectrum_id_pattern, mass_mode, processes params removed — fragment tolerance is now set
  globally via top-level tolerance_value/tolerance_mode (default 0.02 Da, fixing a prior
  fallback bug where it silently defaulted to 20 ppm).
• lower_score_is_better removed — search-engine score direction is now inferred automatically
  per run through spectrum-competed target-decoy evaluation.
• fasta_file, write_rescoring_tables, and the mokapot-specific write_weights, write_txt,
  and protein_kwargs options removed. Rescoring result tables are always written, and
  protein-level results use ristretto's picked-protein competition when id_decoy_pattern is
  configured. While top-level fasta_file option and mokapot-based FASTA protein inference have been removed, this does not affect Mumble's nested psm_generator.mumble.fasta_file option, which is used when validating amino-acid-combination candidates.
• Rescoring can no longer be skipped: rescoring: null is rejected by config validation instead
  of being silently ignored.
• Main PSM output renamed from <prefix>.psms.tsv to <prefix>.tsv; crash-recovery output renamed
  to <prefix>.intermediate.tsv.
• ms2rescore.utils is no longer part of the public API. General internal helpers now reside in
  ms2rescore._utils, with ristretto-specific integration helpers in
  ms2rescore._ristretto_utils.

Changed

• Python 3.11 or newer is now required.
• DeepLC retraining is disabled by default, while HTML report generation is enabled by default.
• max_psm_rank_output > 1 now applies consistently across the main output, rescoring tables,
  and report. Multiple ranked PSMs are retained per spectrum, with q-values and PEPs evaluated
  over the retained rows rather than through full spectrum competition. This mode is intended
  for inspecting ambiguous candidates, such as Mumble candidates, and should not be interpreted
  as a statistically rigorous spectrum-level FDR-controlled count.
• Only post-rescoring result tables are persisted. Pre-rescoring results are reconstructed from
  provenance information in the main PSM TSV, including during standalone report generation.
• DeepLC retraining is disabled by default, while HTML report generation is enabled by default.

New

• ms2 feature generator (Rust, ms2rescore_rs): direct-spectrum features (matched ion
  counts/percentages, hyperscore, intensity ratios), ion series expanded to a/b/c/x/y/z.
• Ristretto rescoring integration through ristretto-ms, with semi-supervised SVM and faster LDA
  models, configurable training FDR, run-aware spectrum competition, and PSM-, peptidoform-,
  peptide-, and protein-level result tables.
• Automatic inference of search-engine score direction, grouped by run so repeated native
  spectrum IDs across multiple input files do not interfere with the inferred direction.
• New top-level report_fdr option controlling console identification counts, HTML report
  statistics and charts, and FlashLFQ filtering.
• ms2rescore-report --fdr option for regenerating reports at a different FDR threshold without
  rerunning feature generation or rescoring.
• • Mumble is now available as an optional extra (ms2rescore[mumble], >=0.3.0) for exploring
    mass-shift candidate peptidoforms pre-rescoring, with a full user-guide page and corrected
    config schema. Mumble remains beta software.
• Configurable IM2Deep reference datasets in CSV, compressed CSV, or Parquet format, containing
  peptidoform and CCS columns.
• Intermediate-file recovery (<prefix>.intermediate.tsv on error) and skip-if-already-present
  feature generators, so a crashed run can resume without recomputing everything.
• Rescoring result tables (<prefix>.psms.tsv, .peptidoforms.tsv, .peptides.tsv,
  .proteins.tsv, .weights.tsv) are always written as plain TSV files, independent of
  rescoring engine internals.
• GUI controls for the ristretto model, centralized fragmentation model, and fragment tolerance;
  GUI runs now also write an HTML log file (<prefix>.log.html), matching the CLI.
• Full per-feature-generator test suite (basic/deeplc/im2deep/ms2/ms2pip), written to catch
  realistic regressions, not just detect change, together with end-to-end rescoring,
  report-generation, and ristretto integration tests.
• Mumble-generated candidates whose matched-ion percentage falls more than 50% below that of the
  corresponding original PSM are removed before rescoring.
• Unknown or removed feature-generator names are rejected before feature generation with a clear
  configuration error listing the available generators.

Fixed

• MS²PIP/DeepLC features silently cross-assigned between PSMs sharing a spectrum ID when
  max_psm_rank_input > 1 (missing per-PSM annotation / missing sort_index() after q-value
  sort).
• Multi-run MS²PIP crash (intensities length X != peak count Y): observed-spectrum dedup keyed
  on spectrum_id alone, which Sage's scan=N IDs repeat across runs — now keyed on
  (run, spectrum_id) (fixed upstream in ms2pip 4.2.0, pulled in here).
• basic feature generator: missing precursor m/z or search-engine score produced NaN instead
  of the documented 0 (None not in ... check never fired against float64 NaN arrays).
• DeepLC multitask model was locked to prediction head 0 regardless of which head best matched
  the run; now selects the best head per run.
• Mumble mass-shift candidates were leaking into DeepLC/IM2Deep RT/CCS calibration-set selection
  (each candidate inherits the original PSM's score/q-value via deepcopy, so calibration
  couldn't tell an unconfirmed candidate from the confirmed hit) — now filtered out via
  get_original_hit_mask().
• Configured IM2Deep reference datasets incorrectly raised FileNotFoundError because the
  file-existence check was inverted.
• Q-value NaN check crashed on None values; processes=-1 (our default) was passed straight to
  DeepLC's num_threads, which requires a positive int or None; numpy 2.0 compatibility
  (np.trapz → np.trapezoid).
• Empty or partial rescoring dictionaries raised KeyError instead of falling back to
  ristretto defaults.
• Ristretto training metadata was discarded during final evaluation, producing empty feature
  weight and iteration outputs.
• Provenance booleans read from TSV were interpreted incorrectly when represented as strings,
  and reconstructing pre-rescoring results could mutate the original PSM objects.
• Report and identification-overlap comparisons collided across runs that reused the same
  spectrum_id; these now consistently key on (run, spectrum_id).
• psm_id_pattern uniqueness validation incorrectly treated identical spectrum IDs in separate
  runs as collisions.
• GUI configuration did not correctly propagate the selected log level, centralized fragment
  annotation settings, or the DeepLC epochs option.
• GUI runs never wrote an HTML log file (<prefix>.log.html), unlike CLI runs — the GUI's
  logging setup only ever attached a plain text-file handler.

Docs

New Mumble user-guide page, corrected config_schema.md/.json (previous Mumble defaults were
wrong and missing several real parameters), "Optional dependencies" section in
installation.rst, "Configuring PSM generators" section in configuration.rst, tie-break
caveat for max_psm_rank_output: 1 with Mumble enabled (candidates can score identically;
selection isn't a meaningful tie-break), and updated configuration and migration documentation
for ristretto, centralized spectrum annotation, output file changes, report_fdr, and automatic
score-direction inference.

Dependencies

Dependencies finalized to ms2pip>=4.2.0, ms2rescore_rs>=0.5.0, im2deep>=2.0.1,
ristretto-ms>=0.3.0, and optional mumble>=0.3.0. deeplc>=4.0.0b1 continues to allow the
DeepLC v4 beta rather than blocking on a stable release. Added pyarrow>=14.