Pre-training crashes with "Too many open files" (EMFILE) on multi-core Macs

[gchindemi]

Summary

Self-supervised pre-training aborts with OSError: [Errno 24] Too many open files on macOS with ~16+ CPU cores. It dies at the first evaluation (end of
epoch 0), when the dev-split DataLoader workers spawn.

Root cause

_configure_dataloaders builds one persistent_workers=True DataLoader per
task, for both the train and dev splits. So an n-task run holds 2 × n_tasks
worker pools alive simultaneously for the whole run. With the 4 self-supervised
tasks (cons,order,shift,warp), that's 8 pools.

num_workers scales with core count, so the peak open-FD count grows with the
machine:

Cores	workers/loader	concurrent workers (4 train + 4 dev)	~peak FDs
8	2	16	~175
14	3	24	~255
16	4	32	~335
32	8	64	~655

macOS ships a default soft RLIMIT_NOFILE of 256 (launchctl limit maxfiles → 256). Training workers spawn first (fine), then the dev workers
spawn at the first _evaluate and push the process past 256 → os.pipe() fails
with EMFILE. This is why it "used to work": single-task fine-tuning only stacks
1–2 loaders, and Linux/SLURM defaults to a 1024+ soft limit.

LISBET never touches RLIMIT_NOFILE, so it inherits whatever the launching
shell provides. This also explains why it works from editor terminals (Zed,
VS Code, etc.) but not a plain Terminal.app session — editors raise the limit,
a login shell doesn't.

Workaround (sufficient)

Raising the open-file limit before launching is enough to fully resolve the
crash — no code change required:

ulimit -n 8192
betman train_model ...

Traceback

  File ".../lisbet/training/core.py", line 439, in train
    _evaluate(model, dev_dataloaders, dev_n_batches, tasks)
  ...
  File ".../torch/utils/data/dataloader.py", line 1169, in __init__
    w.start()
  ...
  File ".../multiprocessing/util.py", line 454, in spawnv_passfds
    errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files

Suggested fix

1. Bump RLIMIT_NOFILE at CLI startup (in app()) so users don't have to
   set ulimit manually. macOS's 256 default is the actual problem; LISBET
   legitimately needs a few hundred FDs. Best-effort, Unix-guarded, no-op when
   the limit is already high enough:

   def ensure_open_file_limit(minimum: int = 8192) -> None:
       try:
           import resource
       except ImportError:
           return  # Windows: no RLIMIT_NOFILE
       soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
       if soft >= minimum:
           return
       target = minimum if hard == resource.RLIM_INFINITY else min(minimum, hard)
       try:
           resource.setrlimit(resource.RLIMIT_NOFILE, (target, hard))
       except (ValueError, OSError) as exc:
           logging.warning("Could not raise open-file limit (%s); training "
                           "may hit 'Too many open files'.", exc)

2. (Optional, defense-in-depth) Make estimate_num_workers budget across all
   2 × n_tasks concurrent pools rather than per-loader. On a 32-core Mac the
   current logic spawns 64 workers for a batch_size=32 job — oversubscribed and
   FD-heavy. This complements (1) but doesn't replace it (16 cores still needs
   ~335 FDs > 256).

Environment

• macOS (Apple silicon), Python 3.12, observed on a 14-core machine at the 256
  edge; reproduces reliably on 16+ core Macs.
• Default ulimit -n = 256.