v3.4.3

[jkool702]

forkrun v3.4.3

Overview

This release focuses on three areas: (1) replacing the experimental MFD_HUGETLB path with proper Transparent Huge Page (THP) support and user-facing guidance, (2) distinguishing clean downstream-pipe-close exits from internal faults so that | head-style consumers no longer trigger spurious checkpoint generation, and (3) fixing two data-integrity bugs in the scanner/indexer paths that could silently corrupt or truncate output on delimiter-free or unterminated-final-line streams.

All changes have passed the full 4,212-test regression suite.

---

Breaking / Removed

• FORKRUN_USE_HUGETLB removed. The MFD_HUGETLB memfd flag and its associated environment variable have been removed. Hugetlbfs-backed huge pages conflict with the shmem semantics forkrun's memfd-backed files depend on (ftruncate, fallocate(PUNCH_HOLE) fine-grained resizing, arbitrary-offset sparse growth). The supported path to hugepage-backed throughput gains is now THP via /sys/kernel/mm/transparent_hugepage/shmem_enabled. The variable is no longer propagated into the cleanroom environment.

---

New Features

Transparent Huge Page (THP) support for ring-state mapping

• The shared ring-state mapping (g_state/state, a MAP_SHARED anonymous mapping backed by kernel shmem) is now madvise(MADV_HUGEPAGE)-hinted at init time, allowing it to be promoted to Shmem Transparent Huge Pages when shmem_enabled is advise or always.
• New startup diagnostic: check_thp_shmem_config() reads /sys/kernel/mm/transparent_hugepage/shmem_enabled at ring_init time. If the policy is never, a one-time notice is printed to stderr explaining the performance implications and recommending advise or always.
• New help section: A PERFORMANCE TIP: TRANSPARENT HUGE PAGES block has been added to frun --help, documenting:
  • The automatic MADV_HUGEPAGE hint on the ring-state mapping.
  • The much larger 50–60% throughput gain on -s/-b/-C modes that requires shmem_enabled=always (since memfd-backed I/O data is accessed via read/write/copy_file_range/sendfile, never mmap, and is therefore not covered by advise mode).
  • Explicit clarification that hugetlbfs-backed hugepages are not supported and are distinct from this setting.

Abort-reason tracking (ring_abort_reason)

• New loadable: ring_abort_reason [VAR] — queries why the global emergency abort was triggered.
  • Returns 0 (unset), 1 (downstream SIGPIPE / consumer closed pipe — clean exit), or 2 (internal fault — checkpoint needed).
• pull_fire_alarm() refactored into pull_fire_alarm_reason(uint8_t reason), which atomically records the reason (via the CAS-protected first-caller-wins pattern) before blasting EOF eventfds. A backward-compatible pull_fire_alarm() wrapper (reason = 2) is retained for all internal-fault call sites.
• New abort_reason field added to GlobalState (cache-line-aligned uint8_t), written once by the first pull_fire_alarm_reason caller and read by the Bash wrapper.
• Bash wrapper updated: The ABORT case in the poll reactor now calls ring_abort_reason to distinguish:
  • Reason 1 (SIGPIPE): Sets NORMAL_EXIT_FLAG=true, exits 0, no checkpoint generated. This is the correct behavior for frun ... | head -c 100 and similar downstream-closed-pipe scenarios.
  • Reason 2 or 0 (internal fault / unset): Sets NORMAL_EXIT_FLAG=false, exits 1, generates a checkpoint for --resume.
• All pull_fire_alarm() calls in ring_order_main (stdout-broken paths) and the ring_claim_main SIGPIPE sentry now use pull_fire_alarm_reason(1), since those represent downstream consumer disconnects, not internal faults.

---

Bug Fixes

NUMA indexer: delimiter-free chunk no longer slices open lines

File: ring_indexer_numa_main

Previously, when a chunk's entire search window contained no delimiter (i.e., a single logical line spans the whole chunk), the indexer defaulted actual_end to the chunk's own raw_offset. The consumer would then treat [prev_boundary, this_chunk_start) as a complete, delimiter-bounded batch, silently slicing a still-open line into fragments.

Now:

• If raw_length == 0 (genuine EOF sentinel): actual_end = raw_offset is emitted as before, so trailing unterminated data is correctly flushed as the final record.
• If raw_length > 0 (real chunk, no delimiter found): the previous chunk's resolved boundary is propagated forward, causing the consumer's existing actual_start >= actual_end skip path to defer this chunk (and any further no-delimiter chunks) until a real delimiter or true EOF is found.

This fix uses the new WAIT_FOR_META_READY macro to safely resolve the previous chunk's boundary.

UMA scanner: delimiter-free streams no longer truncate at one buffer ("4 MB cutoff" bug)

File: core_scanner_loop

Previously, when the UMA (flat) scanner encountered a buffer with no delimiters, it did not advance p past the scanned region. This caused current_p_offset to stall, making the scanner re-read the same chunk_sz window indefinitely and publish only the final buffer's worth of data at EOF — silently truncating all prior content.

Now p is always advanced to end after a delimiter-free scan, so current_p_offset tracks correctly across refills. lines_found is only incremented at true EOF (so pending_lines stays 0 and force_refill keeps firing until the ingest thread signals completion).

UMA scanner: trailing unterminated data now flushed at EOF

File: core_scanner_loop

Added a force-flush at UMA EOF (status == 1 && p >= end) when batch_start < final_off, emitting any remaining unflushed data as a single final batch. This matches the NUMA path's existing chunk_eof_flushed guard and ensures streams without a trailing delimiter are not silently dropped.

---

Internal / Refactoring

WAIT_FOR_META_READY macro

The spin-then-poll wait pattern for ChunkMeta::actual_end (previously hand-inlined in core_scanner_loop) has been extracted into a reusable macro:

WAIT_FOR_META_READY(out_var, meta_ptr, node_var, on_abort_stmt)

• Spins up to 10,000 iterations with cpu_relax(), then registers on the target node's meta_waiters count and blocks via poll() on the meta-ready eventfd.
• Correctly re-stores the freshly-read value into out_var on the post-registration double-check path (fixing a latent bug where the earlier hand-written copy could leave out_var holding a stale zero).
• Accepts a caller-supplied on_abort_stmt for emergency-abort handling (e.g., goto unified_scanner_eof or return EXECUTION_FAILURE).
• Used in both core_scanner_loop (replacing ~25 lines of inline code) and ring_indexer_numa_main (new no-delimiter handling).

Minor

• Removed MFD_HUGETLB #define (no longer used).
• Added MADV_HUGEPAGE #define guard.
• Whitespace normalization in struct IntervalNode.

---

Files Changed

File	Summary
forkrun_ring.c	Version bump, THP support, abort-reason tracking, WAIT_FOR_META_READY macro, NUMA indexer delimiter-free fix, UMA scanner delimiter-free + EOF-flush fixes, ring_abort_reason loadable
frun.bash	Version bump, removed FORKRUN_USE_HUGETLB propagation, added THP help section, updated ABORT handler to query ring_abort_reason

---

Testing

All changes validated against the full 4,212-test regression suite with zero failures.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[sourcery-ai]

Sorry @jkool702, your pull request is larger than the review limit of 150000 diff characters

[sonarqubecloud]

Quality Gate failed

Failed conditions
80.8% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud