Devel branch - next development cycle

[pbuncic]

Summary of changes since 2026-06-10

Covers bits, lcg.bits (recipes), and bits-recipe-tools. Each entry tagged [Feature], [Fix], or [Improvement]; hashes are short SHAs.

build env derived from dependency modulefiles (now the default)

• [Feature] 34a672c --initdotsh-from-modules as a hashed build input (foundation; off-state byte-identical).
• [Feature] 77a6103 from-modules adds the modulefile-equivalent dev env (<PKG>_INCLUDE_DIR, Python site-packages) to init.sh.
• [Feature] e4e713d make --initdotsh-from-modules the default; add --legacy-initdotsh (aliBuild stays legacy, hashes byte-identical).
• [Feature] e2a6169 from-modules also exports CMAKE_PREFIX_PATH (:-separated, read natively by find_package).
• [Improvement] bits-recipe-tools e429a87 gate CMakeRecipe/BitsPython env reconstruction off under from-modules (this commit not landing in the built v0.0.28 briefly broke CMP0144-old packages; fixed by v0.0.29).
• [Improvement] lcg.bits d967f93 drop redundant dependency-env reconstruction; cd609fc drop redundant -DCMAKE_PREFIX_PATH; 93f997c torch_scatter/torch_sparse drop manual PYTHONPATH loop.
• [Improvement] 768bc44 / 900ad31 / 607013f env-diff harness comparing init.sh-derived vs modulefile-derived build env.

--builders scheduler & resource management

• [Feature] 6d64434 unleash the final (sink) package to full -j (default on; memory cap still applies).
• [Feature] 6dd1e4e history-driven critical-path scheduling (default on; --no-critical-path-schedule).
• [Fix] dc2fc07 macOS available-memory was under-reported (subtracted reclaimable inactive-anon), throttling heavy builds (ROOT to -j2 on 24 GB); prefer psutil, else reclaimable vm_stat buckets.
• [Fix] 70d3fca failure logs → LOGS/<arch>/ and 9e3d9cd per-arch bits_build_stats.json — stop different platforms sharing one work area from clobbering each other (stats were also semantically per-platform); b3a1e46/170ff5a bits stats reads the relocated file + docs.
• [Fix] 882ad22 resolve %(version)s in the build-order banner.
• [Improvement] 3162234 use threading.current_thread() (clear deprecation warnings).

Repository providers & init workflow (aliBuild vs native bits)

• [Feature] 2545a0c aliBuild front-end defaults to the legacy path, native bits to the provider path.
• [Feature] 847e8db aliBuild init (no PACKAGE) checks out the recipes (alidist) and exits; d3bc83e bits init <group>.bits checks out a recipe repo from the registry; 55fbd89 develop a package that lives in a required provider repo.
• [Fix] 73a145e load the bootstrap repo's required providers (e.g. alice.bits → alidist.bits — fixed "gsl not found").
• [Improvement] 621dded warn on provider version conflict; 4d92e95 point the "package not found" error at the provider mechanism; ee78182 docs.

CVMFS layout, merged views & relaxed reuse

• [Feature] 0632f28 / b70abac / b1dfc1d / e8d1222 / ea04075 merged symlink-farm view: one-entry-per-var env, opt-in enter/setenv --view, view-aware load/printenv + age-based GC, path remap (fixes PyROOT).
• [Feature] 58f13b6 / c2d99be / f59a547 / 53e91d4 / 80b6a08 published per-build_id views on CVMFS, bits publish --view, per-tree pre-publish primitive, CVMFS layout recorded in .meta.json.
• [Feature] 8f193fa→c52f5d7, cf19966 ADR-0001 import pipeline: modulefile harvest → classify → closure/build_id → overlay → bits import (build-sufficient from modulefiles).
• [Fix] fac7aef review fixes (command injection, partial-view, republish, path traversal, build_id match).
• [Improvement] 44a7ee1 docs for --reuse-policy/--reuse-base/--build-local.

Sync / remote store

• [Fix] 28c6989 upload freshly-built packages to --write-store (S3) when reading from a CVMFS remote (cross-backend DualRemoteSync; was silently dropped).
• [Fix] 8c21990 --aggressive-cleanup dropped the tarball the --write-store upload still needed.
• [Fix] 15f82e4 Boto3 tarball-name crash on specs carrying an architecture key.

Recipe hashing

• [Feature] 5f4a15e untracked_requires — link a dependency without folding it into the consumer's hash (edit a dep without rebuilding the stack above); 20e8ed6 cookbook example.

CLI robustness & bits.rc

• [Fix] fec3303 never exit non-zero without a message (silent-exit safety net for malformed defaults/recipes).
• [Fix] f53c6ee / 0e7abd5 restore search_path for single-package builds; 5dd6b22 use python3.
• [Improvement] a7d0a2f accept flat (header-less) bits.rc; a7afc92 CI README-path fix.

lcg.bits recipes (other) & bits-recipe-tools

• [Fix] 6f30d4f ROOT: use external bits zstd (-Dbuiltin_zstd=OFF) so ROOT 6.40 finds zdict.h.
• [Improvement] a11558a reduce ROOT mem_per_job to 1250; 05a4eef bump bits-recipe-tools version.
• [Improvement] bits-recipe-tools a441d2b ModuleRecipe guards lib/lib64/pkgconfig/site-packages path entries on existence.

[pbuncic]

If there are no objections, I plan to merge this with the main branch. The next iteration will focus on performance improvements (job scheduler), simplification (reducing the number of command-line flags and streamlining documentation for easier reading), and removing dead code (makeflow). After that, the focus would be on integration with the CVFS testbed and, eventually, with a real CVMFS repository to test the entire pipeline and measure performance.

[akritkbehera]

OK, I'll add the "untracked_requires:" field to the recipe, where you can list packages that you can change, and bits will recompile them, but not the rest of the DAG.

if a package isn't built yet and is mentioned in untracked_requires would that be built here?

[akritkbehera]

OK, I'll add the "untracked_requires:" field to the recipe, where you can list packages that you can change, and bits will recompile them, but not the rest of the DAG.

if a package isn't built yet and is mentioned in untracked_requires would that be built here?

All good it works for untracked_requires my setup was broken.

[akritkbehera]

If a package has this defined it doesn't fail it takes up the later requires field that is mentioned. Is this a designed to be that way? @pbuncic

package: demo
version: 1.2.3
requires:
- gcc
- zlib
requires:
- Python

[akritkbehera]

We can also get this merged https://github.com/bitsorg/bits/pull/101/changes

[pbuncic]

We can also get this merged https://github.com/bitsorg/bits/pull/101/changes

I was under the impression that you were OK with the already-provided ways to manage include files in bits. As I wrote earlier, I'd like to start cleaning up the code and adding more ways to do the same thing right now s not what I'd like to do unless you can justify why it has to be done exactly that way.

[pbuncic]

If a package has this defined it doesn't fail it takes up the later requires field that is mentioned. Is this a designed to be that way? @pbuncic

package: demo
version: 1.2.3
requires:
- gcc
- zlib
requires:
- Python

This is the result of merging defaults; the same key in the same defaults will override the first one. Why can't you do it with one require statement? If it helps, you can pass by to discuss what you are doing and if there is a better way of achieving the goal.

[akritkbehera]

We can also get this merged https://github.com/bitsorg/bits/pull/101/changes

I was under the impression that you were OK with the already-provided ways to manage include files in bits. As I wrote earlier, I'd like to start cleaning up the code and adding more ways to do the same thing right now s not what I'd like to do unless you can justify why it has to be done exactly that way.

We are OK with the header part as there already exists a way to !include to do it. In the recipe we can't source an script that has variables defined. So we definitely need this there. We have a Single Build Script for all Integration Builds that we build using SCRAM. We have a single build script that builds out IBs for GCC/Clang ARM/x86_64 LTO/Non-LTO Devel. Only the variables change sometimes like %(scram_compiler)s %(vectorization)s these and certain variables that define flags for nvcc, gcc or clang change. Ideally we want to keep a one script and have these variables defined in defaults which we will chain together to build our IBs.

[pbuncic]

We can also get this merged https://github.com/bitsorg/bits/pull/101/changes

I was under the impression that you were OK with the already-provided ways to manage include files in bits. As I wrote earlier, I'd like to start cleaning up the code and adding more ways to do the same thing right now s not what I'd like to do unless you can justify why it has to be done exactly that way.

We are OK with the header part as there already exists a way to !include to do it. In the recipe we can't source an script that has variables defined. So we definitely need this there. We have a Single Build Script for all Integration Builds that we build using SCRAM. We have a single build script that builds out IBs for GCC/Clang ARM/x86_64 LTO/Non-LTO Devel. Only the variables change sometimes like %(scram_compiler)s %(vectorization)s these and certain variables that define flags for nvcc, gcc or clang change. Ideally we want to keep a one script and have these variables defined in defaults which we will chain together to build our IBs.

The script already sources other scripts, it is a shell and you can do whatever you want. The issue is that variables in sourced script are not expanded. In principle, any variable defined in yaml header is also available as shell variable to the recipe (uppercase). Would this work for you?

[akritkbehera]

Yeah I do think that might work given that they do the same thing in the end. Will be rewrite of recipes but yes!

[akritkbehera]

maybe we could add a prefix for such env variables like VAR_****

[pbuncic]

maybe we could add a prefix for such env variables like VAR_****

I find that odd - $ already denotes a variable, and I prefer to use the same name as in the definition (some of the recipes in lcg.bits use this convention).

[akritkbehera]

On second thought I don't think that will work because we don't add variables to HASH after they resolved we HASH it. We can't add variables to HASH that would change the HASH of all packages even if they didn't use any of the mentioned variables.

[pbuncic]

On second thought I don't think that will work because we don't add variables to HASH after they resolved we HASH it. We can't add variables to HASH that would change the HASH of all packages even if they didn't use any of the mentioned variables.

I am afraid that I do not understand your case. Can you give me a concrete example (using your include syntax) and explain what you expect bits to do?

[akritkbehera]

package: integration_build_gcc
version: 12
variables:
  compiler: gcc
  enable_tools: rocm
---
%(##INCLUDE: cms.bits/scram-build.sh)s

package: integration_build_clang
version: 12
variables:
  compiler: clang
  enable_tools: rocm cuda
---
%(##INCLUDE: cms.bits/scram-build.sh)s

## FILE SCRAM_BUILD
useCompiler=%(compiler)s
extraTools=%(enable_tools)s

So in this example we stay contained to one scram-build.sh but as per our variables that are defined the bits will generate different build recipe by substitution useCompiler, extraTools.

[pbuncic]

bits way of doing it would be:

cat defaults-clang.sh defaults-gcc.sh integration_builds.sh
package: defaults-clang
version: "1"
variables: { compiler: clang, enable_tools: "rocm cuda" }
env:       { CC: clang, CXX: clang++ }
# Appended verbatim to the arch string; the leading '-' is the separator, (bits does not assume one)
append_arch: -clang
---

package: defaults-gcc
version: "1"
variables: { compiler: gcc,   enable_tools: rocm }
env:       { CC: gcc,  CXX: g++ }
# Appended verbatim to the arch string; the leading '-' is the separator, (bits does not assume one)
append_arch: -gcc15-dbg
---

package: integration_builds
version: "1"
expand_recipe: true        # optional but recommended
---
set -vx
useCompiler=%(compiler)s
extraTools=%(enable_tools)s
# .....

And then then

``
bits build --defaults gcc integration_build
bits build --defaults cms::clang integration_build # (if you want to have some common definitions in defaults-cms.sh)

Remember that defaults-release.sh is always processed and that you can chain many defaults using:: syntax.

[akritkbehera]

A. This prevents me from changing the name of package.
B. I can't see how this helps if I am going to add different Patches or Sources.

This basically assumes the entire integration package header always stays the same for all except for the variables part.

[pbuncic]

A. This prevents me from changing the name of package. B. I can't see how this helps if I am going to add different Patches or Sources.

This basically assumes the entire integration package header always stays the same for all except for the variables part.

Right, and what is wrong with that? If I understand well, this is sort of a metapackage anyway, so why are the patches needed here? They belong to the actual buildable software packages. Also, where is the problem in changing the package name? In stacks.bits I started by defining lcg.sh metapackage (long list of dependencies with some exclusions and customisation based on variables (cuda, osx...). After building everything I decided to split this is externals.sh, generators.sh and key4hep.sh - each metapackage built without need to rebuild individual packages that were already build while building lcg.sh.

[pbuncic]

OK, I finally committed this C-like #!include extension - allows to include verbatim another file before variables are expanded:

package: integration_build_gcc
version: 12
variables:
  compiler: gcc
  enable_tools: rocm
---
#!include  <cms.bits/scram-build.sh>

Hope this works for you.