Skip to content

Build EverParse against a F* binary package; configure via ./configure#292

Draft
tahina-pro wants to merge 21 commits into
project-everest:fstar2from
tahina-pro:_taramana_fstar2_binary_package
Draft

Build EverParse against a F* binary package; configure via ./configure#292
tahina-pro wants to merge 21 commits into
project-everest:fstar2from
tahina-pro:_taramana_fstar2_binary_package

Conversation

@tahina-pro

Copy link
Copy Markdown
Member

Motivation

On the fstar2 branch, EverParse builds F* (and, transitively, Karamel and
Pulse) from source: deps.Makefile clones the F* repo under opt/, builds
it, downloads Z3, and threads everything together through a set of
EVERPARSE_* environment variables.

F* now publishes binary packages (the nightly tarballs at
https://github.com/FStarLang/fstar-nightly), which bundle fstar.exe, the
krml (Karamel) executable, the Z3 versions F* needs, and — as of
FStarLang/FStar#4300fstar.lib shipped as OCaml sources + a dune
project
, installable via fstar.exe --install_lib_with_deps.

This PR makes EverParse able to build and test against such a binary package
(no F* source clone, no F* build, no Z3 download), while keeping the existing
build-F*-from-source path fully working. Along the way it replaces the
EVERPARSE_* environment-variable configuration with a ./configure script
that records choices in a generated config.Makefile.

Depends on FStarLang/FStar#4300, which adds fstar.exe --install_lib[_with_deps]
and makes binary packages ship fstar.lib as source. EverParse's
fstarlib.done rule and the binary-package workflow rely on that command.

Configuration: ./configure + config.Makefile (replaces EVERPARSE_* env vars)

  • New ./configure script takes dependency options as command-line
    arguments and produces config.Makefile, which deps.Makefile includes
    via a prerequisite-less rule that runs ./configure with no arguments by
    default. src/z3-version.Makefile also includes config.Makefile, so the Z3
    version can be overridden there. package.sh, CI sparse-checkouts,
    .gitignore and README.md are updated accordingly.
  • Decoupled --use-fstar-exe/--use-krml-exe from the opam root. Supplying
    a pre-existing fstar.exe/krml no longer implies using the ambient opam
    switch. Users who need the ambient switch (e.g. to compile the EverCDDL F*
    plugin against their F*'s OCaml environment) pass --use-opamroot
    explicitly.
  • New --opamroot=PATH option (implies --use-opamroot): records
    OPAMROOT := PATH in config.Makefile; deps.Makefile exports it so
    EverParse can target a dedicated opam root without setting OPAMROOT in the
    environment.

F* binary-package shortcuts

Two ./configure shortcuts point EverParse at a F* binary package:

  • --fstar-bin-package=DIR — use an already-extracted package rooted at
    DIR; equivalent to --fstar-exe=DIR/bin/fstar.exe --krml-exe=DIR/bin/krml.
  • --fstar-bin-package-tar=FILE — extract a package tarball under
    opt/fstar-bin-package/ (deliberately not opt/fstar, to avoid colliding
    with opt/FStar on case-insensitive filesystems) and use the result.

Explicit --fstar-exe/--krml-exe still take precedence; relative paths are
resolved against the invocation directory; the extracted dir is gitignored and
removed by distclean.

Installing fstar.lib for leaf builds

EverParse's 3d/ASN1/cddl tools link F*-extracted OCaml against fstar.lib. In a
binary package fstar.lib is shipped as source, so it must be built/installed
into the configured opam switch:

  • New fstarlib.done rule runs fstar.exe --install_lib_with_deps
    (from Ship fstar.lib as source in binary packages, and add fstar.exe --install_lib[_with_deps] to install it FStarLang/FStar#4300) under the EverParse opam env. It depends on
    $(NEED_FSTAR), so a source build rebuilds F* first; with a binary package
    NEED_FSTAR is empty and the rule just installs fstar.lib. The leaf rules
    that link against fstar.lib (3d-exe, asn1-test, cddl-tool,
    cddl-unit-tests) and the deps aggregate now depend on it.
  • Decoupled opam-root creation from the F sources.* opt/Makefile's
    opam.done previously installed both everparse-deps.opam and
    fstar-deps.opam (a copy of F*'s own opam file) — forcing a F* source clone
    just to populate the opam root. It now installs only everparse-deps.opam;
    F*'s OCaml build deps are installed by the from-source FStar.done rule.
  • Declare the OCaml deps that the F build used to provide transitively.* In
    a source build, menhir/process/batteries were pulled in via the F*
    build; with a binary package they are not, so they are now declared
    explicitly in everparse-deps.opam (used by qd, 3d, ASN1 tools).

Z3 in binary-package mode

The $(FSTAR_EXE) --locate_z3 probe previously ran only in source mode. With a
binary package the probe was skipped, so Z3 was looked up via which z3-VERSION and, if absent from PATH, downloaded into opt/z3 — whose rule
clones the F
sources
*, defeating the purpose of the binary package. The probe
now runs in both modes after FSTAR_EXE is determined: in source mode
fstar.exe doesn't exist yet so it falls back to PATH/download as before; with
a binary package it returns the bundled Z3, so no download or F* clone
happens.

DICE_HOME for the Pulse DICE examples

Binary packages don't ship the Pulse examples that the CDDL DPE tests use. A new
--dice-home configure option sets their location; deps.Makefile keeps
the from-source default and exports DICE_HOME. When DICE_HOME points at a
non-existent dir, src/cddl/tests/dpe populates it via a treeless/shallow/sparse
partial clone of FStarLang/FStar at the commit reported by fstar.exe --version, so the examples match the F* in use.

OCAMLPATH handling in subsidiary Makefiles

The subsidiary OCaml build Makefiles previously forced fstar.lib onto
OCAMLPATH via fstar.exe --ocamlenv wrappers. These were first removed (to let
fstar.lib resolve from the ambient opam switch), then restored, because a
source build keeps fstar.lib under opt/FStar/out/lib (not on
OCAMLPATH) — so without the wrappers dune/ocamlfind couldn't find it and
make test failed. The roundtrip test additionally keeps an explicit
OCAMLPATH entry for evercddl.lib (under $EVERPARSE_HOME/lib), which
--ocamlenv does not provide.

Build hygiene & CI

  • Wire the qd/unit and lowparse Pulse-test clean rules into the top-level
    make clean (their generated artifacts previously leaked); distclean
    removes the new stamps and opt/fstar-bin-package.
  • Disable the release workflow by renaming release.yml
    release.yml.disabled (only .yml/.yaml are processed as workflows;
    matches the repo's existing .disabled convention), while the opam/F*
    dependency wiring is reworked.

Testing the binary-package build: fstarpkg.Dockerfile

New fstarpkg.Dockerfile builds and tests EverParse against a downloaded F*
binary package tarball (fstar*.tar.gz) via --fstar-bin-package-tar, running
make -j$(nproc) -k lowparse (no ADMIT, so it exercises the package's bundled
Z3) followed by ADMIT=1 make -j$(nproc) -k test. fstar-*.tar.gz is gitignored
so the package is not committed. README also notes that clang is required (the
COSE interop test Makefiles hardcode CC := clang).

Files touched (binary-package work)

 .github/workflows/{ci,nightly,nightly-fstar2}.yml   # config.Makefile sparse-checkout
 .github/workflows/release.yml -> release.yml.disabled
 .gitignore                                          # config.Makefile, fstarlib.done, fstar-*.tar.gz, opt/fstar-bin-package
 Makefile                                            # fstarlib.done deps for leaf rules; pulse-test clean rules
 README.md                                           # ./configure usage, binary-package options, clang
 configure                                           # NEW: the configure script
 deps.Makefile                                       # config.Makefile include; fstarlib.done; Z3 probe; DICE_HOME; opam.done decoupling
 fstarpkg.Dockerfile                                 # NEW: binary-package test image
 opt/Makefile                                        # opam.done installs only everparse-deps.opam
 opt/everparse-deps.opam                             # + menhir, process, batteries
 share/everparse/tests/qd/Makefile                   # clean forwarding
 src/cddl/tests/dpe/Makefile                         # DICE_HOME partial-clone rule
 src/package/package.sh                              # config.Makefile awareness
 src/z3-version.Makefile                             # include config.Makefile (Z3 version override)

How to build against a binary package

# extracted package:
./configure --fstar-bin-package=/path/to/fstar
# or an unextracted tarball:
./configure --fstar-bin-package-tar=/path/to/fstar-Linux-x86_64.tar.gz

make -j$(nproc) -k lowparse        # exercises the bundled Z3
ADMIT=1 make -j$(nproc) -k test

Compatibility / before merging

tahina-pro and others added 21 commits June 1, 2026 22:38
…_* env vars

Add a ./configure script that takes dependency options as command-line
arguments and produces config.Makefile, which deps.Makefile includes via
a prerequisite-less rule that invokes ./configure with no argument by
default. src/z3-version.Makefile now also includes config.Makefile (via
$(EVERPARSE_SRC_PATH)/..) so the Z3 version can be overridden. Update
package.sh, CI sparse-checkouts, .gitignore and README accordingly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Using a pre-existing fstar.exe or krml no longer implies using the
ambient opam switch. Users who need the ambient opam switch (e.g. to
compile the EverCDDL F* plugin against their F*'s OCaml environment)
must now pass --use-opamroot explicitly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The new --opamroot=PATH option (which implies --use-opamroot) records
OPAMROOT := PATH in config.Makefile. deps.Makefile exports OPAMROOT in
the opamroot branch so opam-env.sh uses it, letting users point EverParse
at a dedicated opam root without setting OPAMROOT in the environment.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Drop the `fstar.exe --ocamlenv` wrappers (and the pre-existing
`export OCAMLPATH := $(EVERPARSE_HOME)/lib:...` line in the roundtrip
test Makefile) so that the subsidiary Makefiles no longer adjust
OCAMLPATH at all. fstar.lib (and evercddl.lib) are now resolved from the
ambient environment (e.g. the opam switch) instead of being forced onto
OCAMLPATH by F*'s --ocamlenv helper.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a --dice-home configure option so the Pulse DICE examples location
can be set explicitly, which is required when building against an F*
binary package (which does not ship the Pulse examples). configure emits
DICE_HOME to config.Makefile when given; deps.Makefile keeps the
from-source default and exports DICE_HOME.

When DICE_HOME points at a non-existent directory, src/cddl/tests/dpe
now populates it via a partial clone of FStarLang/FStar (treeless,
shallow, sparse on the DICE examples subtree) at the commit reported by
$(FSTAR_EXE) --version, so the examples match the F* in use.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The roundtrip test links against evercddl.lib (built by `make cddl` and
installed under $EVERPARSE_HOME/lib). A previous change removed the
OCAMLPATH entry pointing there along with the --ocamlenv wrappers, which
made `make cddl-unit-tests` fail to find evercddl.lib. Add the EverCDDL
lib path back to OCAMLPATH (fstar.lib stays resolved from the ambient
environment).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The clean targets of share/everparse/tests/qd/unit and
share/everparse/tests/lowparse were never reached by the root
make clean, so their generated artifacts (qd unit extracted/,
lowparse _output/ and example dirs) leaked.

Add a clean rule (forwarding into unit) to the qd tests parent
Makefile, and add clean-quackyducky-pulse-test and
clean-lowparse-pulse-test rules to the top-level Makefile,
appending both to clean_rules.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Commit a13f459 dropped the `fstar.exe --ocamlenv` wrappers from the
subsidiary Makefiles on the assumption that fstar.lib would be resolved
from the ambient opam switch. That holds for a binary/opam install, but
not when EverParse builds F* from source under opt/: there fstar.lib
lives under the F* installation (opt/FStar/out/lib), which is not on
OCAMLPATH, so `dune build`/`ocamlfind` cannot find it and
`ADMIT=1 make -k test` fails.

Restore the --ocamlenv wrappers (which prepend F*'s lib dir to OCAMLPATH)
on all five OCaml build/link commands in src/3d/ocaml, src/ASN1/ocaml,
src/cddl/tool and src/cddl/tests/roundtrip. The roundtrip Makefile keeps
its explicit OCAMLPATH entry for evercddl.lib (under $EVERPARSE_HOME/lib),
which --ocamlenv does not provide.

Verified: `ADMIT=1 make -j16 -k test` now completes with exit 0 (CBOR
det/nondet 363/363, EverCDDL roundtrip dpe/cose_sign/cose_encrypt all
succeed).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
These OCaml packages are required to build EverParse's own tools but were
not listed in the dependency manifest:
  - menhir: used by src/qd/dune and src/3d/ocaml/dune (parser generators)
  - process: used by src/3d/ocaml/dune and src/ASN1/ocaml/dune

Without them, building qd-exe, 3d-exe and asn1-test fails in a fresh opam
switch (e.g. one prepared via 'fstar.exe --install_lib_with_deps', which
only installs fstar.lib's own dependencies).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Introduce a `fstarlib.done` rule in deps.Makefile that runs
`fstar.exe --install_lib_with_deps` (under the EverParse opam env) to
install the F* application library (fstar.lib) and its OCaml dependencies
into the configured opam switch, then touches a stamp file. It depends on
$(NEED_FSTAR) so a source build rebuilds F* first; with a binary F*
package, NEED_FSTAR is empty and the rule simply installs fstar.lib.

Make the leaf rules that link F*-extracted OCaml against fstar.lib
(3d-exe, asn1-test, cddl-tool, cddl-unit-tests) and the deps aggregate
rule depend on fstarlib.done. Remove the stamp in distclean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rename .github/workflows/release.yml to release.yml.disabled so GitHub no
longer processes it (only .yml/.yaml files are treated as workflows). This
follows the repository's existing .disabled suffix convention (cf.
opt/release.Makefile.disabled). The release process is being put on hold
while the opam/F* dependency wiring is reworked.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The rule that populates EverParse's local opam root (opt/Makefile's
opam.done) previously installed both everparse-deps.opam and
fstar-deps.opam. Since fstar-deps.opam is generated by copying F*'s own
opam file, this forced a clone of the F* sources just to create the opam
root — undesirable when F* is supplied as a binary package.

Make opam.done install only everparse-deps.opam (EverParse's own OCaml
tool dependencies), and drop the F*/karamel prerequisites from the
deps.Makefile opam.done wrapper. F*'s OCaml build dependencies are now
installed by the rule that builds F* from source (FStar.done), which
generates fstar-deps.opam and runs `opam install --deps-only` before
building. With an ambient opam root the user remains responsible for
F*'s dependencies, matching the existing opam.done convention.

Also add $(NEED_OPAM) to fstarlib.done so its opam mutation is serialized
after opam-root population under `make -j`.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ages

Add two shortcut options to ./configure pointing EverParse at a F*
binary package (from https://github.com/FStarLang/fstar-nightly):

* --fstar-bin-package=DIR uses an already-extracted package rooted at
  DIR, equivalent to --fstar-exe=DIR/bin/fstar.exe --krml-exe=DIR/bin/krml.

* --fstar-bin-package-tar=FILE extracts an unextracted package tarball
  under opt/fstar-bin-package/ (avoiding an opt/fstar vs opt/FStar
  collision on case-insensitive filesystems) and uses the result.

Explicit --fstar-exe/--krml-exe still take precedence. Relative paths
are resolved against the invocation directory. The extracted directory
is gitignored and removed by distclean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously the $(FSTAR_EXE) --locate_z3 probe ran only when building F*
from source. With a F* binary package (EVERPARSE_USE_FSTAR_EXE=1) the
probe was skipped, so Z3 was looked up via 'which z3-VERSION' and, if
absent from PATH, downloaded into opt/z3 -- whose rule clones the F*
sources, defeating the purpose of the binary package.

Move the probe to run in both modes, after FSTAR_EXE is determined. In
source mode fstar.exe does not exist yet so the probe fails and we fall
back to PATH/download as before; with a binary package it returns the
Z3 bundled in the package, so no download or F* clone is needed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
src/qd/dune lists 'batteries' as a library dependency of
quackyducky.exe, but batteries was not declared in everparse-deps.opam.
In a source build batteries is pulled in transitively by the F* build
(fstar.opam) / fstar.lib, masking the missing dependency. With a F*
binary package, FStar.done is empty and the qd-exe rule depends only on
opam.done (not fstarlib.done), so batteries was never installed before
'dune build' ran, failing with "Library batteries not found".

Declare batteries explicitly so opam.done installs it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The COSE interop and verifiedinterop test Makefiles hardcode CC := clang,
so clang must be installed to run 'make test'. Add it to the list of
additional testing packages.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tests EverParse using a downloaded F* binary package tarball
(fstar*.tar.gz from fstar-nightly) via the new
./configure --fstar-bin-package-tar option, running
`make -j$(nproc) -k lowparse` (which exercises the package's bundled
Z3) followed by `ADMIT=1 make -j$(nproc) -k test`.

Gitignore fstar-*.tar.gz so the binary package is not committed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tahina-pro

Copy link
Copy Markdown
Member Author

fstarpkg.Dockerfile works with the F* binary package obtained from https://github.com/tahina-pro/FStar/actions/runs/26922219139

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant