Build EverParse against a F* binary package; configure via ./configure#292
Draft
tahina-pro wants to merge 21 commits into
Draft
Build EverParse against a F* binary package; configure via ./configure#292tahina-pro wants to merge 21 commits into
./configure#292tahina-pro wants to merge 21 commits into
Conversation
…_* env vars Add a ./configure script that takes dependency options as command-line arguments and produces config.Makefile, which deps.Makefile includes via a prerequisite-less rule that invokes ./configure with no argument by default. src/z3-version.Makefile now also includes config.Makefile (via $(EVERPARSE_SRC_PATH)/..) so the Z3 version can be overridden. Update package.sh, CI sparse-checkouts, .gitignore and README accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Using a pre-existing fstar.exe or krml no longer implies using the ambient opam switch. Users who need the ambient opam switch (e.g. to compile the EverCDDL F* plugin against their F*'s OCaml environment) must now pass --use-opamroot explicitly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The new --opamroot=PATH option (which implies --use-opamroot) records OPAMROOT := PATH in config.Makefile. deps.Makefile exports OPAMROOT in the opamroot branch so opam-env.sh uses it, letting users point EverParse at a dedicated opam root without setting OPAMROOT in the environment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Drop the `fstar.exe --ocamlenv` wrappers (and the pre-existing `export OCAMLPATH := $(EVERPARSE_HOME)/lib:...` line in the roundtrip test Makefile) so that the subsidiary Makefiles no longer adjust OCAMLPATH at all. fstar.lib (and evercddl.lib) are now resolved from the ambient environment (e.g. the opam switch) instead of being forced onto OCAMLPATH by F*'s --ocamlenv helper. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a --dice-home configure option so the Pulse DICE examples location can be set explicitly, which is required when building against an F* binary package (which does not ship the Pulse examples). configure emits DICE_HOME to config.Makefile when given; deps.Makefile keeps the from-source default and exports DICE_HOME. When DICE_HOME points at a non-existent directory, src/cddl/tests/dpe now populates it via a partial clone of FStarLang/FStar (treeless, shallow, sparse on the DICE examples subtree) at the commit reported by $(FSTAR_EXE) --version, so the examples match the F* in use. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The roundtrip test links against evercddl.lib (built by `make cddl` and installed under $EVERPARSE_HOME/lib). A previous change removed the OCAMLPATH entry pointing there along with the --ocamlenv wrappers, which made `make cddl-unit-tests` fail to find evercddl.lib. Add the EverCDDL lib path back to OCAMLPATH (fstar.lib stays resolved from the ambient environment). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The clean targets of share/everparse/tests/qd/unit and share/everparse/tests/lowparse were never reached by the root make clean, so their generated artifacts (qd unit extracted/, lowparse _output/ and example dirs) leaked. Add a clean rule (forwarding into unit) to the qd tests parent Makefile, and add clean-quackyducky-pulse-test and clean-lowparse-pulse-test rules to the top-level Makefile, appending both to clean_rules. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Commit a13f459 dropped the `fstar.exe --ocamlenv` wrappers from the subsidiary Makefiles on the assumption that fstar.lib would be resolved from the ambient opam switch. That holds for a binary/opam install, but not when EverParse builds F* from source under opt/: there fstar.lib lives under the F* installation (opt/FStar/out/lib), which is not on OCAMLPATH, so `dune build`/`ocamlfind` cannot find it and `ADMIT=1 make -k test` fails. Restore the --ocamlenv wrappers (which prepend F*'s lib dir to OCAMLPATH) on all five OCaml build/link commands in src/3d/ocaml, src/ASN1/ocaml, src/cddl/tool and src/cddl/tests/roundtrip. The roundtrip Makefile keeps its explicit OCAMLPATH entry for evercddl.lib (under $EVERPARSE_HOME/lib), which --ocamlenv does not provide. Verified: `ADMIT=1 make -j16 -k test` now completes with exit 0 (CBOR det/nondet 363/363, EverCDDL roundtrip dpe/cose_sign/cose_encrypt all succeed). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
These OCaml packages are required to build EverParse's own tools but were not listed in the dependency manifest: - menhir: used by src/qd/dune and src/3d/ocaml/dune (parser generators) - process: used by src/3d/ocaml/dune and src/ASN1/ocaml/dune Without them, building qd-exe, 3d-exe and asn1-test fails in a fresh opam switch (e.g. one prepared via 'fstar.exe --install_lib_with_deps', which only installs fstar.lib's own dependencies). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Introduce a `fstarlib.done` rule in deps.Makefile that runs `fstar.exe --install_lib_with_deps` (under the EverParse opam env) to install the F* application library (fstar.lib) and its OCaml dependencies into the configured opam switch, then touches a stamp file. It depends on $(NEED_FSTAR) so a source build rebuilds F* first; with a binary F* package, NEED_FSTAR is empty and the rule simply installs fstar.lib. Make the leaf rules that link F*-extracted OCaml against fstar.lib (3d-exe, asn1-test, cddl-tool, cddl-unit-tests) and the deps aggregate rule depend on fstarlib.done. Remove the stamp in distclean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rename .github/workflows/release.yml to release.yml.disabled so GitHub no longer processes it (only .yml/.yaml files are treated as workflows). This follows the repository's existing .disabled suffix convention (cf. opt/release.Makefile.disabled). The release process is being put on hold while the opam/F* dependency wiring is reworked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The rule that populates EverParse's local opam root (opt/Makefile's opam.done) previously installed both everparse-deps.opam and fstar-deps.opam. Since fstar-deps.opam is generated by copying F*'s own opam file, this forced a clone of the F* sources just to create the opam root — undesirable when F* is supplied as a binary package. Make opam.done install only everparse-deps.opam (EverParse's own OCaml tool dependencies), and drop the F*/karamel prerequisites from the deps.Makefile opam.done wrapper. F*'s OCaml build dependencies are now installed by the rule that builds F* from source (FStar.done), which generates fstar-deps.opam and runs `opam install --deps-only` before building. With an ambient opam root the user remains responsible for F*'s dependencies, matching the existing opam.done convention. Also add $(NEED_OPAM) to fstarlib.done so its opam mutation is serialized after opam-root population under `make -j`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ages Add two shortcut options to ./configure pointing EverParse at a F* binary package (from https://github.com/FStarLang/fstar-nightly): * --fstar-bin-package=DIR uses an already-extracted package rooted at DIR, equivalent to --fstar-exe=DIR/bin/fstar.exe --krml-exe=DIR/bin/krml. * --fstar-bin-package-tar=FILE extracts an unextracted package tarball under opt/fstar-bin-package/ (avoiding an opt/fstar vs opt/FStar collision on case-insensitive filesystems) and uses the result. Explicit --fstar-exe/--krml-exe still take precedence. Relative paths are resolved against the invocation directory. The extracted directory is gitignored and removed by distclean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously the $(FSTAR_EXE) --locate_z3 probe ran only when building F* from source. With a F* binary package (EVERPARSE_USE_FSTAR_EXE=1) the probe was skipped, so Z3 was looked up via 'which z3-VERSION' and, if absent from PATH, downloaded into opt/z3 -- whose rule clones the F* sources, defeating the purpose of the binary package. Move the probe to run in both modes, after FSTAR_EXE is determined. In source mode fstar.exe does not exist yet so the probe fails and we fall back to PATH/download as before; with a binary package it returns the Z3 bundled in the package, so no download or F* clone is needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
src/qd/dune lists 'batteries' as a library dependency of quackyducky.exe, but batteries was not declared in everparse-deps.opam. In a source build batteries is pulled in transitively by the F* build (fstar.opam) / fstar.lib, masking the missing dependency. With a F* binary package, FStar.done is empty and the qd-exe rule depends only on opam.done (not fstarlib.done), so batteries was never installed before 'dune build' ran, failing with "Library batteries not found". Declare batteries explicitly so opam.done installs it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The COSE interop and verifiedinterop test Makefiles hardcode CC := clang, so clang must be installed to run 'make test'. Add it to the list of additional testing packages. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tests EverParse using a downloaded F* binary package tarball (fstar*.tar.gz from fstar-nightly) via the new ./configure --fstar-bin-package-tar option, running `make -j$(nproc) -k lowparse` (which exercises the package's bundled Z3) followed by `ADMIT=1 make -j$(nproc) -k test`. Gitignore fstar-*.tar.gz so the binary package is not committed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On the
fstar2branch, EverParse builds F* (and, transitively, Karamel andPulse) from source:
deps.Makefileclones the F* repo underopt/, buildsit, downloads Z3, and threads everything together through a set of
EVERPARSE_*environment variables.F* now publishes binary packages (the nightly tarballs at
https://github.com/FStarLang/fstar-nightly), which bundle
fstar.exe, thekrml(Karamel) executable, the Z3 versions F* needs, and — as ofFStarLang/FStar#4300 —
fstar.libshipped as OCaml sources + a duneproject, installable via
fstar.exe --install_lib_with_deps.This PR makes EverParse able to build and test against such a binary package
(no F* source clone, no F* build, no Z3 download), while keeping the existing
build-F*-from-source path fully working. Along the way it replaces the
EVERPARSE_*environment-variable configuration with a./configurescriptthat records choices in a generated
config.Makefile.Configuration:
./configure+config.Makefile(replacesEVERPARSE_*env vars)./configurescript takes dependency options as command-linearguments and produces
config.Makefile, whichdeps.Makefileincludesvia a prerequisite-less rule that runs
./configurewith no arguments bydefault.
src/z3-version.Makefilealso includesconfig.Makefile, so the Z3version can be overridden there.
package.sh, CI sparse-checkouts,.gitignoreandREADME.mdare updated accordingly.--use-fstar-exe/--use-krml-exefrom the opam root. Supplyinga pre-existing
fstar.exe/krmlno longer implies using the ambient opamswitch. Users who need the ambient switch (e.g. to compile the EverCDDL F*
plugin against their F*'s OCaml environment) pass
--use-opamrootexplicitly.
--opamroot=PATHoption (implies--use-opamroot): recordsOPAMROOT := PATHinconfig.Makefile;deps.Makefileexports it soEverParse can target a dedicated opam root without setting
OPAMROOTin theenvironment.
F* binary-package shortcuts
Two
./configureshortcuts point EverParse at a F* binary package:--fstar-bin-package=DIR— use an already-extracted package rooted atDIR; equivalent to--fstar-exe=DIR/bin/fstar.exe --krml-exe=DIR/bin/krml.--fstar-bin-package-tar=FILE— extract a package tarball underopt/fstar-bin-package/(deliberately notopt/fstar, to avoid collidingwith
opt/FStaron case-insensitive filesystems) and use the result.Explicit
--fstar-exe/--krml-exestill take precedence; relative paths areresolved against the invocation directory; the extracted dir is gitignored and
removed by
distclean.Installing
fstar.libfor leaf buildsEverParse's 3d/ASN1/cddl tools link F*-extracted OCaml against
fstar.lib. In abinary package
fstar.libis shipped as source, so it must be built/installedinto the configured opam switch:
fstarlib.donerule runsfstar.exe --install_lib_with_deps(from Ship
fstar.libas source in binary packages, and addfstar.exe --install_lib[_with_deps]to install it FStarLang/FStar#4300) under the EverParse opam env. It depends on$(NEED_FSTAR), so a source build rebuilds F* first; with a binary packageNEED_FSTARis empty and the rule just installsfstar.lib. The leaf rulesthat link against
fstar.lib(3d-exe,asn1-test,cddl-tool,cddl-unit-tests) and thedepsaggregate now depend on it.opt/Makefile'sopam.donepreviously installed botheverparse-deps.opamandfstar-deps.opam(a copy of F*'s own opam file) — forcing a F* source clonejust to populate the opam root. It now installs only
everparse-deps.opam;F*'s OCaml build deps are installed by the from-source
FStar.donerule.a source build,
menhir/process/batterieswere pulled in via the F*build; with a binary package they are not, so they are now declared
explicitly in
everparse-deps.opam(used byqd,3d,ASN1tools).Z3 in binary-package mode
The
$(FSTAR_EXE) --locate_z3probe previously ran only in source mode. With abinary package the probe was skipped, so Z3 was looked up via
which z3-VERSIONand, if absent fromPATH, downloaded intoopt/z3— whose ruleclones the F sources*, defeating the purpose of the binary package. The probe
now runs in both modes after
FSTAR_EXEis determined: in source modefstar.exedoesn't exist yet so it falls back to PATH/download as before; witha binary package it returns the bundled Z3, so no download or F* clone
happens.
DICE_HOMEfor the Pulse DICE examplesBinary packages don't ship the Pulse examples that the CDDL DPE tests use. A new
--dice-homeconfigure option sets their location;deps.Makefilekeepsthe from-source default and exports
DICE_HOME. WhenDICE_HOMEpoints at anon-existent dir,
src/cddl/tests/dpepopulates it via a treeless/shallow/sparsepartial clone of
FStarLang/FStarat the commit reported byfstar.exe --version, so the examples match the F* in use.OCAMLPATHhandling in subsidiary MakefilesThe subsidiary OCaml build Makefiles previously forced
fstar.libontoOCAMLPATHviafstar.exe --ocamlenvwrappers. These were first removed (to letfstar.libresolve from the ambient opam switch), then restored, because asource build keeps
fstar.libunderopt/FStar/out/lib(not onOCAMLPATH) — so without the wrappersdune/ocamlfindcouldn't find it andmake testfailed. The roundtrip test additionally keeps an explicitOCAMLPATHentry forevercddl.lib(under$EVERPARSE_HOME/lib), which--ocamlenvdoes not provide.Build hygiene & CI
qd/unitandlowparsePulse-test clean rules into the top-levelmake clean(their generated artifacts previously leaked);distcleanremoves the new stamps and
opt/fstar-bin-package.release.yml→release.yml.disabled(only.yml/.yamlare processed as workflows;matches the repo's existing
.disabledconvention), while the opam/F*dependency wiring is reworked.
Testing the binary-package build:
fstarpkg.DockerfileNew
fstarpkg.Dockerfilebuilds and tests EverParse against a downloaded F*binary package tarball (
fstar*.tar.gz) via--fstar-bin-package-tar, runningmake -j$(nproc) -k lowparse(noADMIT, so it exercises the package's bundledZ3) followed by
ADMIT=1 make -j$(nproc) -k test.fstar-*.tar.gzis gitignoredso the package is not committed. README also notes that
clangis required (theCOSE interop test Makefiles hardcode
CC := clang).Files touched (binary-package work)
How to build against a binary package
Compatibility / before merging
./configurewith noarguments reproduces the old defaults, F* is still cloned/built under
opt/,Z3 is still downloaded, and
fstar.libresolves fromopt/FStar/out/lib.(TEMP) use my F* forkcommit(and any hash bumps to the fork) so
opt/points back at the upstream F*repo/commit; merge only after Ship
fstar.libas source in binary packages, and addfstar.exe --install_lib[_with_deps]to install it FStarLang/FStar#4300 lands and is reflected inopt/hashes.Makefile.