feat: replace false CoDel queues with dummynet FQ-CoDel (#532)#6
Open
thedancingdeveloper wants to merge 98 commits into
Open
feat: replace false CoDel queues with dummynet FQ-CoDel (#532)#6thedancingdeveloper wants to merge 98 commits into
thedancingdeveloper wants to merge 98 commits into
Conversation
…rosAndOnesLLC#535) Run the whole target config through the same converters and engine validators the apply path uses, before the first DELETE. A config that would abort mid-apply is now rejected with 400 and zero mutations, instead of relying on snapshot rollback. Covers all entry paths (restore, import, commit-confirm rollback, cluster sync) via checks in both apply_firewall_config and the rollback wrapper. validate_nat_rule and AliasEngine::validate_name become public so the restore path reuses the exact add-time checks rather than duplicates.
…erosAndOnesLLC#535) apply_firewall_config now runs every database mutation — all section clears and re-inserts (rules, NAT, aliases, static routes, geo-IP, WireGuard, IPsec SAs+tunnels, auth, shaping, TLS, HA, DHCP) — on one sqlx transaction. Any DB-stage failure rolls the whole database back automatically; the DB can no longer end up half-restored. pf/kernel/ service applies run strictly after commit, with the snapshot wrapper as their recovery path. Engines gain executor-generic insert variants (insert_*_on) following the existing insert_rule_on/log_on pattern; public add_* methods delegate to them, so validation lives in one place. AliasEngine gains sync_all_strict so post-commit pf table sync stays a required step. Per-row audit entries during restore are replaced by one ConfigChanged row committed atomically with the restore itself.
…AndOnesLLC#535) After the restore transaction commits, verify committed row counts per table match what the restore inserted, then — after the data-plane applies — verify the pf anchors hold exactly the rulesets the engines rendered. RuleEngine/NatEngine/GeoIpEngine gain verify_applied(), with RuleEngine's renderer factored out of apply_rules so apply and verify can't drift. A verification failure surfaces as a required-step error, which the snapshot wrapper turns into a rollback.
…AndOnesLLC#535) commit_confirm_arm_with_snapshot now parses and pre-validates the snapshot before arming — a corrupt rollback target is refused up front instead of failing silently when the timer fires. The expiry path logs loudly instead of skipping an unparseable snapshot. The OPNsense importer aborts before applying when the caller requested commit_confirm but the pre-apply snapshot capture failed, rather than importing without the safety net the caller asked for.
…#535) PfMock gains armed-failure injection (fail_op/clear_fail) and the PfBackend trait gains as_any() so tests reach it through the shared Arc. New table-driven suite injects a failure at each of the 21 DB stages (table swapped for a read-only view so pre-tx migrates can't undo the injection) plus pf-op failures, and asserts all three contract outcomes: abort with prior rows intact, rollback with audit row, and rollback-failed audited when the failure hits both directions.
…erosAndOnesLLC#535) pfctl -sr returns rules in canonical re-rendered form (and omits table definitions), so exact string equality with the loaded source only holds on the mock backend. PfBackend gains echoes_exact_rules(); verification does the exact vector compare on backends that echo the loaded strings and falls back to the emptiness invariant on real pf. Full pfctl-side verification stays tracked under ZerosAndOnesLLC#533. Caught by the FreeBSD functional harness (t05-restore-roundtrip).
…estore-hardening Backup/restore hardening: pre-validation, single DB transaction, post-apply verification (ZerosAndOnesLLC#535)
…tatic-wan-optional-gateway fix: allow isolated static WAN seeds without a gateway
…f-disabled-boot-heal fix: re-enable PF when boot leaves it disabled
…freebsd-enhanced-coverage test: extend FreeBSD packet, API, and boot coverage
…l minisign (ZerosAndOnesLLC#542) - Commit the minisign public key and compile it into the updater via include_str!, so appliances upgraded from builds that never shipped /usr/local/etc/aifw/update-signing.pub still enforce signatures; the on-disk copy is provisioned/refreshed at install time for operators. - Auto-install minisign via the pkg sudo helper when verification runs on an appliance whose running updater predates the manifest entry; a missing/unrunnable minisign now surfaces as VerifyUnavailable instead of a misleading Download error. - release.sh (the local publish gate) signs every *.sha256 with ~/.minisign/aifw-update.key, verifies against the committed public key, refuses to publish unsigned, and uploads .minisig sidecars plus update-signing.pub. - CI: pipe MINISIGN_PASSWORD to minisign stdin (it never read the env var); verify signatures against the committed key in both the sign step and the release job; drop vars.MINISIGN_PUBLIC_KEY provisioning. - SBOM scans the source tree (Cargo.lock, aifw-ui/package-lock.json) instead of xz blobs syft cannot open; SBOM added to provenance subjects. - Remove stale uncompressed-image .sha256 files after xz so unverifiable checksums are never signed or published. - Document key management, rotation, and compromise response in freebsd/RELEASE-SIGNING.md.
…nat64-af-to feat(nat): real NAT64/NAT46 cross-family translation via pf af-to + DNS64
Two windows let untested bits reach the stable channel field appliances auto-pull from (/releases/latest): - build-iso CI published a plain v* tag as STABLE immediately; only -rc/-beta/-alpha tags got the pre-release flag. Now every CI release is a pre-release; promotion to stable is a deliberate manual step (gh release edit <tag> --prerelease=false --latest). - release.sh's release-exists path uploaded assets first and applied the pre-release flag after, so fresh assets briefly sat on a stable release. The flag is now applied strictly before the upload; the stable flip for AIFW_RELEASE_FINAL stays after the upload so a half-uploaded asset set is never marked stable.
…lease-from-start fix(release): pre-release gate applies before any asset is public
Pulls in rDNS#87 (RFC 6147 DNS64 synthesis incl. CNAME-chain chasing + RDNS_LOCK_PATH override) and rDNS#88 (pre-release gate in release.sh). Upstream range 7bd12e2..04d5975 reviewed — all commits from the ZerosAndOnesLLC#531 batch. Appliance builds now ship a DNS64-capable rdns, completing the NAT64 + DNS64 workflow.
…p-rdns-dns64 chore: bump rDNS pin to 04d5975 (DNS64)
Minor bump for the ZerosAndOnesLLC#531 feature batch: real NAT64/NAT46 cross-family translation (pf af-to, FreeBSD 15+), DNS64 in the bundled rDNS (dns64/dns64_prefix wiring end-to-end), NAT page + CLI UX for cross-family rules, ICMPv6 ND pass in the generated pf.conf, and the release pre-release gate. rDNS pinned at 04d5975 (DNS64-capable).
…5.110.0 release: v5.110.0
- IDS alert retention now runs regardless of engine mode: the worker lived inside start(), which the binary skips when IDS is disabled — so disabled-IDS appliances never pruned (2.97M rows in the field, including year-9920 clock-skew rows that age-based pruning can never remove). The binary now spawns it unconditionally. - aifw-sudo-write allowlist gains /usr/local/etc/aifw/pf.conf.aifw — its absence broke pf-tuning's boot apply on every appliance. Guard test asserts every code write target is allowlisted. - watchdog uses /usr/local/sbin/visudo (bare 'visudo' isn't in the daemon(8) PATH, so the sudoers refresh never ran and misreported the failure as a validation error) - ensure_rdr_anchor also self-heals the ICMPv6 ND pass rule into upgraded pf.conf.aifw files, before the first filter anchor - memstats heartbeat caches the 24h ids_alerts count (refresh every 10th tick) instead of a >1s COUNT per minute on large tables - traffic page: stable interface ordering (cards and colors jumped with the WS payload order) and deterministic tie-breaks on the top-talkers/ports lists
…uard CI's test_restart_driver_refreshes_sudoers caught the watchdog change — and rightly flagged that aifw-restart.sh has the identical bare-visudo PATH bug. Both scripts now resolve /usr/local/sbin/visudo explicitly, and the guard test pins the absolute-path form so a bare name can't come back.
…ppliance-sweep fix: appliance post-upgrade sweep findings
Patch release: the ZerosAndOnesLLC#601 appliance sweep fixes — IDS alert retention now runs regardless of engine mode (scrubs the multi-million-row tables and year-9920 clock-skew rows in the field), aifw-sudo-write allowlists pf.conf.aifw so pf-tuning's boot apply works, absolute visudo path in the watchdog/restart sudoers refresh, ICMPv6 ND rule self-heal for upgraded appliances, cached heartbeat alert count, stable traffic-page graph ordering.
…5.110.1 release: v5.110.1
Builds run as root, release.sh runs unprivileged (operator gh auth + minisign key) — so minisign died with Permission denied writing the .minisig files next to root-owned checksums, on every release. - build-local.sh / build-update.sh chown their output to SUDO_USER at the end, handing the artifacts to the operator who will sign them - release.sh preflights writability of the output dir and self-heals older builds' artifacts with sudo -n chown (never prompts; dies with the exact chown command when it can't). Logic verified on FreeBSD (find -writable is GNU-only, so it's a portable test -w loop).
…se-output-ownership fix(release): root-owned build artifacts no longer break signing
Warnings (only visible compiling on FreeBSD — Linux dev/CI never builds the cfg(freebsd) code): docs for the system_apply freebsd_impl functions and the bpf/netmap capture modules; four dead 'use std::process::Command' imports in aifw-setup (all usages are fully qualified). UI restart hangs (seen live installing v5.110.1 from the Updates page): - handleLocalInstall had no .catch and never showed the bounce overlay — when 'restart after install' is set and the API dies before the POST response flushes (restart.log shows aifw_api needing SIGKILL), the fetch never settles and the spinner hangs forever. Failures now surface, and a network-failure-with-restart (or success with restart) hands off to watchRestart's overlay + reload polling. - handleConfirmRestart no longer blocks the overlay on the restart POST completing: an ApiError still reports, but a lost response (API bounced first) proceeds to the overlay, which polls its way back.
The FreeBSD verification pass caught two bare Command::new uses inside create_service_user's cfg block (beyond the range my earlier scan checked) and a fifth dead import — cfg(freebsd) code is invisible to Linux checks, so the errors only surfaced on the build VM.
…ention default (ZerosAndOnesLLC#601 follow-up) Seen live: an appliance sat at a 2.9GB DB file holding 34 alert rows — DELETE alone only freelists pages, and the VACUUM rewrite streams through the WAL, so without a TRUNCATE checkpoint the gigabytes just move into aifw.db-wal. - DELETE /ids/alerts (existing route) now reclaims the space after the purge (VACUUM + WAL truncate) and writes an audit entry - retention worker auto-reclaims space once 100k purged rows accumulate, with retry-on-busy carry-over - alert retention default 30 -> 7 days; UI settings field becomes a preset select (1d/7d/1mo/3mo/1yr) with a hint about the hourly sweep - UI alerts page: Clear All Alerts button with confirm modal - CLI: new 'aifw ids purge-alerts [--yes]' (confirm prompt, reclaim) and 'aifw ids retention [days]' (show/set), degrading gracefully on DBs the IDS subsystem never initialized
Minor release: manual IDS alert purge (UI Clear All + aifw ids purge-alerts CLI), automatic disk-space reclaim after mass alert deletions (VACUUM + WAL truncate — a fielded appliance sat at a 2.9GB DB holding 34 rows), alert retention preset select defaulting to 7 days, FreeBSD-target warning cleanup, and Updates-page restart-hang fixes.
…sd-target-warnings fix: FreeBSD-target warnings + Updates-page restart hangs
…clear-alerts feat(ids): manual alert purge, auto space reclaim, 7-day retention default
…5.111.0 release: v5.111.0
… tarball honors manifest pins The build scripts build companion repos (trafficcop/rDHCP/rDNS/rTIME) in the operator's own working clones: - build-local.sh left them in detached HEAD at the manifest pin after every ISO build — the operator had to manually switch each repo back to main. The pin checkout now records the current ref and restores it after the build (also on build failure); the BUILD_MANIFEST summary reads the pins from manifest.json instead of repo HEADs. - build-update.sh was worse: 'git reset --hard origin/main' silently discarded any uncommitted work in those repos AND bypassed the ZerosAndOnesLLC#538 manifest pins — the update tarball shipped whatever origin/main was at build time, not the reviewed pinned revision. It now uses the same pin-checkout-and-restore as build-local.sh. A dirty companion tree now fails fast with a commit/stash hint instead of being clobbered.
Root-run builds leave root-owned .git objects and target/ artifacts in the operator's companion clones — seen live as 'insufficient permission for adding an object to repository database' blocking git stash. chown back to SUDO_USER after each companion build (mirrors the output-dir ownership fix from ZerosAndOnesLLC#604).
…nion-checkout-restore fix(build): companion checkouts restore the operator's branch; update tarball honors manifest pins
Bumps the ISO build base, CI VM images, release notes, and bug report template from 15.0 to 15.1. The pinned vmactions/freebsd-vm commit already ships conf/15.1.conf, so no action bump is needed. Refs ZerosAndOnesLLC#609
…ebsd-15.1 Move builds to FreeBSD 15.1
The merge with main kept a call to the removed insert_queue method (renamed to insert_queue_on in the config-restore transaction work).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes ZerosAndOnesLLC#532
Validation