feat(config): minimal config persistence — Overrides user-only + sparse save by seeyabye · Pull Request #161 · javinizer/javinizer-go

seeyabye · 2026-07-18T16:56:59Z

Summary

Implements the minimal-config-persistence OpenSpec change: a two-phase plan to fix the structural defect where the first-run flow bakes the full embedded example to disk and Save re-inflates every field on every write, so code-default changes never reach existing installs.

Replaces the reverted PR #158 (which used a defaults_version patch chain). No version tracking — future default changes propagate lazily via sparse persistence.

Phase 1: Overrides becomes user-only

ResolvedSettings(name) on ScrapersConfig — complete default clone for absent overrides, override.Clone() + MergeDefaultsFrom for present
Dropped Finalize seeding loop + pipeline.normalize languageDefaults map
ScraperRegistryConfigFromApp accepts names []string + defaults map (avoids import cycles)
Level 2 mutation API: Override(name), ApplyOverride(name, opts...) (no-op when empty), UserOverride(name) non-creating, WithScrapeActress/WithBrowser option funcs
minimalInitStub() as production first-run (sparse stub instead of full ~90-line example)

Phase 2: Sparse persistence

diffYAMLDocuments(actual, defaults) — node-diff post-MarshalYAML, emits only non-default leaves + alwaysEmitPaths + null clear-markers for cleared omitempty maps
reconcileSparse(dst, sparseTarget, schemaDoc, knownScraperNames) — src-authoritative, key-comment-preserving, recursively deletion-safe, schema-gated
ConfigStorage.SaveSparse — shares afero fs, lock, atomic replace; handles missing/malformed; unchanged-byte skip
SparseSaveContext{Defaults, Schema, KnownScraperNames} with collision rejection
Scraper settings schema (complete, including optional fields + nested ProxyConfig + deprecated aliases)
Prepare split: PrepareForPersistence (excludes translation credentials) + PrepareRuntime (full ValidateConfig)
API PUT /config uses disk-origin snapshot for preserveRedactedSecrets, SaveSparse for persistence, runtimeCfg (post-env) for reload
APIRuntime.SetInitialConfigs(runtimeCfg, diskCfg) + DiskConfigSnapshot() (cloned on set/get)
LoadOrCreate convergence: all 3 branches (existing-file, first-run, migration) apply env + Prepare before returning

No `defaults_version`

No version tracking. Future default-value changes update the code default; existing configs pick it up on their next API save (lazy propagation). Existing baked values persist until users clear them via WebUI.

Test plan

go test -short ./internal/config/... ./internal/api/... — pass
go test -race ./internal/config/... — pass
golangci-lint run ./internal/config/... ./internal/api/... — 0 issues
go build ./... — pass
Full CI suite (will run on push)

Known limitations (follow-up revisions)

Sparse-save defaults use DefaultConfig(nil, nil) (hardcoded priorities) — correct for the standard registry, documented limitation for custom registries
Missing some integration tests (env-secret round trips, startup convergence, Prepare split parity) — to be added as follow-up

OpenSpec

Change: openspec/changes/minimal-config-persistence
Reviewed through 8 cycles by codex-lb/gpt-5.6-sol (medium) — final verdict: PROCEED WITH REVISIONS (0 blockers, 0 majors, 3 minors)

…terialization Phase 1 of minimal-config-persistence: - Add ResolvedSettings(name) on ScrapersConfig (complete default clone for absent, override.Clone() + MergeDefaultsFrom for present) - Drop Finalize seeding loop (Overrides no longer pre-materialized) - Drop pipeline.normalize languageDefaults map (redundant with module Defaults) - Switch ScraperRegistryConfigFromApp to accept names + defaults (avoids import cycles), inline resolve via Clone + MergeDefaultsFrom - Update all 4 call sites (dependencies, hot_reload, e2e) - Fix info command to use ResolvedSettings instead of direct Overrides - Update tests assuming old materialization behavior

…onvergence

…arse semantics

… snapshot + startup re-decode

…+ empty override skip

…e cache

…atic precedence

…ile to proxy schema

…ecated aliases to schema

seeyabye · 2026-07-18T16:57:14Z

@codex review

… size)

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 726e824db2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

seeyabye · 2026-07-18T17:09:20Z

@codex review — please review the latest commit (20ced4a) which fixes the lint file-size issue. The two P1 findings are being addressed in a follow-up commit.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 20ced4ad1e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

seeyabye · 2026-07-18T17:22:03Z

@codex review — please review the latest commit (735192e) which addresses the P1 finding about sparse saves on normalize. The P2 about DefaultConfig(nil,nil) is a known documented limitation for custom registries.

seeyabye · 2026-07-18T17:22:09Z

@codex review

…irst-run sparse)

seeyabye · 2026-07-18T17:34:20Z

@codex review

codecov · 2026-07-18T17:35:40Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.86%. Comparing base (7685a84) to head (b83ef31).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #161      +/-   ##
==========================================
+ Coverage   91.34%   91.86%   +0.52%     
==========================================
  Files         455      457       +2     
  Lines       40593    41381     +788     
==========================================
+ Hits        37078    38013     +935     
+ Misses       2281     2184      -97     
+ Partials     1234     1184      -50

Flag	Coverage Δ
unittests	`91.86% <100.00%> (+0.52%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
cmd/javinizer/commands/info/command.go	`85.05% <100.00%> (ø)`
cmd/javinizer/commands/init/command.go	`85.00% <100.00%> (+11.66%)`	⬆️
cmd/javinizer/commands/scrape/command.go	`91.63% <100.00%> (+0.66%)`	⬆️
internal/api/core/bootstrap.go	`94.73% <100.00%> (+0.45%)`	⬆️
internal/api/core/disk_snapshot.go	`100.00% <100.00%> (ø)`
internal/api/core/hot_reload.go	`94.66% <100.00%> (+0.14%)`	⬆️
internal/api/core/runtime_manager.go	`92.06% <ø> (ø)`
internal/api/system/config.go	`83.09% <100.00%> (+2.67%)`	⬆️
internal/api/system/config_service.go	`100.00% <100.00%> (+28.26%)`	⬆️
internal/commandutil/dependencies.go	`89.67% <100.00%> (+2.81%)`	⬆️
... and 11 more

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 994b36b2ae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

seeyabye · 2026-07-18T17:46:13Z

@codex review — these P2 findings are documented limitations in the OpenSpec design: (1) nested map entry deletion for proxy profiles is a known reconciliation edge case; (2) partial-override presence for scraper Enabled is the documented tri-state limitation. Both are accepted for v1 and tracked as follow-up.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 994b36b2ae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

seeyabye · 2026-07-19T02:37:49Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dad30eb286

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

seeyabye · 2026-07-19T03:32:23Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 643a15262f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

seeyabye · 2026-07-19T04:32:53Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cca16d79c3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

seeyabye · 2026-07-19T05:49:45Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ab7177cf12

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

seeyabye · 2026-07-19T14:54:10Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4fbc7535e4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

seeyabye · 2026-07-19T19:56:19Z

@codex review

chatgpt-codex-connector · 2026-07-19T20:08:12Z

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: b83ef31635

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

seeyabye added 10 commits July 18, 2026 18:37

feat(config): add sparse marshal diff + reconcile

89b40cf

feat(config): add sparse persistence + Prepare split + LoadOrCreate c…

991d7cf

…onvergence

fix(config): wire SaveSparse into API + add mutation API + fix SaveSp…

4b5cb86

…arse semantics

fix(config): wire KnownScraperNames + empty mapping skip + APIRuntime…

0b81bf8

… snapshot + startup re-decode

fix(config): scraper schema for reconciliation + collision rejection …

af809c8

…+ empty override skip

fix(config): complete scraper schema + collision rejection + sync.Onc…

b573006

…e cache

fix(config): remove isZeroScraperMapping + complete proxy schema + st…

4761ba9

…atic precedence

fix(config): skip empty scraper overrides in UnmarshalYAML + add Prof…

a7f9ff5

…ile to proxy schema

fix(config): skip empty scraper overrides in UnmarshalJSON + add depr…

726e824

…ecated aliases to schema

fix(config): split disk snapshot methods to separate file (lint: file…

20ced4a

… size)

chatgpt-codex-connector Bot reviewed Jul 18, 2026

View reviewed changes

Comment thread internal/config/storage.go

Comment thread internal/scraper/registry_config.go

chatgpt-codex-connector Bot reviewed Jul 18, 2026

View reviewed changes

Comment thread internal/config/storage.go

Comment thread internal/config/storage.go

fix(config): use SaveSparse for normalize-triggered saves (P1: keep f…

994b36b

…irst-run sparse)

seeyabye force-pushed the feat/minimal-config-persistence branch from 735192e to 994b36b Compare July 18, 2026 17:24

github-actions Bot added the 🤖 codex: needs work Codex review found issues that need addressing label Jul 18, 2026

github-actions Bot removed the 🤖 codex: needs work Codex review found issues that need addressing label Jul 18, 2026

chatgpt-codex-connector Bot reviewed Jul 18, 2026

View reviewed changes

Comment thread internal/config/sparse_marshal.go

Comment thread internal/scraper/registry_config.go

github-actions Bot added the 🤖 codex: needs work Codex review found issues that need addressing label Jul 18, 2026

github-actions Bot removed the 🤖 codex: needs work Codex review found issues that need addressing label Jul 18, 2026

chatgpt-codex-connector Bot reviewed Jul 18, 2026

View reviewed changes

Comment thread internal/config/storage.go

github-actions Bot added the 🤖 codex: needs work Codex review found issues that need addressing label Jul 18, 2026

fix(init): use SaveSparse to keep config sparse after init

7982bb7

test(config): close sparse persistence coverage gaps

dad30eb

github-actions Bot removed the 🤖 codex: needs work Codex review found issues that need addressing label Jul 19, 2026

chatgpt-codex-connector Bot reviewed Jul 19, 2026

View reviewed changes

Comment thread internal/config/storage.go

github-actions Bot added the 🤖 codex: needs work Codex review found issues that need addressing label Jul 19, 2026

fix(config): compare sparse saves with scraper defaults

643a152

github-actions Bot removed the 🤖 codex: needs work Codex review found issues that need addressing label Jul 19, 2026

chatgpt-codex-connector Bot reviewed Jul 19, 2026

View reviewed changes

Comment thread internal/scraper/registry_config.go

github-actions Bot added the 🤖 codex: needs work Codex review found issues that need addressing label Jul 19, 2026

fix(config): preserve omitted scraper enabled defaults

cca16d7

github-actions Bot removed the 🤖 codex: needs work Codex review found issues that need addressing label Jul 19, 2026

chatgpt-codex-connector Bot reviewed Jul 19, 2026

View reviewed changes

Comment thread internal/config/sparse_marshal.go

Comment thread internal/config/sparse_marshal.go

github-actions Bot added the 🤖 codex: needs work Codex review found issues that need addressing label Jul 19, 2026

fix(config): reconcile sparse scraper state

ab7177c

github-actions Bot removed the 🤖 codex: needs work Codex review found issues that need addressing label Jul 19, 2026

chatgpt-codex-connector Bot reviewed Jul 19, 2026

View reviewed changes

Comment thread internal/scraper/registry_config.go

Comment thread internal/config/sparse_marshal.go Outdated

github-actions Bot added the 🤖 codex: needs work Codex review found issues that need addressing label Jul 19, 2026

fix(config): validate effective scraper overrides

4fbc753

github-actions Bot removed the 🤖 codex: needs work Codex review found issues that need addressing label Jul 19, 2026

chatgpt-codex-connector Bot reviewed Jul 19, 2026

View reviewed changes

Comment thread internal/config/sparse_marshal.go

github-actions Bot added the 🤖 codex: needs work Codex review found issues that need addressing label Jul 19, 2026

fix(config): preserve proxy profile deletions

b83ef31

github-actions Bot removed the 🤖 codex: needs work Codex review found issues that need addressing label Jul 19, 2026

github-actions Bot added the 🤖 codex: ok Codex review found no major issues label Jul 19, 2026

Uh oh!

Conversation

seeyabye commented Jul 18, 2026

Summary

Phase 1: Overrides becomes user-only

Phase 2: Sparse persistence

No defaults_version

Test plan

Known limitations (follow-up revisions)

OpenSpec

Uh oh!

seeyabye commented Jul 18, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

seeyabye commented Jul 18, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

seeyabye commented Jul 18, 2026

Uh oh!

seeyabye commented Jul 18, 2026

Uh oh!

seeyabye commented Jul 18, 2026

Uh oh!

codecov Bot commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

seeyabye commented Jul 18, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

seeyabye commented Jul 19, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

seeyabye commented Jul 19, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

seeyabye commented Jul 19, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

seeyabye commented Jul 19, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

No `defaults_version`

codecov Bot commented Jul 18, 2026 •

edited

Loading