Live bridge: capture, understand, manipulate & verify a company file over the v4 API#260
Open
Benjvandam wants to merge 18 commits into
Open
Live bridge: capture, understand, manipulate & verify a company file over the v4 API#260Benjvandam wants to merge 18 commits into
Benjvandam wants to merge 18 commits into
Conversation
Ad-hoc manipulation of a live company file's custom data, reusing the update*Custom functions. set-custom sets one property (value JSON-parsed when possible); delete-custom soft-deletes via null. Both support --level (inferred from URL), --handle/--account targeting, --file batch, and a confirmation prompt (--yes to skip). Adds lib/customWriter.js + tests.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption. Comment |
Consolidates the live-bridge read commands onto this branch so the four new commands (get-results, capture, set-custom, delete-custom) ship together on top of update-text-properties (#249). get-results reads a live file's computed results+customs; capture snapshots a live file as JSON (scoped, or --full). Adds lib/resultsReader.js, lib/dataCapture.js (+ buildLiquidTest refactor of liquidTestGenerator), and their tests.
This was referenced Jun 29, 2026
Benjvandam
changed the title
Lists a reconciliation's custom inputs with their declared defaults, stored values and live effective values, plus the template's results. Effective values come only from certain sources (stored override, a directly-echoed result, or a literal default); everything else is flagged unavailable rather than re-rendered (a re-render can't faithfully reproduce live company state). Adds lib/inputDescriber.js + tests.
Benjvandam
changed the title
New manifest -h <handle> / -u <url> builds a static data manifest by scanning a reconciliation's Liquid (main + text_parts + shared parts, recursively): own customs, cross-template results/customs, period drop + prior-period depth, company drop, accounts and shared parts. First step of the deep-capture + self-validating render pipeline. Adds lib/templateManifest.js + tests.
Replace the regex-based lib/templateManifest.js with lib/dataScope.js, which shells out to 'silverfin-ls data-scope <main.liquid>' (the maintained tree-sitter language server) and returns its JSON scope. This stops duplicating STL parsing in the CLI and picks up silverfin-ls's AST-accurate, alias-aware analysis — including digit-prefixed handles (2018_*, 275_*) the regex couldn't follow. The 'manifest' command now delegates to it; set SILVERFIN_LS_CMD to point at a specific silverfin-ls binary/build. Removes the regex module + test.
Resolves the effective value of inputs whose default is a DIRECT reference to data created elsewhere — a cross-template result/custom, a period/company custom, optionally a prior period (period.minus_Ny...) — by reading it straight from the live API. lib/defaultResolver.js parses each still-unavailable input's default, captures only the referenced handles + deepest referenced period (via lib/deepCapture.js, driven by the silverfin-ls scope), and fills the value with an effectiveSource of 'captured:<path>'. Computed/conditionally-assigned default variables stay flagged (they need a render). deepCapture now exposes periodOrder so period.minus_Ny maps to the right captured period. Adds resolver unit tests.
Both commands now accept --dry-run, which prints the exact target and the
{namespace, key, value} properties that would be written (value:null for
delete) as JSON and sends no request. Lets the write path be exercised safely
on any firm. Matches the --dry-run already on update-text-properties.
describe-inputs --resolve --compute now computes input-default VARIABLES that reduce to a lookup into captured live data (a cross-template result/custom indexed by a date-derived dynamic key, with branch selection). lib/stlLite.js is a deliberately minimal, SAFE STL evaluator: it supports only assign/capture/ if-elsif-else plus a whitelist of filters (date, default, a few string/arith), tracks all block nesting so it never desyncs on the 6000-line shared parts, and NEVER fabricates — anything using currency/MAX()/infix/loops leaves the variable undefined. lib/offlineDefaultResolver.js builds the context from a deep capture and fills only resolved variable-defaults, labelled 'computed:<var> (offline; validate vs live)'. Verified on 2018_275_A_liquidationreserve: all 6 taxable_yearN defaults computed offline match the live results (5000/0/4000/ 3000/2000/0). Values still require validation against a live render.
deepCapture now records per-period DROP scalars from getPeriods (year_end_date, year_start_date, end_date, fiscal_year) and buildContext exposes them, so the offline evaluator can read period[.minus_Ny].year_start_date and .fiscal_year.* — not just year_end_date. Additive; existing tests unaffected.
Extends the bounded offline evaluator toward account-range and calculated
defaults, keeping the never-fabricate guarantee:
- sfApi.getPeriodAccounts (paginated GET .../periods/{id}/accounts).
- deepCapture fetches period.accounts (number/name/type/value; no debit-credit
arrays) when the template reads accounts (captureAccounts), across all
captured periods.
- buildContext exposes period.accounts (+ minus_Ny + year_end.accounts alias),
period.fiscal_year (stringifies to the year), year_start_date, period.exists,
and flattened company.* drop scalars.
- stlLite executes for/fori loops, supports account range: filtering with
.value/.count/.numbers aggregation, split/size/first/last, currency/percentage
as numeric pass-throughs, at_least/at_most, and SAFE infix arithmetic (any
unresolved operand -> UNRESOLVED, incl. no-space a-b). opening_value stays
UNRESOLVED (not captured) rather than fabricating.
Verified live: the 6 taxable_yearN defaults still compute correctly, and
period.accounts range aggregation matches a manual sum. 613 tests pass.
New 'set-default' command changes an input's DEFAULT (not an override) by writing the upstream custom the default derives from: - lib/provenanceTracer.js reverse-traces a default to a single settable custom, reusing the offline evaluator (an onAssign hook on stlLite) to resolve the taken branch + dynamic keys. Auto-invertible = a direct cross-template custom, or a result that statically echoes a custom; everything else (computed / branch-gated / dynamic result / arithmetic) is reported NOT invertible with the provenance chain, never guessed. - lib/defaultSetter.js orchestrates: light capture (period dates only, fast), trace, resolve the upstream reconciliation, write, verify, and emit a change table with old -> new + why + a blast-radius note when >1 template reads the source. Warns if the target field has a shadowing override. - lib/changeReport.js: shared change table + JSON. Live-verified on 275A (non-invertible dynamic-result path surfaces the chain in ~7s). 624 tests pass.
…writes
Extension: when a default resolves to <handle>.results.<tag> with no static
echo, traceUpstreamResultProduction statically inspects <handle>'s own liquid to
see how it produces that (possibly dynamic) tag — matching its {% result %}
whose tag literal/capture-pattern covers the tag — and auto-inverts when the
value is a direct custom; otherwise it reports the actual production (e.g.
'produces addition_12_2024 from a computed value [addition]') for an actionable
chain.
Fixes from the adversarial review (all false-invertible -> wrong-write risks):
- Reject value-changing filters: a default/RHS with | times/round/currency/… is
NOT invertible (only | default: is safe) — was silently stripped, writing the
raw value so the default recomputed wrong. (2x critical)
- Custom target must be exactly namespace.key; a field access (custom.ns.key.value)
is refused, not truncated to the wrong custom. (critical/high)
- Resolve the upstream period by OFFSET, not by end-date key (two periods can
share an end_date), so we never write to the wrong period. (medium)
Adds safety unit tests. 628 tests pass.
Benjvandam
changed the title
…sult echoes
Many templates echo an input as a result through a variable
(`{% assign v = custom.ns.key | default:… | currency %}{% result 'tag' v %}`),
not a direct `{% result 'tag' custom.ns.key %}`. Previously those inputs were
flagged `unavailable` even though their effective value sits in the live
results. parseAdjacentEchoes now attributes such a result to the nearest
preceding input when the tag relates to the key (and stays on the same side of
the 2026 split), so the value is read straight from the results table — no
silverfin-ls, no re-render. On 2018_275_A_liquidationreserve this fills 19 of 38
inputs from the API (taxable_yearN, withdrawal_N, tax_N …) that were all
`unavailable` before.
- getDataScope now auto-detects the analyzer (SILVERFIN_LS_CMD -> silverfin-ls on PATH -> npx --no-install silverfin-ls) instead of only defaulting to a bare binary, so it works without manual SILVERFIN_LS_CMD when installed. - Every invocation has a 30s timeout, so an outdated silverfin-ls (which would ignore `data-scope` and start its LSP server) can no longer hang the CLI. - Errors are precise: what was tried + how to install/verify + a `silverfin doctor` pointer. - New `silverfin doctor` command probes silverfin-ls end-to-end against a throwaway template and reports OK or the exact failure + install steps.
Inspecting an upstream/underlying template (the ones that feed another's defaults) previously required its URL and didn't cover collection inputs, so an agent had to fall back to reading liquid. Now: - lib/targetResolver.js resolves a handle to its reconciliation instance id (and workflow id + URL) in the same company/period as a given URL. - get-results / describe-inputs / capture accept --handle <handle> to act on that sibling; new 'resolve-handle' command prints the resolved id + URL. - describe-inputs surfaces line-item COLLECTIONS (fori over custom.<ns>) as a 'collections' section with the item fields and the live items. Verified live on firm 1355: describe-inputs --handle liquidation_reserve now shows its custom.items line-items (date/amount/description) — the exact data needed to change a computed addition at its source. 636 tests pass.
Fixes a real failure mode: an agent tried to verify template LOGIC (a display mode -> a note result) by poking live data on an empty shell company where the template is gated off, got 0 results, and wrongly concluded the API was cached. - get-results / describe-inputs now include resultCount (+ an explanatory note when 0): an empty results table means the template isn't computing in this company/period (gated / not set up), NOT that the API is cached and NOT something a custom write will force. - New 'render' command (lib/renderRunner.js) renders a template against its LOCAL liquid-test fixture via the render/test engine and returns a JSON outcome (rendered? / expectation mismatches) — the deterministic, fixture- driven way to verify template logic when a live company lacks the data. Verified live: empty vkt_6_8 -> resultCount 0 + note; liquidation -> 51; render of the liquidation template -> rendered:true, all expectations passed. 638 tests.
# Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The live bridge: drive the full capture → understand → manipulate → verify → push loop on a live Silverfin company file over the v4 API — no browser. Change inputs and see the recomputed results over HTTP, cheaply, and iterate.
This PR grew from the original 5 commands into the full feature and is the hub. Two companion PRs land with it:
data-scopebatch command + digit-leading identifier fix silverfin-ls#1 — thedata-scopecommand this CLI delegates static analysis to.silverfin-live-bridgeskill that teaches the agent the loop.How it fits together
1. Understand (static, via silverfin-ls)
manifest— a template's data scope (own customs, cross-template deps incl. dynamically-keyed, prior-period depth, company drop, accounts, involved files). Delegates tosilverfin-ls data-scope, reusing its maintained tree-sitter parser instead of duplicating STL parsing.describe-inputs— every input with type, declared default, stored value and live effective value, labelling the source (stored override vs. default vs. result-echo). It resolves inputs that are echoed as a result — directly or indirectly (a result echoing a variable derived from the input) — straight from the live results, with no silverfin-ls needed. Only the residual (defaults not surfaced as a result) needs the steps below.--handle <handle>onget-results/describe-inputs/capture— same company/period, no URL to reconstruct — orresolve-handlefor its instance id + URL.describe-inputs --handlealso surfaces line-item collections ({% fori item in custom.items %}…) as acollectionssection with the live dated line-items, so collection-driven inputs (e.g. a working paper's amounts) are visible.2. Capture (live — the oracle)
get-results— computed results + customs (the API returns them directly).capture/capture --full— JSON snapshot of a template's scope, or the whole company across all periods/reconciliations, from firm+company alone.3. Resolve the residual hidden defaults (optional specialist)
For defaults that aren't surfaced as a result:
describe-inputs --resolvefills direct cross-template references from a targeted capture;--computeruns a bounded, never-fabricating offline STL evaluator (lib/stlLite.js) for computed / dynamic-key / account-range defaults, labelledcomputed:… (validate vs live). The platform stays the source of truth; this is the browser-free specialist.4. Manipulate — three deliberate modes
set-custom/delete-custom— override a field (any level),--dry-run,--handleto target an underlying template,--filebatch.set-default— change a field's default by writing the upstream custom it derives from (reverse-traced vialib/provenanceTracer.js), never an override. Auto-proceeds when auto-invertible, prints a change table (old → new + why + blast radius), and otherwise surfaces the provenance chain — never guesses.update-text-properties— bulk-seed from a Liquid Test YAML.5. Verify —
get-resultsagain; compare (its output now includes aresultCount, with anotewhen 0 so an empty results table is self-explanatory — the template isn't computing in that company, not a caching issue). The platform is the source of truth.Live data vs. template logic — for verifying what a template's code produces (not live-data effects), a new
rendercommand renders it against its local liquid-test fixture and returns a JSON outcome (rendered? / expectation mismatches) — deterministic, and it works even where a live company has no data (whereget-resultsis empty).Key modules
lib/dataScope.js(silverfin-ls shell-out + auto-detect +doctor),lib/deepCapture.js(targeted deep capture +period.accounts),lib/offlineDefaultResolver.js+lib/stlLite.js(offline evaluator),lib/defaultResolver.js(direct-ref),lib/provenanceTracer.js+lib/defaultSetter.js+lib/changeReport.js(set-default),lib/inputDescriber.js(incl. indirect/adjacent echo resolution),lib/customWriter.js.Robustness / diagnostics
The CLI auto-detects silverfin-ls (
SILVERFIN_LS_CMD→silverfin-lson PATH →npx --no-install silverfin-ls) and puts a timeout on every call, so an outdated silverfin-ls (which would ignoredata-scopeand start its LSP server) can no longer hang the CLI. A newsilverfin doctorcommand probes silverfin-ls end-to-end and reports OK or the exact failure with install steps. And becausedescribe-inputsresolves echoed inputs from the live results without silverfin-ls, the tool degrades gracefully when it's absent.Safety
Never fabricates a value (the offline evaluator gives up rather than guess);
--dry-runon every write; test-firm default in the skill;set-defaultwas hardened via an adversarial review against value-changing filters, field-access customs and period-collisions to avoid wrong writes.Dependency / release order
The CLI shells out to
silverfin-ls data-scope(those commands error clearly — runsilverfin doctor— if it's absent). Usable order: silverfin-ls merged + released → this CLI merged + released → the skill live.Base / stacking
This branch is based on
implement-reverse-test(#249). Once #249 merges we can retarget this PR tomainso it shows only the live-bridge feature.Tests
633 jest passing, eslint clean. CHANGELOG under
[Unreleased](no self version bump). The silverfin-ls shell-out isn't exercised in CI (unit tests are pure), so CI is green without silverfin-ls installed.