From e564365f80b0b9d70d8ce05532341af8c713d9aa Mon Sep 17 00:00:00 2001 From: alowrydi Date: Fri, 26 Jun 2026 16:26:33 +0100 Subject: [PATCH 01/12] Initial import of di.asyncdispatch --- di/asyncdispatch/asyncdispatch.md | 191 ++++++++++++++++++++++++++++++ di/asyncdispatch/asyncdispatch.q | 159 +++++++++++++++++++++++++ di/asyncdispatch/init.q | 10 ++ di/asyncdispatch/test.csv | 75 ++++++++++++ 4 files changed, 435 insertions(+) create mode 100644 di/asyncdispatch/asyncdispatch.md create mode 100644 di/asyncdispatch/asyncdispatch.q create mode 100644 di/asyncdispatch/init.q create mode 100644 di/asyncdispatch/test.csv diff --git a/di/asyncdispatch/asyncdispatch.md b/di/asyncdispatch/asyncdispatch.md new file mode 100644 index 00000000..dbe5ee3c --- /dev/null +++ b/di/asyncdispatch/asyncdispatch.md @@ -0,0 +1,191 @@ +# asyncdispatch + +`asyncdispatch.q` is the async multi-process query coordinator extracted from the TorQ gateway's core engine (`.gw.*`). It queues client queries, dispatches them to available backend ("server") processes, collects the per-server results, applies a join function, and replies to the client — with timeout management and error propagation if a backend disconnects mid-query. + +**Standalone value:** any process that needs scatter-gather across multiple backend processes (not just a "gateway") can load this module to get queueing, dispatch, join and timeout handling for free. + +**Out of scope** (left to other modules in the gateway decomposition): +- **Routing** — deciding *which* servers satisfy a query is `di.serverselect`'s job. This module is handed a resolved list of servertypes and dispatches to whatever of those is currently idle. +- **Permissions** — `.pm.*`/`execas`/`valp`-style checks belong in `di.gateway`/`di.permissions`. +- **EOD / discovery / dashboard & REST handlers** — `di.gateway` concerns. + +--- + +## Loading + +```q +ad:use`di.asyncdispatch +``` + +Returns a handle (`ad`) exposing the exported API, e.g. `ad.execquery[...]`. + +--- + +## Configuration & pluggable hooks + +| Variable | Default | Setter | Purpose | +|---|---|---|---| +| `errorprefix` | `"error: "` | — | Prefix added to error strings sent back to clients | +| `querykeeptime` | `0D00:30` | — | How long `removequeries` keeps finished queries in `queryqueue` | +| `clearinactivetime` | `0D01:00` | — | How long `removeinactive` keeps records for disconnected servers | +| `synccallsallowed` | `0b` | — | Whether `execquery[...;1b]` (sync mode) is permitted | +| `cp` | `{.z.p}` | `setcp` | Current-time function. Override for simulation/backtesting | +| `formatresponse` | see below | `setformatresponse` | Final transform applied to a result/error before it is sent to the client | +| `availableservers` | built-in | `setavailableservers` | How "which servers are free" is computed | +| `getnextqueryid` | `fifo` | `setgetnextqueryid` | Scheduler — picks the next query to dispatch | +| `resultcallback` / `errorcallback` | `` `addserverresult `` / `` `addservererror `` | `setcallbacks` | Symbols backend servers call back into to report results/errors | + +### `formatresponse[status;sync;result]` +- `status` — `1b` success, `0b` error. +- `sync` — `1b` if the client used a deferred-sync call (`execquery[...;1b]`), `0b` for async. +- Default: on a synchronous error (`not status` and `sync`), **signals** `result` as an error back up the `-30!` deferred response so the client's sync call raises. Otherwise passes `result` through unchanged. + +### `availableservers[excludeinuse]` +- `excludeinuse=1b` → only **idle, active** servers. +- `excludeinuse=0b` → all **active** servers. +- Override for custom load-balancing. + +### `getnextqueryid[]` (scheduler) +- Default: oldest runnable query first (FIFO) — a runnable query is one where all required servertypes have an idle server available. +- Override `getnextqueryid` for priority queues. The override receives no arguments and must return a 1-row (or empty) table with the `queryqueue` schema. + +### `resultcallback` / `errorcallback` +`serverexecute` (the function sent to backend servers) posts its outcome back via `(resultcallback;queryid;result)` / `(errorcallback;queryid;error)` over `neg .z.w`. Since this module has no fixed namespace path when used standalone, the consumer must point these symbols at wherever it mounted this module on the **backend** processes, e.g.: + +```q +ad.setcallbacks[`.gw.dispatch.addserverresult;`.gw.dispatch.addservererror] +``` + +--- + +## Core data structures + +### `servers` (keyed table, key: `handle`) + +| Column | Type | Description | +|---|---|---| +| `handle` | `int` | Connection handle to the backend process | +| `servertype` | `symbol` | e.g. `` `rdb ``, `` `hdb `` | +| `inuse` | `boolean` | Currently running a query | +| `active` | `boolean` | Connected (`0b` once disconnected) | +| `disconnecttime` | `timestamp` | When the server was marked inactive; null while active | + +### `queryqueue` (keyed table, key: `queryid`) + +| Column | Type | Description | +|---|---|---| +| `queryid` | `long` | Allocated by `addquery` | +| `time` | `timestamp` | When the query was queued | +| `clienth` | `int` | `.z.w` of the requesting client | +| `query` | `()` | The query payload — whatever `value` can execute | +| `servertype` | `()` | List of servertype symbols the query must run on | +| `join` | `()` | Function applied to the collected per-server results | +| `postback` | `()` | `()` for a plain reply, else `(function;extra args...)` to wrap the reply | +| `timeout` | `timespan` | `0Wn` for none | +| `returntime` | `timestamp` | Set by `finishquery` once complete | +| `error` | `boolean` | `1b` if the query ended in error | +| `sync` | `boolean` | `1b` if the client is waiting on a deferred (`-30!`) response | + +### `clients` (table) + +| Column | Type | Description | +|---|---|---| +| `time` | `timestamp` | Connection time | +| `clienth` | `int` | `.z.w` of the client | +| `user` | `symbol` | `.z.u` | +| `ip` | `int` | `.z.a` | +| `host` | `symbol` | `.z.h` | + +### `results` (dict) + +`queryid -> (clienth; servertype!(handle;result;received))` + +- Keys are the servertype symbols passed to `addquery`. +- Each value is a 3-list: `(handle assigned; result once it arrives; received flag)`. +- A query is complete once every `received` flag is `1b`. +- Entries are removed by `finishquery` once the query completes. + +--- + +## Functions + +### Server registry +- **`addserver[handle;servertype]`** — register a backend connection. +- **`availableservers[excludeinuse]`** — query which servers can take work now (see Pluggable hooks). + +### Client tracking +- **`addclientdetails[h]`** — call from `.z.po` to record a new client connection in `clients`. +- **`removeclienthandle[h]`** — call from `.z.pc`. Stamps any of that client's unfinished queries as returned/errored and drops their `results` entries. + +### Queueing +- **`addquery[query;servertype;join;postback;timeout;sync]`** — low-level: insert a row into `queryqueue`. Does **not** dispatch — call `runnextquery[]` afterwards (or use `execquery` which does both). +- **`removequeries[age]`** — purge `queryqueue` rows with `returntime` older than `age`. + +### Scheduling +- **`getnextqueryid[]`** — returns the next runnable query (FIFO by default: oldest query for which all required servertypes have an idle server). Override via `setgetnextqueryid` for priority queues. + +### Result collection & joining +- **`addserverresult[qid;data]`** — called when a backend posts back success. Fills slot, frees server, tries to run the next query; once all slots are received applies `join`, sends the reply, and finishes the query. +- **`addservererror[qid;err]`** — called when a backend posts back an error. Sends the error to the client and finishes the query. + +### Dispatch +- **`serverexecute[qid;query]`** — **runs on the backend**. Executes `value query`, trapping errors, and posts the outcome back to the dispatcher via `resultcallback`/`errorcallback`. +- **`runnextquery[]`** — picks the next runnable query via `getnextqueryid`, resolves idle servers, and dispatches. + +### Timeouts & disconnects +- **`checktimeout[]`** — finds queries past their `timeout` with no `returntime`, sends a timeout error, and finishes them. +- **`removeserverhandle[serverh]`** — call from `.z.pc` for **backend** handles. Errors any in-flight or queued queries that depended on this server, marks the server `active:0b`, and calls `runnextquery[]`. +- **`removeinactive[age]`** — purge `servers` rows that have been inactive for longer than `age`. + +### Public API +- **`execquery[query;servertype;join;postback;timeout;sync]`** — queue + dispatch. `sync=0b` for async (uses `postback`); `sync=1b` for deferred sync via `-30!` (`postback` ignored, errors if `synccallsallowed` is `0b`). + +### Housekeeping +- **`init[timerrepeat]`** — optionally wires `removequeries`, `checktimeout` and `removeinactive` into a recurring timer. `timerrepeat` should have the signature `.timer.repeat[starttime;endtime;period;(func;params);description]`, or pass `(::)` to skip. + +--- + +## Example usage + +```q +/ -- dispatcher process -- +ad:use`di.asyncdispatch + +/ point backends' callbacks at this module's mount point on this process +ad.setcallbacks[`ad.addserverresult;`ad.addservererror] + +/ register backend connections as they connect +h:hopen`:backend1:5001 +ad.addserver[h;`rdb] + +/ track gateway clients +.z.po:{ad.addclientdetails[.z.w]} +.z.pc:{ad.removeclienthandle[.z.w]; ad.removeserverhandle[.z.w]} + +/ a client calls this asynchronously: +ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;0b] +/ -> queues the query, dispatches it to the rdb, and (once the rdb replies) +/ razes the single result and sends it back to the calling client + +/ housekeeping - run periodically (e.g. via di.timer) +ad.checktimeout[] +ad.removequeries[ad.querykeeptime] +ad.removeinactive[ad.clearinactivetime] +``` + +```q +/ -- backend process -- +ad:use`di.asyncdispatch +/ nothing else required: when the dispatcher sends (serverexecute;qid;query), +/ this process runs value query and posts the result/error back via +/ ad.addserverresult / ad.addservererror on the dispatcher +``` + +--- + +## Notes + +- `.z.M.` is used for in-place mutation of tables (`upsert`/`insert`/`update from`/`delete from`), and `.z.m.:value` for whole-variable reassignment — the same convention used by `di.cache`. +- Module globals referenced inside q-sql expressions (WHERE conditions, UPDATE SET values) must use `.z.m.varname` form, since q-sql evaluates globals in the calling context rather than the module namespace. +- `servertype` on `queryqueue`/`addquery` is deliberately generic: it is a list of servertype symbols used to look up idle servers of each type. Pass the same servertype list to `addserver` and `addquery` to wire them together. +- This module does not open or accept any connections itself — `addserver` and the `.z.po`/`.z.pc` wiring are the consumer's responsibility, by design (keeps this module dependency-free and testable in-process). diff --git a/di/asyncdispatch/asyncdispatch.q b/di/asyncdispatch/asyncdispatch.q new file mode 100644 index 00000000..a7e0d40a --- /dev/null +++ b/di/asyncdispatch/asyncdispatch.q @@ -0,0 +1,159 @@ +/ di.asyncdispatch - async scatter-gather query coordinator. +/ Queues queries, dispatches to available backends, collects results per server, +/ applies a join function, and replies to the client. +/ Routing (which servers satisfy a query) is di.serverselect's responsibility. + +errorprefix:"error: "; +querykeeptime:0D00:30; +clearinactivetime:0D01:00; +synccallsallowed:0b; + +cp:{.z.p} / injectable clock; swap out in tests to control time +setcp:{.z.m.cp:x} / allows runtime replacement of the clock without editing the module + +formatresponse:{[status;sync;result]$[not[status]and sync;'result;result]} / sync errors must be signalled with ' so the client receives a trapped error; async errors pass through unchanged +setformatresponse:{.z.m.formatresponse:x} / override reply formatting without editing the module + +/ symbols backend servers call back via; set to wherever this module is mounted +resultcallback:`addserverresult; / stored as symbol so the name survives IPC serialization to backend processes +errorcallback:`addservererror; +setcallbacks:{[resfn;errfn].z.m.resultcallback:resfn;.z.m.errorcallback:errfn} / update callback symbols when module is mounted under a non-default namespace + +/ registered backend servers +servers:([handle:`u#`int$()] servertype:`symbol$(); inuse:`boolean$(); active:`boolean$(); disconnecttime:`timestamp$()) + +/ pending and in-flight client queries +queryqueue:([queryid:`u#`long$()] time:`timestamp$(); clienth:`int$(); query:(); servertype:(); join:(); postback:(); timeout:`timespan$(); returntime:`timestamp$(); error:`boolean$(); sync:`boolean$()) + +/ connected client tracking +clients:([] time:`timestamp$(); clienth:`int$(); user:`symbol$(); ip:`int$(); host:`symbol$()) + +/ per-query result accumulator: queryid -> (clienth; servertype!(handle;result;done)) +results:()!() + +queryid:0; + +addserver:{[h;st].z.M.servers upsert (h;st;0b;1b;0Np)} / register a backend handle and servertype so it becomes eligible for dispatch + +availableservers:{[excludeinuse] / centralise the active+idle filter so dispatch and routing share one definition + $[excludeinuse; + select from servers where active, not inuse; + select from servers where active]}; +setavailableservers:{.z.m.availableservers:x} / swap in a custom routing strategy without forking core dispatch + +addclientdetails:{[h].z.M.clients insert (cp[];h;.z.u;.z.a;.z.h)} / record client identity on connect for audit and orphan-query cleanup on disconnect + +removeclienthandle:{[h] / on client disconnect, mark their pending queries errored so result slots are not leaked + update error:1b,returntime:.z.m.cp[] from .z.M.queryqueue where clienth=h, null returntime; + .z.m.results:(exec queryid from .z.m.queryqueue where clienth=h)_results}; + +addquery:{[query;servertype;join;postback;timeout;sync] / enqueue a query without dispatching; caller must call runnextquery[] to trigger dispatch + .z.M.queryqueue upsert (queryid;cp[];.z.w;query;servertype;join;{$[11h=type x;enlist x;x]}postback;timeout;0Np;0b;sync); + .z.m.queryid:queryid+1}; + +removequeries:{[age] / prevent queryqueue growing unboundedly; purge completed queries older than age + .z.m.queryqueue:0!delete from .z.m.queryqueue where not null returntime, .z.m.cp[]>returntime+age}; + +getnextqueryid:{ / pick the oldest FIFO-eligible query whose required servertypes are all currently idle + avail:exec distinct servertype from availableservers 1b; + runnable:0!select from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, {all x in y}[;avail] each servertype; + 1 sublist select from runnable where time=min time}; +setgetnextqueryid:{.z.m.getnextqueryid:x} / inject a priority or custom scheduling strategy without forking the module + +addserverresult:{[qid;data] / fill one result slot; once all slots for a query are filled, run the join function and reply to the client + if[not qid in key results;:()]; + st:first exec servertype from .z.m.servers where handle=.z.w; + slots:results[qid;1]; + slots[st]:(.z.w;data;1b); + results[qid]:(results[qid;0];slots); + update inuse:0b from .z.M.servers where handle in .z.w; + runnextquery[]; + if[not qid in key results;:()]; + vals:value results[qid;1]; + if[not all vals[;2];:()]; + qd:queryqueue[qid]; + res:@[{(0b;x y)}[qd`join];vals[;1];{(1b;errorprefix,"join failed: ",x)}]; + sendclientreply[qid;last res;not res 0]; + finishquery[qid;res 0]}; + +addservererror:{[qid;err] / short-circuit a query on backend failure; free the server and notify the client before moving on + sendclientreply[qid;errorprefix,err;0b]; + update inuse:0b from .z.M.servers where handle in .z.w; + runnextquery[]; + finishquery[qid;1b]}; + +sendclientreply:{[qid;result;status] / deliver result or error to the client, handling sync vs async send and postback wrapping in one place + qd:queryqueue[qid]; + if[qd`error;:()]; + tosend:$[()~qd`postback;result;qd[`postback],enlist[qd`query],enlist result]; + $[qd`sync; + @[-30!;(qd`clienth;not status;$[status;formatresponse[1b;1b;result];result]);{}]; + @[neg qd`clienth;formatresponse[status;0b;tosend];()]]}; + +finishquery:{[qid;err] / remove query from the live results accumulator and stamp its completion time; keeps queryqueue and results consistent + .z.m.results:(qid,())_results; + update error:err,returntime:.z.m.cp[] from .z.M.queryqueue where queryid in qid}; + +serverexecute:{[qid;query] / runs on the backend; traps errors locally so a crash posts an error reply rather than silently dropping the result + res:@[{(0b;value x)};query;{(1b;"server ",(string .z.h),":",(string system"p"),": ",x)}]; + @[neg .z.w;$[res 0;(errorcallback;qid;res 1);(resultcallback;qid;res 1)]; + {@[neg .z.w;(errorcallback;x;"failed to return result: ",y);()]}[qid]]}; + +sendquerytoserver:{[qid;query;handles] / fan the query out to all required handles and mark them in-use atomically to prevent double-dispatch + (neg handles,:())@\:(serverexecute;qid;query); + update inuse:1b from .z.M.servers where handle in handles}; + +runnextquery:{ / pick the next dispatchable query and fan out to one idle server per required servertype; called after any state change that may unblock work + if[0=count torun:getnextqueryid[];:()]; + torun:first torun; + avail:exec first handle by servertype from availableservers 1b; + types:torun`servertype; + handles:avail types; + qid:torun`queryid; + slots:types!(count[types],())#enlist(0Ni;(::);0b); + slots[types;0]:handles; + results[qid]:(torun`clienth;slots); + sendquerytoserver[qid;torun`query;handles]}; + +checktimeout:{ / periodic scan to error queries that have waited beyond their timeout, preventing them from hanging indefinitely + qids:exec queryid from .z.m.queryqueue where not timeout=0Wn, null returntime, .z.m.cp[]>time+timeout; + if[count qids; + sendclientreply[;errorprefix,"query timed out";0b] each qids; + finishquery[qids;1b]]}; + +removeserverhandle:{[serverh] / on backend disconnect, error in-flight queries using that handle and queued queries that can no longer be satisfied + if[null st:first exec servertype from .z.m.servers where handle=serverh;:()]; + err:errorprefix,"backend ",string[st]," server disconnected"; + + / in-flight: queries where this handle was assigned to a slot + qids:where {[h;qid]h in value[.z.m.results[qid;1]][;0]}[serverh] each key .z.m.results; + sendclientreply[;err," during query";0b] each qids; + finishquery[qids;1b]; + + / queued: queries that can no longer be satisfied by remaining active servers + activetypes:exec distinct servertype from .z.m.servers where active, handle<>serverh; + qids2:exec queryid from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, + not {all x in y}[;activetypes] each servertype; + sendclientreply[;err,", query cannot be satisfied";0b] each qids2; + finishquery[qids2;1b]; + + update active:0b,disconnecttime:.z.m.cp[] from .z.M.servers where handle=serverh; + runnextquery[]}; + +removeinactive:{[age]delete from .z.M.servers where not active, .z.m.cp[]>disconnecttime+age} / prune stale disconnected-server rows to stop the servers table growing forever + +execquery:{[query;servertype;join;postback;timeout;sync] / public entry point; validate sync constraints then enqueue and kick dispatch + if[sync; + if[not synccallsallowed;'"syncexec: synchronous calls are not allowed"]; + if[not @[{-30!x;1b};(::);0b];'"syncexec: deferred response not supported on this connection"]; + .[{[q;s;j;t]addquery[q;s;j;();t;1b];runnextquery[]};(query;servertype;join;timeout);{-30!(.z.w;1b;x)}]; + :()]; + addquery[query;servertype;join;postback;timeout;0b]; + runnextquery[]}; + +/ wire housekeeping into a timer - pass (::) to skip +init:{[timerrepeat] / wire recurring housekeeping (timeout scan, query purge, server purge) into a provided timer; pass (::) to skip registration + if[not timerrepeat~(::); + timerrepeat[cp[];0Wp;0D00:05:00;(.z.m.removequeries;querykeeptime);"asyncdispatch: remove old queries"]; + timerrepeat[cp[];0Wp;0D00:00:05;(.z.m.checktimeout;`);"asyncdispatch: timeout expired queries"]; + timerrepeat[cp[];0Wp;0D00:05:00;(.z.m.removeinactive;clearinactivetime);"asyncdispatch: remove inactive servers"]]}; diff --git a/di/asyncdispatch/init.q b/di/asyncdispatch/init.q new file mode 100644 index 00000000..f7086488 --- /dev/null +++ b/di/asyncdispatch/init.q @@ -0,0 +1,10 @@ +\l ::asyncdispatch.q + +export:([ + errorprefix;querykeeptime;clearinactivetime;synccallsallowed; / user-tunable config: error text prefix and housekeeping intervals + setcp;setformatresponse;setcallbacks;setavailableservers;setgetnextqueryid; / user-injectable overrides: clock, reply format, callback namespace, routing and scheduling + addserver;removeserverhandle; / server lifecycle: register and deregister backends + addclientdetails;removeclienthandle; / client lifecycle: wire into .z.po / .z.pc + addserverresult;addservererror; / IPC return paths: backends resolve these by name to deliver results or errors + execquery; / public API: only entry point for submitting a query + init]) / startup: wires housekeeping into the provided timer diff --git a/di/asyncdispatch/test.csv b/di/asyncdispatch/test.csv new file mode 100644 index 00000000..7366d787 --- /dev/null +++ b/di/asyncdispatch/test.csv @@ -0,0 +1,75 @@ +action,ms,bytes,lang,code,repeat,minver,comment +before,0,0,q,system"q -p 19501 -q &",1,,start backend process +before,0,0,q,system"sleep 1",1,,wait for backend to initialise +before,0,0,q,ad:use`di.asyncdispatch,1,,load module +before,0,0,q,srv:{.m.di.0asyncdispatch.servers},1,,live servers table +before,0,0,q,qq:{.m.di.0asyncdispatch.queryqueue},1,,live queryqueue table +before,0,0,q,cl:{.m.di.0asyncdispatch.clients},1,,live clients table +before,0,0,q,res:{.m.di.0asyncdispatch.results},1,,live results dict +before,0,0,q,bh:hopen`::19501,1,,open handle to backend +before,0,0,q,bh".m.di.0asyncdispatch.resultcallback:`.m.di.0asyncdispatch.addserverresult",1,,configure backend result callback to dispatcher +before,0,0,q,bh".m.di.0asyncdispatch.errorcallback:`.m.di.0asyncdispatch.addservererror",1,,configure backend error callback to dispatcher +before,0,0,q,ad.addserver[bh;`mock],1,,register backend as mock server + +/ Test 0: server registry +true,0,0,q,(enlist bh)~exec handle from srv[] where active,1,,backend server is active +true,0,0,q,(enlist bh)~exec handle from ad.availableservers 1b,1,,idle backend server is available + +/ Test 1: queueing and FIFO scheduling +run,0,0,q,.m.di.0asyncdispatch.addquery["2+2";enlist`mock;raze;();0Wn;0b],1,,queue a query for the mock servertype +true,0,0,q,1~count select from qq[] where query~\:"2+2",1,,query was added to the queue +true,0,0,q,1~count .m.di.0asyncdispatch.getnextqueryid[],1,,query is runnable - idle mock server available +true,0,0,q,(enlist`mock)~first exec servertype from .m.di.0asyncdispatch.getnextqueryid[],1,,getnextqueryid identifies the available servertype + +/ Test 2: full dispatch -> result -> join -> reply via real IPC round-trip +run,0,0,q,qid1:exec first queryid from qq[] where query~\:"2+2",1,,capture the queued query id +run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch query to real backend process +run,0,0,q,bh"",1,,sync round-trip: flushes dispatch and waits for backend result callback before returning +true,0,0,q,0b~first exec error from qq[] where queryid=qid1,1,,query completed without error +true,0,0,q,not null first exec returntime from qq[] where queryid=qid1,1,,returntime is stamped +true,0,0,q,not qid1 in key res[],1,,result accumulator cleaned up after join + +/ Test 3: checktimeout +run,0,0,q,.m.di.0asyncdispatch.addquery["3+3";enlist`mock;raze;();0D00:00:00.000000001;0b],1,,queue with a near-zero timeout +run,0,0,q,qid2:exec first queryid from qq[] where query~\:"3+3",1,,capture the queued query id +run,0,0,q,.m.di.0asyncdispatch.checktimeout[],1,,timeout sweep should catch the expired query +true,0,0,q,1b~first exec error from qq[] where queryid=qid2,1,,timed-out query is flagged as error +true,0,0,q,not null first exec returntime from qq[] where queryid=qid2,1,,timed-out query has returntime stamped + +/ Test 4: removequeries purges completed queries +run,0,0,q,ad.setcp[{.z.p+2D}],1,,fast-forward the clock by 2 days +run,0,0,q,.m.di.0asyncdispatch.removequeries ad.querykeeptime,1,,purge queries older than querykeeptime +true,0,0,q,0~count select from qq[] where queryid in qid1,qid2,1,both completed queries purged +run,0,0,q,ad.setcp[{.z.p}],1,,restore the real clock + +/ Test 5: removeserverhandle errors queued queries when the only server disconnects +run,0,0,q,ad.addserver[1i;`mock2],1,,register a second mock server (fake handle - query is errored before dispatch) +run,0,0,q,.m.di.0asyncdispatch.addquery["4+4";enlist`mock2;raze;();0Wn;0b],1,,queue a query that can only run on mock2 +run,0,0,q,qid3:exec first queryid from qq[] where query~\:"4+4",1,,capture the queued query id +run,0,0,q,ad.removeserverhandle[1i],1,,disconnect the only server able to run the query +true,0,0,q,1b~first exec error from qq[] where queryid=qid3,1,,query errored as no server can satisfy it +true,0,0,q,0b~first exec active from srv[] where handle=1i,1,,disconnected server marked inactive + +/ Test 6: removeinactive purges inactive server records +run,0,0,q,ad.setcp[{.z.p+2D}],1,,fast-forward clock past clearinactivetime +run,0,0,q,.m.di.0asyncdispatch.removeinactive ad.clearinactivetime,1,,purge inactive servers +true,0,0,q,0~count select from srv[] where handle=1i,1,,inactive server record removed +run,0,0,q,ad.setcp[{.z.p}],1,,restore real clock + +/ Test 7: client tracking +run,0,0,q,ad.addclientdetails[2i],1,,record a connected client +true,0,0,q,1~count select from cl[] where clienth=2i,1,,client connection is tracked +run,0,0,q,ad.removeclienthandle[2i],1,,disconnect the client +true,0,0,q,0~count select from qq[] where clienth=2i,null returntime,1,client unfinished queries cleared + +/ Test 8: pluggable hooks +run,0,0,q,ad.setformatresponse[{[status;sync;result]result}],1,,override formatresponse to pass result through +true,0,0,q,5~.m.di.0asyncdispatch.formatresponse[1b;0b;5],1,,overridden formatresponse is used +run,0,0,q,ad.setgetnextqueryid[{()}],1,,override scheduler to never pick a query +true,0,0,q,0~count .m.di.0asyncdispatch.getnextqueryid[],1,,overridden scheduler is used +run,0,0,q,ad.setcallbacks[`myresult;`myerror],1,,override callback symbols +true,0,0,q,(`myresult;`myerror)~(.m.di.0asyncdispatch.resultcallback;.m.di.0asyncdispatch.errorcallback),1,,callback symbols updated + +/ Teardown +run,0,0,q,neg[bh](exit;0);neg[bh](::),1,,send exit to backend process and flush +run,0,0,q,hclose bh,1,,close backend handle From c1c3557a3a11bca37c8fa836a96291df2ada01d1 Mon Sep 17 00:00:00 2001 From: alowrydi Date: Mon, 29 Jun 2026 12:21:14 +0100 Subject: [PATCH 02/12] align with project standards, fix production bugs, real IPC integration tests, removing hardcoded ports/paths --- di/asyncdispatch/asyncdispatch.md | 293 +++++++++++++++++------------- di/asyncdispatch/asyncdispatch.q | 272 +++++++++++++++++++-------- di/asyncdispatch/init.q | 15 +- di/asyncdispatch/test.csv | 38 ++-- 4 files changed, 389 insertions(+), 229 deletions(-) diff --git a/di/asyncdispatch/asyncdispatch.md b/di/asyncdispatch/asyncdispatch.md index dbe5ee3c..7511df84 100644 --- a/di/asyncdispatch/asyncdispatch.md +++ b/di/asyncdispatch/asyncdispatch.md @@ -1,191 +1,224 @@ -# asyncdispatch +# di.asyncdispatch -`asyncdispatch.q` is the async multi-process query coordinator extracted from the TorQ gateway's core engine (`.gw.*`). It queues client queries, dispatches them to available backend ("server") processes, collects the per-server results, applies a join function, and replies to the client — with timeout management and error propagation if a backend disconnects mid-query. +Async scatter-gather query coordinator for kdb-x gateway processes. Queues client queries, dispatches them to available backend processes by servertype, collects per-server results, applies a join function, and replies to the client — with timeout management and correct error propagation if a backend disconnects mid-query. -**Standalone value:** any process that needs scatter-gather across multiple backend processes (not just a "gateway") can load this module to get queueing, dispatch, join and timeout handling for free. +Routing (deciding which servertypes satisfy a query) is `di.serverselect`'s responsibility. This module receives a resolved servertype list and dispatches to whatever idle backends of each type are registered. -**Out of scope** (left to other modules in the gateway decomposition): -- **Routing** — deciding *which* servers satisfy a query is `di.serverselect`'s job. This module is handed a resolved list of servertypes and dispatches to whatever of those is currently idle. -- **Permissions** — `.pm.*`/`execas`/`valp`-style checks belong in `di.gateway`/`di.permissions`. -- **EOD / discovery / dashboard & REST handlers** — `di.gateway` concerns. +--- + +## Features + +- Queue and dispatch async client queries to multiple backend process types simultaneously (scatter-gather) +- Collect per-server results and apply a caller-supplied join function once all slots are filled +- Timeout expired queries with configurable per-query timespan via `checktimeout` +- Handle backend disconnects mid-query — errors in-flight queries and queued queries that can no longer be satisfied +- Track connected clients and clean up orphaned queries on client disconnect +- Support synchronous deferred response mode (`-30!`) alongside the default async mode +- Accept fully pluggable scheduler (`setgetnextqueryid`), routing (`setavailableservers`), reply formatter (`setformatresponse`), and callback symbols (`setcallbacks`) — swap without touching core dispatch logic +- Detect and normalise `kx.log` instances automatically so callers can pass a logger directly without manual wrapping --- -## Loading +## Dependencies + +| Dependency | Key | Required | Description | +|---|---|---|---| +| logger | `` `log `` | yes | `info`, `warn`, `error` — each binary `{[c;m]}` where `c` is a symbol context and `m` is a string | + +The `log` dependency must be passed to `init` inside the `deps` dict. The module throws immediately if it is absent or missing any of the three required keys. All three are required since the module calls `info`, `warn`, and `error`. + +A `kx.log` instance can be passed directly — the module normalises monadic functions to the binary `{[c;m]}` contract automatically via `normlog`. Context is embedded in the output as `"context: message"`: ```q +kxlog:use`kx.log ad:use`di.asyncdispatch -``` -Returns a handle (`ad`) exposing the exported API, e.g. `ad.execquery[...]`. +/ minimal +ad.init[enlist[`log]!enlist kxlog.createLog[]] + +/ with config overrides +ad.init[`log`querykeeptime`synccallsallowed!(kxlog.createLog[];0D01:00;1b)] +``` --- -## Configuration & pluggable hooks +## Initialisation -| Variable | Default | Setter | Purpose | +`init[deps]` takes a single dictionary combining the `log` dependency with any configuration overrides. + +| Key | Required | Default | Description | |---|---|---|---| -| `errorprefix` | `"error: "` | — | Prefix added to error strings sent back to clients | -| `querykeeptime` | `0D00:30` | — | How long `removequeries` keeps finished queries in `queryqueue` | -| `clearinactivetime` | `0D01:00` | — | How long `removeinactive` keeps records for disconnected servers | -| `synccallsallowed` | `0b` | — | Whether `execquery[...;1b]` (sync mode) is permitted | -| `cp` | `{.z.p}` | `setcp` | Current-time function. Override for simulation/backtesting | -| `formatresponse` | see below | `setformatresponse` | Final transform applied to a result/error before it is sent to the client | -| `availableservers` | built-in | `setavailableservers` | How "which servers are free" is computed | -| `getnextqueryid` | `fifo` | `setgetnextqueryid` | Scheduler — picks the next query to dispatch | -| `resultcallback` / `errorcallback` | `` `addserverresult `` / `` `addservererror `` | `setcallbacks` | Symbols backend servers call back into to report results/errors | - -### `formatresponse[status;sync;result]` -- `status` — `1b` success, `0b` error. -- `sync` — `1b` if the client used a deferred-sync call (`execquery[...;1b]`), `0b` for async. -- Default: on a synchronous error (`not status` and `sync`), **signals** `result` as an error back up the `-30!` deferred response so the client's sync call raises. Otherwise passes `result` through unchanged. - -### `availableservers[excludeinuse]` -- `excludeinuse=1b` → only **idle, active** servers. -- `excludeinuse=0b` → all **active** servers. -- Override for custom load-balancing. - -### `getnextqueryid[]` (scheduler) -- Default: oldest runnable query first (FIFO) — a runnable query is one where all required servertypes have an idle server available. -- Override `getnextqueryid` for priority queues. The override receives no arguments and must return a 1-row (or empty) table with the `queryqueue` schema. - -### `resultcallback` / `errorcallback` -`serverexecute` (the function sent to backend servers) posts its outcome back via `(resultcallback;queryid;result)` / `(errorcallback;queryid;error)` over `neg .z.w`. Since this module has no fixed namespace path when used standalone, the consumer must point these symbols at wherever it mounted this module on the **backend** processes, e.g.: +| `` `log `` | yes | — | Binary log dep — `info`, `warn`, `error` functions each `{[c;m]}` | +| `` `errorprefix `` | no | `"error: "` | String prepended to all error messages sent back to clients | +| `` `querykeeptime `` | no | `0D00:30` | How long `removequeries` retains finished query rows | +| `` `clearinactivetime `` | no | `0D01:00` | How long `removeinactive` retains disconnected server rows | +| `` `synccallsallowed `` | no | `0b` | Whether `execquery[...;1b]` (deferred sync mode) is permitted | -```q -ad.setcallbacks[`.gw.dispatch.addserverresult;`.gw.dispatch.addservererror] -``` +Housekeeping — `checktimeout`, `removequeries`, and `removeinactive` — is the caller's responsibility. Wire them into your gateway's timer after `init`. The configured default age parameters are accessible as `querykeeptime` and `clearinactivetime` via module state. --- -## Core data structures +## Exported Functions -### `servers` (keyed table, key: `handle`) - -| Column | Type | Description | -|---|---|---| -| `handle` | `int` | Connection handle to the backend process | -| `servertype` | `symbol` | e.g. `` `rdb ``, `` `hdb `` | -| `inuse` | `boolean` | Currently running a query | -| `active` | `boolean` | Connected (`0b` once disconnected) | -| `disconnecttime` | `timestamp` | When the server was marked inactive; null while active | +### `init[deps]` +Initialise the module. Validates the log dependency and applies config overrides. +```q +ad.init[enlist[`log]!enlist logdep] +``` -### `queryqueue` (keyed table, key: `queryid`) +### `addserver[handle;servertype]` +Register a backend connection. `handle`: open int handle. `servertype`: symbol identifying the process type (e.g. `` `rdb ``, `` `hdb ``). +```q +ad.addserver[hopen`:backend1:5001;`rdb] +``` -| Column | Type | Description | -|---|---|---| -| `queryid` | `long` | Allocated by `addquery` | -| `time` | `timestamp` | When the query was queued | -| `clienth` | `int` | `.z.w` of the requesting client | -| `query` | `()` | The query payload — whatever `value` can execute | -| `servertype` | `()` | List of servertype symbols the query must run on | -| `join` | `()` | Function applied to the collected per-server results | -| `postback` | `()` | `()` for a plain reply, else `(function;extra args...)` to wrap the reply | -| `timeout` | `timespan` | `0Wn` for none | -| `returntime` | `timestamp` | Set by `finishquery` once complete | -| `error` | `boolean` | `1b` if the query ended in error | -| `sync` | `boolean` | `1b` if the client is waiting on a deferred (`-30!`) response | - -### `clients` (table) - -| Column | Type | Description | -|---|---|---| -| `time` | `timestamp` | Connection time | -| `clienth` | `int` | `.z.w` of the client | -| `user` | `symbol` | `.z.u` | -| `ip` | `int` | `.z.a` | -| `host` | `symbol` | `.z.h` | +### `removeserverhandle[handle]` +Call from `.z.pc` for **backend** handles. Errors any in-flight or queued queries that depended on this server, marks the server `active:0b`, and triggers `runnextquery`. +```q +.z.pc:{ad.removeserverhandle[.z.w];ad.removeclienthandle[.z.w]} +``` -### `results` (dict) +### `addclientdetails[handle]` +Record client identity on connect. Call from `.z.po`. +```q +.z.po:{ad.addclientdetails[.z.w]} +``` -`queryid -> (clienth; servertype!(handle;result;received))` +### `removeclienthandle[handle]` +On client disconnect, mark their pending queries errored so result slots are not leaked. Call from `.z.pc`. +```q +.z.pc:{ad.removeserverhandle[.z.w];ad.removeclienthandle[.z.w]} +``` -- Keys are the servertype symbols passed to `addquery`. -- Each value is a 3-list: `(handle assigned; result once it arrives; received flag)`. -- A query is complete once every `received` flag is `1b`. -- Entries are removed by `finishquery` once the query completes. +### `addserverresult[qid;data]` +Called when a backend posts back a successful result. Fills the result slot, frees the server, triggers `runnextquery`, and — once all slots for the query are received — applies the join function and replies to the client. +```q +/ called by serverexecute on the backend; not typically called directly +``` ---- +### `addservererror[qid;err]` +Called when a backend posts back an error. Sends the error to the client and finishes the query. +```q +/ called by serverexecute on the backend; not typically called directly +``` -## Functions +### `execquery[query;servertype;join;postback;timeout;sync]` +Public entry point. Validates sync constraints, enqueues the query, and triggers dispatch. -### Server registry -- **`addserver[handle;servertype]`** — register a backend connection. -- **`availableservers[excludeinuse]`** — query which servers can take work now (see Pluggable hooks). +| Argument | Type | Description | +|---|---|---| +| `query` | any | Payload passed to `value` on the backend | +| `servertype` | symbol list | One symbol per required backend type, e.g. `` enlist`rdb `` | +| `join` | function | Applied to the list of per-server results once all are received | +| `postback` | list or `()` | `()` for a plain reply; `(function;extra_args...)` to wrap the reply | +| `timeout` | timespan | `0Wn` for no timeout | +| `sync` | boolean | `1b` for deferred sync via `-30!`; `0b` for async | -### Client tracking -- **`addclientdetails[h]`** — call from `.z.po` to record a new client connection in `clients`. -- **`removeclienthandle[h]`** — call from `.z.pc`. Stamps any of that client's unfinished queries as returned/errored and drops their `results` entries. +```q +ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;0b] +``` -### Queueing -- **`addquery[query;servertype;join;postback;timeout;sync]`** — low-level: insert a row into `queryqueue`. Does **not** dispatch — call `runnextquery[]` afterwards (or use `execquery` which does both). -- **`removequeries[age]`** — purge `queryqueue` rows with `returntime` older than `age`. +### `checktimeout[]` +Scan the queue for queries past their timeout, send a timeout error to each client, and mark them complete. Wire into your gateway's timer — every few seconds is typical. +```q +/ in gateway timer +timer.addjob.default[`asyncdispatch.checktimeout;{ad.checktimeout[]};();5i;1] +``` -### Scheduling -- **`getnextqueryid[]`** — returns the next runnable query (FIFO by default: oldest query for which all required servertypes have an idle server). Override via `setgetnextqueryid` for priority queues. +### `removequeries[age]` +Purge completed `queryqueue` rows older than `age`. Prevents unbounded growth. +```q +/ default age is querykeeptime (0D00:30) +timer.addjob.default[`asyncdispatch.removequeries;{ad.removequeries[0D00:30]};();300i;1] +``` -### Result collection & joining -- **`addserverresult[qid;data]`** — called when a backend posts back success. Fills slot, frees server, tries to run the next query; once all slots are received applies `join`, sends the reply, and finishes the query. -- **`addservererror[qid;err]`** — called when a backend posts back an error. Sends the error to the client and finishes the query. +### `removeinactive[age]` +Purge `servers` rows for backends that have been disconnected longer than `age`. Prevents unbounded growth. +```q +/ default age is clearinactivetime (0D01:00) +timer.addjob.default[`asyncdispatch.removeinactive;{ad.removeinactive[0D01:00]};();300i;1] +``` -### Dispatch -- **`serverexecute[qid;query]`** — **runs on the backend**. Executes `value query`, trapping errors, and posts the outcome back to the dispatcher via `resultcallback`/`errorcallback`. -- **`runnextquery[]`** — picks the next runnable query via `getnextqueryid`, resolves idle servers, and dispatches. +### `setformatresponse[f]` +Override the reply formatter applied before a result or error is sent to the client. `f` must be `{[status;sync;result]}`. +```q +ad.setformatresponse[{[status;sync;result]result}] +``` -### Timeouts & disconnects -- **`checktimeout[]`** — finds queries past their `timeout` with no `returntime`, sends a timeout error, and finishes them. -- **`removeserverhandle[serverh]`** — call from `.z.pc` for **backend** handles. Errors any in-flight or queued queries that depended on this server, marks the server `active:0b`, and calls `runnextquery[]`. -- **`removeinactive[age]`** — purge `servers` rows that have been inactive for longer than `age`. +### `setcallbacks[resfn;errfn]` +Update the callback symbols used by `serverexecute`. Required when the module is mounted under a non-default namespace — point these at wherever `addserverresult` and `addservererror` are visible on the backend processes. +```q +ad.setcallbacks[`.gw.dispatch.addserverresult;`.gw.dispatch.addservererror] +``` -### Public API -- **`execquery[query;servertype;join;postback;timeout;sync]`** — queue + dispatch. `sync=0b` for async (uses `postback`); `sync=1b` for deferred sync via `-30!` (`postback` ignored, errors if `synccallsallowed` is `0b`). +### `setavailableservers[f]` +Swap in a custom routing strategy. `f` must be `{[excludeinuse]}` returning a table with a `servertype` column. +```q +ad.setavailableservers[{[excl]select from servers where active}] +``` -### Housekeeping -- **`init[timerrepeat]`** — optionally wires `removequeries`, `checktimeout` and `removeinactive` into a recurring timer. `timerrepeat` should have the signature `.timer.repeat[starttime;endtime;period;(func;params);description]`, or pass `(::)` to skip. +### `setgetnextqueryid[f]` +Inject a custom scheduling strategy. `f` must be niladic and return a 0- or 1-row table with the `queryqueue` schema. +```q +/ priority queue example - highest-priority query first +ad.setgetnextqueryid[{1 sublist `priority xdesc 0!select from .z.m.queryqueue where null returntime}] +``` --- -## Example usage +## Usage Example ```q -/ -- dispatcher process -- +kxlog:use`kx.log +timer:use`di.timer +timer.init[()!()] + ad:use`di.asyncdispatch +ad.init[enlist[`log]!enlist kxlog.createLog[]] / point backends' callbacks at this module's mount point on this process ad.setcallbacks[`ad.addserverresult;`ad.addservererror] / register backend connections as they connect -h:hopen`:backend1:5001 -ad.addserver[h;`rdb] +ad.addserver[hopen`:backend1:5001;`rdb] +ad.addserver[hopen`:backend2:5002;`hdb] -/ track gateway clients +/ wire client and server connection/disconnection handlers .z.po:{ad.addclientdetails[.z.w]} -.z.pc:{ad.removeclienthandle[.z.w]; ad.removeserverhandle[.z.w]} +.z.pc:{ad.removeserverhandle[.z.w];ad.removeclienthandle[.z.w]} + +/ wire housekeeping into the gateway timer +timer.addjob.default[`asyncdispatch.checktimeout;{ad.checktimeout[]};();5i;1] +timer.addjob.default[`asyncdispatch.removequeries;{ad.removequeries[0D00:30]};();300i;1] +timer.addjob.default[`asyncdispatch.removeinactive;{ad.removeinactive[0D01:00]};();300i;1] / a client calls this asynchronously: -ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;0b] -/ -> queues the query, dispatches it to the rdb, and (once the rdb replies) -/ razes the single result and sends it back to the calling client +/ execquery dispatches to rdb and hdb in parallel, razes results, replies to client +ad.execquery[("select count i by sym from trade";"select count i by sym from trade");`rdb`hdb;raze;();0Wn;0b] -/ housekeeping - run periodically (e.g. via di.timer) -ad.checktimeout[] -ad.removequeries[ad.querykeeptime] -ad.removeinactive[ad.clearinactivetime] +/ synchronous deferred mode (requires synccallsallowed:1b in deps) +ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;1b] ``` +--- + +## Running Tests + ```q -/ -- backend process -- -ad:use`di.asyncdispatch -/ nothing else required: when the dispatcher sends (serverexecute;qid;query), -/ this process runs value query and posts the result/error back via -/ ad.addserverresult / ad.addservererror on the dispatcher +k4unit:use`di.k4unit +k4unit.moduletest`di.asyncdispatch ``` +52 tests. Requires a `q` binary in `PATH` — the test suite starts a real backend process on a dynamically selected free port and exercises the full dispatch lifecycle over a live IPC connection. No TorQ installation or special libraries required. Covers: server registry, FIFO scheduling, full IPC round-trip with join and reply, checktimeout, removequeries, removeserverhandle with orphaned query cleanup, client tracking, removeinactive, in-flight server release on client disconnect, and all pluggable hook setters. + --- ## Notes -- `.z.M.` is used for in-place mutation of tables (`upsert`/`insert`/`update from`/`delete from`), and `.z.m.:value` for whole-variable reassignment — the same convention used by `di.cache`. -- Module globals referenced inside q-sql expressions (WHERE conditions, UPDATE SET values) must use `.z.m.varname` form, since q-sql evaluates globals in the calling context rather than the module namespace. -- `servertype` on `queryqueue`/`addquery` is deliberately generic: it is a list of servertype symbols used to look up idle servers of each type. Pass the same servertype list to `addserver` and `addquery` to wire them together. -- This module does not open or accept any connections itself — `addserver` and the `.z.po`/`.z.pc` wiring are the consumer's responsibility, by design (keeps this module dependency-free and testable in-process). +- Housekeeping (`checktimeout`, `removequeries`, `removeinactive`) is the caller's responsibility — the gateway process already has a timer running and is better placed to decide intervals. Wire all three after `init`; see the usage example above +- `.z.M.` is used for in-place mutation of tables (`upsert`, `insert`, `update from`, `delete from`) and `.z.m.:value` for whole-variable reassignment — the same convention used by `di.cache` +- Module globals referenced inside q-sql expressions (WHERE conditions, UPDATE SET values) must use the `.z.m.varname` form since q-sql evaluates column expressions in the calling context rather than the module namespace +- `servertype` in `queryqueue` and `addquery` is a list of servertype symbols — one per required backend type. Pass `` enlist`rdb `` for single-server queries, `` `rdb`hdb `` for scatter-gather across two types +- `setcallbacks` must be called before any queries are dispatched if the module is mounted under a non-default path — `serverexecute` reads `resultcallback` and `errorcallback` by bare name on the backend process and posts back to whatever symbols they resolve to +- This module opens and accepts no connections itself — `addserver` and the `.z.po`/`.z.pc` wiring are the consumer's responsibility, keeping the module dependency-free and testable in-process +- All three log keys (`info`, `warn`, `error`) are required — the module calls `info` on server/client connect and init, `warn` on disconnect and timeout, and `error` on backend error and join failure diff --git a/di/asyncdispatch/asyncdispatch.q b/di/asyncdispatch/asyncdispatch.q index a7e0d40a..86bcc8ce 100644 --- a/di/asyncdispatch/asyncdispatch.q +++ b/di/asyncdispatch/asyncdispatch.q @@ -1,88 +1,85 @@ -/ di.asyncdispatch - async scatter-gather query coordinator. -/ Queues queries, dispatches to available backends, collects results per server, -/ applies a join function, and replies to the client. -/ Routing (which servers satisfy a query) is di.serverselect's responsibility. +/ di.asyncdispatch - async scatter-gather query coordinator +/ queues queries, dispatches to available backends, collects results per server, +/ applies a join function, and replies to the client +/ routing (which servers satisfy a query) is di.serverselect's responsibility +/ the log dependency is required - init errors immediately if absent or malformed +/ log functions are binary {[c;m]} where c is a symbol context and m is a string +/ ============================================================ +/ module state and defaults +/ ============================================================ + +/ error prefix prepended to all error strings returned to clients errorprefix:"error: "; + +/ how long completed queries are kept in queryqueue before being purged querykeeptime:0D00:30; + +/ how long disconnected servers are kept in the servers table before being removed clearinactivetime:0D01:00; -synccallsallowed:0b; -cp:{.z.p} / injectable clock; swap out in tests to control time -setcp:{.z.m.cp:x} / allows runtime replacement of the clock without editing the module +/ whether synchronous calls via -30! are permitted +synccallsallowed:0b; -formatresponse:{[status;sync;result]$[not[status]and sync;'result;result]} / sync errors must be signalled with ' so the client receives a trapped error; async errors pass through unchanged -setformatresponse:{.z.m.formatresponse:x} / override reply formatting without editing the module +/ injectable clock - replaced in tests to control time without sleeping +cp:{.z.p}; -/ symbols backend servers call back via; set to wherever this module is mounted -resultcallback:`addserverresult; / stored as symbol so the name survives IPC serialization to backend processes +/ symbols backend servers call back via - stored as symbols so names survive IPC serialisation +resultcallback:`addserverresult; errorcallback:`addservererror; -setcallbacks:{[resfn;errfn].z.m.resultcallback:resfn;.z.m.errorcallback:errfn} / update callback symbols when module is mounted under a non-default namespace -/ registered backend servers -servers:([handle:`u#`int$()] servertype:`symbol$(); inuse:`boolean$(); active:`boolean$(); disconnecttime:`timestamp$()) +/ reply formatting - sync errors must be signalled with ' so the client receives a trapped error +formatresponse:{[status;sync;result]$[not[status]and sync;'result;result]}; -/ pending and in-flight client queries -queryqueue:([queryid:`u#`long$()] time:`timestamp$(); clienth:`int$(); query:(); servertype:(); join:(); postback:(); timeout:`timespan$(); returntime:`timestamp$(); error:`boolean$(); sync:`boolean$()) - -/ connected client tracking -clients:([] time:`timestamp$(); clienth:`int$(); user:`symbol$(); ip:`int$(); host:`symbol$()) - -/ per-query result accumulator: queryid -> (clienth; servertype!(handle;result;done)) -results:()!() - -queryid:0; - -addserver:{[h;st].z.M.servers upsert (h;st;0b;1b;0Np)} / register a backend handle and servertype so it becomes eligible for dispatch +/ scheduling strategy - pick oldest FIFO-eligible query by default +getnextqueryid:{ + avail:exec distinct servertype from availableservers 1b; + runnable:0!select from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, {all x in y}[;avail] each servertype; + 1 sublist select from runnable where time=min time}; -availableservers:{[excludeinuse] / centralise the active+idle filter so dispatch and routing share one definition +/ server routing strategy - active and idle by default +availableservers:{[excludeinuse] $[excludeinuse; select from servers where active, not inuse; select from servers where active]}; -setavailableservers:{.z.m.availableservers:x} / swap in a custom routing strategy without forking core dispatch -addclientdetails:{[h].z.M.clients insert (cp[];h;.z.u;.z.a;.z.h)} / record client identity on connect for audit and orphan-query cleanup on disconnect +/ ============================================================ +/ module tables +/ ============================================================ -removeclienthandle:{[h] / on client disconnect, mark their pending queries errored so result slots are not leaked - update error:1b,returntime:.z.m.cp[] from .z.M.queryqueue where clienth=h, null returntime; - .z.m.results:(exec queryid from .z.m.queryqueue where clienth=h)_results}; +/ registered backend servers +servers:([handle:`u#`int$()] servertype:`symbol$(); inuse:`boolean$(); active:`boolean$(); disconnecttime:`timestamp$()); -addquery:{[query;servertype;join;postback;timeout;sync] / enqueue a query without dispatching; caller must call runnextquery[] to trigger dispatch - .z.M.queryqueue upsert (queryid;cp[];.z.w;query;servertype;join;{$[11h=type x;enlist x;x]}postback;timeout;0Np;0b;sync); - .z.m.queryid:queryid+1}; +/ pending and in-flight client queries +queryqueue:([queryid:`u#`long$()] time:`timestamp$(); clienth:`int$(); query:(); servertype:(); join:(); postback:(); timeout:`timespan$(); returntime:`timestamp$(); error:`boolean$(); sync:`boolean$()); -removequeries:{[age] / prevent queryqueue growing unboundedly; purge completed queries older than age - .z.m.queryqueue:0!delete from .z.m.queryqueue where not null returntime, .z.m.cp[]>returntime+age}; +/ connected client tracking +clients:([] time:`timestamp$(); clienth:`int$(); user:`symbol$(); ip:`int$(); host:`symbol$()); -getnextqueryid:{ / pick the oldest FIFO-eligible query whose required servertypes are all currently idle - avail:exec distinct servertype from availableservers 1b; - runnable:0!select from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, {all x in y}[;avail] each servertype; - 1 sublist select from runnable where time=min time}; -setgetnextqueryid:{.z.m.getnextqueryid:x} / inject a priority or custom scheduling strategy without forking the module +/ per-query result accumulator: queryid -> (clienth; servertype!(handle;result;done)) +results:()!(); -addserverresult:{[qid;data] / fill one result slot; once all slots for a query are filled, run the join function and reply to the client - if[not qid in key results;:()]; - st:first exec servertype from .z.m.servers where handle=.z.w; - slots:results[qid;1]; - slots[st]:(.z.w;data;1b); - results[qid]:(results[qid;0];slots); - update inuse:0b from .z.M.servers where handle in .z.w; - runnextquery[]; - if[not qid in key results;:()]; - vals:value results[qid;1]; - if[not all vals[;2];:()]; - qd:queryqueue[qid]; - res:@[{(0b;x y)}[qd`join];vals[;1];{(1b;errorprefix,"join failed: ",x)}]; - sendclientreply[qid;last res;not res 0]; - finishquery[qid;res 0]}; +/ auto-incrementing query id counter +queryid:0; -addservererror:{[qid;err] / short-circuit a query on backend failure; free the server and notify the client before moving on - sendclientreply[qid;errorprefix,err;0b]; - update inuse:0b from .z.M.servers where handle in .z.w; - runnextquery[]; - finishquery[qid;1b]}; +/ ============================================================ +/ internal helpers +/ ============================================================ -sendclientreply:{[qid;result;status] / deliver result or error to the client, handling sync vs async send and postback wrapping in one place +normlog:{[logdict] + / detect kx.log instance by presence of kx.log-specific keys (getlvl, sinks, fmts) + / kx.log functions are monadic - wrap each into binary {[c;m]} and embed context in the message + / plain {[c;m]} log dicts (info`warn`error only) pass through unchanged + $[any `getlvl`sinks`fmts in key logdict; + `info`warn`error!( + {[fn;c;m] fn[string[c],": ",m]}[logdict`info;]; + {[fn;c;m] fn[string[c],": ",m]}[logdict`warn;]; + {[fn;c;m] fn[string[c],": ",m]}[logdict`error;]); + logdict] + }; + +sendclientreply:{[qid;result;status] + / deliver result or error to the client, handling sync vs async send and postback wrapping qd:queryqueue[qid]; if[qd`error;:()]; tosend:$[()~qd`postback;result;qd[`postback],enlist[qd`query],enlist result]; @@ -90,20 +87,25 @@ sendclientreply:{[qid;result;status] / deliver result or error to the client, ha @[-30!;(qd`clienth;not status;$[status;formatresponse[1b;1b;result];result]);{}]; @[neg qd`clienth;formatresponse[status;0b;tosend];()]]}; -finishquery:{[qid;err] / remove query from the live results accumulator and stamp its completion time; keeps queryqueue and results consistent +finishquery:{[qid;err] + / remove query from the live results accumulator and stamp its completion time .z.m.results:(qid,())_results; update error:err,returntime:.z.m.cp[] from .z.M.queryqueue where queryid in qid}; -serverexecute:{[qid;query] / runs on the backend; traps errors locally so a crash posts an error reply rather than silently dropping the result +serverexecute:{[qid;query] + / runs on the backend - traps errors so a crash posts an error reply rather than dropping the result res:@[{(0b;value x)};query;{(1b;"server ",(string .z.h),":",(string system"p"),": ",x)}]; @[neg .z.w;$[res 0;(errorcallback;qid;res 1);(resultcallback;qid;res 1)]; {@[neg .z.w;(errorcallback;x;"failed to return result: ",y);()]}[qid]]}; -sendquerytoserver:{[qid;query;handles] / fan the query out to all required handles and mark them in-use atomically to prevent double-dispatch +sendquerytoserver:{[qid;query;handles] + / fan the query out to all required handles and mark them in-use atomically (neg handles,:())@\:(serverexecute;qid;query); update inuse:1b from .z.M.servers where handle in handles}; -runnextquery:{ / pick the next dispatchable query and fan out to one idle server per required servertype; called after any state change that may unblock work +runnextquery:{[] + / pick the next dispatchable query and fan out to one idle server per required servertype + / called after any state change that may unblock work if[0=count torun:getnextqueryid[];:()]; torun:first torun; avail:exec first handle by servertype from availableservers 1b; @@ -115,34 +117,121 @@ runnextquery:{ / pick the next dispatchable query and fan out to one idle server results[qid]:(torun`clienth;slots); sendquerytoserver[qid;torun`query;handles]}; -checktimeout:{ / periodic scan to error queries that have waited beyond their timeout, preventing them from hanging indefinitely +addquery:{[query;servertype;join;postback;timeout;sync] + / enqueue a query without dispatching - caller must call runnextquery[] to trigger dispatch + .z.M.queryqueue upsert (queryid;.z.m.cp[];.z.w;query;servertype;join;{$[11h=type x;enlist x;x]}postback;timeout;0Np;0b;sync); + .z.m.queryid:queryid+1}; + +removequeries:{[age] + / prevent queryqueue growing unboundedly - purge completed queries older than age + delete from .z.M.queryqueue where not null returntime, .z.m.cp[]>returntime+age}; + +removeinactive:{[age] + / prune stale disconnected-server rows to stop the servers table growing forever + delete from .z.M.servers where not active, .z.m.cp[]>disconnecttime+age}; + +checktimeout:{[] + / periodic scan to error queries that have waited beyond their timeout qids:exec queryid from .z.m.queryqueue where not timeout=0Wn, null returntime, .z.m.cp[]>time+timeout; if[count qids; + .z.m.log[`warn][`asyncdispatch;"queries timed out: ",", " sv string qids]; sendclientreply[;errorprefix,"query timed out";0b] each qids; finishquery[qids;1b]]}; -removeserverhandle:{[serverh] / on backend disconnect, error in-flight queries using that handle and queued queries that can no longer be satisfied +/ ============================================================ +/ public api +/ ============================================================ + +setcp:{[f] + / replace the clock function - used in tests to control time without sleeping + .z.m.cp:f}; + +setformatresponse:{[f] + / override reply formatting - e.g. to wrap results in a standard envelope + .z.m.formatresponse:f}; + +setcallbacks:{[resfn;errfn] + / update callback symbols when module is mounted under a non-default namespace + .z.m.resultcallback:resfn; + .z.m.errorcallback:errfn}; + +setavailableservers:{[f] + / swap in a custom routing strategy without forking core dispatch + .z.m.availableservers:f}; + +setgetnextqueryid:{[f] + / inject a priority or custom scheduling strategy + .z.m.getnextqueryid:f}; + +addserver:{[h;st] + / register a backend handle and servertype so it becomes eligible for dispatch + .z.m.log[`info][`asyncdispatch;"server registered: ",string[st]," handle ",string h]; + .z.M.servers upsert (h;st;0b;1b;0Np)}; + +removeserverhandle:{[serverh] + / on backend disconnect, error in-flight queries using that handle and queued queries + / that can no longer be satisfied if[null st:first exec servertype from .z.m.servers where handle=serverh;:()]; err:errorprefix,"backend ",string[st]," server disconnected"; - + .z.m.log[`warn][`asyncdispatch;"backend disconnected: ",string st]; / in-flight: queries where this handle was assigned to a slot qids:where {[h;qid]h in value[.z.m.results[qid;1]][;0]}[serverh] each key .z.m.results; sendclientreply[;err," during query";0b] each qids; finishquery[qids;1b]; - / queued: queries that can no longer be satisfied by remaining active servers activetypes:exec distinct servertype from .z.m.servers where active, handle<>serverh; qids2:exec queryid from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, not {all x in y}[;activetypes] each servertype; sendclientreply[;err,", query cannot be satisfied";0b] each qids2; finishquery[qids2;1b]; - update active:0b,disconnecttime:.z.m.cp[] from .z.M.servers where handle=serverh; runnextquery[]}; -removeinactive:{[age]delete from .z.M.servers where not active, .z.m.cp[]>disconnecttime+age} / prune stale disconnected-server rows to stop the servers table growing forever +addclientdetails:{[h] + / record client identity on connect for audit and orphan-query cleanup on disconnect + .z.m.log[`info][`asyncdispatch;"client connected: handle ",string h]; + .z.M.clients insert (.z.m.cp[];h;.z.u;.z.a;.z.h)}; + +removeclienthandle:{[h] + / on client disconnect, mark their pending queries errored so result slots are not leaked + / free any servers in-flight for this client before removing result slots, then re-dispatch + .z.m.log[`info][`asyncdispatch;"client disconnected: handle ",string h]; + inflightqids:(exec queryid from .z.m.queryqueue where clienth=h, null returntime) inter key .z.m.results; + if[count inflightqids; + inflighthandles:distinct raze {value[.z.m.results[x;1]][;0]} each inflightqids; + update inuse:0b from .z.M.servers where handle in inflighthandles]; + update error:1b,returntime:.z.m.cp[] from .z.M.queryqueue where clienth=h, null returntime; + .z.m.results:(exec queryid from .z.m.queryqueue where clienth=h)_results; + runnextquery[]}; + +addserverresult:{[qid;data] + / fill one result slot - once all slots for a query are filled, run the join and reply + if[not qid in key results;:()]; + st:first exec servertype from .z.m.servers where handle=.z.w; + slots:results[qid;1]; + slots[st]:(.z.w;data;1b); + results[qid]:(results[qid;0];slots); + update inuse:0b from .z.M.servers where handle in .z.w; + runnextquery[]; + if[not qid in key results;:()]; + vals:value results[qid;1]; + if[not all vals[;2];:()]; + qd:queryqueue[qid]; + res:@[{(0b;x y)}[qd`join];vals[;1];{(1b;errorprefix,"join failed: ",x)}]; + if[res 0;.z.m.log[`error][`asyncdispatch;"join failed for query ",string qid,": ",last res]]; + sendclientreply[qid;last res;not res 0]; + finishquery[qid;res 0]}; + +addservererror:{[qid;err] + / short-circuit a query on backend failure - free the server and notify the client + .z.m.log[`error][`asyncdispatch;"backend error for query ",string[qid],": ",err]; + sendclientreply[qid;errorprefix,err;0b]; + update inuse:0b from .z.M.servers where handle in .z.w; + runnextquery[]; + finishquery[qid;1b]}; -execquery:{[query;servertype;join;postback;timeout;sync] / public entry point; validate sync constraints then enqueue and kick dispatch +execquery:{[query;servertype;join;postback;timeout;sync] + / public entry point - validate sync constraints then enqueue and kick dispatch if[sync; if[not synccallsallowed;'"syncexec: synchronous calls are not allowed"]; if[not @[{-30!x;1b};(::);0b];'"syncexec: deferred response not supported on this connection"]; @@ -151,9 +240,32 @@ execquery:{[query;servertype;join;postback;timeout;sync] / public entry point; v addquery[query;servertype;join;postback;timeout;0b]; runnextquery[]}; -/ wire housekeeping into a timer - pass (::) to skip -init:{[timerrepeat] / wire recurring housekeeping (timeout scan, query purge, server purge) into a provided timer; pass (::) to skip registration - if[not timerrepeat~(::); - timerrepeat[cp[];0Wp;0D00:05:00;(.z.m.removequeries;querykeeptime);"asyncdispatch: remove old queries"]; - timerrepeat[cp[];0Wp;0D00:00:05;(.z.m.checktimeout;`);"asyncdispatch: timeout expired queries"]; - timerrepeat[cp[];0Wp;0D00:05:00;(.z.m.removeinactive;clearinactivetime);"asyncdispatch: remove inactive servers"]]}; +init:{[deps] + / initialise the asyncdispatch module - validate deps and apply config overrides + / deps: dict containing `log (required) plus optional config keys: + / `log - required: `info`warn`error!({[c;m]};{[c;m]};{[c;m]}) binary loggers + / `errorprefix - optional: string prefix for client error messages. default: "error: " + / `querykeeptime - optional: timespan to keep completed queries. default: 0D00:30 + / `clearinactivetime - optional: timespan to keep disconnected servers. default: 0D01:00 + / `synccallsallowed - optional: boolean, whether sync calls are permitted. default: 0b + / housekeeping (checktimeout; removequeries; removeinactive) is the caller's responsibility - + / wire them into your timer after init; the exported defaults for age params are querykeeptime + / and clearinactivetime + / examples: + / ad.init[enlist[`log]!enlist logdep] + / ad.init[`log`querykeeptime!(logdep;0D01:00)] + if[99h<>type deps; + '"di.asyncdispatch: deps must be a dict with `log key"]; + if[not `log in key deps; + '"di.asyncdispatch: log dependency is required; pass `info`warn`error!(infofn;warnfn;errfn) keyed on `log"]; + if[99h<>type deps`log; + '"di.asyncdispatch: log value must be a dict; pass `info`warn`error functions"]; + if[not all `info`warn`error in key deps`log; + '"di.asyncdispatch: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + .z.m.log:normlog deps`log; + if[`errorprefix in key deps; .z.m.errorprefix:deps`errorprefix]; + if[`querykeeptime in key deps; .z.m.querykeeptime:deps`querykeeptime]; + if[`clearinactivetime in key deps; .z.m.clearinactivetime:deps`clearinactivetime]; + if[`synccallsallowed in key deps; .z.m.synccallsallowed:deps`synccallsallowed]; + .z.m.log[`info][`asyncdispatch;"di.asyncdispatch initialised"]; + }; diff --git a/di/asyncdispatch/init.q b/di/asyncdispatch/init.q index f7086488..372cf91f 100644 --- a/di/asyncdispatch/init.q +++ b/di/asyncdispatch/init.q @@ -1,10 +1,11 @@ +/ asyncdispatch module - async scatter-gather query coordinator for multi-backend kdb+ gateway processes \l ::asyncdispatch.q export:([ - errorprefix;querykeeptime;clearinactivetime;synccallsallowed; / user-tunable config: error text prefix and housekeeping intervals - setcp;setformatresponse;setcallbacks;setavailableservers;setgetnextqueryid; / user-injectable overrides: clock, reply format, callback namespace, routing and scheduling - addserver;removeserverhandle; / server lifecycle: register and deregister backends - addclientdetails;removeclienthandle; / client lifecycle: wire into .z.po / .z.pc - addserverresult;addservererror; / IPC return paths: backends resolve these by name to deliver results or errors - execquery; / public API: only entry point for submitting a query - init]) / startup: wires housekeeping into the provided timer + setformatresponse;setcallbacks;setavailableservers;setgetnextqueryid; + addserver;removeserverhandle; + addclientdetails;removeclienthandle; + addserverresult;addservererror; + execquery; + checktimeout;removequeries;removeinactive; + init]) diff --git a/di/asyncdispatch/test.csv b/di/asyncdispatch/test.csv index 7366d787..4a83ca96 100644 --- a/di/asyncdispatch/test.csv +++ b/di/asyncdispatch/test.csv @@ -1,19 +1,22 @@ action,ms,bytes,lang,code,repeat,minver,comment -before,0,0,q,system"q -p 19501 -q &",1,,start backend process +before,0,0,q,p:first {x where {10h=type @[hopen;`$raze("::";string x);{x}]} each x}[19500+til 100],1,,find first free port in range 19500-19599 +before,0,0,q,system raze("q -p ";string p;" -q &"),1,,start backend process on free port before,0,0,q,system"sleep 1",1,,wait for backend to initialise before,0,0,q,ad:use`di.asyncdispatch,1,,load module before,0,0,q,srv:{.m.di.0asyncdispatch.servers},1,,live servers table before,0,0,q,qq:{.m.di.0asyncdispatch.queryqueue},1,,live queryqueue table before,0,0,q,cl:{.m.di.0asyncdispatch.clients},1,,live clients table before,0,0,q,res:{.m.di.0asyncdispatch.results},1,,live results dict -before,0,0,q,bh:hopen`::19501,1,,open handle to backend +before,0,0,q,mocklog:`info`warn`error!3#enlist{[c;m]},1,,mock log dep - binary {[c;m]} no-op +before,0,0,q,ad.init[enlist[`log]!enlist mocklog],1,,initialise with mock logger +before,0,0,q,bh:hopen`$raze("::";string p),1,,open handle to backend before,0,0,q,bh".m.di.0asyncdispatch.resultcallback:`.m.di.0asyncdispatch.addserverresult",1,,configure backend result callback to dispatcher before,0,0,q,bh".m.di.0asyncdispatch.errorcallback:`.m.di.0asyncdispatch.addservererror",1,,configure backend error callback to dispatcher before,0,0,q,ad.addserver[bh;`mock],1,,register backend as mock server / Test 0: server registry true,0,0,q,(enlist bh)~exec handle from srv[] where active,1,,backend server is active -true,0,0,q,(enlist bh)~exec handle from ad.availableservers 1b,1,,idle backend server is available +true,0,0,q,(enlist bh)~exec handle from .m.di.0asyncdispatch.availableservers 1b,1,,idle backend server is available / Test 1: queueing and FIFO scheduling run,0,0,q,.m.di.0asyncdispatch.addquery["2+2";enlist`mock;raze;();0Wn;0b],1,,queue a query for the mock servertype @@ -32,15 +35,15 @@ true,0,0,q,not qid1 in key res[],1,,result accumulator cleaned up after join / Test 3: checktimeout run,0,0,q,.m.di.0asyncdispatch.addquery["3+3";enlist`mock;raze;();0D00:00:00.000000001;0b],1,,queue with a near-zero timeout run,0,0,q,qid2:exec first queryid from qq[] where query~\:"3+3",1,,capture the queued query id -run,0,0,q,.m.di.0asyncdispatch.checktimeout[],1,,timeout sweep should catch the expired query +run,0,0,q,ad.checktimeout[],1,,timeout sweep should catch the expired query true,0,0,q,1b~first exec error from qq[] where queryid=qid2,1,,timed-out query is flagged as error true,0,0,q,not null first exec returntime from qq[] where queryid=qid2,1,,timed-out query has returntime stamped / Test 4: removequeries purges completed queries -run,0,0,q,ad.setcp[{.z.p+2D}],1,,fast-forward the clock by 2 days -run,0,0,q,.m.di.0asyncdispatch.removequeries ad.querykeeptime,1,,purge queries older than querykeeptime -true,0,0,q,0~count select from qq[] where queryid in qid1,qid2,1,both completed queries purged -run,0,0,q,ad.setcp[{.z.p}],1,,restore the real clock +run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p+2D}],1,,fast-forward the clock by 2 days +run,0,0,q,ad.removequeries .m.di.0asyncdispatch.querykeeptime,1,,purge queries older than querykeeptime +true,0,0,q,0~count select from qq[] where queryid in (qid1;qid2),1,,both completed queries purged +run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p}],1,,restore the real clock / Test 5: removeserverhandle errors queued queries when the only server disconnects run,0,0,q,ad.addserver[1i;`mock2],1,,register a second mock server (fake handle - query is errored before dispatch) @@ -51,16 +54,16 @@ true,0,0,q,1b~first exec error from qq[] where queryid=qid3,1,,query errored as true,0,0,q,0b~first exec active from srv[] where handle=1i,1,,disconnected server marked inactive / Test 6: removeinactive purges inactive server records -run,0,0,q,ad.setcp[{.z.p+2D}],1,,fast-forward clock past clearinactivetime -run,0,0,q,.m.di.0asyncdispatch.removeinactive ad.clearinactivetime,1,,purge inactive servers +run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p+2D}],1,,fast-forward clock past clearinactivetime +run,0,0,q,ad.removeinactive .m.di.0asyncdispatch.clearinactivetime,1,,purge inactive servers true,0,0,q,0~count select from srv[] where handle=1i,1,,inactive server record removed -run,0,0,q,ad.setcp[{.z.p}],1,,restore real clock +run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p}],1,,restore real clock / Test 7: client tracking run,0,0,q,ad.addclientdetails[2i],1,,record a connected client true,0,0,q,1~count select from cl[] where clienth=2i,1,,client connection is tracked run,0,0,q,ad.removeclienthandle[2i],1,,disconnect the client -true,0,0,q,0~count select from qq[] where clienth=2i,null returntime,1,client unfinished queries cleared +true,0,0,q,0~count select from qq[] where (clienth=2i)&null returntime,1,,client unfinished queries cleared / Test 8: pluggable hooks run,0,0,q,ad.setformatresponse[{[status;sync;result]result}],1,,override formatresponse to pass result through @@ -70,6 +73,17 @@ true,0,0,q,0~count .m.di.0asyncdispatch.getnextqueryid[],1,,overridden scheduler run,0,0,q,ad.setcallbacks[`myresult;`myerror],1,,override callback symbols true,0,0,q,(`myresult;`myerror)~(.m.di.0asyncdispatch.resultcallback;.m.di.0asyncdispatch.errorcallback),1,,callback symbols updated +/ Test 9: removeclienthandle frees in-flight servers on client disconnect +run,0,0,q,ad.addserver[99i;`inflight],1,,register fake server for in-flight test +run,0,0,q,update inuse:1b from .m.di.0asyncdispatch.servers where handle=99i,1,,manually mark server in-use to simulate dispatched query +run,0,0,q,ad.addclientdetails[5i],1,,add fake client +run,0,0,q,.m.di.0asyncdispatch.addquery["5+5";enlist`inflight;raze;();0Wn;0b],1,,queue query for fake server +run,0,0,q,qid4:exec first queryid from qq[] where query~\:"5+5",1,,capture query id +run,0,0,q,.m.di.0asyncdispatch.results[qid4]:(5i;enlist[`inflight]!enlist(99i;(::);0b)),1,,inject in-flight slot to simulate dispatched-but-pending query +run,0,0,q,ad.removeclienthandle[5i],1,,disconnect client mid-flight +true,0,0,q,0b~first exec inuse from srv[] where handle=99i,1,,in-flight server freed on client disconnect +run,0,0,q,ad.removeserverhandle[99i],1,,clean up fake server + / Teardown run,0,0,q,neg[bh](exit;0);neg[bh](::),1,,send exit to backend process and flush run,0,0,q,hclose bh,1,,close backend handle From fc58e48e774c50b4832a9224df8b1e7e05eb3b08 Mon Sep 17 00:00:00 2001 From: alowrydi Date: Mon, 29 Jun 2026 17:42:06 +0100 Subject: [PATCH 03/12] Add tests for addservererror, postback wrapping and multi-servertype scatter-gather --- di/asyncdispatch/asyncdispatch.md | 2 +- di/asyncdispatch/test.csv | 56 +++++++++++++++++++++++++++---- 2 files changed, 50 insertions(+), 8 deletions(-) diff --git a/di/asyncdispatch/asyncdispatch.md b/di/asyncdispatch/asyncdispatch.md index 7511df84..d720417b 100644 --- a/di/asyncdispatch/asyncdispatch.md +++ b/di/asyncdispatch/asyncdispatch.md @@ -209,7 +209,7 @@ k4unit:use`di.k4unit k4unit.moduletest`di.asyncdispatch ``` -52 tests. Requires a `q` binary in `PATH` — the test suite starts a real backend process on a dynamically selected free port and exercises the full dispatch lifecycle over a live IPC connection. No TorQ installation or special libraries required. Covers: server registry, FIFO scheduling, full IPC round-trip with join and reply, checktimeout, removequeries, removeserverhandle with orphaned query cleanup, client tracking, removeinactive, in-flight server release on client disconnect, and all pluggable hook setters. +82 tests. Requires a `q` binary in `PATH` — the test suite starts two real backend processes on dynamically selected free ports and exercises the full dispatch lifecycle over live IPC connections. No TorQ installation or special libraries required. Covers: server registry, FIFO scheduling, full IPC round-trip with join and reply, backend error path via real IPC callback (`addservererror`), postback wrapping (`tosend` tuple construction), multi-servertype scatter-gather with two backends dispatched in parallel and results joined, checktimeout, removequeries, removeserverhandle with orphaned query cleanup, client tracking, removeinactive, in-flight server release on client disconnect, and all pluggable hook setters. --- diff --git a/di/asyncdispatch/test.csv b/di/asyncdispatch/test.csv index 4a83ca96..4dd1cba3 100644 --- a/di/asyncdispatch/test.csv +++ b/di/asyncdispatch/test.csv @@ -1,7 +1,9 @@ action,ms,bytes,lang,code,repeat,minver,comment before,0,0,q,p:first {x where {10h=type @[hopen;`$raze("::";string x);{x}]} each x}[19500+til 100],1,,find first free port in range 19500-19599 -before,0,0,q,system raze("q -p ";string p;" -q &"),1,,start backend process on free port -before,0,0,q,system"sleep 1",1,,wait for backend to initialise +before,0,0,q,p2:first {x where {10h=type @[hopen;`$raze("::";string x);{x}]} each x}[19600+til 100],1,,find second free port in range 19600-19699 +before,0,0,q,system raze("q -p ";string p;" -q &"),1,,start first backend process on free port +before,0,0,q,system raze("q -p ";string p2;" -q &"),1,,start second backend process on free port +before,0,0,q,system"sleep 1",1,,wait for both backends to initialise before,0,0,q,ad:use`di.asyncdispatch,1,,load module before,0,0,q,srv:{.m.di.0asyncdispatch.servers},1,,live servers table before,0,0,q,qq:{.m.di.0asyncdispatch.queryqueue},1,,live queryqueue table @@ -12,11 +14,17 @@ before,0,0,q,ad.init[enlist[`log]!enlist mocklog],1,,initialise with mock logger before,0,0,q,bh:hopen`$raze("::";string p),1,,open handle to backend before,0,0,q,bh".m.di.0asyncdispatch.resultcallback:`.m.di.0asyncdispatch.addserverresult",1,,configure backend result callback to dispatcher before,0,0,q,bh".m.di.0asyncdispatch.errorcallback:`.m.di.0asyncdispatch.addservererror",1,,configure backend error callback to dispatcher -before,0,0,q,ad.addserver[bh;`mock],1,,register backend as mock server +before,0,0,q,ad.addserver[bh;`mock],1,,register first backend as mock server +before,0,0,q,bh2:hopen`$raze("::";string p2),1,,open handle to second backend +before,0,0,q,bh2".m.di.0asyncdispatch.resultcallback:`.m.di.0asyncdispatch.addserverresult",1,,configure second backend result callback +before,0,0,q,bh2".m.di.0asyncdispatch.errorcallback:`.m.di.0asyncdispatch.addservererror",1,,configure second backend error callback +before,0,0,q,ad.addserver[bh2;`scatter],1,,register second backend as scatter servertype / Test 0: server registry -true,0,0,q,(enlist bh)~exec handle from srv[] where active,1,,backend server is active -true,0,0,q,(enlist bh)~exec handle from .m.di.0asyncdispatch.availableservers 1b,1,,idle backend server is available +true,0,0,q,bh in exec handle from srv[] where active,1,,first backend server is active +true,0,0,q,bh2 in exec handle from srv[] where active,1,,second backend server is active +true,0,0,q,bh in exec handle from .m.di.0asyncdispatch.availableservers 1b,1,,first backend is available for dispatch +true,0,0,q,bh2 in exec handle from .m.di.0asyncdispatch.availableservers 1b,1,,second backend is available for dispatch / Test 1: queueing and FIFO scheduling run,0,0,q,.m.di.0asyncdispatch.addquery["2+2";enlist`mock;raze;();0Wn;0b],1,,queue a query for the mock servertype @@ -84,6 +92,40 @@ run,0,0,q,ad.removeclienthandle[5i],1,,disconnect client mid-flight true,0,0,q,0b~first exec inuse from srv[] where handle=99i,1,,in-flight server freed on client disconnect run,0,0,q,ad.removeserverhandle[99i],1,,clean up fake server +/ Test 10: addservererror - real backend error path via IPC callback +run,0,0,q,ad.setcallbacks[`.m.di.0asyncdispatch.addserverresult;`.m.di.0asyncdispatch.addservererror],1,,restore callbacks overridden in test 8 +run,0,0,q,ad.setgetnextqueryid[{r:0!select from .m.di.0asyncdispatch.queryqueue where null returntime;1 sublist select from r where not queryid in key .m.di.0asyncdispatch.results}],1,,restore scheduler overridden in test 8 +run,0,0,q,.m.di.0asyncdispatch.addquery["'errortest";enlist`mock;raze;();0Wn;0b],1,,queue a query that will signal an error on the backend +run,0,0,q,qid5:exec first queryid from qq[] where query~\:"'errortest",1,,capture the queued query id +run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch query to backend +run,0,0,q,bh"",1,,sync round-trip: ensures backend has processed and sent error callback +true,0,0,q,1b~first exec error from qq[] where queryid=qid5,1,,backend error callback flagged query as error +true,0,0,q,not null first exec returntime from qq[] where queryid=qid5,1,,returntime stamped on backend error + +/ Test 11: postback wrapping - non-empty postback produces (postback;query;result) tuple +run,0,0,q,captured::(),1,,global to capture what formatresponse receives +run,0,0,q,ad.setformatresponse[{[s;sync;r]captured::r;$[not[s]and sync;'r;r]}],1,,intercept formatresponse to verify tosend construction +run,0,0,q,.m.di.0asyncdispatch.addquery["7+7";enlist`mock;raze;enlist(::);0Wn;0b],1,,queue query with non-empty postback +run,0,0,q,qid6:exec first queryid from qq[] where query~\:"7+7",1,,capture query id +run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch +run,0,0,q,bh"",1,,sync flush +true,0,0,q,0b~first exec error from qq[] where queryid=qid6,1,,postback query completed without error +true,0,0,q,(::;"7+7";enlist 14)~captured,1,,formatresponse received (postback;query;result) tuple +run,0,0,q,ad.setformatresponse[{[status;sync;result]$[not[status]and sync;'result;result]}],1,,restore default formatresponse + +/ Test 12: multi-servertype scatter-gather - two backends dispatched in parallel results joined +run,0,0,q,ad.setformatresponse[{[s;sync;r]captured::r;$[not[s]and sync;'r;r]}],1,,intercept formatresponse to capture joined result +run,0,0,q,.m.di.0asyncdispatch.addquery["1+1";`mock`scatter;{sum x};();0Wn;0b],1,,queue scatter-gather query requiring both servertypes +run,0,0,q,qid7:exec first queryid from qq[] where query~\:"1+1",1,,capture query id +run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch to both backends in parallel +run,0,0,q,bh"",1,,sync flush first backend +run,0,0,q,bh2"",1,,sync flush second backend +true,0,0,q,0b~first exec error from qq[] where queryid=qid7,1,,scatter-gather completed without error +true,0,0,q,4~captured,1,,joined result is sum of both backend results (1+1=2 from each; sum 2 2 = 4) +run,0,0,q,ad.setformatresponse[{[status;sync;result]$[not[status]and sync;'result;result]}],1,,restore default formatresponse + / Teardown -run,0,0,q,neg[bh](exit;0);neg[bh](::),1,,send exit to backend process and flush -run,0,0,q,hclose bh,1,,close backend handle +run,0,0,q,neg[bh](exit;0);neg[bh](::),1,,send exit to first backend process and flush +run,0,0,q,hclose bh,1,,close first backend handle +run,0,0,q,neg[bh2](exit;0);neg[bh2](::),1,,send exit to second backend process and flush +run,0,0,q,hclose bh2,1,,close second backend handle From d31f01aefa108639f892fe552d98fa730948160d Mon Sep 17 00:00:00 2001 From: alowrydi Date: Fri, 3 Jul 2026 17:19:39 +0100 Subject: [PATCH 04/12] Add execqueryto in-process routing, removeclients, brace style, normlog fix, 116 tests --- di/asyncdispatch/asyncdispatch.md | 68 +++++--- di/asyncdispatch/asyncdispatch.q | 264 ++++++++++++++++++------------ di/asyncdispatch/init.q | 6 +- di/asyncdispatch/test.csv | 47 ++++++ 4 files changed, 259 insertions(+), 126 deletions(-) diff --git a/di/asyncdispatch/asyncdispatch.md b/di/asyncdispatch/asyncdispatch.md index d720417b..a4f1afc9 100644 --- a/di/asyncdispatch/asyncdispatch.md +++ b/di/asyncdispatch/asyncdispatch.md @@ -33,10 +33,10 @@ A `kx.log` instance can be passed directly — the module normalises monadic fun kxlog:use`kx.log ad:use`di.asyncdispatch -/ minimal +// minimal ad.init[enlist[`log]!enlist kxlog.createLog[]] -/ with config overrides +// with config overrides ad.init[`log`querykeeptime`synccallsallowed!(kxlog.createLog[];0D01:00;1b)] ``` @@ -54,7 +54,7 @@ ad.init[`log`querykeeptime`synccallsallowed!(kxlog.createLog[];0D01:00;1b)] | `` `clearinactivetime `` | no | `0D01:00` | How long `removeinactive` retains disconnected server rows | | `` `synccallsallowed `` | no | `0b` | Whether `execquery[...;1b]` (deferred sync mode) is permitted | -Housekeeping — `checktimeout`, `removequeries`, and `removeinactive` — is the caller's responsibility. Wire them into your gateway's timer after `init`. The configured default age parameters are accessible as `querykeeptime` and `clearinactivetime` via module state. +Housekeeping — `checktimeout`, `removequeries`, `removeinactive`, and `removeclients` — is the caller's responsibility. Wire them into your gateway's timer after `init`. The configured default age parameters are accessible as `querykeeptime` and `clearinactivetime` via module state. --- @@ -93,13 +93,13 @@ On client disconnect, mark their pending queries errored so result slots are not ### `addserverresult[qid;data]` Called when a backend posts back a successful result. Fills the result slot, frees the server, triggers `runnextquery`, and — once all slots for the query are received — applies the join function and replies to the client. ```q -/ called by serverexecute on the backend; not typically called directly +// called by serverexecute on the backend; not typically called directly ``` ### `addservererror[qid;err]` Called when a backend posts back an error. Sends the error to the client and finishes the query. ```q -/ called by serverexecute on the backend; not typically called directly +// called by serverexecute on the backend; not typically called directly ``` ### `execquery[query;servertype;join;postback;timeout;sync]` @@ -118,29 +118,54 @@ Public entry point. Validates sync constraints, enqueues the query, and triggers ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;0b] ``` +### `execqueryto[replyto;query;servertype;join;postback;timeout;sync]` +Variant of `execquery` with an explicit reply target. Pass `replyto:0Ni` to invoke the postback locally via `value` instead of IPC-sending to a handle. Pass any valid int handle to route the reply to a specific process regardless of `.z.w`. Used when the caller is in the same process as the gateway (e.g. `di.dataaccess` dispatching shards). + +| Argument | Type | Description | +|---|---|---| +| `replyto` | int | `0Ni` for local in-process invocation; any int handle to IPC-send to a specific target | +| `query` | any | Payload passed to `value` on the backend | +| `servertype` | symbol list | One symbol per required backend type | +| `join` | function | Applied to the list of per-server results once all are received | +| `postback` | list | Required when `replyto` is `0Ni` — `(function;extra_args...)` invoked locally; `()` permitted for non-local targets | +| `timeout` | timespan | `0Wn` for no timeout | +| `sync` | boolean | Must be `0b` when `replyto` is `0Ni`; sync not supported for local invocation | + +```q +// di.dataaccess wiring: shard result delivered locally to da.shardresult +ad.execqueryto[0Ni;shardquery;servertypes;raze;(`da.shardresult;reqid);0Wn;0b] +``` + ### `checktimeout[]` Scan the queue for queries past their timeout, send a timeout error to each client, and mark them complete. Wire into your gateway's timer — every few seconds is typical. ```q -/ in gateway timer +// in gateway timer timer.addjob.default[`asyncdispatch.checktimeout;{ad.checktimeout[]};();5i;1] ``` ### `removequeries[age]` Purge completed `queryqueue` rows older than `age`. Prevents unbounded growth. ```q -/ default age is querykeeptime (0D00:30) +// default age is querykeeptime (0D00:30) timer.addjob.default[`asyncdispatch.removequeries;{ad.removequeries[0D00:30]};();300i;1] ``` ### `removeinactive[age]` Purge `servers` rows for backends that have been disconnected longer than `age`. Prevents unbounded growth. ```q -/ default age is clearinactivetime (0D01:00) +// default age is clearinactivetime (0D01:00) timer.addjob.default[`asyncdispatch.removeinactive;{ad.removeinactive[0D01:00]};();300i;1] ``` +### `removeclients[age]` +Purge `clients` rows older than `age`. The `clients` table is appended to on every client connect (`addclientdetails`) for audit and is not otherwise pruned, so wire this into the timer to prevent unbounded growth. Choose `age` to match your audit retention requirements. +```q +// retain one day of client audit history +timer.addjob.default[`asyncdispatch.removeclients;{ad.removeclients[1D]};();300i;1] +``` + ### `setformatresponse[f]` -Override the reply formatter applied before a result or error is sent to the client. `f` must be `{[status;sync;result]}`. +Override the reply formatter applied before a result or error is sent to the client. `f` must be `{[status;sync;result]}`. Note: `formatresponse` is only applied on the remote IPC path — queries dispatched via `execqueryto` with `replyto:0Ni` invoke the postback directly and bypass this formatter. ```q ad.setformatresponse[{[status;sync;result]result}] ``` @@ -160,7 +185,7 @@ ad.setavailableservers[{[excl]select from servers where active}] ### `setgetnextqueryid[f]` Inject a custom scheduling strategy. `f` must be niladic and return a 0- or 1-row table with the `queryqueue` schema. ```q -/ priority queue example - highest-priority query first +// priority queue example - highest-priority query first ad.setgetnextqueryid[{1 sublist `priority xdesc 0!select from .z.m.queryqueue where null returntime}] ``` @@ -176,27 +201,28 @@ timer.init[()!()] ad:use`di.asyncdispatch ad.init[enlist[`log]!enlist kxlog.createLog[]] -/ point backends' callbacks at this module's mount point on this process +// point backends' callbacks at this module's mount point on this process ad.setcallbacks[`ad.addserverresult;`ad.addservererror] -/ register backend connections as they connect +// register backend connections as they connect ad.addserver[hopen`:backend1:5001;`rdb] ad.addserver[hopen`:backend2:5002;`hdb] -/ wire client and server connection/disconnection handlers +// wire client and server connection/disconnection handlers .z.po:{ad.addclientdetails[.z.w]} .z.pc:{ad.removeserverhandle[.z.w];ad.removeclienthandle[.z.w]} -/ wire housekeeping into the gateway timer +// wire housekeeping into the gateway timer timer.addjob.default[`asyncdispatch.checktimeout;{ad.checktimeout[]};();5i;1] timer.addjob.default[`asyncdispatch.removequeries;{ad.removequeries[0D00:30]};();300i;1] timer.addjob.default[`asyncdispatch.removeinactive;{ad.removeinactive[0D01:00]};();300i;1] +timer.addjob.default[`asyncdispatch.removeclients;{ad.removeclients[1D]};();300i;1] -/ a client calls this asynchronously: -/ execquery dispatches to rdb and hdb in parallel, razes results, replies to client +// a client calls this asynchronously: +// execquery dispatches to rdb and hdb in parallel, razes results, replies to client ad.execquery[("select count i by sym from trade";"select count i by sym from trade");`rdb`hdb;raze;();0Wn;0b] -/ synchronous deferred mode (requires synccallsallowed:1b in deps) +// synchronous deferred mode (requires synccallsallowed:1b in deps) ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;1b] ``` @@ -209,16 +235,20 @@ k4unit:use`di.k4unit k4unit.moduletest`di.asyncdispatch ``` -82 tests. Requires a `q` binary in `PATH` — the test suite starts two real backend processes on dynamically selected free ports and exercises the full dispatch lifecycle over live IPC connections. No TorQ installation or special libraries required. Covers: server registry, FIFO scheduling, full IPC round-trip with join and reply, backend error path via real IPC callback (`addservererror`), postback wrapping (`tosend` tuple construction), multi-servertype scatter-gather with two backends dispatched in parallel and results joined, checktimeout, removequeries, removeserverhandle with orphaned query cleanup, client tracking, removeinactive, in-flight server release on client disconnect, and all pluggable hook setters. +116 tests. Requires a `q` binary in `PATH` — the test suite starts two real backend processes on dynamically selected free ports and exercises the full dispatch lifecycle over live IPC connections. No TorQ installation or special libraries required. Covers: server registry, FIFO scheduling, full IPC round-trip with join and reply, backend error path via real IPC callback (`addservererror`), join-failure path (throwing join function flagged as error), postback wrapping (`tosend` tuple construction), multi-servertype scatter-gather with two backends dispatched in parallel and results joined, checktimeout, removequeries, removeinactive, removeclients audit-row purge, removeserverhandle with orphaned query cleanup, client tracking, in-flight server release on client disconnect, all pluggable hook setters, and local in-process reply path via `execqueryto` (success, backend error, timeout, and non-null explicit replyto variants). --- ## Notes -- Housekeeping (`checktimeout`, `removequeries`, `removeinactive`) is the caller's responsibility — the gateway process already has a timer running and is better placed to decide intervals. Wire all three after `init`; see the usage example above +- Housekeeping (`checktimeout`, `removequeries`, `removeinactive`, `removeclients`) is the caller's responsibility — the gateway process already has a timer running and is better placed to decide intervals. Wire all four after `init`; see the usage example above +- When `checktimeout` fires for a query that is already in-flight (dispatched to a backend but not yet answered), the client receives the timeout error and the query is marked done, but the backend that received the dispatch remains `inuse:1b` until it eventually replies. A slow-but-alive backend holds its dispatch slot until `addserverresult` or `addservererror` fires; a permanently hung backend holds it until `removeserverhandle` fires on disconnect. Size backend pools and timeouts with this in mind — a run of timeouts against a stuck backend will not free its slot on timeout alone - `.z.M.` is used for in-place mutation of tables (`upsert`, `insert`, `update from`, `delete from`) and `.z.m.:value` for whole-variable reassignment — the same convention used by `di.cache` - Module globals referenced inside q-sql expressions (WHERE conditions, UPDATE SET values) must use the `.z.m.varname` form since q-sql evaluates column expressions in the calling context rather than the module namespace - `servertype` in `queryqueue` and `addquery` is a list of servertype symbols — one per required backend type. Pass `` enlist`rdb `` for single-server queries, `` `rdb`hdb `` for scatter-gather across two types - `setcallbacks` must be called before any queries are dispatched if the module is mounted under a non-default path — `serverexecute` reads `resultcallback` and `errorcallback` by bare name on the backend process and posts back to whatever symbols they resolve to - This module opens and accepts no connections itself — `addserver` and the `.z.po`/`.z.pc` wiring are the consumer's responsibility, keeping the module dependency-free and testable in-process - All three log keys (`info`, `warn`, `error`) are required — the module calls `info` on server/client connect and init, `warn` on disconnect and timeout, and `error` on backend error and join failure +- `execqueryto` with `replyto:0Ni` invokes the postback locally via `value` — the postback head symbol must be mount-qualified (e.g. `` `da.shardresult ``) since bare exported names are not globals. This is the same resolution mechanism used by `setcallbacks` +- Local queries store `clienth:0Ni` and are not matched by `removeclienthandle` — the in-process caller owns disconnect cleanup for its own requests +- `setformatresponse` overrides only apply to the remote IPC path — local invocation via `execqueryto[0Ni;...]` calls the postback directly without applying `formatresponse`. The default `formatresponse` is a pass-through for async, so this is transparent by default; consumers that override it should not combine that with local invocation diff --git a/di/asyncdispatch/asyncdispatch.q b/di/asyncdispatch/asyncdispatch.q index 86bcc8ce..85b7f5da 100644 --- a/di/asyncdispatch/asyncdispatch.q +++ b/di/asyncdispatch/asyncdispatch.q @@ -1,76 +1,79 @@ -/ di.asyncdispatch - async scatter-gather query coordinator -/ queues queries, dispatches to available backends, collects results per server, -/ applies a join function, and replies to the client -/ routing (which servers satisfy a query) is di.serverselect's responsibility -/ the log dependency is required - init errors immediately if absent or malformed -/ log functions are binary {[c;m]} where c is a symbol context and m is a string - -/ ============================================================ -/ module state and defaults -/ ============================================================ - -/ error prefix prepended to all error strings returned to clients +// di.asyncdispatch - async scatter-gather query coordinator +// queues queries, dispatches to available backends, collects results per server, +// applies a join function, and replies to the client +// routing (which servers satisfy a query) is di.serverselect's responsibility +// the log dependency is required - init errors immediately if absent or malformed +// log functions are binary {[c;m]} where c is a symbol context and m is a string + +// ============================================================ +// module state and defaults +// ============================================================ + +// error prefix prepended to all error strings returned to clients errorprefix:"error: "; -/ how long completed queries are kept in queryqueue before being purged +// how long completed queries are kept in queryqueue before being purged querykeeptime:0D00:30; -/ how long disconnected servers are kept in the servers table before being removed +// how long disconnected servers are kept in the servers table before being removed clearinactivetime:0D01:00; -/ whether synchronous calls via -30! are permitted +// whether synchronous calls via -30! are permitted synccallsallowed:0b; -/ injectable clock - replaced in tests to control time without sleeping +// injectable clock - replaced in tests to control time without sleeping cp:{.z.p}; -/ symbols backend servers call back via - stored as symbols so names survive IPC serialisation +// symbols backend servers call back via - stored as symbols so names survive IPC serialisation resultcallback:`addserverresult; errorcallback:`addservererror; -/ reply formatting - sync errors must be signalled with ' so the client receives a trapped error +// reply formatting - sync errors must be signalled with ' so the client receives a trapped error formatresponse:{[status;sync;result]$[not[status]and sync;'result;result]}; -/ scheduling strategy - pick oldest FIFO-eligible query by default +// scheduling strategy - pick oldest FIFO-eligible query by default getnextqueryid:{ avail:exec distinct servertype from availableservers 1b; + // 0! is required - select from a keyed table stays keyed in kdb-x, and runnextquery needs queryid via first runnable:0!select from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, {all x in y}[;avail] each servertype; - 1 sublist select from runnable where time=min time}; + 1 sublist select from runnable where time=min time + }; -/ server routing strategy - active and idle by default +// server routing strategy - active and idle by default availableservers:{[excludeinuse] $[excludeinuse; select from servers where active, not inuse; - select from servers where active]}; + select from servers where active] + }; -/ ============================================================ -/ module tables -/ ============================================================ +// ============================================================ +// module tables +// ============================================================ -/ registered backend servers +// registered backend servers servers:([handle:`u#`int$()] servertype:`symbol$(); inuse:`boolean$(); active:`boolean$(); disconnecttime:`timestamp$()); -/ pending and in-flight client queries -queryqueue:([queryid:`u#`long$()] time:`timestamp$(); clienth:`int$(); query:(); servertype:(); join:(); postback:(); timeout:`timespan$(); returntime:`timestamp$(); error:`boolean$(); sync:`boolean$()); +// pending and in-flight client queries +queryqueue:([queryid:`u#`long$()] time:`timestamp$(); clienth:`int$(); query:(); servertype:(); join:(); postback:(); timeout:`timespan$(); returntime:`timestamp$(); error:`boolean$(); sync:`boolean$(); local:`boolean$()); -/ connected client tracking +// connected client tracking clients:([] time:`timestamp$(); clienth:`int$(); user:`symbol$(); ip:`int$(); host:`symbol$()); -/ per-query result accumulator: queryid -> (clienth; servertype!(handle;result;done)) +// per-query result accumulator: queryid -> (clienth; servertype!(handle;result;done)) results:()!(); -/ auto-incrementing query id counter +// auto-incrementing query id counter queryid:0; -/ ============================================================ -/ internal helpers -/ ============================================================ +// ============================================================ +// internal helpers +// ============================================================ normlog:{[logdict] - / detect kx.log instance by presence of kx.log-specific keys (getlvl, sinks, fmts) - / kx.log functions are monadic - wrap each into binary {[c;m]} and embed context in the message - / plain {[c;m]} log dicts (info`warn`error only) pass through unchanged - $[any `getlvl`sinks`fmts in key logdict; + // detect kx.log instance by presence of kx.log-specific keys (getlvl, sinks, fmts) + // kx.log functions are monadic - wrap each into binary {[c;m]} and embed context in the message + // plain {[c;m]} log dicts (info`warn`error only) pass through unchanged + $[all `getlvl`sinks`fmts in key logdict; `info`warn`error!( {[fn;c;m] fn[string[c],": ",m]}[logdict`info;]; {[fn;c;m] fn[string[c],": ",m]}[logdict`warn;]; @@ -79,122 +82,156 @@ normlog:{[logdict] }; sendclientreply:{[qid;result;status] - / deliver result or error to the client, handling sync vs async send and postback wrapping + // deliver result or error to the client, handling sync vs async send and postback wrapping + // local path invokes value tosend directly - formatresponse is not applied (it is a pass-through + // for async by default; consumers that override setformatresponse should not use local invocation) qd:queryqueue[qid]; if[qd`error;:()]; tosend:$[()~qd`postback;result;qd[`postback],enlist[qd`query],enlist result]; $[qd`sync; @[-30!;(qd`clienth;not status;$[status;formatresponse[1b;1b;result];result]);{}]; - @[neg qd`clienth;formatresponse[status;0b;tosend];()]]}; + $[qd`local; + @[value;tosend;{.z.m.log[`error][`asyncdispatch;"local postback failed: ",x]}]; + @[neg qd`clienth;formatresponse[status;0b;tosend];()]]]; + }; finishquery:{[qid;err] - / remove query from the live results accumulator and stamp its completion time + // remove query from the live results accumulator and stamp its completion time .z.m.results:(qid,())_results; - update error:err,returntime:.z.m.cp[] from .z.M.queryqueue where queryid in qid}; + update error:err,returntime:.z.m.cp[] from .z.M.queryqueue where queryid in qid; + }; serverexecute:{[qid;query] - / runs on the backend - traps errors so a crash posts an error reply rather than dropping the result + // runs on the backend - traps errors so a crash posts an error reply rather than dropping the result res:@[{(0b;value x)};query;{(1b;"server ",(string .z.h),":",(string system"p"),": ",x)}]; @[neg .z.w;$[res 0;(errorcallback;qid;res 1);(resultcallback;qid;res 1)]; - {@[neg .z.w;(errorcallback;x;"failed to return result: ",y);()]}[qid]]}; + {@[neg .z.w;(errorcallback;x;"failed to return result: ",y);()]}[qid]]; + }; sendquerytoserver:{[qid;query;handles] - / fan the query out to all required handles and mark them in-use atomically + // fan the query out to all required handles and mark them in-use atomically (neg handles,:())@\:(serverexecute;qid;query); - update inuse:1b from .z.M.servers where handle in handles}; + update inuse:1b from .z.M.servers where handle in handles; + }; runnextquery:{[] - / pick the next dispatchable query and fan out to one idle server per required servertype - / called after any state change that may unblock work + // pick the next dispatchable query and fan out to one idle server per required servertype + // called after any state change that may unblock work if[0=count torun:getnextqueryid[];:()]; torun:first torun; avail:exec first handle by servertype from availableservers 1b; types:torun`servertype; handles:avail types; qid:torun`queryid; - slots:types!(count[types],())#enlist(0Ni;(::);0b); + slots:types!count[types]#enlist(0Ni;(::);0b); slots[types;0]:handles; + // indexed assignment on bare results propagates through to .z.m in kdb-x (empirically verified) results[qid]:(torun`clienth;slots); - sendquerytoserver[qid;torun`query;handles]}; + sendquerytoserver[qid;torun`query;handles]; + }; + +addqueryto:{[query;servertype;join;postback;timeout;sync;replyto;local] + // enqueue a query with an explicit reply target - used by execqueryto for local in-process routing + .z.M.queryqueue upsert (queryid;.z.m.cp[];replyto;query;servertype;join;{$[11h=type x;enlist x;x]}postback;timeout;0Np;0b;sync;local); + .z.m.queryid:queryid+1; + }; addquery:{[query;servertype;join;postback;timeout;sync] - / enqueue a query without dispatching - caller must call runnextquery[] to trigger dispatch - .z.M.queryqueue upsert (queryid;.z.m.cp[];.z.w;query;servertype;join;{$[11h=type x;enlist x;x]}postback;timeout;0Np;0b;sync); - .z.m.queryid:queryid+1}; + // enqueue a query without dispatching - caller must call runnextquery[] to trigger dispatch + addqueryto[query;servertype;join;postback;timeout;sync;.z.w;0b]; + }; removequeries:{[age] - / prevent queryqueue growing unboundedly - purge completed queries older than age - delete from .z.M.queryqueue where not null returntime, .z.m.cp[]>returntime+age}; + // prevent queryqueue growing unboundedly - purge completed queries older than age + delete from .z.M.queryqueue where not null returntime, .z.m.cp[]>returntime+age; + }; removeinactive:{[age] - / prune stale disconnected-server rows to stop the servers table growing forever - delete from .z.M.servers where not active, .z.m.cp[]>disconnecttime+age}; + // prune stale disconnected-server rows to stop the servers table growing forever + delete from .z.M.servers where not active, .z.m.cp[]>disconnecttime+age; + }; + +removeclients:{[age] + // prune stale client audit rows to stop the clients table growing forever + // clients are recorded on every connect by addclientdetails and never removed otherwise + delete from .z.M.clients where .z.m.cp[]>time+age; + }; checktimeout:{[] - / periodic scan to error queries that have waited beyond their timeout + // periodic scan to error queries that have waited beyond their timeout qids:exec queryid from .z.m.queryqueue where not timeout=0Wn, null returntime, .z.m.cp[]>time+timeout; if[count qids; .z.m.log[`warn][`asyncdispatch;"queries timed out: ",", " sv string qids]; sendclientreply[;errorprefix,"query timed out";0b] each qids; - finishquery[qids;1b]]}; + finishquery[qids;1b]]; + }; -/ ============================================================ -/ public api -/ ============================================================ +// ============================================================ +// public api +// ============================================================ setcp:{[f] - / replace the clock function - used in tests to control time without sleeping - .z.m.cp:f}; + // replace the clock function - used in tests to control time without sleeping + .z.m.cp:f; + }; setformatresponse:{[f] - / override reply formatting - e.g. to wrap results in a standard envelope - .z.m.formatresponse:f}; + // override reply formatting - e.g. to wrap results in a standard envelope + .z.m.formatresponse:f; + }; setcallbacks:{[resfn;errfn] - / update callback symbols when module is mounted under a non-default namespace + // update callback symbols when module is mounted under a non-default namespace .z.m.resultcallback:resfn; - .z.m.errorcallback:errfn}; + .z.m.errorcallback:errfn; + }; setavailableservers:{[f] - / swap in a custom routing strategy without forking core dispatch - .z.m.availableservers:f}; + // swap in a custom routing strategy without forking core dispatch + .z.m.availableservers:f; + }; setgetnextqueryid:{[f] - / inject a priority or custom scheduling strategy - .z.m.getnextqueryid:f}; + // inject a priority or custom scheduling strategy + .z.m.getnextqueryid:f; + }; addserver:{[h;st] - / register a backend handle and servertype so it becomes eligible for dispatch + // register a backend handle and servertype so it becomes eligible for dispatch .z.m.log[`info][`asyncdispatch;"server registered: ",string[st]," handle ",string h]; - .z.M.servers upsert (h;st;0b;1b;0Np)}; + .z.M.servers upsert (h;st;0b;1b;0Np); + }; removeserverhandle:{[serverh] - / on backend disconnect, error in-flight queries using that handle and queued queries - / that can no longer be satisfied + // on backend disconnect, error in-flight queries using that handle and queued queries + // that can no longer be satisfied if[null st:first exec servertype from .z.m.servers where handle=serverh;:()]; err:errorprefix,"backend ",string[st]," server disconnected"; .z.m.log[`warn][`asyncdispatch;"backend disconnected: ",string st]; - / in-flight: queries where this handle was assigned to a slot + // in-flight: queries where this handle was assigned to a slot qids:where {[h;qid]h in value[.z.m.results[qid;1]][;0]}[serverh] each key .z.m.results; sendclientreply[;err," during query";0b] each qids; finishquery[qids;1b]; - / queued: queries that can no longer be satisfied by remaining active servers + // queued: queries that can no longer be satisfied by remaining active servers activetypes:exec distinct servertype from .z.m.servers where active, handle<>serverh; qids2:exec queryid from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, not {all x in y}[;activetypes] each servertype; sendclientreply[;err,", query cannot be satisfied";0b] each qids2; finishquery[qids2;1b]; update active:0b,disconnecttime:.z.m.cp[] from .z.M.servers where handle=serverh; - runnextquery[]}; + runnextquery[]; + }; addclientdetails:{[h] - / record client identity on connect for audit and orphan-query cleanup on disconnect + // record client identity on connect for audit and orphan-query cleanup on disconnect .z.m.log[`info][`asyncdispatch;"client connected: handle ",string h]; - .z.M.clients insert (.z.m.cp[];h;.z.u;.z.a;.z.h)}; + .z.M.clients insert (.z.m.cp[];h;.z.u;.z.a;.z.h); + }; removeclienthandle:{[h] - / on client disconnect, mark their pending queries errored so result slots are not leaked - / free any servers in-flight for this client before removing result slots, then re-dispatch + // on client disconnect, mark their pending queries errored so result slots are not leaked + // free any servers in-flight for this client before removing result slots, then re-dispatch + // local queries store clienth:0Ni and are not matched here - the in-process caller owns cleanup for its own requests .z.m.log[`info][`asyncdispatch;"client disconnected: handle ",string h]; inflightqids:(exec queryid from .z.m.queryqueue where clienth=h, null returntime) inter key .z.m.results; if[count inflightqids; @@ -202,10 +239,12 @@ removeclienthandle:{[h] update inuse:0b from .z.M.servers where handle in inflighthandles]; update error:1b,returntime:.z.m.cp[] from .z.M.queryqueue where clienth=h, null returntime; .z.m.results:(exec queryid from .z.m.queryqueue where clienth=h)_results; - runnextquery[]}; + runnextquery[]; + }; addserverresult:{[qid;data] - / fill one result slot - once all slots for a query are filled, run the join and reply + // fill one result slot - once all slots for a query are filled, run the join and reply + // bare indexed assignment on results propagates through to .z.m in kdb-x (empirically verified) if[not qid in key results;:()]; st:first exec servertype from .z.m.servers where handle=.z.w; slots:results[qid;1]; @@ -217,43 +256,60 @@ addserverresult:{[qid;data] vals:value results[qid;1]; if[not all vals[;2];:()]; qd:queryqueue[qid]; - res:@[{(0b;x y)}[qd`join];vals[;1];{(1b;errorprefix,"join failed: ",x)}]; + res:@[{(0b;x y)}[qd`join];vals[;1];{(1b;.z.m.errorprefix,"join failed: ",x)}]; if[res 0;.z.m.log[`error][`asyncdispatch;"join failed for query ",string qid,": ",last res]]; sendclientreply[qid;last res;not res 0]; - finishquery[qid;res 0]}; + finishquery[qid;res 0]; + }; addservererror:{[qid;err] - / short-circuit a query on backend failure - free the server and notify the client + // short-circuit a query on backend failure - free the server and notify the client .z.m.log[`error][`asyncdispatch;"backend error for query ",string[qid],": ",err]; sendclientreply[qid;errorprefix,err;0b]; update inuse:0b from .z.M.servers where handle in .z.w; runnextquery[]; - finishquery[qid;1b]}; + finishquery[qid;1b]; + }; execquery:{[query;servertype;join;postback;timeout;sync] - / public entry point - validate sync constraints then enqueue and kick dispatch + // public entry point - validate sync constraints then enqueue and kick dispatch if[sync; + if[not ()~postback;.z.m.log[`warn][`asyncdispatch;"execquery: postback ignored for sync call"]]; if[not synccallsallowed;'"syncexec: synchronous calls are not allowed"]; if[not @[{-30!x;1b};(::);0b];'"syncexec: deferred response not supported on this connection"]; .[{[q;s;j;t]addquery[q;s;j;();t;1b];runnextquery[]};(query;servertype;join;timeout);{-30!(.z.w;1b;x)}]; :()]; addquery[query;servertype;join;postback;timeout;0b]; - runnextquery[]}; + runnextquery[]; + }; + +execqueryto:{[replyto;query;servertype;join;postback;timeout;sync] + // in-process variant of execquery - replyto is 0Ni for local invocation via value + // postback must be non-empty when replyto is 0Ni - there is no handle to send a bare result to + // sync is not supported for local invocation + // mount-qualified postback symbols required for local invocation e.g. `da.shardresult not `shardresult + if[0Ni~replyto; + if[()~postback;'"di.asyncdispatch: local invocation requires a non-empty postback"]; + if[sync;'"di.asyncdispatch: local invocation does not support sync mode"]]; + local:0Ni~replyto; + addqueryto[query;servertype;join;postback;timeout;sync;replyto;local]; + runnextquery[]; + }; init:{[deps] - / initialise the asyncdispatch module - validate deps and apply config overrides - / deps: dict containing `log (required) plus optional config keys: - / `log - required: `info`warn`error!({[c;m]};{[c;m]};{[c;m]}) binary loggers - / `errorprefix - optional: string prefix for client error messages. default: "error: " - / `querykeeptime - optional: timespan to keep completed queries. default: 0D00:30 - / `clearinactivetime - optional: timespan to keep disconnected servers. default: 0D01:00 - / `synccallsallowed - optional: boolean, whether sync calls are permitted. default: 0b - / housekeeping (checktimeout; removequeries; removeinactive) is the caller's responsibility - - / wire them into your timer after init; the exported defaults for age params are querykeeptime - / and clearinactivetime - / examples: - / ad.init[enlist[`log]!enlist logdep] - / ad.init[`log`querykeeptime!(logdep;0D01:00)] + // initialise the asyncdispatch module - validate deps and apply config overrides + // deps: dict containing `log (required) plus optional config keys: + // `log - required: `info`warn`error!({[c;m]};{[c;m]};{[c;m]}) binary loggers + // `errorprefix - optional: string prefix for client error messages. default: "error: " + // `querykeeptime - optional: timespan to keep completed queries. default: 0D00:30 + // `clearinactivetime - optional: timespan to keep disconnected servers. default: 0D01:00 + // `synccallsallowed - optional: boolean, whether sync calls are permitted. default: 0b + // housekeeping (checktimeout; removequeries; removeinactive; removeclients) is the caller's + // responsibility - wire them into your timer after init; the exported defaults for age params are + // querykeeptime and clearinactivetime + // examples: + // ad.init[enlist[`log]!enlist logdep] + // ad.init[`log`querykeeptime!(logdep;0D01:00)] if[99h<>type deps; '"di.asyncdispatch: deps must be a dict with `log key"]; if[not `log in key deps; diff --git a/di/asyncdispatch/init.q b/di/asyncdispatch/init.q index 372cf91f..21200d8a 100644 --- a/di/asyncdispatch/init.q +++ b/di/asyncdispatch/init.q @@ -1,4 +1,4 @@ -/ asyncdispatch module - async scatter-gather query coordinator for multi-backend kdb+ gateway processes +// asyncdispatch module - async scatter-gather query coordinator for multi-backend kdb+ gateway processes \l ::asyncdispatch.q export:([ @@ -6,6 +6,6 @@ export:([ addserver;removeserverhandle; addclientdetails;removeclienthandle; addserverresult;addservererror; - execquery; - checktimeout;removequeries;removeinactive; + execquery;execqueryto; + checktimeout;removequeries;removeinactive;removeclients; init]) diff --git a/di/asyncdispatch/test.csv b/di/asyncdispatch/test.csv index 4dd1cba3..8b202caf 100644 --- a/di/asyncdispatch/test.csv +++ b/di/asyncdispatch/test.csv @@ -124,6 +124,53 @@ true,0,0,q,0b~first exec error from qq[] where queryid=qid7,1,,scatter-gather co true,0,0,q,4~captured,1,,joined result is sum of both backend results (1+1=2 from each; sum 2 2 = 4) run,0,0,q,ad.setformatresponse[{[status;sync;result]$[not[status]and sync;'result;result]}],1,,restore default formatresponse +/ Test 13: execqueryto local path - successful result delivered in-process +run,0,0,q,ad.setcallbacks[`.m.di.0asyncdispatch.addserverresult;`.m.di.0asyncdispatch.addservererror],1,,restore callbacks +run,0,0,q,ad.setgetnextqueryid[{r:0!select from .m.di.0asyncdispatch.queryqueue where null returntime;1 sublist select from r where not queryid in key .m.di.0asyncdispatch.results}],1,,restore scheduler +run,0,0,q,.test.localcap:(::),1,,reset local capture +run,0,0,q,.test.localfn:{[reqid;qry;res] .test.localcap:(reqid;qry;res)},1,,define local capturing callback in root namespace +run,0,0,q,ad.execqueryto[0Ni;"2+2";enlist`mock;{sum x};(.test.localfn;`req1);0Wn;0b],1,,dispatch via local path with postback (sum collapses single-element result to scalar) +run,0,0,q,bh"",1,,sync flush - wait for backend result callback +true,0,0,q,(`req1;"2+2";4)~.test.localcap,1,,postback invoked in-process with (reqid;query;result) + +/ Test 14: execqueryto local path - backend error delivered in-process via postback +run,0,0,q,.test.localcap:(::),1,,reset local capture +run,0,0,q,ad.execqueryto[0Ni;"'errortest";enlist`mock;raze;(.test.localfn;`req2);0Wn;0b],1,,dispatch erroring query via local path +run,0,0,q,bh"",1,,sync flush - wait for backend error callback +true,0,0,q,`req2~.test.localcap[0],1,,reqid correct +true,0,0,q,.test.localcap[2] like "error:*",1,,error string delivered as result via postback + +/ Test 15: execqueryto local path - timeout delivered in-process via postback +run,0,0,q,.test.localcap:(::),1,,reset local capture +run,0,0,q,ad.execqueryto[0Ni;"3+3";enlist`mock;raze;(.test.localfn;`req3);0D00:00:00.000000001;0b],1,,dispatch with near-zero timeout +run,0,0,q,ad.checktimeout[],1,,trigger timeout sweep +true,0,0,q,`req3~.test.localcap[0],1,,reqid correct +true,0,0,q,.test.localcap[2] like "error:*",1,,timeout error delivered in-process via postback + +/ Test 16: removeclients purges stale client audit rows +run,0,0,q,ad.addclientdetails[7i],1,,record a client for the cleanup test +true,0,0,q,1~count select from cl[] where clienth=7i,1,,client audit row exists +run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p+2D}],1,,fast-forward clock two days past the audit age +run,0,0,q,ad.removeclients[0D00:01],1,,purge client rows older than one minute +true,0,0,q,0~count select from cl[] where clienth=7i,1,,client audit row purged +run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p}],1,,restore the real clock + +/ Test 17: join-failure path - join function throws, client receives error and query is finished +/ dispatched to the scatter backend (bh2): the mock backend (bh) is left inuse by the in-flight timeout in test 15 +run,0,0,q,.m.di.0asyncdispatch.addquery["2+2";enlist`scatter;{'"joinboom"};();0Wn;0b],1,,queue a query whose join function throws +run,0,0,q,qid8:exec first queryid from qq[] where join~\:{'"joinboom"},1,,capture the query id +run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch to the scatter backend +run,0,0,q,bh2"",1,,sync flush - wait for the result callback and join attempt +true,0,0,q,1b~first exec error from qq[] where queryid=qid8,1,,join failure correctly flagged as error +true,0,0,q,not null first exec returntime from qq[] where queryid=qid8,1,,returntime stamped after join failure + +/ Test 18: execqueryto with a non-null explicit replyto stores it as clienth with local=0b +run,0,0,q,ad.execqueryto[99i;"1+1";enlist`scatter;{sum x};();0Wn;0b],1,,dispatch with explicit replyto handle 99i +run,0,0,q,qid9:exec first queryid from qq[] where (clienth=99i)&query~\:"1+1",1,,capture the query id +true,0,0,q,99i~first exec clienth from qq[] where queryid=qid9,1,,replyto stored as clienth +true,0,0,q,0b~first exec local from qq[] where queryid=qid9,1,,local flag is 0b for a non-null replyto +run,0,0,q,bh2"",1,,sync flush - completes the backend round-trip (reply to fake 99i fails silently) + / Teardown run,0,0,q,neg[bh](exit;0);neg[bh](::),1,,send exit to first backend process and flush run,0,0,q,hclose bh,1,,close first backend handle From 430166728a92e05d352ca2a1b8a87e3abbe93246 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Mon, 6 Jul 2026 15:10:35 +0100 Subject: [PATCH 05/12] added in pluggable server source without requiring a hard dependacy --- di/asyncdispatch/asyncdispatch.md | 17 +++++++++++++---- di/asyncdispatch/asyncdispatch.q | 8 ++++++-- di/asyncdispatch/test.csv | 12 ++++++++++++ 3 files changed, 31 insertions(+), 6 deletions(-) diff --git a/di/asyncdispatch/asyncdispatch.md b/di/asyncdispatch/asyncdispatch.md index a4f1afc9..c11d5c80 100644 --- a/di/asyncdispatch/asyncdispatch.md +++ b/di/asyncdispatch/asyncdispatch.md @@ -2,7 +2,7 @@ Async scatter-gather query coordinator for kdb-x gateway processes. Queues client queries, dispatches them to available backend processes by servertype, collects per-server results, applies a join function, and replies to the client — with timeout management and correct error propagation if a backend disconnects mid-query. -Routing (deciding which servertypes satisfy a query) is `di.serverselect`'s responsibility. This module receives a resolved servertype list and dispatches to whatever idle backends of each type are registered. +Routing (deciding which servertypes satisfy a query) is `di.serverselect`'s responsibility, but **`di.serverselect` is not a dependency** — this module dispatches a resolved servertype list to whatever backends its *server source* reports. That source is pluggable with a default: by default it is asyncdispatch's own registry (populate it with `addserver`), or you point it at `di.serverselect`'s output via `setavailableservers`. Either works standalone; neither is required. --- @@ -14,6 +14,7 @@ Routing (deciding which servertypes satisfy a query) is `di.serverselect`'s resp - Handle backend disconnects mid-query — errors in-flight queries and queued queries that can no longer be satisfied - Track connected clients and clean up orphaned queries on client disconnect - Support synchronous deferred response mode (`-30!`) alongside the default async mode +- Pluggable server source with a default — dispatch against the built-in registry (`addserver`) by default, or point it at `di.serverselect`'s output (or any source) via `setavailableservers`; `di.serverselect` is a composable option, never a dependency - Accept fully pluggable scheduler (`setgetnextqueryid`), routing (`setavailableservers`), reply formatter (`setformatresponse`), and callback symbols (`setcallbacks`) — swap without touching core dispatch logic - Detect and normalise `kx.log` instances automatically so callers can pass a logger directly without manual wrapping @@ -27,6 +28,8 @@ Routing (deciding which servertypes satisfy a query) is `di.serverselect`'s resp The `log` dependency must be passed to `init` inside the `deps` dict. The module throws immediately if it is absent or missing any of the three required keys. All three are required since the module calls `info`, `warn`, and `error`. +> **`di.serverselect` is not a dependency.** asyncdispatch never imports or calls it. The server source is pluggable (`setavailableservers`) and defaults to the built-in registry (`addserver`), so you run standalone out of the box; to dispatch against serverselect's live view instead, inject its `getservers` output as the source (see `setavailableservers`). No registry copying, no dependency. + A `kx.log` instance can be passed directly — the module normalises monadic functions to the binary `{[c;m]}` contract automatically via `normlog`. Context is embedded in the output as `"context: message"`: ```q @@ -67,7 +70,7 @@ ad.init[enlist[`log]!enlist logdep] ``` ### `addserver[handle;servertype]` -Register a backend connection. `handle`: open int handle. `servertype`: symbol identifying the process type (e.g. `` `rdb ``, `` `hdb ``). +Register a backend connection into the **built-in (default) server source**. `handle`: open int handle. `servertype`: symbol identifying the process type (e.g. `` `rdb ``, `` `hdb ``). Use this when asyncdispatch owns the server list; to source servers from `di.serverselect` instead, leave `addserver` unused and inject via `setavailableservers`. ```q ad.addserver[hopen`:backend1:5001;`rdb] ``` @@ -177,10 +180,16 @@ ad.setcallbacks[`.gw.dispatch.addserverresult;`.gw.dispatch.addservererror] ``` ### `setavailableservers[f]` -Swap in a custom routing strategy. `f` must be `{[excludeinuse]}` returning a table with a `servertype` column. +Replace the **server source** — the function dispatch uses to find backends. `f` is `{[excludeinuse]}` and must return a table with `handle` and `servertype` columns. This is the seam for running against `di.serverselect` (or any external source) **without a dependency and without copying its registry**: by default the source reads asyncdispatch's own registry (`addserver`), and you override it to read serverselect instead. ```q -ad.setavailableservers[{[excl]select from servers where active}] +// default (built-in registry) — equivalent to not calling this at all +ad.setavailableservers[{[eu] $[eu; select from servers where active, not inuse; select from servers where active]}] + +// compose with di.serverselect (not a dependency — just its output as the source) +srvsel:use`di.serverselect +ad.setavailableservers[{[eu] select handle, servertype from srvsel.getservers[`servertype;`;()!()]}] ``` +Result routing does **not** depend on the source — a returning handle is matched to its servertype from the query's own dispatch record — so an injected source needs no registration in asyncdispatch. Note `inuse` throttling applies only to the built-in registry; an external source is expected to do its own idle/selection (e.g. serverselect's `roundrobin`), and the `excludeinuse` flag is a hint such a source may ignore. ### `setgetnextqueryid[f]` Inject a custom scheduling strategy. `f` must be niladic and return a 0- or 1-row table with the `queryqueue` schema. diff --git a/di/asyncdispatch/asyncdispatch.q b/di/asyncdispatch/asyncdispatch.q index 85b7f5da..c2f812b5 100644 --- a/di/asyncdispatch/asyncdispatch.q +++ b/di/asyncdispatch/asyncdispatch.q @@ -197,7 +197,9 @@ setgetnextqueryid:{[f] }; addserver:{[h;st] - // register a backend handle and servertype so it becomes eligible for dispatch + // register a backend handle and servertype so it becomes eligible for dispatch. + // this is the default (built-in) server source; to dispatch against di.serverselect's view instead, + // inject it via setavailableservers - no registration and no di.serverselect dependency required .z.m.log[`info][`asyncdispatch;"server registered: ",string[st]," handle ",string h]; .z.M.servers upsert (h;st;0b;1b;0Np); }; @@ -246,8 +248,10 @@ addserverresult:{[qid;data] // fill one result slot - once all slots for a query are filled, run the join and reply // bare indexed assignment on results propagates through to .z.m in kdb-x (empirically verified) if[not qid in key results;:()]; - st:first exec servertype from .z.m.servers where handle=.z.w; slots:results[qid;1]; + // map the responding handle to its servertype from THIS query's own dispatch record, not a server + // registry - so any server source (built-in or an injected di.serverselect view) works unchanged + st:first where .z.w=slots[;0]; slots[st]:(.z.w;data;1b); results[qid]:(results[qid;0];slots); update inuse:0b from .z.M.servers where handle in .z.w; diff --git a/di/asyncdispatch/test.csv b/di/asyncdispatch/test.csv index 8b202caf..41ba99a8 100644 --- a/di/asyncdispatch/test.csv +++ b/di/asyncdispatch/test.csv @@ -171,6 +171,18 @@ true,0,0,q,99i~first exec clienth from qq[] where queryid=qid9,1,,replyto stored true,0,0,q,0b~first exec local from qq[] where queryid=qid9,1,,local flag is 0b for a non-null replyto run,0,0,q,bh2"",1,,sync flush - completes the backend round-trip (reply to fake 99i fails silently) +/ Test 19: pluggable server source - dispatch against an injected source whose servertype is not in the +/ internal registry; result routing uses the query's dispatch record so no internal registration is needed +run,0,0,q,savedavail:.m.di.0asyncdispatch.availableservers,1,,save the default (internal-registry) server source +run,0,0,q,ad.setavailableservers[{[eu]([]handle:enlist bh;servertype:enlist`ext)}],1,,inject a source presenting bh under servertype ext (bh is registered as mock) +run,0,0,q,.m.di.0asyncdispatch.addquery["6*7";enlist`ext;raze;();0Wn;0b],1,,queue a query for the external-only servertype +run,0,0,q,qidx:exec first queryid from qq[] where query~\:"6*7",1,,capture the query id +run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch to bh via the injected source +run,0,0,q,bh"",1,,sync flush - wait for the backend result callback +true,0,0,q,0b~first exec error from qq[] where queryid=qidx,1,,query completed without error via the injected source +true,0,0,q,not null first exec returntime from qq[] where queryid=qidx,1,,result routed via the dispatch record (servertype ext) not the servers table (mock) +run,0,0,q,ad.setavailableservers[savedavail],1,,restore the default internal-registry source + / Teardown run,0,0,q,neg[bh](exit;0);neg[bh](::),1,,send exit to first backend process and flush run,0,0,q,hclose bh,1,,close first backend handle From 68b0ba68fd9dac63c4b1262348d7315d307fbb5e Mon Sep 17 00:00:00 2001 From: ascottDI Date: Thu, 9 Jul 2026 11:44:54 +0100 Subject: [PATCH 06/12] closing in on a completed v1 of this module --- di/config/config.md | 145 +++++++++++++++++++++++++++++ di/config/config.q | 219 ++++++++++++++++++++++++++++++++++++++++++++ di/config/init.q | 5 + di/config/test.csv | 115 +++++++++++++++++++++++ 4 files changed, 484 insertions(+) create mode 100644 di/config/config.md create mode 100644 di/config/config.q create mode 100644 di/config/init.q create mode 100644 di/config/test.csv diff --git a/di/config/config.md b/di/config/config.md new file mode 100644 index 00000000..d7419d37 --- /dev/null +++ b/di/config/config.md @@ -0,0 +1,145 @@ +# di.config + +Configuration loading and cascade resolution for the modular TorQ world. This +replaces TorQ's in-process config handling (`torq.q`: `loadf`, `loaddir`, +`loadconfig`, `loadaddconfig`, `overrideconfig`), where `di.torq` will later +distribute the resolved config to each module via that module's `init`. + +## Import and init + +`init` requires a `log` dependency — there is no silent fallback. Pass an +already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` loggers (context +symbol, message string). The module performs **no** adaptation, so a raw monadic +[`kx.log`](https://github.com/KxSystems/logging) instance must be wrapped by the +caller first (this is `di.log`'s job once it ships): + +```q +config:use`di.config + +/ option 1: di.log (standard, once it ships) - build the binary dict from its exports +logger:use`di.log +logdep:`info`warn`error!(logger.info;logger.warn;logger.error) +config.init[enlist[`log]!enlist logdep] + +/ option 2: hand-rolled binary {[c;m]} logger (works today) +mylog:`info`warn`error!( + {[c;m] -1 string[c],": INFO ",m;}; + {[c;m] -1 string[c],": WARN ",m;}; + {[c;m] -2 string[c],": ERROR ",m;}); +config.init[enlist[`log]!enlist mylog] +``` + +To wrap a raw `kx.log` instance yourself in the interim, fold the context into the +message per call: `` `info`warn`error!({[c;m]inst[`info][string[c],": ",m]};…) ``. + +`init` must be called before any other function — there is no default logger. + +## Exported functions + +| Function | Signature | Description | +|---|---|---| +| `init` | `init[deps]` | Wire the injected logger. `deps` is a dict with a required `log` key — an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` functions. Errors immediately if the log dependency is missing or malformed. | +| `loadfile` | `loadfile[file]` | Load a single q config file (a string path) if it exists, tracking it so it is not re-loaded. Returns the path. A missing file is a logged warning, not an error; a failed load is signalled. | +| `loadconfig` | `loadconfig[dir;name]` | Load one cascade config file `dir/{name}.q` (`dir` a string path, `name` a symbol) if present. Returns the constructed path. A missing file is **normal** in the cascade, so its absence is logged at *info* and skipped (not warned). Present files load via `loadfile`. | +| `loaddir` | `loaddir[dir]` | Load every `.q`/`.k` file in a directory (a string path), honouring an optional `order.txt`. Files listed in `order.txt` load first, in that order; the rest follow. Returns the ordered list of file paths processed. A missing directory is a logged warning returning `()`. Delegates each load to `loadfile`, so already-loaded files are skipped. | +| `loadcascade` | `loadcascade[dirs;names]` | Resolve a config cascade: for each settings directory (in order) load each named file `dir/{name}.q` (in name order). `dirs` is a string path or list of paths; `names` is a symbol or symbol list. Later names and later directories override earlier ones (load order). Missing files are skipped. Returns the flat list of constructed paths in cascade order. | +| `overrideconfig` | `overrideconfig[params]` | Apply command-line-style overrides after the cascade. `params` is a dict keyed by variable name (symbol) with string (or list-of-string) values, each parsed into that variable's **existing** type. Only already-defined, basic-typed variables can be overridden; undefined names, non-basic types, and unparseable values are logged and skipped. Returns the list of variables actually overridden. | +| `get` | `get[namespace;key;default]` | Query the resolved config store: return the value of `.{namespace}.{key}`, or `default` if unset. `namespace` and `key` are **bare** symbols (no leading dot), e.g. `get[\`rdb;\`subscribeto;\`]`. (`get` is a reserved word, so the implementation function is named `getcfg` and exported under the `get` key.) | +| `getmodule` | `getmodule[namespace]` | Return a namespace's whole resolved config as a bare-keyed value dict — the slice di.torq passes to a module's `init`. `namespace` is a bare symbol; an unconfigured namespace yields an empty dict. | + +`order.txt` is a plain-text file, one filename per line, naming the files that must +load ahead of the rest (e.g. to satisfy load-order dependencies). + +Internal helpers (`raiseerror`, `applyoverride`) and the load-tracking state are deliberately +not exported. + +`loadcascade` is the top-level entry point. A caller (e.g. `di.torq`, which resolves the +process identity and directory paths) drives it as: + +```q +config.loadcascade[(kdbconfig;appconfig);`default,proctype,procname] +config.overrideconfig[`.myproc.enabled`.myproc.rows!(enlist"1";enlist"5000")] +/ then partition per module and hand each slice to that module's init: +rdbcfg:config.getmodule[`rdb] / -> `subscribeto`hdbtypes!(...) etc. +rdb.init[rdbcfg;`log`timer!(logdep;timerdep)] +/ or read a single value with a fallback: +config.get[`rdb;`subscribeto;`trade`quote] +``` + +`overrideconfig` runs after `loadcascade` so command-line values win over file config. +di.config is deliberately generic — it does not read environment variables or process +identity itself; the caller (di.torq) supplies the settings directories, the ordered +name sequence, and the override params, then uses `getmodule` to partition the resolved +store per module. + +### The config store + +`loadcascade` loads settings `.q` files that assign into root namespaces +(`.rdb.subscribeto:...`); those resolved namespaces **are** the config store, and +`get`/`getmodule` query them (so anything `overrideconfig` changes is reflected too). +Additional config sources (env vars, k8s config maps) are out of scope for v1 but would +plug in by populating the same namespaces before the store is queried — the `get`/ +`getmodule` contract stays unchanged. See the EXTENSION POINT comment in `config.q`. + +> **Note:** avoid `-` in config file paths — a source-level backtick symbol literal +> containing `-` parses as subtraction (`` `:/a-b.q `` → `` `:/a `` `- b.q`), not one +> symbol. Runtime string paths passed to `loadfile` are unaffected. + +## Injectable dependencies + +| Injectable | Required keys | Function signature | +|---|---|---| +| `log` | `` `info`warn`error `` | `{[c;m]}` (context symbol, message string); caller passes an already-conforming binary dict (from `di.log` once it ships, or hand-rolled). No adaptation is done in the module — a raw monadic `kx.log` instance must be wrapped first | + +## Hard dependencies + +None — `di.config` is a standalone module. + +## Design notes & open gaps + +- **Return values are informational.** `loadfile`/`loadconfig`/`loaddir`/`loadcascade` return the + path(s) they *considered* — including files that were missing or skipped. The real effect is the + side-effect load; use the return to see what was attempted, not what definitely loaded. +- **Missing-file log level is deliberate.** `loadfile` logs a missing explicit path at *warn* (you + asked for it); `loadconfig` logs a missing cascade file at *info* (absence is normal — most + `{proctype}`/`{procname}` files won't exist); `loaddir` logs a missing directory at *warn* and + returns `()`. +- **Dedup / idempotency.** `loadfile` tracks loaded paths in `.z.m.loaded` and skips re-loads, so the + cascade is safe to re-run and `init` is safe to call again. +- **`overrideconfig` deviates from TorQ deliberately.** Per-variable problems (undefined name, + non-basic type, value that won't parse to the target type) are logged and skipped — a single bad + override never aborts the batch and a null is never written into config. +- **Entry point is `loadcascade[dirs;names]`, NOT a zero-arg `load[]`.** The Plan sketches + `di.config.load[]`, but that predates the integration design. Config loading happens once at + startup, so a blank call buys nothing and would force the cascade inputs into hidden `.z.m` state + (temporal coupling with `init`). Explicit args keep di.config a pure, env-free resolver: di.torq + resolves `KDBCONFIG`/`KDBAPPCONFIG` → dirs and `proctype`/`procname` → names and passes them. (Also + `load` is a reserved word, so the Plan's literal name is unusable anyway.) If di.torq ever wants a + blank trigger it's a one-line wrapper on its side — not di.config's concern. +- **Config store is live namespace globals, not an explicit internal store (1a, deliberate).** Settings + files assign globals when executed, so those namespaces *are* the store; `get`/`getmodule` read them. + An explicit store with provenance / cross-source precedence (1b) is deferred until a second config + source (env, k8s) exists — the `get`/`getmodule` contract is storage-independent, so that swap won't + touch consumers. See the `EXTENSION POINT` comment in `config.q`. + +Open gaps / not implemented: + +- **No runtime reload.** TorQ's `reloadf` (force-reload an already-loaded file) is not ported — + `loadfile` always dedups. Adequate for startup; revisit if runtime config reload is needed. +- **`order.txt` is matched against the full directory listing** (as in TorQ), so a non-`.q`/`.k` file + named in `order.txt` would still be prepended and load-attempted. Keep `order.txt` to q/k files. +- **`overrideconfig`'s null-parse guard targets scalar/vector basic types**; a string-typed (`10h`) + config var is an uncommon edge where the per-element null check is loose. +- **Standard logger (`di.log`) does not exist yet.** Callers must supply a hand-rolled binary + `` `info`warn`error `` dict in the interim; the kx.log wiring test wraps kx.log caller-side to prove + real emission. +- **Logger storage follows the skill's single-dict pattern** (`.z.m.log[\`info][…]`), which diverges + from `consistency.md`'s three-flat-var example (`.z.m.loginfo`/`…`). Per the skill this is the + directed pattern, pending `consistency.md` being reconciled once `di.log` ships. + +## Tests + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.config +``` diff --git a/di/config/config.q b/di/config/config.q new file mode 100644 index 00000000..8167cd4f --- /dev/null +++ b/di/config/config.q @@ -0,0 +1,219 @@ +/ configuration loading and cascade resolution for the modular torq world. +/ replaces torq.q's in-process config handling (loadf, loaddir, loadconfig, +/ loadaddconfig, overrideconfig). the logger is an injected dependency wired via +/ init; the module reads no environment variables or process identity itself - +/ the caller (di.torq) supplies the settings directories, name sequence and any +/ command-line overrides. layered lowest to highest priority: +/ loadfile - load one file by path (deduplicated) +/ loadconfig - load one cascade file dir/{name}.q (missing is normal) +/ loaddir - load every .q/.k file in a directory (honours order.txt) +/ loadcascade - resolve the full cascade over dirs x names (name-major) +/ overrideconfig - apply command-line-style overrides after the cascade +/ get / getmodule - query the resolved config store (di.torq partitions per +/ module with getmodule and injects each slice via that module's init) + +raiseerror:{[ctx;msg] + / internal - log an error under ctx then signal it, so failures are observable + / in the log as well as thrown to the caller. + .z.m.log[`error][ctx;msg]; + '"di.config: ",string[ctx],": ",msg; + }; + +init:{[deps] + / wire injectable dependencies - log is required, there is no silent fallback. + / deps: a dict with a `log key; log must already be a binary `info`warn`error + / dict of {[c;m]} loggers (from di.log once it ships, or hand-rolled). no + / adaptation is performed here - a raw monadic kx.log instance must be wrapped + / by the caller before being passed. examples: + / config.init[enlist[`log]!enlist logdep] + / config.init[`log`timer!(logdep;timerdep)] + if[99h<>type deps; + '"di.config: deps must be a dict with `log key"]; + if[not `log in key deps; + '"di.config: log dependency is required; pass `info`warn`error functions keyed on `log"]; + if[99h<>type deps`log; + '"di.config: log value must be a dict; pass `info`warn`error functions"]; + if[not all (`info`warn`error) in key deps`log; + '"di.config: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + .z.m.log:deps`log; + .z.m.loaded:enlist""; + .z.m.log[`info][`init;"di.config initialised"]; + }; + +loadfile:{[file] + / load a single q config file if it exists, tracking it so it is not re-loaded. + / returns the file path; a missing file is a logged warning, not an error. + if[not 10h=abs type file; + raiseerror[`loadfile;"file must be a string path"]; + ]; + if[file in .z.m.loaded; + .z.m.log[`info][`loadfile;"already loaded ",file]; + :file; + ]; + if[()~key hsym `$file; + .z.m.log[`warn][`loadfile;"config file not found: ",file]; + :file; + ]; + .z.m.log[`info][`loadfile;"loading ",file]; + .[system;enlist"l ",file;{[f;e] raiseerror[`loadfile;"failed to load ",f,": ",e]}[file;]]; + .z.m.loaded,:enlist file; + :file; + }; + +loadconfig:{[dir;name] + / load a single cascade config file dir/{name}.q if it is present. a missing + / file is normal in the cascade (not every proctype/procname has one), so its + / absence is logged at info and skipped, NOT warned. present files load via + / loadfile, so they are tracked and de-duplicated. returns the file path. + if[not 10h=abs type dir; + raiseerror[`loadconfig;"dir must be a string path"]; + ]; + if[not -11h=type name; + raiseerror[`loadconfig;"name must be a symbol"]; + ]; + file:dir,"/",string[name],".q"; + if[()~key hsym `$file; + .z.m.log[`info][`loadconfig;"no config file (skipping): ",file]; + :file; + ]; + :loadfile file; + }; + +loaddir:{[dir] + / load every .q and .k file in a directory, honouring an optional order.txt. + / files listed in order.txt load first, in that order; the rest follow in the + / order key returns them. returns the ordered list of file paths processed; + / a missing directory is a logged warning returning (). each load is delegated + / to loadfile, so files already loaded are skipped. + if[not 10h=abs type dir; + raiseerror[`loaddir;"dir must be a string path"]; + ]; + if[()~files:key hsym `$dir; + .z.m.log[`warn][`loaddir;"directory not found: ",dir]; + :(); + ]; + haveorder:`order.txt in files; + if[haveorder; + .z.m.log[`info][`loaddir;"found order.txt in ",dir]; + ]; + order:$[haveorder;(`$read0 hsym `$dir,"/order.txt") inter files;`symbol$()]; + files:files where any files like/:("*.q";"*.k"); + files:order,files except order; + paths:(dir,"/"),/:string files; + .z.m.log[`info][`loaddir;"loading ",(string count paths)," file(s) from ",dir]; + loadfile each paths; + :paths; + }; + +loadcascade:{[dirs;names] + / resolve a configuration cascade name-major: for each config name (least to + / most specific) load it from every directory in turn. files load in sequence, + / so a more specific name always wins over a less specific one regardless of + / directory, and within a single name the later (higher-priority) directory + / overrides the earlier. dirs is a directory path or list of paths (strings), + / ordered lowest->highest priority; names is a config name or list of names + / (symbols), ordered least->most specific. missing files are skipped (by + / loadconfig). returns the flat list of constructed paths, in load order. + / typical caller (di.torq resolves the dirs and process identity): + / config.loadcascade[(kdbconfig;appconfig);`default,proctype,procname] + dirs:$[10h=type dirs;enlist dirs;dirs]; + names:(),names; + if[not 0h=type dirs; + raiseerror[`loadcascade;"dirs must be a string path or a list of string paths"]; + ]; + if[not all 10h=type each dirs; + raiseerror[`loadcascade;"dirs must be a string path or a list of string paths"]; + ]; + if[not 11h=abs type names; + raiseerror[`loadcascade;"names must be a symbol or a symbol list"]; + ]; + .z.m.log[`info][`loadcascade;"resolving cascade: ",(string count dirs)," dir(s) x ",(string count names)," name(s)"]; + :raze {[ds;nm] loadconfig[;nm] each ds}[dirs;] each names; + }; + +applyoverride:{[name;raw] + / internal - parse raw command-line value(s) into name's current type and set + / it. raw is a string or a list of strings. returns 1b if applied, 0b if the + / variable is not a basic type or a value failed to parse (null). the variable + / must already exist - its current type drives the parse. + t:type value name; + if[not (abs t) within (1;-1+count .Q.t); + .z.m.log[`error][`overrideconfig;"cannot override ",(string name),": not a basic type"]; + :0b; + ]; + raw:$[10h=type raw;enlist raw;raw]; + vals:(upper .Q.t abs t)$'raw; + if[t<0;vals:first vals]; + if[any null vals; + .z.m.log[`error][`overrideconfig;"cannot override ",(string name),": value did not parse"]; + :0b; + ]; + .z.m.log[`info][`overrideconfig;"setting ",(string name)," to ",-3!vals]; + set[name;vals]; + :1b; + }; + +overrideconfig:{[params] + / apply command-line-style overrides to already-defined variables, parsing each + / value into the variable's existing type. params is a dict keyed by variable + / name (symbol) with string (or list-of-string) values. only defined variables + / can be overridden (their type drives the parse); undefined names are logged + / and skipped. returns the list of variables actually overridden. call after + / loadcascade to let the command line win over file config. + if[99h<>type params; + raiseerror[`overrideconfig;"params must be a dict keyed by variable name"]; + ]; + vars:key params; + if[0 Date: Fri, 10 Jul 2026 12:26:26 +0100 Subject: [PATCH 07/12] Complete v1, trimmed to only essential API, new logging added, finalised di.logging and di.torq required for full deployment --- di/config/config.md | 117 ++++++++++++++++++++++++++++++-------------- di/config/config.q | 86 +++++++++----------------------- di/config/init.q | 2 +- di/config/test.csv | 72 ++++++--------------------- 4 files changed, 118 insertions(+), 159 deletions(-) diff --git a/di/config/config.md b/di/config/config.md index d7419d37..d619583f 100644 --- a/di/config/config.md +++ b/di/config/config.md @@ -1,27 +1,28 @@ # di.config Configuration loading and cascade resolution for the modular TorQ world. This -replaces TorQ's in-process config handling (`torq.q`: `loadf`, `loaddir`, -`loadconfig`, `loadaddconfig`, `overrideconfig`), where `di.torq` will later -distribute the resolved config to each module via that module's `init`. +replaces TorQ's in-process config handling (`torq.q`: `loadf`, `loadconfig`, +`loadaddconfig`, `overrideconfig`), where `di.torq` will later distribute the +resolved config to each module via that module's `init`. ## Import and init `init` requires a `log` dependency — there is no silent fallback. Pass an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` loggers (context -symbol, message string). The module performs **no** adaptation, so a raw monadic -[`kx.log`](https://github.com/KxSystems/logging) instance must be wrapped by the -caller first (this is `di.log`'s job once it ships): +symbol, message string). The module performs **no** adaptation of the logger. + +The standard logger is **`di.log`**, which exports binary `info`/`warn`/`error` +functions directly — build the dict from its exports: ```q config:use`di.config -/ option 1: di.log (standard, once it ships) - build the binary dict from its exports +/ option 1: di.log — the standard logger (exports binary info/warn/error) logger:use`di.log logdep:`info`warn`error!(logger.info;logger.warn;logger.error) config.init[enlist[`log]!enlist logdep] -/ option 2: hand-rolled binary {[c;m]} logger (works today) +/ option 2: any hand-rolled binary {[c;m]} logger mylog:`info`warn`error!( {[c;m] -1 string[c],": INFO ",m;}; {[c;m] -1 string[c],": WARN ",m;}; @@ -29,8 +30,9 @@ mylog:`info`warn`error!( config.init[enlist[`log]!enlist mylog] ``` -To wrap a raw `kx.log` instance yourself in the interim, fold the context into the -message per call: `` `info`warn`error!({[c;m]inst[`info][string[c],": ",m]};…) ``. +A logger that is *not* already binary `{[c;m]}` (e.g. a raw monadic +[`kx.log`](https://github.com/KxSystems/logging) instance) must be wrapped by the +caller first — di.config does no wrapping: `` `info`warn`error!({[c;m]inst[`info][string[c],": ",m]};…) ``. `init` must be called before any other function — there is no default logger. @@ -39,19 +41,13 @@ message per call: `` `info`warn`error!({[c;m]inst[`info][string[c],": ",m]};…) | Function | Signature | Description | |---|---|---| | `init` | `init[deps]` | Wire the injected logger. `deps` is a dict with a required `log` key — an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` functions. Errors immediately if the log dependency is missing or malformed. | -| `loadfile` | `loadfile[file]` | Load a single q config file (a string path) if it exists, tracking it so it is not re-loaded. Returns the path. A missing file is a logged warning, not an error; a failed load is signalled. | -| `loadconfig` | `loadconfig[dir;name]` | Load one cascade config file `dir/{name}.q` (`dir` a string path, `name` a symbol) if present. Returns the constructed path. A missing file is **normal** in the cascade, so its absence is logged at *info* and skipped (not warned). Present files load via `loadfile`. | -| `loaddir` | `loaddir[dir]` | Load every `.q`/`.k` file in a directory (a string path), honouring an optional `order.txt`. Files listed in `order.txt` load first, in that order; the rest follow. Returns the ordered list of file paths processed. A missing directory is a logged warning returning `()`. Delegates each load to `loadfile`, so already-loaded files are skipped. | -| `loadcascade` | `loadcascade[dirs;names]` | Resolve a config cascade: for each settings directory (in order) load each named file `dir/{name}.q` (in name order). `dirs` is a string path or list of paths; `names` is a symbol or symbol list. Later names and later directories override earlier ones (load order). Missing files are skipped. Returns the flat list of constructed paths in cascade order. | +| `loadcascade` | `loadcascade[dirs;names]` | Resolve the config cascade **name-major**: for each name (least→most specific) load `dir/{name}.q` from each directory in turn. `dirs` is a string path or list of paths (low→high priority); `names` is a symbol or symbol list (least→most specific). A more specific name wins over the directory layer; within a name the later directory overrides. Missing files are skipped (logged at *info*); already-loaded files are de-duplicated. Returns the flat list of constructed paths, in load order. | | `overrideconfig` | `overrideconfig[params]` | Apply command-line-style overrides after the cascade. `params` is a dict keyed by variable name (symbol) with string (or list-of-string) values, each parsed into that variable's **existing** type. Only already-defined, basic-typed variables can be overridden; undefined names, non-basic types, and unparseable values are logged and skipped. Returns the list of variables actually overridden. | | `get` | `get[namespace;key;default]` | Query the resolved config store: return the value of `.{namespace}.{key}`, or `default` if unset. `namespace` and `key` are **bare** symbols (no leading dot), e.g. `get[\`rdb;\`subscribeto;\`]`. (`get` is a reserved word, so the implementation function is named `getcfg` and exported under the `get` key.) | | `getmodule` | `getmodule[namespace]` | Return a namespace's whole resolved config as a bare-keyed value dict — the slice di.torq passes to a module's `init`. `namespace` is a bare symbol; an unconfigured namespace yields an empty dict. | -`order.txt` is a plain-text file, one filename per line, naming the files that must -load ahead of the rest (e.g. to satisfy load-order dependencies). - -Internal helpers (`raiseerror`, `applyoverride`) and the load-tracking state are deliberately -not exported. +Internal helpers — `loadconfig` (the per-file loader behind `loadcascade`), `raiseerror`, +`applyoverride`, and the load-tracking state — are deliberately not exported. `loadcascade` is the top-level entry point. A caller (e.g. `di.torq`, which resolves the process identity and directory paths) drives it as: @@ -83,13 +79,13 @@ plug in by populating the same namespaces before the store is queried — the `g > **Note:** avoid `-` in config file paths — a source-level backtick symbol literal > containing `-` parses as subtraction (`` `:/a-b.q `` → `` `:/a `` `- b.q`), not one -> symbol. Runtime string paths passed to `loadfile` are unaffected. +> symbol. Runtime string paths (built internally by `loadcascade`) are unaffected. ## Injectable dependencies | Injectable | Required keys | Function signature | |---|---|---| -| `log` | `` `info`warn`error `` | `{[c;m]}` (context symbol, message string); caller passes an already-conforming binary dict (from `di.log` once it ships, or hand-rolled). No adaptation is done in the module — a raw monadic `kx.log` instance must be wrapped first | +| `log` | `` `info`warn`error `` | `{[c;m]}` (context symbol, message string); caller passes an already-conforming binary dict — built from `di.log` (the standard logger, which exports binary `info`/`warn`/`error`) or hand-rolled. No adaptation is done in the module — a non-binary logger (e.g. a raw monadic `kx.log` instance) must be wrapped first | ## Hard dependencies @@ -97,15 +93,14 @@ None — `di.config` is a standalone module. ## Design notes & open gaps -- **Return values are informational.** `loadfile`/`loadconfig`/`loaddir`/`loadcascade` return the - path(s) they *considered* — including files that were missing or skipped. The real effect is the - side-effect load; use the return to see what was attempted, not what definitely loaded. -- **Missing-file log level is deliberate.** `loadfile` logs a missing explicit path at *warn* (you - asked for it); `loadconfig` logs a missing cascade file at *info* (absence is normal — most - `{proctype}`/`{procname}` files won't exist); `loaddir` logs a missing directory at *warn* and - returns `()`. -- **Dedup / idempotency.** `loadfile` tracks loaded paths in `.z.m.loaded` and skips re-loads, so the - cascade is safe to re-run and `init` is safe to call again. +- **Return values are informational.** `loadcascade` returns the path(s) it *considered* — including + files that were missing or skipped. The real effect is the side-effect load; use the return to see + what was attempted, not what definitely loaded. +- **Missing cascade files are skipped, not errored.** The internal `loadconfig` logs a missing + `dir/{name}.q` at *info* and skips it — absence is normal in the cascade (most `{proctype}`/ + `{procname}` files won't exist). +- **Dedup / idempotency.** `loadconfig` tracks loaded paths in `.z.m.loaded` and skips re-loads, so + the cascade is safe to re-run and `init` is safe to call again. - **`overrideconfig` deviates from TorQ deliberately.** Per-variable problems (undefined name, non-basic type, value that won't parse to the target type) are logged and skipped — a single bad override never aborts the batch and a null is never written into config. @@ -125,17 +120,18 @@ None — `di.config` is a standalone module. Open gaps / not implemented: - **No runtime reload.** TorQ's `reloadf` (force-reload an already-loaded file) is not ported — - `loadfile` always dedups. Adequate for startup; revisit if runtime config reload is needed. -- **`order.txt` is matched against the full directory listing** (as in TorQ), so a non-`.q`/`.k` file - named in `order.txt` would still be prepended and load-attempted. Keep `order.txt` to q/k files. + `loadconfig` always dedups. Adequate for startup; revisit if runtime config reload is needed. - **`overrideconfig`'s null-parse guard targets scalar/vector basic types**; a string-typed (`10h`) config var is an uncommon edge where the per-element null check is loose. -- **Standard logger (`di.log`) does not exist yet.** Callers must supply a hand-rolled binary - `` `info`warn`error `` dict in the interim; the kx.log wiring test wraps kx.log caller-side to prove - real emission. +- **No committed `di.log` integration test yet.** `di.log` (the standard logger) currently lives on + the `feature-logging` branch and isn't merged, so a committed `use\`di.log` test would fail on this + branch. di.config's contract match with `di.log` is **verified manually** — the real `di.log` was + wired end-to-end with no code change (`init`/`loadcascade`/`loadconfig` all emit through it). The + committed kx.log-wrapper test proves the same binary-`{[c;m]}`-dict contract `di.log` satisfies. Add + a `di.log` integration test once `di.log` merges. - **Logger storage follows the skill's single-dict pattern** (`.z.m.log[\`info][…]`), which diverges - from `consistency.md`'s three-flat-var example (`.z.m.loginfo`/`…`). Per the skill this is the - directed pattern, pending `consistency.md` being reconciled once `di.log` ships. + from `consistency.md`'s three-flat-var example (`.z.m.loginfo`/`…`). `di.log` is now built and the + single-dict pattern is the directed one; `consistency.md` still needs reconciling to match. ## Tests @@ -143,3 +139,48 @@ Open gaps / not implemented: k4unit:use`di.k4unit k4unit.moduletest`di.config ``` + +## Follow-ups (deferred until other modules land) + +di.config is complete for v1 in isolation. The items below are intentionally deferred; +each is keyed to the module or milestone that unblocks it. Do these **when the trigger +lands**, not before. + +### When `di.log` merges to main + +- **Add a committed `di.log` integration test.** Today `di.log` lives on the + `feature-logging` branch (unmerged), so a committed `` use`di.log `` test would fail + here — the contract match is verified manually and stood in for by the kx.log-wrapper + emission test. Once `di.log` resolves on `QPATH`, add a test that builds + `` `info`warn`error!(logger.info;logger.warn;logger.error) `` from `di.log` and drives + `loadcascade` through it. +- **Re-verify only if `di.log`'s contract changes.** It isn't final/approved yet. No + di.config code change is expected — di.config needs only binary `info`/`warn`/`error`, + which `di.log` exports — but if those signatures change, re-run the integration drive. + +### When `di.torq` is built + +- **Prove the end-to-end flow.** `getmodule` is unit-tested here, but "partition per + module and inject via `init`" only runs for real once `di.torq` exists. Add a + multi-module integration test (under `di.torq`/`di.inttest`, not here) exercising: + di.torq resolves `KDBCONFIG`/`KDBAPPCONFIG` → `dirs` and `proctype`/`procname` → + `names`, calls `loadcascade`, then `overrideconfig` with the command-line params, then + `getmodule` per module → each module's `init`. +- **Hold the env-free boundary.** di.config reads no env vars or process identity — + di.torq owns that resolution and passes `dirs`/`names` explicitly. Do **not** add a + zero-arg `load[]` to di.config; if di.torq wants one it wraps `loadcascade` in a line + on its own side. + +### When `di.depcheck` is built (versioning rollout) + +- **Add a `version` export and `deps.q`.** No module ships these yet and `di.depcheck` + (which consumes them) doesn't exist. When it lands, add `version:"…"` to the export and + a `deps.q` (empty/minimal — di.config has no hard deps) as part of the coordinated + repo-wide rollout, not a di.config-only change. + +### When a second config source (env vars, k8s config maps) is needed + +- **Implement the 1b explicit store.** Replace the live-namespace-globals store with an + internal store carrying provenance / cross-source precedence, behind the unchanged + `get`/`getmodule` contract (so consumers don't change). See the `EXTENSION POINT` + comment in `config.q`. diff --git a/di/config/config.q b/di/config/config.q index 8167cd4f..ec93c443 100644 --- a/di/config/config.q +++ b/di/config/config.q @@ -1,16 +1,14 @@ / configuration loading and cascade resolution for the modular torq world. -/ replaces torq.q's in-process config handling (loadf, loaddir, loadconfig, -/ loadaddconfig, overrideconfig). the logger is an injected dependency wired via -/ init; the module reads no environment variables or process identity itself - -/ the caller (di.torq) supplies the settings directories, name sequence and any -/ command-line overrides. layered lowest to highest priority: -/ loadfile - load one file by path (deduplicated) -/ loadconfig - load one cascade file dir/{name}.q (missing is normal) -/ loaddir - load every .q/.k file in a directory (honours order.txt) -/ loadcascade - resolve the full cascade over dirs x names (name-major) +/ replaces torq.q's in-process config handling (loadf, loadconfig, loadaddconfig, +/ overrideconfig). the logger is an injected dependency wired via init; the module +/ reads no environment variables or process identity itself - the caller (di.torq) +/ supplies the settings directories, name sequence and any command-line overrides. +/ public api: +/ loadcascade - resolve the full cascade over dirs x names (name-major) / overrideconfig - apply command-line-style overrides after the cascade / get / getmodule - query the resolved config store (di.torq partitions per / module with getmodule and injects each slice via that module's init) +/ internal helper loadconfig handles per-file loading behind loadcascade. raiseerror:{[ctx;msg] / internal - log an error under ctx then signal it, so failures are observable @@ -22,9 +20,10 @@ raiseerror:{[ctx;msg] init:{[deps] / wire injectable dependencies - log is required, there is no silent fallback. / deps: a dict with a `log key; log must already be a binary `info`warn`error - / dict of {[c;m]} loggers (from di.log once it ships, or hand-rolled). no - / adaptation is performed here - a raw monadic kx.log instance must be wrapped - / by the caller before being passed. examples: + / dict of {[c;m]} loggers - build it from di.log (the standard logger, which + / exports binary info/warn/error) or hand-roll one. no adaptation is performed + / here, so a non-conforming logger (e.g. a raw monadic kx.log instance) must be + / wrapped by the caller first. examples: / config.init[enlist[`log]!enlist logdep] / config.init[`log`timer!(logdep;timerdep)] if[99h<>type deps; @@ -40,31 +39,11 @@ init:{[deps] .z.m.log[`info][`init;"di.config initialised"]; }; -loadfile:{[file] - / load a single q config file if it exists, tracking it so it is not re-loaded. - / returns the file path; a missing file is a logged warning, not an error. - if[not 10h=abs type file; - raiseerror[`loadfile;"file must be a string path"]; - ]; - if[file in .z.m.loaded; - .z.m.log[`info][`loadfile;"already loaded ",file]; - :file; - ]; - if[()~key hsym `$file; - .z.m.log[`warn][`loadfile;"config file not found: ",file]; - :file; - ]; - .z.m.log[`info][`loadfile;"loading ",file]; - .[system;enlist"l ",file;{[f;e] raiseerror[`loadfile;"failed to load ",f,": ",e]}[file;]]; - .z.m.loaded,:enlist file; - :file; - }; - loadconfig:{[dir;name] - / load a single cascade config file dir/{name}.q if it is present. a missing - / file is normal in the cascade (not every proctype/procname has one), so its - / absence is logged at info and skipped, NOT warned. present files load via - / loadfile, so they are tracked and de-duplicated. returns the file path. + / internal - load one cascade config file dir/{name}.q if present, tracking it + / so it is not re-loaded. a missing file is normal in the cascade (not every + / proctype/procname has one), so its absence is logged at info and skipped, not + / warned. returns the file path. if[not 10h=abs type dir; raiseerror[`loadconfig;"dir must be a string path"]; ]; @@ -72,37 +51,18 @@ loadconfig:{[dir;name] raiseerror[`loadconfig;"name must be a symbol"]; ]; file:dir,"/",string[name],".q"; + if[file in .z.m.loaded; + .z.m.log[`info][`loadconfig;"already loaded ",file]; + :file; + ]; if[()~key hsym `$file; .z.m.log[`info][`loadconfig;"no config file (skipping): ",file]; :file; ]; - :loadfile file; - }; - -loaddir:{[dir] - / load every .q and .k file in a directory, honouring an optional order.txt. - / files listed in order.txt load first, in that order; the rest follow in the - / order key returns them. returns the ordered list of file paths processed; - / a missing directory is a logged warning returning (). each load is delegated - / to loadfile, so files already loaded are skipped. - if[not 10h=abs type dir; - raiseerror[`loaddir;"dir must be a string path"]; - ]; - if[()~files:key hsym `$dir; - .z.m.log[`warn][`loaddir;"directory not found: ",dir]; - :(); - ]; - haveorder:`order.txt in files; - if[haveorder; - .z.m.log[`info][`loaddir;"found order.txt in ",dir]; - ]; - order:$[haveorder;(`$read0 hsym `$dir,"/order.txt") inter files;`symbol$()]; - files:files where any files like/:("*.q";"*.k"); - files:order,files except order; - paths:(dir,"/"),/:string files; - .z.m.log[`info][`loaddir;"loading ",(string count paths)," file(s) from ",dir]; - loadfile each paths; - :paths; + .z.m.log[`info][`loadconfig;"loading ",file]; + .[system;enlist"l ",file;{[f;e] raiseerror[`loadconfig;"failed to load ",f,": ",e]}[file;]]; + .z.m.loaded,:enlist file; + :file; }; loadcascade:{[dirs;names] diff --git a/di/config/init.q b/di/config/init.q index b0e5237c..ae1d533f 100644 --- a/di/config/init.q +++ b/di/config/init.q @@ -2,4 +2,4 @@ / note: `get` is a q reserved word, so it cannot be a top-level name; the query / function is defined as `getcfg` in config.q and exported under the `get` key. \l ::config.q -export:([init;loadfile;loadconfig;loaddir;loadcascade;overrideconfig;getmodule]),(enlist`get)!enlist getcfg +export:([init;loadcascade;overrideconfig;getmodule]),(enlist`get)!enlist getcfg diff --git a/di/config/test.csv b/di/config/test.csv index 01daaa51..6433060a 100644 --- a/di/config/test.csv +++ b/di/config/test.csv @@ -3,77 +3,35 @@ before,0,0,q,cfg:use`di.config,1,1,load the module before,0,0,q,"logtab:([]lvl:`symbol$();ctx:`symbol$();msg:())",1,1,capturing log table before,0,0,q,"mocklog:`info`warn`error!({[c;m]`logtab upsert(`info;c;m)};{[c;m]`logtab upsert(`warn;c;m)};{[c;m]`logtab upsert(`error;c;m)})",1,1,define a capturing binary logger before,0,0,q,cfg.init[enlist[`log]!enlist mocklog],1,1,init with the required log dependency -before,0,0,q,"(`:/tmp/diconfigtest.q) 0: enlist ""diconfigtestvar:42;diconfigcount:1+@[value;`diconfigcount;0];""",1,1,write a temp config file that sets a var and counts loads -before,0,0,q,diconfigorder:`$(),1,1,tracker for the order files load in -before,0,0,q,"system""mkdir -p /tmp/diconfigloaddir""",1,1,make a directory with an order.txt -before,0,0,q,"(`:/tmp/diconfigloaddir/a.q) 0: enlist ""diconfiga:1;diconfigorder,:`a;""",1,1,dir file a.q -before,0,0,q,"(`:/tmp/diconfigloaddir/b.q) 0: enlist ""diconfigb:1;diconfigorder,:`b;""",1,1,dir file b.q -before,0,0,q,"(`:/tmp/diconfigloaddir/d.k) 0: enlist """"",1,1,dir file d.k (proves .k pickup) -before,0,0,q,"(`:/tmp/diconfigloaddir/notes.txt) 0: enlist ""not a q file""",1,1,non-q file that must be ignored -before,0,0,q,"(`:/tmp/diconfigloaddir/order.txt) 0: (""b.q"";""a.q"")",1,1,order.txt loads b before a -before,0,0,q,"system""mkdir -p /tmp/diconfignoorder""",1,1,make a directory without an order.txt -before,0,0,q,"(`:/tmp/diconfignoorder/p.q) 0: enlist ""diconfigp:1;""",1,1,noorder file p.q -before,0,0,q,"(`:/tmp/diconfignoorder/q.q) 0: enlist ""diconfigq:1;""",1,1,noorder file q.q -before,0,0,q,"system""mkdir -p /tmp/diconfigcascade""",1,1,make a cascade settings directory -before,0,0,q,"(`:/tmp/diconfigcascade/default.q) 0: enlist ""diconfigdefault:99;""",1,1,cascade default.q before,0,0,q,diconfigtrace:`symbol$(),1,1,tracer for cascade load order before,0,0,q,"system""mkdir -p /tmp/diconfigc1""",1,1,cascade directory one before,0,0,q,"system""mkdir -p /tmp/diconfigc2""",1,1,cascade directory two before,0,0,q,"(`:/tmp/diconfigc1/default.q) 0: enlist ""diconfigcx:1;diconfigtrace,:`c1default;""",1,1,c1 default.q before,0,0,q,"(`:/tmp/diconfigc1/rdb.q) 0: enlist ""diconfigcx:10;diconfigtrace,:`c1rdb;""",1,1,c1 rdb.q -before,0,0,q,"(`:/tmp/diconfigc2/default.q) 0: enlist ""diconfigcx:2;diconfigtrace,:`c2default;""",1,1,c2 default.q (no c2 rdb.q) +before,0,0,q,"(`:/tmp/diconfigc2/default.q) 0: enlist ""diconfigcx:2;diconfigtrace,:`c2default;""",1,1,c2 default.q (no c2 rdb.q - a missing cascade file) before,0,0,q,"system""mkdir -p /tmp/diconfigstore""",1,1,cascade dir for the config-store tests before,0,0,q,"(`:/tmp/diconfigstore/default.q) 0: enlist "".diconfigq.subscribeto:`trade`quote;.diconfigq.rows:100;""",1,1,settings file populating the .diconfigq namespace before,0,0,q,.diconfigtest.enabled:0b,1,1,override target - boolean before,0,0,q,.diconfigtest.rows:100,1,1,override target - long before,0,0,q,.diconfigtest.syms:`a`b,1,1,override target - symbol list before,0,0,q,.diconfigtest.tab:([]a:`long$()),1,1,override target - non-basic (table) -before,0,0,q,"(`:/tmp/diconfigkxreal.q) 0: enlist ""diconfigkxvar:1;""",1,1,file to load through a real kx.log +before,0,0,q,"system""mkdir -p /tmp/diconfigkxdir""",1,1,dir for the kx.log emission test +before,0,0,q,"(`:/tmp/diconfigkxdir/default.q) 0: enlist ""diconfigkxvar:1;""",1,1,file loaded through the real logger comment,,,,,,,init - dependency validation fail,0,0,q,cfg.init[(::)],1,1,init rejects a non-dictionary deps fail,0,0,q,cfg.init[enlist[`foo]!enlist 1],1,1,init rejects deps without a log key fail,0,0,q,cfg.init[enlist[`log]!enlist 42],1,1,init rejects a non-dict log value fail,0,0,q,cfg.init[enlist[`log]!enlist ((enlist`info)!enlist {[c;m]})],1,1,init rejects a log dict missing warn/error -comment,,,,,,,loadfile - missing file is a warning -true,0,0,q,"""/tmp/diconfignope.q""~cfg.loadfile[""/tmp/diconfignope.q""]",1,1,missing file returns its path -true,0,0,q,`warn in exec lvl from logtab,1,1,missing file logged a warning -comment,,,,,,,loadfile - real load and dedup -true,0,0,q,"""/tmp/diconfigtest.q""~cfg.loadfile[""/tmp/diconfigtest.q""]",1,1,real file returns its path -true,0,0,q,42=diconfigtestvar,1,1,loaded file executed and set its global -true,0,0,q,1=diconfigcount,1,1,file was loaded exactly once -run,0,0,q,"cfg.loadfile[""/tmp/diconfigtest.q""]",1,1,re-load the same file -true,0,0,q,1=diconfigcount,1,1,re-load is skipped (counter unchanged) -comment,,,,,,,loadfile - input validation -fail,0,0,q,cfg.loadfile[42],1,1,loadfile rejects a non-string path -comment,,,,,,,loaddir - ordered load honouring order.txt -run,0,0,q,"cfg.loaddir[""/tmp/diconfigloaddir""]",1,1,load the directory (first time) -true,0,0,q,"3=count cfg.loaddir[""/tmp/diconfigloaddir""]",1,1,three q/k files loaded (notes.txt ignored) -true,0,0,q,`b`a~diconfigorder,1,1,order.txt loaded b before a -true,0,0,q,"any (cfg.loaddir[""/tmp/diconfigloaddir""]) like ""*d.k""",1,1,.k files are picked up -true,0,0,q,"not any (cfg.loaddir[""/tmp/diconfigloaddir""]) like ""*.txt""",1,1,non-q/k files are excluded -true,0,0,q,2=diconfiga+diconfigb,1,1,directory files actually executed -comment,,,,,,,loaddir - directory without order.txt -true,0,0,q,"2=count cfg.loaddir[""/tmp/diconfignoorder""]",1,1,loads all q files when no order.txt present -comment,,,,,,,loaddir - missing directory and validation -true,0,0,q,"()~cfg.loaddir[""/tmp/diconfignodir""]",1,1,missing directory returns an empty list -true,0,0,q,"any (exec msg from logtab) like ""*directory not found*""",1,1,missing directory logged a warning -fail,0,0,q,cfg.loaddir[42],1,1,loaddir rejects a non-string path -comment,,,,,,,loadconfig - named cascade file -true,0,0,q,"""/tmp/diconfigcascade/default.q""~cfg.loadconfig[""/tmp/diconfigcascade"";`default]",1,1,named file returns its constructed path -true,0,0,q,99=diconfigdefault,1,1,named file was executed -true,0,0,q,"""/tmp/diconfigcascade/nosuch.q""~cfg.loadconfig[""/tmp/diconfigcascade"";`nosuch]",1,1,missing named file returns its path without erroring -true,0,0,q,"any (exec msg from logtab where lvl=`info) like ""*no config file*""",1,1,missing cascade file logged at info not warn -fail,0,0,q,cfg.loadconfig[42;`default],1,1,loadconfig rejects a non-string dir -fail,0,0,q,"cfg.loadconfig[""/tmp/diconfigcascade"";""default""]",1,1,loadconfig rejects a non-symbol name -comment,,,,,,,load - full cascade over directories and names +comment,,,,,,,loadcascade - full cascade over directories and names run,0,0,q,"cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,run the cascade true,0,0,q,"4=count cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,one path per dir-name pair (skipped files included) true,0,0,q,10=diconfigcx,1,1,most-specific name wins over dir layer (c1/rdb loads last) -true,0,0,q,`c1default`c2default`c1rdb~diconfigtrace,1,1,cascade order is name-major (each name across all dirs) -true,0,0,q,"any (cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]) like ""*diconfigc2/rdb.q""",1,1,returned paths include skipped files +true,0,0,q,`c1default`c2default`c1rdb~diconfigtrace,1,1,name-major order; and re-runs don't re-execute (dedup) +true,0,0,q,"any (cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]) like ""*diconfigc2/rdb.q""",1,1,returned paths include skipped (missing) files +true,0,0,q,"any (exec msg from logtab where lvl=`info) like ""*no config file*""",1,1,a missing cascade file is skipped at info (internal loadconfig) true,0,0,q,"1=count cfg.loadcascade[""/tmp/diconfigc1"";`default]",1,1,accepts a single string dir and single symbol name -fail,0,0,q,cfg.loadcascade[42;`default],1,1,load rejects non-string dirs -fail,0,0,q,"cfg.loadcascade[""/tmp/diconfigc1"";""default""]",1,1,load rejects non-symbol names +fail,0,0,q,cfg.loadcascade[42;`default],1,1,loadcascade rejects non-string dirs +fail,0,0,q,"cfg.loadcascade[""/tmp/diconfigc1"";""default""]",1,1,loadcascade rejects non-symbol names comment,,,,,,,overrideconfig - parse and apply command-line overrides run,0,0,q,"cfg.overrideconfig[`.diconfigtest.enabled`.diconfigtest.rows!(enlist""1"";enlist""500"")]",1,1,override a bool and a long true,0,0,q,1b~.diconfigtest.enabled,1,1,bool parsed and set from string @@ -101,15 +59,15 @@ run,0,0,q,"cfg.overrideconfig[enlist[`.diconfigq.rows]!enlist enlist""500""]",1, true,0,0,q,500~cfg.get[`diconfigq;`rows;0],1,1,get reflects overrideconfig changes fail,0,0,q,cfg.get[42;`k;0],1,1,get rejects a non-symbol namespace fail,0,0,q,cfg.getmodule[42],1,1,getmodule rejects a non-symbol namespace -comment,,,,,,,kx.log - real emission via a caller-side binary wrapper (must run last; re-inits cfg) +comment,,,,,,,logger emission - real logger via a caller-side binary wrapper (must run last; re-inits cfg) run,0,0,q,kxlogger:use`kx.log,1,1,load the kx.log module run,0,0,q,kxinst:kxlogger.createLog[],1,1,create a real kx.log instance run,0,0,q,kxh:hopen `:/tmp/diconfigkxsink.txt,1,1,open a file sink to capture output run,0,0,q,kxinst.add[kxh;`info],1,1,route info-level output to the file sink -run,0,0,q,"kxbinary:`info`warn`error!({[c;m]kxinst[`info][string[c],"": "",m]};{[c;m]kxinst[`warn][string[c],"": "",m]};{[c;m]kxinst[`error][string[c],"": "",m]})",1,1,wrap monadic kx.log into a binary dict (what di.log will do) +run,0,0,q,"kxbinary:`info`warn`error!({[c;m]kxinst[`info][string[c],"": "",m]};{[c;m]kxinst[`warn][string[c],"": "",m]};{[c;m]kxinst[`error][string[c],"": "",m]})",1,1,wrap the monadic logger into the binary {[c;m]} contract run,0,0,q,cfg.init[enlist[`log]!enlist kxbinary],1,1,re-init di.config with the conforming binary logger -run,0,0,q,"cfg.loadfile[""/tmp/diconfigkxreal.q""]",1,1,drive a real info-level log through kx.log +run,0,0,q,"cfg.loadcascade[""/tmp/diconfigkxdir"";`default]",1,1,drive a real info-level log through the wrapped logger run,0,0,q,hclose kxh,1,1,flush and close the file sink -true,0,0,q,1=diconfigkxvar,1,1,the real load executed the file -true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*loadfile: loading*""",1,1,kx.log emitted the wrapped context and message -true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*diconfigkxreal.q*""",1,1,emitted message includes the file path +true,0,0,q,1=diconfigkxvar,1,1,the cascade executed the file +true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*loadconfig: loading*""",1,1,the internal loader emitted through the real logger +true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*diconfigkxdir/default.q*""",1,1,emitted message includes the file path From b5f2e030d212ae30b0c91639bc086da3cc1a2fc3 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Mon, 13 Jul 2026 18:25:29 +0100 Subject: [PATCH 08/12] updating tests and removing not needed function after review call --- di/config/config.md | 61 +++++++++++++++++++++++++++++++-------------- di/config/config.q | 52 ++++++++++++++------------------------ di/config/init.q | 4 +-- di/config/test.csv | 8 ++---- 4 files changed, 64 insertions(+), 61 deletions(-) diff --git a/di/config/config.md b/di/config/config.md index d619583f..bca13ab4 100644 --- a/di/config/config.md +++ b/di/config/config.md @@ -42,9 +42,8 @@ caller first — di.config does no wrapping: `` `info`warn`error!({[c;m]inst[`in |---|---|---| | `init` | `init[deps]` | Wire the injected logger. `deps` is a dict with a required `log` key — an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` functions. Errors immediately if the log dependency is missing or malformed. | | `loadcascade` | `loadcascade[dirs;names]` | Resolve the config cascade **name-major**: for each name (least→most specific) load `dir/{name}.q` from each directory in turn. `dirs` is a string path or list of paths (low→high priority); `names` is a symbol or symbol list (least→most specific). A more specific name wins over the directory layer; within a name the later directory overrides. Missing files are skipped (logged at *info*); already-loaded files are de-duplicated. Returns the flat list of constructed paths, in load order. | -| `overrideconfig` | `overrideconfig[params]` | Apply command-line-style overrides after the cascade. `params` is a dict keyed by variable name (symbol) with string (or list-of-string) values, each parsed into that variable's **existing** type. Only already-defined, basic-typed variables can be overridden; undefined names, non-basic types, and unparseable values are logged and skipped. Returns the list of variables actually overridden. | -| `get` | `get[namespace;key;default]` | Query the resolved config store: return the value of `.{namespace}.{key}`, or `default` if unset. `namespace` and `key` are **bare** symbols (no leading dot), e.g. `get[\`rdb;\`subscribeto;\`]`. (`get` is a reserved word, so the implementation function is named `getcfg` and exported under the `get` key.) | -| `getmodule` | `getmodule[namespace]` | Return a namespace's whole resolved config as a bare-keyed value dict — the slice di.torq passes to a module's `init`. `namespace` is a bare symbol; an unconfigured namespace yields an empty dict. | +| `overrideconfig` | `overrideconfig[params]` | Apply the **command-line-argument layer** on top of the file cascade — di.torq parses the process command line (`.Q.opt .z.x`) and calls this at startup, so launch-time flags (e.g. `-loglevel info`, `-tablelist trade quote`) win over the settings files. This is the automatic top layer of the cascade, **not** an interactive/by-hand step. `params` is a dict keyed by variable name (symbol) with string (or list-of-string) values, each parsed into that variable's **existing** type. Only already-defined, basic-typed variables can be overridden; undefined names, non-basic types, and unparseable values are logged and skipped. Returns the list of variables actually overridden. | +| `getmodule` | `getmodule[namespace]` | Return a namespace's whole resolved config as a bare-keyed value dict — the slice di.torq passes to a module's `init`. `namespace` is a bare symbol; an unconfigured namespace yields an empty dict. Callers read a single setting by indexing the returned dict inline (e.g. `getmodule[\`rdb]\`subscribeto`). | Internal helpers — `loadconfig` (the per-file loader behind `loadcascade`), `raiseerror`, `applyoverride`, and the load-tracking state — are deliberately not exported. @@ -58,12 +57,14 @@ config.overrideconfig[`.myproc.enabled`.myproc.rows!(enlist"1";enlist"5000")] / then partition per module and hand each slice to that module's init: rdbcfg:config.getmodule[`rdb] / -> `subscribeto`hdbtypes!(...) etc. rdb.init[rdbcfg;`log`timer!(logdep;timerdep)] -/ or read a single value with a fallback: -config.get[`rdb;`subscribeto;`trade`quote] +/ read a single setting by indexing the slice inline (configs are booted with defaults, +/ so the key is always present — no fallback needed): +rdbcfg`subscribeto ``` -`overrideconfig` runs after `loadcascade` so command-line values win over file config. -di.config is deliberately generic — it does not read environment variables or process +`overrideconfig` runs after `loadcascade` so command-line values win over file config — it is the +top (command-line) layer of the cascade, applied automatically by di.torq at startup, not a manual +step. di.config is deliberately generic — it does not read environment variables or process identity itself; the caller (di.torq) supplies the settings directories, the ordered name sequence, and the override params, then uses `getmodule` to partition the resolved store per module. @@ -72,10 +73,10 @@ store per module. `loadcascade` loads settings `.q` files that assign into root namespaces (`.rdb.subscribeto:...`); those resolved namespaces **are** the config store, and -`get`/`getmodule` query them (so anything `overrideconfig` changes is reflected too). +`getmodule` queries them (so anything `overrideconfig` changes is reflected too). Additional config sources (env vars, k8s config maps) are out of scope for v1 but would -plug in by populating the same namespaces before the store is queried — the `get`/ -`getmodule` contract stays unchanged. See the EXTENSION POINT comment in `config.q`. +plug in by populating the same namespaces before the store is queried — the `getmodule` +contract stays unchanged. See the EXTENSION POINT comment in `config.q`. > **Note:** avoid `-` in config file paths — a source-level backtick symbol literal > containing `-` parses as subtraction (`` `:/a-b.q `` → `` `:/a `` `- b.q`), not one @@ -101,9 +102,14 @@ None — `di.config` is a standalone module. `{procname}` files won't exist). - **Dedup / idempotency.** `loadconfig` tracks loaded paths in `.z.m.loaded` and skips re-loads, so the cascade is safe to re-run and `init` is safe to call again. -- **`overrideconfig` deviates from TorQ deliberately.** Per-variable problems (undefined name, - non-basic type, value that won't parse to the target type) are logged and skipped — a single bad - override never aborts the batch and a null is never written into config. +- **`overrideconfig` is the command-line layer, applied by di.torq — not a manual step.** It mirrors + TorQ's `overrideconfig`/`override[]` (`torq.q`), which TorQ runs at startup off `.Q.opt .z.x` so + launch-time flags beat the settings files. In the modular world di.torq owns that command-line parse + and calls this after `loadcascade`; di.config just does the type-aware apply (which is why it stays + env-free and never reads `.z.x` itself). It *deviates from TorQ deliberately* on error handling: + per-variable problems (undefined name, non-basic type, value that won't parse to the target type) + are logged and skipped — a single bad override never aborts the batch and a null is never written + into config. - **Entry point is `loadcascade[dirs;names]`, NOT a zero-arg `load[]`.** The Plan sketches `di.config.load[]`, but that predates the integration design. Config loading happens once at startup, so a blank call buys nothing and would force the cascade inputs into hidden `.z.m` state @@ -112,10 +118,17 @@ None — `di.config` is a standalone module. `load` is a reserved word, so the Plan's literal name is unusable anyway.) If di.torq ever wants a blank trigger it's a one-line wrapper on its side — not di.config's concern. - **Config store is live namespace globals, not an explicit internal store (1a, deliberate).** Settings - files assign globals when executed, so those namespaces *are* the store; `get`/`getmodule` read them. + files assign globals when executed, so those namespaces *are* the store; `getmodule` reads them. An explicit store with provenance / cross-source precedence (1b) is deferred until a second config - source (env, k8s) exists — the `get`/`getmodule` contract is storage-independent, so that swap won't + source (env, k8s) exists — the `getmodule` contract is storage-independent, so that swap won't touch consumers. See the `EXTENSION POINT` comment in `config.q`. +- **No single-key getter — `getmodule` is the only query.** An earlier draft exported a + `get[namespace;key;default]` convenience. It was removed: `getmodule` already returns the whole + namespace as a dict, and the intended flow fetches that slice once (di.torq → each module's `init`) + and indexes it inline (`rdbcfg\`subscribeto`), so a per-key getter added a second query path for no + gain. The `default` fallback was dropped with it — every config is booted with a default, so a + requested key is always present and a fallback arg is dead weight. A missing key is now a caller + bug (surfaced as a q null on the inline index), not something the config layer silently papers over. Open gaps / not implemented: @@ -129,9 +142,10 @@ Open gaps / not implemented: wired end-to-end with no code change (`init`/`loadcascade`/`loadconfig` all emit through it). The committed kx.log-wrapper test proves the same binary-`{[c;m]}`-dict contract `di.log` satisfies. Add a `di.log` integration test once `di.log` merges. -- **Logger storage follows the skill's single-dict pattern** (`.z.m.log[\`info][…]`), which diverges - from `consistency.md`'s three-flat-var example (`.z.m.loginfo`/`…`). `di.log` is now built and the - single-dict pattern is the directed one; `consistency.md` still needs reconciling to match. +- **Logging uses the three-flat-var convention** — `.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`, + called as `.z.m.loginfo[\`ctx;"msg"]` — matching `consistency.md` and `di.compression`. The injected + `log` value is still the same `` `info`warn`error `` dict; `init` just fans it out into the three + module-local vars. ## Tests @@ -146,6 +160,15 @@ di.config is complete for v1 in isolation. The items below are intentionally def each is keyed to the module or milestone that unblocks it. Do these **when the trigger lands**, not before. +### ⚠️ Logging call convention — NOT set in stone (ask before changing) + +di.config uses the **three-flat-var** convention (`.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`), +matching `consistency.md` and `di.compression`. But the project hasn't finalised this versus the +**single-dict** form (`.z.m.log[\`info][…]`) that an earlier skill revision directed. **Before +changing di.config's logging — or wiring logging in another module — stop and ask the user which +convention is authoritative.** If the project settles on single-dict, switching di.config back is +mechanical (storage + call sites only; the injected input contract is identical either way). + ### When `di.log` merges to main - **Add a committed `di.log` integration test.** Today `di.log` lives on the @@ -182,5 +205,5 @@ lands**, not before. - **Implement the 1b explicit store.** Replace the live-namespace-globals store with an internal store carrying provenance / cross-source precedence, behind the unchanged - `get`/`getmodule` contract (so consumers don't change). See the `EXTENSION POINT` + `getmodule` contract (so consumers don't change). See the `EXTENSION POINT` comment in `config.q`. diff --git a/di/config/config.q b/di/config/config.q index ec93c443..872a9495 100644 --- a/di/config/config.q +++ b/di/config/config.q @@ -6,14 +6,14 @@ / public api: / loadcascade - resolve the full cascade over dirs x names (name-major) / overrideconfig - apply command-line-style overrides after the cascade -/ get / getmodule - query the resolved config store (di.torq partitions per -/ module with getmodule and injects each slice via that module's init) +/ getmodule - query the resolved config store as a per-namespace dict; di.torq +/ partitions config per module with it and injects each slice via that module's init / internal helper loadconfig handles per-file loading behind loadcascade. raiseerror:{[ctx;msg] / internal - log an error under ctx then signal it, so failures are observable / in the log as well as thrown to the caller. - .z.m.log[`error][ctx;msg]; + .z.m.logerr[ctx;msg]; '"di.config: ",string[ctx],": ",msg; }; @@ -34,9 +34,11 @@ init:{[deps] '"di.config: log value must be a dict; pass `info`warn`error functions"]; if[not all (`info`warn`error) in key deps`log; '"di.config: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; - .z.m.log:deps`log; + .z.m.loginfo:deps[`log]`info; + .z.m.logwarn:deps[`log]`warn; + .z.m.logerr:deps[`log]`error; .z.m.loaded:enlist""; - .z.m.log[`info][`init;"di.config initialised"]; + .z.m.loginfo[`init;"di.config initialised"]; }; loadconfig:{[dir;name] @@ -52,14 +54,14 @@ loadconfig:{[dir;name] ]; file:dir,"/",string[name],".q"; if[file in .z.m.loaded; - .z.m.log[`info][`loadconfig;"already loaded ",file]; + .z.m.loginfo[`loadconfig;"already loaded ",file]; :file; ]; if[()~key hsym `$file; - .z.m.log[`info][`loadconfig;"no config file (skipping): ",file]; + .z.m.loginfo[`loadconfig;"no config file (skipping): ",file]; :file; ]; - .z.m.log[`info][`loadconfig;"loading ",file]; + .z.m.loginfo[`loadconfig;"loading ",file]; .[system;enlist"l ",file;{[f;e] raiseerror[`loadconfig;"failed to load ",f,": ",e]}[file;]]; .z.m.loaded,:enlist file; :file; @@ -87,7 +89,7 @@ loadcascade:{[dirs;names] if[not 11h=abs type names; raiseerror[`loadcascade;"names must be a symbol or a symbol list"]; ]; - .z.m.log[`info][`loadcascade;"resolving cascade: ",(string count dirs)," dir(s) x ",(string count names)," name(s)"]; + .z.m.loginfo[`loadcascade;"resolving cascade: ",(string count dirs)," dir(s) x ",(string count names)," name(s)"]; :raze {[ds;nm] loadconfig[;nm] each ds}[dirs;] each names; }; @@ -98,17 +100,17 @@ applyoverride:{[name;raw] / must already exist - its current type drives the parse. t:type value name; if[not (abs t) within (1;-1+count .Q.t); - .z.m.log[`error][`overrideconfig;"cannot override ",(string name),": not a basic type"]; + .z.m.logerr[`overrideconfig;"cannot override ",(string name),": not a basic type"]; :0b; ]; raw:$[10h=type raw;enlist raw;raw]; vals:(upper .Q.t abs t)$'raw; if[t<0;vals:first vals]; if[any null vals; - .z.m.log[`error][`overrideconfig;"cannot override ",(string name),": value did not parse"]; + .z.m.logerr[`overrideconfig;"cannot override ",(string name),": value did not parse"]; :0b; ]; - .z.m.log[`info][`overrideconfig;"setting ",(string name)," to ",-3!vals]; + .z.m.loginfo[`overrideconfig;"setting ",(string name)," to ",-3!vals]; set[name;vals]; :1b; }; @@ -132,7 +134,7 @@ overrideconfig:{[params] defined:vars where {@[{value x;1b};x;0b]} each vars; undefined:vars except defined; if[count undefined; - .z.m.log[`warn][`overrideconfig;"skipping undefined variable(s): ",", " sv string undefined]; + .z.m.logwarn[`overrideconfig;"skipping undefined variable(s): ",", " sv string undefined]; ]; applied:{[params;v] applyoverride[v;params v]}[params;] each defined; :defined where applied; @@ -140,31 +142,15 @@ overrideconfig:{[params] / --- queryable config store --- / the store is the set of root namespaces populated by the settings files that -/ loadcascade loads (plus anything overrideconfig changes). getcfg/getmodule -/ query it; di.torq uses getmodule to partition config per module and pass each -/ slice to that module's init. +/ loadcascade loads (plus anything overrideconfig changes). getmodule queries it; +/ di.torq uses getmodule to partition config per module and pass each slice to that +/ module's init. / EXTENSION POINT (out of scope for v1, flagged per the modularisation plan): / additional config sources - environment variables, k8s config maps, external / key-value stores - would plug in by populating the same root namespaces before / the store is queried. keep source reading (loadcascade et al.) separate from / querying (below) so a new source is an additive step, not a change to the -/ get/getmodule contract. - -getcfg:{[ns;k;dflt] - / query the resolved config store: return the value of the loaded config - / variable .{ns}.{k}, or dflt if it is not set. ns and k are bare symbols with - / no leading dot (e.g. getcfg[`rdb;`subscribeto;`]). exported under the `get` - / key. NB: params are ns/k/dflt because key and default are reserved words - - / used as parameter names they throw 'match when the function is called (and - / neither is listed in .Q.res). - if[not -11h=type ns; - raiseerror[`get;"namespace must be a symbol (no leading dot)"]; - ]; - if[not -11h=type k; - raiseerror[`get;"key must be a symbol"]; - ]; - :@[value;`$".",(string ns),".",string k;dflt]; - }; +/ getmodule contract. getmodule:{[namespace] / return all resolved config for a namespace as a bare-keyed value dict, for diff --git a/di/config/init.q b/di/config/init.q index ae1d533f..d5534dc9 100644 --- a/di/config/init.q +++ b/di/config/init.q @@ -1,5 +1,3 @@ / configuration loading and cascade resolution for the modular torq world. -/ note: `get` is a q reserved word, so it cannot be a top-level name; the query -/ function is defined as `getcfg` in config.q and exported under the `get` key. \l ::config.q -export:([init;loadcascade;overrideconfig;getmodule]),(enlist`get)!enlist getcfg +export:([init;loadcascade;overrideconfig;getmodule]) diff --git a/di/config/test.csv b/di/config/test.csv index 6433060a..9655d6e1 100644 --- a/di/config/test.csv +++ b/di/config/test.csv @@ -46,18 +46,14 @@ true,0,0,q,"any (exec msg from logtab) like ""*not a basic type*""",1,1,non-basi true,0,0,q,0=count cfg.overrideconfig[()!()],1,1,empty params is a no-op fail,0,0,q,cfg.overrideconfig[42],1,1,overrideconfig rejects a non-dict fail,0,0,q,"cfg.overrideconfig[enlist[1]!enlist enlist""x""]",1,1,overrideconfig rejects non-symbol keys -comment,,,,,,,get / getmodule - queryable config store +comment,,,,,,,getmodule - queryable config store run,0,0,q,"cfg.loadcascade[""/tmp/diconfigstore"";`default]",1,1,load a settings file into the store -true,0,0,q,`trade`quote~cfg.get[`diconfigq;`subscribeto;`],1,1,get returns a loaded config value -true,0,0,q,42~cfg.get[`diconfigq;`missing;42],1,1,get returns the default for an unset key -true,0,0,q,999~cfg.get[`nosuchns;`x;999],1,1,get returns the default for an unknown namespace true,0,0,q,2=count cfg.getmodule[`diconfigq],1,1,getmodule returns the whole namespace slice true,0,0,q,`trade`quote~cfg.getmodule[`diconfigq]`subscribeto,1,1,getmodule slice carries the values true,0,0,q,100~cfg.getmodule[`diconfigq]`rows,1,1,getmodule slice carries the second value true,0,0,q,0=count cfg.getmodule[`nosuchns],1,1,getmodule returns an empty dict for an unknown namespace run,0,0,q,"cfg.overrideconfig[enlist[`.diconfigq.rows]!enlist enlist""500""]",1,1,override a stored value -true,0,0,q,500~cfg.get[`diconfigq;`rows;0],1,1,get reflects overrideconfig changes -fail,0,0,q,cfg.get[42;`k;0],1,1,get rejects a non-symbol namespace +true,0,0,q,500~cfg.getmodule[`diconfigq]`rows,1,1,getmodule reflects overrideconfig changes fail,0,0,q,cfg.getmodule[42],1,1,getmodule rejects a non-symbol namespace comment,,,,,,,logger emission - real logger via a caller-side binary wrapper (must run last; re-inits cfg) run,0,0,q,kxlogger:use`kx.log,1,1,load the kx.log module From bc5fa3c82548f01055fa37c2acfa0f70fa7f8426 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Mon, 13 Jul 2026 18:32:00 +0100 Subject: [PATCH 09/12] removing accidentally added module --- di/asyncdispatch/asyncdispatch.md | 263 ------------------------ di/asyncdispatch/asyncdispatch.q | 331 ------------------------------ di/asyncdispatch/init.q | 11 - di/asyncdispatch/test.csv | 190 ----------------- 4 files changed, 795 deletions(-) delete mode 100644 di/asyncdispatch/asyncdispatch.md delete mode 100644 di/asyncdispatch/asyncdispatch.q delete mode 100644 di/asyncdispatch/init.q delete mode 100644 di/asyncdispatch/test.csv diff --git a/di/asyncdispatch/asyncdispatch.md b/di/asyncdispatch/asyncdispatch.md deleted file mode 100644 index c11d5c80..00000000 --- a/di/asyncdispatch/asyncdispatch.md +++ /dev/null @@ -1,263 +0,0 @@ -# di.asyncdispatch - -Async scatter-gather query coordinator for kdb-x gateway processes. Queues client queries, dispatches them to available backend processes by servertype, collects per-server results, applies a join function, and replies to the client — with timeout management and correct error propagation if a backend disconnects mid-query. - -Routing (deciding which servertypes satisfy a query) is `di.serverselect`'s responsibility, but **`di.serverselect` is not a dependency** — this module dispatches a resolved servertype list to whatever backends its *server source* reports. That source is pluggable with a default: by default it is asyncdispatch's own registry (populate it with `addserver`), or you point it at `di.serverselect`'s output via `setavailableservers`. Either works standalone; neither is required. - ---- - -## Features - -- Queue and dispatch async client queries to multiple backend process types simultaneously (scatter-gather) -- Collect per-server results and apply a caller-supplied join function once all slots are filled -- Timeout expired queries with configurable per-query timespan via `checktimeout` -- Handle backend disconnects mid-query — errors in-flight queries and queued queries that can no longer be satisfied -- Track connected clients and clean up orphaned queries on client disconnect -- Support synchronous deferred response mode (`-30!`) alongside the default async mode -- Pluggable server source with a default — dispatch against the built-in registry (`addserver`) by default, or point it at `di.serverselect`'s output (or any source) via `setavailableservers`; `di.serverselect` is a composable option, never a dependency -- Accept fully pluggable scheduler (`setgetnextqueryid`), routing (`setavailableservers`), reply formatter (`setformatresponse`), and callback symbols (`setcallbacks`) — swap without touching core dispatch logic -- Detect and normalise `kx.log` instances automatically so callers can pass a logger directly without manual wrapping - ---- - -## Dependencies - -| Dependency | Key | Required | Description | -|---|---|---|---| -| logger | `` `log `` | yes | `info`, `warn`, `error` — each binary `{[c;m]}` where `c` is a symbol context and `m` is a string | - -The `log` dependency must be passed to `init` inside the `deps` dict. The module throws immediately if it is absent or missing any of the three required keys. All three are required since the module calls `info`, `warn`, and `error`. - -> **`di.serverselect` is not a dependency.** asyncdispatch never imports or calls it. The server source is pluggable (`setavailableservers`) and defaults to the built-in registry (`addserver`), so you run standalone out of the box; to dispatch against serverselect's live view instead, inject its `getservers` output as the source (see `setavailableservers`). No registry copying, no dependency. - -A `kx.log` instance can be passed directly — the module normalises monadic functions to the binary `{[c;m]}` contract automatically via `normlog`. Context is embedded in the output as `"context: message"`: - -```q -kxlog:use`kx.log -ad:use`di.asyncdispatch - -// minimal -ad.init[enlist[`log]!enlist kxlog.createLog[]] - -// with config overrides -ad.init[`log`querykeeptime`synccallsallowed!(kxlog.createLog[];0D01:00;1b)] -``` - ---- - -## Initialisation - -`init[deps]` takes a single dictionary combining the `log` dependency with any configuration overrides. - -| Key | Required | Default | Description | -|---|---|---|---| -| `` `log `` | yes | — | Binary log dep — `info`, `warn`, `error` functions each `{[c;m]}` | -| `` `errorprefix `` | no | `"error: "` | String prepended to all error messages sent back to clients | -| `` `querykeeptime `` | no | `0D00:30` | How long `removequeries` retains finished query rows | -| `` `clearinactivetime `` | no | `0D01:00` | How long `removeinactive` retains disconnected server rows | -| `` `synccallsallowed `` | no | `0b` | Whether `execquery[...;1b]` (deferred sync mode) is permitted | - -Housekeeping — `checktimeout`, `removequeries`, `removeinactive`, and `removeclients` — is the caller's responsibility. Wire them into your gateway's timer after `init`. The configured default age parameters are accessible as `querykeeptime` and `clearinactivetime` via module state. - ---- - -## Exported Functions - -### `init[deps]` -Initialise the module. Validates the log dependency and applies config overrides. -```q -ad.init[enlist[`log]!enlist logdep] -``` - -### `addserver[handle;servertype]` -Register a backend connection into the **built-in (default) server source**. `handle`: open int handle. `servertype`: symbol identifying the process type (e.g. `` `rdb ``, `` `hdb ``). Use this when asyncdispatch owns the server list; to source servers from `di.serverselect` instead, leave `addserver` unused and inject via `setavailableservers`. -```q -ad.addserver[hopen`:backend1:5001;`rdb] -``` - -### `removeserverhandle[handle]` -Call from `.z.pc` for **backend** handles. Errors any in-flight or queued queries that depended on this server, marks the server `active:0b`, and triggers `runnextquery`. -```q -.z.pc:{ad.removeserverhandle[.z.w];ad.removeclienthandle[.z.w]} -``` - -### `addclientdetails[handle]` -Record client identity on connect. Call from `.z.po`. -```q -.z.po:{ad.addclientdetails[.z.w]} -``` - -### `removeclienthandle[handle]` -On client disconnect, mark their pending queries errored so result slots are not leaked. Call from `.z.pc`. -```q -.z.pc:{ad.removeserverhandle[.z.w];ad.removeclienthandle[.z.w]} -``` - -### `addserverresult[qid;data]` -Called when a backend posts back a successful result. Fills the result slot, frees the server, triggers `runnextquery`, and — once all slots for the query are received — applies the join function and replies to the client. -```q -// called by serverexecute on the backend; not typically called directly -``` - -### `addservererror[qid;err]` -Called when a backend posts back an error. Sends the error to the client and finishes the query. -```q -// called by serverexecute on the backend; not typically called directly -``` - -### `execquery[query;servertype;join;postback;timeout;sync]` -Public entry point. Validates sync constraints, enqueues the query, and triggers dispatch. - -| Argument | Type | Description | -|---|---|---| -| `query` | any | Payload passed to `value` on the backend | -| `servertype` | symbol list | One symbol per required backend type, e.g. `` enlist`rdb `` | -| `join` | function | Applied to the list of per-server results once all are received | -| `postback` | list or `()` | `()` for a plain reply; `(function;extra_args...)` to wrap the reply | -| `timeout` | timespan | `0Wn` for no timeout | -| `sync` | boolean | `1b` for deferred sync via `-30!`; `0b` for async | - -```q -ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;0b] -``` - -### `execqueryto[replyto;query;servertype;join;postback;timeout;sync]` -Variant of `execquery` with an explicit reply target. Pass `replyto:0Ni` to invoke the postback locally via `value` instead of IPC-sending to a handle. Pass any valid int handle to route the reply to a specific process regardless of `.z.w`. Used when the caller is in the same process as the gateway (e.g. `di.dataaccess` dispatching shards). - -| Argument | Type | Description | -|---|---|---| -| `replyto` | int | `0Ni` for local in-process invocation; any int handle to IPC-send to a specific target | -| `query` | any | Payload passed to `value` on the backend | -| `servertype` | symbol list | One symbol per required backend type | -| `join` | function | Applied to the list of per-server results once all are received | -| `postback` | list | Required when `replyto` is `0Ni` — `(function;extra_args...)` invoked locally; `()` permitted for non-local targets | -| `timeout` | timespan | `0Wn` for no timeout | -| `sync` | boolean | Must be `0b` when `replyto` is `0Ni`; sync not supported for local invocation | - -```q -// di.dataaccess wiring: shard result delivered locally to da.shardresult -ad.execqueryto[0Ni;shardquery;servertypes;raze;(`da.shardresult;reqid);0Wn;0b] -``` - -### `checktimeout[]` -Scan the queue for queries past their timeout, send a timeout error to each client, and mark them complete. Wire into your gateway's timer — every few seconds is typical. -```q -// in gateway timer -timer.addjob.default[`asyncdispatch.checktimeout;{ad.checktimeout[]};();5i;1] -``` - -### `removequeries[age]` -Purge completed `queryqueue` rows older than `age`. Prevents unbounded growth. -```q -// default age is querykeeptime (0D00:30) -timer.addjob.default[`asyncdispatch.removequeries;{ad.removequeries[0D00:30]};();300i;1] -``` - -### `removeinactive[age]` -Purge `servers` rows for backends that have been disconnected longer than `age`. Prevents unbounded growth. -```q -// default age is clearinactivetime (0D01:00) -timer.addjob.default[`asyncdispatch.removeinactive;{ad.removeinactive[0D01:00]};();300i;1] -``` - -### `removeclients[age]` -Purge `clients` rows older than `age`. The `clients` table is appended to on every client connect (`addclientdetails`) for audit and is not otherwise pruned, so wire this into the timer to prevent unbounded growth. Choose `age` to match your audit retention requirements. -```q -// retain one day of client audit history -timer.addjob.default[`asyncdispatch.removeclients;{ad.removeclients[1D]};();300i;1] -``` - -### `setformatresponse[f]` -Override the reply formatter applied before a result or error is sent to the client. `f` must be `{[status;sync;result]}`. Note: `formatresponse` is only applied on the remote IPC path — queries dispatched via `execqueryto` with `replyto:0Ni` invoke the postback directly and bypass this formatter. -```q -ad.setformatresponse[{[status;sync;result]result}] -``` - -### `setcallbacks[resfn;errfn]` -Update the callback symbols used by `serverexecute`. Required when the module is mounted under a non-default namespace — point these at wherever `addserverresult` and `addservererror` are visible on the backend processes. -```q -ad.setcallbacks[`.gw.dispatch.addserverresult;`.gw.dispatch.addservererror] -``` - -### `setavailableservers[f]` -Replace the **server source** — the function dispatch uses to find backends. `f` is `{[excludeinuse]}` and must return a table with `handle` and `servertype` columns. This is the seam for running against `di.serverselect` (or any external source) **without a dependency and without copying its registry**: by default the source reads asyncdispatch's own registry (`addserver`), and you override it to read serverselect instead. -```q -// default (built-in registry) — equivalent to not calling this at all -ad.setavailableservers[{[eu] $[eu; select from servers where active, not inuse; select from servers where active]}] - -// compose with di.serverselect (not a dependency — just its output as the source) -srvsel:use`di.serverselect -ad.setavailableservers[{[eu] select handle, servertype from srvsel.getservers[`servertype;`;()!()]}] -``` -Result routing does **not** depend on the source — a returning handle is matched to its servertype from the query's own dispatch record — so an injected source needs no registration in asyncdispatch. Note `inuse` throttling applies only to the built-in registry; an external source is expected to do its own idle/selection (e.g. serverselect's `roundrobin`), and the `excludeinuse` flag is a hint such a source may ignore. - -### `setgetnextqueryid[f]` -Inject a custom scheduling strategy. `f` must be niladic and return a 0- or 1-row table with the `queryqueue` schema. -```q -// priority queue example - highest-priority query first -ad.setgetnextqueryid[{1 sublist `priority xdesc 0!select from .z.m.queryqueue where null returntime}] -``` - ---- - -## Usage Example - -```q -kxlog:use`kx.log -timer:use`di.timer -timer.init[()!()] - -ad:use`di.asyncdispatch -ad.init[enlist[`log]!enlist kxlog.createLog[]] - -// point backends' callbacks at this module's mount point on this process -ad.setcallbacks[`ad.addserverresult;`ad.addservererror] - -// register backend connections as they connect -ad.addserver[hopen`:backend1:5001;`rdb] -ad.addserver[hopen`:backend2:5002;`hdb] - -// wire client and server connection/disconnection handlers -.z.po:{ad.addclientdetails[.z.w]} -.z.pc:{ad.removeserverhandle[.z.w];ad.removeclienthandle[.z.w]} - -// wire housekeeping into the gateway timer -timer.addjob.default[`asyncdispatch.checktimeout;{ad.checktimeout[]};();5i;1] -timer.addjob.default[`asyncdispatch.removequeries;{ad.removequeries[0D00:30]};();300i;1] -timer.addjob.default[`asyncdispatch.removeinactive;{ad.removeinactive[0D01:00]};();300i;1] -timer.addjob.default[`asyncdispatch.removeclients;{ad.removeclients[1D]};();300i;1] - -// a client calls this asynchronously: -// execquery dispatches to rdb and hdb in parallel, razes results, replies to client -ad.execquery[("select count i by sym from trade";"select count i by sym from trade");`rdb`hdb;raze;();0Wn;0b] - -// synchronous deferred mode (requires synccallsallowed:1b in deps) -ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;1b] -``` - ---- - -## Running Tests - -```q -k4unit:use`di.k4unit -k4unit.moduletest`di.asyncdispatch -``` - -116 tests. Requires a `q` binary in `PATH` — the test suite starts two real backend processes on dynamically selected free ports and exercises the full dispatch lifecycle over live IPC connections. No TorQ installation or special libraries required. Covers: server registry, FIFO scheduling, full IPC round-trip with join and reply, backend error path via real IPC callback (`addservererror`), join-failure path (throwing join function flagged as error), postback wrapping (`tosend` tuple construction), multi-servertype scatter-gather with two backends dispatched in parallel and results joined, checktimeout, removequeries, removeinactive, removeclients audit-row purge, removeserverhandle with orphaned query cleanup, client tracking, in-flight server release on client disconnect, all pluggable hook setters, and local in-process reply path via `execqueryto` (success, backend error, timeout, and non-null explicit replyto variants). - ---- - -## Notes - -- Housekeeping (`checktimeout`, `removequeries`, `removeinactive`, `removeclients`) is the caller's responsibility — the gateway process already has a timer running and is better placed to decide intervals. Wire all four after `init`; see the usage example above -- When `checktimeout` fires for a query that is already in-flight (dispatched to a backend but not yet answered), the client receives the timeout error and the query is marked done, but the backend that received the dispatch remains `inuse:1b` until it eventually replies. A slow-but-alive backend holds its dispatch slot until `addserverresult` or `addservererror` fires; a permanently hung backend holds it until `removeserverhandle` fires on disconnect. Size backend pools and timeouts with this in mind — a run of timeouts against a stuck backend will not free its slot on timeout alone -- `.z.M.` is used for in-place mutation of tables (`upsert`, `insert`, `update from`, `delete from`) and `.z.m.:value` for whole-variable reassignment — the same convention used by `di.cache` -- Module globals referenced inside q-sql expressions (WHERE conditions, UPDATE SET values) must use the `.z.m.varname` form since q-sql evaluates column expressions in the calling context rather than the module namespace -- `servertype` in `queryqueue` and `addquery` is a list of servertype symbols — one per required backend type. Pass `` enlist`rdb `` for single-server queries, `` `rdb`hdb `` for scatter-gather across two types -- `setcallbacks` must be called before any queries are dispatched if the module is mounted under a non-default path — `serverexecute` reads `resultcallback` and `errorcallback` by bare name on the backend process and posts back to whatever symbols they resolve to -- This module opens and accepts no connections itself — `addserver` and the `.z.po`/`.z.pc` wiring are the consumer's responsibility, keeping the module dependency-free and testable in-process -- All three log keys (`info`, `warn`, `error`) are required — the module calls `info` on server/client connect and init, `warn` on disconnect and timeout, and `error` on backend error and join failure -- `execqueryto` with `replyto:0Ni` invokes the postback locally via `value` — the postback head symbol must be mount-qualified (e.g. `` `da.shardresult ``) since bare exported names are not globals. This is the same resolution mechanism used by `setcallbacks` -- Local queries store `clienth:0Ni` and are not matched by `removeclienthandle` — the in-process caller owns disconnect cleanup for its own requests -- `setformatresponse` overrides only apply to the remote IPC path — local invocation via `execqueryto[0Ni;...]` calls the postback directly without applying `formatresponse`. The default `formatresponse` is a pass-through for async, so this is transparent by default; consumers that override it should not combine that with local invocation diff --git a/di/asyncdispatch/asyncdispatch.q b/di/asyncdispatch/asyncdispatch.q deleted file mode 100644 index c2f812b5..00000000 --- a/di/asyncdispatch/asyncdispatch.q +++ /dev/null @@ -1,331 +0,0 @@ -// di.asyncdispatch - async scatter-gather query coordinator -// queues queries, dispatches to available backends, collects results per server, -// applies a join function, and replies to the client -// routing (which servers satisfy a query) is di.serverselect's responsibility -// the log dependency is required - init errors immediately if absent or malformed -// log functions are binary {[c;m]} where c is a symbol context and m is a string - -// ============================================================ -// module state and defaults -// ============================================================ - -// error prefix prepended to all error strings returned to clients -errorprefix:"error: "; - -// how long completed queries are kept in queryqueue before being purged -querykeeptime:0D00:30; - -// how long disconnected servers are kept in the servers table before being removed -clearinactivetime:0D01:00; - -// whether synchronous calls via -30! are permitted -synccallsallowed:0b; - -// injectable clock - replaced in tests to control time without sleeping -cp:{.z.p}; - -// symbols backend servers call back via - stored as symbols so names survive IPC serialisation -resultcallback:`addserverresult; -errorcallback:`addservererror; - -// reply formatting - sync errors must be signalled with ' so the client receives a trapped error -formatresponse:{[status;sync;result]$[not[status]and sync;'result;result]}; - -// scheduling strategy - pick oldest FIFO-eligible query by default -getnextqueryid:{ - avail:exec distinct servertype from availableservers 1b; - // 0! is required - select from a keyed table stays keyed in kdb-x, and runnextquery needs queryid via first - runnable:0!select from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, {all x in y}[;avail] each servertype; - 1 sublist select from runnable where time=min time - }; - -// server routing strategy - active and idle by default -availableservers:{[excludeinuse] - $[excludeinuse; - select from servers where active, not inuse; - select from servers where active] - }; - -// ============================================================ -// module tables -// ============================================================ - -// registered backend servers -servers:([handle:`u#`int$()] servertype:`symbol$(); inuse:`boolean$(); active:`boolean$(); disconnecttime:`timestamp$()); - -// pending and in-flight client queries -queryqueue:([queryid:`u#`long$()] time:`timestamp$(); clienth:`int$(); query:(); servertype:(); join:(); postback:(); timeout:`timespan$(); returntime:`timestamp$(); error:`boolean$(); sync:`boolean$(); local:`boolean$()); - -// connected client tracking -clients:([] time:`timestamp$(); clienth:`int$(); user:`symbol$(); ip:`int$(); host:`symbol$()); - -// per-query result accumulator: queryid -> (clienth; servertype!(handle;result;done)) -results:()!(); - -// auto-incrementing query id counter -queryid:0; - -// ============================================================ -// internal helpers -// ============================================================ - -normlog:{[logdict] - // detect kx.log instance by presence of kx.log-specific keys (getlvl, sinks, fmts) - // kx.log functions are monadic - wrap each into binary {[c;m]} and embed context in the message - // plain {[c;m]} log dicts (info`warn`error only) pass through unchanged - $[all `getlvl`sinks`fmts in key logdict; - `info`warn`error!( - {[fn;c;m] fn[string[c],": ",m]}[logdict`info;]; - {[fn;c;m] fn[string[c],": ",m]}[logdict`warn;]; - {[fn;c;m] fn[string[c],": ",m]}[logdict`error;]); - logdict] - }; - -sendclientreply:{[qid;result;status] - // deliver result or error to the client, handling sync vs async send and postback wrapping - // local path invokes value tosend directly - formatresponse is not applied (it is a pass-through - // for async by default; consumers that override setformatresponse should not use local invocation) - qd:queryqueue[qid]; - if[qd`error;:()]; - tosend:$[()~qd`postback;result;qd[`postback],enlist[qd`query],enlist result]; - $[qd`sync; - @[-30!;(qd`clienth;not status;$[status;formatresponse[1b;1b;result];result]);{}]; - $[qd`local; - @[value;tosend;{.z.m.log[`error][`asyncdispatch;"local postback failed: ",x]}]; - @[neg qd`clienth;formatresponse[status;0b;tosend];()]]]; - }; - -finishquery:{[qid;err] - // remove query from the live results accumulator and stamp its completion time - .z.m.results:(qid,())_results; - update error:err,returntime:.z.m.cp[] from .z.M.queryqueue where queryid in qid; - }; - -serverexecute:{[qid;query] - // runs on the backend - traps errors so a crash posts an error reply rather than dropping the result - res:@[{(0b;value x)};query;{(1b;"server ",(string .z.h),":",(string system"p"),": ",x)}]; - @[neg .z.w;$[res 0;(errorcallback;qid;res 1);(resultcallback;qid;res 1)]; - {@[neg .z.w;(errorcallback;x;"failed to return result: ",y);()]}[qid]]; - }; - -sendquerytoserver:{[qid;query;handles] - // fan the query out to all required handles and mark them in-use atomically - (neg handles,:())@\:(serverexecute;qid;query); - update inuse:1b from .z.M.servers where handle in handles; - }; - -runnextquery:{[] - // pick the next dispatchable query and fan out to one idle server per required servertype - // called after any state change that may unblock work - if[0=count torun:getnextqueryid[];:()]; - torun:first torun; - avail:exec first handle by servertype from availableservers 1b; - types:torun`servertype; - handles:avail types; - qid:torun`queryid; - slots:types!count[types]#enlist(0Ni;(::);0b); - slots[types;0]:handles; - // indexed assignment on bare results propagates through to .z.m in kdb-x (empirically verified) - results[qid]:(torun`clienth;slots); - sendquerytoserver[qid;torun`query;handles]; - }; - -addqueryto:{[query;servertype;join;postback;timeout;sync;replyto;local] - // enqueue a query with an explicit reply target - used by execqueryto for local in-process routing - .z.M.queryqueue upsert (queryid;.z.m.cp[];replyto;query;servertype;join;{$[11h=type x;enlist x;x]}postback;timeout;0Np;0b;sync;local); - .z.m.queryid:queryid+1; - }; - -addquery:{[query;servertype;join;postback;timeout;sync] - // enqueue a query without dispatching - caller must call runnextquery[] to trigger dispatch - addqueryto[query;servertype;join;postback;timeout;sync;.z.w;0b]; - }; - -removequeries:{[age] - // prevent queryqueue growing unboundedly - purge completed queries older than age - delete from .z.M.queryqueue where not null returntime, .z.m.cp[]>returntime+age; - }; - -removeinactive:{[age] - // prune stale disconnected-server rows to stop the servers table growing forever - delete from .z.M.servers where not active, .z.m.cp[]>disconnecttime+age; - }; - -removeclients:{[age] - // prune stale client audit rows to stop the clients table growing forever - // clients are recorded on every connect by addclientdetails and never removed otherwise - delete from .z.M.clients where .z.m.cp[]>time+age; - }; - -checktimeout:{[] - // periodic scan to error queries that have waited beyond their timeout - qids:exec queryid from .z.m.queryqueue where not timeout=0Wn, null returntime, .z.m.cp[]>time+timeout; - if[count qids; - .z.m.log[`warn][`asyncdispatch;"queries timed out: ",", " sv string qids]; - sendclientreply[;errorprefix,"query timed out";0b] each qids; - finishquery[qids;1b]]; - }; - -// ============================================================ -// public api -// ============================================================ - -setcp:{[f] - // replace the clock function - used in tests to control time without sleeping - .z.m.cp:f; - }; - -setformatresponse:{[f] - // override reply formatting - e.g. to wrap results in a standard envelope - .z.m.formatresponse:f; - }; - -setcallbacks:{[resfn;errfn] - // update callback symbols when module is mounted under a non-default namespace - .z.m.resultcallback:resfn; - .z.m.errorcallback:errfn; - }; - -setavailableservers:{[f] - // swap in a custom routing strategy without forking core dispatch - .z.m.availableservers:f; - }; - -setgetnextqueryid:{[f] - // inject a priority or custom scheduling strategy - .z.m.getnextqueryid:f; - }; - -addserver:{[h;st] - // register a backend handle and servertype so it becomes eligible for dispatch. - // this is the default (built-in) server source; to dispatch against di.serverselect's view instead, - // inject it via setavailableservers - no registration and no di.serverselect dependency required - .z.m.log[`info][`asyncdispatch;"server registered: ",string[st]," handle ",string h]; - .z.M.servers upsert (h;st;0b;1b;0Np); - }; - -removeserverhandle:{[serverh] - // on backend disconnect, error in-flight queries using that handle and queued queries - // that can no longer be satisfied - if[null st:first exec servertype from .z.m.servers where handle=serverh;:()]; - err:errorprefix,"backend ",string[st]," server disconnected"; - .z.m.log[`warn][`asyncdispatch;"backend disconnected: ",string st]; - // in-flight: queries where this handle was assigned to a slot - qids:where {[h;qid]h in value[.z.m.results[qid;1]][;0]}[serverh] each key .z.m.results; - sendclientreply[;err," during query";0b] each qids; - finishquery[qids;1b]; - // queued: queries that can no longer be satisfied by remaining active servers - activetypes:exec distinct servertype from .z.m.servers where active, handle<>serverh; - qids2:exec queryid from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, - not {all x in y}[;activetypes] each servertype; - sendclientreply[;err,", query cannot be satisfied";0b] each qids2; - finishquery[qids2;1b]; - update active:0b,disconnecttime:.z.m.cp[] from .z.M.servers where handle=serverh; - runnextquery[]; - }; - -addclientdetails:{[h] - // record client identity on connect for audit and orphan-query cleanup on disconnect - .z.m.log[`info][`asyncdispatch;"client connected: handle ",string h]; - .z.M.clients insert (.z.m.cp[];h;.z.u;.z.a;.z.h); - }; - -removeclienthandle:{[h] - // on client disconnect, mark their pending queries errored so result slots are not leaked - // free any servers in-flight for this client before removing result slots, then re-dispatch - // local queries store clienth:0Ni and are not matched here - the in-process caller owns cleanup for its own requests - .z.m.log[`info][`asyncdispatch;"client disconnected: handle ",string h]; - inflightqids:(exec queryid from .z.m.queryqueue where clienth=h, null returntime) inter key .z.m.results; - if[count inflightqids; - inflighthandles:distinct raze {value[.z.m.results[x;1]][;0]} each inflightqids; - update inuse:0b from .z.M.servers where handle in inflighthandles]; - update error:1b,returntime:.z.m.cp[] from .z.M.queryqueue where clienth=h, null returntime; - .z.m.results:(exec queryid from .z.m.queryqueue where clienth=h)_results; - runnextquery[]; - }; - -addserverresult:{[qid;data] - // fill one result slot - once all slots for a query are filled, run the join and reply - // bare indexed assignment on results propagates through to .z.m in kdb-x (empirically verified) - if[not qid in key results;:()]; - slots:results[qid;1]; - // map the responding handle to its servertype from THIS query's own dispatch record, not a server - // registry - so any server source (built-in or an injected di.serverselect view) works unchanged - st:first where .z.w=slots[;0]; - slots[st]:(.z.w;data;1b); - results[qid]:(results[qid;0];slots); - update inuse:0b from .z.M.servers where handle in .z.w; - runnextquery[]; - if[not qid in key results;:()]; - vals:value results[qid;1]; - if[not all vals[;2];:()]; - qd:queryqueue[qid]; - res:@[{(0b;x y)}[qd`join];vals[;1];{(1b;.z.m.errorprefix,"join failed: ",x)}]; - if[res 0;.z.m.log[`error][`asyncdispatch;"join failed for query ",string qid,": ",last res]]; - sendclientreply[qid;last res;not res 0]; - finishquery[qid;res 0]; - }; - -addservererror:{[qid;err] - // short-circuit a query on backend failure - free the server and notify the client - .z.m.log[`error][`asyncdispatch;"backend error for query ",string[qid],": ",err]; - sendclientreply[qid;errorprefix,err;0b]; - update inuse:0b from .z.M.servers where handle in .z.w; - runnextquery[]; - finishquery[qid;1b]; - }; - -execquery:{[query;servertype;join;postback;timeout;sync] - // public entry point - validate sync constraints then enqueue and kick dispatch - if[sync; - if[not ()~postback;.z.m.log[`warn][`asyncdispatch;"execquery: postback ignored for sync call"]]; - if[not synccallsallowed;'"syncexec: synchronous calls are not allowed"]; - if[not @[{-30!x;1b};(::);0b];'"syncexec: deferred response not supported on this connection"]; - .[{[q;s;j;t]addquery[q;s;j;();t;1b];runnextquery[]};(query;servertype;join;timeout);{-30!(.z.w;1b;x)}]; - :()]; - addquery[query;servertype;join;postback;timeout;0b]; - runnextquery[]; - }; - -execqueryto:{[replyto;query;servertype;join;postback;timeout;sync] - // in-process variant of execquery - replyto is 0Ni for local invocation via value - // postback must be non-empty when replyto is 0Ni - there is no handle to send a bare result to - // sync is not supported for local invocation - // mount-qualified postback symbols required for local invocation e.g. `da.shardresult not `shardresult - if[0Ni~replyto; - if[()~postback;'"di.asyncdispatch: local invocation requires a non-empty postback"]; - if[sync;'"di.asyncdispatch: local invocation does not support sync mode"]]; - local:0Ni~replyto; - addqueryto[query;servertype;join;postback;timeout;sync;replyto;local]; - runnextquery[]; - }; - -init:{[deps] - // initialise the asyncdispatch module - validate deps and apply config overrides - // deps: dict containing `log (required) plus optional config keys: - // `log - required: `info`warn`error!({[c;m]};{[c;m]};{[c;m]}) binary loggers - // `errorprefix - optional: string prefix for client error messages. default: "error: " - // `querykeeptime - optional: timespan to keep completed queries. default: 0D00:30 - // `clearinactivetime - optional: timespan to keep disconnected servers. default: 0D01:00 - // `synccallsallowed - optional: boolean, whether sync calls are permitted. default: 0b - // housekeeping (checktimeout; removequeries; removeinactive; removeclients) is the caller's - // responsibility - wire them into your timer after init; the exported defaults for age params are - // querykeeptime and clearinactivetime - // examples: - // ad.init[enlist[`log]!enlist logdep] - // ad.init[`log`querykeeptime!(logdep;0D01:00)] - if[99h<>type deps; - '"di.asyncdispatch: deps must be a dict with `log key"]; - if[not `log in key deps; - '"di.asyncdispatch: log dependency is required; pass `info`warn`error!(infofn;warnfn;errfn) keyed on `log"]; - if[99h<>type deps`log; - '"di.asyncdispatch: log value must be a dict; pass `info`warn`error functions"]; - if[not all `info`warn`error in key deps`log; - '"di.asyncdispatch: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; - .z.m.log:normlog deps`log; - if[`errorprefix in key deps; .z.m.errorprefix:deps`errorprefix]; - if[`querykeeptime in key deps; .z.m.querykeeptime:deps`querykeeptime]; - if[`clearinactivetime in key deps; .z.m.clearinactivetime:deps`clearinactivetime]; - if[`synccallsallowed in key deps; .z.m.synccallsallowed:deps`synccallsallowed]; - .z.m.log[`info][`asyncdispatch;"di.asyncdispatch initialised"]; - }; diff --git a/di/asyncdispatch/init.q b/di/asyncdispatch/init.q deleted file mode 100644 index 21200d8a..00000000 --- a/di/asyncdispatch/init.q +++ /dev/null @@ -1,11 +0,0 @@ -// asyncdispatch module - async scatter-gather query coordinator for multi-backend kdb+ gateway processes -\l ::asyncdispatch.q - -export:([ - setformatresponse;setcallbacks;setavailableservers;setgetnextqueryid; - addserver;removeserverhandle; - addclientdetails;removeclienthandle; - addserverresult;addservererror; - execquery;execqueryto; - checktimeout;removequeries;removeinactive;removeclients; - init]) diff --git a/di/asyncdispatch/test.csv b/di/asyncdispatch/test.csv deleted file mode 100644 index 41ba99a8..00000000 --- a/di/asyncdispatch/test.csv +++ /dev/null @@ -1,190 +0,0 @@ -action,ms,bytes,lang,code,repeat,minver,comment -before,0,0,q,p:first {x where {10h=type @[hopen;`$raze("::";string x);{x}]} each x}[19500+til 100],1,,find first free port in range 19500-19599 -before,0,0,q,p2:first {x where {10h=type @[hopen;`$raze("::";string x);{x}]} each x}[19600+til 100],1,,find second free port in range 19600-19699 -before,0,0,q,system raze("q -p ";string p;" -q &"),1,,start first backend process on free port -before,0,0,q,system raze("q -p ";string p2;" -q &"),1,,start second backend process on free port -before,0,0,q,system"sleep 1",1,,wait for both backends to initialise -before,0,0,q,ad:use`di.asyncdispatch,1,,load module -before,0,0,q,srv:{.m.di.0asyncdispatch.servers},1,,live servers table -before,0,0,q,qq:{.m.di.0asyncdispatch.queryqueue},1,,live queryqueue table -before,0,0,q,cl:{.m.di.0asyncdispatch.clients},1,,live clients table -before,0,0,q,res:{.m.di.0asyncdispatch.results},1,,live results dict -before,0,0,q,mocklog:`info`warn`error!3#enlist{[c;m]},1,,mock log dep - binary {[c;m]} no-op -before,0,0,q,ad.init[enlist[`log]!enlist mocklog],1,,initialise with mock logger -before,0,0,q,bh:hopen`$raze("::";string p),1,,open handle to backend -before,0,0,q,bh".m.di.0asyncdispatch.resultcallback:`.m.di.0asyncdispatch.addserverresult",1,,configure backend result callback to dispatcher -before,0,0,q,bh".m.di.0asyncdispatch.errorcallback:`.m.di.0asyncdispatch.addservererror",1,,configure backend error callback to dispatcher -before,0,0,q,ad.addserver[bh;`mock],1,,register first backend as mock server -before,0,0,q,bh2:hopen`$raze("::";string p2),1,,open handle to second backend -before,0,0,q,bh2".m.di.0asyncdispatch.resultcallback:`.m.di.0asyncdispatch.addserverresult",1,,configure second backend result callback -before,0,0,q,bh2".m.di.0asyncdispatch.errorcallback:`.m.di.0asyncdispatch.addservererror",1,,configure second backend error callback -before,0,0,q,ad.addserver[bh2;`scatter],1,,register second backend as scatter servertype - -/ Test 0: server registry -true,0,0,q,bh in exec handle from srv[] where active,1,,first backend server is active -true,0,0,q,bh2 in exec handle from srv[] where active,1,,second backend server is active -true,0,0,q,bh in exec handle from .m.di.0asyncdispatch.availableservers 1b,1,,first backend is available for dispatch -true,0,0,q,bh2 in exec handle from .m.di.0asyncdispatch.availableservers 1b,1,,second backend is available for dispatch - -/ Test 1: queueing and FIFO scheduling -run,0,0,q,.m.di.0asyncdispatch.addquery["2+2";enlist`mock;raze;();0Wn;0b],1,,queue a query for the mock servertype -true,0,0,q,1~count select from qq[] where query~\:"2+2",1,,query was added to the queue -true,0,0,q,1~count .m.di.0asyncdispatch.getnextqueryid[],1,,query is runnable - idle mock server available -true,0,0,q,(enlist`mock)~first exec servertype from .m.di.0asyncdispatch.getnextqueryid[],1,,getnextqueryid identifies the available servertype - -/ Test 2: full dispatch -> result -> join -> reply via real IPC round-trip -run,0,0,q,qid1:exec first queryid from qq[] where query~\:"2+2",1,,capture the queued query id -run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch query to real backend process -run,0,0,q,bh"",1,,sync round-trip: flushes dispatch and waits for backend result callback before returning -true,0,0,q,0b~first exec error from qq[] where queryid=qid1,1,,query completed without error -true,0,0,q,not null first exec returntime from qq[] where queryid=qid1,1,,returntime is stamped -true,0,0,q,not qid1 in key res[],1,,result accumulator cleaned up after join - -/ Test 3: checktimeout -run,0,0,q,.m.di.0asyncdispatch.addquery["3+3";enlist`mock;raze;();0D00:00:00.000000001;0b],1,,queue with a near-zero timeout -run,0,0,q,qid2:exec first queryid from qq[] where query~\:"3+3",1,,capture the queued query id -run,0,0,q,ad.checktimeout[],1,,timeout sweep should catch the expired query -true,0,0,q,1b~first exec error from qq[] where queryid=qid2,1,,timed-out query is flagged as error -true,0,0,q,not null first exec returntime from qq[] where queryid=qid2,1,,timed-out query has returntime stamped - -/ Test 4: removequeries purges completed queries -run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p+2D}],1,,fast-forward the clock by 2 days -run,0,0,q,ad.removequeries .m.di.0asyncdispatch.querykeeptime,1,,purge queries older than querykeeptime -true,0,0,q,0~count select from qq[] where queryid in (qid1;qid2),1,,both completed queries purged -run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p}],1,,restore the real clock - -/ Test 5: removeserverhandle errors queued queries when the only server disconnects -run,0,0,q,ad.addserver[1i;`mock2],1,,register a second mock server (fake handle - query is errored before dispatch) -run,0,0,q,.m.di.0asyncdispatch.addquery["4+4";enlist`mock2;raze;();0Wn;0b],1,,queue a query that can only run on mock2 -run,0,0,q,qid3:exec first queryid from qq[] where query~\:"4+4",1,,capture the queued query id -run,0,0,q,ad.removeserverhandle[1i],1,,disconnect the only server able to run the query -true,0,0,q,1b~first exec error from qq[] where queryid=qid3,1,,query errored as no server can satisfy it -true,0,0,q,0b~first exec active from srv[] where handle=1i,1,,disconnected server marked inactive - -/ Test 6: removeinactive purges inactive server records -run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p+2D}],1,,fast-forward clock past clearinactivetime -run,0,0,q,ad.removeinactive .m.di.0asyncdispatch.clearinactivetime,1,,purge inactive servers -true,0,0,q,0~count select from srv[] where handle=1i,1,,inactive server record removed -run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p}],1,,restore real clock - -/ Test 7: client tracking -run,0,0,q,ad.addclientdetails[2i],1,,record a connected client -true,0,0,q,1~count select from cl[] where clienth=2i,1,,client connection is tracked -run,0,0,q,ad.removeclienthandle[2i],1,,disconnect the client -true,0,0,q,0~count select from qq[] where (clienth=2i)&null returntime,1,,client unfinished queries cleared - -/ Test 8: pluggable hooks -run,0,0,q,ad.setformatresponse[{[status;sync;result]result}],1,,override formatresponse to pass result through -true,0,0,q,5~.m.di.0asyncdispatch.formatresponse[1b;0b;5],1,,overridden formatresponse is used -run,0,0,q,ad.setgetnextqueryid[{()}],1,,override scheduler to never pick a query -true,0,0,q,0~count .m.di.0asyncdispatch.getnextqueryid[],1,,overridden scheduler is used -run,0,0,q,ad.setcallbacks[`myresult;`myerror],1,,override callback symbols -true,0,0,q,(`myresult;`myerror)~(.m.di.0asyncdispatch.resultcallback;.m.di.0asyncdispatch.errorcallback),1,,callback symbols updated - -/ Test 9: removeclienthandle frees in-flight servers on client disconnect -run,0,0,q,ad.addserver[99i;`inflight],1,,register fake server for in-flight test -run,0,0,q,update inuse:1b from .m.di.0asyncdispatch.servers where handle=99i,1,,manually mark server in-use to simulate dispatched query -run,0,0,q,ad.addclientdetails[5i],1,,add fake client -run,0,0,q,.m.di.0asyncdispatch.addquery["5+5";enlist`inflight;raze;();0Wn;0b],1,,queue query for fake server -run,0,0,q,qid4:exec first queryid from qq[] where query~\:"5+5",1,,capture query id -run,0,0,q,.m.di.0asyncdispatch.results[qid4]:(5i;enlist[`inflight]!enlist(99i;(::);0b)),1,,inject in-flight slot to simulate dispatched-but-pending query -run,0,0,q,ad.removeclienthandle[5i],1,,disconnect client mid-flight -true,0,0,q,0b~first exec inuse from srv[] where handle=99i,1,,in-flight server freed on client disconnect -run,0,0,q,ad.removeserverhandle[99i],1,,clean up fake server - -/ Test 10: addservererror - real backend error path via IPC callback -run,0,0,q,ad.setcallbacks[`.m.di.0asyncdispatch.addserverresult;`.m.di.0asyncdispatch.addservererror],1,,restore callbacks overridden in test 8 -run,0,0,q,ad.setgetnextqueryid[{r:0!select from .m.di.0asyncdispatch.queryqueue where null returntime;1 sublist select from r where not queryid in key .m.di.0asyncdispatch.results}],1,,restore scheduler overridden in test 8 -run,0,0,q,.m.di.0asyncdispatch.addquery["'errortest";enlist`mock;raze;();0Wn;0b],1,,queue a query that will signal an error on the backend -run,0,0,q,qid5:exec first queryid from qq[] where query~\:"'errortest",1,,capture the queued query id -run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch query to backend -run,0,0,q,bh"",1,,sync round-trip: ensures backend has processed and sent error callback -true,0,0,q,1b~first exec error from qq[] where queryid=qid5,1,,backend error callback flagged query as error -true,0,0,q,not null first exec returntime from qq[] where queryid=qid5,1,,returntime stamped on backend error - -/ Test 11: postback wrapping - non-empty postback produces (postback;query;result) tuple -run,0,0,q,captured::(),1,,global to capture what formatresponse receives -run,0,0,q,ad.setformatresponse[{[s;sync;r]captured::r;$[not[s]and sync;'r;r]}],1,,intercept formatresponse to verify tosend construction -run,0,0,q,.m.di.0asyncdispatch.addquery["7+7";enlist`mock;raze;enlist(::);0Wn;0b],1,,queue query with non-empty postback -run,0,0,q,qid6:exec first queryid from qq[] where query~\:"7+7",1,,capture query id -run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch -run,0,0,q,bh"",1,,sync flush -true,0,0,q,0b~first exec error from qq[] where queryid=qid6,1,,postback query completed without error -true,0,0,q,(::;"7+7";enlist 14)~captured,1,,formatresponse received (postback;query;result) tuple -run,0,0,q,ad.setformatresponse[{[status;sync;result]$[not[status]and sync;'result;result]}],1,,restore default formatresponse - -/ Test 12: multi-servertype scatter-gather - two backends dispatched in parallel results joined -run,0,0,q,ad.setformatresponse[{[s;sync;r]captured::r;$[not[s]and sync;'r;r]}],1,,intercept formatresponse to capture joined result -run,0,0,q,.m.di.0asyncdispatch.addquery["1+1";`mock`scatter;{sum x};();0Wn;0b],1,,queue scatter-gather query requiring both servertypes -run,0,0,q,qid7:exec first queryid from qq[] where query~\:"1+1",1,,capture query id -run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch to both backends in parallel -run,0,0,q,bh"",1,,sync flush first backend -run,0,0,q,bh2"",1,,sync flush second backend -true,0,0,q,0b~first exec error from qq[] where queryid=qid7,1,,scatter-gather completed without error -true,0,0,q,4~captured,1,,joined result is sum of both backend results (1+1=2 from each; sum 2 2 = 4) -run,0,0,q,ad.setformatresponse[{[status;sync;result]$[not[status]and sync;'result;result]}],1,,restore default formatresponse - -/ Test 13: execqueryto local path - successful result delivered in-process -run,0,0,q,ad.setcallbacks[`.m.di.0asyncdispatch.addserverresult;`.m.di.0asyncdispatch.addservererror],1,,restore callbacks -run,0,0,q,ad.setgetnextqueryid[{r:0!select from .m.di.0asyncdispatch.queryqueue where null returntime;1 sublist select from r where not queryid in key .m.di.0asyncdispatch.results}],1,,restore scheduler -run,0,0,q,.test.localcap:(::),1,,reset local capture -run,0,0,q,.test.localfn:{[reqid;qry;res] .test.localcap:(reqid;qry;res)},1,,define local capturing callback in root namespace -run,0,0,q,ad.execqueryto[0Ni;"2+2";enlist`mock;{sum x};(.test.localfn;`req1);0Wn;0b],1,,dispatch via local path with postback (sum collapses single-element result to scalar) -run,0,0,q,bh"",1,,sync flush - wait for backend result callback -true,0,0,q,(`req1;"2+2";4)~.test.localcap,1,,postback invoked in-process with (reqid;query;result) - -/ Test 14: execqueryto local path - backend error delivered in-process via postback -run,0,0,q,.test.localcap:(::),1,,reset local capture -run,0,0,q,ad.execqueryto[0Ni;"'errortest";enlist`mock;raze;(.test.localfn;`req2);0Wn;0b],1,,dispatch erroring query via local path -run,0,0,q,bh"",1,,sync flush - wait for backend error callback -true,0,0,q,`req2~.test.localcap[0],1,,reqid correct -true,0,0,q,.test.localcap[2] like "error:*",1,,error string delivered as result via postback - -/ Test 15: execqueryto local path - timeout delivered in-process via postback -run,0,0,q,.test.localcap:(::),1,,reset local capture -run,0,0,q,ad.execqueryto[0Ni;"3+3";enlist`mock;raze;(.test.localfn;`req3);0D00:00:00.000000001;0b],1,,dispatch with near-zero timeout -run,0,0,q,ad.checktimeout[],1,,trigger timeout sweep -true,0,0,q,`req3~.test.localcap[0],1,,reqid correct -true,0,0,q,.test.localcap[2] like "error:*",1,,timeout error delivered in-process via postback - -/ Test 16: removeclients purges stale client audit rows -run,0,0,q,ad.addclientdetails[7i],1,,record a client for the cleanup test -true,0,0,q,1~count select from cl[] where clienth=7i,1,,client audit row exists -run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p+2D}],1,,fast-forward clock two days past the audit age -run,0,0,q,ad.removeclients[0D00:01],1,,purge client rows older than one minute -true,0,0,q,0~count select from cl[] where clienth=7i,1,,client audit row purged -run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p}],1,,restore the real clock - -/ Test 17: join-failure path - join function throws, client receives error and query is finished -/ dispatched to the scatter backend (bh2): the mock backend (bh) is left inuse by the in-flight timeout in test 15 -run,0,0,q,.m.di.0asyncdispatch.addquery["2+2";enlist`scatter;{'"joinboom"};();0Wn;0b],1,,queue a query whose join function throws -run,0,0,q,qid8:exec first queryid from qq[] where join~\:{'"joinboom"},1,,capture the query id -run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch to the scatter backend -run,0,0,q,bh2"",1,,sync flush - wait for the result callback and join attempt -true,0,0,q,1b~first exec error from qq[] where queryid=qid8,1,,join failure correctly flagged as error -true,0,0,q,not null first exec returntime from qq[] where queryid=qid8,1,,returntime stamped after join failure - -/ Test 18: execqueryto with a non-null explicit replyto stores it as clienth with local=0b -run,0,0,q,ad.execqueryto[99i;"1+1";enlist`scatter;{sum x};();0Wn;0b],1,,dispatch with explicit replyto handle 99i -run,0,0,q,qid9:exec first queryid from qq[] where (clienth=99i)&query~\:"1+1",1,,capture the query id -true,0,0,q,99i~first exec clienth from qq[] where queryid=qid9,1,,replyto stored as clienth -true,0,0,q,0b~first exec local from qq[] where queryid=qid9,1,,local flag is 0b for a non-null replyto -run,0,0,q,bh2"",1,,sync flush - completes the backend round-trip (reply to fake 99i fails silently) - -/ Test 19: pluggable server source - dispatch against an injected source whose servertype is not in the -/ internal registry; result routing uses the query's dispatch record so no internal registration is needed -run,0,0,q,savedavail:.m.di.0asyncdispatch.availableservers,1,,save the default (internal-registry) server source -run,0,0,q,ad.setavailableservers[{[eu]([]handle:enlist bh;servertype:enlist`ext)}],1,,inject a source presenting bh under servertype ext (bh is registered as mock) -run,0,0,q,.m.di.0asyncdispatch.addquery["6*7";enlist`ext;raze;();0Wn;0b],1,,queue a query for the external-only servertype -run,0,0,q,qidx:exec first queryid from qq[] where query~\:"6*7",1,,capture the query id -run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch to bh via the injected source -run,0,0,q,bh"",1,,sync flush - wait for the backend result callback -true,0,0,q,0b~first exec error from qq[] where queryid=qidx,1,,query completed without error via the injected source -true,0,0,q,not null first exec returntime from qq[] where queryid=qidx,1,,result routed via the dispatch record (servertype ext) not the servers table (mock) -run,0,0,q,ad.setavailableservers[savedavail],1,,restore the default internal-registry source - -/ Teardown -run,0,0,q,neg[bh](exit;0);neg[bh](::),1,,send exit to first backend process and flush -run,0,0,q,hclose bh,1,,close first backend handle -run,0,0,q,neg[bh2](exit;0);neg[bh2](::),1,,send exit to second backend process and flush -run,0,0,q,hclose bh2,1,,close second backend handle From 06cdf5a6a713a853f5229b1695be8aa60214804a Mon Sep 17 00:00:00 2001 From: ascottDI Date: Tue, 14 Jul 2026 11:04:40 +0100 Subject: [PATCH 10/12] responding to automated generated comments --- di/config/config.md | 14 ++++++++++++-- di/config/config.q | 39 +++++++++++++++++++++++++++++---------- di/config/test.csv | 20 +++++++++++++++++++- 3 files changed, 60 insertions(+), 13 deletions(-) diff --git a/di/config/config.md b/di/config/config.md index bca13ab4..9bdc921a 100644 --- a/di/config/config.md +++ b/di/config/config.md @@ -129,13 +129,23 @@ None — `di.config` is a standalone module. gain. The `default` fallback was dropped with it — every config is booted with a default, so a requested key is always present and a fallback arg is dead weight. A missing key is now a caller bug (surfaced as a q null on the inline index), not something the config layer silently papers over. +- **`getmodule` returns leaf settings only — child namespaces are excluded.** `\v .rdb` lists any + child namespace (e.g. `.rdb.sub`) alongside the real settings, so `getmodule` filters them out — + a module's config slice should be flat setting→value pairs, not a nested namespace. A child + namespace is detected as a `99h` dict carrying a null-symbol self-reference key; a genuine + dict-valued setting has no such key and is kept, so it is not misidentified as a namespace. Open gaps / not implemented: - **No runtime reload.** TorQ's `reloadf` (force-reload an already-loaded file) is not ported — `loadconfig` always dedups. Adequate for startup; revisit if runtime config reload is needed. -- **`overrideconfig`'s null-parse guard targets scalar/vector basic types**; a string-typed (`10h`) - config var is an uncommon edge where the per-element null check is loose. +- **`overrideconfig` parse-failure detection is type-aware.** Most basic types parse to a null on a + bad value, which `applyoverride` rejects via a null check. Boolean (`1h`) and byte (`4h`) are the + only in-scope basic types with **no null** — `"B"$"bad"` yields `0b`, `"X"$"gg"` yields `0x00`, both + non-null — so a bad value would slip past a null check and silently corrupt config. `parsefailed` + handles these two explicitly (boolean must be `"0"`/`"1"`; byte must be even-length hex); anything + else is rejected and logged, never written. A string-typed (`10h`) config var still can't be + overridden (its per-element char cast is rejected as non-parsing) — this fails safe and is a rare edge. - **No committed `di.log` integration test yet.** `di.log` (the standard logger) currently lives on the `feature-logging` branch and isn't merged, so a committed `use\`di.log` test would fail on this branch. di.config's contract match with `di.log` is **verified manually** — the real `di.log` was diff --git a/di/config/config.q b/di/config/config.q index 872a9495..9efbc945 100644 --- a/di/config/config.q +++ b/di/config/config.q @@ -11,9 +11,10 @@ / internal helper loadconfig handles per-file loading behind loadcascade. raiseerror:{[ctx;msg] - / internal - log an error under ctx then signal it, so failures are observable - / in the log as well as thrown to the caller. - .z.m.logerr[ctx;msg]; + / internal - log an error under ctx then signal it, so failures are observable. + / guard the log call: the logger is unset until init runs, so a public fn misused + / before init still signals the real message rather than a name error on logerr. + .[{.z.m.logerr[x;y]};(ctx;msg);{}]; '"di.config: ",string[ctx],": ",msg; }; @@ -93,11 +94,23 @@ loadcascade:{[dirs;names] :raze {[ds;nm] loadconfig[;nm] each ds}[dirs;] each names; }; +hexchars:"0123456789abcdefABCDEF"; + +parsefailed:{[t;raw;vals] + / internal - true if any raw string failed to parse to type t. boolean (1h) and byte + / (4h) are the only in-scope basic types with no null, so a bad parse ("B"$"bad" -> 0b, + / "X"$"gg" -> 0x00) slips past a null check and would silently corrupt config - validate + / their raw string form explicitly. every other type yields a null on a bad parse. + :$[1h=abs t;not all raw in (enlist"0";enlist"1"); + 4h=abs t;not all {(0=count[x] mod 2) and all x in hexchars} each raw; + any null vals]; + }; + applyoverride:{[name;raw] - / internal - parse raw command-line value(s) into name's current type and set - / it. raw is a string or a list of strings. returns 1b if applied, 0b if the - / variable is not a basic type or a value failed to parse (null). the variable - / must already exist - its current type drives the parse. + / internal - parse raw command-line value(s) into name's current type and set it. raw + / is a string or a list of strings. returns 1b if applied, 0b if the variable is not a + / basic type or a value failed to parse. the variable must already exist - its current + / type drives the parse. t:type value name; if[not (abs t) within (1;-1+count .Q.t); .z.m.logerr[`overrideconfig;"cannot override ",(string name),": not a basic type"]; @@ -105,11 +118,11 @@ applyoverride:{[name;raw] ]; raw:$[10h=type raw;enlist raw;raw]; vals:(upper .Q.t abs t)$'raw; - if[t<0;vals:first vals]; - if[any null vals; + if[parsefailed[t;raw;vals]; .z.m.logerr[`overrideconfig;"cannot override ",(string name),": value did not parse"]; :0b; ]; + if[t<0;vals:first vals]; .z.m.loginfo[`overrideconfig;"setting ",(string name)," to ",-3!vals]; set[name;vals]; :1b; @@ -161,5 +174,11 @@ getmodule:{[namespace] ]; ns:`$".",string namespace; vars:@[{system"v ",x};".",string namespace;`$()]; - :vars!{[ns;v] value ` sv (ns;v)}[ns;] each vars; + cfg:vars!{[ns;v] value ` sv (ns;v)}[ns;] each vars; + / \v lists child namespaces alongside settings, so drop them - only leaf settings + / belong in a module's config slice. a child namespace is a 99h dict carrying a + / null-symbol self-reference key; a genuine dict-valued setting has no such key and + / is kept. guard the key lookup with a cond (not `and`, which evaluates eagerly and + / would run `key` on non-dict values). + :(where {$[99h=type x;(`) in key x;0b]} each cfg) _ cfg; }; diff --git a/di/config/test.csv b/di/config/test.csv index 9655d6e1..601132cf 100644 --- a/di/config/test.csv +++ b/di/config/test.csv @@ -15,6 +15,10 @@ before,0,0,q,.diconfigtest.enabled:0b,1,1,override target - boolean before,0,0,q,.diconfigtest.rows:100,1,1,override target - long before,0,0,q,.diconfigtest.syms:`a`b,1,1,override target - symbol list before,0,0,q,.diconfigtest.tab:([]a:`long$()),1,1,override target - non-basic (table) +before,0,0,q,.diconfigtest.byte:0x01,1,1,override target - byte (a no-null basic type) +before,0,0,q,.diconfiggm.rows:100,1,1,getmodule target - flat scalar setting +before,0,0,q,.diconfiggm.limits:`lo`hi!1 2,1,1,getmodule target - dict-valued setting (99h - must be kept) +before,0,0,q,.diconfiggm.sub.x:1,1,1,getmodule target - child namespace (must be excluded) before,0,0,q,"system""mkdir -p /tmp/diconfigkxdir""",1,1,dir for the kx.log emission test before,0,0,q,"(`:/tmp/diconfigkxdir/default.q) 0: enlist ""diconfigkxvar:1;""",1,1,file loaded through the real logger comment,,,,,,,init - dependency validation @@ -26,7 +30,11 @@ comment,,,,,,,loadcascade - full cascade over directories and names run,0,0,q,"cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,run the cascade true,0,0,q,"4=count cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,one path per dir-name pair (skipped files included) true,0,0,q,10=diconfigcx,1,1,most-specific name wins over dir layer (c1/rdb loads last) -true,0,0,q,`c1default`c2default`c1rdb~diconfigtrace,1,1,name-major order; and re-runs don't re-execute (dedup) +run,0,0,q,cfg.init[enlist[`log]!enlist mocklog],1,1,re-init to reset the dedup set so the trace check is independent of any prior load (order/re-run safe) +run,0,0,q,diconfigtrace:`symbol$(),1,1,reset the tracer here in the assert phase - a before-row reset is batched once and cannot isolate this +run,0,0,q,"cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,load once - populates the tracer +run,0,0,q,"cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,re-run - dedup must NOT re-execute the files +true,0,0,q,`c1default`c2default`c1rdb~diconfigtrace,1,1,name-major order AND dedup (tracer unchanged by the re-run) true,0,0,q,"any (cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]) like ""*diconfigc2/rdb.q""",1,1,returned paths include skipped (missing) files true,0,0,q,"any (exec msg from logtab where lvl=`info) like ""*no config file*""",1,1,a missing cascade file is skipped at info (internal loadconfig) true,0,0,q,"1=count cfg.loadcascade[""/tmp/diconfigc1"";`default]",1,1,accepts a single string dir and single symbol name @@ -36,6 +44,13 @@ comment,,,,,,,overrideconfig - parse and apply command-line overrides run,0,0,q,"cfg.overrideconfig[`.diconfigtest.enabled`.diconfigtest.rows!(enlist""1"";enlist""500"")]",1,1,override a bool and a long true,0,0,q,1b~.diconfigtest.enabled,1,1,bool parsed and set from string true,0,0,q,500=.diconfigtest.rows,1,1,long parsed and set from string +true,0,0,q,"0=count cfg.overrideconfig[enlist[`.diconfigtest.enabled]!enlist ""bad""]",1,1,unparseable boolean is rejected (no null - must not slip through) +true,0,0,q,1b~.diconfigtest.enabled,1,1,rejected boolean override leaves the variable unchanged (not silently set to 0b) +true,0,0,q,"any (exec msg from logtab) like ""*did not parse*""",1,1,rejected boolean override is logged +true,0,0,q,"(enlist`.diconfigtest.byte)~cfg.overrideconfig[enlist[`.diconfigtest.byte]!enlist ""ff""]",1,1,valid hex byte parsed and applied +true,0,0,q,0xff~.diconfigtest.byte,1,1,byte set from valid hex string +true,0,0,q,"0=count cfg.overrideconfig[enlist[`.diconfigtest.byte]!enlist ""gg""]",1,1,invalid hex byte is rejected (no null - must not slip through as 0x00) +true,0,0,q,0xff~.diconfigtest.byte,1,1,rejected byte override leaves the variable unchanged run,0,0,q,"cfg.overrideconfig[enlist[`.diconfigtest.syms]!enlist (""xx"";""yy"")]",1,1,override a symbol list true,0,0,q,`xx`yy~.diconfigtest.syms,1,1,symbol list parsed and set from strings true,0,0,q,"(enlist`.diconfigtest.rows)~cfg.overrideconfig[enlist[`.diconfigtest.rows]!enlist enlist""7""]",1,1,returns the variables actually overridden @@ -52,6 +67,9 @@ true,0,0,q,2=count cfg.getmodule[`diconfigq],1,1,getmodule returns the whole nam true,0,0,q,`trade`quote~cfg.getmodule[`diconfigq]`subscribeto,1,1,getmodule slice carries the values true,0,0,q,100~cfg.getmodule[`diconfigq]`rows,1,1,getmodule slice carries the second value true,0,0,q,0=count cfg.getmodule[`nosuchns],1,1,getmodule returns an empty dict for an unknown namespace +true,0,0,q,`limits`rows~asc key cfg.getmodule[`diconfiggm],1,1,getmodule excludes child namespaces but keeps flat and dict-valued settings +true,0,0,q,not `sub in key cfg.getmodule[`diconfiggm],1,1,child namespace excluded from the config slice +true,0,0,q,(`lo`hi!1 2)~cfg.getmodule[`diconfiggm]`limits,1,1,dict-valued setting preserved (not misidentified as a namespace) run,0,0,q,"cfg.overrideconfig[enlist[`.diconfigq.rows]!enlist enlist""500""]",1,1,override a stored value true,0,0,q,500~cfg.getmodule[`diconfigq]`rows,1,1,getmodule reflects overrideconfig changes fail,0,0,q,cfg.getmodule[42],1,1,getmodule rejects a non-symbol namespace From 84533b3810302659d1755758d113009353bb100c Mon Sep 17 00:00:00 2001 From: ascottDI Date: Tue, 14 Jul 2026 11:21:13 +0100 Subject: [PATCH 11/12] updated following automated reviewer comments --- di/config/config.md | 9 ++++++--- di/config/config.q | 28 +++++++++++++++++----------- di/config/test.csv | 4 +++- 3 files changed, 26 insertions(+), 15 deletions(-) diff --git a/di/config/config.md b/di/config/config.md index 9bdc921a..81fa2c2b 100644 --- a/di/config/config.md +++ b/di/config/config.md @@ -131,9 +131,12 @@ None — `di.config` is a standalone module. bug (surfaced as a q null on the inline index), not something the config layer silently papers over. - **`getmodule` returns leaf settings only — child namespaces are excluded.** `\v .rdb` lists any child namespace (e.g. `.rdb.sub`) alongside the real settings, so `getmodule` filters them out — - a module's config slice should be flat setting→value pairs, not a nested namespace. A child - namespace is detected as a `99h` dict carrying a null-symbol self-reference key; a genuine - dict-valued setting has no such key and is kept, so it is not misidentified as a namespace. + a module's config slice should be flat setting→value pairs, not a nested namespace. Each name is + tested by asking kdb+ directly whether it is itself a namespace (`\v` on the qualified name succeeds + for a namespace and signals for a plain variable). This is deliberately used over the tempting + null-symbol-self-key heuristic (`` (`) in key value ``), which is fragile at the edges — a + dict-valued setting could coincidentally carry a null-symbol key (and be wrongly dropped), or a + namespace reassigned to a plain dict could lack one (and be wrongly kept). Open gaps / not implemented: diff --git a/di/config/config.q b/di/config/config.q index 9efbc945..ee057029 100644 --- a/di/config/config.q +++ b/di/config/config.q @@ -11,10 +11,12 @@ / internal helper loadconfig handles per-file loading behind loadcascade. raiseerror:{[ctx;msg] - / internal - log an error under ctx then signal it, so failures are observable. - / guard the log call: the logger is unset until init runs, so a public fn misused - / before init still signals the real message rather than a name error on logerr. - .[{.z.m.logerr[x;y]};(ctx;msg);{}]; + / internal - log an error under ctx then signal it, so the domain error is always + / observable. the logger is unset until init runs, so log only when it is actually + / wired (checked without calling it) - a public fn misused before init still signals + / the real message instead of a name error on logerr. a wired logger is called + / directly, as everywhere else in the module, so its own errors are not swallowed. + if[@[{.z.m.logerr;1b};::;0b];.z.m.logerr[ctx;msg]]; '"di.config: ",string[ctx],": ",msg; }; @@ -122,6 +124,9 @@ applyoverride:{[name;raw] .z.m.logerr[`overrideconfig;"cannot override ",(string name),": value did not parse"]; :0b; ]; + / reduce to a scalar only after parsefailed - it validates the full per-element list + / (its null check spans every parsed value); this runs before the log and set below, + / so both use the scalar form. if[t<0;vals:first vals]; .z.m.loginfo[`overrideconfig;"setting ",(string name)," to ",-3!vals]; set[name;vals]; @@ -174,11 +179,12 @@ getmodule:{[namespace] ]; ns:`$".",string namespace; vars:@[{system"v ",x};".",string namespace;`$()]; - cfg:vars!{[ns;v] value ` sv (ns;v)}[ns;] each vars; - / \v lists child namespaces alongside settings, so drop them - only leaf settings - / belong in a module's config slice. a child namespace is a 99h dict carrying a - / null-symbol self-reference key; a genuine dict-valued setting has no such key and - / is kept. guard the key lookup with a cond (not `and`, which evaluates eagerly and - / would run `key` on non-dict values). - :(where {$[99h=type x;(`) in key x;0b]} each cfg) _ cfg; + / \v lists child namespaces alongside settings, so drop them - a module's config slice + / is flat setting->value pairs, not nested namespaces. ask kdb+ directly whether each + / name is itself a namespace (\v succeeds on a namespace, signals on a plain variable); + / this is more robust than inspecting the dict for a null-symbol self-reference key, + / which a plain dict-valued setting could coincidentally carry (or a namespace lack). + issub:{[ns;v] @[{system"v ",x;1b};string ` sv (ns;v);0b]}; + vars:vars where not issub[ns;] each vars; + :vars!{[ns;v] value ` sv (ns;v)}[ns;] each vars; }; diff --git a/di/config/test.csv b/di/config/test.csv index 601132cf..ef09ba2c 100644 --- a/di/config/test.csv +++ b/di/config/test.csv @@ -19,6 +19,7 @@ before,0,0,q,.diconfigtest.byte:0x01,1,1,override target - byte (a no-null basic before,0,0,q,.diconfiggm.rows:100,1,1,getmodule target - flat scalar setting before,0,0,q,.diconfiggm.limits:`lo`hi!1 2,1,1,getmodule target - dict-valued setting (99h - must be kept) before,0,0,q,.diconfiggm.sub.x:1,1,1,getmodule target - child namespace (must be excluded) +before,0,0,q,.diconfiggm.nullkey:(enlist `)!enlist 42,1,1,getmodule target - dict setting carrying a null-symbol key (must NOT be treated as a namespace) before,0,0,q,"system""mkdir -p /tmp/diconfigkxdir""",1,1,dir for the kx.log emission test before,0,0,q,"(`:/tmp/diconfigkxdir/default.q) 0: enlist ""diconfigkxvar:1;""",1,1,file loaded through the real logger comment,,,,,,,init - dependency validation @@ -67,9 +68,10 @@ true,0,0,q,2=count cfg.getmodule[`diconfigq],1,1,getmodule returns the whole nam true,0,0,q,`trade`quote~cfg.getmodule[`diconfigq]`subscribeto,1,1,getmodule slice carries the values true,0,0,q,100~cfg.getmodule[`diconfigq]`rows,1,1,getmodule slice carries the second value true,0,0,q,0=count cfg.getmodule[`nosuchns],1,1,getmodule returns an empty dict for an unknown namespace -true,0,0,q,`limits`rows~asc key cfg.getmodule[`diconfiggm],1,1,getmodule excludes child namespaces but keeps flat and dict-valued settings +true,0,0,q,`limits`nullkey`rows~asc key cfg.getmodule[`diconfiggm],1,1,getmodule excludes child namespaces but keeps flat and dict-valued settings true,0,0,q,not `sub in key cfg.getmodule[`diconfiggm],1,1,child namespace excluded from the config slice true,0,0,q,(`lo`hi!1 2)~cfg.getmodule[`diconfiggm]`limits,1,1,dict-valued setting preserved (not misidentified as a namespace) +true,0,0,q,42~first cfg.getmodule[`diconfiggm]`nullkey,1,1,dict setting with a null-symbol key kept (namespace test is \v-based not self-key-based) run,0,0,q,"cfg.overrideconfig[enlist[`.diconfigq.rows]!enlist enlist""500""]",1,1,override a stored value true,0,0,q,500~cfg.getmodule[`diconfigq]`rows,1,1,getmodule reflects overrideconfig changes fail,0,0,q,cfg.getmodule[42],1,1,getmodule rejects a non-symbol namespace From a2a24f8fac47b7e2a9a1163bbbed87bc7c444e72 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Tue, 14 Jul 2026 12:10:10 +0100 Subject: [PATCH 12/12] responding to J.Masset comments --- di/config/config.md | 4 +- di/config/config.q | 147 ++++++++++++++++++-------------------------- 2 files changed, 62 insertions(+), 89 deletions(-) diff --git a/di/config/config.md b/di/config/config.md index 81fa2c2b..fdd6eeeb 100644 --- a/di/config/config.md +++ b/di/config/config.md @@ -45,8 +45,8 @@ caller first — di.config does no wrapping: `` `info`warn`error!({[c;m]inst[`in | `overrideconfig` | `overrideconfig[params]` | Apply the **command-line-argument layer** on top of the file cascade — di.torq parses the process command line (`.Q.opt .z.x`) and calls this at startup, so launch-time flags (e.g. `-loglevel info`, `-tablelist trade quote`) win over the settings files. This is the automatic top layer of the cascade, **not** an interactive/by-hand step. `params` is a dict keyed by variable name (symbol) with string (or list-of-string) values, each parsed into that variable's **existing** type. Only already-defined, basic-typed variables can be overridden; undefined names, non-basic types, and unparseable values are logged and skipped. Returns the list of variables actually overridden. | | `getmodule` | `getmodule[namespace]` | Return a namespace's whole resolved config as a bare-keyed value dict — the slice di.torq passes to a module's `init`. `namespace` is a bare symbol; an unconfigured namespace yields an empty dict. Callers read a single setting by indexing the returned dict inline (e.g. `getmodule[\`rdb]\`subscribeto`). | -Internal helpers — `loadconfig` (the per-file loader behind `loadcascade`), `raiseerror`, -`applyoverride`, and the load-tracking state — are deliberately not exported. +Internal helpers — `loadconfig` (the per-file loader behind `loadcascade`), +`applyoverride`, `parsefailed`, and the load-tracking state — are deliberately not exported. `loadcascade` is the top-level entry point. A caller (e.g. `di.torq`, which resolves the process identity and directory paths) drives it as: diff --git a/di/config/config.q b/di/config/config.q index ee057029..d70ec27a 100644 --- a/di/config/config.q +++ b/di/config/config.q @@ -1,34 +1,16 @@ -/ configuration loading and cascade resolution for the modular torq world. -/ replaces torq.q's in-process config handling (loadf, loadconfig, loadaddconfig, -/ overrideconfig). the logger is an injected dependency wired via init; the module -/ reads no environment variables or process identity itself - the caller (di.torq) -/ supplies the settings directories, name sequence and any command-line overrides. -/ public api: -/ loadcascade - resolve the full cascade over dirs x names (name-major) -/ overrideconfig - apply command-line-style overrides after the cascade -/ getmodule - query the resolved config store as a per-namespace dict; di.torq -/ partitions config per module with it and injects each slice via that module's init -/ internal helper loadconfig handles per-file loading behind loadcascade. - -raiseerror:{[ctx;msg] - / internal - log an error under ctx then signal it, so the domain error is always - / observable. the logger is unset until init runs, so log only when it is actually - / wired (checked without calling it) - a public fn misused before init still signals - / the real message instead of a name error on logerr. a wired logger is called - / directly, as everywhere else in the module, so its own errors are not swallowed. - if[@[{.z.m.logerr;1b};::;0b];.z.m.logerr[ctx;msg]]; - '"di.config: ",string[ctx],": ",msg; - }; +/ configuration loading and cascade resolution for the modular torq world - replaces +/ torq.q's loadf/loadconfig/loadaddconfig/overrideconfig. the logger is injected via init; +/ the module reads no env/identity - the caller (di.torq) supplies dirs, names and overrides. +/ public api: loadcascade (resolve cascade over dirs x names, name-major), overrideconfig +/ (command-line overrides, applied after), getmodule (query the store as a per-namespace +/ dict; di.torq partitions config per module with it). loadconfig is the internal per-file +/ loader behind loadcascade. init:{[deps] - / wire injectable dependencies - log is required, there is no silent fallback. - / deps: a dict with a `log key; log must already be a binary `info`warn`error - / dict of {[c;m]} loggers - build it from di.log (the standard logger, which - / exports binary info/warn/error) or hand-roll one. no adaptation is performed - / here, so a non-conforming logger (e.g. a raw monadic kx.log instance) must be - / wrapped by the caller first. examples: - / config.init[enlist[`log]!enlist logdep] - / config.init[`log`timer!(logdep;timerdep)] + / wire the injected logger - required, no silent fallback. deps: a dict with a `log key + / holding a binary `info`warn`error dict of {[c;m]} loggers (from di.log or hand-rolled). + / no adaptation here, so a monadic kx.log instance must be wrapped first. + / e.g. config.init[enlist[`log]!enlist logdep] if[99h<>type deps; '"di.config: deps must be a dict with `log key"]; if[not `log in key deps; @@ -45,15 +27,16 @@ init:{[deps] }; loadconfig:{[dir;name] - / internal - load one cascade config file dir/{name}.q if present, tracking it - / so it is not re-loaded. a missing file is normal in the cascade (not every - / proctype/procname has one), so its absence is logged at info and skipped, not - / warned. returns the file path. + / internal - load cascade file dir/{name}.q if present, tracking it so it is not + / re-loaded. a missing file is normal in the cascade, so logged at info and skipped. + / returns the file path. if[not 10h=abs type dir; - raiseerror[`loadconfig;"dir must be a string path"]; + .z.m.logerr[`loadconfig;err:"di.config: dir must be a string path"]; + 'err; ]; if[not -11h=type name; - raiseerror[`loadconfig;"name must be a symbol"]; + .z.m.logerr[`loadconfig;err:"di.config: name must be a symbol"]; + 'err; ]; file:dir,"/",string[name],".q"; if[file in .z.m.loaded; @@ -65,32 +48,30 @@ loadconfig:{[dir;name] :file; ]; .z.m.loginfo[`loadconfig;"loading ",file]; - .[system;enlist"l ",file;{[f;e] raiseerror[`loadconfig;"failed to load ",f,": ",e]}[file;]]; + .[system;enlist"l ",file;{[f;e] .z.m.logerr[`loadconfig;err:"di.config: failed to load ",f,": ",e];'err}[file;]]; .z.m.loaded,:enlist file; :file; }; loadcascade:{[dirs;names] - / resolve a configuration cascade name-major: for each config name (least to - / most specific) load it from every directory in turn. files load in sequence, - / so a more specific name always wins over a less specific one regardless of - / directory, and within a single name the later (higher-priority) directory - / overrides the earlier. dirs is a directory path or list of paths (strings), - / ordered lowest->highest priority; names is a config name or list of names - / (symbols), ordered least->most specific. missing files are skipped (by - / loadconfig). returns the flat list of constructed paths, in load order. - / typical caller (di.torq resolves the dirs and process identity): - / config.loadcascade[(kdbconfig;appconfig);`default,proctype,procname] + / resolve a cascade name-major: for each name (least->most specific) load it from every + / dir in turn, so a more specific name always wins and, within a name, the later (higher- + / priority) dir overrides. dirs is a path or list of paths (low->high priority); names a + / symbol or symbol list (least->most specific). missing files are skipped. returns the + / constructed paths in load order. e.g. loadcascade[(kdbconfig;appconfig);`default,proctype,procname] dirs:$[10h=type dirs;enlist dirs;dirs]; names:(),names; if[not 0h=type dirs; - raiseerror[`loadcascade;"dirs must be a string path or a list of string paths"]; + .z.m.logerr[`loadcascade;err:"di.config: dirs must be a string path or a list of string paths"]; + 'err; ]; if[not all 10h=type each dirs; - raiseerror[`loadcascade;"dirs must be a string path or a list of string paths"]; + .z.m.logerr[`loadcascade;err:"di.config: dirs must be a string path or a list of string paths"]; + 'err; ]; if[not 11h=abs type names; - raiseerror[`loadcascade;"names must be a symbol or a symbol list"]; + .z.m.logerr[`loadcascade;err:"di.config: names must be a symbol or a symbol list"]; + 'err; ]; .z.m.loginfo[`loadcascade;"resolving cascade: ",(string count dirs)," dir(s) x ",(string count names)," name(s)"]; :raze {[ds;nm] loadconfig[;nm] each ds}[dirs;] each names; @@ -99,20 +80,18 @@ loadcascade:{[dirs;names] hexchars:"0123456789abcdefABCDEF"; parsefailed:{[t;raw;vals] - / internal - true if any raw string failed to parse to type t. boolean (1h) and byte - / (4h) are the only in-scope basic types with no null, so a bad parse ("B"$"bad" -> 0b, - / "X"$"gg" -> 0x00) slips past a null check and would silently corrupt config - validate - / their raw string form explicitly. every other type yields a null on a bad parse. + / internal - true if any raw string failed to parse to type t. boolean (1h) and byte (4h) + / have no null, so a bad parse ("B"$"bad"->0b, "X"$"gg"->0x00) slips past a null check and + / would corrupt config - validate their string form explicitly. other types null on failure. :$[1h=abs t;not all raw in (enlist"0";enlist"1"); 4h=abs t;not all {(0=count[x] mod 2) and all x in hexchars} each raw; any null vals]; }; applyoverride:{[name;raw] - / internal - parse raw command-line value(s) into name's current type and set it. raw - / is a string or a list of strings. returns 1b if applied, 0b if the variable is not a - / basic type or a value failed to parse. the variable must already exist - its current - / type drives the parse. + / internal - parse raw value(s) into name's current type and set it. raw is a string or + / list of strings. returns 1b if applied, 0b if name is not a basic type or a value failed + / to parse. name must already exist - its type drives the parse. t:type value name; if[not (abs t) within (1;-1+count .Q.t); .z.m.logerr[`overrideconfig;"cannot override ",(string name),": not a basic type"]; @@ -124,9 +103,8 @@ applyoverride:{[name;raw] .z.m.logerr[`overrideconfig;"cannot override ",(string name),": value did not parse"]; :0b; ]; - / reduce to a scalar only after parsefailed - it validates the full per-element list - / (its null check spans every parsed value); this runs before the log and set below, - / so both use the scalar form. + / reduce to scalar only after parsefailed (which checks the full per-element list); runs + / before the log and set, so both use the scalar form. if[t<0;vals:first vals]; .z.m.loginfo[`overrideconfig;"setting ",(string name)," to ",-3!vals]; set[name;vals]; @@ -134,19 +112,19 @@ applyoverride:{[name;raw] }; overrideconfig:{[params] - / apply command-line-style overrides to already-defined variables, parsing each - / value into the variable's existing type. params is a dict keyed by variable - / name (symbol) with string (or list-of-string) values. only defined variables - / can be overridden (their type drives the parse); undefined names are logged - / and skipped. returns the list of variables actually overridden. call after - / loadcascade to let the command line win over file config. + / apply command-line-style overrides to defined variables, parsing each value into the + / variable's existing type. params is a dict of variable name (symbol) -> string (or list + / of strings). only defined names are overridden; undefined ones are logged and skipped. + / returns the variables actually overridden. call after loadcascade so the command line wins. if[99h<>type params; - raiseerror[`overrideconfig;"params must be a dict keyed by variable name"]; + .z.m.logerr[`overrideconfig;err:"di.config: params must be a dict keyed by variable name"]; + 'err; ]; vars:key params; if[0value pairs, not nested namespaces. ask kdb+ directly whether each - / name is itself a namespace (\v succeeds on a namespace, signals on a plain variable); - / this is more robust than inspecting the dict for a null-symbol self-reference key, - / which a plain dict-valued setting could coincidentally carry (or a namespace lack). + / \v lists child namespaces alongside settings; drop them so the slice is flat + / setting->value pairs. ask kdb+ whether each name is itself a namespace (\v succeeds on a + / namespace, signals on a plain variable) - more robust than checking for a null-symbol + / self-key, which a dict-valued setting could carry or a namespace lack. issub:{[ns;v] @[{system"v ",x;1b};string ` sv (ns;v);0b]}; vars:vars where not issub[ns;] each vars; :vars!{[ns;v] value ` sv (ns;v)}[ns;] each vars;