diff --git a/CHANGELOG.md b/CHANGELOG.md index 4e99bca..989da44 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,27 @@ All notable changes to EigenScript are documented here. ### Added +- **`eigenscript --api [--json]`: the machine-readable surface index + (#734).** One call answers "does X exist, and is it builtin, + extension, or lib" instead of a text search across 729 lines of + BUILTINS.md plus 1491 of STDLIB.md — the missing third leg beside + `--lint --json` and `--test --json`. Builtins are enumerated from the + live registry (never a hand list — the #459 lesson), extensions from + `ext_names.h` by group (`gfx`/`http`/`http_request`/`db`/`model`/ + `net` — the LANGUAGE surface, independent of this binary's build + flags), and every public `lib/*.eigs` define ships WITH its parameter + list as written (defaults included; the no-paren form reports its + implicit `n`), scraped from the same directories the import resolver + searches. Text mode is greppable (`builtin sort_by`, + `extension net net_dial`, `lib list.filter(items, fn)`); suite [131] + pins the kinds, the params, JSON validity, and the + sort_by-is-a-builtin / filter-is-lib split that misled agents. + Discoverability fixes from the same issue: `CLAUDE.md` now routes + `.eigs`-writing work to `docs/llms.txt` (the repo's best AI-facing + reference was reachable only from README line 424), and BUILTINS.md's + builtins-vs-library note now names `map`/`filter`/`reduce` explicitly + — the exact greps that returned zero hits. + - **Lint `W022`: over-arity literal calls (#733).** With `define two(a, b)`, `two of [1, 2, 99]` binds `a=1, b=2` and silently DISCARDS `99` — rc=0, no diagnostic at any stage (under-arity at diff --git a/CLAUDE.md b/CLAUDE.md index ad8182e..3f55f60 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -116,7 +116,12 @@ Always-on: sites" checklists — silent capture/replay/ABI bugs). - **Changing the AOT compiler** (separate `ouroboros` repo)? → the **`aot-differential`** skill (VM as byte-exact oracle). -- **Writing `.eigs` code**? → the **`write-eigenscript`** skill. +- **Writing `.eigs` code**? → the **`write-eigenscript`** skill, and + **`docs/llms.txt`** — the whole language in one 190-line file (call + syntax, scope, observer, validation ladder); an agent primed with it + has written correct programs from it alone (#734). Resolve "does + function X exist" with `eigenscript --api` (or `--api --json`) — the + full builtin/extension/lib surface index in one call. - **Cutting a release** (tag/dispatch path, the doc-drift "Latest release" gate, the tap)? → the **`release`** skill. diff --git a/docs/BUILTINS.md b/docs/BUILTINS.md index 48e7249..32661a7 100644 --- a/docs/BUILTINS.md +++ b/docs/BUILTINS.md @@ -9,10 +9,16 @@ gfx, audio) require a full build or the `gfx` target. > layer (DataFrames, stats, distributions, matrices, sets, sorting, > the GUI toolkit, the STEM shelf, ...) lives in > **[STDLIB.md](STDLIB.md)** — start at its "Finding Things" index if you -> know the task but not the function. Rule of thumb: **file/process/network, +> know the task but not the function. In particular **`map`, `filter`, +> and `reduce` are NOT builtins** — they are `lib/list.eigs` +> (`import list` → `list.filter of [xs, fn]`, or +> `load_file of "lib/list.eigs"` for bare names), even though `sort_by` +> IS a builtin (#734). Rule of thumb: **file/process/network, > tensor math, JSON encode-decode, regex, channels/tasks, and the > interrogatives are builtins (here); everything you `import` is library -> (there).** +> (there).** To resolve a name mechanically, `eigenscript --api [--json]` +> dumps the full surface — every builtin, extension (by group), and lib +> function with its parameter list — in one call. New since 0.8.1: concurrency (`spawn`, `thread_join`, `channel`, `send`, `recv`, `try_recv`, `recv_timeout`, `close_channel`, `channel_closed`), diff --git a/docs/llms.txt b/docs/llms.txt index 3e2da8c..5414ccf 100644 --- a/docs/llms.txt +++ b/docs/llms.txt @@ -175,7 +175,9 @@ import list # namespaced: list.filter of [xs, fn] load_file of "lib/list.eigs" # bare names: filter of [xs, fn] ``` -See docs/BUILTINS.md and docs/STDLIB.md before writing your own. +See docs/BUILTINS.md and docs/STDLIB.md before writing your own — or resolve +a name mechanically: `eigenscript --api --json` dumps every builtin, +extension (by group), and lib function with its parameter list in one call. ## The generation-validation ladder (always run what you generate) diff --git a/src/eigenscript.h b/src/eigenscript.h index c14e2af..55135ff 100644 --- a/src/eigenscript.h +++ b/src/eigenscript.h @@ -1339,5 +1339,10 @@ int lint_collect(ASTNode *ast, const char *path, const char *source, * the loaded-by directive feeds its INPUT via the `source` param above. */ int lint_file_allows(const char *source, const char *code); int eigenscript_lint(const char *path, int json_mode, int fail_on_warning); +/* #734: the --api surface index — builtins from the live registry, + * extensions from ext_names.h by group, lib/*.eigs public defines with + * their parameter lists. Name resolution in one call; conventions stay + * in docs/BUILTINS.md / docs/STDLIB.md. Hosted-only (lint.c). */ +int eigs_api_dump(FILE *out, int json); #endif /* EIGENSCRIPT_H */ diff --git a/src/lint.c b/src/lint.c index d9c4681..4d99c28 100644 --- a/src/lint.c +++ b/src/lint.c @@ -6,6 +6,15 @@ #include "eigenscript.h" #include "ext_names.h" +/* #734: eigs_api_dump stamps the surface index with the version. The + * Makefile passes -DEIGENSCRIPT_VERSION for its own targets, but the + * amalgamation/embed build (build/eigenscript_all.c, no main.c) does not — + * same per-file fallback the other amalgamated TUs use (main.c, trace.c, + * eigenlsp.c). */ +#ifndef EIGENSCRIPT_VERSION +#define EIGENSCRIPT_VERSION "dev" +#endif + /* ---- Lint warning storage ---- */ typedef struct { @@ -958,22 +967,20 @@ static void w021_scan_dir(const char *dirpath) { free(ents); } -/* Build (or reuse) the name table for the linted file's base dir. Candidate - * lib dirs mirror the import resolver's priority order for "lib/.eigs" - * (resolve_eigenscript_file_from): CWD first, then the linted file's dir and - * its parent, the exe's install root, the user install — first module found - * wins, as it does at runtime. */ -static void w021_build(const char *base) { - if (g_w021_built && strcmp(g_w021_base, base) == 0) return; - w021_free_table(); - snprintf(g_w021_base, sizeof(g_w021_base), "%s", base); - g_w021_built = 1; - +/* Candidate lib dirs, mirroring the import resolver's priority order for + * "lib/.eigs" (resolve_eigenscript_file_from): CWD first, then the + * base file's dir and its parent, the exe's install root, the user + * install. Realpath-deduplicated; first module found wins, as at runtime. + * Shared by W021 and the --api index (#734). Returns the number of + * resolved dirs written into `out` (each 4096 bytes). */ +static int lib_candidate_dirs(const char *base, char (*out)[4096], int max) { char cand[6][4096]; int nc = 0; snprintf(cand[nc++], 4096, "lib"); - snprintf(cand[nc++], 4096, "%.4000s/lib", base); - snprintf(cand[nc++], 4096, "%.4000s/../lib", base); + if (base && base[0]) { + snprintf(cand[nc++], 4096, "%.4000s/lib", base); + snprintf(cand[nc++], 4096, "%.4000s/../lib", base); + } if (g_exe_dir[0]) { snprintf(cand[nc++], 4096, "%.4000s/../lib", g_exe_dir); snprintf(cand[nc++], 4096, "%.4000s/../lib/eigenscript", g_exe_dir); @@ -982,18 +989,261 @@ static void w021_build(const char *base) { if (home) snprintf(cand[nc++], 4096, "%.2000s/.local/lib/eigenscript", home); - char seen[6][4096]; int ns = 0; - for (int i = 0; i < nc; i++) { + for (int i = 0; i < nc && ns < max; i++) { char real[4096]; if (!realpath(cand[i], real)) continue; int dup = 0; for (int j = 0; j < ns; j++) - if (strcmp(seen[j], real) == 0) { dup = 1; break; } + if (strcmp(out[j], real) == 0) { dup = 1; break; } + if (dup) continue; + snprintf(out[ns++], 4096, "%s", real); + } + return ns; +} + +/* Build (or reuse) the name table for the linted file's base dir. */ +static void w021_build(const char *base) { + if (g_w021_built && strcmp(g_w021_base, base) == 0) return; + w021_free_table(); + snprintf(g_w021_base, sizeof(g_w021_base), "%s", base); + g_w021_built = 1; + + char dirs[6][4096]; + int nd = lib_candidate_dirs(base, dirs, 6); + for (int i = 0; i < nd; i++) + w021_scan_dir(dirs[i]); +} + +/* ---- #734: --api — the machine-readable surface index ---- */ + +/* One call answers "does X exist, and is it builtin, extension, or lib": + * - "builtins": the compiled-in core registry, enumerated from a scratch + * register_builtins env — never a hand list (#459: the old hand-copied + * array drifted ~120 names behind the binary); + * - "extensions": the ext_names.h surface BY GROUP (gfx/http/db/model/ + * net) — the surface of the LANGUAGE, independent of this binary's + * build flags (a name here may need `make http`/`make gfx`); + * - "lib": every public top-level define of lib/*.eigs WITH its + * parameter list as written (defaults included), scraped from the same + * directories the import resolver searches, first module wins. + * Name resolution only — calling conventions live in docs/BUILTINS.md + * (builtins/extensions) and docs/STDLIB.md (lib). */ + +typedef struct { char *module; char *name; char *params; } ApiLibFn; + +static int api_mod_seen(char **mods, int count, const char *m) { + for (int i = 0; i < count; i++) + if (strcmp(mods[i], m) == 0) return 1; + return 0; +} + +static void api_scan_file(const char *dirpath, const char *filename, + ApiLibFn **fns, int *fc, int *fcap, + char ***mods, int *mc, int *mcap) { + size_t nl = strlen(filename); + if (nl <= 5 || nl - 5 >= 512) return; + char module[512]; + memcpy(module, filename, nl - 5); + module[nl - 5] = '\0'; + if (api_mod_seen(*mods, *mc, module)) return; /* higher dir won */ + + char full[8192]; + snprintf(full, sizeof(full), "%.4000s/%.500s", dirpath, filename); + FILE *f = fopen(full, "r"); + if (!f) return; + if (*mc == *mcap) { + *mcap = *mcap ? *mcap * 2 : 64; + *mods = xrealloc(*mods, (size_t)*mcap * sizeof(char *)); + } + (*mods)[(*mc)++] = xstrdup(module); + + char line[2048]; + while (fgets(line, sizeof(line), f)) { + if (strncmp(line, "define ", 7) != 0) continue; + const char *p = line + 7; + char name[256]; + size_t o = 0; + while (w021_ident_char(*p) && o + 1 < sizeof(name)) name[o++] = *p++; + name[o] = '\0'; + if (o == 0 || name[0] == '_') continue; /* private by convention */ + /* Parameter list as written; a define with no paren list carries + * the implicit single parameter `n` (parser.c) — say so. */ + char params[1024]; + if (*p == '(') { + p++; + size_t q = 0; + while (*p && *p != ')' && q + 1 < sizeof(params)) params[q++] = *p++; + params[q] = '\0'; + } else { + snprintf(params, sizeof(params), "n"); + } + int dup = 0; /* within-module redefine: first wins */ + for (int i = 0; i < *fc; i++) + if (strcmp((*fns)[i].module, module) == 0 && + strcmp((*fns)[i].name, name) == 0) { dup = 1; break; } if (dup) continue; - snprintf(seen[ns++], 4096, "%s", real); - w021_scan_dir(real); + if (*fc == *fcap) { + *fcap = *fcap ? *fcap * 2 : 256; + *fns = xrealloc(*fns, (size_t)*fcap * sizeof(ApiLibFn)); + } + (*fns)[*fc].module = xstrdup(module); + (*fns)[*fc].name = xstrdup(name); + (*fns)[*fc].params = xstrdup(params); + (*fc)++; } + fclose(f); +} + +/* Emit one params string as a JSON array of trimmed comma-split entries. */ +static void api_emit_params_json(FILE *out, const char *params) { + fprintf(out, "["); + const char *p = params; + int first = 1; + while (*p) { + while (*p == ' ' || *p == '\t') p++; + const char *start = p; + while (*p && *p != ',') p++; + const char *end = p; + while (end > start && (end[-1] == ' ' || end[-1] == '\t')) end--; + if (end > start) { + char raw[512], esc[1100]; + size_t l = (size_t)(end - start); + if (l >= sizeof(raw)) l = sizeof(raw) - 1; + memcpy(raw, start, l); + raw[l] = '\0'; + json_escape(raw, esc, sizeof(esc)); + fprintf(out, "%s\"%s\"", first ? "" : ", ", esc); + first = 0; + } + if (*p == ',') p++; + } + fprintf(out, "]"); +} + +/* Is `nm` an extension-surface name (any ext_names.h group)? Used to keep + * the builtins list build-INDEPENDENT: in a build that compiled an + * extension in (e.g. make net registers net_dial), that name is in the + * core env too, but --api must still report it as an extension, not a + * plain builtin — the ext groups are the LANGUAGE surface, not this + * binary's flags, so the same name lands in the same section every build. */ +static int api_is_ext_name(const char *nm) { +#define API_ISX(n, fn) if (strcmp(nm, #n) == 0) return 1; + EIGS_GFX_BUILTINS(API_ISX) + EIGS_HTTP_BUILTINS(API_ISX) + EIGS_HTTP_REQUEST_BUILTINS(API_ISX) + EIGS_DB_BUILTINS(API_ISX) + EIGS_MODEL_BUILTINS(API_ISX) + EIGS_NET_BUILTINS(API_ISX) +#undef API_ISX + return 0; +} + +int eigs_api_dump(FILE *out, int json) { + /* Core registry on a scratch env (#459: never a hand list). */ + Env *core = env_new(NULL); + register_builtins(core); + register_store_builtins(core); + + /* Lib scan over the resolver's candidate dirs. */ + ApiLibFn *fns = NULL; + int fc = 0, fcap = 0; + char **mods = NULL; + int mc = 0, mcap = 0; + char dirs[6][4096]; + int nd = lib_candidate_dirs(".", dirs, 6); + for (int d = 0; d < nd; d++) { + struct dirent **ents = NULL; + int n = scandir(dirs[d], &ents, NULL, alphasort); + if (n < 0) continue; + for (int i = 0; i < n; i++) { + const char *nm = ents[i]->d_name; + size_t l = strlen(nm); + if (l > 5 && strcmp(nm + l - 5, ".eigs") == 0) + api_scan_file(dirs[d], nm, &fns, &fc, &fcap, + &mods, &mc, &mcap); + free(ents[i]); + } + free(ents); + } + + if (json) { + fprintf(out, "{\"version\": \"%s\",\n \"builtins\": [", + EIGENSCRIPT_VERSION); + int b_first = 1; + for (int i = 0; i < core->count; i++) { + char esc[600]; + if (!core->names[i] || api_is_ext_name(core->names[i])) continue; + json_escape(core->names[i], esc, sizeof(esc)); + fprintf(out, "%s\"%s\"", b_first ? "" : ", ", esc); + b_first = 0; + } + fprintf(out, "],\n \"extensions\": {"); + int g_first = 1; +#define API_GROUP(label, LIST) do { \ + fprintf(out, "%s\"%s\": [", g_first ? "" : ", ", label); \ + g_first = 0; \ + int e_first = 1; \ + LIST(API_X) \ + fprintf(out, "]"); \ + } while (0) +#define API_X(nm, fn) do { \ + fprintf(out, "%s\"%s\"", e_first ? "" : ", ", #nm); \ + e_first = 0; \ + } while (0); + API_GROUP("gfx", EIGS_GFX_BUILTINS); + API_GROUP("http", EIGS_HTTP_BUILTINS); + API_GROUP("http_request", EIGS_HTTP_REQUEST_BUILTINS); + API_GROUP("db", EIGS_DB_BUILTINS); + API_GROUP("model", EIGS_MODEL_BUILTINS); + API_GROUP("net", EIGS_NET_BUILTINS); +#undef API_X +#undef API_GROUP + fprintf(out, "},\n \"lib\": ["); + for (int i = 0; i < fc; i++) { + char me[600], ne[600]; + json_escape(fns[i].module, me, sizeof(me)); + json_escape(fns[i].name, ne, sizeof(ne)); + fprintf(out, "%s\n {\"module\": \"%s\", \"name\": \"%s\", " + "\"params\": ", i ? "," : "", me, ne); + api_emit_params_json(out, fns[i].params); + fprintf(out, "}"); + } + fprintf(out, "\n]}\n"); + } else { + for (int i = 0; i < core->count; i++) + if (core->names[i] && !api_is_ext_name(core->names[i])) + fprintf(out, "builtin %s\n", core->names[i]); +#define API_X(nm, fn) \ + fprintf(out, "extension %s %s\n", api_group_label, #nm); +#define API_GROUP(label, LIST) do { \ + const char *api_group_label = label; \ + (void)api_group_label; \ + LIST(API_X) \ + } while (0) + API_GROUP("gfx", EIGS_GFX_BUILTINS); + API_GROUP("http", EIGS_HTTP_BUILTINS); + API_GROUP("http_request", EIGS_HTTP_REQUEST_BUILTINS); + API_GROUP("db", EIGS_DB_BUILTINS); + API_GROUP("model", EIGS_MODEL_BUILTINS); + API_GROUP("net", EIGS_NET_BUILTINS); +#undef API_GROUP +#undef API_X + for (int i = 0; i < fc; i++) + fprintf(out, "lib %s.%s(%s)\n", + fns[i].module, fns[i].name, fns[i].params); + } + + for (int i = 0; i < fc; i++) { + free(fns[i].module); free(fns[i].name); free(fns[i].params); + } + free(fns); + for (int i = 0; i < mc; i++) free(mods[i]); + free(mods); + /* LeakSanitizer cannot trace the env's tagged slots — free explicitly + * (same rule as builtin_name_env_free above). */ + env_decref(core); + return 0; } /* Module names the linted file imported (AST-owned strings). */ diff --git a/src/main.c b/src/main.c index 1708cde..12fa51d 100644 --- a/src/main.c +++ b/src/main.c @@ -93,6 +93,8 @@ int main(int argc, char **argv) { " eigenscript --lint [--json] [--lint-level error|warning] \n" " lint a source file (--json machine-readable; --lint-level error = warnings advisory;\n" " '# lint: allow W001' suppresses a code inline)\n" + " eigenscript --api [--json] surface index: every builtin, extension (by group), and\n" + " lib function with params — 'does X exist' in one call\n" " eigenscript --test [paths...] run test_*.eigs files (--json for machine-readable results)\n" " ... --trace-on-fail record each test; a failure prints its EIGS_REPLAY reproducer\n" " eigenscript --trace run and record a replay tape (CLI twin of EIGS_TRACE)\n" @@ -218,6 +220,18 @@ int main(int argc, char **argv) { return rc; } + /* #734 --api: the machine-readable surface index. Answers "does X + * exist, and is it builtin / extension / lib" in one call instead of + * a text search across BUILTINS.md + STDLIB.md. Same shape as --lint: + * --json for machines, plain lines (greppable) otherwise. */ + if (argc >= 2 && strcmp(argv[1], "--api") == 0) { + int api_json = (argc >= 3 && strcmp(argv[2], "--json") == 0); + int rc = eigs_api_dump(stdout, api_json); + eigs_thread_detach(); + eigs_state_destroy(eigs_st); + return rc; + } + /* --pkg flag: dispatch to lib/pkg.eigs with the rest of the argv as * its args. `args of null` returns argv[2:], so the EigenScript * side sees [subcommand, ...rest]. The script reads ./eigs.json and diff --git a/tests/run_all_tests.sh b/tests/run_all_tests.sh index 5b35cb7..64570b7 100755 --- a/tests/run_all_tests.sh +++ b/tests/run_all_tests.sh @@ -3425,6 +3425,53 @@ else fi echo "" +# [131] --api surface index (#734). One call answers "does X exist, and is +# it builtin / extension / lib". Builtins come from the live registry +# (never a hand list, #459), extensions from ext_names.h by group, lib +# functions from lib/*.eigs with their parameter lists. Pins the three +# kinds, the params capture, the sort_by-is-a-builtin / filter-is-lib +# split that misled agents, and that --json is valid JSON when python3 +# is present. +echo "[131] --api Surface Index (#734)" +API_OUT=$(./eigenscript --api 2>&1); API_RC=$? +TOTAL=$((TOTAL + 1)) +if [ "$API_RC" = "0" ] \ + && echo "$API_OUT" | grep -q "^builtin sort_by$" \ + && echo "$API_OUT" | grep -q "^extension net net_dial$" \ + && echo "$API_OUT" | grep -q "^lib list.filter(items, fn)$" \ + && ! echo "$API_OUT" | grep -q "^builtin filter$"; then + PASS=$((PASS + 1)) + echo " PASS: builtin/extension/lib kinds + params + the filter/sort_by split" +else + FAIL=$((FAIL + 1)) + echo " FAIL: --api text surface (rc=$API_RC)" + echo "$API_OUT" | head -3 +fi +TOTAL=$((TOTAL + 1)) +if command -v python3 >/dev/null 2>&1; then + API_JSON_OK=$(./eigenscript --api --json 2>/dev/null | python3 -c " +import json, sys +d = json.load(sys.stdin) +ok = ('sort_by' in d['builtins'] + and 'filter' not in d['builtins'] + and 'net_dial' in d['extensions']['net'] + and any(e['module'] == 'list' and e['name'] == 'filter' + and e['params'] == ['items', 'fn'] for e in d['lib'])) +print('OK' if ok else 'BAD') +" 2>&1) + if [ "$API_JSON_OK" = "OK" ]; then + PASS=$((PASS + 1)) + echo " PASS: --api --json parses and carries the same facts" + else + FAIL=$((FAIL + 1)) + echo " FAIL: --api --json ($API_JSON_OK)" + fi +else + PASS=$((PASS + 1)) + echo " SKIP (counted as pass): python3 not available for JSON validation" +fi +echo "" + # [89] Executable documentation — every eigenscript/output block pair in # docs/SPEC.md and docs/COMPARISON.md runs and must match exactly, so # the spec cannot drift from the implementation. Skips without python3.