parser: leftover-token hints diagnose the actual mistake (#732)

CHANGELOG.md

@@ -6,6 +6,28 @@ All notable changes to EigenScript are documented here.
 
 ### Fixed
 
+- **The leftover-token parse error diagnoses the actual mistake instead
+  of always saying "one statement per line" (#732).** The four most
+  likely model/newcomer mistakes — `print("hello")`, `def f(a):`,
+  `f is x => x + 1`, `say x` — all landed in `p_end_statement`, whose
+  blanket hint named a rule none of them broke (and `docs/llms.txt`
+  taught the same misdiagnosis, so the reference and the diagnostic
+  wrongly confirmed each other). The hint is now specialized on the
+  leftover/predecessor tokens: paren-style calls get "calls use 'of':
+  write f of x", a bare `=>` gets the parenthesized-lambda form, a
+  lone-identifier statement gets a name-check hint (with `def`
+  specifically pointed at `define … as:`), and everything else keeps the
+  original text — `x is 2 x is 3` still says "one statement per line".
+  Also from the same pass: a statement that already reported a parse
+  error no longer stacks a spurious "unexpected X after statement" on
+  its debris (`add2 of (1, 2)` now gets exactly one error, the real
+  one), and the llms.txt validation-ladder entries for this signature
+  now describe the specialized hints (including that the caret sits one
+  token AFTER the mistake). The recorded `--lint --json`/LSP message is
+  unchanged (`unexpected X after statement`) — the hint is
+  display-layer. Five new `examples/errors/` demos pin the hints, plus
+  an `# expect-error-count:` harness extension pinning the
+  single-error cascade behavior (suite [90], now 19 checks).
 - **`json_decode` no longer reads past the allocation on a string ending
   in an escape backslash (#776).** The escape branch stepped onto the
   terminating NUL, appended it, and the loop guard then read one byte

docs/llms.txt

@@ -161,13 +161,20 @@ right" is often wrong at exactly the traps above. Grade it mechanically, cheapes
 check first — this is the same parse→compile→sandbox ladder iLambdaAi uses:
 
 1. **Parse / lint** (no execution): `eigenscript --lint --json prog.eigs`.
-   Empty `[]` = clean; `E002` = parse error (usually two statements on one line,
-   or a soft-keyword bound with `is`); `W###` = warnings (`--lint-level error`
+   Empty `[]` = clean; `E002` = parse error — the stderr hint names the likely
+   fix (a paren-style call `f(x)` → `f of x`, `def` → `define … as:`,
+   unparenthesized lambda params `x => …` → `(x) => …`, an unknown word used
+   like a keyword, a soft-keyword bound with `is`, or genuinely two statements
+   on one line); `W###` = warnings (`--lint-level error`
    to keep them advisory, `# lint: allow W001` to suppress one inline).
 2. **Run it**: `eigenscript prog.eigs`. The three failure signatures:
    `undefined variable` (a missing `local`, or a scope/soft-keyword surprise),
-   `unexpected … after statement` (two statements on one line), and **silently
-   wrong numbers** (an `f of [x]` that didn't spread — no error, bad output).
+   `unexpected … after statement` (READ THE HINT after the dash — it
+   diagnoses the specific cause: paren-style calls, `def`, bare lambda
+   params, an unknown statement word, or two statements on one line; the
+   caret sits on the leftover token, i.e. one token AFTER the mistake), and
+   **silently wrong numbers** (an `f of [x]` that didn't spread — no error,
+   bad output).
 3. **Test suites**: name files `test_*.eigs`, `assert of [cond, msg]` on
    failure, and run `eigenscript --test [dir]`. Add `--trace-on-fail` and a
    failing test prints an `EIGS_REPLAY=<tape> …` line that reproduces it

examples/errors/bare_lambda_params.eigs

@@ -0,0 +1,5 @@
+# ERROR DEMO — unparenthesized lambda parameter (#732).
+# `x => x + 1` is the JS reflex; EigenScript lambda params need parens.
+# The statement parses as `f is x`, leaving `=>` as the leftover token.
+# expect-error: lambda parameters need parentheses: (x) => ...
+f is x => x + 1

examples/errors/def_not_define.eigs

@@ -0,0 +1,7 @@
+# ERROR DEMO — `def` instead of `define ... as:` (#732).
+# `def` is not a keyword, so the statement parses as a lone identifier and
+# the function name is the leftover token; the hint now points at the
+# actual definition syntax instead of "one statement per line".
+# expect-error: function definitions are written 'define f(...) as:'
+def f(a):
+    return a + 1

examples/errors/paren_call.eigs

@@ -0,0 +1,6 @@
+# ERROR DEMO — paren-style call (#732).
+# The single most likely model/newcomer mistake: Python/JS call syntax.
+# The old blanket hint said "one statement per line", which names a rule
+# this input did not break; the hint now states the actual fix.
+# expect-error: calls use 'of', not parentheses: write print of x
+print("hello")

examples/errors/paren_tuple_call.eigs

@@ -0,0 +1,11 @@
+# ERROR DEMO — parenthesized "tuple" argument list (#732 note 1).
+# `add2 of (1, 2)` gets exactly ONE error: the real one. Before #732 the
+# leftover tokens after the failed expression re-triggered a second,
+# spurious "unexpected number after statement" with the misleading
+# one-statement-per-line hint; p_end_statement now resyncs silently when
+# the statement already reported.
+# expect-error: expected ')', got ','
+# expect-error-count: 1
+define add2(a, b) as:
+    return a + b
+add2 of (1, 2)

examples/errors/unknown_statement_word.eigs

@@ -0,0 +1,6 @@
+# ERROR DEMO — a bare word used like a statement keyword (#732).
+# `say` is not a builtin or keyword; the statement parses as the lone
+# identifier `say` and `x` is the leftover token. The hint suggests the
+# call form and a name check instead of "one statement per line".
+# expect-error: 'say' is not a statement keyword; a call is written 'say of x'
+say x

src/parser.c

@@ -83,16 +83,71 @@ static void p_print_caret(int line, int col) {
     eigs_print_caret_src(stderr, g_parse_caret_src, line, col);
 }
 
+/* #732: statement-entry snapshots, set by parse_statement_inner.
+ * g_stmt_err_base lets p_end_statement tell "this statement already
+ * reported" (suppress the cascade); g_stmt_start_pos lets it tell "the
+ * whole statement was one bare identifier" (the `say x` / `def f(a):`
+ * shapes). __thread like the lexer/JSON parser state — eval can run on
+ * worker threads. */
+static __thread int g_stmt_err_base  = 0;
+static __thread int g_stmt_start_pos = -1;
+
 static void p_end_statement(Parser *p) {
     if (p_match(p, TOK_NEWLINE)) return;
     TokType t = p_cur(p)->type;
     if (t == TOK_EOF || t == TOK_DEDENT) return;
     Token *tok = p_cur(p);
+    /* #732: if this statement already reported a parse error, the leftover
+     * tokens are that error's debris, not a second statement — resync
+     * silently instead of stacking a spurious "unexpected X after
+     * statement" on top (the `add2 of (1, 2)` shape: "expected ')', got
+     * ','" was followed by exactly that). */
+    if (g_parse_errors > g_stmt_err_base) {
+        while (p_cur(p)->type != TOK_NEWLINE && p_cur(p)->type != TOK_EOF &&
+               p_cur(p)->type != TOK_DEDENT)
+            p_advance(p);
+        p_match(p, TOK_NEWLINE);
+        return;
+    }
+    /* #732: specialize the hint on what is actually in hand. The four most
+     * likely model-generated mistakes (paren calls, `def`, bare-word
+     * statements, unparenthesized lambda params) all land here, and the
+     * blanket "one statement per line" named a rule none of them broke —
+     * while docs/llms.txt taught the same misdiagnosis, so the reference
+     * and the diagnostic wrongly confirmed each other. */
+    Token *prev = (p->pos > 0) ? &p->tl->tokens[p->pos - 1] : NULL;
+    const char *prev_name = (prev && prev->type == TOK_IDENT && prev->str_val)
+                                ? prev->str_val : NULL;
+    int lone_ident_stmt = prev_name && (p->pos == g_stmt_start_pos + 1);
+    char hint[192];
+    if (t == TOK_LPAREN && prev_name) {
+        snprintf(hint, sizeof hint,
+                 " — calls use 'of', not parentheses: write %.32s of x "
+                 "(or %.32s of [a, b] for several arguments)",
+                 prev_name, prev_name);
+    } else if (t == TOK_ARROW) {
+        snprintf(hint, sizeof hint,
+                 " — lambda parameters need parentheses: (x) => ...");
+    } else if (t == TOK_IDENT && lone_ident_stmt &&
+               strcmp(prev_name, "def") == 0) {
+        snprintf(hint, sizeof hint,
+                 " — function definitions are written 'define %.32s(...) as:'",
+                 tok->str_val ? tok->str_val : "name");
+    } else if (t == TOK_IDENT && lone_ident_stmt) {
+        snprintf(hint, sizeof hint,
+                 " — '%.32s' is not a statement keyword; a call is written "
+                 "'%.32s of %.32s', and check the name for typos",
+                 prev_name, prev_name, tok->str_val ? tok->str_val : "x");
+    } else {
+        snprintf(hint, sizeof hint, " — one statement per line");
+    }
     fprintf(stderr, "Parse error line %d:%d: unexpected %s after statement",
             tok->line, tok->col + 1, tok_type_name(tok->type));
     if (tok->str_val) fprintf(stderr, " ('%s')", tok->str_val);
-    fprintf(stderr, " — one statement per line\n");
+    fprintf(stderr, "%s\n", hint);
     {
+        /* The recorded (--lint --json / LSP) message stays the stable
+         * "unexpected X after statement" — the hint is display-layer. */
         char m[160];
         snprintf(m, sizeof(m), "unexpected %s after statement",
                  tok_type_name(tok->type));
@@ -1330,6 +1385,14 @@ static ASTNode* parse_statement_inner(Parser *p) {
 
     if (t->type == TOK_EOF || t->type == TOK_DEDENT) return NULL;
 
+    /* #732: snapshots for p_end_statement's cascade guard + hint
+     * specialization. Nested statements (block bodies) overwrite these;
+     * that is fine — each p_end_statement call belongs to the innermost
+     * statement being terminated, and a stale-low err_base after an
+     * erroring nested block only suppresses a lower-confidence cascade. */
+    g_stmt_err_base  = g_parse_errors;
+    g_stmt_start_pos = p->pos;
+
     if (t->type == TOK_DEFINE) {
         p_advance(p);
         Token *name_tok = p_cur(p);

tests/run_all_tests.sh

@@ -3449,7 +3449,7 @@ echo ""
 
 # [90] Error examples — examples/errors/*.eigs must exit nonzero and
 # print their declared '# expect-error:' message.
-echo "[90] Error Examples (14 checks)"
+echo "[90] Error Examples (19 checks)"
 ERR_OUTPUT=$(bash "$TESTS_DIR/test_error_examples.sh" 2>&1)
 ERR_PASS=$(echo "$ERR_OUTPUT" | grep -c "  PASS:" || true)
 ERR_FAIL=$(echo "$ERR_OUTPUT" | grep -c "  FAIL:" || true)

tests/test_error_examples.sh

@@ -25,6 +25,10 @@ for f in "$ROOT"/examples/errors/*.eigs; do
     #     # expect-caret: <exact caret line, from the pipe on>
     # so a column-attribution regression moves the caret and fails here.
     caret=$(grep -m1 "^# expect-caret:" "$f" | sed 's/^# expect-caret: //')
+    # #732: cascade cases pin the NUMBER of "Parse error" lines with
+    #     # expect-error-count: <n>
+    # so a suppressed spurious follow-on error can't silently come back.
+    errcount=$(grep -m1 "^# expect-error-count:" "$f" | sed 's/^# expect-error-count: //')
     out=$("$EIGS" "$f" </dev/null 2>&1)
     rc=$?
     if [ "$rc" = "0" ]; then
@@ -35,6 +39,12 @@ for f in "$ROOT"/examples/errors/*.eigs; do
         echo "    expected substring: $expected"
         printf '%s\n' "$out" | head -3 | sed 's/^/    got: /'
         FAIL=$((FAIL + 1))
+    elif [ -n "$errcount" ] && \
+         [ "$(printf '%s\n' "$out" | grep -c 'Parse error')" != "$errcount" ]; then
+        echo "  FAIL: $name (wrong parse-error count)"
+        echo "    expected $errcount, got $(printf '%s\n' "$out" | grep -c 'Parse error')"
+        printf '%s\n' "$out" | head -4 | sed 's/^/    got: /'
+        FAIL=$((FAIL + 1))
     elif [ -n "$caret" ] && ! printf '%s' "$out" | grep -qF "$caret"; then
         echo "  FAIL: $name (caret not at the declared column)"
         echo "    expected caret line: $caret"