LALR(1) parser from official MySQL grammar#429
Merged
Merged
Conversation
Contributor
🤖 Lexer benchmarkChanges to lexer-related files were detected and triggered a benchmark:
Note: Hosted runners are noisy, and absolute numbers vary. Treat the results with caution and verify them locally. To reproduce locally: |
JanJakes
changed the title
JanJakes
force-pushed
the
lalr-parser
branch
9 times, most recently
from
df8874b to
70d642d
Compare
JanJakes
changed the title
JanJakes
force-pushed
the
lalr-parser
branch
4 times, most recently
from
b3c39da to
1f88932
Compare
Add a new monorepo package for a MySQL parser generated from the official MySQL grammar. This commit sets up the package metadata; the source, tooling, and documentation follow in later commits.
Bring the MySQL lexer and the token and node classes over from the mysql-on-sqlite package unchanged, so the later adaptation to the official grammar is reviewable as a focused diff, and register src/ as the package Composer classmap (the WordPress-style file names rule out PSR-4).
JanJakes
force-pushed
the
lalr-parser
branch
6 times, most recently
from
296a9c5 to
0f841c5
Compare
Compile the grammar from the official MySQL sources: fetch sql_yacc.yy and lex.h at a pinned, checksum-verified mysql-server tag; run a pinned Bison build (Docker, version-asserted) to produce the automaton; compact the automaton into plain PHP ACTION/GOTO tables (about 7% of the dense cells); and derive the keyword table and token constants from lex.h, failing the build on any unresolved terminal. bin/build-grammar (composer run build-grammar) runs the pipeline end to end.
JanJakes
force-pushed
the
lalr-parser
branch
2 times, most recently
from
a9f8619 to
aca2c38
Compare
Commit the LALR(1) parse table produced by bin/build-grammar: a plain PHP array that compacts the grammar's dense ACTION/GOTO automaton to about 7% of its cells. Regenerate with composer run build-grammar. The token-level data (keyword table, paren-gated function keywords, and token constants) is generated into the lexer itself; see the next commit.
Make the lexer emit the grammar's own token numbers, with the keyword table generated from lex.h: keyword synonyms, paren-gated function keywords, and dropped keywords all follow MySQL's own data. Diagnostic token names are derived on demand instead of shipping a name map. The lexer produces MySQL's grammar token stream directly, the way MySQL's own lexer does, rather than scanning a different token model and reconciling it in a separate pass: "@" is a standalone terminal followed by its name, "WITH ROLLUP" is contracted via a one-token lookahead, NOT becomes NOT2 under HIGH_NOT_PRECEDENCE, and the input ends with END_OF_INPUT and Bison's end marker (omitted on invalid input). The pull iterator (next_token/get_token) and remaining_tokens() both yield this single stream; the scanner's internal sentinels stay private and never reach it.
A table-driven LALR(1) shift-reduce runtime (WP_Parser) over a WP_Parser_Grammar that expands a compact, generated ACTION/GOTO parse table, building a WP_Parser_Node AST. The grammar is unambiguous for LALR(1), so the loop is deterministic, with no conflict handling or backtracking. A rule that matches nothing produces no node, so empty optional rules are absent from the tree. This is grammar-agnostic: it knows nothing about MySQL, only how to run an LALR(1) parse table. Adapt the copied parse-tree primitives to the package: the runtime builds each node in a single step, so the old recursive parser's merge_fragment() is dropped, and the node and token docblocks no longer reference that parser.
Wire the generated MySQL parse table into the generic LALR(1) runtime through a factory: WP_MySQL_Parser_Factory::create_parser() builds a WP_Parser over a WP_Parser_Grammar loaded from src/mysql-parse-table.php. The grammar is expanded once and shared between created parsers; create_grammar() exposes a fresh grammar for callers that want their own. This is the only piece that knows the parser is being used for MySQL.
Cut the generated parse table from 190 KB to 177 KB (-7%) with no behavior change: most shifts on a given terminal go to the same successor state, so those cells are stored as bare token lists (action_row_shift_tokens) and restored from a per-terminal target table (action_shift_targets) when the grammar is constructed. The smaller file also parses faster on a cold opcache.
Bring the query corpus extracted from the MySQL server test suite, with the tooling that generates it, into the package: data/mysql-server-query-corpus/ plus a bin/build-corpus orchestrator (composer run build-corpus) that fetches the mysql-test directory at the pinned tag and extracts the queries. The SQLite driver package keeps its own copy for now; it will be retired when the driver is ported to this package.
Measure the corpus parse rate and end-to-end (lex + parse) throughput, with warmup and timed passes. The parser accepts 99.76% of the ~69k corpus queries.
Cover the token stream, the scanner (the exhaustive unit suite ported from the SQLite driver), the parser runtime, token value and name resolution, generated grammar-data invariants, and a corpus regression test pinning the exact acceptance tally. Run the suite on the oldest and newest supported PHP versions in CI.
Tokenizing a whole statement routed every token through the pull iterator (next_token -> produce -> scan_lexeme -> read_next_token -> enqueue_token), adding ~4 method calls plus token-queue bookkeeping per token over a plain scan-and-emit loop. Give remaining_tokens() a tight fast path that emits the common single-token lexemes inline and delegates only the rare multi-token ones (@, WITH ROLLUP, end markers) to the buffered producers. The pull API is unchanged and the output is byte-identical; ~24% faster (no JIT) / ~16% (JIT) end-to-end over the MySQL server corpus.
The pull iterator buffered produced tokens in a dynamic $token_queue drained by index. A scan step yields at most two grammar tokens, so a single $pending_token slot suffices: next_token() returns the first and holds the second. The multi-token producers (@, WITH ROLLUP, end markers) now append to a caller-supplied array, shared directly by both next_token() and remaining_tokens() — removing the queue bookkeeping and the duplicated drain in the fast path. A make_token() helper unifies token construction. Output is byte-identical and throughput is unchanged (the multi-token cases were already off the hot path); this is a structural cleanup.
adamziel
reviewed
Jun 23, 2026
| list( $type, $start, $length ) = $this->lookahead; | ||
| $this->lookahead = null; | ||
| } else { | ||
| // Inlined scan_lexeme(): skip whitespace and comments, then scan. |
Collaborator
There was a problem hiding this comment.
we could remove all comments like that one, LLMs love to leave them for continuity between commits, but it doesn't really mean anything anymore
adamziel
reviewed
Jun 23, 2026
| // Token constants share the class with the lexer's own constants; | ||
| // grammar tokens are the non-negative ints that are not SQL modes | ||
| // (the scanner sentinels are negative, the SQL modes are bit flags). | ||
| foreach ( ( new ReflectionClass( self::class ) )->getConstants() as $name => $value ) { |
Collaborator
There was a problem hiding this comment.
Reflections look concerning, especially in such a prominent code path. What's the performance hit?
adamziel
reviewed
Jun 23, 2026
| $type = self::LESS_OR_EQUAL_OPERATOR; | ||
| } | ||
| } elseif ( '>' === $next_byte ) { | ||
| $this->bytes_already_read += 1; // Consume the '>'. |
Collaborator
There was a problem hiding this comment.
These comments seem pretty redundant.
adamziel
reviewed
Jun 23, 2026
| // in the range of U+0080 to U+FFFF before looking at further bytes. | ||
| // If it can't, bail out early to avoid unnecessary UTF-8 decoding. | ||
| // Identifiers are usually ASCII-only, so we can optimize for that. | ||
| $byte_1 = ord( |
Collaborator
There was a problem hiding this comment.
An inline utf-8 parser? :D Let's see if it makes any sense to adapt the one @dmsnell built for WordPress core, that's less maintenance, less opportunity to have an error, and probably more performance.
adamziel
reviewed
Jun 23, 2026
| } | ||
|
|
||
| private function read_number(): ?int { | ||
| // @TODO: Support numeric-only identifier parts after "." (e.g., 1ea10.1). |
Collaborator
There was a problem hiding this comment.
Should this be addressed before merging?
adamziel
reviewed
Jun 23, 2026
| * A backslash with any other character represents the character itself. | ||
| * That is, \x evaluates to x, \\ evaluates to \, and \🙂 evaluates to 🙂. | ||
| */ | ||
| $preg_quoted_backslash = preg_quote( $backslash ); |
Collaborator
There was a problem hiding this comment.
Can we just compute this on paper and inline it in the next line? :-)
adamziel
reviewed
Jun 23, 2026
| * https://github.com/mysql/mysql-workbench/blob/8.0.38/library/parsers/grammars/MySQLLexer.g4 | ||
| * https://github.com/mysql/mysql-workbench/blob/8.0.38/library/parsers/mysql/MySQLBaseLexer.cpp | ||
| */ | ||
| class WP_MySQL_Lexer { |
Collaborator
There was a problem hiding this comment.
I'd love to understand better how was this was adapted from the default lexer
Collaborator
|
Really good work here @JanJakes! I've left some non-blocking comments and will proceed with merging 🎉 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
The changed line numbers are misleading—about 115,000 added lines is just a testing query corpus.
(Copied to the new
mysql-parserpackage frommysql-on-sqlite.)LALR(1) parser from official MySQL grammar
A new experimental
packages/mysql-parserpackage that implements a universal LALR(1) parser and builds a MySQL parse table from the official MySQL grammar.This is the initial implementation, not used anywhere in the driver yet.
A full driver migration to this new parser is AI-prototyped in #432.
What it does
What it doesn't do yet
Benchmarks
Measured on MacBook Pro M4 Max on PHP 8.4, the package's 8.4.10 corpus ~70k queries, end-to-end (lex + parse), best of 5 timed passes after 2 warmups:
This parser is over 5× faster without JIT and over 4.5× faster with JIT. Cold boot is a bit slower; warm boot is faster. The memory footprint is a bit higher, and the overall size about 14 KB higher.
Recognize-only
The same lex+parse runs but building no AST, measuring only recognition without AST allocation:
Dropping AST construction lifts both by ~1.5–2×, but the gap stays around ~4.2–5.8×.