Can we avoid the need for the hard cut operator?

[Nadrieril]

The Notation section says:

The hard cut operator is necessary because some tokens in Rust begin with a prefix that is itself a valid token. [...] Without the cut, if c"\0" failed to lex as a C string literal (because null bytes are not allowed in C strings), the parser could backtrack and lex it as two tokens [...]

My question is: why do we exclude null bytes in string literals as part of lexing instead of as a separate "c strings may not contain null bytes" rule? Could this make an observable different to e.g. macros? I'd quite like to remove that operator to make the grammar simpler.

[ehuss]

Unfortunately that is quite difficult. One of the goals is to have a fully declarative grammar (in this case using PEG) without the need for tacking on additional things (like rules written in English, or some other language like Rust). This allows us to validate the grammar and to concisely describe exactly what the grammar is.

Macros are often a reason these kinds of things are observable, as well as other APIs that do tokenization or parsing (like proc-macro APIs), and also things like #[cfg] (if you don't consider that a macro).

There were a variety of alternatives we considered, but they were either unworkable or added significant complexity.

Is there a particular tool or situation you're trying to work with?

[Nadrieril]

Yeah, I'm turning the grammar into a real parser as part of my desugaring-based specification experiment, and I want to make it as legible as possible. In my case it's quite natural to include extra post-processing in Rust and I think a better choice for understandability if I can entirely avoid the hard cut operator.