how to require at least one space between 2 tokens?

[shi-yan]

I want to match

"aaa bbb"

there must be at least one space between the two tokens "aaa" and "bbb"

the following works,

WHITESPACE = _{ " " | "\t" }
accept = ${"aaa" ~ WHITESPACE+  ~ "bbb" ~ NEWLINE}

but is very inconvenient, because $ disables implicit whitespaces. So I have to manually add whitespace+ between all tokens. (this is fine, but I also need to add comments /**/ , as according to the doc, $ disables both )

Then I tried this,

WHITESPACE = _{ " " | "\t" }
accept = {"aaa"  ~ "bbb" ~ NEWLINE}

this works too, however it also matches "aaabbb". I want to make sure there is at least one space.

In my opinion, the correct way should be

WHITESPACE = _{ " " | "\t" }
accept = {"aaa" ~ WHITESPACE  ~ "bbb" ~ NEWLINE}

I thought this would enforce at least one space between "aaa" and "bbb".

however I got parsing error:

let pairs = IdentParser::parse(Rule::accept, "aaa bbb
").unwrap_or_else(|e| panic!("{}", e));

thread 'main' panicked at ' --> 1:1
  |
1 | aaa bbb␊
  | ^---
  |
  = expected accept', src/main.rs:29:23

This I don't understand?

[siberianbluerobin]

Maybe it's too late, but the problem is the newline in the example code

let pairs = IdentParser::parse(Rule::accept, "aaa bbb
").unwrap_or_else(|e| panic!("{}", e));

I think that if you use raw strings the newline will be processed correctly, but aside from that the last grammar works correctly because I tested in pest editor

[dragazo]

@siberianbluerobin I just tried a simplified version of the last grammar in the editor and it fails to parse even when there is whitespace.

WHITESPACE = _{ " " | "\t" }
accept = { "aaa" ~ WHITESPACE ~ "bbb"}

aaa bbb

 --> 1:1
  |
1 | aaa bbb
  | ^---
  |
  = expected accept

It seems the ~ connection consumes all the whitespace greedily, so it fails to match WHITESPACE. If you change the rule to atomic and use WHITESPACE+ it works as intended, but as already mentioned, atomics are somewhat annoying if you want to allow whitespace... You said you used the last grammar unchanged and it worked?

[siberianbluerobin]

@dragazo My mistake, you are right, when I tested the grammar I didn't copy and paste just write short names for the identifiers

This works

MYWHITESPACE = _{ " " | "\t" }
accept = { "aaa" ~ MYWHITESPACE ~ "bbb"}

Sound like the WHITESPACE rule isn't replaced

[dragazo]

@siberianbluerobin Yeah, by not defining WHITESPACE that's basically atomic, which should work. I think I'll just start making normal non-atomic rules for most things and have atomic helpers for special cases like this. A bit annoying, but it'll do : /

[max-sixty]

I'm facing a similar situation.

Assuming we can't make rules both non-atomic, and silent, something that would be helpful for this is a rule type which is atomic only for that rule — no inheritance.

So:

accept = #{"aaa" ~ WHITESPACE+  ~ other_rule ~ NEWLINE}

...forces whitespace between aaa and whatever other_rule is, without disallowing whitespace in other_rule, and without having to make other_rule non-atomic.

If other_rule could be both non-atomic and silent, that would also be fine — it would there's lots of non-atomic rules as @dragazo suggests, but that's a fairly small cost.

[clouds56]

I'd like to use NEWLINE instead of ";" as statement delimiter,

FILE = { SOI ~ (stmt ~ NEWLINE)* ~ EOI }
WHITESPACE = _{ " " | "\t" | NEWLINE }

and it failed to consume newline between stmt,
with error

Parse(Error { variant: ParsingError { positives: [COMMENT], negatives: [] }, location: Pos(302), line_col: Pos((12, 1)), path: None, line: "_str1=\"0x120312\"", continued_line: None })

Is it an option to let parser take WHITESPACE non-greedily?

[Tartasprint]

I don't know if that is possible. But you could do something like:

FILE = @{ SOI ~ (stmt ~ NEWLINE)* ~ EOI }

and if you want your stmt not to be atomic mark it as non-atomic with:

stmt = !{...}

This should work if your stmt rule doesn't end by eating all the newlines.
A problem with the above solution is that it forces to have one empty line in the end of the file. To avoid that you can write this:

FILE = @{ SOI ~ (stmt ~ NEWLINE)* ~ stmt? ~ EOI }

Since the FILE rule is now atomic it doesn't allow whitespace everywhere as you may want. You can do something like:

FILE = @{ SOI ~ WHITESPACE ~ (stmt ~ NEWLINE ~ WHITESPACE)* ~ stmt? ~ WHITESPACE ~ EOI }

(and if you want a very little optimization, replace (stmt ~ NEWLINE)* ~ stmt? by (stmt ~ (NEWLINE~stmt)* | "") which should avoid parsing stmt twice in the last stmt)

[davawen]

Having space delimited identifiers is really common and it's a shame to have to write what amount to a huge amount of boilerplate if you want this effect.
I guess there wouldn't be this problem with a separate lexer/parser combo either...
I think the best way to fix this would be to allow specific ~ operators to not greedily consume whitespace, something like @~ or $~ ("atomic sequence"):

a = { "a" ~ "b" ~ "c" }
// "desugares" to
a = { "a" ~ (WHITESPACE | COMMENT)* ~ "b" ~ (WHITESPACE | COMMENT)* ~ "c" }

b = {"a" @~ "b" ~ "c"}
// "desugares" to
b = { "a" ~ "b" ~ (WHITESPACE | COMMENT)* ~ "c" }

For me what makes it heavy is the lack of fine grained atomicity control inside the rule itself, since I often only want this behaviour in one specific place.
Plus this change would be backwards compatible.