BUG?: Comments and string literals

[pms1969]

Describe the bug
I have a grammar that includes the capability to use single line comments like

// this is a comment

and string literals inside double quotes or in the rust style raw strings; like

"fist quote of the day"
r#"rust style raw string where there can be anynumber of # to start and close the string"#

when I put a url inside the string literal, I get a parsing error. in fact, it need only be what the comment start is to make this error; so "//" and r#"//"# both error.

To Reproduce
Steps to reproduce the behavior:
A minimal grammar looks like so:

string_literal = { raw_string | "\"" ~ string_content ~ "\"" }
string_content = { (!("\"") ~ ANY)* }

// raw strings was taken directly from 
// https://pest.rs/book/grammars/syntax.html#the-stack-wip
raw_string = _{
    "r" ~ PUSH("#"*) ~ "\""    // push the number signs onto the stack
    ~ raw_string_interior
    ~ "\"" ~ POP               // match a quotation mark and the number signs
}

raw_string_interior = {
    (
        !("\"" ~ PEEK)    // unless the next character is a quotation mark
                          // followed by the correct amount of number signs,
        ~ ANY             // consume one character
    )*
}

WHITESPACE = _{ " " | "\t" | NEWLINE }
COMMENT = _{ "//" ~ (!NEWLINE ~ ANY)* }

and a test to see it in action:

use pest_derive::Parser;

extern crate pest;

#[derive(Parser)]
#[grammar = "grammar.pest"]
pub struct MyParser;

#[cfg(test)]
mod string_tests {
    use pest::Parser;

    use super::*;
    fn check(input: &str) {
        let parse_tree = MyParser::parse(Rule::string_literal, input);
        match parse_tree {
            Ok(_) => {}
            Err(e) => {
                println!("{:#?}", e);
                assert!(false)
            }
        }
    }

    #[test]
    fn can_parse() {
        let input = r##""this is a test""##;
        check(input);

        let input = r##""try something /\/ new""##;
        check(input);

        let input = r##""//""##;
        check(input);
    }
}

Expected behavior
A clear and concise description of what you expected to happen.
I would expect the parsing to succeed and a string with the // contents to be returned in the pair.

Additional context
Add any other context about the problem here.

[pms1969]

oh, just noticed the editor on the website;
so here is my problem link
and if I just remove the capitalisation for COMMENT, it works. I know that's because COMMENT is special. I don't think it should interfere with any other token tho.

[tomtau]

I assume it's likely due to the greedy PEG nature in combination with that COMMENT rule; you can try https://github.com/pest-parser/awesome-pest/blob/main/debugger.md to see what's going on

[pms1969]

OK, so leaving out the stuff that adds no value, this is what I get using the provided grammar

❯ pest_debugger
pest_debugger is up to date.
pest_debugger v2.6.0

> g src/grammar.pest
> id "https://this.should.not.be/a/comment"
> r string_literal
 --> 1:1
  |
1 | "https://this.should.not.be/a/comment"
  | ^---
  |
  = parsing string_literal
> c
 --> 1:1
  |
1 | "https://this.should.not.be/a/comment"
  | ^---
  |
  = expected "string_literal"
> ba
> r string_literal
 --> 1:1
  |
1 | "https://this.should.not.be/a/comment"
  | ^---
  |
  = parsing string_literal
> c
 --> 1:1
  |
1 | "https://this.should.not.be/a/comment"
  | ^---
  |
  = parsing raw_string
> c
 --> 1:2
  |
1 | "https://this.should.not.be/a/comment"
  |  ^---
  |
  = parsing WHITESPACE
> c
 --> 1:2
  |
1 | "https://this.should.not.be/a/comment"
  |  ^---
  |
  = parsing COMMENT
> c
 --> 1:2
  |
1 | "https://this.should.not.be/a/comment"
  |  ^---
  |
  = parsing string_content
> c
 --> 1:2
  |
1 | "https://this.should.not.be/a/comment"
  |  ^---
  |
  = parsing WHITESPACE
> c
 --> 1:2
  |
1 | "https://this.should.not.be/a/comment"
  |  ^---
  |
  = parsing COMMENT
...
> c
 --> 1:8
  |
1 | "https://this.should.not.be/a/comment"
  |        ^---
  |
  = parsing WHITESPACE
> c
 --> 1:8
  |
1 | "https://this.should.not.be/a/comment"
  |        ^---
  |
  = parsing COMMENT
> c
 --> 1:39
  |
1 | "https://this.should.not.be/a/comment"
  |                                       ^---
  |
  = parsing WHITESPACE
> c
 --> 1:39
  |
1 | "https://this.should.not.be/a/comment"
  |                                       ^---
  |
  = parsing COMMENT
> c
 --> 1:39
  |
1 | "https://this.should.not.be/a/comment"
  |                                       ^---
  |
  = parsing WHITESPACE
> c
 --> 1:39
  |
1 | "https://this.should.not.be/a/comment"
  |                                       ^---
  |
  = parsing COMMENT
> c
 --> 1:8
  |
1 | "https://this.should.not.be/a/comment"
  |        ^---
  |
  = parsing WHITESPACE
> c
 --> 1:8
  |
1 | "https://this.should.not.be/a/comment"
  |        ^---
  |
  = parsing COMMENT
> c
 --> 1:39
  |
1 | "https://this.should.not.be/a/comment"
  |                                       ^---
  |
  = parsing WHITESPACE
> c
 --> 1:39
  |
1 | "https://this.should.not.be/a/comment"
  |                                       ^---
  |
  = parsing COMMENT
> c
 --> 1:1
  |
1 | "https://this.should.not.be/a/comment"
  | ^---
  |
  = expected "string_literal"
> c
Error: End-of-input reached

My rather limited analysis of this is that it recognises the double quote, starts parsing the string content, but still looks for WHITESPACE and COMMENT; and presumably any other specially treated tokens. It's doing that while being asked to check for an ANY character as part of the string content.
I think there's a pretty strong case to suspend examining for WHITESPACE and COMMENT while searching for an ANY, but not really sure what else that would break.

[pms1969]

OK, I think I've got to the bottom of this. If i make the string_literal rule atomic, it prevents the processing. You have got that in the documents, so I'll just slink away into my corner of the internet. Thanks for the pointer.

[skull-squadron]

I also just encountered this surprise. This behavior should be made clearer at the beginning of the introduction and include a table or diagram about behavior options.