You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This actually might be tricky to get right, since our tokens in lfortran are based on just a regex, so a "program" keyword and a variable are both of type "keyword". To do that, we need to have the correct type for each token. We know the correct parsing at the AST level, but we do not have tokens anymore.
It is also possible to obtain the correct tokens using our fixed-form tokenizer which is a recursive descent parser. We could write a similar parser for free-form also. However unless we use it for the actual parsing, it would be duplicate work, so not sure it is worth it.
So we can start by providing a list of tokens and their types, although some variable names might be mistyped as keywords, since that's what the tokenizer sees.
The text was updated successfully, but these errors were encountered:
This actually might be tricky to get right, since our tokens in lfortran are based on just a regex, so a "program" keyword and a variable are both of type "keyword". To do that, we need to have the correct type for each token. We know the correct parsing at the AST level, but we do not have tokens anymore.
It is also possible to obtain the correct tokens using our fixed-form tokenizer which is a recursive descent parser. We could write a similar parser for free-form also. However unless we use it for the actual parsing, it would be duplicate work, so not sure it is worth it.
So we can start by providing a list of tokens and their types, although some variable names might be mistyped as keywords, since that's what the tokenizer sees.
The text was updated successfully, but these errors were encountered: