Should this be used for creating a programming language "from scratch"?

[devlzcode]

I want to try and attempt to create a basic programming language from scratch and I'm very interested in using pest but I'm coming to the conclusion that while this can be used for such purpose, its better off for simpler grammar like json, toml, yaml, etc. What benefits as well as the trade offs with using pest to parse grammar to create a programming language? Also what is the difference between this and a lexer and a tokenizer? Do they all mean the same?

[tomtau]

A few projects listed here are programming language implementations: https://github.com/pest-parser/pest#projects-using-pest

What benefits as well as the trade offs with using pest to parse grammar to create a programming language?

Benefits: simplicity (writing PEG is similar to writing regex), correctness (unlike handwritten parsers, parser code is automatically generated), and relatively good performance (case-by-case, but generally on par or slightly worse than handwritten parsers)

Tradeoffs: less flexibility (you may hit limitations of the current grammar), the current API may not fit well some use cases in programming language implementations (e.g. for IDE, one may want a separate lexer or parsing that does not terminate on the first error) and possibly worse performance for some cases

Also what is the difference between this and a lexer and a tokenizer? Do they all mean the same?

This is a parser generator, and the output of a generated parser is a "parse tree", whereas the output of a lexer or a tokeniser is a "sequence of tokens". So, they do not mean the same; the current version of pest's generated code combines lexing and parsing.