Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source map support #163

Merged
merged 21 commits into from
Oct 18, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
769a432
source-map: Create SourceNodes for functions and initializers
Mingun May 31, 2021
fcae22a
source-map: Generate tables of constants as a SourceNode
Mingun May 31, 2021
de0ad72
source-map: Remove excess `.join("\n")`s
Mingun May 31, 2021
4ad80de
source-map: Do not join arrays prematurely
Mingun May 31, 2021
9bdf8eb
source-map: Remove yet another premature join
Mingun May 31, 2021
fa5da47
source-map: Remove useless function
Mingun May 31, 2021
d482541
source-map: Generate top-level as a SourceNode
Mingun May 31, 2021
6eece33
source-map: Generate module wrappers as a SourceNode
Mingun May 31, 2021
8b7d69d
source-map: Represent generated code in the AST as a SourceNode
Mingun Jun 1, 2021
b352e92
source-map: Don't Repeat Yourself - introduce new interface SourceOpt…
Mingun Jul 25, 2021
c08e4d8
source-map: Implement API for producing source maps
Mingun Jun 1, 2021
cadc935
source-map: Run CLI tests in the test/cli directory
Mingun Aug 20, 2021
ef9f014
source-map: Do not break process until generated code/source map will…
Mingun Oct 16, 2021
e47b5f4
source-map: Add new `-m/--source-map` parameter to the CLI
Mingun Jul 25, 2021
e5b1ebf
source-map: Introduce helper for generate a parse rule function name
Mingun Jul 25, 2021
1ee49ab
source-map: Add mapping for the rule names to the names of the parse …
Mingun Jul 25, 2021
fe39cbf
source-map: Cover ast2SourceNode completely
Mingun Oct 16, 2021
6f57b4a
source-map: Alternative API: new output mode "source-and-map" instead…
Mingun Jul 25, 2021
4ee3472
source-map: Make source-map work in the browser
hildjj Oct 17, 2021
0ac78fd
Update all dependencies
Mingun Oct 18, 2021
53f3d12
Only run lint on latest node, rather than holding back eslint to an o…
hildjj Oct 18, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/node.js.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ jobs:
- name: Install dependencies
run: npm install
- name: Check coding standards
if: matrix.node-version == '16.x' && matrix.os == 'ubuntu-latest'
run: npm run lint
- name: Static analysis - check types
run: npm run ts
Expand Down
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ This file documents all notable changes to Peggy.

Released: TBD

### Major Changes

- Add support for generating source maps.
[@Mingun](https://github.com/peggyjs/peggy/pull/163)

### Minor Changes

- New CLI [@hildjj](https://github.com/peggyjs/peggy/pull/167)
Expand Down
71 changes: 64 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ Follow these steps to upgrade:
— more powerful than traditional LL(_k_) and LR(_k_) parsers
- Usable [from your browser](https://peggyjs.org/online), from the command line,
or via JavaScript API
- [Source map](https://developer.mozilla.org/en-US/docs/Tools/Debugger/How_to/Use_a_source_map) support
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This worked when I added a //# sourceMappingURL= comment to the end of the generated .js file, which I think should be done automatically.

This pointed out that the filenames in the .map file might need to be tweaked; from the project root, try ./bin/peggy.js examples/fizzbuzz.peggy --source-map. The map file includes "sources":["examples/fizzbuzz.peggy"] which should probably instead be `"sources":["fizzbuzz.peggy"]". The filename should be relative to the .map file.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure that automatic appending of this line will be desired behavior in each case, mostly because I don't known how people will use source-map feature. Source maps could be hosted not in the same directory where it was generated and only the user know the right place. Adding this line is a trivial task, so I leaved it out of scope of my PR. We can firstly gather the feedback and add this line later if users will complain about it. Or, if you have a clear vision, I can add it now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the line should be auto-appended in the CLI only. We can add an option for inlining it later.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The filename should be relative to the .map file

Done.

I think the line should be auto-appended in the CLI only. We can add an option for inlining it later.

I'll leave that for the next PRs. There a many options to forming such URL. For example, someone can want to form a data URL embedding generated source map in it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is going to take a few more PRs before we're ready to release, and I don't want to docs to be wrong for that long a time, so I'm going to create a "1.3" branch that we can merge this into, and start to merge other work into as well.

After this lands in the 1.3 branch, and we add in #199, we should have a better way to talk about how to test whether the maps are working as expected. Make sense @Mingun?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think this is a good plan


## Getting Started

Expand Down Expand Up @@ -120,13 +121,16 @@ You can tweak the generated parser with several options:
name with extension changed to `.js`, or stdout if no input file is given.
- `--plugin <module>` — makes Peggy use a specified plugin (can be specified
multiple times)
- `-t`, `--test <text>` - Test the parser with the given text, outputting the
result of running the parser against this input
- `-T`, `--test-file <filename>` - Test the parser with the contents of the
given file, outputting the result of running the parser against this input
- `-t`, `--test <text>` — Test the parser with the given text, outputting the
result of running the parser against this input.
If the input to be tested is not parsed, the CLI will exit with code 2
- `-T`, `--test-file <filename>` — Test the parser with the contents of the
given file, outputting the result of running the parser against this input.
If the input to be tested is not parsed, the CLI will exit with code 2
- `--source-map` — generate a source map file with an optionally specified name
- `--trace` — makes the parser trace its progress
- `-v`, `--version` - output the version number
- `-h`, `--help` - display help for command
- `-v`, `--version` output the version number
- `-h`, `--help` display help for command

If you specify options using `-c <file>` or `--extra-options-file <file>`, you
will need to ensure you are using the correct types. In particular, you may
Expand All @@ -150,6 +154,50 @@ module.exports = {
};
```

You can test generated parser immediately if you specify the `-t/--test` or `-T/--test-file`
option. This option conflicts with the option `-m/--source-map` unless `-o/--output` is
also specified.

The CLI will exit with the code:
- `0` if all was success
- `1` if you supply incorrect or conflicting parameters
- `2` if all parameters is correct, you specify the `-t/--test` or `-T/--test-file` option
and specified input does not parsed with the specified grammar

Examples:

```console
# - write test results to stdout (42)
# - exit with the code 0
echo "foo = '1' { return 42 }" | peggy --test 1

# - write a parser error to stdout (Expected "1" but "2" found)
# - exit with the code 2
echo "foo = '1' { return 42 }" | peggy --test 2

# - write an error to stdout (Generation of the source map is useless if you don't
# store a generated parser code, perhaps you forgot to add an `-o/--output` option?)
# - exit with the code 1
echo "foo = '1' { return 42 }" | peggy --source-map --test 1

# - write an error to stdout (Generation of the source map is useless if you don't
# store a generated parser code, perhaps you forgot to add an `-o/--output` option?)
# - exit with the code 1
echo "foo = '1' { return 42 }" | peggy --source-map --test 2

# - write an output to `parser.js`,
# - write a source map to `parser.js.map`
# - write test results to stdout (42)
# - exit with the code 0
echo "foo = '1' { return 42 }" | peggy --output parser.js --source-map --test 1

# - write an output to `parser.js`,
# - write a source map to `parser.js.map`
# - write a parser error to stdout (Expected "1" but "2" found)
# - exit with the code 2
echo "foo = '1' { return 42 }" | peggy --output parser.js --source-map --test 2
```

### JavaScript API

In Node.js, require the Peggy parser generator module:
Expand Down Expand Up @@ -207,11 +255,20 @@ object to `peg.generate`. The following options are supported:
`options.grammarSource` is redefined in the grammar. It is useful to attach
the file information to the errors, for example
- `output` — if set to `"parser"`, the method will return generated parser
object; if set to `"source"`, it will return parser source code as a string
object; if set to `"source"`, it will return parser source code as a string.
If set to `"source-and-map"`, it will return a [`SourceNode`] object; you can
get source code by calling `toString()` method or source code and mapping by
calling `toStringWithSourceMap()` method, see the [`SourceNode`] documentation
(default: `"parser"`)

> **Note**: because of bug [source-map/444] you should also set `grammarSource` to
> a not-empty string if you set this value to `"source-and-map"`
- `plugins` — plugins to use. See the [Plugins API](#plugins-api) section
- `trace` — makes the parser trace its progress (default: `false`)

[`SourceNode`]: https://github.com/mozilla/source-map#sourcenode
[source-map/444]: https://github.com/mozilla/source-map/issues/444

## Using the Parser

To use the generated parser, call its `parse` method and pass an input string
Expand Down
Loading