C# source generator for docopt usage #77

atifaziz · 2021-04-04T16:15:47Z

This PR implements a C# source generator for docopt. It generates strong-typed C# given a *.docopt.txt file.

There is an example of NavalFate using the source generator.

atifaziz · 2021-04-04T16:21:34Z

There is no concept or specification for the final shape of this feature. In its first draft, the source generator simply generates C# source code similar to the T4 template, but it would be a shame to leave it there. It can and should go much further; it can be a lot smarter given that the source generator has access to the syntactic and semantic model of a project at generation-time.

The PR should help to explore and develop the feature.

Meanwhile, take it for a spin:

dotnet run -p src\Examples\SourceGenerator\NavalFate -- ship new foo bar baz

voieducode · 2021-04-05T09:52:10Z

I love this - lots to unpack for me, I will take it for a spin!

atifaziz · 2021-04-05T10:19:50Z

I love this - lots to unpack for me, I will take it for a spin!

Cool and needless to say, feature ideas welcome!

I am not sure how well this is doable/mappable, but I'd like to toy with the idea that the source generate can generate code to bind the following usage:

naval_fate ship new <name>...
naval_fate ship <name> move <x> <y> [--speed=<kn>]
naval_fate ship shoot <x> <y>
naval_fate mine (set|remove) <x> <y> [--moored|--drifting]

to the following method signatures:

int ShipNew(string[] name);
int ShipMove(string name, int x, int y, int speed);
int ShipShoot(int x, int y, int);
int Mine(Choice<Set, Remove> action, int x, int y, Choice<Set, Remove>? disposition);

// generated types:

enum Set { Set }
enum Remove { Remove }
enum Moored { Moored }
enum Drifting { Drifting }

Conflicts resolved: - eg/SourceGenerator/NavalFate/NavalFate.csproj - eg/SourceGenerator/NavalFate/Program.cs - eg/SourceGenerator/NavalFate/Program.docopt.txt

For example: // Required: // Either: // Required: // Command(ship, False) -> CommandNode ship Bool // Command(new, False) -> CommandNode new Bool // OneOrMore: // Argument(<name>, []) -> ArgumentNode <name> List // Required: // Command(ship, False) -> CommandNode ship Bool // Argument(<name>, []) -> ArgumentNode <name> List // Command(move, False) -> CommandNode move Bool // Argument(<x>, ) -> ArgumentNode <x> String // Argument(<y>, ) -> ArgumentNode <y> String // Optional: // Option(,--speed,1,10) -> OptionNode speed String // Required: // Command(ship, False) -> CommandNode ship Bool // Command(shoot, False) -> CommandNode shoot Bool // Argument(<x>, ) -> ArgumentNode <x> String // Argument(<y>, ) -> ArgumentNode <y> String // Required: // Command(mine, False) -> CommandNode mine Bool // Required: // Either: // Command(set, False) -> CommandNode set Bool // Command(remove, False) -> CommandNode remove Bool // Argument(<x>, ) -> ArgumentNode <x> String // Argument(<y>, ) -> ArgumentNode <y> String // Optional: // Either: // Option(,--moored,0,False) -> OptionNode moored Bool // Option(,--drifting,0,False) -> OptionNode drifting Bool // Required: // Required: // Option(-h,--help,0,False) -> OptionNode help Bool // Required: // Option(,--version,0,False) -> OptionNode version Bool

Example of what's generated: static readonly Pattern Pattern = new Required(ImmutableArray.Create<Pattern>( new Either(ImmutableArray.Create<Pattern>( new Required(ImmutableArray.Create<Pattern>( new Command("ship"), new Command("new"), new OneOrMore( new Argument("<name>", null)))), new Required(ImmutableArray.Create<Pattern>( new Command("ship"), new Argument("<name>", null), new Command("move"), new Argument("<x>", null), new Argument("<y>", null), new Optional(ImmutableArray.Create<Pattern>( new Option("", "--speed", 1, null))))), new Required(ImmutableArray.Create<Pattern>( new Command("ship"), new Command("shoot"), new Argument("<x>", null), new Argument("<y>", null))), new Required(ImmutableArray.Create<Pattern>( new Command("mine"), new Required(ImmutableArray.Create<Pattern>( new Either(ImmutableArray.Create<Pattern>( new Command("set"), new Command("remove"))))), new Argument("<x>", null), new Argument("<y>", null), new Optional(ImmutableArray.Create<Pattern>( new Either(ImmutableArray.Create<Pattern>( new Option("", "--moored", 0, null), new Option("", "--drifting", 0, null))))))), new Required(ImmutableArray.Create<Pattern>( new Required(ImmutableArray.Create<Pattern>( new Option("-h", "--help", 0, null))))), new Required(ImmutableArray.Create<Pattern>( new Option("", "--version", 0, null)))))));

atifaziz · 2021-04-05T15:23:45Z

With 41bf0c0 and 54825a7, the source generator now creates a static version of the pattern tree created from the usage text:

// Required:
//   Either:
//     Required:
//       Command(ship, False) -> CommandNode ship Bool
//       Command(new, False) -> CommandNode new Bool
//       OneOrMore:
//         Argument(<name>, []) -> ArgumentNode <name> List
//     Required:
//       Command(ship, False) -> CommandNode ship Bool
//       Argument(<name>, []) -> ArgumentNode <name> List
//       Command(move, False) -> CommandNode move Bool
//       Argument(<x>, ) -> ArgumentNode <x> String
//       Argument(<y>, ) -> ArgumentNode <y> String
//       Optional:
//         Option(,--speed,1,10) -> OptionNode speed String
//     Required:
//       Command(ship, False) -> CommandNode ship Bool
//       Command(shoot, False) -> CommandNode shoot Bool
//       Argument(<x>, ) -> ArgumentNode <x> String
//       Argument(<y>, ) -> ArgumentNode <y> String
//     Required:
//       Command(mine, False) -> CommandNode mine Bool
//       Required:
//         Either:
//           Command(set, False) -> CommandNode set Bool
//           Command(remove, False) -> CommandNode remove Bool
//       Argument(<x>, ) -> ArgumentNode <x> String
//       Argument(<y>, ) -> ArgumentNode <y> String
//       Optional:
//         Either:
//           Option(,--moored,0,False) -> OptionNode moored Bool
//           Option(,--drifting,0,False) -> OptionNode drifting Bool
//     Required:
//       Required:
//         Option(-h,--help,0,False) -> OptionNode help Bool
//     Required:
//       Option(,--version,0,False) -> OptionNode version Bool

static readonly Pattern Pattern =
    new Required(ImmutableArray.Create<Pattern>(
        new Either(ImmutableArray.Create<Pattern>(
            new Required(ImmutableArray.Create<Pattern>(
                new Command("ship"),
                new Command("new"),
                new OneOrMore(
                    new Argument("<name>", null)))),
            new Required(ImmutableArray.Create<Pattern>(
                new Command("ship"),
                new Argument("<name>", null),
                new Command("move"),
                new Argument("<x>", null),
                new Argument("<y>", null),
                new Optional(ImmutableArray.Create<Pattern>(
                    new Option("", "--speed", 1, null))))),
            new Required(ImmutableArray.Create<Pattern>(
                new Command("ship"),
                new Command("shoot"),
                new Argument("<x>", null),
                new Argument("<y>", null))),
            new Required(ImmutableArray.Create<Pattern>(
                new Command("mine"),
                new Required(ImmutableArray.Create<Pattern>(
                    new Either(ImmutableArray.Create<Pattern>(
                        new Command("set"),
                        new Command("remove"))))),
                new Argument("<x>", null),
                new Argument("<y>", null),
                new Optional(ImmutableArray.Create<Pattern>(
                    new Either(ImmutableArray.Create<Pattern>(
                        new Option("", "--moored", 0, null),
                        new Option("", "--drifting", 0, null))))))),
            new Required(ImmutableArray.Create<Pattern>(
                new Required(ImmutableArray.Create<Pattern>(
                    new Option("-h", "--help", 0, null))))),
            new Required(ImmutableArray.Create<Pattern>(
                new Option("", "--version", 0, null)))))));

This means that there is zero run-time cost of parsing the document usage, which is now all passed to build-time. With the above tree, one technically only needs ParseArgv:

docopt.net/src/DocoptNet/Docopt.cs

Lines 166 to 167 in ed57edb

    
           internal static IList<LeafPattern> ParseArgv(Tokens tokens, ICollection<Option> options, 
        
               bool optionsFirst = false)

and the PatternMatcher added with PR #82 for the final bit. The runtime cost of these two functions should be very minimal. Ideally, even the PatternMatcher can be statically computed/generated code from the tree, one downside of which would be maintenance of duplicate code (one used for DocoptNet as library code and the other for source generation). The duplication could be removed by some intermediate language that's interpreted at run-time but can also be used to generate code; though that might be too fancy for patterns that are not going to change much given that the docopt specification has been stable for a very long time now.

atifaziz · 2021-05-01T14:00:12Z

0100f0f adds the first generated version of Docopt.Apply with the entire pattern matching tree in-lined as code. During development, I will be sharing the generated code for those interested in following the progress or providing feedback.

At this stage, I am not even sure the generated code works so the next step is to add tests, fix bugs and then refactor (with tests avoiding regressions).

atifaziz · 2021-09-02T17:21:58Z

Notes for reviewers

There are a large number (75+) of files changed/added in this PR, but keep in mind that 50 of those belong to examples added (from the reference Python implementation) that helped to shape & drive the generated code and testing:

PS> git diff --name-only "$(git merge-base --fork-point master)..HEAD" eg/ | grep -n .

The examples include the generated code and technically don't need to be committed/tracked. They were added nonetheless to help with online review as well as regressions during development. They should/will be removed before the final merge of this PR. There are 14 such files:

PS> git diff --name-only "$(git merge-base --fork-point master)..HEAD" eg/*/obj/* | grep -n .

When you exclude the examples, then changes to review only amount to a third (~25 files):

PS> git diff --name-only "$(git merge-base --fork-point master)..HEAD" | grep -vF eg/SourceGenerator/ | grep -n .

There are even fewer when you consider core project changes (i.e. excluding tests):

PS> git diff --name-only "$(git merge-base --fork-point master)..HEAD" | grep -F src/ | grep -n .
1:src/DocoptNet.CodeGeneration/AnalyzerReleases.Shipped.md
2:src/DocoptNet.CodeGeneration/AnalyzerReleases.Unshipped.md
3:src/DocoptNet.CodeGeneration/CSharpSourceBuilder.cs
4:src/DocoptNet.CodeGeneration/DocoptNet.CodeGeneration.csproj
5:src/DocoptNet.CodeGeneration/DocoptNet.CodeGeneration.targets
6:src/DocoptNet.CodeGeneration/Extensions.cs
7:src/DocoptNet.CodeGeneration/SourceGenerator.cs
8:src/DocoptNet.CodeGeneration/StringBuilderSourceText.cs
9:src/DocoptNet/BranchPatterns.cs
10:src/DocoptNet/Docopt.cs
11:src/DocoptNet/GenerateCodeHelper.cs
12:src/DocoptNet/GeneratedSourceModule.cs
13:src/DocoptNet/Pattern.cs
14:src/DocoptNet/PatternMatcher.cs
15:src/DocoptNet/ValueObject.cs

In the very core project (i.e. excluding the source generator project), the bulk of the changes went into PatternMatcher.cs:

PS> git diff --numstat "$(git merge-base --fork-point master)..HEAD" | grep -F src/DocoptNet/
6       1       src/DocoptNet/BranchPatterns.cs
1       1       src/DocoptNet/Docopt.cs
27      5       src/DocoptNet/GenerateCodeHelper.cs
28      0       src/DocoptNet/GeneratedSourceModule.cs
1       1       src/DocoptNet/Pattern.cs
279     106     src/DocoptNet/PatternMatcher.cs
2       0       src/DocoptNet/ValueObject.cs

PatternMatcher.cs was refactored extensively (by consolidating duplication) so that the generated code could efficiently share as much of the pattern matching logic as possible with the runtime/library version.

Apart from the technical changes, reviewers should focus on the overall design of the generated code in terms of:

Simplicity
Experience
Above all, hindering any customisation or extensibility in the future

voieducode · 2021-09-02T17:48:53Z

Super helpful notes, thank you @atifaziz

atifaziz · 2021-09-05T10:20:07Z

In my own review, I noticed that the exit argument of Apply was never used in the generated code and the implementation always acted as if exit: true was the supplied argument. With, fa0c013 (+ 783d813), I've removed the exit parameter entirely. Eventually, I think all the parameters will go away except args, but those tweaks can come in post merge of this PR.

voieducode

After our offline conversation about the PR, I have no additional questions or comments - thanks!

voieducode · 2021-09-03T09:19:28Z

eg/SourceGenerator/ArgumentsExample/Program.cs

+foreach (var (name, value) in arguments)
+    Console.WriteLine($"{name} = {value}");
+
+Console.WriteLine($@"{{


This is as clean and simple as the Python version. Someone on my team was commenting that there was a lot of magic happening. The good news is that magic is not as costly as one would expect.

eg/SourceGenerator/ArgumentsExample/Program.cs

voieducode · 2021-09-03T09:34:05Z

...erated/DocoptNet.CodeGeneration/DocoptNet.CodeGeneration.SourceGenerator/ProgramArguments.cs

@@ -0,0 +1,290 @@
+#nullable enable annotations


Source generator is one step towards addressing the ask from issue #8. What do you think @prabirshrestha?

I think #76 kind of addressed that already.

voieducode · 2021-09-03T10:21:36Z

...erated/DocoptNet.CodeGeneration/DocoptNet.CodeGeneration.SourceGenerator/ProgramArguments.cs

+
+namespace ArgumentsExample
+{
+    partial class ProgramArguments : IEnumerable<KeyValuePair<string, object?>>


This generated code is so legible! 😮 - And there are so many ways we could extend this to support new scenarios, really exciting stuff.

voieducode · 2021-09-03T11:29:51Z

eg/SourceGenerator/Git/Git.docopt.txt

@@ -0,0 +1,21 @@
+usage: git [--version] [--exec-path=<path>] [--html-path]


I have posted issue #114 to capture a weird scenario where exec-path is both a boolean and a string.

Did you see my reply there?

voieducode · 2021-09-13T16:23:15Z

Congratulations for the merge!!! 🎉🎉🎉

atifaziz · 2021-09-13T18:54:49Z

Congratulations for the merge!!! 🎉🎉🎉

Thanks! Slowly, but surely, made it to the finishing line! 🏁 Appreciate all your trust, patience and support during the months it took to work through this PR! 😄

Add C# source generator from docopt usage

61f41da

atifaziz added 2 commits April 4, 2021 18:34

Allow packaging of source generator

5bea89f

Remove project reference since DocoptNet is embedded

04f7336

This was referenced Apr 5, 2021

Make public API predicated on "DOCOPTNET_PUBLIC" #79

Merged

Refactor pattern matching to be static & stand alone #82

Merged

atifaziz added 4 commits April 5, 2021 15:00

Merge branch 'master' into source-generator

c271e21

Conflicts resolved: - eg/SourceGenerator/NavalFate/NavalFate.csproj - eg/SourceGenerator/NavalFate/Program.cs - eg/SourceGenerator/NavalFate/Program.docopt.txt

Merge remote-tracking branch 'upstream/master' into source-generator

490e650

Merge branch 'master' into source-generator

0dc9902

atifaziz mentioned this pull request Apr 5, 2021

T4 template doesn't run in "T4DocoptNetHostApp" #75

Closed

atifaziz added 8 commits April 5, 2021 20:56

Merge remote-tracking branch 'upstream/master' into source-generator

9c67277

Add extension for invariant formatting

ca85081

Add option value (like default) that was missing

eb3425c

Merge branch 'master' into source-generator

addc09a

Merge branch 'master' into source-generator

ddf1f5a

Merge remote-tracking branch 'upstream/master' into source-generator

cd5ebac

Initial rough-cut of "Apply" as generated code

0100f0f

Add generated code to share results (during development)

7f02925

atifaziz added 6 commits May 2, 2021 23:05

Add initial scaffolding for source generation tests

0897db4

Validate & move static part of source generation into resources

836079b

Track only the generated code

a174cd7

Merge remote-tracking branch 'upstream/master' into source-generator

1c1f5af

Split common into module and patterns

e12b25a

Embed all of DocoptNet into generated code for max reuse

5606825

atifaziz added 5 commits September 2, 2021 12:31

Fix code formatting

127205c

Remove unused "Value" to "ValueObject" conversion

8b50984

Revert unneeded "Value" accessibility edits

b54f6ab

Revert unused (T4) code generation changes

01943e6

Merge branch 'master' into source-generator

3ef3138

atifaziz added 4 commits September 3, 2021 13:35

Add missing linked file extension in project

922165d

Remove need for SOURCE_GENERATION conditional symbol in tests

0c5b203

Remove "exit" option; for now implemented as "exit: true"

fa0c013

Fix tests with respect to removal of "exit" parameter

783d813

atifaziz added 9 commits September 5, 2021 14:57

Combine the "SourceGeneratorTests" parts

adf7e0c

Remove ".dll" extension from test assembly name

adbdefe

Use project's convention for naming test cases

c2e3cc8

Remove extra blank line

1cf45cd

Wrap "AdditionalFiles" attributes in targets file

20d2410

Use "docopt" all lower for "SourceItemType"

7834bf0

Merge branch 'master' into source-generator

b00100b

Merge branch 'master' into source-generator

83e09ec

Revert duplicate package refs in test project

ede640e

voieducode approved these changes Sep 7, 2021

View reviewed changes

Remove the generated code

4de329c

Change diagnostic ID prefix to "DCPT" (remove vowels)

0adc6f1

atifaziz merged commit a8bcd5e into docopt:master Sep 13, 2021

atifaziz deleted the source-generator branch September 13, 2021 15:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

C# source generator for docopt usage #77

C# source generator for docopt usage #77

atifaziz commented Apr 4, 2021 •

edited

Loading

atifaziz commented Apr 4, 2021 •

edited

Loading

voieducode commented Apr 5, 2021

atifaziz commented Apr 5, 2021 •

edited

Loading

atifaziz commented Apr 5, 2021 •

edited

Loading

atifaziz commented May 1, 2021 •

edited

Loading

atifaziz commented Sep 2, 2021

voieducode commented Sep 2, 2021

atifaziz commented Sep 5, 2021

voieducode left a comment

voieducode Sep 3, 2021

voieducode Sep 3, 2021

atifaziz Sep 9, 2021

voieducode Sep 3, 2021

voieducode Sep 3, 2021

atifaziz Sep 9, 2021

voieducode commented Sep 13, 2021

atifaziz commented Sep 13, 2021

		@@ -0,0 +1,21 @@
		usage: git [--version] [--exec-path=<path>] [--html-path]

C# source generator for docopt usage #77

C# source generator for docopt usage #77

Conversation

atifaziz commented Apr 4, 2021 • edited Loading

atifaziz commented Apr 4, 2021 • edited Loading

voieducode commented Apr 5, 2021

atifaziz commented Apr 5, 2021 • edited Loading

atifaziz commented Apr 5, 2021 • edited Loading

atifaziz commented May 1, 2021 • edited Loading

atifaziz commented Sep 2, 2021

Notes for reviewers

voieducode commented Sep 2, 2021

atifaziz commented Sep 5, 2021

voieducode left a comment

Choose a reason for hiding this comment

voieducode Sep 3, 2021

Choose a reason for hiding this comment

voieducode Sep 3, 2021

Choose a reason for hiding this comment

atifaziz Sep 9, 2021

Choose a reason for hiding this comment

voieducode Sep 3, 2021

Choose a reason for hiding this comment

voieducode Sep 3, 2021

Choose a reason for hiding this comment

atifaziz Sep 9, 2021

Choose a reason for hiding this comment

voieducode commented Sep 13, 2021

atifaziz commented Sep 13, 2021

atifaziz commented Apr 4, 2021 •

edited

Loading

atifaziz commented Apr 4, 2021 •

edited

Loading

atifaziz commented Apr 5, 2021 •

edited

Loading

atifaziz commented Apr 5, 2021 •

edited

Loading

atifaziz commented May 1, 2021 •

edited

Loading