Creating a Python codegenerator

[adriangb]

Hi folks,

Before I knew this package existed, I implemented a similar AST-like structure over at python-betterproto. I actually think it's remarkable that we independently arrived at almost exactly the same solution. Our AST-like-structure is here, the AST builder is here and the codegen template which walks the built AST and compiles it to Python is here (jinja2).

I really like what I see here (PG*), and I think it is much more developed and thought out than what we have there. I am especially interested in being able to parse custom options for use in protoc-gen-valdiate, which does not work in python-betterproto as of now.

From what I can gather, there are two steps I will need to take:

1. Write something like the lang/go subpackge (let's call it lang/python) that is able to correctly format python method/class/attribute names and such. This would be similar to what we have here.
2. Write a Module to process the AST. This can be done either by generating the code by hand or using text/template (which would be similar to the jinja template we use there).

Am I on track up to this point?

One additional question that I think may impact the direction to take:
To add protoc-gen-valdiate type validation, would the best way be to re-use the lang/python package but re-write the Module portion to insert validation directly (i.e. copy and past but modify)? Or would it be better to chain Modules and generate separate validation files based on the same AST that import and extend the vanilla classes/methods? Let's call these option No.1 and No.2. They would look something like:

No1.

In: final.py

class MyMessage(betterproto.Message):
    my_field = ...
    
    def my_field_validator(self, value):
        ...

No2.

In: vanilla.py

class MyMessage(betterproto.Message):
    my_field = ...

In: final.py

from vanilla import MyMessage as MyMessageOriginal

class MyMessage(MyMessageOriginal):
    def my_field_validator(self, value):
        ...

Thank you for your great work here!

[adriangb]

I've been able to make some progress with this, reproducing much of lang/go but for Python (at least the re-casing for classes/functions part and type mapping). I still have to figure out some of the more complicated parts, like AddImportMapping.

One issue I ran into that I think is a possible bug/oversight is that protoc-gen-debug requires/will require the go_package option, which does not make sense for Python packages. While it could be kept in there just for compatibility, it would be nice if this wasn't a requirement, or at least if it was documented that this can be safely ignored (I think it can).