FAQ

Q. How to manipulate the AST?

Suppose I have a pyll object, e.g. a model spec generator. Suppose I want to programmatically replace one value in it with something determined by a function from some inputs. E.g. I want to set a key "num_filters" = pyll.scope.choice(function(some_inputs)), where the "num_filters" key is a node that appears somewhere down the object tree. I suspect I need to use replace_input? But the exact way to construct the arguments escapes me. Or do I need to do something else?

Similarly, suppose I want to add a key?

I guess I'm in general asking: how do I do simple transformations on pyll ASTs of this kind? If you could point me to the relevant place to figure it out, that would be great.

Answer

A bit of background: the basic data type in Pyll is the Apply. It represents the application of a function. It's defined in base.py. It has three important attributes:

• name - normally the function's name, like "sin" or "dict"

• pos_args - a list of Apply instances used as positional arguments to this function application.

• named_args - a list of [key, arg] pairs passed as named arguments to this function application.

• args - (TODO) a dictionary mapping parameter names to arguments using the same logic as the Python interpreter. This makes the most sense for Apply's of Python functions, but I think it can be reasonably well-defined in all cases.

So to rephrase your question, if you want to modify the 'num_filters' "key" then you probably want to modify the named_args of one or more Apply's with the property name == 'dict'. You can explicitly do this as follows:

root = ... # your top-level expression node
newnode = scope.choice(function(some_inputs))
for orig in dfs(root):
    if orig.name == 'dict':

       orig.named_args = [[n, newnode if n == 'num_filters' else a]
           for (n, a) in orig.named_args]

N.B. that the new named_args is a list of lists with no tuples. It's important that this container be mutable.

I can imagine a couple of types of AST filtering that could have shorthand notations, but as long as the Apply is the basic data structure they would need to actually do this sort of logic to perform an in-place modification to the AST.