Generating syntax ASTNode objects automatically from grammar

[brando90]

I wanted to transform strings to their python object representation automatically. e.g. I have a small toy grammar like this:

term : "λ" var "." body
    | term term
 
body : term

var : /[a-zA-Z]/ /[0-9]/ /[_ | ']+/

%import common.WS
%ignore WS

And wanted to create objects automatically e.g.

    text = "λ x : T. x"
    # goes to
    expected_term = Abs(x=Var('x'), body=Var('x'))

I think it's not hard given a lark_ast to build them if I know the grammar but I would have to change my ast_lark parser to objects every time I change the grammar and I wanted to avoid that. Is there a lark automatic way around this?

Thanks!

[MegaIng]

There is the relatively new ast_utils Example here.

[brando90]

Thanks! Will take a look at it

[brando90]

if I already have some classes pre-built, is there a way to use those? (e.g. so that it doesn't create unexpected classes for me)

[MegaIng]

It doesn't create any classes for you. You have to define them yourself. It automatically constructs a Transformer for you that uses your classes. Just adding ast_utils.AST should be enough. (as long as the names are ok)

[brando90]

It doesn't create any classes for you. You have to define them yourself. It automatically constructs a Transformer for you that uses your classes. Just adding ast_utils.AST should be enough. (as long as the names are ok)

ok so this seems to be the same as a converted from CSTs (concrete syntax trees) to ASTs (abstract syntax trees), right? reference: https://eli.thegreenplace.net/2009/02/16/abstract-vs-concrete-syntax-trees#id4

[erezsh]

It's not so simple. Lark can generate CSTs, but also supports built-in tree transformations (like ?rule, _rule, etc.), which means the result doesn't exactly map to the grammar. But also, after constructing the "AST" with ast_utiles, you might still need to do additional post-processing / transformations to get it into the right form.

To conclude, I would say the CST->AST is a continuum, and ast_utils can help you along it. Sometimes it will bridge the entire gap, and often it won't.