Mini-languages in Python
Question:
I’m after creating a simple mini-language parser in Python, programming close to the problem domain and all that.
Anyway, I was wondering how the people on here would go around doing that – what are the preferred ways of doing this kind of thing in Python?
I’m not going to give specific details of what I’m after because at the moment I’m just investigating how easy this whole field is in Python.
Answers:
I have limited but positive experience with PLY (Python Lex-Yacc). It combines Lex and Yacc functionality in a single Python class. You may want to check it out.
Fellow Stackoverflow’er Ned Batchelder has a nice overview of available tools on his website. There’s also an overview on the Python website itself.
In order to be productive, I’d always use a parser generator like CocoPy (Tutorial) to have your grammar transformed into a (correct) parser (unless you want to implement the parser manually for the sake of learning).
The rest is writing the actual interpreter/compiler (Create stack-based byte code or memory AST to be interpreted and then evaluate it).
Python is such a wonderfully simple and extensible language that I’d suggest merely creating a comprehensive python module, and coding against that.
I see that while I typed up the above, PLY has already been mentioned.
Pyparsing is handy for writing “little languages”. I gave a presentation at PyCon’06 on writing a simple adventure game engine, in which the language being parsed and interpreted was the game command set (“inventory”, “take sword”, “drop book”, etc.). (Source code here.)
You can also find links to other pyparsing articles at the pyparsing wiki.
I would recommend funcparserlib. It was written especially for parsing little languages and DSLs and it is faster and smaller than pyparsing (see stats on its homepage). Minimalists and functional programmers should like funcparserlib.
Edit: By the way, I’m the author of this library, so my opinion may be biased.
If you ask me this now, I would try the textx library for python. You can very easily create a dsl in that with python! Advantages are that it creates an AST for you, and lexing and parsing is combined.
I’m after creating a simple mini-language parser in Python, programming close to the problem domain and all that.
Anyway, I was wondering how the people on here would go around doing that – what are the preferred ways of doing this kind of thing in Python?
I’m not going to give specific details of what I’m after because at the moment I’m just investigating how easy this whole field is in Python.
I have limited but positive experience with PLY (Python Lex-Yacc). It combines Lex and Yacc functionality in a single Python class. You may want to check it out.
Fellow Stackoverflow’er Ned Batchelder has a nice overview of available tools on his website. There’s also an overview on the Python website itself.
In order to be productive, I’d always use a parser generator like CocoPy (Tutorial) to have your grammar transformed into a (correct) parser (unless you want to implement the parser manually for the sake of learning).
The rest is writing the actual interpreter/compiler (Create stack-based byte code or memory AST to be interpreted and then evaluate it).
Python is such a wonderfully simple and extensible language that I’d suggest merely creating a comprehensive python module, and coding against that.
I see that while I typed up the above, PLY has already been mentioned.
Pyparsing is handy for writing “little languages”. I gave a presentation at PyCon’06 on writing a simple adventure game engine, in which the language being parsed and interpreted was the game command set (“inventory”, “take sword”, “drop book”, etc.). (Source code here.)
You can also find links to other pyparsing articles at the pyparsing wiki.
I would recommend funcparserlib. It was written especially for parsing little languages and DSLs and it is faster and smaller than pyparsing (see stats on its homepage). Minimalists and functional programmers should like funcparserlib.
Edit: By the way, I’m the author of this library, so my opinion may be biased.
If you ask me this now, I would try the textx library for python. You can very easily create a dsl in that with python! Advantages are that it creates an AST for you, and lexing and parsing is combined.