How to parse multiple statements in sly?

Question

I am trying to parse some code using sly. I would like to separate the statements with a semicolon.

I have defined a token called SEMI which represents a semicolon:

class MyLexer(Lexer):
    tokens = {
        ...,
        SEMI
    }

    SEMI = r";"
    ...

If I use SEMI inside the parser class like so:

class MyParser(Parser):
    ...
    @_("OUTPUT expr SEMI")
    def statement(self, p):
        return ("output", p.expr)

and put multiple statements in the code I’m trying to parse separated with a semicolon like so:

output 1;output 2;

I get the following error:

sly: Syntax error at line 1, token=OUTPUT

Does anyone know how to make sly parse multiple statements which are separated with a semicolon (or any other character, such as a newline)?

Asked By: macic

||

Source

Answer 1

If you just say that a statement has the form output <expr> ;, and you tell the parser to parse a statement, then it will parse a statement. Not "some number of statements". One statement. The second statement in the input doesn’t match the grammar.

If you want to parse a program consisting of a number of statements, you have to do that explicitly:

@_("{ statement }")
def program(self, p):
    return p.statement

Note that the parser will attempt to parse the non-terminal produced by the first rule in the grammar, unless you configure a start symbol. Do make sure your grammar starts with the non-terminal you want to match.

Note:

The version of Sly currently on Github (which, according to Sly’s author, is no longer being maintained or extended) includes partial implementation of EBNF optional and repeating elements, which I used in the above code. I apologise for using the wrong syntax in the first version of this answer.

Answered By: rici

Answer 2

By default the parser only parses one statement. To parse multiple statements:

@_('statements')
def program(self, p):
    return p.statements

@_('statement')
def statements(self, p):
    return (p.statement, )

@_('statements statement')
def statements(self, p):
    return p.statements + (p.statement, )

Answered By: macic

How to parse multiple statements in sly?

Question:

Answers:

Note: