Pyparsing Precedence breaks with Unary operator
Question:
I’m trying to implement a subset of Python’s operators for arithmetic parsing using pyparsing. I have the following code implementing my parser:
variable_names = pyparsing.Combine(pyparsing.Literal('$') + pyparsing.Word(pyparsing.alphanums + '_'))
integer = pyparsing.Word(pyparsing.nums)
double = pyparsing.Combine(pyparsing.Word(pyparsing.nums) + '.' + pyparsing.Word(pyparsing.nums))
parser = pyparsing.operatorPrecedence(variable_names | double | integer, [
('**', 2, pyparsing.opAssoc.RIGHT),
('-', 1, pyparsing.opAssoc.RIGHT),
(pyparsing.oneOf('* / // %'), 2, pyparsing.opAssoc.LEFT),
(pyparsing.oneOf('+ -'), 2, pyparsing.opAssoc.LEFT),
(pyparsing.oneOf('> >= < <= == !='), 2, pyparsing.opAssoc.LEFT),
('not', 1, pyparsing.opAssoc.RIGHT),
('and', 2, pyparsing.opAssoc.LEFT),
('or', 2, pyparsing.opAssoc.LEFT)])
For the most part, this works fine, although sometimes it breaks when I use the unary -. Specifically, I think (I may be wrong) it breaks if I use - after higher precedence operands, which in this case is just **. The following examples show the issue:
parsing 5 * 10 * -2 yields: ['5', '*', '10', '*', ['-', '2']]
parsing 5 * 10 ** -2 yields: ['5', '*', '10'] # Wrong
parsing 5 * 10 ** (-2) yields: ['5', '*', ['10', '**', ['-', '2']]]
parsing 5 and not 8 yields: ['5', 'and', ['not', '8']]
parsing 5 and - 8 yields: ['5', 'and', ['-', '8']]
Is there any reason why this is happening? What am I missing?
Answers:
As for me you should define - as higher then **
('-', 1, pyparsing.opAssoc.RIGHT),
('**', 2, pyparsing.opAssoc.RIGHT),
and this should resolve your problem.
Minimal working code
import pyparsing
variable_names = pyparsing.Combine(pyparsing.Literal('$') + pyparsing.Word(pyparsing.alphanums + '_'))
integer = pyparsing.Word(pyparsing.nums)
double = pyparsing.Combine(pyparsing.Word(pyparsing.nums) + '.' + pyparsing.Word(pyparsing.nums))
parser = pyparsing.operatorPrecedence(
variable_names | double | integer,
[
('-', 1, pyparsing.opAssoc.RIGHT),
('**', 2, pyparsing.opAssoc.RIGHT),
(pyparsing.oneOf('* / // %'), 2, pyparsing.opAssoc.LEFT),
(pyparsing.oneOf('+ -'), 2, pyparsing.opAssoc.LEFT),
(pyparsing.oneOf('> >= < <= == !='), 2, pyparsing.opAssoc.LEFT),
('not', 1, pyparsing.opAssoc.RIGHT),
('and', 2, pyparsing.opAssoc.LEFT),
('or', 2, pyparsing.opAssoc.LEFT)
]
)
examples = [
"5 * 10 ** -2",
"5 * 10 * -2",
"5 * 10 ** (-2)",
"5 * -10 ** 2",
"5 * (-10) ** 2",
"5 and not 8",
"5 and -8",
"1 ** -2",
"-1 ** 2",
]
longest = max(map(len, examples))
for ex in examples:
result = parser.parseString(ex)
print(f'{ex:{longest}} <=> {result}')
Results:
5 * 10 ** -2 <=> [['5', '*', ['10', '**', ['-', '2']]]]
5 * 10 * -2 <=> [['5', '*', '10', '*', ['-', '2']]]
5 * 10 ** (-2) <=> [['5', '*', ['10', '**', ['-', '2']]]]
5 * -10 ** 2 <=> [['5', '*', [['-', '10'], '**', '2']]]
5 * (-10) ** 2 <=> [['5', '*', [['-', '10'], '**', '2']]]
5 and not 8 <=> [['5', 'and', ['not', '8']]]
5 and -8 <=> [['5', 'and', ['-', '8']]]
1 ** -2 <=> [['1', '**', ['-', '2']]]
-1 ** 2 <=> [[['-', '1'], '**', '2']]
BTW: for comparision: C Operator Precedence and Python – Operator precedence
EDIT:
I can get -500 for 5 * -10 ** 2 ([[5, '*', ['-', [10, '**', 2]]]]) when I keep ** before - but I use
integer = pyparsing.pyparsing_common.signed_integer
import pyparsing
variable_names = pyparsing.Combine(pyparsing.Literal('$') + pyparsing.Word(pyparsing.alphanums + '_'))
#integer = pyparsing.Word(pyparsing.nums)
integer = pyparsing.pyparsing_common.signed_integer
double = pyparsing.Combine(pyparsing.Word(pyparsing.nums) + '.' + pyparsing.Word(pyparsing.nums))
parser = pyparsing.operatorPrecedence(
variable_names | double | integer,
[
('**', 2, pyparsing.opAssoc.RIGHT),
('-', 1, pyparsing.opAssoc.RIGHT),
(pyparsing.oneOf('* / // %'), 2, pyparsing.opAssoc.LEFT),
(pyparsing.oneOf('+ -'), 2, pyparsing.opAssoc.LEFT),
(pyparsing.oneOf('> >= < <= == !='), 2, pyparsing.opAssoc.LEFT),
('not', 1, pyparsing.opAssoc.RIGHT),
('and', 2, pyparsing.opAssoc.LEFT),
('or', 2, pyparsing.opAssoc.LEFT)
]
)
examples = [
"5 * 10 ** -2",
"5 * 10 * -2",
"5 * 10 ** (-2)",
"5 * -10 ** 2",
"5 * (-10) ** 2",
"5 and not 8",
"5 and -8",
"1 ** -2",
"-1 ** 2",
]
longest = max(map(len, examples))
for ex in examples:
result = parser.parseString(ex)
print(f'{ex:{longest}} <=> {result}')
Result:
5 * 10 ** -2 <=> [[5, '*', [10, '**', -2]]]
5 * 10 * -2 <=> [[5, '*', 10, '*', ['-', 2]]]
5 * 10 ** (-2) <=> [[5, '*', [10, '**', ['-', 2]]]]
5 * -10 ** 2 <=> [[5, '*', ['-', [10, '**', 2]]]]
5 * (-10) ** 2 <=> [[5, '*', [['-', 10], '**', 2]]]
5 and not 8 <=> [[5, 'and', ['not', 8]]]
5 and -8 <=> [[5, 'and', ['-', 8]]]
1 ** -2 <=> [[1, '**', -2]]
-1 ** 2 <=> [['-', [1, '**', 2]]]
Doc for pyparsing_common with other predefined expressions
sorry for bringing back such an old subject but I’m writing a very similar parser for my project which mix Boolean logic and mathematical operator and I ended up with such a similar code that it seemed appropriate.
Since the parser given here encounter the same issue than mine, I’m going to use it to show my problem.
I can’t make it parse "$true == not $false", and basically, any "not" after a comparaison won’t work without parenthesis.
$true == $false <=> [['$true', '==', '$false']]
$true == not $false <=> ['$true']
$true == (not $false) <=> [['$true', '==', ['not', '$false']]]
As you can see without the "not" or with parenthesis, it parse fine, but with a simple "not" it seems to ignore everything after the "=="
What kind of fix it is to move the "not" operator first in the infix_notation (previously operatorPrecedence). Then I got those results:
$true == not $false <=> [['$true', '==', ['not', '$false']]]
$true == (not $false) <=> [['$true', '==', ['not', '$false']]]
Which are greats, but of course it broke operator precedence for things like "$true and not 10 == 9" which I’d like to be parse "$true and not (10 == 9)" like in python and instead parse like that:
$true and not 10 == 9 <=> [['$true', 'and', [['not', 10], '==', 9]]]
I was wondering if you encounters those use case in the implementation of your parser and have found a way around them.
I’m trying to implement a subset of Python’s operators for arithmetic parsing using pyparsing. I have the following code implementing my parser:
variable_names = pyparsing.Combine(pyparsing.Literal('$') + pyparsing.Word(pyparsing.alphanums + '_'))
integer = pyparsing.Word(pyparsing.nums)
double = pyparsing.Combine(pyparsing.Word(pyparsing.nums) + '.' + pyparsing.Word(pyparsing.nums))
parser = pyparsing.operatorPrecedence(variable_names | double | integer, [
('**', 2, pyparsing.opAssoc.RIGHT),
('-', 1, pyparsing.opAssoc.RIGHT),
(pyparsing.oneOf('* / // %'), 2, pyparsing.opAssoc.LEFT),
(pyparsing.oneOf('+ -'), 2, pyparsing.opAssoc.LEFT),
(pyparsing.oneOf('> >= < <= == !='), 2, pyparsing.opAssoc.LEFT),
('not', 1, pyparsing.opAssoc.RIGHT),
('and', 2, pyparsing.opAssoc.LEFT),
('or', 2, pyparsing.opAssoc.LEFT)])
For the most part, this works fine, although sometimes it breaks when I use the unary -. Specifically, I think (I may be wrong) it breaks if I use - after higher precedence operands, which in this case is just **. The following examples show the issue:
parsing 5 * 10 * -2 yields: ['5', '*', '10', '*', ['-', '2']]
parsing 5 * 10 ** -2 yields: ['5', '*', '10'] # Wrong
parsing 5 * 10 ** (-2) yields: ['5', '*', ['10', '**', ['-', '2']]]
parsing 5 and not 8 yields: ['5', 'and', ['not', '8']]
parsing 5 and - 8 yields: ['5', 'and', ['-', '8']]
Is there any reason why this is happening? What am I missing?
As for me you should define - as higher then **
('-', 1, pyparsing.opAssoc.RIGHT),
('**', 2, pyparsing.opAssoc.RIGHT),
and this should resolve your problem.
Minimal working code
import pyparsing
variable_names = pyparsing.Combine(pyparsing.Literal('$') + pyparsing.Word(pyparsing.alphanums + '_'))
integer = pyparsing.Word(pyparsing.nums)
double = pyparsing.Combine(pyparsing.Word(pyparsing.nums) + '.' + pyparsing.Word(pyparsing.nums))
parser = pyparsing.operatorPrecedence(
variable_names | double | integer,
[
('-', 1, pyparsing.opAssoc.RIGHT),
('**', 2, pyparsing.opAssoc.RIGHT),
(pyparsing.oneOf('* / // %'), 2, pyparsing.opAssoc.LEFT),
(pyparsing.oneOf('+ -'), 2, pyparsing.opAssoc.LEFT),
(pyparsing.oneOf('> >= < <= == !='), 2, pyparsing.opAssoc.LEFT),
('not', 1, pyparsing.opAssoc.RIGHT),
('and', 2, pyparsing.opAssoc.LEFT),
('or', 2, pyparsing.opAssoc.LEFT)
]
)
examples = [
"5 * 10 ** -2",
"5 * 10 * -2",
"5 * 10 ** (-2)",
"5 * -10 ** 2",
"5 * (-10) ** 2",
"5 and not 8",
"5 and -8",
"1 ** -2",
"-1 ** 2",
]
longest = max(map(len, examples))
for ex in examples:
result = parser.parseString(ex)
print(f'{ex:{longest}} <=> {result}')
Results:
5 * 10 ** -2 <=> [['5', '*', ['10', '**', ['-', '2']]]]
5 * 10 * -2 <=> [['5', '*', '10', '*', ['-', '2']]]
5 * 10 ** (-2) <=> [['5', '*', ['10', '**', ['-', '2']]]]
5 * -10 ** 2 <=> [['5', '*', [['-', '10'], '**', '2']]]
5 * (-10) ** 2 <=> [['5', '*', [['-', '10'], '**', '2']]]
5 and not 8 <=> [['5', 'and', ['not', '8']]]
5 and -8 <=> [['5', 'and', ['-', '8']]]
1 ** -2 <=> [['1', '**', ['-', '2']]]
-1 ** 2 <=> [[['-', '1'], '**', '2']]
BTW: for comparision: C Operator Precedence and Python – Operator precedence
EDIT:
I can get -500 for 5 * -10 ** 2 ([[5, '*', ['-', [10, '**', 2]]]]) when I keep ** before - but I use
integer = pyparsing.pyparsing_common.signed_integer
import pyparsing
variable_names = pyparsing.Combine(pyparsing.Literal('$') + pyparsing.Word(pyparsing.alphanums + '_'))
#integer = pyparsing.Word(pyparsing.nums)
integer = pyparsing.pyparsing_common.signed_integer
double = pyparsing.Combine(pyparsing.Word(pyparsing.nums) + '.' + pyparsing.Word(pyparsing.nums))
parser = pyparsing.operatorPrecedence(
variable_names | double | integer,
[
('**', 2, pyparsing.opAssoc.RIGHT),
('-', 1, pyparsing.opAssoc.RIGHT),
(pyparsing.oneOf('* / // %'), 2, pyparsing.opAssoc.LEFT),
(pyparsing.oneOf('+ -'), 2, pyparsing.opAssoc.LEFT),
(pyparsing.oneOf('> >= < <= == !='), 2, pyparsing.opAssoc.LEFT),
('not', 1, pyparsing.opAssoc.RIGHT),
('and', 2, pyparsing.opAssoc.LEFT),
('or', 2, pyparsing.opAssoc.LEFT)
]
)
examples = [
"5 * 10 ** -2",
"5 * 10 * -2",
"5 * 10 ** (-2)",
"5 * -10 ** 2",
"5 * (-10) ** 2",
"5 and not 8",
"5 and -8",
"1 ** -2",
"-1 ** 2",
]
longest = max(map(len, examples))
for ex in examples:
result = parser.parseString(ex)
print(f'{ex:{longest}} <=> {result}')
Result:
5 * 10 ** -2 <=> [[5, '*', [10, '**', -2]]]
5 * 10 * -2 <=> [[5, '*', 10, '*', ['-', 2]]]
5 * 10 ** (-2) <=> [[5, '*', [10, '**', ['-', 2]]]]
5 * -10 ** 2 <=> [[5, '*', ['-', [10, '**', 2]]]]
5 * (-10) ** 2 <=> [[5, '*', [['-', 10], '**', 2]]]
5 and not 8 <=> [[5, 'and', ['not', 8]]]
5 and -8 <=> [[5, 'and', ['-', 8]]]
1 ** -2 <=> [[1, '**', -2]]
-1 ** 2 <=> [['-', [1, '**', 2]]]
Doc for pyparsing_common with other predefined expressions
sorry for bringing back such an old subject but I’m writing a very similar parser for my project which mix Boolean logic and mathematical operator and I ended up with such a similar code that it seemed appropriate.
Since the parser given here encounter the same issue than mine, I’m going to use it to show my problem.
I can’t make it parse "$true == not $false", and basically, any "not" after a comparaison won’t work without parenthesis.
$true == $false <=> [['$true', '==', '$false']]
$true == not $false <=> ['$true']
$true == (not $false) <=> [['$true', '==', ['not', '$false']]]
As you can see without the "not" or with parenthesis, it parse fine, but with a simple "not" it seems to ignore everything after the "=="
What kind of fix it is to move the "not" operator first in the infix_notation (previously operatorPrecedence). Then I got those results:
$true == not $false <=> [['$true', '==', ['not', '$false']]]
$true == (not $false) <=> [['$true', '==', ['not', '$false']]]
Which are greats, but of course it broke operator precedence for things like "$true and not 10 == 9" which I’d like to be parse "$true and not (10 == 9)" like in python and instead parse like that:
$true and not 10 == 9 <=> [['$true', 'and', [['not', 10], '==', 9]]]
I was wondering if you encounters those use case in the implementation of your parser and have found a way around them.