Using python's eval() vs. ast.literal_eval()
Question:
I have a situation with some code where eval() came up as a possible solution. Now I have never had to use eval() before but, I have come across plenty of information about the potential danger it can cause. That said, I’m very wary about using it.
My situation is that I have input being given by a user:
datamap = input('Provide some data here: ')
Where datamap needs to be a dictionary. I searched around and found that eval() could work this out. I thought that I might be able to check the type of the input before trying to use the data and that would be a viable security precaution.
datamap = eval(input('Provide some data here: ')
if not isinstance(datamap, dict):
return
I read through the docs and I am still unclear if this would be safe or not. Does eval evaluate the data as soon as its entered or after the datamap variable is called?
Is the ast module’s .literal_eval() the only safe option?
Answers:
Python’s eager in its evaluation, so eval(input(...)) (Python 3) will evaluate the user’s input as soon as it hits the eval, regardless of what you do with the data afterwards. Therefore, this is not safe, especially when you eval user input.
Use ast.literal_eval.
As an example, entering this at the prompt could be very bad for you:
__import__('os').system('rm -rf /a-path-you-really-care-about')
datamap = eval(input('Provide some data here: ')) means that you actually evaluate the code before you deem it to be unsafe or not. It evaluates the code as soon as the function is called. See also the dangers of eval.
ast.literal_eval raises an exception if the input isn’t a valid Python datatype, so the code won’t be executed if it’s not.
Use ast.literal_eval whenever you need eval. You shouldn’t usually evaluate literal Python statements.
ast.literal_eval() only considers a small subset of Python’s syntax to be valid:
The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None.
Passing __import__('os').system('rm -rf /a-path-you-really-care-about') into ast.literal_eval() will raise an error, but eval() will happily delete your files.
Since it looks like you’re only letting the user input a plain dictionary, use ast.literal_eval(). It safely does what you want and nothing more.
eval:
This is very powerful, but is also very dangerous if you accept strings to evaluate from untrusted input. Suppose the string being evaluated is “os.system(‘rm -rf /’)” ? It will really start deleting all the files on your computer.
ast.literal_eval:
Safely evaluate an expression node or a string containing a Python literal or container display. The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, None, bytes and sets.
Syntax:
eval(expression, globals=None, locals=None)
import ast
ast.literal_eval(node_or_string)
Example:
# python 2.x - doesn't accept operators in string format
import ast
ast.literal_eval('[1, 2, 3]') # output: [1, 2, 3]
ast.literal_eval('1+1') # output: ValueError: malformed string
# python 3.0 -3.6
import ast
ast.literal_eval("1+1") # output : 2
ast.literal_eval("{'a': 2, 'b': 3, 3:'xyz'}") # output : {'a': 2, 'b': 3, 3:'xyz'}
# type dictionary
ast.literal_eval("",{}) # output : Syntax Error required only one parameter
ast.literal_eval("__import__('os').system('rm -rf /')") # output : error
eval("__import__('os').system('rm -rf /')")
# output : start deleting all the files on your computer.
# restricting using global and local variables
eval("__import__('os').system('rm -rf /')",{'__builtins__':{}},{})
# output : Error due to blocked imports by passing '__builtins__':{} in global
# But still eval is not safe. we can access and break the code as given below
s = """
(lambda fc=(
lambda n: [
c for c in
().__class__.__bases__[0].__subclasses__()
if c.__name__ == n
][0]
):
fc("function")(
fc("code")(
0,0,0,0,"KABOOM",(),(),(),"","",0,""
),{}
)()
)()
"""
eval(s, {'__builtins__':{}})
In the above code ().__class__.__bases__[0] nothing but object itself.
Now we instantiated all the subclasses, here our main enter code hereobjective is to find one class named n from it.
We need to code object and function object from instantiated subclasses. This is an alternative way from CPython to access subclasses of object and attach the system.
From python 3.7 ast.literal_eval() is now stricter. Addition and subtraction of arbitrary numbers are no longer allowed. link
If all you need is a user provided dictionary, a possible better solution is json.loads. The main limitation is that JSON dicts ("objects") require string keys. Also you can only provide literal data, but that is also the case for ast.literal_eval.
I was stuck with ast.literal_eval(). I was trying it in IntelliJ IDEA debugger, and it kept returning None on debugger output.
But later when I assigned its output to a variable and printed it in code, it worked fine. Sharing code example:
import ast
sample_string = '[{"id":"XYZ_GTTC_TYR", "name":"Suction"}]'
output_value = ast.literal_eval(sample_string)
print(output_value)
It’s python version 3.6.
In recent Python3 ast.literal_eval() no longer parses simple strings, instead you are supposed to use the ast.parse() method to create an AST then interpret it.
This is a complete example of using ast.parse() correctly in Python 3.6+ to evaluate simple arithmetic expressions safely.
import ast, operator, math
import logging
logger = logging.getLogger(__file__)
def safe_eval(s):
def checkmath(x, *args):
if x not in [x for x in dir(math) if not "__" in x]:
raise SyntaxError(f"Unknown func {x}()")
fun = getattr(math, x)
return fun(*args)
binOps = {
ast.Add: operator.add,
ast.Sub: operator.sub,
ast.Mult: operator.mul,
ast.Div: operator.truediv,
ast.Mod: operator.mod,
ast.Pow: operator.pow,
ast.Call: checkmath,
ast.BinOp: ast.BinOp,
}
unOps = {
ast.USub: operator.neg,
ast.UAdd: operator.pos,
ast.UnaryOp: ast.UnaryOp,
}
ops = tuple(binOps) + tuple(unOps)
tree = ast.parse(s, mode='eval')
def _eval(node):
if isinstance(node, ast.Expression):
logger.debug("Expr")
return _eval(node.body)
elif isinstance(node, ast.Str):
logger.debug("Str")
return node.s
elif isinstance(node, ast.Num):
logger.debug("Num")
return node.value
elif isinstance(node, ast.Constant):
logger.info("Const")
return node.value
elif isinstance(node, ast.BinOp):
logger.debug("BinOp")
if isinstance(node.left, ops):
left = _eval(node.left)
else:
left = node.left.value
if isinstance(node.right, ops):
right = _eval(node.right)
else:
right = node.right.value
return binOps[type(node.op)](left, right)
elif isinstance(node, ast.UnaryOp):
logger.debug("UpOp")
if isinstance(node.operand, ops):
operand = _eval(node.operand)
else:
operand = node.operand.value
return unOps[type(node.op)](operand)
elif isinstance(node, ast.Call):
args = [_eval(x) for x in node.args]
r = checkmath(node.func.id, *args)
return r
else:
raise SyntaxError(f"Bad syntax, {type(node)}")
return _eval(tree)
if __name__ == "__main__":
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
logger.addHandler(ch)
assert safe_eval("1+1") == 2
assert safe_eval("1+-5") == -4
assert safe_eval("-1") == -1
assert safe_eval("-+1") == -1
assert safe_eval("(100*10)+6") == 1006
assert safe_eval("100*(10+6)") == 1600
assert safe_eval("2**4") == 2**4
assert safe_eval("sqrt(16)+1") == math.sqrt(16) + 1
assert safe_eval("1.2345 * 10") == 1.2345 * 10
print("Tests pass")
I have a situation with some code where eval() came up as a possible solution. Now I have never had to use eval() before but, I have come across plenty of information about the potential danger it can cause. That said, I’m very wary about using it.
My situation is that I have input being given by a user:
datamap = input('Provide some data here: ')
Where datamap needs to be a dictionary. I searched around and found that eval() could work this out. I thought that I might be able to check the type of the input before trying to use the data and that would be a viable security precaution.
datamap = eval(input('Provide some data here: ')
if not isinstance(datamap, dict):
return
I read through the docs and I am still unclear if this would be safe or not. Does eval evaluate the data as soon as its entered or after the datamap variable is called?
Is the ast module’s .literal_eval() the only safe option?
Python’s eager in its evaluation, so eval(input(...)) (Python 3) will evaluate the user’s input as soon as it hits the eval, regardless of what you do with the data afterwards. Therefore, this is not safe, especially when you eval user input.
Use ast.literal_eval.
As an example, entering this at the prompt could be very bad for you:
__import__('os').system('rm -rf /a-path-you-really-care-about')
datamap = eval(input('Provide some data here: ')) means that you actually evaluate the code before you deem it to be unsafe or not. It evaluates the code as soon as the function is called. See also the dangers of eval.
ast.literal_eval raises an exception if the input isn’t a valid Python datatype, so the code won’t be executed if it’s not.
Use ast.literal_eval whenever you need eval. You shouldn’t usually evaluate literal Python statements.
ast.literal_eval() only considers a small subset of Python’s syntax to be valid:
The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and
None.
Passing __import__('os').system('rm -rf /a-path-you-really-care-about') into ast.literal_eval() will raise an error, but eval() will happily delete your files.
Since it looks like you’re only letting the user input a plain dictionary, use ast.literal_eval(). It safely does what you want and nothing more.
eval:
This is very powerful, but is also very dangerous if you accept strings to evaluate from untrusted input. Suppose the string being evaluated is “os.system(‘rm -rf /’)” ? It will really start deleting all the files on your computer.
ast.literal_eval:
Safely evaluate an expression node or a string containing a Python literal or container display. The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, None, bytes and sets.
Syntax:
eval(expression, globals=None, locals=None)
import ast
ast.literal_eval(node_or_string)
Example:
# python 2.x - doesn't accept operators in string format
import ast
ast.literal_eval('[1, 2, 3]') # output: [1, 2, 3]
ast.literal_eval('1+1') # output: ValueError: malformed string
# python 3.0 -3.6
import ast
ast.literal_eval("1+1") # output : 2
ast.literal_eval("{'a': 2, 'b': 3, 3:'xyz'}") # output : {'a': 2, 'b': 3, 3:'xyz'}
# type dictionary
ast.literal_eval("",{}) # output : Syntax Error required only one parameter
ast.literal_eval("__import__('os').system('rm -rf /')") # output : error
eval("__import__('os').system('rm -rf /')")
# output : start deleting all the files on your computer.
# restricting using global and local variables
eval("__import__('os').system('rm -rf /')",{'__builtins__':{}},{})
# output : Error due to blocked imports by passing '__builtins__':{} in global
# But still eval is not safe. we can access and break the code as given below
s = """
(lambda fc=(
lambda n: [
c for c in
().__class__.__bases__[0].__subclasses__()
if c.__name__ == n
][0]
):
fc("function")(
fc("code")(
0,0,0,0,"KABOOM",(),(),(),"","",0,""
),{}
)()
)()
"""
eval(s, {'__builtins__':{}})
In the above code ().__class__.__bases__[0] nothing but object itself.
Now we instantiated all the subclasses, here our main enter code hereobjective is to find one class named n from it.
We need to code object and function object from instantiated subclasses. This is an alternative way from CPython to access subclasses of object and attach the system.
From python 3.7 ast.literal_eval() is now stricter. Addition and subtraction of arbitrary numbers are no longer allowed. link
If all you need is a user provided dictionary, a possible better solution is json.loads. The main limitation is that JSON dicts ("objects") require string keys. Also you can only provide literal data, but that is also the case for ast.literal_eval.
I was stuck with ast.literal_eval(). I was trying it in IntelliJ IDEA debugger, and it kept returning None on debugger output.
But later when I assigned its output to a variable and printed it in code, it worked fine. Sharing code example:
import ast
sample_string = '[{"id":"XYZ_GTTC_TYR", "name":"Suction"}]'
output_value = ast.literal_eval(sample_string)
print(output_value)
It’s python version 3.6.
In recent Python3 ast.literal_eval() no longer parses simple strings, instead you are supposed to use the ast.parse() method to create an AST then interpret it.
This is a complete example of using ast.parse() correctly in Python 3.6+ to evaluate simple arithmetic expressions safely.
import ast, operator, math
import logging
logger = logging.getLogger(__file__)
def safe_eval(s):
def checkmath(x, *args):
if x not in [x for x in dir(math) if not "__" in x]:
raise SyntaxError(f"Unknown func {x}()")
fun = getattr(math, x)
return fun(*args)
binOps = {
ast.Add: operator.add,
ast.Sub: operator.sub,
ast.Mult: operator.mul,
ast.Div: operator.truediv,
ast.Mod: operator.mod,
ast.Pow: operator.pow,
ast.Call: checkmath,
ast.BinOp: ast.BinOp,
}
unOps = {
ast.USub: operator.neg,
ast.UAdd: operator.pos,
ast.UnaryOp: ast.UnaryOp,
}
ops = tuple(binOps) + tuple(unOps)
tree = ast.parse(s, mode='eval')
def _eval(node):
if isinstance(node, ast.Expression):
logger.debug("Expr")
return _eval(node.body)
elif isinstance(node, ast.Str):
logger.debug("Str")
return node.s
elif isinstance(node, ast.Num):
logger.debug("Num")
return node.value
elif isinstance(node, ast.Constant):
logger.info("Const")
return node.value
elif isinstance(node, ast.BinOp):
logger.debug("BinOp")
if isinstance(node.left, ops):
left = _eval(node.left)
else:
left = node.left.value
if isinstance(node.right, ops):
right = _eval(node.right)
else:
right = node.right.value
return binOps[type(node.op)](left, right)
elif isinstance(node, ast.UnaryOp):
logger.debug("UpOp")
if isinstance(node.operand, ops):
operand = _eval(node.operand)
else:
operand = node.operand.value
return unOps[type(node.op)](operand)
elif isinstance(node, ast.Call):
args = [_eval(x) for x in node.args]
r = checkmath(node.func.id, *args)
return r
else:
raise SyntaxError(f"Bad syntax, {type(node)}")
return _eval(tree)
if __name__ == "__main__":
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
logger.addHandler(ch)
assert safe_eval("1+1") == 2
assert safe_eval("1+-5") == -4
assert safe_eval("-1") == -1
assert safe_eval("-+1") == -1
assert safe_eval("(100*10)+6") == 1006
assert safe_eval("100*(10+6)") == 1600
assert safe_eval("2**4") == 2**4
assert safe_eval("sqrt(16)+1") == math.sqrt(16) + 1
assert safe_eval("1.2345 * 10") == 1.2345 * 10
print("Tests pass")