Best way to retrieve variable values from a text file?
Question:
Referring on this question, I have a similar -but not the same- problem..
On my way, I’ll have some text file, structured like:
var_a: 'home'
var_b: 'car'
var_c: 15.5
And I need that python read the file and then create a variable named var_a with value ‘home’, and so on.
Example:
#python stuff over here
getVarFromFile(filename) #this is the function that im looking for
print var_b
#output: car, as string
print var_c
#output 15.5, as number.
Is this possible, I mean, even keep the var type?
Notice that I have the full freedom to the text file structure, I can use the format I like if the one I proposed isn’t the best.
EDIT: the ConfigParser can be a solution, but I don’t like it so much, because in my script I’ll have then to refer to the variables in the file with
config.get("set", "var_name")
But what I’ll love is to refer to the variable directly, as I declared it in the python script…
There is a way to import the file as a python dictionary?
Oh, last thing, keep in mind that I don’t know exactly how many variables would I have in the text file.
Edit 2: I’m very interested at stephan’s JSON solution, because in that way the text file could be read simply with others languages (PHP, then via AJAX JavaScript, for example), but I fail in something while acting that solution:
#for the example, i dont load the file but create a var with the supposed file content
file_content = "'var_a': 4, 'var_b': 'a string'"
mydict = dict(file_content)
#Error: ValueError: dictionary update sequence element #0 has length 1; 2 is required
file_content_2 = "{'var_a': 4, 'var_b': 'a string'}"
mydict_2 = dict(json.dump(file_content_2, True))
#Error:
#Traceback (most recent call last):
#File "<pyshell#5>", line 1, in <module>
#mydict_2 = dict(json.dump(file_content_2, True))
#File "C:Python26libjson__init__.py", line 181, in dump
#fp.write(chunk)
#AttributeError: 'bool' object has no attribute 'write'
In what kind of issues can I fall with the JSON format?
And, how can I read a JSON array in a text file, and transform it in a python dict?
P.S: I don’t like the solution using .py files; I’ll prefer .txt, .inc, .whatever is not restrictive to one language.
Answers:
You can treat your text file as a python module and load it dynamically using imp.load_source:
import imp
imp.load_source( name, pathname[, file])
Example:
// mydata.txt
var1 = 'hi'
var2 = 'how are you?'
var3 = { 1:'elem1', 2:'elem2' }
//...
// In your script file
def getVarFromFile(filename):
import imp
f = open(filename)
global data
data = imp.load_source('data', '', f)
f.close()
# path to "config" file
getVarFromFile('c:/mydata.txt')
print data.var1
print data.var2
print data.var3
...
Use ConfigParser.
Your config:
[myvars]
var_a: 'home'
var_b: 'car'
var_c: 15.5
Your python code:
import ConfigParser
config = ConfigParser.ConfigParser()
config.read("config.ini")
var_a = config.get("myvars", "var_a")
var_b = config.get("myvars", "var_b")
var_c = config.get("myvars", "var_c")
Load your file with JSON or PyYAML into a dictionary the_dict (see doc for JSON or PyYAML for this step, both can store data type) and add the dictionary to your globals dictionary, e.g. using globals().update(the_dict).
If you want it in a local dictionary instead (e.g. inside a function), you can do it like this:
for (n, v) in the_dict.items():
exec('%s=%s' % (n, repr(v)))
as long as it is safe to use exec. If not, you can use the dictionary directly.
But what i’ll love is to refer to the variable direclty, as i declared it in the python script..
Assuming you’re happy to change your syntax slightly, just use python and import the “config” module.
# myconfig.py:
var_a = 'home'
var_b = 'car'
var_c = 15.5
Then do
from myconfig import *
And you can reference them by name in your current context.
What you want appear to want is the following, but this is NOT RECOMMENDED:
>>> for line in open('dangerous.txt'):
... exec('%s = %s' % tuple(line.split(':', 1)))
...
>>> var_a
'home'
This creates somewhat similar behavior to PHP’s register_globals and hence has the same security issues. Additionally, the use of exec that I showed allows arbitrary code execution. Only use this if you are absolutely sure that the contents of the text file can be trusted under all circumstances.
You should really consider binding the variables not to the local scope, but to an object, and use a library that parses the file contents such that no code is executed. So: go with any of the other solutions provided here.
(Please note: I added this answer not as a solution, but as an explicit non-solution.)
How reliable is your format? If the seperator is always exactly ‘: ‘, the following works. If not, a comparatively simple regex should do the job.
As long as you’re working with fairly simple variable types, Python’s eval function makes persisting variables to files surprisingly easy.
(The below gives you a dictionary, btw, which you mentioned was one of your prefered solutions).
def read_config(filename):
f = open(filename)
config_dict = {}
for lines in f:
items = lines.split(': ', 1)
config_dict[items[0]] = eval(items[1])
return config_dict
Suppose that you have a file Called “test.txt” with:
a=1.251
b=2.65415
c=3.54
d=549.5645
e=4684.65489
And you want to find a variable (a,b,c,d or e):
ffile=open('test.txt','r').read()
variable=raw_input('Wich is the variable you are looking for?n')
ini=ffile.find(variable)+(len(variable)+1)
rest=ffile[ini:]
search_enter=rest.find('n')
number=float(rest[:search_enter])
print "value:",number
The other solutions posted here didn’t work for me, because:
- i just needed parameters from a file for a normal script
import * didn’t work for me, as i need a way to override them by choosing another file- Just a file with a dict wasn’t fine, as I needed comments in it.
So I ended up using Configparser and globals().update()
Test file:
#File parametertest.cfg:
[Settings]
#Comments are no Problem
test= True
bla= False #Here neither
#that neither
And that’s my demo script:
import ConfigParser
cfg = ConfigParser.RawConfigParser()
cfg.read('parametertest.cfg') # Read file
#print cfg.getboolean('Settings','bla') # Manual Way to acess them
par=dict(cfg.items("Settings"))
for p in par:
par[p]=par[p].split("#",1)[0].strip() # To get rid of inline comments
globals().update(par) #Make them availible globally
print bla
It’s just for a file with one section now, but that will be easy to adopt.
Hope it will be helpful for someone š
hbn’s answer won’t work out of the box if the file to load is in a subdirectory or is named with dashes.
In such a case you may consider this alternativeĀ :
exec open(myconfig.py).read()
Or the simpler but deprecated in python3Ā :
execfile(myconfig.py)
I guess Stephan202’s warning applies to both options, though, and maybe the loop on lines is safer.
If your data has a regular structure you can read a file in line by line and populate your favorite container. For example:
Let’s say your data has 3 variables: x, y, i.
A file contains n of these data, each variable on its own line (3 lines per record). Here are two records:
384
198
0
255
444
2
Here’s how you read your file data into a list. (Reading from text, so cast accordingly.)
data = []
try:
with open(dataFilename, "r") as file:
# read data until end of file
x = file.readline()
while x != "":
x = int(x.strip()) # remove n, cast as int
y = file.readline()
y = int(y.strip())
i = file.readline()
i = int(i.strip())
data.append([x,y,i])
x = file.readline()
except FileNotFoundError as e:
print("File not found:", e)
return(data)
A simple way of reading variables from a text file using the standard library:
# Get vars from conf file
var = {}
with open("myvars.txt") as conf:
for line in conf:
if ":" in line:
name, value = line.split(":")
var[name] = str(value).rstrip()
globals().update(var)
By default this reads each variable as a string. If you need them converted as literal variables(e.g lists/tuples/bool), then you would need to use the ast module:
import ast
ast.literal_eval(varname)
Referring on this question, I have a similar -but not the same- problem..
On my way, I’ll have some text file, structured like:
var_a: 'home'
var_b: 'car'
var_c: 15.5
And I need that python read the file and then create a variable named var_a with value ‘home’, and so on.
Example:
#python stuff over here
getVarFromFile(filename) #this is the function that im looking for
print var_b
#output: car, as string
print var_c
#output 15.5, as number.
Is this possible, I mean, even keep the var type?
Notice that I have the full freedom to the text file structure, I can use the format I like if the one I proposed isn’t the best.
EDIT: the ConfigParser can be a solution, but I don’t like it so much, because in my script I’ll have then to refer to the variables in the file with
config.get("set", "var_name")
But what I’ll love is to refer to the variable directly, as I declared it in the python script…
There is a way to import the file as a python dictionary?
Oh, last thing, keep in mind that I don’t know exactly how many variables would I have in the text file.
Edit 2: I’m very interested at stephan’s JSON solution, because in that way the text file could be read simply with others languages (PHP, then via AJAX JavaScript, for example), but I fail in something while acting that solution:
#for the example, i dont load the file but create a var with the supposed file content
file_content = "'var_a': 4, 'var_b': 'a string'"
mydict = dict(file_content)
#Error: ValueError: dictionary update sequence element #0 has length 1; 2 is required
file_content_2 = "{'var_a': 4, 'var_b': 'a string'}"
mydict_2 = dict(json.dump(file_content_2, True))
#Error:
#Traceback (most recent call last):
#File "<pyshell#5>", line 1, in <module>
#mydict_2 = dict(json.dump(file_content_2, True))
#File "C:Python26libjson__init__.py", line 181, in dump
#fp.write(chunk)
#AttributeError: 'bool' object has no attribute 'write'
In what kind of issues can I fall with the JSON format?
And, how can I read a JSON array in a text file, and transform it in a python dict?
P.S: I don’t like the solution using .py files; I’ll prefer .txt, .inc, .whatever is not restrictive to one language.
You can treat your text file as a python module and load it dynamically using imp.load_source:
import imp
imp.load_source( name, pathname[, file])
Example:
// mydata.txt
var1 = 'hi'
var2 = 'how are you?'
var3 = { 1:'elem1', 2:'elem2' }
//...
// In your script file
def getVarFromFile(filename):
import imp
f = open(filename)
global data
data = imp.load_source('data', '', f)
f.close()
# path to "config" file
getVarFromFile('c:/mydata.txt')
print data.var1
print data.var2
print data.var3
...
Use ConfigParser.
Your config:
[myvars]
var_a: 'home'
var_b: 'car'
var_c: 15.5
Your python code:
import ConfigParser
config = ConfigParser.ConfigParser()
config.read("config.ini")
var_a = config.get("myvars", "var_a")
var_b = config.get("myvars", "var_b")
var_c = config.get("myvars", "var_c")
Load your file with JSON or PyYAML into a dictionary the_dict (see doc for JSON or PyYAML for this step, both can store data type) and add the dictionary to your globals dictionary, e.g. using globals().update(the_dict).
If you want it in a local dictionary instead (e.g. inside a function), you can do it like this:
for (n, v) in the_dict.items():
exec('%s=%s' % (n, repr(v)))
as long as it is safe to use exec. If not, you can use the dictionary directly.
But what i’ll love is to refer to the variable direclty, as i declared it in the python script..
Assuming you’re happy to change your syntax slightly, just use python and import the “config” module.
# myconfig.py:
var_a = 'home'
var_b = 'car'
var_c = 15.5
Then do
from myconfig import *
And you can reference them by name in your current context.
What you want appear to want is the following, but this is NOT RECOMMENDED:
>>> for line in open('dangerous.txt'):
... exec('%s = %s' % tuple(line.split(':', 1)))
...
>>> var_a
'home'
This creates somewhat similar behavior to PHP’s register_globals and hence has the same security issues. Additionally, the use of exec that I showed allows arbitrary code execution. Only use this if you are absolutely sure that the contents of the text file can be trusted under all circumstances.
You should really consider binding the variables not to the local scope, but to an object, and use a library that parses the file contents such that no code is executed. So: go with any of the other solutions provided here.
(Please note: I added this answer not as a solution, but as an explicit non-solution.)
How reliable is your format? If the seperator is always exactly ‘: ‘, the following works. If not, a comparatively simple regex should do the job.
As long as you’re working with fairly simple variable types, Python’s eval function makes persisting variables to files surprisingly easy.
(The below gives you a dictionary, btw, which you mentioned was one of your prefered solutions).
def read_config(filename):
f = open(filename)
config_dict = {}
for lines in f:
items = lines.split(': ', 1)
config_dict[items[0]] = eval(items[1])
return config_dict
Suppose that you have a file Called “test.txt” with:
a=1.251
b=2.65415
c=3.54
d=549.5645
e=4684.65489
And you want to find a variable (a,b,c,d or e):
ffile=open('test.txt','r').read()
variable=raw_input('Wich is the variable you are looking for?n')
ini=ffile.find(variable)+(len(variable)+1)
rest=ffile[ini:]
search_enter=rest.find('n')
number=float(rest[:search_enter])
print "value:",number
The other solutions posted here didn’t work for me, because:
- i just needed parameters from a file for a normal script
import *didn’t work for me, as i need a way to override them by choosing another file- Just a file with a dict wasn’t fine, as I needed comments in it.
So I ended up using Configparser and globals().update()
Test file:
#File parametertest.cfg:
[Settings]
#Comments are no Problem
test= True
bla= False #Here neither
#that neither
And that’s my demo script:
import ConfigParser
cfg = ConfigParser.RawConfigParser()
cfg.read('parametertest.cfg') # Read file
#print cfg.getboolean('Settings','bla') # Manual Way to acess them
par=dict(cfg.items("Settings"))
for p in par:
par[p]=par[p].split("#",1)[0].strip() # To get rid of inline comments
globals().update(par) #Make them availible globally
print bla
It’s just for a file with one section now, but that will be easy to adopt.
Hope it will be helpful for someone š
hbn’s answer won’t work out of the box if the file to load is in a subdirectory or is named with dashes.
In such a case you may consider this alternativeĀ :
exec open(myconfig.py).read()
Or the simpler but deprecated in python3Ā :
execfile(myconfig.py)
I guess Stephan202’s warning applies to both options, though, and maybe the loop on lines is safer.
If your data has a regular structure you can read a file in line by line and populate your favorite container. For example:
Let’s say your data has 3 variables: x, y, i.
A file contains n of these data, each variable on its own line (3 lines per record). Here are two records:
384
198
0
255
444
2
Here’s how you read your file data into a list. (Reading from text, so cast accordingly.)
data = []
try:
with open(dataFilename, "r") as file:
# read data until end of file
x = file.readline()
while x != "":
x = int(x.strip()) # remove n, cast as int
y = file.readline()
y = int(y.strip())
i = file.readline()
i = int(i.strip())
data.append([x,y,i])
x = file.readline()
except FileNotFoundError as e:
print("File not found:", e)
return(data)
A simple way of reading variables from a text file using the standard library:
# Get vars from conf file
var = {}
with open("myvars.txt") as conf:
for line in conf:
if ":" in line:
name, value = line.split(":")
var[name] = str(value).rstrip()
globals().update(var)
By default this reads each variable as a string. If you need them converted as literal variables(e.g lists/tuples/bool), then you would need to use the ast module:
import ast
ast.literal_eval(varname)