How to save a list to a file and read it as a list type?
Question:
Say I have the list score = [1,2,3,4,5]
and it gets changed while my program is running. How could I save it to a file so that next time the program is run I can access the changed list as a list
type?
I have tried:
score=[1,2,3,4,5]
with open("file.txt", 'w') as f:
for s in score:
f.write(str(s) + 'n')
with open("file.txt", 'r') as f:
score = [line.rstrip('n') for line in f]
print(score)
But this results in the elements in the list being strings not integers.
Answers:
You can use the pickle
module for that.
This module has two methods,
- Pickling(dump): Convert Python objects into a string representation.
- Unpickling(load): Retrieving original objects from a stored string representation.
https://docs.python.org/3.3/library/pickle.html
Code:
>>> import pickle
>>> l = [1,2,3,4]
>>> with open("test", "wb") as fp: #Pickling
... pickle.dump(l, fp)
...
>>> with open("test", "rb") as fp: # Unpickling
... b = pickle.load(fp)
...
>>> b
[1, 2, 3, 4]
Also Json
- dump/dumps: Serialize
- load/loads: Deserialize
https://docs.python.org/3/library/json.html
Code:
>>> import json
>>> with open("test", "w") as fp:
... json.dump(l, fp)
...
>>> with open("test", "r") as fp:
... b = json.load(fp)
...
>>> b
[1, 2, 3, 4]
pickle
and other serialization packages work. So does writing it to a .py
file that you can then import.
>>> score = [1,2,3,4,5]
>>>
>>> with open('file.py', 'w') as f:
... f.write('score = %s' % score)
...
>>> from file import score as my_list
>>> print(my_list)
[1, 2, 3, 4, 5]
I decided I didn’t want to use a pickle because I wanted to be able to open the text file and change its contents easily during testing. Therefore, I did this:
score = [1,2,3,4,5]
with open("file.txt", "w") as f:
for s in score:
f.write(str(s) +"n")
score = []
with open("file.txt", "r") as f:
for line in f:
score.append(int(line.strip()))
So the items in the file are read as integers, despite being stored to the file as strings.
If you don’t want to use pickle, you can store the list as text and then evaluate it:
data = [0,1,2,3,4,5]
with open("test.txt", "w") as file:
file.write(str(data))
with open("test.txt", "r") as file:
data2 = eval(file.readline())
# Let's see if data and types are same.
print(data, type(data), type(data[0]))
print(data2, type(data2), type(data2[0]))
[0, 1, 2, 3, 4, 5] class ‘list’ class ‘int’
[0, 1, 2, 3, 4, 5] class ‘list’ class ‘int’
I am using pandas.
import pandas as pd
x = pd.Series([1,2,3,4,5])
x.to_excel('temp.xlsx')
y = list(pd.read_excel('temp.xlsx')[0])
print(y)
Use this if you are anyway importing pandas for other computations.
If you want you can use numpy’s save function to save the list as file.
Say you have two lists
sampleList1=['z','x','a','b']
sampleList2=[[1,2],[4,5]]
here’s the function to save the list as file, remember you need to keep the extension .npy
def saveList(myList,filename):
# the filename should mention the extension 'npy'
np.save(filename,myList)
print("Saved successfully!")
and here’s the function to load the file into a list
def loadList(filename):
# the filename should mention the extension 'npy'
tempNumpyArray=np.load(filename)
return tempNumpyArray.tolist()
a working example
>>> saveList(sampleList1,'sampleList1.npy')
>>> Saved successfully!
>>> saveList(sampleList2,'sampleList2.npy')
>>> Saved successfully!
# loading the list now
>>> loadedList1=loadList('sampleList1.npy')
>>> loadedList2=loadList('sampleList2.npy')
>>> loadedList1==sampleList1
>>> True
>>> print(loadedList1,sampleList1)
>>> ['z', 'x', 'a', 'b'] ['z', 'x', 'a', 'b']
What I did not like with many answers is that it makes way too many system calls by writing to the file line per line. Imho it is best to join list with ‘n’ (line return) and then write it only once to the file:
mylist = ["abc", "def", "ghi"]
myfile = "file.txt"
with open(myfile, 'w') as f:
f.write("n".join(mylist))
and then to open it and get your list again:
with open(myfile, 'r') as f:
mystring = f.read()
my_list = mystring.split("n")
While the accepted answer works, you should really be using python’s json
module (see end of post for comparison with pickle
):
import json
score=[1,2,3,4,5]
with open("file.json", 'w') as f:
# indent=2 is not needed but makes the file human-readable
# if the data is nested
json.dump(score, f, indent=2)
with open("file.json", 'r') as f:
score = json.load(f)
print(score)
Advantages:
json
is a widely adopted and standardized data format, so non-python programs can easily read and understand the json files
json
files are human-readable and easy to edit (plain text)
- Any nested or non-nested list/dictionary structure can be saved to a
json
file (as long as all the contents are serializable).
Disadvantages:
- The data is stored in plain-text (ie it’s uncompressed), which makes it a slow and space-inefficient option for large amounts of data.
- The contents of a list/dictionary need to be serializable before you can save it as a json. The
json
module will let you save strings, ints, floats, boolean, and None values, you’ll need to write custom serialization and deserialization code to save objects, classes, and functions.
pickle
vs json
, which one should I use?:
- If you want to store something you know you’re only ever going to use in the context of a python program, use
pickle
- If you need to save data that isn’t serializable by default (ie objects), save yourself the trouble and use
pickle
- If you need a platform agnostic solution, use
json
- If you need to be able to inspect and edit the data directly, use
json
- If you need something robust and long-term, use
json
(pickle
won’t work correctly if you change the location of classes/files or make breaking changes to the code)
errorlist = ['aaaa', 'bbbb', 'cccc', 'dddd']
f = open("filee.txt", "w")
f.writelines(nthstring + 'n' for nthstring in errorlist)
f = open("filee.txt", "r")
cont = f.read()
contentlist = cont.split()
print(contentlist)
I had similar problem where I needed to read list saved as text file. The list had multiple layers so using split would not help.
For example:
list1.txt
[(1,2,3),['a','b'],'a1']
so what I did , I changed list.txt to list.py and then imported list from python file.
For example:
list1.py
a = [(1,2,3),['a','b'],'a1']
Then:
from list1 import a
print(a)
Say I have the list score = [1,2,3,4,5]
and it gets changed while my program is running. How could I save it to a file so that next time the program is run I can access the changed list as a list
type?
I have tried:
score=[1,2,3,4,5]
with open("file.txt", 'w') as f:
for s in score:
f.write(str(s) + 'n')
with open("file.txt", 'r') as f:
score = [line.rstrip('n') for line in f]
print(score)
But this results in the elements in the list being strings not integers.
You can use the pickle
module for that.
This module has two methods,
- Pickling(dump): Convert Python objects into a string representation.
- Unpickling(load): Retrieving original objects from a stored string representation.
https://docs.python.org/3.3/library/pickle.html
Code:
>>> import pickle
>>> l = [1,2,3,4]
>>> with open("test", "wb") as fp: #Pickling
... pickle.dump(l, fp)
...
>>> with open("test", "rb") as fp: # Unpickling
... b = pickle.load(fp)
...
>>> b
[1, 2, 3, 4]
Also Json
- dump/dumps: Serialize
- load/loads: Deserialize
https://docs.python.org/3/library/json.html
Code:
>>> import json
>>> with open("test", "w") as fp:
... json.dump(l, fp)
...
>>> with open("test", "r") as fp:
... b = json.load(fp)
...
>>> b
[1, 2, 3, 4]
pickle
and other serialization packages work. So does writing it to a .py
file that you can then import.
>>> score = [1,2,3,4,5]
>>>
>>> with open('file.py', 'w') as f:
... f.write('score = %s' % score)
...
>>> from file import score as my_list
>>> print(my_list)
[1, 2, 3, 4, 5]
I decided I didn’t want to use a pickle because I wanted to be able to open the text file and change its contents easily during testing. Therefore, I did this:
score = [1,2,3,4,5]
with open("file.txt", "w") as f:
for s in score:
f.write(str(s) +"n")
score = []
with open("file.txt", "r") as f:
for line in f:
score.append(int(line.strip()))
So the items in the file are read as integers, despite being stored to the file as strings.
If you don’t want to use pickle, you can store the list as text and then evaluate it:
data = [0,1,2,3,4,5]
with open("test.txt", "w") as file:
file.write(str(data))
with open("test.txt", "r") as file:
data2 = eval(file.readline())
# Let's see if data and types are same.
print(data, type(data), type(data[0]))
print(data2, type(data2), type(data2[0]))
[0, 1, 2, 3, 4, 5] class ‘list’ class ‘int’
[0, 1, 2, 3, 4, 5] class ‘list’ class ‘int’
I am using pandas.
import pandas as pd
x = pd.Series([1,2,3,4,5])
x.to_excel('temp.xlsx')
y = list(pd.read_excel('temp.xlsx')[0])
print(y)
Use this if you are anyway importing pandas for other computations.
If you want you can use numpy’s save function to save the list as file.
Say you have two lists
sampleList1=['z','x','a','b']
sampleList2=[[1,2],[4,5]]
here’s the function to save the list as file, remember you need to keep the extension .npy
def saveList(myList,filename):
# the filename should mention the extension 'npy'
np.save(filename,myList)
print("Saved successfully!")
and here’s the function to load the file into a list
def loadList(filename):
# the filename should mention the extension 'npy'
tempNumpyArray=np.load(filename)
return tempNumpyArray.tolist()
a working example
>>> saveList(sampleList1,'sampleList1.npy')
>>> Saved successfully!
>>> saveList(sampleList2,'sampleList2.npy')
>>> Saved successfully!
# loading the list now
>>> loadedList1=loadList('sampleList1.npy')
>>> loadedList2=loadList('sampleList2.npy')
>>> loadedList1==sampleList1
>>> True
>>> print(loadedList1,sampleList1)
>>> ['z', 'x', 'a', 'b'] ['z', 'x', 'a', 'b']
What I did not like with many answers is that it makes way too many system calls by writing to the file line per line. Imho it is best to join list with ‘n’ (line return) and then write it only once to the file:
mylist = ["abc", "def", "ghi"]
myfile = "file.txt"
with open(myfile, 'w') as f:
f.write("n".join(mylist))
and then to open it and get your list again:
with open(myfile, 'r') as f:
mystring = f.read()
my_list = mystring.split("n")
While the accepted answer works, you should really be using python’s json
module (see end of post for comparison with pickle
):
import json
score=[1,2,3,4,5]
with open("file.json", 'w') as f:
# indent=2 is not needed but makes the file human-readable
# if the data is nested
json.dump(score, f, indent=2)
with open("file.json", 'r') as f:
score = json.load(f)
print(score)
Advantages:
json
is a widely adopted and standardized data format, so non-python programs can easily read and understand the json filesjson
files are human-readable and easy to edit (plain text)- Any nested or non-nested list/dictionary structure can be saved to a
json
file (as long as all the contents are serializable).
Disadvantages:
- The data is stored in plain-text (ie it’s uncompressed), which makes it a slow and space-inefficient option for large amounts of data.
- The contents of a list/dictionary need to be serializable before you can save it as a json. The
json
module will let you save strings, ints, floats, boolean, and None values, you’ll need to write custom serialization and deserialization code to save objects, classes, and functions.
pickle
vs json
, which one should I use?:
- If you want to store something you know you’re only ever going to use in the context of a python program, use
pickle
- If you need to save data that isn’t serializable by default (ie objects), save yourself the trouble and use
pickle
- If you need a platform agnostic solution, use
json
- If you need to be able to inspect and edit the data directly, use
json
- If you need something robust and long-term, use
json
(pickle
won’t work correctly if you change the location of classes/files or make breaking changes to the code)
errorlist = ['aaaa', 'bbbb', 'cccc', 'dddd']
f = open("filee.txt", "w")
f.writelines(nthstring + 'n' for nthstring in errorlist)
f = open("filee.txt", "r")
cont = f.read()
contentlist = cont.split()
print(contentlist)
I had similar problem where I needed to read list saved as text file. The list had multiple layers so using split would not help.
For example:
list1.txt
[(1,2,3),['a','b'],'a1']
so what I did , I changed list.txt to list.py and then imported list from python file.
For example:
list1.py
a = [(1,2,3),['a','b'],'a1']
Then:
from list1 import a
print(a)