How to do virtual file processing?
Question:
So for creating files I use the following:
fileHandle = open('fileName', 'w')
then write the contents to the file, close the file.
In the next step I process the file.
At the end of the program, I end up with a “physical file” that I need to delete.
Is there a way to write a “virtual” file that behaves exactly like a “physical” one (allowing it to be manipulated the same way) but does not exist at the end of the run in Python?
Answers:
You have StringIO and BytesIO in the io module.
StringIO behaves like a file opened in text mode – reading and writing unicode strings (equivalent to opening a file with io.open(filename, mode, encoding='...')), and the BytesIO behaves like a file opened in binary mode (mode='[rw]b'), and can read write bytes.
Python 2:
In [4]: f = io.BytesIO('test')
In [5]: type(f.read())
Out[5]: str
In [6]: f = io.StringIO(u'test')
In [7]: type(f.read())
Out[7]: unicode
Python 3:
In [2]: f = io.BytesIO(b'test')
In [3]: type(f.read())
Out[3]: builtins.bytes
In [4]: f = io.StringIO('test')
In [5]: type(f.read())
Out[5]: builtins.str
You can use StringIO as a virtual file ,
from official documentation
from io import StringIO
output = StringIO()
output.write('First line.n')
print >>output, 'Second line.'
# Retrieve file contents -- this will be
# 'First line.nSecond line.n'
contents = output.getvalue()
# Close object and discard memory buffer --
# .getvalue() will now raise an exception.
output.close()
There is the StringIO module, read its documentation, it should be easy to use.
Bear in mind, though, that this would keep the “file’s” contents in memory. If you have too much data, it would probably be better to create a real file, e.g. in /tmp, and delete it afterwards.
You might want to consider using a tempfile.SpooledTemporaryFile which gives you the best of both worlds in the sense that it will create a temporary memory-based virtual file initially but will automatically switch to a physical disk-based file if the data held in memory exceeds a specified size.
Another nice feature is that (when using memory) it will automatically use either an io.BytesIO or io.StringIO depending on what mode is being used—allowing you to either read and write Unicode strings or binary data (bytes) to it.
The only tricky part might be the fact that you’ll need to avoid closing the file between steps because doing so would cause it to be deleted from memory or disk. Instead you can just rewind it back to the beginning with a file seek(0) method call.
When you are completely done with the file and close it, it will automatically be deleted from disk if the amount of data in it caused it to be rolled-over to a physical file.
If you mean writing to memory instead of a file, you can simply write the text to a buffer and use the following function:
def write(text):
global buffer
buffer += text + 'n' # Add a linefeed as you would if you were writing to a file
buffer = "" # Initialize the buffer
write("My name is Steve Grafton")
At the end, you will have a buffer that will be the same as if you had written your stuff to a file and then open the file and read all its contents to a buffer!
Moreover, you can use the buffer during the process (before having finished your writing) and do searches in it, as if you had created a file for both reading and writing, only that in this case your pointer will
So for creating files I use the following:
fileHandle = open('fileName', 'w')
then write the contents to the file, close the file.
In the next step I process the file.
At the end of the program, I end up with a “physical file” that I need to delete.
Is there a way to write a “virtual” file that behaves exactly like a “physical” one (allowing it to be manipulated the same way) but does not exist at the end of the run in Python?
You have StringIO and BytesIO in the io module.
StringIO behaves like a file opened in text mode – reading and writing unicode strings (equivalent to opening a file with io.open(filename, mode, encoding='...')), and the BytesIO behaves like a file opened in binary mode (mode='[rw]b'), and can read write bytes.
Python 2:
In [4]: f = io.BytesIO('test')
In [5]: type(f.read())
Out[5]: str
In [6]: f = io.StringIO(u'test')
In [7]: type(f.read())
Out[7]: unicode
Python 3:
In [2]: f = io.BytesIO(b'test')
In [3]: type(f.read())
Out[3]: builtins.bytes
In [4]: f = io.StringIO('test')
In [5]: type(f.read())
Out[5]: builtins.str
You can use StringIO as a virtual file ,
from official documentation
from io import StringIO
output = StringIO()
output.write('First line.n')
print >>output, 'Second line.'
# Retrieve file contents -- this will be
# 'First line.nSecond line.n'
contents = output.getvalue()
# Close object and discard memory buffer --
# .getvalue() will now raise an exception.
output.close()
There is the StringIO module, read its documentation, it should be easy to use.
Bear in mind, though, that this would keep the “file’s” contents in memory. If you have too much data, it would probably be better to create a real file, e.g. in /tmp, and delete it afterwards.
You might want to consider using a tempfile.SpooledTemporaryFile which gives you the best of both worlds in the sense that it will create a temporary memory-based virtual file initially but will automatically switch to a physical disk-based file if the data held in memory exceeds a specified size.
Another nice feature is that (when using memory) it will automatically use either an io.BytesIO or io.StringIO depending on what mode is being used—allowing you to either read and write Unicode strings or binary data (bytes) to it.
The only tricky part might be the fact that you’ll need to avoid closing the file between steps because doing so would cause it to be deleted from memory or disk. Instead you can just rewind it back to the beginning with a file seek(0) method call.
When you are completely done with the file and close it, it will automatically be deleted from disk if the amount of data in it caused it to be rolled-over to a physical file.
If you mean writing to memory instead of a file, you can simply write the text to a buffer and use the following function:
def write(text):
global buffer
buffer += text + 'n' # Add a linefeed as you would if you were writing to a file
buffer = "" # Initialize the buffer
write("My name is Steve Grafton")
At the end, you will have a buffer that will be the same as if you had written your stuff to a file and then open the file and read all its contents to a buffer!
Moreover, you can use the buffer during the process (before having finished your writing) and do searches in it, as if you had created a file for both reading and writing, only that in this case your pointer will