What is StringIO in python used for in reality?
Question:
What exactly is StringIO used for?
I have been looking around the internet for some examples. However, almost all of the examples are very abstract. And they just show "how" to use it. But none of them show "why" and "in which circumstances" one should/will use it?
p.s. not to be confused with this question on stackoverflow: StringIO Usage which compares string and StringIo.
Answers:
StringIO gives you file-like access to strings, so you can use an existing module that deals with a file and change almost nothing and make it work with strings.
For example, say you have a logger that writes things to a file and you want to instead send the log output over the network. You can read the file and write its contents to the network, or you can write the log to a StringIO object and ship it off to its network destination without touching the filesystem. StringIO makes it easy to do it the first way then switch to the second way.
It’s used when you have some API that only takes files, but you need to use a string. For example, to compress a string using the gzip module in Python 2:
import gzip
import StringIO
stringio = StringIO.StringIO()
gzip_file = gzip.GzipFile(fileobj=stringio, mode='w')
gzip_file.write('Hello World')
gzip_file.close()
stringio.getvalue()
In cases where you want a file-like object that ACTS like a file, but is writing to an in-memory string buffer: StringIO is the tool. If you’re building large strings, such as plain-text documents, and doing a lot of string concatenation, you might find it easier to just use StringIO instead of a bunch of mystr += 'more stuffn' type of operations.
Couple of things I personally have used it for:
-
Whole-file caching. I have a script that reads PDFs and does validation of various things about them. The PDF library I’m using takes an open file in its document constructor. I originally just opened the PDF I was interested in reading, however when I changed it to read the entire file at once into memory then pass a StringIO object to the PDF library, the running time of my script was cut in half.
-
Deferred printing. Same script prints a header before every PDF it reads. However, I can specify on the command line whether to ignore certain tests that are in its configuration file, or to only include certain ones. If I ignore all tests for a given PDF I don’t want the header printed, but I won’t know how many tests I ran until I’m done running the tests (the tests can be defined dynamically as well). So I capture the header into a StringIO object by changing sys.stdout to point to it, and each time I run a test I check to see whether that object has anything in it. If so, I print it then and reset it to empty. Voila, only PDFs that have tests have headers printed.
I’ve just used StringIO in practice for two things:
- To unit-test a script that does a lot of
printing, by redirecting sys.stdout to a StringIO instance for easy analysis;
- To create a guaranteed well-formed XML document (a custom API request) using
ElementTree and then write it for sending via a HTTP connection.
Not that you need StringIO often, but sometimes it’s pretty useful.
Django has a function call_command which is used to call management commands. This function prints output to stdout and doesn’t return any value. If you want to know whether the command ran successfully or not, you have to look into output and decide.
Using StringIO, you can capture output and check if it is desired output or not.
with io.StringIO() as output:
call_command('custom_command', stdout=output)
if 'Success' not in output.getvalue():
print('Custom command failed...')
I’ve used it in place of text files for unit-testing.
For example, to make a csv ‘file’ for testing with pandas (Python 3):
import io
import pandas as pd
f = io.StringIO("id,namen1,briann2,amandan3,zoeyn")
df = pd.read_csv(f) # pandas takes a file path or a file-like object
From the documentation here:
An in-memory stream for text I/O. The text buffer is discarded when the close() method is called.
The initial value of the buffer can be set by providing initial_value.
method getvalue(): Return a str containing the entire contents of the buffer.
Here’s a concrete example of a use case for StringIO: writing some data directly to aws s3, without needing to create a file on local disk:
import csv
import io
import boto3
data = [
["test", "data", "headers etc", "123","",],
["blah", "123", "35", "blah","",],
["abc", "def", "blah", "yep", "blah"]
]
bucket_name = 'bucket_name_here'
session = boto3.Session(
aws_access_key_id = "fake Access ID"),
aws_secret_access_key = "fake Secret key"),
region_name = "ap-southeast-2")
)
s3 = session.resource('s3')
with io.StringIO() as f:
writer = csv.writer(f, delimiter=",")
writer.writerows(data)
resp = s3.Object(bucket_name, "test.csv").put(Body=f.getvalue())
Enjoy your new csv on S3, without having written anything to your local disk!
What exactly is StringIO used for?
I have been looking around the internet for some examples. However, almost all of the examples are very abstract. And they just show "how" to use it. But none of them show "why" and "in which circumstances" one should/will use it?
p.s. not to be confused with this question on stackoverflow: StringIO Usage which compares string and StringIo.
StringIO gives you file-like access to strings, so you can use an existing module that deals with a file and change almost nothing and make it work with strings.
For example, say you have a logger that writes things to a file and you want to instead send the log output over the network. You can read the file and write its contents to the network, or you can write the log to a StringIO object and ship it off to its network destination without touching the filesystem. StringIO makes it easy to do it the first way then switch to the second way.
It’s used when you have some API that only takes files, but you need to use a string. For example, to compress a string using the gzip module in Python 2:
import gzip
import StringIO
stringio = StringIO.StringIO()
gzip_file = gzip.GzipFile(fileobj=stringio, mode='w')
gzip_file.write('Hello World')
gzip_file.close()
stringio.getvalue()
In cases where you want a file-like object that ACTS like a file, but is writing to an in-memory string buffer: StringIO is the tool. If you’re building large strings, such as plain-text documents, and doing a lot of string concatenation, you might find it easier to just use StringIO instead of a bunch of mystr += 'more stuffn' type of operations.
Couple of things I personally have used it for:
-
Whole-file caching. I have a script that reads PDFs and does validation of various things about them. The PDF library I’m using takes an open file in its document constructor. I originally just opened the PDF I was interested in reading, however when I changed it to read the entire file at once into memory then pass a StringIO object to the PDF library, the running time of my script was cut in half.
-
Deferred printing. Same script prints a header before every PDF it reads. However, I can specify on the command line whether to ignore certain tests that are in its configuration file, or to only include certain ones. If I ignore all tests for a given PDF I don’t want the header printed, but I won’t know how many tests I ran until I’m done running the tests (the tests can be defined dynamically as well). So I capture the header into a StringIO object by changing
sys.stdoutto point to it, and each time I run a test I check to see whether that object has anything in it. If so, I print it then and reset it to empty. Voila, only PDFs that have tests have headers printed.
I’ve just used StringIO in practice for two things:
- To unit-test a script that does a lot of
printing, by redirectingsys.stdoutto aStringIOinstance for easy analysis; - To create a guaranteed well-formed XML document (a custom API request) using
ElementTreeand thenwriteit for sending via a HTTP connection.
Not that you need StringIO often, but sometimes it’s pretty useful.
Django has a function call_command which is used to call management commands. This function prints output to stdout and doesn’t return any value. If you want to know whether the command ran successfully or not, you have to look into output and decide.
Using StringIO, you can capture output and check if it is desired output or not.
with io.StringIO() as output:
call_command('custom_command', stdout=output)
if 'Success' not in output.getvalue():
print('Custom command failed...')
I’ve used it in place of text files for unit-testing.
For example, to make a csv ‘file’ for testing with pandas (Python 3):
import io
import pandas as pd
f = io.StringIO("id,namen1,briann2,amandan3,zoeyn")
df = pd.read_csv(f) # pandas takes a file path or a file-like object
From the documentation here:
An in-memory stream for text I/O. The text buffer is discarded when the close() method is called.
The initial value of the buffer can be set by providing initial_value.
method getvalue(): Return a str containing the entire contents of the buffer.
Here’s a concrete example of a use case for StringIO: writing some data directly to aws s3, without needing to create a file on local disk:
import csv
import io
import boto3
data = [
["test", "data", "headers etc", "123","",],
["blah", "123", "35", "blah","",],
["abc", "def", "blah", "yep", "blah"]
]
bucket_name = 'bucket_name_here'
session = boto3.Session(
aws_access_key_id = "fake Access ID"),
aws_secret_access_key = "fake Secret key"),
region_name = "ap-southeast-2")
)
s3 = session.resource('s3')
with io.StringIO() as f:
writer = csv.writer(f, delimiter=",")
writer.writerows(data)
resp = s3.Object(bucket_name, "test.csv").put(Body=f.getvalue())
Enjoy your new csv on S3, without having written anything to your local disk!