For Python open() function, what's more efficient?

Question

I’m curious if there’s performance or resource difference between calling open() function for file write/append whenever I need AND having a connection open throughout a whole program.

So it basically

with open('random.log', 'w') as f: 
  f.write('new log1')
    .
    .
    .
with open('random.log', 'w') as f: 
  f.write('new log2')
    .
    .
    .
with open('random.log', 'w') as f: 
  f.write('new log3')

vs.

with open('random.log', 'w') as f: 
  f.write('new log1')
    .
    .
    .
  f.write('new log2')
    .
    .
    .
  f.write('new log3')

I would like to know the best practice of using open() method in Python when during a situation that multiple file operations are needed.

Asked By: Jay Kim

||

Source

Answer 1

It is a lot more efficient to hold the file open until you don’t need it anymore, so if you need to keep it open for the entire program, then it shouldn’t be closed until the end of the program.

Each call to open() makes a syscall (short for "system call") to your operating system, asking it to do something special for your program such as open/close a file, get data from the network, or a variety of other things. Syscalls have more overhead because they have to go through the operating system instead of just the Python standard library, so only make them when you have to.

To prove this to you, I timed both of your examples with timeit:

import timeit

print(timeit.timeit("""
with open('random.log', 'w') as f: 
  f.write('new log1')

with open('random.log', 'w') as f: 
  f.write('new log2')

with open('random.log', 'w') as f: 
  f.write('new log3')
                    """, number=1000))

print(timeit.timeit("""
with open('random2.log', 'w') as f: 
  f.write('new log1')
  f.write('new log2')
  f.write('new log3')
                    """, number=1000))

and here was the output:

0.8310379990143701
0.27580801700241864

Calling open() once was nearly 3 times faster!

Answered By: Michael M.

Answer 2

with open(‘random.log’, ‘w’) as f:
f.write(‘new log1’)
.
.
.
with open(‘random.log’, ‘w’) as f:
f.write(‘new log2’)
.
.
.
with open(‘random.log’, ‘w’) as f:
f.write(‘new log3’)
vs.

with open(‘random.log’, ‘w’) as f:
f.write(‘new log1’)
.
.
.
f.write(‘new log2’)
.
.
.
f.write(‘new log3’)

Answered By: Nakul yadav

Answer 3

I ran this code to test :

import time

t0 = time.time()

for i in range(10000):
    with open('random.log', 'w') as f: 
        f.write('new log1')
    with open('random.log', 'w') as f: 
        f.write('new log2')
    with open('random.log', 'w') as f: 
        f.write('new log3')

print(time.time() - t0)

t0 = time.time()

for i in range(10000):
    with open('random.log', 'w') as f: 
        f.write('new log1')
        f.write('new log2')
        f.write('new log3')

print(time.time() - t0)

This gives me this output :

30.80701231956482
11.21492624282837

Im not surprised because "opening" a file requires some time, the less you do it, the faster the code.

Answered By: Sabrebar

Answer 4

The only difference I see is that at first, you are opening the same file MULTIPLE times and writing to it. In the second scenario, you’re opening it once and writing and then performing some operations, and then writing to it again without closing it. In my opinion, it’s better to open the file once and perform the operations and then close it using close(). The first approach might take up some extra time because you are opening the same file multiple times (and assuming that you close them each time). However, neither approach should eat up too many resources or take you too long unless you’re taking the first approach and there is a lot of reading and writing and other operations involved. This was my opinion, but here are some resources for further info:
Quora Q/A

Answered By: Adrino Rosario

For Python open() function, what's more efficient?

Question:

Answers: