Pylint R1732 ("Consider using 'with'") for one-liner: is it really good advice?
Question:
On a line such as
r = open(path, encoding="utf-8").read()
Pylint 2.14.5 provides the following advise:
submodules-dedup.py:71:32: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
If I understand correctly, the suggestion is to change it to
with open(path, encoding="utf-8") as f:
r = f.read()
But is this really better in any way?
Personally I don’t find it any more readable, and as for other concerns, wouldn’t the file be closed at the same time thanks to how reference counting works anyhow?
Answers:
This suggestion is to ensure the resource is closed or freed when it exits the context. This is the point of using a context manager.
Of course using a context manager breaks in some extent the one-liner style but it brings better/safer code. No chance to forget the close statement. Indeed it is a trade off between readability and good coding practice.
The question is: Is the second line with close statement more readable?
Python documentation states it explicitly:
If you’re not using the with keyword, then you should call f.close()
to close the file and immediately free up any system resources used by
it.
Warning: Calling f.write() without using the with keyword or calling
f.close() might result in the arguments of f.write() not being
completely written to the disk, even if the program exits
successfully.
Anyway the resource should be released when your program exists but it may not be in the state you think it should be.
If the resource is not critical or you think that explicitly write the close statement afterward does not break the one liner style you may ignore this warning.
The risk of keeping files opened are few but you may consider it:
- Dead lock if the resource is locked when opened, it will prevent other process to access it until the lock is released;
- Corruption and unattended behaviour when writing to the resource;
- Reaching the limit of number of files that can be opened by the OS;
The same will happen with database connection:
- Reaching the connection limit due to unclosed connections leading to a service denial.
So, IMHO using the context manager is the good choice to take as it ensures resource to be released as soon as possible, it keeps the code clean and prevent you to forget the required close statement that anyway will break the one-liner style.
Complementary to the other answer: FWIW pathlib provides utility methods which nicely replace open().read(), without the issues that might have on alternative implementations:
r = open(path, encoding="utf-8").read()
can be written as
r = Path(path).read_text("utf-8")
The end result is the same, but the implementation will ensure the file is properly closed before returning. And the Path() invocation is not necessary if you work with path objects to start with, which is a pretty nice thing to do in modern python as the API is not the worst (though not all of the os and os.path utility functions are available through pathlib, sadly).
wouldn’t the file be closed at the same time thanks to how reference counting works anyhow?
There are more implementations than cpython, and thus other memory reclamation schemes than refcounting (none of pypy, jython, and ironpython use refcounting). pylint should not assume you’re using cpython, and those alternate implementations are a big part of why context managers were introduced to start with.
On a line such as
r = open(path, encoding="utf-8").read()
Pylint 2.14.5 provides the following advise:
submodules-dedup.py:71:32: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
If I understand correctly, the suggestion is to change it to
with open(path, encoding="utf-8") as f:
r = f.read()
But is this really better in any way?
Personally I don’t find it any more readable, and as for other concerns, wouldn’t the file be closed at the same time thanks to how reference counting works anyhow?
This suggestion is to ensure the resource is closed or freed when it exits the context. This is the point of using a context manager.
Of course using a context manager breaks in some extent the one-liner style but it brings better/safer code. No chance to forget the close statement. Indeed it is a trade off between readability and good coding practice.
The question is: Is the second line with close statement more readable?
Python documentation states it explicitly:
If you’re not using the with keyword, then you should call f.close()
to close the file and immediately free up any system resources used by
it.Warning: Calling f.write() without using the with keyword or calling
f.close() might result in the arguments of f.write() not being
completely written to the disk, even if the program exits
successfully.
Anyway the resource should be released when your program exists but it may not be in the state you think it should be.
If the resource is not critical or you think that explicitly write the close statement afterward does not break the one liner style you may ignore this warning.
The risk of keeping files opened are few but you may consider it:
- Dead lock if the resource is locked when opened, it will prevent other process to access it until the lock is released;
- Corruption and unattended behaviour when writing to the resource;
- Reaching the limit of number of files that can be opened by the OS;
The same will happen with database connection:
- Reaching the connection limit due to unclosed connections leading to a service denial.
So, IMHO using the context manager is the good choice to take as it ensures resource to be released as soon as possible, it keeps the code clean and prevent you to forget the required close statement that anyway will break the one-liner style.
Complementary to the other answer: FWIW pathlib provides utility methods which nicely replace open().read(), without the issues that might have on alternative implementations:
r = open(path, encoding="utf-8").read()
can be written as
r = Path(path).read_text("utf-8")
The end result is the same, but the implementation will ensure the file is properly closed before returning. And the Path() invocation is not necessary if you work with path objects to start with, which is a pretty nice thing to do in modern python as the API is not the worst (though not all of the os and os.path utility functions are available through pathlib, sadly).
wouldn’t the file be closed at the same time thanks to how reference counting works anyhow?
There are more implementations than cpython, and thus other memory reclamation schemes than refcounting (none of pypy, jython, and ironpython use refcounting). pylint should not assume you’re using cpython, and those alternate implementations are a big part of why context managers were introduced to start with.