Seeking from end of file throwing unsupported exception
Question:
I have this code snippet and I’m trying to seek backwards from the end of file using python:
f=open('D:SGStat.txt','a');
f.seek(0,2)
f.seek(-3,2)
This throws the following exception while running:
f.seek(-3,2)
io.UnsupportedOperation: can't do nonzero end-relative seeks
Am i missing something here?
Answers:
From the documentation for Python 3.2 and up:
In text files (those opened without a b in the mode string), only seeks relative to the beginning of the file are allowed (the exception being seeking to the very file end with seek(0, 2)).
This is because text files do not have a 1-to-1 correspondence between encoded bytes and the characters they represent, so seek can’t tell where to jump to in the file to move by a certain number of characters.
If your program is okay with working in terms of raw bytes, you can change your program to read:
f = open('D:SGStat.txt', 'ab')
f.seek(-3, 2)
Note the b in the mode string, for a binary file. (Also note the removal of the redundant f.seek(0, 2) call.)
However, you should be aware that adding the b flag when you are reading or writing text can have unintended consequences (with multibyte encoding for example), and in fact changes the type of data read or written.
In order to use seek from current position and end you have to open the text file in binary mode. See this example where I have created a file “nums.txt” and have put “ABCDEFGHIJKLMNOPQRSTUVWXYZ” in the file. I read the letters of the string “PYTHON” from the file and display the same. See the code I’ve run in python 3.6 windows in anaconda 4.2
>>> file=open('nums.txt','rb')
>>> file.seek(15,0)
15
>>> file.read(1).decode('utf-8')
'P'
>>> file.seek(8,1)
24
>>> file.read(1).decode('utf-8')
'Y'
>>> file.seek(-7,2)
19
>>> file.read(1).decode('utf-8')
'T'
>>> file.seek(7,0)
7
>>> file.read(1).decode('utf-8')
'H'
>>> file.seek(6,1)
14
>>> file.read(1).decode('utf-8')
'O'
>>> file.seek(-2,1)
13
>>> file.read(1).decode('utf-8')
'N'
The existing answers do answer the question, but provide no solution.
As pointed out in the comments, this answer is based on undefined behavior and does not handle UnicodeDecodeError, which you may encounter with UTF-8 files. It works fine with ASCII and other fixed-width encodings as long as you seek to the beginning of a character. Please see Philip’s answer which includes a workaround and further comments discussing why seeking backwards in UTF-8 is a problem.
From readthedocs:
If the file is opened in text mode (without b), only offsets returned by tell() are legal. Use of other offsets causes undefined behavior.
This is supported by the documentation, which says that:
In text files (those opened without a b in the mode string), only seeks relative to the beginning of the file [os.SEEK_SET] are allowed…
This means if you have this code from old Python:
f.seek(-1, 1) # seek -1 from current position
it would look like this in Python 3:
f.seek(f.tell() - 1, os.SEEK_SET) # os.SEEK_SET == 0
Solution
Putting this information together we can achieve the goal of the OP:
f.seek(0, os.SEEK_END) # seek to end of file; f.seek(0, 2) is legal
f.seek(f.tell() - 3, os.SEEK_SET) # go backwards 3 bytes
Eric Lindsey’s answer does not work because UTF-8 files can have more than one byte per character. Worse, for those of us who speak English as a first language and work with English only files, it might work just long enough to get out into production code and really break things.
The following answer is based on undefined behavior
… but it does work for now for UTF-8 in Python 3.7.
To seek backwards through a file in text mode, you can do so as long as you correctly handle the UnicodeDecodeError caused by seeking to a byte which is not the start of a UTF-8 Character. Since we are seeking backwards we can simply seek back an extra byte until we find the start of the character.
The result of f.tell() is still the byte position in the file for UTF-8 files, at-least for now. So an f.seek() to an invalid offset will raise a UnicodeDecodeError when you subsequently f.read() and this can be corrected by f.seek() again to a different offset. At least this works for now.
Eg, seeking to the beginning of a line (just after the n):
pos = f.tell() - 1
if pos < 0:
pos = 0
f.seek(pos, os.SEEK_SET)
while pos > 0:
try:
character = f.read(1)
if character == 'n':
break
except UnicodeDecodeError:
pass
pos -= 1
f.seek(pos, os.SEEK_SET)
I have this code snippet and I’m trying to seek backwards from the end of file using python:
f=open('D:SGStat.txt','a');
f.seek(0,2)
f.seek(-3,2)
This throws the following exception while running:
f.seek(-3,2)
io.UnsupportedOperation: can't do nonzero end-relative seeks
Am i missing something here?
From the documentation for Python 3.2 and up:
In text files (those opened without a
bin the mode string), only seeks relative to the beginning of the file are allowed (the exception being seeking to the very file end withseek(0, 2)).
This is because text files do not have a 1-to-1 correspondence between encoded bytes and the characters they represent, so seek can’t tell where to jump to in the file to move by a certain number of characters.
If your program is okay with working in terms of raw bytes, you can change your program to read:
f = open('D:SGStat.txt', 'ab')
f.seek(-3, 2)
Note the b in the mode string, for a binary file. (Also note the removal of the redundant f.seek(0, 2) call.)
However, you should be aware that adding the b flag when you are reading or writing text can have unintended consequences (with multibyte encoding for example), and in fact changes the type of data read or written.
In order to use seek from current position and end you have to open the text file in binary mode. See this example where I have created a file “nums.txt” and have put “ABCDEFGHIJKLMNOPQRSTUVWXYZ” in the file. I read the letters of the string “PYTHON” from the file and display the same. See the code I’ve run in python 3.6 windows in anaconda 4.2
>>> file=open('nums.txt','rb')
>>> file.seek(15,0)
15
>>> file.read(1).decode('utf-8')
'P'
>>> file.seek(8,1)
24
>>> file.read(1).decode('utf-8')
'Y'
>>> file.seek(-7,2)
19
>>> file.read(1).decode('utf-8')
'T'
>>> file.seek(7,0)
7
>>> file.read(1).decode('utf-8')
'H'
>>> file.seek(6,1)
14
>>> file.read(1).decode('utf-8')
'O'
>>> file.seek(-2,1)
13
>>> file.read(1).decode('utf-8')
'N'
The existing answers do answer the question, but provide no solution.
As pointed out in the comments, this answer is based on undefined behavior and does not handle UnicodeDecodeError, which you may encounter with UTF-8 files. It works fine with ASCII and other fixed-width encodings as long as you seek to the beginning of a character. Please see Philip’s answer which includes a workaround and further comments discussing why seeking backwards in UTF-8 is a problem.
From readthedocs:
If the file is opened in text mode (without
b), only offsets returned bytell()are legal. Use of other offsets causes undefined behavior.
This is supported by the documentation, which says that:
In text files (those opened without a
bin the mode string), only seeks relative to the beginning of the file [os.SEEK_SET] are allowed…
This means if you have this code from old Python:
f.seek(-1, 1) # seek -1 from current position
it would look like this in Python 3:
f.seek(f.tell() - 1, os.SEEK_SET) # os.SEEK_SET == 0
Solution
Putting this information together we can achieve the goal of the OP:
f.seek(0, os.SEEK_END) # seek to end of file; f.seek(0, 2) is legal
f.seek(f.tell() - 3, os.SEEK_SET) # go backwards 3 bytes
Eric Lindsey’s answer does not work because UTF-8 files can have more than one byte per character. Worse, for those of us who speak English as a first language and work with English only files, it might work just long enough to get out into production code and really break things.
The following answer is based on undefined behavior
… but it does work for now for UTF-8 in Python 3.7.
To seek backwards through a file in text mode, you can do so as long as you correctly handle the UnicodeDecodeError caused by seeking to a byte which is not the start of a UTF-8 Character. Since we are seeking backwards we can simply seek back an extra byte until we find the start of the character.
The result of f.tell() is still the byte position in the file for UTF-8 files, at-least for now. So an f.seek() to an invalid offset will raise a UnicodeDecodeError when you subsequently f.read() and this can be corrected by f.seek() again to a different offset. At least this works for now.
Eg, seeking to the beginning of a line (just after the n):
pos = f.tell() - 1
if pos < 0:
pos = 0
f.seek(pos, os.SEEK_SET)
while pos > 0:
try:
character = f.read(1)
if character == 'n':
break
except UnicodeDecodeError:
pass
pos -= 1
f.seek(pos, os.SEEK_SET)