Python can't open symlinked file
Question:
Python is unable to open my simlinked file. I made sure the file exists and I can access it. I was under the impression that symlinks are resolved on the OS level so Python would never know about it.
therold@therold:~/Programming/ML/deeptank/dataset/inc (master)$ ls -lisa /Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png
7870541 8 lrwxr-xr-x 1 therold staff 46 13 Mai 16:44 /Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png -> ../training_data/matilda-iv/matilda-iv_689.png
OSX is clearly able resolve the symlinked image file. However the Python open() method fails me:
therold@therold:~/Programming/ML/deeptank/dataset/inc (master)$ python
Python 2.7.10 (default, Sep 23 2015, 04:34:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.72)] on darwin
>>> open("/Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory: '/Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png'
>>>
Any ideas what I am doing wrong? Help is much appreciated.
Answers:
Any ideas what I am doing wrong?
The target of the symbolic link doesn’t exist.
I don’t understand why I was able to resolve it in the ls statement in my question.
You weren’t.
The ls command by default operates on the link itself, not on the target of the link. Absent the -L option, ls never attempts to resolve the symbolic link.
Consider a directory with these two files:
$ ls -l
-rw-rw-r-- 1 me me 6 May 13 11:58 a
lrwxrwxrwx 1 me me 3 May 13 12:00 b -> ./a
a is a text file containing the six bytes 'hellon'. b is a link file containing the three bytes to its target path: './a'. ls is able to describe the properties of the link without dereferencing the link itself.
In contrast, use the -L option:
$ ls -lL
-rw-rw-r-- 1 me me 6 May 13 11:58 a
-rw-rw-r-- 1 me me 6 May 13 11:58 b
Now ls has resolved the link in b, and displays information about the linked-to file. With -L, ls now claims that b is also a six-byte text file.
Finally, consider this:
$ rm a
$ ls -l
lrwxrwxrwx 1 me me 3 May 13 12:00 b -> ./a
$ ls -lL
ls: cannot access b: No such file or directory
l????????? ? ? ? ? ? b
Now b is a link that resolves to a file that no longer exists. Since ls -l never attempts the resolve the link, its output is unchanged from the previous test. (b is a link file, 3 bytes long, contents './a'.)
But ls -lL attempts to resolve the link, fails, and displays the failure information.
… don’t forget to use complete paths. On Raspberry Pi with Python 3.7.3 I had to include the whole paths for a working link:
os.symlink("Desktop/a.txt","Desktop/a_link.txt") # WRONG !
python3 did not recognize the file a.txt because the paths of the standard shell are not active.
os.symllink("/home/pi/Desktop/a.txt","/home/pi/Desktop/a_link.txt")
works well.
I’m logging an answer because I came to this question with the same problem, but the target of my symlink did exist.
Trying to open a file across a symlink in pandas gave an unhelpful not-found message:
⇒ ipython.exe # this is a clue to the answer below...
In [1]: import pandas as pd
In [2]: codes = pd.read_excel('codes-no-agg-children.xlsx', ...)
---------------------------------------------------------------------------
<...lots of traceback stuff...>
FileNotFoundError: [Errno 2] No such file or directory: 'codes-no-agg-children.xlsx'
Since the target did exist, the answer above didn’t help me, but the question prompted me to try a regular python open instead of through pandas:
In [3]: import os
In [4]: os.system('cp ./nvivo-win/good-extracts/codes-no-agg-children.xlsx .')
\wsl$Ubuntuhomesigfriedccvsanalysis'
CMD.EXE was started with the above path as the current directory.
UNC paths are not supported. Defaulting to windows directory.
cp: cannot stat './nvivo-win/good-extracts/codes-no-agg-children.xlsx': No such file or directory
I don’t know what all that means, but I assume the cause is:
Python for windows can’t follow Ubuntu (WSL 2) symlinks (or, at least it can’t open Ubuntu symlinks back to a windows file.
Hope this helps someone.
I had the same issue-Fedora 32 system, with Jupyter’s Notebook folder containing my files having symlinks to json files. I beat my head against the proverbial wall for a few days, googling came up short.
In the end, copying the files over fixed the issue (though there are other options like hardlinks).
I hit something similar today: when symlinking to a file, and reading lines into an array, some strings of digits would get broken into separate list elements??
This behaviour disappears when opening the original file directly (instead of via the symlink):
Original file -> Python list
123456 -> '1', '23', '456', ''
123456789 -> '1234567', '89'
Hex editor must be following symlinks properly
$ head rockyou.txt | xxd
00000000: 3132 3334 3536 0a31 3233 3435 0a31 3233 123456.12345.123
00000010: 3435 3637 3839 0a70 6173 7377 6f72 640a 456789.password.
00000020: 696c 6f76 6579 6f75 0a70 7269 6e63 6573 iloveyou.princes
00000030: 730a 3132 3334 3536 370a 726f 636b 796f s.1234567.rockyo
00000040: 750a 3132 3334 3536 3738 0a61 6263 3132 u.12345678.abc12
00000050: 330a 3.
$ head rockyou.txt.short | xxd
00000000: 3132 3334 3536 0a31 3233 3435 0a31 3233 123456.12345.123
00000010: 3435 3637 3839 0a70 6173 7377 6f72 640a 456789.password.
00000020: 696c 6f76 6579 6f75 0a70 7269 6e63 6573 iloveyou.princes
00000030: 730a 3132 3334 3536 370a 726f 636b 796f s.1234567.rockyo
00000040: 750a 3132 3334 3536 3738 0a61 6263 3132 u.12345678.abc12
00000050: 330a 3.
$ ls -lah rock*
lrwxr-xr-x 1 u g 17B Apr 16 10:56 rockyou.txt -> rockyou.txt.short
-rw-r--r-- 1 u g 945B Apr 16 11:39 rockyou.txt.short
MacOS Catalina, Python 3.7, 3.8
Note that given a file_path,
os.path.realpath(file_path)
will interpret symlinks and give you the real path to the actual file. So if you are thinking its a problem of the symlink, using realpath will resolve the issue.
I seemed to have the same problem, but for an "existing" file.
Turns out this is a question that may have multiple answers, as another option is that the filename string is somehow malformed (extra space, wrong file separator etc.)…
In my case e.g. from jupyter:
!ls -alh {file_path}
!cat -alh {file_path}
were finding the file, so I was thinking it was python’s not following the symlink.
What I realized, was that I had an extra space in the filename e.g.:
file_path = '/foo/bar/file.txt '
^notice the extra space here
As a result, ls etc. were working fine, but I could not find the file from python… 😀
Python is unable to open my simlinked file. I made sure the file exists and I can access it. I was under the impression that symlinks are resolved on the OS level so Python would never know about it.
therold@therold:~/Programming/ML/deeptank/dataset/inc (master)$ ls -lisa /Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png
7870541 8 lrwxr-xr-x 1 therold staff 46 13 Mai 16:44 /Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png -> ../training_data/matilda-iv/matilda-iv_689.png
OSX is clearly able resolve the symlinked image file. However the Python open() method fails me:
therold@therold:~/Programming/ML/deeptank/dataset/inc (master)$ python
Python 2.7.10 (default, Sep 23 2015, 04:34:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.72)] on darwin
>>> open("/Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory: '/Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png'
>>>
Any ideas what I am doing wrong? Help is much appreciated.
Any ideas what I am doing wrong?
The target of the symbolic link doesn’t exist.
I don’t understand why I was able to resolve it in the ls statement in my question.
You weren’t.
The ls command by default operates on the link itself, not on the target of the link. Absent the -L option, ls never attempts to resolve the symbolic link.
Consider a directory with these two files:
$ ls -l
-rw-rw-r-- 1 me me 6 May 13 11:58 a
lrwxrwxrwx 1 me me 3 May 13 12:00 b -> ./a
a is a text file containing the six bytes 'hellon'. b is a link file containing the three bytes to its target path: './a'. ls is able to describe the properties of the link without dereferencing the link itself.
In contrast, use the -L option:
$ ls -lL
-rw-rw-r-- 1 me me 6 May 13 11:58 a
-rw-rw-r-- 1 me me 6 May 13 11:58 b
Now ls has resolved the link in b, and displays information about the linked-to file. With -L, ls now claims that b is also a six-byte text file.
Finally, consider this:
$ rm a
$ ls -l
lrwxrwxrwx 1 me me 3 May 13 12:00 b -> ./a
$ ls -lL
ls: cannot access b: No such file or directory
l????????? ? ? ? ? ? b
Now b is a link that resolves to a file that no longer exists. Since ls -l never attempts the resolve the link, its output is unchanged from the previous test. (b is a link file, 3 bytes long, contents './a'.)
But ls -lL attempts to resolve the link, fails, and displays the failure information.
… don’t forget to use complete paths. On Raspberry Pi with Python 3.7.3 I had to include the whole paths for a working link:
os.symlink("Desktop/a.txt","Desktop/a_link.txt") # WRONG !
python3 did not recognize the file a.txt because the paths of the standard shell are not active.
os.symllink("/home/pi/Desktop/a.txt","/home/pi/Desktop/a_link.txt")
works well.
I’m logging an answer because I came to this question with the same problem, but the target of my symlink did exist.
Trying to open a file across a symlink in pandas gave an unhelpful not-found message:
⇒ ipython.exe # this is a clue to the answer below...
In [1]: import pandas as pd
In [2]: codes = pd.read_excel('codes-no-agg-children.xlsx', ...)
---------------------------------------------------------------------------
<...lots of traceback stuff...>
FileNotFoundError: [Errno 2] No such file or directory: 'codes-no-agg-children.xlsx'
Since the target did exist, the answer above didn’t help me, but the question prompted me to try a regular python open instead of through pandas:
In [3]: import os
In [4]: os.system('cp ./nvivo-win/good-extracts/codes-no-agg-children.xlsx .')
\wsl$Ubuntuhomesigfriedccvsanalysis'
CMD.EXE was started with the above path as the current directory.
UNC paths are not supported. Defaulting to windows directory.
cp: cannot stat './nvivo-win/good-extracts/codes-no-agg-children.xlsx': No such file or directory
I don’t know what all that means, but I assume the cause is:
Python for windows can’t follow Ubuntu (WSL 2) symlinks (or, at least it can’t open Ubuntu symlinks back to a windows file.
Hope this helps someone.
I had the same issue-Fedora 32 system, with Jupyter’s Notebook folder containing my files having symlinks to json files. I beat my head against the proverbial wall for a few days, googling came up short.
In the end, copying the files over fixed the issue (though there are other options like hardlinks).
I hit something similar today: when symlinking to a file, and reading lines into an array, some strings of digits would get broken into separate list elements??
This behaviour disappears when opening the original file directly (instead of via the symlink):
Original file -> Python list
123456 -> '1', '23', '456', ''
123456789 -> '1234567', '89'
Hex editor must be following symlinks properly
$ head rockyou.txt | xxd
00000000: 3132 3334 3536 0a31 3233 3435 0a31 3233 123456.12345.123
00000010: 3435 3637 3839 0a70 6173 7377 6f72 640a 456789.password.
00000020: 696c 6f76 6579 6f75 0a70 7269 6e63 6573 iloveyou.princes
00000030: 730a 3132 3334 3536 370a 726f 636b 796f s.1234567.rockyo
00000040: 750a 3132 3334 3536 3738 0a61 6263 3132 u.12345678.abc12
00000050: 330a 3.
$ head rockyou.txt.short | xxd
00000000: 3132 3334 3536 0a31 3233 3435 0a31 3233 123456.12345.123
00000010: 3435 3637 3839 0a70 6173 7377 6f72 640a 456789.password.
00000020: 696c 6f76 6579 6f75 0a70 7269 6e63 6573 iloveyou.princes
00000030: 730a 3132 3334 3536 370a 726f 636b 796f s.1234567.rockyo
00000040: 750a 3132 3334 3536 3738 0a61 6263 3132 u.12345678.abc12
00000050: 330a 3.
$ ls -lah rock*
lrwxr-xr-x 1 u g 17B Apr 16 10:56 rockyou.txt -> rockyou.txt.short
-rw-r--r-- 1 u g 945B Apr 16 11:39 rockyou.txt.short
MacOS Catalina, Python 3.7, 3.8
Note that given a file_path,
os.path.realpath(file_path)
will interpret symlinks and give you the real path to the actual file. So if you are thinking its a problem of the symlink, using realpath will resolve the issue.
I seemed to have the same problem, but for an "existing" file.
Turns out this is a question that may have multiple answers, as another option is that the filename string is somehow malformed (extra space, wrong file separator etc.)…
In my case e.g. from jupyter:
!ls -alh {file_path}
!cat -alh {file_path}
were finding the file, so I was thinking it was python’s not following the symlink.
What I realized, was that I had an extra space in the filename e.g.:
file_path = '/foo/bar/file.txt '
^notice the extra space here
As a result, ls etc. were working fine, but I could not find the file from python… 😀