Unable to locate files with long names on Windows with Python
Question:
I need to walk through folders with long file names in Windows.
I tried using os.listdir(), but it crashes with long pathnames, which is bad.
I tried using os.walk(), but it ignores the pathnames longer than ~256, which is worse.
I tried the magic word workaround described here, but it only works with mapped drives, not with UNC pathnames.
Here is an example with short pathnames, that shows that UNC pathnames don’t work with the magic word trick.
>>> os.listdir('c:\drivers')
['nusb3hub.cat', 'nusb3hub.inf', 'nusb3hub.sys', 'nusb3xhc.cat', 'nusb3xhc.inf', 'nusb3xhc.sys']
>>> os.listdir('\\Uni-hq-srv6\router')
['2009-04-0210', '2010-11-0909', ... ]
>>> mw=u'\\?\'
>>> os.listdir(mw+'c:\drivers')
[u'nusb3hub.cat', u'nusb3hub.inf', u'nusb3hub.sys', u'nusb3xhc.cat', u'nusb3xhc.inf', u'nusb3xhc.sys']
>>> os.listdir(mw+'\\Uni-hq-srv6\router')
Traceback (most recent call last):
File "<pyshell#160>", line 1, in <module>
os.listdir(mw+'\\Uni-hq-srv6\router')
WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: u'\\?\\\Uni-hq-srv6\router\*.*'
Any idea on how to deal with long pathnames or with unicode UNC pathnames?
Edit:
Following the suggestion of the comments below, I created some test functions to compare Python 2.7 and 3.3, and I added the test of glob.glob and os.listdir after os.chdir.
The os.chdir didn’t help as expected (see this comment).
The glob.glob is the only one that in Python 3.3 works better, but only in one condition: using the magic word and with the drive name.
Here is the code I used (it works on both 2.7 and 3.3). I am learning Python now, and I hope these tests make sense:
from __future__ import print_function
import os, glob
mw = u'\\?\'
def walk(root):
n = 0
for root, dirs, files in os.walk(root):
n += len(files)
return n
def walk_mw(root):
n = 0
for root, dirs, files in os.walk(mw + root):
n += len(files)
return n
def listdir(root):
try:
folders = [f for f in os.listdir(root) if os.path.isdir(os.path.join(root, f))]
files = [f for f in os.listdir(root) if os.path.isfile(os.path.join(root, f))]
n = len(files)
for f in folders:
n += listdir(os.path.join(root, f))
return n
except:
return 'Crash'
def listdir_mw(root):
if not root.startswith(mw):
root = mw + root
try:
folders = [f for f in os.listdir(root) if os.path.isdir(os.path.join(root, f))]
files = [f for f in os.listdir(root) if os.path.isfile(os.path.join(root, f))]
n = len(files)
for f in folders:
n += listdir_mw(os.path.join(root, f))
return n
except:
return 'Crash'
def listdir_cd(root):
try:
os.chdir(root)
folders = [f for f in os.listdir('.') if os.path.isdir(os.path.join(f))]
files = [f for f in os.listdir('.') if os.path.isfile(os.path.join(f))]
n = len(files)
for f in folders:
n += listdir_cd(f)
return n
except:
return 'Crash'
def listdir_mw_cd(root):
if not root.startswith(mw):
root = mw + root
try:
os.chdir(root)
folders = [f for f in os.listdir('.') if os.path.isdir(os.path.join(f))]
files = [f for f in os.listdir('.') if os.path.isfile(os.path.join(f))]
n = len(files)
for f in folders:
n += listdir_cd(f) # the magic word can only be added the first time
return n
except:
return 'Crash'
def glb(root):
folders = [f for f in glob.glob(root + '\*') if os.path.isdir(os.path.join(root, f))]
files = [f for f in glob.glob(root + '\*') if os.path.isfile(os.path.join(root, f))]
n = len(files)
for f in folders:
n += glb(os.path.join(root, f))
return n
def glb_mw(root):
if not root.startswith(mw):
root = mw + root
folders = [f for f in glob.glob(root + '\*') if os.path.isdir(os.path.join(root, f))]
files = [f for f in glob.glob(root + '\*') if os.path.isfile(os.path.join(root, f))]
n = len(files)
for f in folders:
n += glb_mw(os.path.join(root, f))
return n
def test():
for txt1, root in [('drive ', r'C:test'),
('UNC ', r'\Uni-hq-srv6routertest')]:
for txt2, func in [('walk ', walk),
('walk magic word ', walk_mw),
('listdir ', listdir),
('listdir magic word ', listdir_mw),
('listdir cd ', listdir_cd),
('listdir magic word cd ', listdir_mw_cd),
('glob ', glb),
('glob magic word ', glb_mw)]:
print(txt1, txt2, func(root))
test()
And here is the result:
- The number 8 means all the files were found
- The number 0 means it didn’t even try without crashing
- Any number between 1 and 7 means it failed half way without crashing
- The word
Crash means it crashed
–
Python 2.7
drive walk 5
drive walk magic word 8 * GOOD *
drive listdir Crash
drive listdir magic word 8 * GOOD *
drive listdir cd Crash
drive listdir magic word cd 5
drive glob 5
drive glob magic word 0
UNC walk 6
UNC walk magic word 0
UNC listdir 5
UNC listdir magic word Crash
UNC listdir cd 5
UNC listdir magic word cd Crash
UNC glob 5
UNC glob magic word 0
Python 3.3
drive walk 5
drive walk magic word 8 * GOOD *
drive listdir Crash
drive listdir magic word 8 * GOOD *
drive listdir cd Crash
drive listdir magic word cd 5
drive glob 5
drive glob magic word 8 * GOOD *
UNC walk 6
UNC walk magic word 0
UNC listdir 5
UNC listdir magic word Crash
UNC listdir cd 5
UNC listdir magic word cd Crash
UNC glob 5
UNC glob magic word 0
Answers:
Use the 8.3 fallback to avoid the long pathname, browsing in Win7 explorer this seems to be what windows itself does, ie every long paths has a shorter ‘true name’:
>>> long_unc="\\K53\Users\Tolan\testing\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\xxxxxxxxxxxxxxxxxxxxxxxxdddddddddddddddddddddwgggggggggggggggggggggggggggggggggggxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\esssssssssssssssssssssggggggggggggggggggggggggggggggggggggggggggggggeee"
>>> os.listdir(long_unc)
FileNotFoundError: [WinError 3]
but you can use win32api (pywin32) to ‘build’ up a shorter version, ie
short_unc=win32api.GetShortPathName(win32api.GetShortPathName(win32api.GetShortPathName("\\K53\Users\Tolan\testing\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")+"\xxxxxxxxxxxxxxxxxxxxxxxxdddddddddddddddddddddwgggggggggggggggggggggggggggggggggggxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx") + "\esssssssssssssssssssssggggggggggggggggggggggggggggggggggggggggggggggeee")
>>> print(short_unc)
\K53UsersTolantestingXXXXXX~1XXXXXX~1ESSSSS~1
>>> import os
>>> os.listdir(short_unc)
['test.txt']
clearly you can just fold the win32api.GetShortPathName call into you dir exploration rather than nesting as in my example.
I’ve done it like this with 3 calls because if you’ve already got a ‘too long’ path then win32api.GetShortPathName wont cope with it either, but you can do it per dir and stay below the limit.
To locate files on UNC paths, the the magic prefix is \?UNC rather than just \?.
Reference: https://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx#maxpath
So to access //server/share/really/deep/path/etc/etc, you’d need to
- Convert it to unicode (use the
unicode() constructor)
- Add the magic prefix (
"\?\UNC"), and
- Ensure all directory separators are
"" (see os.path.normpath())
Resulting unicode string: \?UNCserversharereallydeeppathetcetc
I’ve only experimented a little (much less than @stenci did) but with Python 2.7 it seems to work OK with os.walk(), and to fail with os.listdir().
Caveat: It only works with os.walk() if the starting path for the traversal is within the MAX_PATH limit, and none of the sub directories in the starting path would push it over the limit either. This is because as os.walk() uses os.listdir() on the top directory.
In my previous comment I said that the nested recursive call of GetShortPathName is not required. I found it is not required most of the times, but once in a while it crashes. I wasn’t able to figure out when, so I made this little function that has been working smoothly for some time:
This is the function that I use now:
def short_name(name):
try:
return win32api.GetShortPathName(name)
except win32api.error:
dirname = os.path.dirname(name)
basename = os.path.basename(name)
short_dirname = win32api.GetShortPathName(dirname)
return win32api.GetShortPathName(os.path.join(short_dirname, basename))
try:
mtime = os.path.getmtime(name)
except FileNotFoundError:
name = short_name(name)
mtime = os.path.getmtime(name)
I need to walk through folders with long file names in Windows.
I tried using os.listdir(), but it crashes with long pathnames, which is bad.
I tried using os.walk(), but it ignores the pathnames longer than ~256, which is worse.
I tried the magic word workaround described here, but it only works with mapped drives, not with UNC pathnames.
Here is an example with short pathnames, that shows that UNC pathnames don’t work with the magic word trick.
>>> os.listdir('c:\drivers')
['nusb3hub.cat', 'nusb3hub.inf', 'nusb3hub.sys', 'nusb3xhc.cat', 'nusb3xhc.inf', 'nusb3xhc.sys']
>>> os.listdir('\\Uni-hq-srv6\router')
['2009-04-0210', '2010-11-0909', ... ]
>>> mw=u'\\?\'
>>> os.listdir(mw+'c:\drivers')
[u'nusb3hub.cat', u'nusb3hub.inf', u'nusb3hub.sys', u'nusb3xhc.cat', u'nusb3xhc.inf', u'nusb3xhc.sys']
>>> os.listdir(mw+'\\Uni-hq-srv6\router')
Traceback (most recent call last):
File "<pyshell#160>", line 1, in <module>
os.listdir(mw+'\\Uni-hq-srv6\router')
WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: u'\\?\\\Uni-hq-srv6\router\*.*'
Any idea on how to deal with long pathnames or with unicode UNC pathnames?
Edit:
Following the suggestion of the comments below, I created some test functions to compare Python 2.7 and 3.3, and I added the test of glob.glob and os.listdir after os.chdir.
The os.chdir didn’t help as expected (see this comment).
The glob.glob is the only one that in Python 3.3 works better, but only in one condition: using the magic word and with the drive name.
Here is the code I used (it works on both 2.7 and 3.3). I am learning Python now, and I hope these tests make sense:
from __future__ import print_function
import os, glob
mw = u'\\?\'
def walk(root):
n = 0
for root, dirs, files in os.walk(root):
n += len(files)
return n
def walk_mw(root):
n = 0
for root, dirs, files in os.walk(mw + root):
n += len(files)
return n
def listdir(root):
try:
folders = [f for f in os.listdir(root) if os.path.isdir(os.path.join(root, f))]
files = [f for f in os.listdir(root) if os.path.isfile(os.path.join(root, f))]
n = len(files)
for f in folders:
n += listdir(os.path.join(root, f))
return n
except:
return 'Crash'
def listdir_mw(root):
if not root.startswith(mw):
root = mw + root
try:
folders = [f for f in os.listdir(root) if os.path.isdir(os.path.join(root, f))]
files = [f for f in os.listdir(root) if os.path.isfile(os.path.join(root, f))]
n = len(files)
for f in folders:
n += listdir_mw(os.path.join(root, f))
return n
except:
return 'Crash'
def listdir_cd(root):
try:
os.chdir(root)
folders = [f for f in os.listdir('.') if os.path.isdir(os.path.join(f))]
files = [f for f in os.listdir('.') if os.path.isfile(os.path.join(f))]
n = len(files)
for f in folders:
n += listdir_cd(f)
return n
except:
return 'Crash'
def listdir_mw_cd(root):
if not root.startswith(mw):
root = mw + root
try:
os.chdir(root)
folders = [f for f in os.listdir('.') if os.path.isdir(os.path.join(f))]
files = [f for f in os.listdir('.') if os.path.isfile(os.path.join(f))]
n = len(files)
for f in folders:
n += listdir_cd(f) # the magic word can only be added the first time
return n
except:
return 'Crash'
def glb(root):
folders = [f for f in glob.glob(root + '\*') if os.path.isdir(os.path.join(root, f))]
files = [f for f in glob.glob(root + '\*') if os.path.isfile(os.path.join(root, f))]
n = len(files)
for f in folders:
n += glb(os.path.join(root, f))
return n
def glb_mw(root):
if not root.startswith(mw):
root = mw + root
folders = [f for f in glob.glob(root + '\*') if os.path.isdir(os.path.join(root, f))]
files = [f for f in glob.glob(root + '\*') if os.path.isfile(os.path.join(root, f))]
n = len(files)
for f in folders:
n += glb_mw(os.path.join(root, f))
return n
def test():
for txt1, root in [('drive ', r'C:test'),
('UNC ', r'\Uni-hq-srv6routertest')]:
for txt2, func in [('walk ', walk),
('walk magic word ', walk_mw),
('listdir ', listdir),
('listdir magic word ', listdir_mw),
('listdir cd ', listdir_cd),
('listdir magic word cd ', listdir_mw_cd),
('glob ', glb),
('glob magic word ', glb_mw)]:
print(txt1, txt2, func(root))
test()
And here is the result:
- The number 8 means all the files were found
- The number 0 means it didn’t even try without crashing
- Any number between 1 and 7 means it failed half way without crashing
- The word
Crashmeans it crashed
–
Python 2.7
drive walk 5
drive walk magic word 8 * GOOD *
drive listdir Crash
drive listdir magic word 8 * GOOD *
drive listdir cd Crash
drive listdir magic word cd 5
drive glob 5
drive glob magic word 0
UNC walk 6
UNC walk magic word 0
UNC listdir 5
UNC listdir magic word Crash
UNC listdir cd 5
UNC listdir magic word cd Crash
UNC glob 5
UNC glob magic word 0
Python 3.3
drive walk 5
drive walk magic word 8 * GOOD *
drive listdir Crash
drive listdir magic word 8 * GOOD *
drive listdir cd Crash
drive listdir magic word cd 5
drive glob 5
drive glob magic word 8 * GOOD *
UNC walk 6
UNC walk magic word 0
UNC listdir 5
UNC listdir magic word Crash
UNC listdir cd 5
UNC listdir magic word cd Crash
UNC glob 5
UNC glob magic word 0
Use the 8.3 fallback to avoid the long pathname, browsing in Win7 explorer this seems to be what windows itself does, ie every long paths has a shorter ‘true name’:
>>> long_unc="\\K53\Users\Tolan\testing\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\xxxxxxxxxxxxxxxxxxxxxxxxdddddddddddddddddddddwgggggggggggggggggggggggggggggggggggxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\esssssssssssssssssssssggggggggggggggggggggggggggggggggggggggggggggggeee"
>>> os.listdir(long_unc)
FileNotFoundError: [WinError 3]
but you can use win32api (pywin32) to ‘build’ up a shorter version, ie
short_unc=win32api.GetShortPathName(win32api.GetShortPathName(win32api.GetShortPathName("\\K53\Users\Tolan\testing\xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")+"\xxxxxxxxxxxxxxxxxxxxxxxxdddddddddddddddddddddwgggggggggggggggggggggggggggggggggggxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx") + "\esssssssssssssssssssssggggggggggggggggggggggggggggggggggggggggggggggeee")
>>> print(short_unc)
\K53UsersTolantestingXXXXXX~1XXXXXX~1ESSSSS~1
>>> import os
>>> os.listdir(short_unc)
['test.txt']
clearly you can just fold the win32api.GetShortPathName call into you dir exploration rather than nesting as in my example.
I’ve done it like this with 3 calls because if you’ve already got a ‘too long’ path then win32api.GetShortPathName wont cope with it either, but you can do it per dir and stay below the limit.
To locate files on UNC paths, the the magic prefix is \?UNC rather than just \?.
Reference: https://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx#maxpath
So to access //server/share/really/deep/path/etc/etc, you’d need to
- Convert it to unicode (use the
unicode()constructor) - Add the magic prefix (
"\?\UNC"), and - Ensure all directory separators are
""(seeos.path.normpath())
Resulting unicode string: \?UNCserversharereallydeeppathetcetc
I’ve only experimented a little (much less than @stenci did) but with Python 2.7 it seems to work OK with os.walk(), and to fail with os.listdir().
Caveat: It only works with os.walk() if the starting path for the traversal is within the MAX_PATH limit, and none of the sub directories in the starting path would push it over the limit either. This is because as os.walk() uses os.listdir() on the top directory.
In my previous comment I said that the nested recursive call of GetShortPathName is not required. I found it is not required most of the times, but once in a while it crashes. I wasn’t able to figure out when, so I made this little function that has been working smoothly for some time:
This is the function that I use now:
def short_name(name):
try:
return win32api.GetShortPathName(name)
except win32api.error:
dirname = os.path.dirname(name)
basename = os.path.basename(name)
short_dirname = win32api.GetShortPathName(dirname)
return win32api.GetShortPathName(os.path.join(short_dirname, basename))
try:
mtime = os.path.getmtime(name)
except FileNotFoundError:
name = short_name(name)
mtime = os.path.getmtime(name)