Recursive directory download with Paramiko?
Question:
I want to download a directory with unknown contents recursively via SSH and have been trying Paramiko. I have seen several examples how to upload directories but none that covers recursive download.
I can list all items in a directory but haven’t been able to find a way of knowing if the item is a file (to download) or a directory (to call recursively).
transport = paramiko.Transport((MY_IP, 22))
transport.connect(username=MY_NAME, password=MY_PASS)
sftp = paramiko.SFTPClient.from_transport(transport)
file_list = sftp.listdir(path='/home/MY_HOME_DIR')
for item in file_list:
# Here is an item name... but is it a file or directory?
print(item)
sftp.close()
transport.close()
So how do I know if an item is a file or if it is a directory?
Answers:
If u using Linux or Unix-like. U can use ‘file’ utility with popen.
Or simple u can use os.path.isdir() =)
You can use the stat() method of your sftp object:
http://www.lag.net/paramiko/docs/paramiko.SFTPClient-class.html
from stat import S_ISDIR
def isdir(path):
try:
return S_ISDIR(sftp.stat(path).st_mode)
except IOError:
#Path does not exist, so by definition not a directory
return False
…assuming sftp
is an open Paramiko SFTP connection.
stat() method among other attributes returns permissions. d (for example drwxrwxrwx) shows that it is directory.
As example:
dir = oct(sftp.stat(path).st_mode)
print dir[0:2]
output interpritation:
01 fifo
02 character special
04 directory
06 block special
10 regular file
12 symbolic link
14 socket
An old question, but a solution I came up with that works quite well, it’s a little bit sloppy (typecasting and slashes and all) – but it does work.
Note this uses fabric.api.local
to make the directories in the destination.
def sftp_get_recursive(path, dest, sftp=sftp):
item_list = sftp.listdir(path)
dest = str(dest)
if not os.path.isdir(dest):
local("mkdir %s" % dest)
for item in item_list:
item = str(item)
if is_directory(path + "/" + item, sftp):
sftp_get_recursive(path + "/" + item, dest + "/" + item, sftp)
else:
sftp.get(path + "/" + item, dest + "/" + item)
Paramiko does not support recursive operations.
But it’s easy to implement:
import os
from stat import S_ISDIR, S_ISREG
def get_r_portable(sftp, remotedir, localdir):
for entry in sftp.listdir_attr(remotedir):
remotepath = remotedir + "/" + entry.filename
localpath = os.path.join(localdir, entry.filename)
mode = entry.st_mode
if S_ISDIR(mode):
try:
os.mkdir(localpath)
except OSError:
pass
get_r_portable(sftp, remotepath, localpath)
elif S_ISREG(mode):
sftp.get(remotepath, localpath)
You also can use pysftp. It’s a wrapper around Paramiko that has more Python-ish look and feel and supports recursive operations. See
Or see my answer to Python pysftp get_r from Linux works fine on Linux but not on Windows.
But pysftp seems to be an abandoned project, so you better stick with Paramiko.
A small update to Dan LaManna’s answer that works in 2021.
import paramiko
import os
from stat import S_ISDIR, S_ISREG
def sftp_get_recursive(path, dest, sftp):
item_list = sftp.listdir_attr(path)
dest = str(dest)
if not os.path.isdir(dest):
os.makedirs(dest, exist_ok=True)
for item in item_list:
mode = item.st_mode
if S_ISDIR(mode):
sftp_get_recursive(path + "/" + item.filename, dest + "/" + item.filename, sftp)
else:
sftp.get(path + "/" + item.filename, dest + "/" + item.filename)
transport = paramiko.Transport((host, port))
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
sftp_get_recursive(remote_path, local_path, sftp)
sftp.close()
Here is an answer for 2022.
pysftp is abandoned. paramiko does not implement recursivity for SFTP. So I made a library named sftputil (to make a parallel with ftputil) based on Paramiko.
sftputil implements functionalities such as walk
, glob
and sync
. To copy an entire directory, the easiest way is to use the synchronization feature :
from sftputil import SFTP
sftp = SFTP("hostname", "username", password="password")
# To copy recursively
sftp.sync_pull("remote/path", "local/dir")
# To copy only the files
sftp.sync_pull("remote/path", "local/dir", recursive=False)
I want to download a directory with unknown contents recursively via SSH and have been trying Paramiko. I have seen several examples how to upload directories but none that covers recursive download.
I can list all items in a directory but haven’t been able to find a way of knowing if the item is a file (to download) or a directory (to call recursively).
transport = paramiko.Transport((MY_IP, 22))
transport.connect(username=MY_NAME, password=MY_PASS)
sftp = paramiko.SFTPClient.from_transport(transport)
file_list = sftp.listdir(path='/home/MY_HOME_DIR')
for item in file_list:
# Here is an item name... but is it a file or directory?
print(item)
sftp.close()
transport.close()
So how do I know if an item is a file or if it is a directory?
If u using Linux or Unix-like. U can use ‘file’ utility with popen.
Or simple u can use os.path.isdir() =)
You can use the stat() method of your sftp object:
http://www.lag.net/paramiko/docs/paramiko.SFTPClient-class.html
from stat import S_ISDIR
def isdir(path):
try:
return S_ISDIR(sftp.stat(path).st_mode)
except IOError:
#Path does not exist, so by definition not a directory
return False
…assuming sftp
is an open Paramiko SFTP connection.
stat() method among other attributes returns permissions. d (for example drwxrwxrwx) shows that it is directory.
As example:
dir = oct(sftp.stat(path).st_mode)
print dir[0:2]
output interpritation:
01 fifo
02 character special
04 directory
06 block special
10 regular file
12 symbolic link
14 socket
An old question, but a solution I came up with that works quite well, it’s a little bit sloppy (typecasting and slashes and all) – but it does work.
Note this uses fabric.api.local
to make the directories in the destination.
def sftp_get_recursive(path, dest, sftp=sftp):
item_list = sftp.listdir(path)
dest = str(dest)
if not os.path.isdir(dest):
local("mkdir %s" % dest)
for item in item_list:
item = str(item)
if is_directory(path + "/" + item, sftp):
sftp_get_recursive(path + "/" + item, dest + "/" + item, sftp)
else:
sftp.get(path + "/" + item, dest + "/" + item)
Paramiko does not support recursive operations.
But it’s easy to implement:
import os
from stat import S_ISDIR, S_ISREG
def get_r_portable(sftp, remotedir, localdir):
for entry in sftp.listdir_attr(remotedir):
remotepath = remotedir + "/" + entry.filename
localpath = os.path.join(localdir, entry.filename)
mode = entry.st_mode
if S_ISDIR(mode):
try:
os.mkdir(localpath)
except OSError:
pass
get_r_portable(sftp, remotepath, localpath)
elif S_ISREG(mode):
sftp.get(remotepath, localpath)
You also can use pysftp. It’s a wrapper around Paramiko that has more Python-ish look and feel and supports recursive operations. See
Or see my answer to Python pysftp get_r from Linux works fine on Linux but not on Windows.
But pysftp seems to be an abandoned project, so you better stick with Paramiko.
A small update to Dan LaManna’s answer that works in 2021.
import paramiko
import os
from stat import S_ISDIR, S_ISREG
def sftp_get_recursive(path, dest, sftp):
item_list = sftp.listdir_attr(path)
dest = str(dest)
if not os.path.isdir(dest):
os.makedirs(dest, exist_ok=True)
for item in item_list:
mode = item.st_mode
if S_ISDIR(mode):
sftp_get_recursive(path + "/" + item.filename, dest + "/" + item.filename, sftp)
else:
sftp.get(path + "/" + item.filename, dest + "/" + item.filename)
transport = paramiko.Transport((host, port))
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
sftp_get_recursive(remote_path, local_path, sftp)
sftp.close()
Here is an answer for 2022.
pysftp is abandoned. paramiko does not implement recursivity for SFTP. So I made a library named sftputil (to make a parallel with ftputil) based on Paramiko.
sftputil implements functionalities such as walk
, glob
and sync
. To copy an entire directory, the easiest way is to use the synchronization feature :
from sftputil import SFTP
sftp = SFTP("hostname", "username", password="password")
# To copy recursively
sftp.sync_pull("remote/path", "local/dir")
# To copy only the files
sftp.sync_pull("remote/path", "local/dir", recursive=False)