Transfer file from SFTP to S3 using Paramiko

Question

I am using Paramiko to access a remote SFTP folder, and I’m trying to write code that transfers files from a path in SFTP (with a simple logic using the file metadata to check it’s last modified date) to AWS S3 bucket.
I have set the connection to S3 using Boto3, but I still can’t seem to write a working code that transfers the files without downloading them to a local directory first. Here is some code I tried using Paramiko’s getfo() method. But it doesn’t work.

for f in files:
    # get last modified from file metadata
    last_modified = sftp.stat(remote_path + f).st_mtime
    last_modified_date = datetime.fromtimestamp(last_modified).date()
    if last_modified_date > date_limit:  # check limit
       print('getting ' + f)
       full_path = f"{folder_path}{f}"
       fo = sftp.getfo(remote_path + f,f)
       s3_conn.put_object(Body=fo,Bucket=s3_bucket, Key=full_path)

Thank you!

Asked By: Tomer Shalhon

||

Source

Answer 1

Use Paramiko SFTPClient.open to get a file-like object that you can pass to Boto3 Client.put_object:

with sftp.open(remote_path + f, "r") as f:
    f.prefetch()
    s3_conn.put_object(Body=f)

For the purpose of the f.prefetch(), see Reading file opened with Python Paramiko SFTPClient.open method is slow.

For the opposite direction, see:
Transfer file from AWS S3 to SFTP using Boto 3

Answered By: Martin Prikryl

Transfer file from SFTP to S3 using Paramiko

Question:

Answers: