Python pysftp.put raises "No such file" exception although file is uploaded

Question

I am using pysftp to connect to a server and upload a file to it.

cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
self.sftp = pysftp.Connection(host=self.serverConnectionAuth['host'], port=self.serverConnectionAuth['port'],
                              username=self.serverConnectionAuth['username'], password=self.serverConnectionAuth['password'], 
                              cnopts=cnopts)
self.sftp.put(localpath=self.filepath+filename, remotepath=filename)

Sometimes it does okay with no error, but sometime it puts the file correctly, BUT raises the following exception. The file is read and processed by another program running on the server, so I can see that the file is there and it is not corrupted

  File "E:Anacondaenvspy35libsite-packagespysftp__init__.py", line 364, in put
    confirm=confirm)
  File "E:Anacondaenvspy35libsite-packagesparamikosftp_client.py", line 727, in put
    return self.putfo(fl, remotepath, file_size, callback, confirm)
  File "E:Anacondaenvspy35libsite-packagesparamikosftp_client.py", line 689, in putfo
    s = self.stat(remotepath)
  File "E:Anacondaenvspy35libsite-packagesparamikosftp_client.py", line 460, in stat
    t, msg = self._request(CMD_STAT, path)
  File "E:Anacondaenvspy35libsite-packagesparamikosftp_client.py", line 780, in _request
    return self._read_response(num)
  File "E:Anacondaenvspy35libsite-packagesparamikosftp_client.py", line 832, in _read_response
    self._convert_status(msg)
  File "E:Anacondaenvspy35libsite-packagesparamikosftp_client.py", line 861, in _convert_status
    raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file

How can I prevent the exception?

Asked By: Amir Zare

||

Source

Answer 1

From the described behaviour, I assume that the file is removed very shortly after it is uploaded by some server-side process.

By default pysftp.Connection.put verifies the upload by checking a size of the target file. If the server-side processes manages to remove the file too fast, reading the file size would fail.

You can disable the post-upload check by setting confirm parameter to False:

self.sftp.put(localpath=self.filepath+filename, remotepath=filename, confirm=False)

I believe the check is redundant anyway, see
How to perform checksums during a SFTP file transfer for data integrity?

For a similar question about Paramiko (which pysftp uses internally), see:
Paramiko put method throws "[Errno 2] File not found" if SFTP server has trigger to automatically move file upon upload

And actually, you should use Paramiko directory, as pysftp is a dead project. See pysftp vs. Paramiko.

Answered By: Martin Prikryl

Answer 2

Also had this issue of the file automatically getting moved before paramiko could do an os.stat on the uploaded file and compare the local and uploaded file sizes.

@Martin_Prikryl solution works works fine for removing the error by passing in confirm=False when using sftp.put or sftp.putfo

If you want this check to still run to verify the file has been uploaded fully you can run something along these lines. For this to work you will need to know the moved file location and have the ability to read the file.

import os

sftp.putfo(source_file_object, destination_file, confirm=False)
upload_size = sftp.stat(moved_path).st_size
local_size = os.stat(source_file_object).st_size
if upload_size != local_size:
    raise IOError(
        "size mismatch in put!  {} != {}".format(upload_size, local_size)
    )

Both checks use os.stat

Answered By: mRyan

Python pysftp.put raises "No such file" exception although file is uploaded

Question:

Answers: