I want it to skip the missing files and continue executing, while downloading a file via get request in python

Question:

I am not familiar with error handling in python/get.req,
I am trying to write a function which downloads files from a dynamic URL which has dynamic {date}.
When I run it in loop i.e date range, It stops where the files are missing.
I want it to skip the missing files and continue executing.

Code:

def download(url: str, dest_folder: str):
    if not os.path.exists(dest_folder):
        os.makedirs(dest_folder) 

    filename = url.split('/')[-1].replace(" ", "_")  
    file_path = os.path.join(dest_folder, filename)

    r = requests.get(url, stream=True)
    if r.ok:
        print("saving to", os.path.abspath(file_path))
        with open(file_path, 'wb') as f:
            for chunk in r.iter_content(chunk_size=1024 * 8):
                if chunk:
                    f.write(chunk)
                    f.flush()
                    os.fsync(f.fileno())
    else:  # HTTP status code 4XX/5XX
        print("Download failed: status code {}n{}".format(r.status_code, r.text))
Asked By: Apurva

||
Source

Answers:

Welcome to StackOverflow! Maybe you can use try/except method. Can you try your codes like this:

try:
  if not os.path.exists(dest_folder):
    os.makedirs(dest_folder) 

    filename = url.split('/')[-1].replace(" ", "_")  
    file_path = os.path.join(dest_folder, filename)

    r = requests.get(url, stream=True)
    if r.ok:
      print("saving to", os.path.abspath(file_path))
      with open(file_path, 'wb') as f:
        for chunk in r.iter_content(chunk_size=1024 * 8):
          if chunk:
            f.write(chunk)
            f.flush()
            os.fsync(f.fileno())
except:
    print("Download failed: status code {}n{}".format(r.status_code, r.text))
Answered By: uozcan12

To handle an error, you need try and except.

try is like an if: you put a try: and then after it you put the whole code (with the correct indentation).

Then, you put an except: (like an else), and the code after the except is going to be executed when an error occurs.

With this being said, it’s time for the alternative code (in case of error).

To terminate the program when the error occurs, you can use, for example, os._exit(0), which exits with no error.

If you want to ignore the error (in case you just want to test or see what happens next), you must change except: for except Exception as e: and then add print(e). You can also add text to the print(), i.e. print("An error has occurred: " + e)/print(f"An error has occurred: {e}").

Now I read your code and I see I was wrong. If you want to use an if check, um… I don’t understand it very well. What exactly do you want to do?

Code with continue (which might work, but I didn’t test it):

def download(url: str, dest_folder: str):
    try:
        if not os.path.exists(dest_folder):
            os.makedirs(dest_folder) 

        filename = url.split('/')[-1].replace(" ", "_")  
        file_path = os.path.join(dest_folder, filename)

        r = requests.get(url, stream=True)
        if r.ok:
            print("saving to", os.path.abspath(file_path))
            with open(file_path, 'wb') as f:
                for chunk in r.iter_content(chunk_size=1024 * 8):
                    if chunk:
                        f.write(chunk)
                        f.flush()
                        os.fsync(f.fileno())
        else: # HTTP status code 4XX/5XX
            print("Download failed: status code {}n{}".format(r.status_code, r.text))
            continue
    except Exception as e:
        print("Failed: error {}".format(e))
        continue
Answered By: aritz331_