Saving images while crawling website in Selenium

Question

I would like to download images like those that can be found on this page.

I need to download all of the images, each one once.

Here’s the code I’m using:

links = []
wait = WebDriverWait(driver, 5)
all_images = wait.until(
    EC.presence_of_all_elements_located((By.XPATH, "//div[contains(@class,'swiper-button-next swiper-button-white')]")))

for image in all_images:
    a = image.get_attribute('style')
    b = a.split("(")[1].split(")")[0].replace('"', '')
    links.append(b)

all_images = wait.until(
    EC.presence_of_all_elements_located((By.XPATH, "//div[contains(@class,'swiper-slide swiper-slide-visible swiper-slide-active swiper-slide-thumb-active')]")))

for image in all_images:
    a = image.get_attribute('style')
    b = a.split("(")[1].split(")")[0].replace('"', '')
    links.append(b)

all_images = wait.until(
    EC.presence_of_all_elements_located((By.XPATH, "//div[contains(@class,'swiper-slide swiper-slide-visible')]")))

for image in all_images:
    a = image.get_attribute('style')
    b = a.split("(")[1].split(")")[0].replace('"', '')
    links.append(b)

index = 1
for i in range(len(links)//2 + 1):
    with open(title.replace(' ', '-') + str(index) + '.jpg', 'wb') as file:
        im = requests.get(links[i])
        file.write(im.content)
        print('Saving image.. ', title + str(index))
    index += 1

The problem is that this saves images repeatedly, and doesn’t save some others, and I don’t know how to fix it.

Asked By: pickle rick

||

Source

Answer 1

You are using a wrong locator.
Additionally, presence_of_all_elements_located doesn’t wait for ALL the elements, it waits for presence of at least 1 element.
Also, presence of element waits for element presence, while this may be not enought. It is recommended to use visibility_of_element_located instead.
I think the following code will work better:

links = []
wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.XPATH, "//div[contains(@class,'swiper-slide')]")))
time.sleep(0.5)
all_images = driver.find_elements_by_xpath("//div[contains(@class,'swiper-slide')]")

Answered By: Prophet

Saving images while crawling website in Selenium

Question:

Answers: