Python selenium crashed

Question:

I use "selenium" to locate info with Python3.9.6 64-bit.Sometimes it does output some correct results, but then suddenly crashes after finishing a few of the 10 tasks. But sometimes it doesn’t work at all.Codes are like this:

from selenium.webdriver import Chrome
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import time

web = Chrome()
web.get("http://lagou.com")

web.find_element(By.XPATH, '//*[@id="cboxClose"]').click()

time.sleep(1)

web.find_element(By.XPATH, '//*[@id="search_input"]').send_keys("python", Keys.ENTER)

for i in range(1, 11):
    web.find_element(By.XPATH, f'//*[@id="jobList"]/div[1]/div[{i}]/div[1]/div[1]/div[1]/a').click()

    web.switch_to.window(web.window_handles[-1])

    details = web.find_element(By.XPATH, '//*[@id="job_detail"]/dd[2]/div').text
    print(details)

    web.close()
    web.switch_to.window(web.window_handles[0])

And the crach info are like this

File "/Users/xxxx/Library/Mobile Documents/com~apple~CloudDocs/xxxx/Code/Python/Project/selenium(2).py", line 20, in <module>
    details = web.find_element(By.XPATH, '//*[@id="job_detail"]/dd[2]/div').text
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 1244, in find_element
    return self.execute(Command.FIND_ELEMENT, {
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 424, in execute
    self.error_handler.check_response(response)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="job_detail"]/dd[2]/div"}
  (Session info: chrome=97.0.4692.71)
Stacktrace:
0   chromedriver                        0x000000010fa0fe69 chromedriver + 5160553
1   chromedriver                        0x000000010f99a593 chromedriver + 4679059
2   chromedriver                        0x000000010f54e2c8 chromedriver + 172744
3   chromedriver                        0x000000010f583b62 chromedriver + 392034
4   chromedriver                        0x000000010f583d21 chromedriver + 392481
5   chromedriver                        0x000000010f5b6304 chromedriver + 598788
6   chromedriver                        0x000000010f5a13fd chromedriver + 513021
7   chromedriver                        0x000000010f5b40ab chromedriver + 589995
8   chromedriver                        0x000000010f5a1623 chromedriver + 513571
9   chromedriver                        0x000000010f576dce chromedriver + 339406
10  chromedriver                        0x000000010f578105 chromedriver + 344325
11  chromedriver                        0x000000010f9cb23e chromedriver + 4878910
12  chromedriver                        0x000000010f9e2d17 chromedriver + 4975895
13  chromedriver                        0x000000010f9e8a3f chromedriver + 4999743
14  chromedriver                        0x000000010f9e361a chromedriver + 4978202
15  chromedriver                        0x000000010f9bfbb1 chromedriver + 4832177
16  chromedriver                        0x000000010f9fffd8 chromedriver + 5095384
17  chromedriver                        0x000000010fa00161 chromedriver + 5095777
18  chromedriver                        0x000000010fa172a8 chromedriver + 5190312
19  libsystem_pthread.dylib             0x00007fff204ab8fc _pthread_start + 224
20  libsystem_pthread.dylib             0x00007fff204a7443 thread_start + 15

How was that going?

Asked By: xzajyjs

||
Source

Answers:

I’ve run your code many times and yes it crashes after random iterations. and that is an indication of some code some ware didn’t finish its execution. also, one of my runs finished all the 10 iterations. So. that supports my theory.

So. the solution is to add this

time.sleep(3)

at the end of the loop. or just add this line before going to the link. and this is better for future updates.

web.implicitly_wait(10)

you can read more about this at selenium wait documentation

Answered By: Mohaned AbdElMonsef

I used the SeleniumBase Python framework to make the script reliable: Run pip install seleniumbase and then run the following test with pytest:

from seleniumbase import BaseCase

class MyTestClass(BaseCase):
    def test_base(self):
        self.open("http://lagou.com")
        self.click("#cboxClose")
        self.type("#search_input", "pythonn")
        for i in range(1, 11):
            self.click('#jobList > div > div:nth-of-type(%s) a' % i)
            details = self.get_text("#job_detail > dd:nth-of-type(2) > div")
            print(details)
            self.driver.close()
            self.switch_to_window(0)

Additionally, since the website is in Chinese, you might like the script in the Chinese translation of SeleniumBase:

from seleniumbase.translate.chinese import 硒测试用例

class MyTestClass(硒测试用例):
    def test_base(self):
        self.开启("http://lagou.com")
        self.单击("#cboxClose")
        self.输入文本("#search_input", "pythonn")
        for i in range(1, 11):
            self.单击('#jobList > div > div:nth-of-type(%s) a' % i)
            details = self.获取文本("#job_detail > dd:nth-of-type(2) > div")
            print(details)
            self.driver.close()
            self.切换到窗口(0)

Both files will run the same, reliably.

Answered By: Michael Mintz

The trick is to use the wait method in between orders so the page has time to load completely, also use implicitly_wait

time.sleep(5)
driver.implicitly_wait(5)
Answered By: Crisriel