How can I capture realtime request url's to a local file with selenium using python

Question

I have a selenium script set up to open a particular website, the site requests multiple URL’s every 30 seconds and I need to somehow capture those requested URL and copy them to a file locally on my pc.

I’ve done lots of googling and found people recommending browsermob-proxy but it didn’t seem to fit my needs as I need to write the URL to a file in realtime. I’ll give an example photo in chrome’s network developer tool of what I am talking about that I need to have copied to a file.

The site is http://itemalert.com

This is an example of the links I need to fetch every 30 seconds:

An example of the popups:

An example of the inspect elements:

Asked By: tych0brahe

||

Source

Answer 1

This should help u:

from selenium import webdriver
import copy
import threading

global driver
    
def retrieve_urls():    
    
    threading.Timer(30, retrieve_urls).start() #Repeats the task every 30 seconds
    
    driver.get('webpage url') #Enter the url of the webpage here
    
    urls = driver.find_elements_by_class_name('pull-left epnlink')
    
    url_list = []
    
    for element in urls:
        url_list.append(element.get_attribute('href'))

    path = "Urls.txt" #Enter the path of the text file here
    
    f = open(path,'r')
    
    txt = f.readlines()
    
    f.close()
    
    for x in range(len(txt)):
        txt[x] = txt[x].strip()
    
    final_txt = ""
    
    f = open(path,'w')
    
    dup = copy.deepcopy(url_list)
    
    for x in dup:
        if x not in txt:
            final_txt += x + "n"
            url_list.remove(x)
    
    for x in url_list:
        final_txt += x + "n"
    
    f.write(final_txt)        
    
    f.close()

driver = webdriver.Chrome()

retrieve_urls()

If u get any errors, pls let me know. It might be because I did not test my program (cuz I don’t have an account on this website). But I hope that this should work.

Answered By: Sushil

How can I capture realtime request url's to a local file with selenium using python

Question:

Answers: