urllib2

How to get specific part of any url using urlparse()? Question: I have an url like this url = ‘https://grabagun.com/firearms/handguns/semi-automatic-handguns/glock-19-gen-5-polished-nickel-9mm-4-02-inch-barrel-15-rounds-exclusive.html’ When I use urlparse() function, I am getting result like this: >>> url = urlparse(url) >>> url.path ‘/firearms/handguns/semi-automatic-handguns/glock-19-gen-5-polished-nickel-9mm-4-02-inch-barrel-15-rounds-exclusive.html’ Is it possible to get something like this: path1 = "firearms" path2 = "handguns" path3 = "semi-automatic-handguns" …

urllib.error.HTTPError: HTTP Error 403: Forbidden with urllib.requests Question: I am trying to read an image URL from the internet and be able to get the image onto my machine via python, I used example used in this blog post https://www.geeksforgeeks.org/how-to-open-an-image-from-the-url-in-pil/ which was https://media.geeksforgeeks.org/wp-content/uploads/20210318103632/gfg-300×300.png, however, when I try my own example it just doesn’t seem to …

Python: download files from google drive using url Question: I am trying to download files from google drive and all I have is the drive’s URL. I have read about google API that talks about some drive_service and MedioIO, which also requires some credentials( mainly JSON file/OAuth). But I am unable to get any idea …

Web scraping – how to access content rendered in JavaScript via Angular.js? Question: I’m trying to scrape data from the public site asx.com.au The page http://www.asx.com.au/asx/research/company.do#!/ACB/details contains a div with class ‘view-content’, which has the information I need: But when I try to view this page via Python’s urllib2.urlopen that div is empty: import urllib2 …

Python3 error: initial_value must be str or None, with StringIO Question: While porting code from python2 to 3, I get this error when reading from a URL TypeError: initial_value must be str or None, not bytes. import urllib import json import gzip from urllib.parse import urlencode from urllib.request import Request service_url = ‘https://babelfy.io/v1/disambiguate’ text = …

Python 3 urllib produces TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str Question: I am trying to convert working Python 2.7 code into Python 3 code and I am receiving a type error from the urllib request module. I used the inbuilt 2to3 Python tool to …

Python percent encoding only certain characters in a URL Question: I have to percent encode only # character if it appears in a given url. I know that we can encode a URL using urllib.quote. It takes a safe keyword to set a particular character to be safe for the URL. I am looking for …

Parse XML from URL into python object Question: The goodreads website has this API for accessing a user’s ‘shelves:’ https://www.goodreads.com/review/list/20990068.xml?key=nGvCqaQ6tn9w4HNpW8kquw&v=2&shelf=toread It returns XML. I’m trying to create a django project that shows books on a shelf from this API. I’m looking to find out how (or if there is a better way than) to write …

Python handling username and password for URL Question: Messing with Python and I’m trying to use this https://updates.opendns.com/nic/update?hostname=, when you got to the URL it will prompt a username and password. I’ve been looking around and I found something about password managers, so I came up with this: urll = “http://url.com” username = “username” password …

Python web scraping gives wrong source code Question: I want to extract some data from Amazon(link in the following code) Here is my code: import urllib2 url=”http://www.amazon.com/s/ref=sr_nr_n_11?rh=n%3A283155%2Cn%3A%2144258011%2Cn%3A2205237011%2Cp_n_feature_browse-bin%3A2656020011%2Cn%3A173507&bbn=2205237011&sort=titlerank&ie=UTF8&qid=1393984161&rnid=1000″ webpage=urllib2.urlopen(url).read() doc=open(“test.html”,”w”) doc.write(webpage) doc.close() When I open the test.html, the content of my page is different from the website in the Internet. Asked By: Hossein Dehghani || Source …