python-requests

python lxml.etree generating empty tree when given requests.get().text Question: I am trying to build a web scraper for TEDTalks and am running into an issue when generating transcripts. Python 3.10.4, lxml 4.9.2. First I am generating the html response like this. text = requests.get(‘https://www.ted.com/talks/ted_countdown_how_do_we_get_the_world_off_fossil_fuels_quickly_and_fairly/transcript’, headers={}).text when checking the value of text, it shows me that …

No impact of payload on scrappy request Question: I am facing strange issue. url = ["https://nr.aws-achat.info/_extranet/index.cfm?fuseaction=mEnt.lister"] payload = { ‘rechInputCPV’:’03000000-1′, ‘rechInputMetier’:”, ‘texte’:”, ‘btnSub’:’Afficher’ } yield scrapy.Request(url[0],method=’POST’,body=json.dumps(payload),callback=self.parse) in above case of scrappy request, the response is same as if I pass payload as blank dict. Expectation : if pass rechInputCPV’:’03000000- I should get 60 rows of …

Get ALL strings from html Question: I send get requests to different sites. In response I get HTML pages. How can I get only strings from the HTML page? I mean all strings in general (the ones colored white in my screenshot). I understand how I can get "div", "code", "a", and etc tags. But …

How to download chromedriver instead of webpage with python requests library Question: Today I’m creating a Python script to find a specific version of chromedriver here. My intention is to make a request to the URL containing the specific version of chromedriver I need to download and save it to the same directory as the …

How can I fix invaild format specifier in python? Question: I can’t find what I am doing wrong… I am trying to send a post req with custom values but cant seem to make it work always getting invaild format specifier. Here is the original request: headers = header data = { ‘source’: ‘src_1MnugIHWW88ELdDvkI8CNwa1’, ‘final_cres’: …

Select specific values from JSON output Question: I am querying a REST API and I need to select 2 fields from the adapter output below. I basically need to make variables from OUT_Detailed Description and OUT_Vendor Ticket Number: Code: headers = {‘content-type’: ‘application/json’, ‘Authentication-Token’: authToken} response = requests.post(‘http://dev.startschools.local:2031/baocdp/rest/process/:ITSM_Interface:IncidentManagement:QueryIncident/execute’, headers=headers, data=json.dumps(get_query_inc_json())) print(response.text) json_format = json.loads(response) Description …

Parse a table from wikipedia that is hidden Question: I’am pretty new here. I want to parse a table from wikipedia from a following link: https://en.wikipedia.org/wiki/MIUI I was able to parse first table, but I can’t figure out how to get the information from the second table there, the information that contains "version history" of …

How to add images with Discord API and Python Requests Question: I want to send an image that I have stored on my computer through this code import requests import time headers = {‘authorization’: TOKEN, ‘Content-Type’: ‘application/json’} url = f"https://discord.com/api/v9/channels/{CHANNEL}/messages" data = { "content": message } files = { ‘file’: (open(‘image.png’, ‘rb’)), } r = …

POST request returning null values Question: I’m trying to make some POST requests to a website and, while I get 200 responses every time I do so, the values of said responses are all either null or 0’s. I’ve tried using all the headers visible when inspecting the website, just a combination of them, insertint …

Python requests stop redirects does not work Question: I want to access the content of a web page, but I’m being redirected to another page even though I’ve set allow_redirects to False in my requests call. Here’s an example code snippet: import requests from bs4 import BeautifulSoup headers = {‘User-Agent’: user_agent} # assume I inserted …