Request for data that generates chart always empty

Question

I am trying to scrape data that generates a chart on a website using python’s request module.

My code currently looks like this:

# load modules
import os
import json
import requests as r

# url to send the call to
postURL = <insert website>

# utiliz get to pull cookie data
cookie_intel = r.get(postURL, verify = False)

# get cookies
search_cookies = cookie_intel.cookies

#### Request Information ####

# API request data
post_data = <insert request json>

# header information
headers = {"user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"}

# results 
results_post = r.post(postURL, data = post_data, cookies = search_cookies, headers = headers, verify = False)

# result
print(results_post.json())

As a quick summary, I first loaded the site to then inspect it, from there I identified the url for the request in the network tab and then checked the required request data in the payload tab. Then I took the user-agent from the request headers tab.

The request itself works, however, it is always empty. I have tried altering all sorts of inputs but without success. I would highly appreciate any sort of tips that would help me to solve this issue. Thank you in advance!

Asked By: fabla

||

Source

Answer 1

in this case you have to use json= instead of data= when making the post request according to the requests documentation . By replacing this part of your code you should get the expected response.

results_post = r.post(postURL, json = post_data, cookies = search_cookies, headers = headers, verify = False)

You can also try other scraping tools like Scrapy to crawl these data and maybe running the crawler on the cloud using estela.

Answered By: Alexander Yanque

Request for data that generates chart always empty

Question:

Answers: