How to identify elements needed for requests in python?

Question

I have a web url https://www.blibli.com/backend/content/promotions/blm-ranch-market
where when I opened it on chrome returns a json.

However when I tried using requests in python and tried using headers including the cookie from headers it does not work. What is missing?
How do we know what’s needed for the request?I check in sources there’s no payload only headers, I assumed by this code it will work. However using the same headers does not work.

import requests

resp = requests.get('https://www.blibli.com/backend/content/promotions/blm-ranch-market')
if resp == 200:
    print(resp.json())

Asked By: Halim stack

||

Source

Answer 1

If you want to get the contents of the page, you might want to try resp.content.

Answered By: ankitgupta5294

Answer 2

It’s because that site does not support HTTP/1.1 and requests(and its backend http client, http.client in the standard library) does not support HTTP/2.

If you need a HTTP/2 client library, try other library, such as HTTPX.

Answered By: relent95

Answer 3

It worked fine for me after adding a User-Agent header.

import requests

headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
}

response = requests.get('https://www.blibli.com/backend/content/promotions/blm-ranch-market', headers=headers)

print(response.status_code)
print(response.json())

output:

200
{'code': 200,
 'data': {'components': [{'id': '8ab055d680424814018045bccc5e42ce',
                          'name': 'PAGE_ANCHOR_TABBING',
                          'parameters': [{'alt': 'Produk Terlaris',
                                          'id': '8ab055d680424814018045bccc5e42d2',
                                          'image': 'https://www.static-src.com/siva/asset//02_2022/Icon_TBL_musthave.png',
                                          'imageRatio': '1:1',
                                          'isMerchant': False,
                                          'name': 'image_4',
                                          'newTab': False,
                                          'requireLocation': False,
                                          'sequence': 0,
                                          'showProductTag': False,
                                          'sumOfProduct': 0,
                                          'title': 'Produk Terlaris',
                                          'type': 'IMAGE',
                                          'url': 'PROMO_1_BANNER-24',
                                          'use2HdProduct': False},
                                         {'alt': 'Produk Pilihan',
                                          'id': '8ab055d680424814018045bccc5e42d3',
                                          'image': 'https://www.static-src.com/siva/asset//02_2022/Icon_TBL_popular.png',
                                          'imageRatio': '1:1',
                                          'isMerchant': False,
                                          'name': 'image_5',
                                          'newTab': False,
                                          'requireLocation': False,
                                          'sequence': 0,
                                          'showProductTag': False,
                                          'sumOfProduct': 0,
                                          'title': 'Produk Pilihan',
                                          'type': 'IMAGE',
                                          'url': 'PROMO_1_BANNER-26',
                                          'use2HdProduct': False},
                                         {'alt': 'Vouchers',
                                          'id': '8ab055d680424814018045bccc5e42cf',
                                          'image': 'https://www.static-src.com/siva/asset//02_2022/BPD-FEB-ICON-CASHBACK.jpg',
                                          'imageRatio': '1:1',
                                          'isMerchant': False,
                                          'name': 'image_1',
                                          'newTab': False,
                                          'requireLocation': False,
                                          'sequence': 0,
                                          'showProductTag': False,
                                          'sumOfProduct': 0,
                                          'title': 'Vouchers',
                                          'type': 'IMAGE',
                                          'url': 'VOUCHERS_GEN_2-6',
                                          'use2HdProduct': False},
...

Answered By: Alexander

How to identify elements needed for requests in python?

Question:

Answers: