python error 403 while scrapping angle list website

Question

i have watched other questions on stakeoverflow regarding HTTP 403 error however, have not found solution there.

i would like to change error from 403 to 200

trying to scrape this url https://angel.co/startups.

`import requests
 import random
 my_session = requests.session()

 for_cookies = my_session.get('https://angel.co/startups')
 cookies = for_cookies.cookies

user_agents_list = [
    'Mozilla/5.0 (iPad; CPU OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) 
     Mobile/15E148',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) 
     Chrome/99.0.4844.83 Safari/537.36',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 
     Chrome/99.0.4844.51 Safari/537.36',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 
     Chrome/105.0.0.0 Safari/537.36',

]

response = my_session.get('https://angel.co/startups',cookies=cookies, headers={'User-Agent': 
random.choice(user_agents_list)})
print(response.text)
response.status_code #403

while running this code i am getting 403 error and instead of whole HTML page.

Asked By: vish

||

Source

Answer 1

It may be due to cloudflare protection or some sort of protection.

So, use cloudscraper to bypass it.

import cloudscraper

url = "https://angel.co/startups"
scraper = cloudscraper.create_scraper()
response = scraper.get(url)
text = response.text
print(response.status_code)

Output

200

Answered By: Sagun Devkota

python error 403 while scrapping angle list website

Question:

Answers: