How to bypass Cloudflare with Python on GET requests?

Question

I want to bypass Cloudflare on a GET request I have tried using Cloudscraper which worked for me in the past but now seems decreped.

I tried:

import cloudscraper
import requests
ses = requests.Session()
ses.headers = {
    'referer': 'https://magiceden.io/',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36',
    'accept': 'application/json'
}
scraper = cloudscraper.create_scraper(sess=ses)
hookLink = f"https://magiceden.io/launchpad/planetarians"
meG = scraper.get("https://api-mainnet.magiceden.io/launchpads/planetarians")
print(meG.status_code)
print(meG.text)

The issue seems to be that I’m getting a captcha on the request

Asked By: jonAvery22

||

Source

Answer 1

The python library works well (I never knew about it), the issue is your user agent. Cloudflare uses some sort of extra checks to determine whether you’re faking it.

For me, any of the following works:

ses.headers = {
    'referer': 'https://magiceden.io/',
    'accept': 'application/json'
}

ses.headers = {
    'accept': 'application/json'
}

And also just:

scraper = cloudscraper.create_scraper()
meG = scraper.get("https://api-mainnet.magiceden.io/launchpads/planetarians")

EDIT:

You can use this dict syntax instead to fake the user agent (as per the manual)

scraper = cloudscraper.create_scraper(
    browser={
        'browser': 'chrome',
        'platform': 'windows',
        'desktop': True
    }
)

Answered By: swaggg

How to bypass Cloudflare with Python on GET requests?

Question:

Answers: