How to programmatically generate new headers from URL?

Question

I’m writing a script to pull financial data from a website everyday however the script stops working after a few hours because the cookies expire. I need to generate new headers (I think just the x-xsrf-token and cookies) each time I run the script (once a day), otherwise I get a 401 status code.

This is the page I’m trying to pull data from:

https://www.barchart.com/futures/quotes/CLZ22/futures-prices?viewName=main&timeFrame=current

And this is the XHR response url that I’m trying to scrape:

https://www.barchart.com/proxies/core-api/v1/quotes/get?fields=symbol%2CcontractSymbol%2ClastPrice%2CpriceChange%2CopenPrice%2ChighPrice%2ClowPrice%2CpreviousPrice%2Cvolume%2CopenInterest%2CtradeTime%2CsymbolCode%2CsymbolType%2ChasOptions&lists=futures.contractInRoot&root=CL&meta=field.shortName%2Cfield.type%2Cfield.description%2Clists.lastUpdate&hasOptions=true&page=1&limit=100&raw=1

The only way I know how to currently do that is to go to the website and copy the XHR request as cURL (bash) then paste this into Postman and manually paste these headers into my existing code.

Below is the code generated from Postman. I’ve been trying to figure out how to generate the headers from the URL so I don’t have to run to Postman and manually refresh the headers everyday.

import requests

url = "https://www.barchart.com/proxies/core-api/v1/quotes/get?fields=symbol%2CcontractSymbol%2ClastPrice%2CpriceChange%2CopenPrice%2ChighPrice%2ClowPrice%2CpreviousPrice%2Cvolume%2CopenInterest%2CtradeTime%2CsymbolCode%2CsymbolType%2ChasOptions&lists=futures.contractInRoot&root=CL&meta=field.shortName%2Cfield.type%2Cfield.description%2Clists.lastUpdate&hasOptions=true&page=1&limit=100&raw=1"

payload={}
headers = {
  'authority': 'www.barchart.com',
  'accept': 'application/json',
  'accept-language': 'en-US,en;q=0.9',
  'cookie': 'webinar124WebinarClosed=true; market=eyJpdiI6IkovREZvUVlZMGFzM2x3b05wb3V4cGc9PSIsInZhbHVlIjoiSVJoT00rMTdWUFFYRlJiOG53OU12dTdjcUhEL3FKTW5XUy9FZFNjc1Z2VWkwdjV2RkNrWXpGZzYzMUNpK2IxbyIsIm1hYyI6IjkzNDYxOTg5OWQ5MzgxYjhlMGI4ODg4NDRlMDA1NWE3MjUxYTNmOTMzNzllYjBjYjhmNGM1ZGZiYjA0Yzk5ODEifQ%3D%3D; bcFreeUserPageView=0; laravel_token=eyJpdiI6ImR3bGlHVTY3WEhGdkdEWlBleGtkMFE9PSIsInZhbHVlIjoiWnFJMml3dis3cVN0d2VIdDRCbFQvczRmVGZxcjFYYTF4YWpBd09NSVJBVXRQYVVLdWxnaUlTM3dXTElUaUJHK1VoQkxaQkdsRHNlTzZRU3c2R3NhZzVROUYvRHM4TTQ3V2srcHZLZG9Ra3BzOUZndXhxME4rSmtYODZHTWtmN3pmOENtRGZWQmdhUEZFc0FiZ0dSV1BEbC9acTVVQnBTOUl1Y2ZleW50WVAxSmYvMTdQQVZlN0lRQ25qR1BKQWZUMU1XbE5rcW14ZTYvTkpVbkpmcXc2RVRHUmtrUHlTNithNkJiY1ZTNG1rWkl1cHkxeVRWUU9zZUE2RFhoN2VYeGFnZytPN2RBZ3VPS0tJdVQxZUw4eHB1d2FZN3JKNlJ3QmllYWx2N21nUGlFb25OYXM0aFhjbFBCS0Q0ajJTSmMiLCJtYWMiOiJkODgwMjliMzM4MGI4M2E4Njk4MmE3ODYzMDY2ZmRkYjRmN2MzZGExYThhMTliMTE2YjNiZDQ1YzkzZWMzMWQwIn0%3D; XSRF-TOKEN=eyJpdiI6InQyUll2aHRCaXFlQkZIRXV0TjdaVGc9PSIsInZhbHVlIjoiSmtZaXlTbmVrTkJNVmEyUHQrUDFZN1RWNCt5cmFSanMxcnpTTW8vTjdrTU1RVlZQWktXNnhtakJjeVJ6Y0h3cFpkaWl4UnBvS28vTHNCUzNsM0ZRcXN2ZG9tWnFLTUVwdUZHY2VhNmxSRFg0ajhXU0lobFRZaFZRanhHZis4STkiLCJtYWMiOiI2MDc3NjIzNTAwMmY5MjlkNjRkMTVkYTZjYmNiM2RiNjg4ZDI1MmUzZWEzYjc1NWY0ZDNiZGNjNzY0ZGY2NGY5In0%3D; laravel_session=eyJpdiI6IlhPVGVDbTVURlpWRDcvNWVMWUgxclE9PSIsInZhbHVlIjoiVEJvTUVIVkRHOFlQUXNKcUJRaGtmZ2U4aVcrbE9JNDV3bG1adG1DLzVpSzI5Z0lqYlk2NU5TQkE5ZTAzMHZPL1VoVjJlZU9kSkYvT1VERFBsK1BnRUVzaGMzVlNiRFFTQzFPblEyMUFXSjM3dmdRQXhnTXFSaVYwSkNkZ3ZJS3UiLCJtYWMiOiJjMzUzMzAyMjEzYzYwZGZmM2M3OTMwMGE0OGM3NTJmM2M3MzhkNDUyYjE2OTI4Njg5ODQxNDM3NjcyMzM0ZWE5In0%3D',
  'referer': 'https://www.barchart.com/futures/quotes/CLZ22/futures-prices?viewName=main&timeFrame=current',
  'sec-ch-ua': '"Chromium";v="106", "Google Chrome";v="106", "Not;A=Brand";v="99"',
  'sec-ch-ua-mobile': '?0',
  'sec-ch-ua-platform': '"Windows"',
  'sec-fetch-dest': 'empty',
  'sec-fetch-mode': 'cors',
  'sec-fetch-site': 'same-origin',
  'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36',
  'x-xsrf-token': 'eyJpdiI6InQyUll2aHRCaXFlQkZIRXV0TjdaVGc9PSIsInZhbHVlIjoiSmtZaXlTbmVrTkJNVmEyUHQrUDFZN1RWNCt5cmFSanMxcnpTTW8vTjdrTU1RVlZQWktXNnhtakJjeVJ6Y0h3cFpkaWl4UnBvS28vTHNCUzNsM0ZRcXN2ZG9tWnFLTUVwdUZHY2VhNmxSRFg0ajhXU0lobFRZaFZRanhHZis4STkiLCJtYWMiOiI2MDc3NjIzNTAwMmY5MjlkNjRkMTVkYTZjYmNiM2RiNjg4ZDI1MmUzZWEzYjc1NWY0ZDNiZGNjNzY0ZGY2NGY5In0='
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Is there a way to generate these headers in Python from xhr response url that I can then use when sending my GET request?

Asked By: grahamrob

||

Source

Answer 1

The cookies are the bottleneck. You first have to fetch them and then pass them along with the request:

import requests
from urllib.parse import unquote

ua_headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0"
}

params = {
    'fields': 'symbol,contractSymbol,lastPrice,priceChange,openPrice,highPrice,lowPrice,previousPrice,volume,openInterest,tradeTime,symbolCode,symbolType,hasOptions',
    'lists': 'futures.contractInRoot',
    'root': 'CL',
    'meta': 'field.shortName,field.type,field.description,lists.lastUpdate',
    'hasOptions': 'true',
    'page': '1',
    'limit': '100',
    'raw': '1',
}

with requests.Session() as s:
    # get cookies
    s.get("https://www.barchart.com/options/iv-rank-percentile/stocks", headers=ua_headers)
    # use one cookie as HTTP header
    headers["X-XSRF-TOKEN"] = unquote(s.cookies["XSRF-TOKEN"])
    response = s.get('https://www.barchart.com/proxies/core-api/v1/quotes/get', params=params, headers=headers)
    print(response.json())

Answered By: RJ Adriaansen

How to programmatically generate new headers from URL?

Question:

Answers: