Looping through script based off three arrays

Question

I’ve been working on this script and it’s finally producing results, however to get what I really need from it I have to run it 42 separate times and change the values for exchange, instrument, interval. Basically I need all intervals(7) from each exchange(3) from both instruments(2). so 7 * 3 * 2 = 42 files.

How would looping through all three work? Would I store exchange, instrument, interval as arrays and then loop through them? How would that work would that be a loop within a loop within a loop? It’s hurting my brain just thinking about it. Would there be a better solution to loops? This project is a learning experience but I’m outside of my current comfort zone.

import requests
import json
import pandas as pd
import time
import os

exchange = 'cbse'
instrument = 'btc-usd'  
interval = '15m'

#what I need to loop through
#instrument = ('btc-usd','eth-usd')
#exchange = ('cbse','krkn', 'gmni')  
#interval = ('15s','1m', '3m', '5m', '10m', '15m', '1h')  

page_size = '1000'
url = f'https://us.market-api.kaiko.io/v2/data/trades.v1/exchanges/{exchange}/spot/{instrument}/aggregations/count_ohlcv_vwap'
params = {'interval': interval, 'page_size': page_size}
KEY = 'xxx'
headers = {
    "X-Api-Key": KEY,
    "Accept": "application/json",
    "Accept-Encoding": "gzip"
}
csv_file = f"{exchange}-{instrument}-{interval}.csv"
c_token = True

while(c_token):
    res = requests.get(url, params=params, headers=headers)
    j_data = res.json()
    parse_data = j_data['data']
    c_token = j_data.get('continuation_token')
    today = time.strftime("%Y-%m-%d")
    params = {'continuation_token': c_token}

    if c_token:   
        url = f'https://us.market-api.kaiko.io/v2/data/trades.v1/exchanges/cbse/spot/btc-usd/aggregations/count_ohlcv_vwap?continuation_token={c_token}'        

    # create dataframe
    df = pd.DataFrame.from_dict(pd.json_normalize(parse_data), orient='columns')
    df.insert(1, 'time', pd.to_datetime(df.timestamp.astype(int),unit='ms'))    

    #print(url)
    if(csv_file in os.listdir()): #that means file already exists need to append
        csv_string = df.to_csv(index=False, encoding='utf-8', header=False)
        with open(csv_file, 'a') as f:
            f.write(csv_string)
    else: #that means writing file for the first time
        csv_string = df.to_csv(index=False, encoding='utf-8')
        with open(csv_file, 'w') as f:
            f.write(csv_string)

Asked By: robothead

||

Source

Answer 1

I find for loops to work well for this kind of thing. You can leave them as sets just as they are.

for i in exchange:
    for j in instrument:
        for k in interval:
           # Do stuff

Replace any reference to {exchange}, {instrument}, and {interval} to i, j, and k, respectively

Answered By: goalie1998

Answer 2

If want avoid 3 times loops you can create product for combination ao all lists and then loop in one for like:

from  itertools import product

#renamed for clearness
instrumentL = ('btc-usd','eth-usd')
exchangeL = ('cbse','krkn', 'gmni')  
intervalL = ('15s','1m', '3m', '5m', '10m', '15m', '1h')  
    
for (instrument, exchange, interval) in product(instrumentL, exchangeL, intervalL):
    print (instrument, exchange, interval)
    
    page_size = '1000'
    url = f'https://us.market-api.kaiko.io/v2/data/trades.v1/exchanges/{exchange}/spot/{instrument}/aggregations/count_ohlcv_vwap'
    params = {'interval': interval, 'page_size': page_size}
    KEY = 'xxx'
    headers = {
        "X-Api-Key": KEY,
        "Accept": "application/json",
        "Accept-Encoding": "gzip"
    }
    csv_file = f"{exchange}-{instrument}-{interval}.csv"
    c_token = True
    ...
    ...

Answered By: jezrael

Looping through script based off three arrays

Question:

Answers: