python list of dictionaries find paired values
Question:
I have this csv containing some paired rows such as:
LabebStoreId,catalog_uuid,lang,cat_0_name,cat_1_name,cat_2_name,cat_3_name,catalogname,description,properties,price,price_before_discount,externallink,Rating,delivery,discount,instock
6021,89028,en,Electronics & Appliances,Batteries & Power,Batteries,Alkaline Batteries,Energizer Max AA Alkaline Battery E91BP Pack of 8,,"{""Number of batteries included"": ""8"", ""Battery voltage"": ""1.5""}",41.5,,https://www.carrefouruae.com//mafuae/en/alkaline-batteries/energizer-max-alke91bp-batery-8-aa/p/89028,,,,
6021,89028,ar,الإلكترونيات والأجهزة المنزلية,البطاريات ومولدات الطاقة,بطاريات,بطاريات الكالاين,ENERGIZER BATTERY AAX8 MAX,,"{""Number of batteries included"": ""8"", ""طاقة البطارية"": ""1.5""}",41.5,,https://www.carrefouruae.com//mafuae/ar/alkaline-batteries/energizer-max-alke91bp-batery-8-aa/p/89028,,,,
6021,742553,en,Electronics & Appliances,Batteries & Power,Batteries,Alkaline Batteries,Energizer Max AAA Alkaline Battery EP2BP Pack of 8,,"{""Number of batteries included"": ""8"", ""Battery voltage"": ""1.5""}",33.0,,https://www.carrefouruae.com//mafuae/en/alkaline-batteries/energizer-max-alke92bp-batery-8-aaa/p/742553,,,,
6021,742553,ar,الإلكترونيات والأجهزة المنزلية,البطاريات ومولدات الطاقة,بطاريات,بطاريات الكالاين,ENERGIZER AAA/8 MAX ALKALINE,,"{""Number of batteries included"": ""8"", ""طاقة البطارية"": ""1.5""}",33.0,,https://www.carrefouruae.com//mafuae/ar/alkaline-batteries/energizer-max-alke92bp-batery-8-aaa/p/742553,,,,
6021,893379,en,Electronics & Appliances,Large Appliances,Fridges,Fridge 101L to 200L,First1 Free Standing 2 shelves Fridge FR-130L,,"{""Dimensions WxDxH"": ""W 49cm x D 45cm x H 83.5cm""}",549.0,,https://www.carrefouruae.com//mafuae/en/fridge-101l-to-200l/mychoice-first1-fridge-fr-130l-94l/p/893379,,Free delivery,,1.0
6021,893379,ar,الإلكترونيات والأجهزة المنزلية,أجهزة منزلية كبيرة,الثلاجات,ثلاجة 101L إلى 200L,FIRST1 FR-130L FRIDGE,,"{""Dimensions WxDxH"": ""W 49cm x D 45cm x H 83.5cm""}",549.0,,https://www.carrefouruae.com//mafuae/ar/fridge-101l-to-200l/mychoice-first1-fridge-fr-130l-94l/p/893379,,,,1.0
6021,915909,en,Electronics & Appliances,Batteries & Power,Power Adaptors & Sockets,Adaptors & Sockets,Elexon Universal Travel Adaptor PP7974A White,,{},14.5,,https://www.carrefouruae.com//mafuae/en/adaptors-sockets/elexon-universal-travel-adaptor/p/915909,,,,
6021,945671,en,Electronics & Appliances,Small Appliances,Food Preparation,Rice Cooker,Mychoice Rice Cooker 350W F-06RC White,,{},69.0,,https://www.carrefouruae.com//mafuae/en/rice-cooker/mychoice-first1-rice-cooker-f-06rc/p/945671,,,,
6021,945671,ar,الإلكترونيات والأجهزة المنزلية,أجهزة منزلية صغيرة,تحضير الطعام,طبخ الأرز,جهاز طهي الأرز الأول F-06RC 350 وات 0.6 لتر,,{},69.0,,https://www.carrefouruae.com//mafuae/ar/rice-cooker/mychoice-first1-rice-cooker-f-06rc/p/945671,,,,
one row for English and one for Arabic and I’d like to setup some check like this:
if catalog_uuid contains en and ar:
payload = {
"row": {
"LabebStoreId": d["LabebStoreId"],
"catalog_uuid": d["catalog_uuid"],
"lang": d["lang"],
"cat_0_name": d["cat_0_name"],
"cat_1_name": d["cat_1_name"],
"cat_2_name": d["cat_2_name"],
"cat_3_name": d["cat_3_name"],
"catalogname": d["catalogname"],
"description": d["description"],
"properties": d["properties"],
"price": d["price"],
"price_before_discount": d["price_before_discount"],
"externallink": d["externallink"],
"Rating": d["Rating"],
"delivery": d["delivery"],
"discount": d["discount"],
"instock": d["instock"],
}
}
payload = {
"nextRow": {
"LabebStoreId": d["LabebStoreId"],
"catalog_uuid": d["catalog_uuid"],
"lang": d["lang"],
"cat_0_name": d["cat_0_name"],
"cat_1_name": d["cat_1_name"],
"cat_2_name": d["cat_2_name"],
"cat_3_name": d["cat_3_name"],
"catalogname": d["catalogname"],
"description": d["description"],
"properties": d["properties"],
"price": d["price"],
"price_before_discount": d["price_before_discount"],
"externallink": d["externallink"],
"Rating": d["Rating"],
"delivery": d["delivery"],
"discount": d["discount"],
"instock": d["instock"],
}
}
elif catalog_uuid contains only en:
payload = {
"row": {
"LabebStoreId": d["LabebStoreId"],
"catalog_uuid": d["catalog_uuid"],
"lang": d["lang"],
"cat_0_name": d["cat_0_name"],
"cat_1_name": d["cat_1_name"],
"cat_2_name": d["cat_2_name"],
"cat_3_name": d["cat_3_name"],
"catalogname": d["catalogname"],
"description": d["description"],
"properties": d["properties"],
"price": d["price"],
"price_before_discount": d["price_before_discount"],
"externallink": d["externallink"],
"Rating": d["Rating"],
"delivery": d["delivery"],
"discount": d["discount"],
"instock": d["instock"],
}
}
elif catalog_uuid contains only ar:
payload = {
"row": {
"LabebStoreId": d["LabebStoreId"],
"catalog_uuid": d["catalog_uuid"],
"lang": d["lang"],
"cat_0_name": d["cat_0_name"],
"cat_1_name": d["cat_1_name"],
"cat_2_name": d["cat_2_name"],
"cat_3_name": d["cat_3_name"],
"catalogname": d["catalogname"],
"description": d["description"],
"properties": d["properties"],
"price": d["price"],
"price_before_discount": d["price_before_discount"],
"externallink": d["externallink"],
"Rating": d["Rating"],
"delivery": d["delivery"],
"discount": d["discount"],
"instock": d["instock"],
}
}
some rows only contains English and some only contains Arabic so I have to cover both aspects. I am using pandas to read the csv and transforming the dataframe to pd.to_json(orient="index") because I need to post that rows in a POST request as a json.
Here is my recent try:
import pandas as pd
import requests
import json
import ast
HEADERS = {"Accept": "*/*", "Content-Type": "application/json"}
df = pd.read_csv("carrefour-uae-items-final.csv").fillna("").astype(str)
dict_data = json.loads(df.to_json(orient="index"))
list_dict = [d for d in dict_data.values()]
for d in list_dict:
cat_id = d["catalog_uuid"]
if next(item for item in list_dict if item["catalog_uuid"] == cat_id):
payload_rows = ["row", "nextRow"]
for row in payload_rows:
payload = {
row: {
"LabebStoreId": d["LabebStoreId"],
"catalog_uuid": d["catalog_uuid"],
"lang": d["lang"],
"cat_0_name": d["cat_0_name"],
"cat_1_name": d["cat_1_name"],
"cat_2_name": d["cat_2_name"],
"cat_3_name": d["cat_3_name"],
"catalogname": d["catalogname"],
"description": d["description"],
"properties": d["properties"],
"price": d["price"],
"price_before_discount": d["price_before_discount"],
"externallink": d["externallink"],
"Rating": d["Rating"],
"delivery": d["delivery"],
"discount": d["discount"],
"instock": d["instock"],
"images": ast.literal_eval(d["encoded_images"]),
}
}
print(json.dumps(payload, indent=4))
response = requests.post(
"http://crawlerapi.labeb.com/api/PCCrawler/Crawl?StoreId=6021",
headers=HEADERS,
json=payload,
)
print(response.content.decode())
else:
payload = {
"row": {
"LabebStoreId": d["LabebStoreId"],
"catalog_uuid": d["catalog_uuid"],
"lang": d["lang"],
"cat_0_name": d["cat_0_name"],
"cat_1_name": d["cat_1_name"],
"cat_2_name": d["cat_2_name"],
"cat_3_name": d["cat_3_name"],
"catalogname": d["catalogname"],
"description": d["description"],
"properties": d["properties"],
"price": d["price"],
"price_before_discount": d["price_before_discount"],
"externallink": d["externallink"],
"Rating": d["Rating"],
"delivery": d["delivery"],
"discount": d["discount"],
"instock": d["instock"],
"images": ast.literal_eval(d["encoded_images"]),
}
}
print(json.dumps(payload, indent=4))
response = requests.post(
"http://crawlerapi.labeb.com/api/PCCrawler/Crawl?StoreId=6021",
headers=HEADERS,
json=payload,
)
print(response.content.decode())
Can anyone please help me figure out how to achieve the above logic? Thanks in advance!
Answers:
As I understand correctly, you can group the data by LabebStoreId and catalog_uuid and then you make payloads according each group:
import csv
data = {}
with open("data.csv", "r") as f_in:
reader = csv.DictReader(f_in)
for row in reader:
data.setdefault((row["LabebStoreId"], row["catalog_uuid"]), []).append(
row
)
for payload_no, v in enumerate(data.values(), 1):
print("Payload number", payload_no)
for d in v:
print("Language", d["lang"])
print("t", d)
print("-" * 80)
Prints:
Payload number 1
Language en
{'LabebStoreId': '6021', 'catalog_uuid': '89028', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Batteries & Power', 'cat_2_name': 'Batteries', 'cat_3_name': 'Alkaline Batteries', 'catalogname': 'Energizer Max AA Alkaline Battery E91BP Pack of 8', 'description': '', 'properties': '{"Number of batteries included": "8", "Battery voltage": "1.5"}', 'price': '41.5', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/alkaline-batteries/energizer-max-alke91bp-batery-8-aa/p/89028', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}
Language ar
{'LabebStoreId': '6021', 'catalog_uuid': '89028', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'البطاريات ومولدات الطاقة', 'cat_2_name': 'بطاريات', 'cat_3_name': 'بطاريات الكالاين', 'catalogname': 'ENERGIZER BATTERY AAX8 MAX', 'description': '', 'properties': '{"Number of batteries included": "8", "طاقة البطارية": "1.5"}', 'price': '41.5', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/alkaline-batteries/energizer-max-alke91bp-batery-8-aa/p/89028', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}
--------------------------------------------------------------------------------
Payload number 2
Language en
{'LabebStoreId': '6021', 'catalog_uuid': '742553', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Batteries & Power', 'cat_2_name': 'Batteries', 'cat_3_name': 'Alkaline Batteries', 'catalogname': 'Energizer Max AAA Alkaline Battery EP2BP Pack of 8', 'description': '', 'properties': '{"Number of batteries included": "8", "Battery voltage": "1.5"}', 'price': '33.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/alkaline-batteries/energizer-max-alke92bp-batery-8-aaa/p/742553', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}
Language ar
{'LabebStoreId': '6021', 'catalog_uuid': '742553', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'البطاريات ومولدات الطاقة', 'cat_2_name': 'بطاريات', 'cat_3_name': 'بطاريات الكالاين', 'catalogname': 'ENERGIZER AAA/8 MAX ALKALINE', 'description': '', 'properties': '{"Number of batteries included": "8", "طاقة البطارية": "1.5"}', 'price': '33.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/alkaline-batteries/energizer-max-alke92bp-batery-8-aaa/p/742553', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}
--------------------------------------------------------------------------------
Payload number 3
Language en
{'LabebStoreId': '6021', 'catalog_uuid': '893379', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Large Appliances', 'cat_2_name': 'Fridges', 'cat_3_name': 'Fridge 101L to 200L', 'catalogname': 'First1 Free Standing 2 shelves Fridge FR-130L', 'description': '', 'properties': '{"Dimensions WxDxH": "W 49cm x D 45cm x H 83.5cm"}', 'price': '549.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/fridge-101l-to-200l/mychoice-first1-fridge-fr-130l-94l/p/893379', 'Rating': '', 'delivery': 'Free delivery', 'discount': '', 'instock': '1.0'}
Language ar
{'LabebStoreId': '6021', 'catalog_uuid': '893379', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'أجهزة منزلية كبيرة', 'cat_2_name': 'الثلاجات', 'cat_3_name': 'ثلاجة 101L إلى 200L', 'catalogname': 'FIRST1 FR-130L FRIDGE', 'description': '', 'properties': '{"Dimensions WxDxH": "W 49cm x D 45cm x H 83.5cm"}', 'price': '549.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/fridge-101l-to-200l/mychoice-first1-fridge-fr-130l-94l/p/893379', 'Rating': '', 'delivery': '', 'discount': '', 'instock': '1.0'}
--------------------------------------------------------------------------------
Payload number 4
Language en
{'LabebStoreId': '6021', 'catalog_uuid': '915909', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Batteries & Power', 'cat_2_name': 'Power Adaptors & Sockets', 'cat_3_name': 'Adaptors & Sockets', 'catalogname': 'Elexon Universal Travel Adaptor PP7974A White', 'description': '', 'properties': '{}', 'price': '14.5', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/adaptors-sockets/elexon-universal-travel-adaptor/p/915909', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}
--------------------------------------------------------------------------------
Payload number 5
Language en
{'LabebStoreId': '6021', 'catalog_uuid': '945671', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Small Appliances', 'cat_2_name': 'Food Preparation', 'cat_3_name': 'Rice Cooker', 'catalogname': 'Mychoice Rice Cooker 350W F-06RC White', 'description': '', 'properties': '{}', 'price': '69.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/rice-cooker/mychoice-first1-rice-cooker-f-06rc/p/945671', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}
Language ar
{'LabebStoreId': '6021', 'catalog_uuid': '945671', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'أجهزة منزلية صغيرة', 'cat_2_name': 'تحضير الطعام', 'cat_3_name': 'طبخ الأرز', 'catalogname': 'جهاز طهي الأرز الأول F-06RC 350 وات 0.6 لتر', 'description': '', 'properties': '{}', 'price': '69.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/rice-cooker/mychoice-first1-rice-cooker-f-06rc/p/945671', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}
--------------------------------------------------------------------------------
To have {'row':...} or {'row':..., 'nextRow':...} you can check lenght of the group:
import csv
data = {}
with open("data.csv", "r") as f_in:
reader = csv.DictReader(f_in)
for row in reader:
data.setdefault((row["LabebStoreId"], row["catalog_uuid"]), []).append(
row
)
for payload_no, v in enumerate(data.values(), 1):
print("Payload number", payload_no)
if len(v) == 1:
payload = {"row": v[0]}
else:
payload = {"row": v[0], "nextRow": v[1]}
print("t", payload)
print("-" * 80)
Prints:
Payload number 1
{'row': {'LabebStoreId': '6021', 'catalog_uuid': '89028', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Batteries & Power', 'cat_2_name': 'Batteries', 'cat_3_name': 'Alkaline Batteries', 'catalogname': 'Energizer Max AA Alkaline Battery E91BP Pack of 8', 'description': '', 'properties': '{"Number of batteries included": "8", "Battery voltage": "1.5"}', 'price': '41.5', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/alkaline-batteries/energizer-max-alke91bp-batery-8-aa/p/89028', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}, 'nextRow': {'LabebStoreId': '6021', 'catalog_uuid': '89028', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'البطاريات ومولدات الطاقة', 'cat_2_name': 'بطاريات', 'cat_3_name': 'بطاريات الكالاين', 'catalogname': 'ENERGIZER BATTERY AAX8 MAX', 'description': '', 'properties': '{"Number of batteries included": "8", "طاقة البطارية": "1.5"}', 'price': '41.5', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/alkaline-batteries/energizer-max-alke91bp-batery-8-aa/p/89028', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}}
--------------------------------------------------------------------------------
Payload number 2
{'row': {'LabebStoreId': '6021', 'catalog_uuid': '742553', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Batteries & Power', 'cat_2_name': 'Batteries', 'cat_3_name': 'Alkaline Batteries', 'catalogname': 'Energizer Max AAA Alkaline Battery EP2BP Pack of 8', 'description': '', 'properties': '{"Number of batteries included": "8", "Battery voltage": "1.5"}', 'price': '33.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/alkaline-batteries/energizer-max-alke92bp-batery-8-aaa/p/742553', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}, 'nextRow': {'LabebStoreId': '6021', 'catalog_uuid': '742553', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'البطاريات ومولدات الطاقة', 'cat_2_name': 'بطاريات', 'cat_3_name': 'بطاريات الكالاين', 'catalogname': 'ENERGIZER AAA/8 MAX ALKALINE', 'description': '', 'properties': '{"Number of batteries included": "8", "طاقة البطارية": "1.5"}', 'price': '33.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/alkaline-batteries/energizer-max-alke92bp-batery-8-aaa/p/742553', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}}
--------------------------------------------------------------------------------
Payload number 3
{'row': {'LabebStoreId': '6021', 'catalog_uuid': '893379', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Large Appliances', 'cat_2_name': 'Fridges', 'cat_3_name': 'Fridge 101L to 200L', 'catalogname': 'First1 Free Standing 2 shelves Fridge FR-130L', 'description': '', 'properties': '{"Dimensions WxDxH": "W 49cm x D 45cm x H 83.5cm"}', 'price': '549.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/fridge-101l-to-200l/mychoice-first1-fridge-fr-130l-94l/p/893379', 'Rating': '', 'delivery': 'Free delivery', 'discount': '', 'instock': '1.0'}, 'nextRow': {'LabebStoreId': '6021', 'catalog_uuid': '893379', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'أجهزة منزلية كبيرة', 'cat_2_name': 'الثلاجات', 'cat_3_name': 'ثلاجة 101L إلى 200L', 'catalogname': 'FIRST1 FR-130L FRIDGE', 'description': '', 'properties': '{"Dimensions WxDxH": "W 49cm x D 45cm x H 83.5cm"}', 'price': '549.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/fridge-101l-to-200l/mychoice-first1-fridge-fr-130l-94l/p/893379', 'Rating': '', 'delivery': '', 'discount': '', 'instock': '1.0'}}
--------------------------------------------------------------------------------
Payload number 4
{'row': {'LabebStoreId': '6021', 'catalog_uuid': '915909', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Batteries & Power', 'cat_2_name': 'Power Adaptors & Sockets', 'cat_3_name': 'Adaptors & Sockets', 'catalogname': 'Elexon Universal Travel Adaptor PP7974A White', 'description': '', 'properties': '{}', 'price': '14.5', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/adaptors-sockets/elexon-universal-travel-adaptor/p/915909', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}}
--------------------------------------------------------------------------------
Payload number 5
{'row': {'LabebStoreId': '6021', 'catalog_uuid': '945671', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Small Appliances', 'cat_2_name': 'Food Preparation', 'cat_3_name': 'Rice Cooker', 'catalogname': 'Mychoice Rice Cooker 350W F-06RC White', 'description': '', 'properties': '{}', 'price': '69.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/rice-cooker/mychoice-first1-rice-cooker-f-06rc/p/945671', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}, 'nextRow': {'LabebStoreId': '6021', 'catalog_uuid': '945671', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'أجهزة منزلية صغيرة', 'cat_2_name': 'تحضير الطعام', 'cat_3_name': 'طبخ الأرز', 'catalogname': 'جهاز طهي الأرز الأول F-06RC 350 وات 0.6 لتر', 'description': '', 'properties': '{}', 'price': '69.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/rice-cooker/mychoice-first1-rice-cooker-f-06rc/p/945671', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}}
--------------------------------------------------------------------------------
I have this csv containing some paired rows such as:
LabebStoreId,catalog_uuid,lang,cat_0_name,cat_1_name,cat_2_name,cat_3_name,catalogname,description,properties,price,price_before_discount,externallink,Rating,delivery,discount,instock
6021,89028,en,Electronics & Appliances,Batteries & Power,Batteries,Alkaline Batteries,Energizer Max AA Alkaline Battery E91BP Pack of 8,,"{""Number of batteries included"": ""8"", ""Battery voltage"": ""1.5""}",41.5,,https://www.carrefouruae.com//mafuae/en/alkaline-batteries/energizer-max-alke91bp-batery-8-aa/p/89028,,,,
6021,89028,ar,الإلكترونيات والأجهزة المنزلية,البطاريات ومولدات الطاقة,بطاريات,بطاريات الكالاين,ENERGIZER BATTERY AAX8 MAX,,"{""Number of batteries included"": ""8"", ""طاقة البطارية"": ""1.5""}",41.5,,https://www.carrefouruae.com//mafuae/ar/alkaline-batteries/energizer-max-alke91bp-batery-8-aa/p/89028,,,,
6021,742553,en,Electronics & Appliances,Batteries & Power,Batteries,Alkaline Batteries,Energizer Max AAA Alkaline Battery EP2BP Pack of 8,,"{""Number of batteries included"": ""8"", ""Battery voltage"": ""1.5""}",33.0,,https://www.carrefouruae.com//mafuae/en/alkaline-batteries/energizer-max-alke92bp-batery-8-aaa/p/742553,,,,
6021,742553,ar,الإلكترونيات والأجهزة المنزلية,البطاريات ومولدات الطاقة,بطاريات,بطاريات الكالاين,ENERGIZER AAA/8 MAX ALKALINE,,"{""Number of batteries included"": ""8"", ""طاقة البطارية"": ""1.5""}",33.0,,https://www.carrefouruae.com//mafuae/ar/alkaline-batteries/energizer-max-alke92bp-batery-8-aaa/p/742553,,,,
6021,893379,en,Electronics & Appliances,Large Appliances,Fridges,Fridge 101L to 200L,First1 Free Standing 2 shelves Fridge FR-130L,,"{""Dimensions WxDxH"": ""W 49cm x D 45cm x H 83.5cm""}",549.0,,https://www.carrefouruae.com//mafuae/en/fridge-101l-to-200l/mychoice-first1-fridge-fr-130l-94l/p/893379,,Free delivery,,1.0
6021,893379,ar,الإلكترونيات والأجهزة المنزلية,أجهزة منزلية كبيرة,الثلاجات,ثلاجة 101L إلى 200L,FIRST1 FR-130L FRIDGE,,"{""Dimensions WxDxH"": ""W 49cm x D 45cm x H 83.5cm""}",549.0,,https://www.carrefouruae.com//mafuae/ar/fridge-101l-to-200l/mychoice-first1-fridge-fr-130l-94l/p/893379,,,,1.0
6021,915909,en,Electronics & Appliances,Batteries & Power,Power Adaptors & Sockets,Adaptors & Sockets,Elexon Universal Travel Adaptor PP7974A White,,{},14.5,,https://www.carrefouruae.com//mafuae/en/adaptors-sockets/elexon-universal-travel-adaptor/p/915909,,,,
6021,945671,en,Electronics & Appliances,Small Appliances,Food Preparation,Rice Cooker,Mychoice Rice Cooker 350W F-06RC White,,{},69.0,,https://www.carrefouruae.com//mafuae/en/rice-cooker/mychoice-first1-rice-cooker-f-06rc/p/945671,,,,
6021,945671,ar,الإلكترونيات والأجهزة المنزلية,أجهزة منزلية صغيرة,تحضير الطعام,طبخ الأرز,جهاز طهي الأرز الأول F-06RC 350 وات 0.6 لتر,,{},69.0,,https://www.carrefouruae.com//mafuae/ar/rice-cooker/mychoice-first1-rice-cooker-f-06rc/p/945671,,,,
one row for English and one for Arabic and I’d like to setup some check like this:
if catalog_uuid contains en and ar:
payload = {
"row": {
"LabebStoreId": d["LabebStoreId"],
"catalog_uuid": d["catalog_uuid"],
"lang": d["lang"],
"cat_0_name": d["cat_0_name"],
"cat_1_name": d["cat_1_name"],
"cat_2_name": d["cat_2_name"],
"cat_3_name": d["cat_3_name"],
"catalogname": d["catalogname"],
"description": d["description"],
"properties": d["properties"],
"price": d["price"],
"price_before_discount": d["price_before_discount"],
"externallink": d["externallink"],
"Rating": d["Rating"],
"delivery": d["delivery"],
"discount": d["discount"],
"instock": d["instock"],
}
}
payload = {
"nextRow": {
"LabebStoreId": d["LabebStoreId"],
"catalog_uuid": d["catalog_uuid"],
"lang": d["lang"],
"cat_0_name": d["cat_0_name"],
"cat_1_name": d["cat_1_name"],
"cat_2_name": d["cat_2_name"],
"cat_3_name": d["cat_3_name"],
"catalogname": d["catalogname"],
"description": d["description"],
"properties": d["properties"],
"price": d["price"],
"price_before_discount": d["price_before_discount"],
"externallink": d["externallink"],
"Rating": d["Rating"],
"delivery": d["delivery"],
"discount": d["discount"],
"instock": d["instock"],
}
}
elif catalog_uuid contains only en:
payload = {
"row": {
"LabebStoreId": d["LabebStoreId"],
"catalog_uuid": d["catalog_uuid"],
"lang": d["lang"],
"cat_0_name": d["cat_0_name"],
"cat_1_name": d["cat_1_name"],
"cat_2_name": d["cat_2_name"],
"cat_3_name": d["cat_3_name"],
"catalogname": d["catalogname"],
"description": d["description"],
"properties": d["properties"],
"price": d["price"],
"price_before_discount": d["price_before_discount"],
"externallink": d["externallink"],
"Rating": d["Rating"],
"delivery": d["delivery"],
"discount": d["discount"],
"instock": d["instock"],
}
}
elif catalog_uuid contains only ar:
payload = {
"row": {
"LabebStoreId": d["LabebStoreId"],
"catalog_uuid": d["catalog_uuid"],
"lang": d["lang"],
"cat_0_name": d["cat_0_name"],
"cat_1_name": d["cat_1_name"],
"cat_2_name": d["cat_2_name"],
"cat_3_name": d["cat_3_name"],
"catalogname": d["catalogname"],
"description": d["description"],
"properties": d["properties"],
"price": d["price"],
"price_before_discount": d["price_before_discount"],
"externallink": d["externallink"],
"Rating": d["Rating"],
"delivery": d["delivery"],
"discount": d["discount"],
"instock": d["instock"],
}
}
some rows only contains English and some only contains Arabic so I have to cover both aspects. I am using pandas to read the csv and transforming the dataframe to pd.to_json(orient="index") because I need to post that rows in a POST request as a json.
Here is my recent try:
import pandas as pd
import requests
import json
import ast
HEADERS = {"Accept": "*/*", "Content-Type": "application/json"}
df = pd.read_csv("carrefour-uae-items-final.csv").fillna("").astype(str)
dict_data = json.loads(df.to_json(orient="index"))
list_dict = [d for d in dict_data.values()]
for d in list_dict:
cat_id = d["catalog_uuid"]
if next(item for item in list_dict if item["catalog_uuid"] == cat_id):
payload_rows = ["row", "nextRow"]
for row in payload_rows:
payload = {
row: {
"LabebStoreId": d["LabebStoreId"],
"catalog_uuid": d["catalog_uuid"],
"lang": d["lang"],
"cat_0_name": d["cat_0_name"],
"cat_1_name": d["cat_1_name"],
"cat_2_name": d["cat_2_name"],
"cat_3_name": d["cat_3_name"],
"catalogname": d["catalogname"],
"description": d["description"],
"properties": d["properties"],
"price": d["price"],
"price_before_discount": d["price_before_discount"],
"externallink": d["externallink"],
"Rating": d["Rating"],
"delivery": d["delivery"],
"discount": d["discount"],
"instock": d["instock"],
"images": ast.literal_eval(d["encoded_images"]),
}
}
print(json.dumps(payload, indent=4))
response = requests.post(
"http://crawlerapi.labeb.com/api/PCCrawler/Crawl?StoreId=6021",
headers=HEADERS,
json=payload,
)
print(response.content.decode())
else:
payload = {
"row": {
"LabebStoreId": d["LabebStoreId"],
"catalog_uuid": d["catalog_uuid"],
"lang": d["lang"],
"cat_0_name": d["cat_0_name"],
"cat_1_name": d["cat_1_name"],
"cat_2_name": d["cat_2_name"],
"cat_3_name": d["cat_3_name"],
"catalogname": d["catalogname"],
"description": d["description"],
"properties": d["properties"],
"price": d["price"],
"price_before_discount": d["price_before_discount"],
"externallink": d["externallink"],
"Rating": d["Rating"],
"delivery": d["delivery"],
"discount": d["discount"],
"instock": d["instock"],
"images": ast.literal_eval(d["encoded_images"]),
}
}
print(json.dumps(payload, indent=4))
response = requests.post(
"http://crawlerapi.labeb.com/api/PCCrawler/Crawl?StoreId=6021",
headers=HEADERS,
json=payload,
)
print(response.content.decode())
Can anyone please help me figure out how to achieve the above logic? Thanks in advance!
As I understand correctly, you can group the data by LabebStoreId and catalog_uuid and then you make payloads according each group:
import csv
data = {}
with open("data.csv", "r") as f_in:
reader = csv.DictReader(f_in)
for row in reader:
data.setdefault((row["LabebStoreId"], row["catalog_uuid"]), []).append(
row
)
for payload_no, v in enumerate(data.values(), 1):
print("Payload number", payload_no)
for d in v:
print("Language", d["lang"])
print("t", d)
print("-" * 80)
Prints:
Payload number 1
Language en
{'LabebStoreId': '6021', 'catalog_uuid': '89028', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Batteries & Power', 'cat_2_name': 'Batteries', 'cat_3_name': 'Alkaline Batteries', 'catalogname': 'Energizer Max AA Alkaline Battery E91BP Pack of 8', 'description': '', 'properties': '{"Number of batteries included": "8", "Battery voltage": "1.5"}', 'price': '41.5', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/alkaline-batteries/energizer-max-alke91bp-batery-8-aa/p/89028', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}
Language ar
{'LabebStoreId': '6021', 'catalog_uuid': '89028', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'البطاريات ومولدات الطاقة', 'cat_2_name': 'بطاريات', 'cat_3_name': 'بطاريات الكالاين', 'catalogname': 'ENERGIZER BATTERY AAX8 MAX', 'description': '', 'properties': '{"Number of batteries included": "8", "طاقة البطارية": "1.5"}', 'price': '41.5', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/alkaline-batteries/energizer-max-alke91bp-batery-8-aa/p/89028', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}
--------------------------------------------------------------------------------
Payload number 2
Language en
{'LabebStoreId': '6021', 'catalog_uuid': '742553', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Batteries & Power', 'cat_2_name': 'Batteries', 'cat_3_name': 'Alkaline Batteries', 'catalogname': 'Energizer Max AAA Alkaline Battery EP2BP Pack of 8', 'description': '', 'properties': '{"Number of batteries included": "8", "Battery voltage": "1.5"}', 'price': '33.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/alkaline-batteries/energizer-max-alke92bp-batery-8-aaa/p/742553', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}
Language ar
{'LabebStoreId': '6021', 'catalog_uuid': '742553', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'البطاريات ومولدات الطاقة', 'cat_2_name': 'بطاريات', 'cat_3_name': 'بطاريات الكالاين', 'catalogname': 'ENERGIZER AAA/8 MAX ALKALINE', 'description': '', 'properties': '{"Number of batteries included": "8", "طاقة البطارية": "1.5"}', 'price': '33.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/alkaline-batteries/energizer-max-alke92bp-batery-8-aaa/p/742553', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}
--------------------------------------------------------------------------------
Payload number 3
Language en
{'LabebStoreId': '6021', 'catalog_uuid': '893379', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Large Appliances', 'cat_2_name': 'Fridges', 'cat_3_name': 'Fridge 101L to 200L', 'catalogname': 'First1 Free Standing 2 shelves Fridge FR-130L', 'description': '', 'properties': '{"Dimensions WxDxH": "W 49cm x D 45cm x H 83.5cm"}', 'price': '549.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/fridge-101l-to-200l/mychoice-first1-fridge-fr-130l-94l/p/893379', 'Rating': '', 'delivery': 'Free delivery', 'discount': '', 'instock': '1.0'}
Language ar
{'LabebStoreId': '6021', 'catalog_uuid': '893379', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'أجهزة منزلية كبيرة', 'cat_2_name': 'الثلاجات', 'cat_3_name': 'ثلاجة 101L إلى 200L', 'catalogname': 'FIRST1 FR-130L FRIDGE', 'description': '', 'properties': '{"Dimensions WxDxH": "W 49cm x D 45cm x H 83.5cm"}', 'price': '549.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/fridge-101l-to-200l/mychoice-first1-fridge-fr-130l-94l/p/893379', 'Rating': '', 'delivery': '', 'discount': '', 'instock': '1.0'}
--------------------------------------------------------------------------------
Payload number 4
Language en
{'LabebStoreId': '6021', 'catalog_uuid': '915909', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Batteries & Power', 'cat_2_name': 'Power Adaptors & Sockets', 'cat_3_name': 'Adaptors & Sockets', 'catalogname': 'Elexon Universal Travel Adaptor PP7974A White', 'description': '', 'properties': '{}', 'price': '14.5', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/adaptors-sockets/elexon-universal-travel-adaptor/p/915909', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}
--------------------------------------------------------------------------------
Payload number 5
Language en
{'LabebStoreId': '6021', 'catalog_uuid': '945671', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Small Appliances', 'cat_2_name': 'Food Preparation', 'cat_3_name': 'Rice Cooker', 'catalogname': 'Mychoice Rice Cooker 350W F-06RC White', 'description': '', 'properties': '{}', 'price': '69.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/rice-cooker/mychoice-first1-rice-cooker-f-06rc/p/945671', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}
Language ar
{'LabebStoreId': '6021', 'catalog_uuid': '945671', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'أجهزة منزلية صغيرة', 'cat_2_name': 'تحضير الطعام', 'cat_3_name': 'طبخ الأرز', 'catalogname': 'جهاز طهي الأرز الأول F-06RC 350 وات 0.6 لتر', 'description': '', 'properties': '{}', 'price': '69.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/rice-cooker/mychoice-first1-rice-cooker-f-06rc/p/945671', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}
--------------------------------------------------------------------------------
To have {'row':...} or {'row':..., 'nextRow':...} you can check lenght of the group:
import csv
data = {}
with open("data.csv", "r") as f_in:
reader = csv.DictReader(f_in)
for row in reader:
data.setdefault((row["LabebStoreId"], row["catalog_uuid"]), []).append(
row
)
for payload_no, v in enumerate(data.values(), 1):
print("Payload number", payload_no)
if len(v) == 1:
payload = {"row": v[0]}
else:
payload = {"row": v[0], "nextRow": v[1]}
print("t", payload)
print("-" * 80)
Prints:
Payload number 1
{'row': {'LabebStoreId': '6021', 'catalog_uuid': '89028', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Batteries & Power', 'cat_2_name': 'Batteries', 'cat_3_name': 'Alkaline Batteries', 'catalogname': 'Energizer Max AA Alkaline Battery E91BP Pack of 8', 'description': '', 'properties': '{"Number of batteries included": "8", "Battery voltage": "1.5"}', 'price': '41.5', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/alkaline-batteries/energizer-max-alke91bp-batery-8-aa/p/89028', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}, 'nextRow': {'LabebStoreId': '6021', 'catalog_uuid': '89028', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'البطاريات ومولدات الطاقة', 'cat_2_name': 'بطاريات', 'cat_3_name': 'بطاريات الكالاين', 'catalogname': 'ENERGIZER BATTERY AAX8 MAX', 'description': '', 'properties': '{"Number of batteries included": "8", "طاقة البطارية": "1.5"}', 'price': '41.5', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/alkaline-batteries/energizer-max-alke91bp-batery-8-aa/p/89028', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}}
--------------------------------------------------------------------------------
Payload number 2
{'row': {'LabebStoreId': '6021', 'catalog_uuid': '742553', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Batteries & Power', 'cat_2_name': 'Batteries', 'cat_3_name': 'Alkaline Batteries', 'catalogname': 'Energizer Max AAA Alkaline Battery EP2BP Pack of 8', 'description': '', 'properties': '{"Number of batteries included": "8", "Battery voltage": "1.5"}', 'price': '33.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/alkaline-batteries/energizer-max-alke92bp-batery-8-aaa/p/742553', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}, 'nextRow': {'LabebStoreId': '6021', 'catalog_uuid': '742553', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'البطاريات ومولدات الطاقة', 'cat_2_name': 'بطاريات', 'cat_3_name': 'بطاريات الكالاين', 'catalogname': 'ENERGIZER AAA/8 MAX ALKALINE', 'description': '', 'properties': '{"Number of batteries included": "8", "طاقة البطارية": "1.5"}', 'price': '33.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/alkaline-batteries/energizer-max-alke92bp-batery-8-aaa/p/742553', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}}
--------------------------------------------------------------------------------
Payload number 3
{'row': {'LabebStoreId': '6021', 'catalog_uuid': '893379', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Large Appliances', 'cat_2_name': 'Fridges', 'cat_3_name': 'Fridge 101L to 200L', 'catalogname': 'First1 Free Standing 2 shelves Fridge FR-130L', 'description': '', 'properties': '{"Dimensions WxDxH": "W 49cm x D 45cm x H 83.5cm"}', 'price': '549.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/fridge-101l-to-200l/mychoice-first1-fridge-fr-130l-94l/p/893379', 'Rating': '', 'delivery': 'Free delivery', 'discount': '', 'instock': '1.0'}, 'nextRow': {'LabebStoreId': '6021', 'catalog_uuid': '893379', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'أجهزة منزلية كبيرة', 'cat_2_name': 'الثلاجات', 'cat_3_name': 'ثلاجة 101L إلى 200L', 'catalogname': 'FIRST1 FR-130L FRIDGE', 'description': '', 'properties': '{"Dimensions WxDxH": "W 49cm x D 45cm x H 83.5cm"}', 'price': '549.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/fridge-101l-to-200l/mychoice-first1-fridge-fr-130l-94l/p/893379', 'Rating': '', 'delivery': '', 'discount': '', 'instock': '1.0'}}
--------------------------------------------------------------------------------
Payload number 4
{'row': {'LabebStoreId': '6021', 'catalog_uuid': '915909', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Batteries & Power', 'cat_2_name': 'Power Adaptors & Sockets', 'cat_3_name': 'Adaptors & Sockets', 'catalogname': 'Elexon Universal Travel Adaptor PP7974A White', 'description': '', 'properties': '{}', 'price': '14.5', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/adaptors-sockets/elexon-universal-travel-adaptor/p/915909', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}}
--------------------------------------------------------------------------------
Payload number 5
{'row': {'LabebStoreId': '6021', 'catalog_uuid': '945671', 'lang': 'en', 'cat_0_name': 'Electronics & Appliances', 'cat_1_name': 'Small Appliances', 'cat_2_name': 'Food Preparation', 'cat_3_name': 'Rice Cooker', 'catalogname': 'Mychoice Rice Cooker 350W F-06RC White', 'description': '', 'properties': '{}', 'price': '69.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/en/rice-cooker/mychoice-first1-rice-cooker-f-06rc/p/945671', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}, 'nextRow': {'LabebStoreId': '6021', 'catalog_uuid': '945671', 'lang': 'ar', 'cat_0_name': 'الإلكترونيات والأجهزة المنزلية', 'cat_1_name': 'أجهزة منزلية صغيرة', 'cat_2_name': 'تحضير الطعام', 'cat_3_name': 'طبخ الأرز', 'catalogname': 'جهاز طهي الأرز الأول F-06RC 350 وات 0.6 لتر', 'description': '', 'properties': '{}', 'price': '69.0', 'price_before_discount': '', 'externallink': 'https://www.carrefouruae.com//mafuae/ar/rice-cooker/mychoice-first1-rice-cooker-f-06rc/p/945671', 'Rating': '', 'delivery': '', 'discount': '', 'instock': ''}}
--------------------------------------------------------------------------------