Compare two JSON results and make new JSON of new entries (Python)
Question:
I’m querying an API to try to find new entries in the JSON, I am stuck on how to "Substract" the two lists so that I get a new JSON object that has ONLY the NEW ITEMS between the two lists.
What I’m doing now is:
- I’m querying the API.
- Storing JSON results into a variable.
- Wait 30 seconds.
- Query the API again.
- Compare if PreviousJSON == NewJSON
- If there’s a difference THEN:
^^ this is where I’m stuck, I can compare values but I don’t know how to find only NEW values and iterate over them.
I’m basically monitoring this API:
https://qzlsklfacc.medianetwork.cloud/nft_for_sale?collection=galacticgeckospacegarage
And trying to find new listings and new sold items.
Right now to compare differences between existing items I’m doing the following:
# check if new listings are the same as the old listings
if prevListings != newListings:
await bot.get_channel(893292122298540083).send("New listings found!")
for listingOld in prevListings:
for listingNew in newListings:
if listingNew['id'] == listingOld['id']:
if listingNew['lastSoldPrice'] is not None:
if listingNew['lastSoldPrice'] != listingOld['lastSoldPrice']:
Do something.
So the two questions are:
- Is there a better way to compare two identical entries in two JSON objects (Not sorted)
- Is there a way to get a 3rd JSON containing only new entries that do not exist in PrevListings.
This is the JSON Structure of each item in the JSON response
0000:
'id':1155682
'token_add':'HV4duJjY67DD4doX9ffvzfoaENAAkFuwz9qtbwAweDAG'
'number':1
'currency':'SOL'
'price':29
'link_img':'https://www.arweave.net/iZs-LiMAg5mIqaqxrd--EtcKhHTNtPZFPrZyIK95nUc?ext=jpeg'
'for_sale':1
'programId':'CJsLwbP1iu5DuUikHEJnLfANgKy6stB2uFgvBBHoyxwz'
'name':'Galactic Gecko #6376'
'description':'undefined'
'escrowAdd':'4TULtxsixV4NLdZWqY45d4Mz5XrE4q4CfTECNcvaZDY1'
'seller_address':'Er6QJPusC1JsUqevTjFKXtYHbgCtJkyo1DNjEBWevWut'
'attributes':'Faction: Barada,Body: Light Pink,Armor: Planetary Kirtle,Mouth: Pensive,Helmet: Signal Deflector,Ears: Planetary Muffler,Eyes: Augmented Surveyor'
'skin':None
'type':'galacticgeckospacegarage'
'ranking':None
'buyerAdd':None
'blockhash':None
'lastSoldPrice':13.99
I’m using Python 3.8
Thank you!
Answers:
The approach you take depends on whether the top-level keys (e.g. 0000) are unique to the individual objects or not. If they are, you can use these keys to filter the returned object, if not you need to do a little more work.
top level key is ID
If the top level key is unique to the object, you can simply iterate the new dictionary and see if its key is in the existing one. In the code below first is the initial dictionary, and second is our second response. The output is stored in the dictionary third.
third = {} # empty dictionary for our output.
for key, data in second.items():
if key not in first: # is new
third[key] = data
So we iterate over the second dictionary, checking if each key is in the first dictionary. If it isn’t we add it to the third.
You can also do this using dictionary comprehension (same result).
third = {key:value for key, value in second.items() if key not in first}
id is in data
If the keys aren’t IDs then you have a problem: you need to filter the second dictionary by something nested in the data of the first. We could iterate the first dictionary for each entry in second, but that’s slow.
Since we only want to know if it’s in there, we can extract the value we need to check (e.g. id) into a new variable. A set is good for this, since lookups are fast.
# build a set of values from the first dictionary.
first_ids = set(o['id'] for o in first.values())
third = {}
for key, data in second.items():
if data['id'] not in first_ids:
third[key] = data
Or, again with the dict comprehension
first_ids = set(o['id'] for o in first.values())
third = {key:value for key, value in second.items() if value['id'] not in first_ids}
As you can see, we iterate over the first dictionary once to build our set of ids. This can then be used to quickly test if new data received in the second dictionary is new. If it is, we add it to the dictionary third.
Try the below (data_1 and data_2 represent the results of 2 API calls)
data_1 = [{'lastSoldPrice':12,'id':14},{'lastSoldPrice':18,'id':15},{'lastSoldPrice':149,'id':146}]
data_2 = [{'lastSoldPrice':12,'id':17},{'lastSoldPrice':18,'id':15},{'lastSoldPrice':142,'id':146},{'lastSoldPrice':1422,'id':166}]
# convert list to dict
data_1 = {entry.get('id'): entry for entry in data_1}
data_2 = {entry.get('id'): entry for entry in data_2}
# set operation to find new and common
new_entries = data_2.keys() - data_1.keys()
print(f'New entry ids: {new_entries}')
for _id in new_entries:
print(f't {data_2.get(_id)}')
common_entries = data_2.keys() & (data_1.keys())
print(f'Common entry ids: {common_entries}')
# loop over common entries and see if there is a price change
for _id in common_entries:
p1 = data_1.get(_id)['lastSoldPrice']
p2= data_2.get(_id)['lastSoldPrice']
if p1 != p2:
print(f'Price change for id {_id}. Old price: {p1}, new price: {p2}')
output
New entry ids: {17, 166}
{'lastSoldPrice': 12, 'id': 17}
{'lastSoldPrice': 1422, 'id': 166}
Common entry ids: {146, 15}
Price change for id 146. Old price: 149, new price: 142
Here is a pythonic solution:
old = {'a': 1, 'b': 2, 'c': 8}
new = {'a': 2, 'b': 2, 'c': -5}
old = {k: v for k, v in old.items() if not isinstance(v, (list, dict))}
new = {k: v for k, v in new.items() if not isinstance(v, (list, dict))}
new_items = new_items = {k: v for (k, v) in set(new.items()) - set(old.items())}
old_items = {k: old.get(k) for k, v in new_items.items()}
res = [{'field': k,
'old_value': old_items[k],
'new_value': new_items[k]} for k in new_items]
return res
Returns
>>> res
[{'field': 'c', 'old_value': 8, 'new_value': -5},
{'field': 'a', 'old_value': 1, 'new_value': 2}]
I’m querying an API to try to find new entries in the JSON, I am stuck on how to "Substract" the two lists so that I get a new JSON object that has ONLY the NEW ITEMS between the two lists.
What I’m doing now is:
- I’m querying the API.
- Storing JSON results into a variable.
- Wait 30 seconds.
- Query the API again.
- Compare if PreviousJSON == NewJSON
- If there’s a difference THEN:
^^ this is where I’m stuck, I can compare values but I don’t know how to find only NEW values and iterate over them.
I’m basically monitoring this API:
https://qzlsklfacc.medianetwork.cloud/nft_for_sale?collection=galacticgeckospacegarage
And trying to find new listings and new sold items.
Right now to compare differences between existing items I’m doing the following:
# check if new listings are the same as the old listings
if prevListings != newListings:
await bot.get_channel(893292122298540083).send("New listings found!")
for listingOld in prevListings:
for listingNew in newListings:
if listingNew['id'] == listingOld['id']:
if listingNew['lastSoldPrice'] is not None:
if listingNew['lastSoldPrice'] != listingOld['lastSoldPrice']:
Do something.
So the two questions are:
- Is there a better way to compare two identical entries in two JSON objects (Not sorted)
- Is there a way to get a 3rd JSON containing only new entries that do not exist in PrevListings.
This is the JSON Structure of each item in the JSON response
0000:
'id':1155682
'token_add':'HV4duJjY67DD4doX9ffvzfoaENAAkFuwz9qtbwAweDAG'
'number':1
'currency':'SOL'
'price':29
'link_img':'https://www.arweave.net/iZs-LiMAg5mIqaqxrd--EtcKhHTNtPZFPrZyIK95nUc?ext=jpeg'
'for_sale':1
'programId':'CJsLwbP1iu5DuUikHEJnLfANgKy6stB2uFgvBBHoyxwz'
'name':'Galactic Gecko #6376'
'description':'undefined'
'escrowAdd':'4TULtxsixV4NLdZWqY45d4Mz5XrE4q4CfTECNcvaZDY1'
'seller_address':'Er6QJPusC1JsUqevTjFKXtYHbgCtJkyo1DNjEBWevWut'
'attributes':'Faction: Barada,Body: Light Pink,Armor: Planetary Kirtle,Mouth: Pensive,Helmet: Signal Deflector,Ears: Planetary Muffler,Eyes: Augmented Surveyor'
'skin':None
'type':'galacticgeckospacegarage'
'ranking':None
'buyerAdd':None
'blockhash':None
'lastSoldPrice':13.99
I’m using Python 3.8
Thank you!
The approach you take depends on whether the top-level keys (e.g. 0000) are unique to the individual objects or not. If they are, you can use these keys to filter the returned object, if not you need to do a little more work.
top level key is ID
If the top level key is unique to the object, you can simply iterate the new dictionary and see if its key is in the existing one. In the code below first is the initial dictionary, and second is our second response. The output is stored in the dictionary third.
third = {} # empty dictionary for our output.
for key, data in second.items():
if key not in first: # is new
third[key] = data
So we iterate over the second dictionary, checking if each key is in the first dictionary. If it isn’t we add it to the third.
You can also do this using dictionary comprehension (same result).
third = {key:value for key, value in second.items() if key not in first}
id is in data
If the keys aren’t IDs then you have a problem: you need to filter the second dictionary by something nested in the data of the first. We could iterate the first dictionary for each entry in second, but that’s slow.
Since we only want to know if it’s in there, we can extract the value we need to check (e.g. id) into a new variable. A set is good for this, since lookups are fast.
# build a set of values from the first dictionary.
first_ids = set(o['id'] for o in first.values())
third = {}
for key, data in second.items():
if data['id'] not in first_ids:
third[key] = data
Or, again with the dict comprehension
first_ids = set(o['id'] for o in first.values())
third = {key:value for key, value in second.items() if value['id'] not in first_ids}
As you can see, we iterate over the first dictionary once to build our set of ids. This can then be used to quickly test if new data received in the second dictionary is new. If it is, we add it to the dictionary third.
Try the below (data_1 and data_2 represent the results of 2 API calls)
data_1 = [{'lastSoldPrice':12,'id':14},{'lastSoldPrice':18,'id':15},{'lastSoldPrice':149,'id':146}]
data_2 = [{'lastSoldPrice':12,'id':17},{'lastSoldPrice':18,'id':15},{'lastSoldPrice':142,'id':146},{'lastSoldPrice':1422,'id':166}]
# convert list to dict
data_1 = {entry.get('id'): entry for entry in data_1}
data_2 = {entry.get('id'): entry for entry in data_2}
# set operation to find new and common
new_entries = data_2.keys() - data_1.keys()
print(f'New entry ids: {new_entries}')
for _id in new_entries:
print(f't {data_2.get(_id)}')
common_entries = data_2.keys() & (data_1.keys())
print(f'Common entry ids: {common_entries}')
# loop over common entries and see if there is a price change
for _id in common_entries:
p1 = data_1.get(_id)['lastSoldPrice']
p2= data_2.get(_id)['lastSoldPrice']
if p1 != p2:
print(f'Price change for id {_id}. Old price: {p1}, new price: {p2}')
output
New entry ids: {17, 166}
{'lastSoldPrice': 12, 'id': 17}
{'lastSoldPrice': 1422, 'id': 166}
Common entry ids: {146, 15}
Price change for id 146. Old price: 149, new price: 142
Here is a pythonic solution:
old = {'a': 1, 'b': 2, 'c': 8}
new = {'a': 2, 'b': 2, 'c': -5}
old = {k: v for k, v in old.items() if not isinstance(v, (list, dict))}
new = {k: v for k, v in new.items() if not isinstance(v, (list, dict))}
new_items = new_items = {k: v for (k, v) in set(new.items()) - set(old.items())}
old_items = {k: old.get(k) for k, v in new_items.items()}
res = [{'field': k,
'old_value': old_items[k],
'new_value': new_items[k]} for k in new_items]
return res
Returns
>>> res
[{'field': 'c', 'old_value': 8, 'new_value': -5},
{'field': 'a', 'old_value': 1, 'new_value': 2}]