How can i remove the duplicate elements in my list of dictionnary

Question

I have a list with 5 dictionaries and I would like to filter them with the id.
If they have the same id, I delete it.
I tried something like this but it doesn’t work

res = []

for y in self.dict_trie_csv_infos:
    #check if  same 'event_id' before
    if float(y['event_id']) not in res:
        res.append(y)

The id of the dicts are "1", "2", "3", "4", "4".

Only the id is different and i have to delete it.
I tried to do a double loop too but it did nothing.

When I do a print(len(self.dict_trie_csv_infos) it gives me 5
and same when I print my res.

There is the print of my first element:

{'event_id': 1,
 'collection_name': 'Mouse On',
 'smart_contract': {'multisig': 'KT1Aer6TxNwoMJejoqsNP8TEN7J6STgMtJcA',
 'sale_params': {
 'is_presale': False,
 'metadata_list': [],
 'price_per_token': 4,
 'max_mint_per_user': 5,
 'sale_size': 500,
 'sale_currency': {'xtz': None},
 'start_time': 1656626400,
 'end_time': 1657490400,
 }}}

Asked By: guiguilecodeur

||

Source

Answer 1

Here’s a way to do what your question asks:

class Foo:
    def __init__(self):
        dict_trie_csv_infos = []
    def remove_duplicate_ids(self):
        res = []
        ids = set()
        for y in self.dict_trie_csv_infos:
            if y['event_id'] not in ids:
                ids.add(y['event_id'])
                res.append(y)
        return res


f = Foo()
f.dict_trie_csv_infos = [
    {'event_id': 1, 'collection_name': 'Mouse On', 'smart_contract': {'multisig': 'KT1Aer6TxNwoMJejoqsNP8TEN7J6STgMtJcA', 'sale_params': {'is_presale': False, 'metadata_list': [], 'price_per_token': 4, 'max_mint_per_user': 5, 'sale_size': 500, 'sale_currency': {'xtz': None}, 'start_time': 1656626400, 'end_time': 1657490400}}},
    {'event_id': 2, 'collection_name': 'Mouse On', 'smart_contract': {'multisig': 'KT1Aer6TxNwoMJejoqsNP8TEN7J6STgMtJcA', 'sale_params': {'is_presale': False, 'metadata_list': [], 'price_per_token': 4, 'max_mint_per_user': 5, 'sale_size': 500, 'sale_currency': {'xtz': None}, 'start_time': 1656626400, 'end_time': 1657490400}}},
    {'event_id': 3, 'collection_name': 'Mouse On', 'smart_contract': {'multisig': 'KT1Aer6TxNwoMJejoqsNP8TEN7J6STgMtJcA', 'sale_params': {'is_presale': False, 'metadata_list': [], 'price_per_token': 4, 'max_mint_per_user': 5, 'sale_size': 500, 'sale_currency': {'xtz': None}, 'start_time': 1656626400, 'end_time': 1657490400}}},
    {'event_id': 4, 'collection_name': 'Mouse On', 'smart_contract': {'multisig': 'KT1Aer6TxNwoMJejoqsNP8TEN7J6STgMtJcA', 'sale_params': {'is_presale': False, 'metadata_list': [], 'price_per_token': 4, 'max_mint_per_user': 5, 'sale_size': 500, 'sale_currency': {'xtz': None}, 'start_time': 1656626400, 'end_time': 1657490400}}},
    {'event_id': 4, 'collection_name': 'Mouse On', 'smart_contract': {'multisig': 'KT1Aer6TxNwoMJejoqsNP8TEN7J6STgMtJcA', 'sale_params': {'is_presale': False, 'metadata_list': [], 'price_per_token': 4, 'max_mint_per_user': 5, 'sale_size': 500, 'sale_currency': {'xtz': None}, 'start_time': 1656626400, 'end_time': 1657490400}}}
]

res = f.remove_duplicate_ids()
[print(elem, 'n') for elem in res]

Output:

{'event_id': 1, 'collection_name': 'Mouse On', 'smart_contract': {'multisig': 'KT1Aer6TxNwoMJejoqsNP8TEN7J6STgMtJcA', 'sale_params': {'is_presale': False, 'metadata_list': [], 'price_per_token': 4, 'max_mint_per_user': 5, 'sale_size': 500, 'sale_currency': {'xtz': None}, 'start_time': 1656626400, 'end_time': 1657490400}}}

{'event_id': 2, 'collection_name': 'Mouse On', 'smart_contract': {'multisig': 'KT1Aer6TxNwoMJejoqsNP8TEN7J6STgMtJcA', 'sale_params': {'is_presale': False, 'metadata_list': [], 'price_per_token': 4, 'max_mint_per_user': 5, 'sale_size': 500, 'sale_currency': {'xtz': None}, 'start_time': 1656626400, 'end_time': 1657490400}}}

{'event_id': 3, 'collection_name': 'Mouse On', 'smart_contract': {'multisig': 'KT1Aer6TxNwoMJejoqsNP8TEN7J6STgMtJcA', 'sale_params': {'is_presale': False, 'metadata_list': [], 'price_per_token': 4, 'max_mint_per_user': 5, 'sale_size': 500, 'sale_currency': {'xtz': None}, 'start_time': 1656626400, 'end_time': 1657490400}}}

{'event_id': 4, 'collection_name': 'Mouse On', 'smart_contract': {'multisig': 'KT1Aer6TxNwoMJejoqsNP8TEN7J6STgMtJcA', 'sale_params': {'is_presale': False, 'metadata_list': [], 'price_per_token': 4, 'max_mint_per_user': 5, 'sale_size': 500, 'sale_currency': {'xtz': None}, 'start_time': 1656626400, 'end_time': 1657490400}}}

Explanation:

For testing, create a simple class Foo with instance attribute dict_trie_csv_infos and instance method remove_duplicate_ids
Create a variable f of type Foo and assign a list with dict elements like the example in the question, with values for event_id of 1, 2, 3, 4, and 4
Call f.remove_duplicate_ids()
In remove_duplicate_ids(), which contains a modified version of the code in your question, use a set named ids to detect duplicates, and build result variable res to contain rows with unique event_id values

Answered By: constantstranger

Answer 2

It looks like every object in the first list is being added to the second list. This is because float(y['event_id']) not in res is always evaluating to True.

You might want to try something like

res = []

for y in self.dict_trie_csv_infos:

    # check if  same 'event_id' before
    found = False
    for x in res:
      if y['event_id'] == x['event_id']:
        found = True
        break
    if(not found)
      res.append(y)

Answered By: Alex Oakley

How can i remove the duplicate elements in my list of dictionnary

Question:

Answers: