deleting repeating value in nested dictionary

You can remove duplicates and None values from the dictionary like this

for k,v in d['C4QY10_e'].items():
    d['C4QY10_e'][k] = list(set(filter(None, v)))

Output

{'C4QY10_e': {'protein accession': ['C4QY10_e'],
  'sequence length': ['1879'],
  'analysis': ['Pfam'],
  'signature accession': ['PF18325',
   'PF18314',
   'PF00109',
   'PF01648',
   'PF02801'],
  'signature description': ['Beta-ketoacyl synthase',
   'Fatty acid synthase type I helical domain',
   'Fatty acid synthase subunit alpha Acyl carrier domain',
   "4'-phosphopantetheinyl transferase superfamily"],
  'start location': ['1761', '139', '328'],
  'stop location': ['300', '528', '1861'],
  'e-value': ['1.3E-72', '4.7E-73', '1.4E-18'],
  'interpro accession': ['IPR008278', 'IPR041550', 'IPR040899'],
  'interpro description': ['Fatty acid synthase type I',
   "4'-phosphopantetheinyl transferase domain",
   'Fatty acid synthase subunit alpha'],
  'nunique': [1],
  'domain_count': [5]}}

There are no keys that are repeated, keys are repeated at different levels.

{'C4QY10_e': {'analysis': ['Pfam', 'Pfam', 'Pfam', 'Pfam', 'Pfam'],
              'domain_count': [5, 5, 5, 5, 5],
              'e-value': ['4.7E-73', None, '1.3E-72', None, '1.4E-18'],
              'interpro accession': ['IPR041550',
                                     None,
                                     'IPR040899',
                                     None,
                                     'IPR008278'],
              'interpro description': ['Fatty acid synthase type I',
                                       None,
                                       'Fatty acid synthase subunit alpha',
                                       None,
                                       "4'-phosphopantetheinyl transferase "
                                       'domain'],
              'nunique': [1, 1, 1, 1, 1],
              'protein accession': ['C4QY10_e',
                                    'C4QY10_e',
                                    'C4QY10_e',
                                    'C4QY10_e',
                                    'C4QY10_e'],
              'sequence length': ['1879', '1879', '1879', '1879', '1879'],
              'signature accession': ['PF18314',
                                      'PF02801',
                                      'PF18325',
                                      'PF00109',
                                      'PF01648'],
              'signature description': ['Fatty acid synthase type I helical '
                                        'domain',
                                        'Beta-ketoacyl synthase',
                                        'Fatty acid synthase subunit alpha '
                                        'Acyl carrier domain',
                                        'Beta-ketoacyl synthase',
                                        "4'-phosphopantetheinyl transferase "
                                        'superfamily'],
              'start location': ['328', None, '139', None, '1761'],
              'stop location': ['528', None, '300', None, '1861']}}

You can check on the level here.

And to remove the None values from the list you can do this,

In [1]: list(filter(None, ['328', None, '139', None, '1761']))
Out[1]: ['328', '139', '1761']

Since some lists contain duplicates you can use this code to construct an ordered list

ordered_list = list(dict.fromkeys(duplicated_list))

I think, you want to avoid duplication and None values in arrays. For this try the function:

def remove_duplication_from_arr_of_nested_dict(input_dict):
   for key, value in input_dict.items():
      if type(value) is dict:
          input_dict[key] = remove_duplication_from_arr_of_nested_dict(value)
      elif type(value) is list:
          value = set(value)
          if None in value:
              value.remove(None)
          input_dict[key] = list(value)
      return input_dict

Just pass your dictionary to the function, it will remove the duplicates and Nones from lists. If you want to remove duplicated keys at different levels, use same approach to list down all the existing keys and check whether key is in the list or not and delete accordingly.

You can convert a list to a set then back to a list to remove duplicates. However, you need to bear in mind that the original order of the list may not be retained.

Removal of None values can be dealt with in the same list comprehension that’s used to reconstruct the list.

def fix(d):
    for v in d.values():
        if isinstance(v, dict):
            fix(v)
        elif isinstance(v, list):
            v[:] = [e for e in set(v) if e is not None]
    return d

print(fix(data))

If keeping order in the lists is important then:

def fix(d):
    for v in d.values():
        if isinstance(v, dict):
            fix(v)
        elif isinstance(v, list):
            v[:] = list({e: None for e in v if e is not None})
    return d

print(fix(data))