Python list comprehension: list sub-items without duplicates

Question

I am trying to print all the letters in all the words in a list, without duplicates. I tried:

>>> wordlist = ['cat','dog','rabbit']
>>> letterlist = []
>>> [[letterlist.append(x) for x in y] for y in wordlist]
[[None, None, None], [None, None, None], [None, None, None, None, None, None]]
>>> letterlist
['c', 'a', 't', 'd', 'o', 'g', 'r', 'a', 'b', 'b', 'i', 't']

The desired result is ['c', 'a', 't', 'd', 'o', 'g', 'r', 'b', 'i'] instead.

How do I modify the list comprehension to remove duplicates?

Asked By: zhaoy

||

Source

Answer 1

You can use set to remove the duplicates but the order is not maintained.

>>> letterlist = list({x for y in wordlist for x in y})
>>> letterlist
['a', 'c', 'b', 'd', 'g', 'i', 'o', 'r', 't']
>>>

Answered By: zhangyangyu

Answer 2

If you want to edit you own code:

[[letterlist.append(x) for x in y if x not in letterlist] for y in wordlist]

or

list(set([[letterlist.append(x) for x in y if x not in letterlist] for y in wordlist]))

else:

list(set(''.join(wordlist)))

Answered By: rnbguy

Answer 3

wordlist = ['cat','dog','rabbit']
s = set()
[[s.add(x) for x in y] for y in wordlist]

Answered By: Udy

Answer 4

Do you care about maintaining order?

>>> wordlist = ['cat','dog','rabbit']
>>> set(''.join(wordlist))
{'o', 'i', 'g', 'd', 'c', 'b', 'a', 't', 'r'}

Answered By: roippi

Answer 5

While all other answers don’t maintain order, this code does:

from collections import OrderedDict
letterlist = list(OrderedDict.fromkeys(letterlist))

See also, an article about several ways with benchmarks: Fastest way to uniqify a list in Python.

Answered By: Moayad Mardini

Answer 6

Two approaches:

Preserving order:

>>> from itertools import chain
>>> from collections import OrderedDict
>>> list(OrderedDict.fromkeys(chain.from_iterable(wordlist)))
['c', 'a', 't', 'd', 'o', 'g', 'r', 'b', 'i']

If you’re not fussed about order:

>>> list(set().union(*wordlist))
['a', 'c', 'b', 'd', 'g', 'i', 'o', 'r', 't']

Neither of this are using list-comps for side effects, eg:

[[letterlist.append(x) for x in y] for y in wordlist]

Is building a list of lists of Nones purely to mutate letterlist

Answered By: Jon Clements

Python list comprehension: list sub-items without duplicates

Question:

Answers: