Converting a list to a set changes element order
Question:
Recently I noticed that when I am converting a list
to set
the order of elements is changed and is sorted by character.
Consider this example:
x=[1,2,20,6,210]
print(x)
# [1, 2, 20, 6, 210] # the order is same as initial order
set(x)
# set([1, 2, 20, 210, 6]) # in the set(x) output order is sorted
My questions are –
 Why is this happening?
 How can I do set operations (especially set difference) without losing the initial order?
Answers:

A
set
is an unordered data structure, so it does not preserve the insertion order. 
This depends on your requirements. If you have an normal list, and want to remove some set of elements while preserving the order of the list, you can do this with a list comprehension:
>>> a = [1, 2, 20, 6, 210] >>> b = set([6, 20, 1]) >>> [x for x in a if x not in b] [2, 210]
If you need a data structure that supports both fast membership tests and preservation of insertion order, you can use the keys of a Python dictionary, which starting from Python 3.7 is guaranteed to preserve the insertion order:
>>> a = dict.fromkeys([1, 2, 20, 6, 210]) >>> b = dict.fromkeys([6, 20, 1]) >>> dict.fromkeys(x for x in a if x not in b) {2: None, 210: None}
b
doesn’t really need to be ordered here – you could use aset
as well. Note thata.keys()  b.keys()
returns the set difference as aset
, so it won’t preserve the insertion order.In older versions of Python, you can use
collections.OrderedDict
instead:>>> a = collections.OrderedDict.fromkeys([1, 2, 20, 6, 210]) >>> b = collections.OrderedDict.fromkeys([6, 20, 1]) >>> collections.OrderedDict.fromkeys(x for x in a if x not in b) OrderedDict([(2, None), (210, None)])
Answering your first question, a set is a data structure optimized for set operations. Like a mathematical set, it does not enforce or maintain any particular order of the elements. The abstract concept of a set does not enforce order, so the implementation is not required to. When you create a set from a list, Python has the liberty to change the order of the elements for the needs of the internal implementation it uses for a set, which is able to perform set operations efficiently.
As denoted in other answers, sets are data structures (and mathematical concepts) that do not preserve the element order –
However, by using a combination of sets and dictionaries, it is possible that you can achieve wathever you want – try using these snippets:
# save the element order in a dict:
x_dict = dict(x,y for y, x in enumerate(my_list) )
x_set = set(my_list)
#perform desired set operations
...
#retrieve ordered list from the set:
new_list = [None] * len(new_set)
for element in new_set:
new_list[x_dict[element]] = element
Building on Sven’s answer, I found using collections.OrderedDict like so helped me accomplish what you want plus allow me to add more items to the dict:
import collections
x=[1,2,20,6,210]
z=collections.OrderedDict.fromkeys(x)
z
OrderedDict([(1, None), (2, None), (20, None), (6, None), (210, None)])
If you want to add items but still treat it like a set you can just do:
z['nextitem']=None
And you can perform an operation like z.keys() on the dict and get the set:
z.keys()
[1, 2, 20, 6, 210]
Here’s an easy way to do it:
x=[1,2,20,6,210]
print sorted(set(x))
In Python 3.6, there is another solution for Python 2 and 3:set()
now should keep the order, but
>>> x = [1, 2, 20, 6, 210]
>>> sorted(set(x), key=x.index)
[1, 2, 20, 6, 210]
An implementation of the highest score concept above that brings it back to a list:
def SetOfListInOrder(incominglist):
from collections import OrderedDict
outtemp = OrderedDict()
for item in incominglist:
outtemp[item] = None
return(list(outtemp))
Tested (briefly) on Python 3.6 and Python 2.7.
In case you have a small number of elements in your two initial lists on which you want to do set difference operation, instead of using collections.OrderedDict
which complicates the implementation and makes it less readable, you can use:
# initial lists on which you want to do set difference
>>> nums = [1,2,2,3,3,4,4,5]
>>> evens = [2,4,4,6]
>>> evens_set = set(evens)
>>> result = []
>>> for n in nums:
... if not n in evens_set and not n in result:
... result.append(n)
...
>>> result
[1, 3, 5]
Its time complexity is not that good but it is neat and easy to read.
Remove duplicates and preserve order by below function
def unique(sequence):
seen = set()
return [x for x in sequence if not (x in seen or seen.add(x))]
How to remove duplicates from a list while preserving order in Python
In mathematics, there are sets and ordered sets (osets).
 set: an unordered container of unique elements (Implemented)
 oset: an ordered container of unique elements (NotImplemented)
In Python, only sets are directly implemented. We can emulate osets with regular dict keys (3.7+).
Given
a = [1, 2, 20, 6, 210, 2, 1]
b = {2, 6}
Code
oset = dict.fromkeys(a).keys()
# dict_keys([1, 2, 20, 6, 210])
Demo
Replicates are removed, insertionorder is preserved.
list(oset)
# [1, 2, 20, 6, 210]
Setlike operations on dict keys.
oset  b
# {1, 20, 210}
oset  b
# {1, 2, 5, 6, 20, 210}
oset & b
# {2, 6}
oset ^ b
# {1, 5, 20, 210}
Details
Note: an unordered structure does not preclude ordered elements. Rather, maintained order is not guaranteed. Example:
assert {1, 2, 3} == {2, 3, 1} # sets (order is ignored)
assert [1, 2, 3] != [2, 3, 1] # lists (order is guaranteed)
One may be pleased to discover that a list and multiset (mset) are two more fascinating, mathematical data structures:
 list: an ordered container of elements that permits replicates (Implemented)
 mset: an unordered container of elements that permits replicates (NotImplemented)*
Summary
Container  Ordered  Unique  Implemented

set  n  y  y
oset  y  y  n
list  y  n  y
mset  n  n  n*
^{*A multiset can be indirectly emulated with collections.Counter(), a dictlike mapping of multiplicities (counts).}
It’s interesting that people always use ‘real world problem’ to make joke on the definition in theoretical science.
If set has order, you first need to figure out the following problems.
If your list has duplicate elements, what should the order be when you turn it into a set? What is the order if we union two sets? What is the order if we intersect two sets with different order on the same elements?
Plus, set is much faster in searching for a particular key which is very good in sets operation (and that’s why you need a set, but not list).
If you really care about the index, just keep it as a list. If you still want to do set operation on the elements in many lists, the simplest way is creating a dictionary for each list with the same keys in the set along with a value of list containing all the index of the key in the original list.
def indx_dic(l):
dic = {}
for i in range(len(l)):
if l[i] in dic:
dic.get(l[i]).append(i)
else:
dic[l[i]] = [i]
return(dic)
a = [1,2,3,4,5,1,3,2]
set_a = set(a)
dic_a = indx_dic(a)
print(dic_a)
# {1: [0, 5], 2: [1, 7], 3: [2, 6], 4: [3], 5: [4]}
print(set_a)
# {1, 2, 3, 4, 5}
You can remove the duplicated values and keep the list order of insertion with one line of code, Python 3.8.2
mylist = ['b', 'b', 'a', 'd', 'd', 'c'] results = list({value:"" for value in mylist}) print(results) >>> ['b', 'a', 'd', 'c'] results = list(dict.fromkeys(mylist)) print(results) >>> ['b', 'a', 'd', 'c']
We can use collections.Counter for this:
# tested on python 3.7
>>> from collections import Counter
>>> lst = ["1", "2", "20", "6", "210"]
>>> for i in Counter(lst):
>>> print(i, end=" ")
1 2 20 6 210
>>> for i in set(lst):
>>> print(i, end=" ")
20 6 2 1 210
You can remove the duplicated values and keep the list order of insertion, if you want
lst = [1,2,1,3]
new_lst = []
for num in lst :
if num not in new_lst :
new_lst.append(num)
# new_lst = [1,2,3]
don’t use ‘sets’ for removing duplicate if ‘order’ is something you want,
use sets for searching i.e.
x in list
takes O(n) time
where
x in set
takes O(1) time *most cases
Late to answer but you can use Pandas, pd.Series to convert list while preserving the order:
import pandas as pd
x = pd.Series([1, 2, 20, 6, 210, 2, 1])
print(pd.unique(x))
Output:
array([ 1, 2, 20, 6, 210])
Works for a list of strings
x = pd.Series(['c', 'k', 'q', 'n', 'p','c', 'n'])
print(pd.unique(x))
Output
[‘c’ ‘k’ ‘q’ ‘n’ ‘p’]
One more simpler way can be two create a empty list ,let’s say "unique_list" for adding the unique elements from the original list, for example:
unique_list=[]
for i in original_list:
if i not in unique_list:
unique_list.append(i)
else:
pass
This will give you all the unique elements as well as maintain the order.