Creating a list of dictionaries results in a list of copies of the same dictionary

Question

I want to get all the iframe from a webpage.

Code:

site = "http://" + url
f = urllib2.urlopen(site)
web_content =  f.read()

soup = BeautifulSoup(web_content)
info = {}
content = []
for iframe in soup.find_all('iframe'):
    info['src'] = iframe.get('src')
    info['height'] = iframe.get('height')
    info['width'] = iframe.get('width')
    content.append(info)
    print(info)       

pprint(content)

result of print(info):

{'src': u'abc.com', 'width': u'0', 'height': u'0'}
{'src': u'xyz.com', 'width': u'0', 'height': u'0'}
{'src': u'http://www.detik.com', 'width': u'1000', 'height': u'600'}

result of pprint(content):

[{'height': u'600', 'src': u'http://www.detik.com', 'width': u'1000'},
{'height': u'600', 'src': u'http://www.detik.com', 'width': u'1000'},
{'height': u'600', 'src': u'http://www.detik.com', 'width': u'1000'}]

Why is the value of the content not right? It’s suppose to be the same as the value when I print(info).

Asked By: l1th1um

||

Source

Answer 1

info is a pointer to a dictionary – you keep adding the same pointer to your list contact.

Insert info = {} into the loop and it should solve the problem:

...
content = []
for iframe in soup.find_all('iframe'):
    info = {}
    info['src'] = iframe.get('src')
    info['height'] = iframe.get('height')
    info['width'] = iframe.get('width')
...

Answered By: zenpoy

Answer 2

You are not creating a separate dictionary for each iframe, you just keep modifying the same dictionary over and over, and you keep adding additional references to that dictionary in your list.

Remember, when you do something like content.append(info), you aren’t making a copy of the data, you are simply appending a reference to the data.

You need to create a new dictionary for each iframe.

for iframe in soup.find_all('iframe'):
    info = {}
    ...

Even better, you don’t need to create an empty dictionary first. Just create it all at once:

for iframe in soup.find_all('iframe'):
    info = {
        "src": iframe.get('src'),
        "height": iframe.get('height'),
        "width": iframe.get('width'),
    }
    content.append(info)

There are other ways to accomplish this, such as iterating over a list of attributes, or using list or dictionary comprehensions, but it’s hard to improve upon the clarity of the above code.

Answered By: Bryan Oakley

Answer 3

You have misunderstood the Python list object. It is similar to a C pointer-array. It does not actually “copy” the object which you append to it. Instead, it just store a “pointer” to that object.

Try the following code:

>>> d={}
>>> dlist=[]
>>> for i in xrange(0,3):
    d['data']=i
    dlist.append(d)
    print(d)

{'data': 0}
{'data': 1}
{'data': 2}
>>> print(dlist)
[{'data': 2}, {'data': 2}, {'data': 2}]

So why is print(dlist) not the same as print(d)?

The following code shows you the reason:

>>> for i in dlist:
    print "the list item point to object:", id(i)

the list item point to object: 47472232
the list item point to object: 47472232
the list item point to object: 47472232

So you can see all the items in the dlist is actually pointing to the same dict object.

The real answer to this question will be to append the “copy” of the target item, by using d.copy().

>>> dlist=[]
>>> for i in xrange(0,3):
    d['data']=i
    dlist.append(d.copy())
    print(d)

{'data': 0}
{'data': 1}
{'data': 2}
>>> print dlist
[{'data': 0}, {'data': 1}, {'data': 2}]

Try the id() trick, you can see the list items actually point to completely different objects.

>>> for i in dlist:
    print "the list item points to object:", id(i)

the list item points to object: 33861576
the list item points to object: 47472520
the list item points to object: 47458120

Answered By: Wang

Answer 4

If you want one line:

list_of_dict = [{} for i in range(list_len)]

Answered By: XU Bin

Creating a list of dictionaries results in a list of copies of the same dictionary

Question:

Answers: