Unable to extract full src attribute from img tag

Question

^{[From trying to scrape Google Search results with requests+bs4, like if soup = BeautifulSoup(requests.get('https://www.google.com/search?q=best+laptops+2022').content), then any img tag you might find with soup.select('img[id^="dimg"]').]}

It seemed simple enough at first – just use soup.find and then .get('src') or .attrs['src'], but major chunks of the src has been replaced with "/////"

The value is actually much longer:
DevTools screenshot

What’s baffling me is that I saved str(soup) as a html file, and also used display(HTML(str(soup))) and with both, the image is being rendered just fine – I can even copy the full src from inspecting the file.
Colab output with fully rendered images

But even

str(soup).split('id="dimg_179" src="')[1].split('"')[0]

produces the same data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==.

I would very much appreciate any explanation of this behavior and/or some suggestions of how to extract the actual src.

Asked By: Driftr95

||

Source

Answer 1

I figured it out as I was writing in the question about inspecting the saved html file – I only used devtools and didn’t open and see all the saved html code itself.

When I open the html on vscode, there are actually two occurrences of the id dimg_179 – the one in the img tag itself, and another in a script tag:

<script nonce="w8n56Ul9BxlnjUkznIswGw">(function(){var s='data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAkGBwgHBgkIBwgKCgkLDRYPDQwMDRsUFRAWIB0iIiAdHx8kKDQsJCYxJx8fLT0tMTU3Ojo6Iys/RD84QzQ5OjcBCgoKDQwNGg8PGjclHyU3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3N//AABEIAEIAdwMBIgACEQEDEQH/xAAbAAACAwEBAQAAAAAAAAAAAAACAwAEBQYBB//EADYQAAEDAgQDBQYFBQEAAAAAAAEAAgMEEQUhMUESUZETUmFxgQYiQmKh8COxwdHhFTIzNJIH/8QAFwEBAQEBAAAAAAAAAAAAAAAAAAIBA//EABsRAQEBAQEBAQEAAAAAAAAAAAABESECURJB/9oADAMBAAIRAxEAPwDouIX2CfG830BQcR7xPqUbX2XJ0WGuPdTWvdySA8b26J0ZaeXQIHCR1rL0PduChbwd1vRGCwbNWseXdffovRI7fL0VukopqrOOMBnfcLD+Vr0uF08BDngSyDdwyHkFsiWRSUFVVWLBwRn43D8ua0n4XBHTObm+UjJ7jofJaZcEqTNVkNcbKXNu1wIINkk6rVxmAxTdq2/C/J1uazSeL4iud4qFkHnZecJ5jqjJHNASPsLNUAhyiIlu9v8AlRNFFvu/ynMft+iSHHl9U1hvshphva7dR9U2CUP1Nj5oGE8h0Ks4bLQsxAQ1kd3vbxRXyaba5dOq2dZbFujpaipP4LPd77tAtykwuCGzprSyeIs0eiY2dvCAywbsAj7RdJ5xNqzxC1gvC5I7Rc37Q+3GEYBilJh9bUNM9Q5rOzY4F7C5wDS4GwDcySSdt7qkuqLkBcldpfdCXoE18LaiBzDuMvNcjVSmE2e03vYm+i7F7lzWPUo7YS/C/J3mo9+dX5uKTi5wBubbZoT5nqlnJoAfpzQEnvBRjdh+R1Jt5qKvxHmOiiz81v6JDj8qYxzuTeiQ1/ypzXju/Vaw9rjy+iTiMMktMHw/7ELu0jtzG3roj4/l+qJsmoLL+qdZxuYFibK6ijladRmOS12y5L5/h9V/TcZfDm2nqbvjF8g74h+vquwhqOJuq66lpiRcV7Yf+c4V7VYxDiVTPPTyNAZOIbfjNGmuh2vnlsun7a268NRbdA7D6eDDsPpqGlBbBTRNijBNyGtFhn6JjpgFnTVzWDN1lj4hj0UDbukAubDxPIcz4Jo6CasazUrIxXGKVkRZO4DjyaBm5x8BqT5LDNVX1xu0Opoe+9t3nyboPM9E2np6emcXtjc+UizpZDxPPry8NFN9fG4dxv4QS21x0S3PN9E0vadWBKcWnRqiNLMmai8e0P8AhsvVRDI6aZzQ5sUjgdCGnNQsLHcLg5pGoI0VqHtDAHguDGtYCBfMEi6r1cv451vwt/u1/tGq5dVxGWumBv3ZIEiMSD7C3pxWxagNVSHsSBPGe0iPzD99PVOwPFxU0rHOycBZzTqCrDZABt0XJ41N/RMRNQz/AAVRvYaCTceuvVdPN+osdrJiLWDNyya32hjjyDs72HiVhUlPieLEPkcaSnPxSNPER4N/ey6DD8NoMO/EYOObQzSZu/j0sl9EimxuKYj71jTRH4pB7xHg3b16K9S4ZBTO7QNL5iLGWTNx9dh4CwV4TNcAWkEHdeOf4KdqshThmlSOa21y65NhZNcSdMkpznN0sm0xOEnn6oSzxUdI7x6JbpHX3KdOCLAN/wA1EPEfsKJ04COonZ7rZpGgaAPOS94nPPE9xc46km5Xiin+BjQOSJRRIGRbIyootBtJsUJztfPNeKLQSiiiAXIFFFlCyTdeO0CiiADqoootH//Z';var i=['dimg_179'];_setImagesSrc(i,s);})();</script>

so I can extract it with a bit of extra effort:

dimgTags = soup.find_all(lambda l: 'dimg_179' in str(l))

# get rid of parent tags
dimgTags = [d for d in dimgTags if not [t for t in dimgTags if d in t.parents]]
dimgTags = [d for d in dimgTags if d.name != 'img']

if dimgTags:
    dimgSrc = str(dimgTags[0]).split("var s='")[1].split("'")[0]
    display(HTML(f'<img src="{dimgSrc}">'))

I’d still love to know if there are better ways though!

Answered By: Driftr95

Unable to extract full src attribute from img tag

Question:

Answers: