How to get all the html from google search page using requests?
Question:
I’d like to get the html from this url (“https://www.google.com/search?q=urban+outfitters+facebook“), so that I can parse it to get all the links on the page that show up, so that I can ultimately get the facebook username from the first link that shows up (https://www.facebook.com/urbanoutfitters)
I’m able to get all the html I need from a page using requests, but I can’t seem to get all the text from google.
For example, see below for my code:
import requests
url = "https://www.google.com/search?q=urban+outfitters+facebook"
print requests.get(url).text
Also, I’ve looked into the API, but I think just using requests is simpler. I was able to do this using Selenium, so I don’t understand why I can’t do this using requests.
This is my response using requests:
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." name="description"><meta content="noodp" name="robots"><meta content="/images/google_favicon_128.png" itemprop="image"><title>Google</title><script>(function(){window.google={kEI:'DM7KVKwpivaCBJWRg6gL',kEXPI:'4010073,4011559,4020346,4020562,4020873,4021587,4021598,4024625,4025891,4027899,4028063,4028126,4028129,4028468,4028508,4028519,4028585,4028940,8300111,8500393,8500852,8501130,10200083,10200855,10200905',authuser:0,kSID:'DM7KVKwpivaCBJWRg6gL'};google.kHL='en';})();(function(){google.lc=[];google.li=0;google.getEI=function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute("eid")));)a=a.parentNode;return b||google.kEI};google.https=function(){return"https:"==window.location.protocol};google.ml=function(){};google.time=function(){return(new Date).getTime()};google.log=function(a,b,d,e,k){var c=new Image,h=google.lc,f=google.li,g="",l=google.ls||"";c.onerror=c.onload=c.onabort=function(){delete h[f]};h[f]=c;d||-1!=b.search("&ei=")||(e=google.getEI(e),g="&ei="+e,e!=google.kEI&&(g+="&lei="+google.kEI));a=d||"/"+(k||"gen_204")+"?atyp=i&ct="+a+"&cad="+b+g+l+"&zx="+google.time();/^http:/i.test(a)&&google.https()?(google.ml(Error("a"),!1,{src:a,glmm:1}),delete h[f]):(c.src=a,google.li=f+1)};google.y={};google.x=function(a,b){google.y[a.id]=[a,b];return!1};google.load=function(a,b,d){google.x({id:a+m++},function(){google.load(a,b,d)})};var m=0;})();google.kCSI={};var _gjwl=location;function _gjuc(){var a=_gjwl.href.indexOf("#");if(0<=a&&(a=_gjwl.href.substring(a),0<a.indexOf("&q=")||0<=a.indexOf("#q="))&&(a=a.substring(1),-1==a.indexOf("#"))){for(var d=0;d<a.length;){var b=d;"&"==a.charAt(b)&&++b;var c=a.indexOf("&",b);-1==c&&(c=a.length);b=a.substring(b,c);if(0==b.indexOf("fp="))a=a.substring(0,d)+a.substring(c,a.length),c=d;else if("cad=h"==b)return 0;d=c}_gjwl.href="/search?"+a+"&cad=h";return 1}return 0}
function _gjh(){!_gjuc()&&window.google&&google.x&&google.x({id:"GJH"},function(){google.nav&&google.nav.gjh&&google.nav.gjh()})};window._gjh&&_gjh();</script><style>#gbar,#guser{font-size:13px;padding-top:1px !important;}#gbar{height:22px}#guser{padding-bottom:7px !important;text-align:right}.gbh,.gbd{border-top:1px solid #c9d7f1;font-size:1px}.gbh{height:0;position:absolute;top:24px;width:100%}@media all{.gb1{height:22px;margin-right:.5em;vertical-align:top}#gbar{float:left}}a.gb1,a.gb4{text-decoration:underline !important}a.gb1,a.gb4{color:#00c !important}.gbi .gb4{color:#dd8e27 !important}.gbf .gb4{color:#900 !important}</style><style>body,td,a,p,.h{font-family:arial,sans-serif}body{margin:0;overflow-y:scroll}#gog{padding:3px 8px 0}td{line-height:.8em}.gac_m td{line-height:17px}form{margin-bottom:20px}.h{color:#36c}.q{color:#00c}.ts td{padding:0}.ts{border-collapse:collapse}em{font-weight:bold;font-style:normal}.lst{height:25px;width:496px}.gsfi,.lst{font:18px arial,sans-serif}.gsfs{font:17px arial,sans-serif}.ds{display:inline-box;display:inline-block;margin:3px 0 4px;margin-left:4px}input{font-family:inherit}a.gb1,a.gb2,a.gb3,a.gb4{color:#11c !important}body{background:#fff;color:black}a{color:#11c;text-decoration:none}a:hover,a:active{text-decoration:underline}.fl a{color:#36c}a:visited{color:#551a8b}a.gb1,a.gb4{text-decoration:underline}a.gb3:hover{text-decoration:none}#ghead a.gb2:hover{color:#fff !important}.sblc{padding-top:5px}.sblc a{display:block;margin:2px 0;margin-left:13px;font-size:11px}.lsbb{background:#eee;border:solid 1px;border-color:#ccc #999 #999 #ccc;height:30px}.lsbb{display:block}.ftl,#fll a{display:inline-block;margin:0 12px}.lsb{background:url(/images/srpr/nav_logo80.png) 0 -258px repeat-x;border:none;color:#000;cursor:pointer;height:30px;margin:0;outline:0;font:15px arial,sans-serif;vertical-align:top}.lsb:active{background:#ccc}.lst:focus{outline:none}</style><script></script></head><body bgcolor="#fff"><script>(function(){var src='/images/nav_logo176.png';var iesg=false;document.body.onload = function(){window.n && window.n();if (document.images){new Image().src=src;}
if (!iesg){document.f&&document.f.q.focus();document.gbqf&&document.gbqf.q.focus();}
}
})();</script><div id="mngb"> <div id=gbar><nobr><b class=gb1>Search</b> <a class=gb1 href="https://www.google.com/imghp?hl=en&tab=wi">Images</a> <a class=gb1 href="https://maps.google.com/maps?hl=en&tab=wl">Maps</a> <a class=gb1 href="https://play.google.com/?hl=en&tab=w8">Play</a> <a class=gb1 href="https://www.youtube.com/?tab=w1">YouTube</a> <a class=gb1 href="https://news.google.com/nwshp?hl=en&tab=wn">News</a> <a class=gb1 href="https://mail.google.com/mail/?tab=wm">Gmail</a> <a class=gb1 href="https://drive.google.com/?tab=wo">Drive</a> <a class=gb1 style="text-decoration:none" href="http://www.google.com/intl/en/options/"><u>More</u> »</a></nobr></div><div id=guser width=100%><nobr><span id=gbn class=gbi></span><span id=gbf class=gbf></span><span id=gbe></span><a href="http://www.google.com/history/optout?hl=en" class=gb4>Web History</a> | <a href="/preferences?hl=en" class=gb4>Settings</a> | <a target=_top id=gb_70 href="https://accounts.google.com/ServiceLogin?hl=en&continue=https://www.google.com/" class=gb4>Sign in</a></nobr></div><div class=gbh style=left:0></div><div class=gbh style=right:0></div> </div><center><span id="prt" style="display:block"> <div><style>.pmoabs{background-color:#fff;border:1px solid #E5E5E5;color:#666;font-size:13px;padding-bottom:20px;position:absolute;right:2px;top:3px;z-index:986}#pmolnk{border-radius:2px;-moz-border-radius:2px;-webkit-border-radius:2px}.kd-button-submit{border:1px solid #3079ed;background-color:#4d90fe;background-image:-webkit-gradient(linear,left top,left bottom,from(#4d90fe),to(#4787ed));background-image:-webkit-linear-gradient(top,#4d90fe,#4787ed);background-image:-moz-linear-gradient(top,#4d90fe,#4787ed);background-image:-ms-linear-gradient(top,#4d90fe,#4787ed);background-image:-o-linear-gradient(top,#4d90fe,#4787ed);background-image:linear-gradient(top,#4d90fe,#4787ed);filter:progid:DXImageTransform.Microsoft.gradient(startColorStr='#4d90fe',EndColorStr='#4787ed')}.kd-button-submit:hover{border:1px solid #2f5bb7;background-color:#357ae8;background-image:-webkit-gradient(linear,left top,left bottom,from(#4d90fe),to(#357ae8));background-image:-webkit-linear-gradient(top,#4d90fe,#357ae8);background-image:-moz-linear-gradient(top,#4d90fe,#357ae8);background-image:-ms-linear-gradient(top,#4d90fe,#357ae8);background-image:-o-linear-gradient(top,#4d90fe,#357ae8);background-image:linear-gradient(top,#4d90fe,#357ae8);filter:progid:DXImageTransform.Microsoft.gradient(startColorStr='#4d90fe',EndColorStr='#357ae8')}.kd-button-submit:active{-webkit-box-shadow:inset 0 1px 2px rgba(0,0,0,0.3);-moz-box-shadow:inset 0 1px 2px rgba(0,0,0,0.3);box-shadow:inset 0 1px 2px rgba(0,0,0,0.3)}#pmolnk a{color:#fff;display:inline-block;font-weight:bold;padding:5px 20px;text-decoration:none;white-space:nowrap}.xbtn{color:#999;cursor:pointer;font-size:23px;line-height:5px;padding-top:5px}.padi{padding:0 8px 0 10px}.padt{padding:5px 20px 0 0;color:#444}.pads{text-align:left;max-width:200px}</style> <div class="pmoabs" id="pmocntr2" style="behavior:url(#default#userdata);display:none"> <table border="0"> <tr> <td colspan="2"> <div class="xbtn" onclick="google.promos&&google.promos.toast&& google.promos.toast.cpc()" style="float:right">×</div> </td> </tr> <tr> <td class="padi" rowspan="2"> <img src="/images/icons/product/chrome-48.png"> </td> <td class="pads">A faster way to browse the web</td> </tr> <tr> <td class="padt"> <div class="kd-button-submit" id="pmolnk"> <a href="/chrome/index.html?hl=en&brand=CHNG&utm_source=en-hpp&utm_medium=hpp&utm_campaign=en" onclick="google.promos&&google.promos.toast&& google.promos.toast.cl()">Install Google Chrome</a> </div> </td> </tr> </table> </div> <script type="text/javascript">(function(){var a={o:{}};a.o.Pa=50;a.o.Oa=10;a.o.ca="body";a.o.La=!0;a.o.Ea=function(b,c){var d=a.o.Ja();a.o.Ka(d,b,c);a.o.Na(d);a.o.La&&a.o.Ma(d)};a.o.Na=function(b){(b=a.o.ba(b))&&0<b.forms.length&&b.forms[0].submit()};a.o.Ja=function(){var b=document.createElement("iframe");b.height=0;b.width=0;b.style.overflow="hidden";b.style.top=b.style.left="-100px";b.style.position="absolute";document.body.appendChild(b);return b};a.o.ba=function(b){return b.contentDocument||b.contentWindow.document};a.o.Ka=function(b,c,d){b=a.o.ba(b);b.open();d=["<",a.o.ca,'><form method=POST action="',d,'">'];for(var e in c)c.hasOwnProperty(e)&&d.push('<textarea name="',e,'">',c[e],"</textarea>");d.push("</form></",a.o.ca,">");b.write(d.join(""));b.close()};a.o.aa=function(b,c){c>a.o.Oa?google&&google.ml&&google.ml(Error("ogcdr"),!1,{cause:"timeout"}):b.contentWindow?a.o.Qa(b):window.setTimeout(function(){a.o.aa(b,c+1)},a.o.Pa)};a.o.Qa=function(b){document.body.removeChild(b)};a.o.Ma=function(b){a.o.Ra(b,"load",function(){a.o.aa(b,0)})};a.o.Ra=function(b,c,d){b.addEventListener?b.addEventListener(c,d,!1):b.attachEvent&&b.attachEvent("on"+c,d)};var m={Va:0,D:1,F:2,S:5};a.k={};a.k.T={Ha:"i",X:"d",Ia:"l"};a.k.A={R:"0",H:"1"};a.k.U={O:1,X:2,P:3};a.k.w={ta:"a",wa:"g",C:"c",za:"u",ya:"t",R:"p",xa:"pid",va:"eid",Aa:"at"};a.k.Ca=window.location.protocol+"//www.google.com/_/og/promos/";a.k.Ba="g";a.k.Da="z";a.k.Q=function(b,c,d,e){var f=null;switch(c){case m.D:f=window.gbar.up.gpd(b,d,!0);break;case m.S:f=window.gbar.up.gcc(e)}return null==f?0:parseInt(f,10)};a.k.ia=function(b,c,d){return c==m.D?null!=window.gbar.up.gpd(b,d,!0):!1};a.k.V=function(b,c,d,e,f,h,k,l){var g={};g[a.k.w.R]=b;g[a.k.w.wa]=c;g[a.k.w.ta]=d;g[a.k.w.Aa]=e;g[a.k.w.va]=f;g[a.k.w.xa]=1;k&&(g[a.k.w.C]=k);l&&(g[a.k.w.za]=l);if(h)g[a.k.w.ya]=h;else return google.ml(Error("knu"),!1,{cause:"Token is not found"}),null;return g};a.k.W=function(b,c,d){if(b){var e=c?a.k.Ba:a.k.Da;c&&d&&(e+="?authuser="+d);a.o.Ea(b,a.k.Ca+e)}};a.k.Ga=function(b,c,d,e,f,h,k){b=a.k.V(c,b,a.k.T.X,a.k.U.X,d,f,null,e);a.k.W(b,h,k)};a.k.Fa=function(b,c,d,e,f,h,k){b=a.k.V(c,b,a.k.T.Ha,a.k.U.O,d,f,e,null);a.k.W(b,h,k)};a.k.la=function(b,c,d,e,f,h,k,l,g,n){switch(c){case m.S:window.gbar.up.dpc(e,f);break;case m.D:window.gbar.up.spd(b,d,1,!0);break;case m.F:g=g||!1,l=l||"",h=h||0,k=k||a.k.A.H,n=n||0,a.k.Ga(e,h,k,f,l,g,n)}};a.k.ja=function(b,c,d,e,f){return c==m.D?0<d&&a.k.Q(b,c,e,f)>=d:!1};a.k.ga=function(b,c,d,e,f,h,k,l,g,n){switch(c){case m.S:window.gbar.up.iic(e,f);break;case m.D:c=a.k.Q(b,c,d,e)+1;window.gbar.up.spd(b,d,c.toString(),!0);break;case m.F:g=g||!1,l=l||"",h=h||0,k=k||a.k.A.R,n=n||0,a.k.Fa(e,h,k,1,l,g,n)}};a.k.na=function(b,c,d,e,f,h){b=a.k.V(c,b,a.k.T.Ia,a.k.U.P,d,e,null,null);a.k.W(b,f,h)};var p={Ta:"a",Wa:"l",Ua:"c",ka:"d",P:"h",O:"i",gb:"n",H:"x",cb:"ma",eb:"mc",fb:"mi",Xa:"pa",Ya:"pc",$a:"pi",bb:"pn",ab:"px",Za:"pd",hb:"gpa",jb:"gpi",kb:"gpn",lb:"gpx",ib:"gpd"};a.i={};a.i.v={oa:"hplogo",Sa:"pmocntr2"};a.i.A={ea:"0",H:"1",ma:"2"};a.i.p=document.getElementById(a.i.v.Sa);a.i.pa=16;a.i.qa=2;a.i.ra=20;google.promos=google.promos||{};google.promos.toast=google.promos.toast||{};a.i.G=function(b){a.i.p&&(a.i.p.style.display=b?"":"none",a.i.p.parentNode&&(a.i.p.parentNode.style.position=b?"relative":""))};a.i.$=function(b){try{if(a.i.p&&b&&b.es&&b.es.m){var c=window.gbar.rtl(document.body)?"left":"right";a.i.p.style[c]=b.es.m-a.i.pa+a.i.qa+"px";a.i.p.style.top=a.i.ra+"px"}}catch(d){google.ml(d,!1,{cause:a.i.s+"_PT"})}};google.promos.toast.cl=function(){try{a.i.I==m.F&&a.k.na(a.i.J,a.i.B,a.i.A.ma,a.i.N,a.i.L,a.i.M),window.gbar.up.sl(a.i.B,a.i.s,p.P,a.i.K(),1)}catch(b){google.ml(b,!1,{cause:a.i.s+"_CL"})}};google.promos.toast.cpc=function(){try{a.i.p&&(a.i.G(!1),a.k.la(a.i.p,a.i.I,a.i.v.Y,a.i.J,a.i.da,a.i.B,a.i.A.H,a.i.N,a.i.L,a.i.M),window.gbar.up.sl(a.i.B,a.i.s,p.ka,a.i.K(),1))}catch(b){google.ml(b,!1,{cause:a.i.s+"_CPC"})}};a.i.Z=function(){try{if(a.i.p){var b=276,c=document.getElementById(a.i.v.oa);c&&(b=Math.max(b,c.offsetWidth));var d=parseInt(a.i.p.style.right,10)||0;a.i.p.style.visibility=2*(a.i.p.offsetWidth+d)+b>document.body.clientWidth?"hidden":""}}catch(e){google.ml(e,!1,{cause:a.i.s+"_HOSW"})}};a.i.fa=function(){var b=["gpd","spd","aeh","sl"];if(!window.gbar||!window.gbar.up)return!1;for(var c=0,d;d=b[c];c++)if(!(d in window.gbar.up))return!1;return!0};a.i.ha=function(){return a.i.p.currentStyle&&"absolute"!=a.i.p.currentStyle.position};google.promos.toast.init=function(b,c,d,e,f,h,k,l,g,n,q,r){try{a.i.fa()?a.i.p&&(e==m.F&&!l==!g?(google.ml(Error("tku"),!1,{cause:"zwieback: "+g+", gaia: "+l}),a.i.G(!1)):(a.i.v.C="toast_count_"+c+(q?"_"+q:""),a.i.v.Y="toast_dp_"+c+(r?"_"+r:""),a.i.s=d,a.i.B=b,a.i.I=e,a.i.J=c,a.i.da=f,a.i.N=l?l:g,a.i.L=!!l,a.i.M=k,a.k.ia(a.i.p,e,a.i.v.Y,c)||a.k.ja(a.i.p,e,h,a.i.v.C,c)||a.i.ha()?a.i.G(!1):(a.k.ga(a.i.p,e,a.i.v.C,c,f,a.i.B,a.i.A.ea,a.i.N,a.i.L,a.i.M),n||(window.gbar.up.aeh(window,"resize",a.i.Z),window.lol=
a.i.Z,window.gbar.elr&&a.i.$(window.gbar.elr()),window.gbar.elc&&window.gbar.elc(a.i.$),a.i.G(!0)),window.gbar.up.sl(a.i.B,a.i.s,p.O,a.i.K())))):google.ml(Error("apa"),!1,{cause:a.i.s+"_INIT"})}catch(t){google.ml(t,!1,{cause:a.i.s+"_INIT"})}};a.i.K=function(){var b=a.k.Q(a.i.p,a.i.I,a.i.v.C,a.i.J);return"ic="+b};})();</script> <script type="text/javascript">(function(){var sourceWebappPromoID=144002;var sourceWebappGroupID=5;var payloadType=5;var cookieMaxAgeSec=2592000;var dismissalType=5;var impressionCap=25;var gaiaXsrfToken='';var zwbkXsrfToken='';var kansasDismissalEnabled=false;var sessionIndex=0;var invisible=false;window.gbar&&gbar.up&&gbar.up.r&&gbar.up.r(payloadType,function(show){if (show){google.promos.toast.init(sourceWebappPromoID,sourceWebappGroupID,payloadType,dismissalType,cookieMaxAgeSec,impressionCap,sessionIndex,gaiaXsrfToken,zwbkXsrfToken,invisible,'0612');}
});})();</script> </div> </span><br clear="all" id="lgpd"><div id="lga"><img alt="Google" height="95" src="/images/srpr/logo9w.png" style="padding:28px 0 14px" width="269" id="hplogo" onload="window.lol&&lol()"><br><br></div><form action="/search" name="f"><table cellpadding="0" cellspacing="0"><tr valign="top"><td width="25%"> </td><td align="center" nowrap=""><input name="ie" value="ISO-8859-1" type="hidden"><input value="en" name="hl" type="hidden"><input name="source" type="hidden" value="hp"><div class="ds" style="height:32px;margin:4px 0"><input style="color:#000;margin:0;padding:5px 8px 0 6px;vertical-align:top" autocomplete="off" class="lst" value="" title="Google Search" maxlength="2048" name="q" size="57"></div><br style="line-height:0"><span class="ds"><span class="lsbb"><input class="lsb" value="Google Search" name="btnG" type="submit"></span></span><span class="ds"><span class="lsbb"><input class="lsb" value="I'm Feeling Lucky" name="btnI" onclick="if(this.form.q.value)this.checked=1; else top.location='/doodles/'" type="submit"></span></span></td><td class="fl sblc" align="left" nowrap="" width="25%"><a href="/advanced_search?hl=en&authuser=0">Advanced search</a><a href="/language_tools?hl=en&authuser=0">Language tools</a></td></tr></table><input id="gbv" name="gbv" type="hidden" value="1"></form><div id="gac_scont"></div><div style="font-size:83%;min-height:3.5em"><br></div><span id="footer"><div style="font-size:10pt"><div style="margin:19px auto;text-align:center" id="fll"><a href="/intl/en/ads/">Advertising Programs</a><a href="/services/">Business Solutions</a><a href="https://plus.google.com/116899029375914044550" rel="publisher">+Google</a><a href="/intl/en/about.html">About Google</a></div></div><p style="color:#767676;font-size:8pt">© 2015 - <a href="/intl/en/policies/privacy/">Privacy</a> - <a href="/intl/en/policies/terms/">Terms</a></p></span></center><div id="xjsd"></div><div id="xjsi" data-jiis="bp"><script>(function(){function c(b){window.setTimeout(function(){var a=document.createElement("script");a.src=b;document.getElementById("xjsd").appendChild(a)},0)}google.dljp=function(b,a){google.xjsu=b;c(a)};google.dlj=c;})();(function(){window.google.xjsrm=[];})();if(google.y)google.y.first=[];if(!google.xjs){window._=window._||{};window._._DumpException=function(e){throw e};if(google.timers&&google.timers.load.t){google.timers.load.t.xjsls=new Date().getTime();}google.dljp('/xjs/_/js/kx3dxjs.hp.en_US.4dB-kXZgo4g.O/mx3dsb_he,d/rtx3dj/dx3d1/tx3dzcms/rsx3dACT90oFyTgnV60GhNLdstOIcFET3IVANCA','/xjs/_/js/kx3dxjs.hp.en_US.4dB-kXZgo4g.O/mx3dsb_he,d/rtx3dj/dx3d1/tx3dzcms/rsx3dACT90oFyTgnV60GhNLdstOIcFET3IVANCA');google.xjs=1;}google.pmc={"sb_he":{"agen":true,"cgen":true,"client":"heirloom-hp","dh":true,"ds":"","exp":"msedr","fl":true,"host":"google.com","jam":0,"jsonp":true,"msgs":{"cibl":"Clear Search","dym":"Did you mean:","lcky":"Iu0026#39;m Feeling Lucky","lml":"Learn more","oskt":"Input tools","psrc":"This search was removed from your u003Ca href="/history"u003EWeb Historyu003C/au003E","psrl":"Remove","sbit":"Search by image","srch":"Google Search"},"ovr":{},"pq":"","refoq":true,"scd":10,"sce":5,"stok":"iXw-xWnUXlH7Fp6SrUErmgr3X8g"},"d":{}};google.y.first.push(function(){if(google.med){google.med('init');google.initHistory();google.med('history');}});if(google.j&&google.j.en&&google.j.xi){window.setTimeout(google.j.xi,0);}
</script></div></body></html>
Answers:
UPDATE 2021: Looks like this no longer works. Google deprecated this a while back. They instead recommend you use their new Custom Search API but this now requires you to input an API key, which is less convenient than the old public endpoint.
Replace the beginning of your url with http://ajax.googleapis.com/ajax/services/search/web?v=1.0
It should now looks like this:
http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=urban+outfitters+facebook
Using the Python json parser you can retrieve the first url.
import requests
import json
url = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=urban+outfitters+facebook"
google_result = json.loads(requests.get(url).text)
print google_result["responseData"]["results"][0]["url"]
Since Laurent’s old answer didn’t work (and I didn’t want to use the API) I searched for another method and found the one from here: https://hackernoon.com/how-to-scrape-google-with-python-bo7d2tal
import requests
from bs4 import BeautifulSoup
query = "hackernoon How To Scrape Google With Python"
query = query.replace(' ', '+')
URL = f"https://google.com/search?q={query}"
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0"
headers = {"user-agent" : USER_AGENT}
resp = requests.get(URL, headers=headers)
if resp.status_code == 200:
soup = BeautifulSoup(resp.content, "html.parser")
Using the ‘soup’ everything else can be extracted from the html.
Make sure you’re using user-agent because Google treats your script as python-requests. That was a problem you faced. You need to fake a real user visit with user-agent. List of user-agents.
Code and example in online IDE:
from bs4 import BeautifulSoup
import requests, lxml
headers = {
'User-agent':
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}
html = requests.get('https://www.google.com/search?q=urban outfitters facebook', headers=headers).text
soup = BeautifulSoup(html, 'lxml')
for result in soup.select('.yuRUbf'):
title = result.select_one('.DKV0Md').text
link = result.select_one('a')['href']
print(f'{title}n{link}n')
Output:
Urban Outfitters - Verified Page | Facebook
https://www.facebook.com/urbanoutfitters/
Urban Outfitters - Home | Facebook
https://www.facebook.com/urbanoutfitterseurope/
Urban Outfitters - Facebook - Urban Outfitters - Blog
https://blog.urbanoutfitters.com/facebook
Contact Us - Urban Outfitters
https://www.urbanoutfitters.com/en-gb/help/contact-us
Frank & Funny Facebook Card | Urban Outfitters
https://www.urbanoutfitters.com/shop/frank--funny-facebook-card?color=&parentid=SALE_APT&quantity=1&type=REGULAR
Urban Outfitters
https://www.urbanoutfitters.com/
Urban Outfitters (@urbanoutfitters) • Instagram photos and ...
https://www.instagram.com/urbanoutfitters/
Alternatively, you can achieve these results using Google Search Engine Results API from SerpApi. It’s a paid API with a free plan to test out.
If you want to get just a RAW-HTML link, you can call ['search_metadata']['raw_html_file'] or just print(results) to get all data.
Code to integrate:
import os
from serpapi import GoogleSearch
params = {
"engine": "google",
"q": "urban outfitters facebook",
"api_key": os.getenv("API_KEY"),
}
search = GoogleSearch(params)
results = search.get_dict()
html = results['search_metadata']['raw_html_file']
print(f'Raw HTML: {html}')
print(json.dumps(results, indent = 2, ensure_ascii = False))
Output:
https://serpapi.com/searches/f4b6c93572fd22f8/609cb4f5c640d28919b34dad.html
...
"organic_results": [
{
"position": 1,
"title": "Urban Outfitters - Verified Page | Facebook",
"link": "https://www.facebook.com/urbanoutfitters/",
"displayed_link": "https://www.facebook.com › ... › Clothing (Brand)",
"snippet": "Urban Outfitters. 2187043 likes · 4157 talking about this · 169423 were here. Visit us at www.urbanoutfitters.com. Always open, always awesome.",
"sitelinks": {
"expanded": [
{
"title": "Urban Outfitters",
"link": "https://en-gb.facebook.com/urbanoutfitters",
"snippet": "Urban Outfitters. 2187043 likes · 4259 talking about this ..."
},
{
"title": "Instagram",
"link": "https://www.facebook.com/urbanoutfitters/app/168188869963563/",
"snippet": "Block Page. More. Send Message. See more of Urban Outfitters on ..."
},
{
"title": "About",
"link": "https://www.facebook.com/urbanoutfitters/about/",
"snippet": "The official Facebook page of Urban Outfitters. Questions ..."
},
{
"title": "Events",
"link": "https://www.facebook.com/urbanoutfitters/events/",
"snippet": "Urban Outfitters does not have any upcoming events. Past Events ..."
}
]
}
}
]...
Disclaimer, I work for SerpApi.
I’d like to get the html from this url (“https://www.google.com/search?q=urban+outfitters+facebook“), so that I can parse it to get all the links on the page that show up, so that I can ultimately get the facebook username from the first link that shows up (https://www.facebook.com/urbanoutfitters)
I’m able to get all the html I need from a page using requests, but I can’t seem to get all the text from google.
For example, see below for my code:
import requests
url = "https://www.google.com/search?q=urban+outfitters+facebook"
print requests.get(url).text
Also, I’ve looked into the API, but I think just using requests is simpler. I was able to do this using Selenium, so I don’t understand why I can’t do this using requests.
This is my response using requests:
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." name="description"><meta content="noodp" name="robots"><meta content="/images/google_favicon_128.png" itemprop="image"><title>Google</title><script>(function(){window.google={kEI:'DM7KVKwpivaCBJWRg6gL',kEXPI:'4010073,4011559,4020346,4020562,4020873,4021587,4021598,4024625,4025891,4027899,4028063,4028126,4028129,4028468,4028508,4028519,4028585,4028940,8300111,8500393,8500852,8501130,10200083,10200855,10200905',authuser:0,kSID:'DM7KVKwpivaCBJWRg6gL'};google.kHL='en';})();(function(){google.lc=[];google.li=0;google.getEI=function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute("eid")));)a=a.parentNode;return b||google.kEI};google.https=function(){return"https:"==window.location.protocol};google.ml=function(){};google.time=function(){return(new Date).getTime()};google.log=function(a,b,d,e,k){var c=new Image,h=google.lc,f=google.li,g="",l=google.ls||"";c.onerror=c.onload=c.onabort=function(){delete h[f]};h[f]=c;d||-1!=b.search("&ei=")||(e=google.getEI(e),g="&ei="+e,e!=google.kEI&&(g+="&lei="+google.kEI));a=d||"/"+(k||"gen_204")+"?atyp=i&ct="+a+"&cad="+b+g+l+"&zx="+google.time();/^http:/i.test(a)&&google.https()?(google.ml(Error("a"),!1,{src:a,glmm:1}),delete h[f]):(c.src=a,google.li=f+1)};google.y={};google.x=function(a,b){google.y[a.id]=[a,b];return!1};google.load=function(a,b,d){google.x({id:a+m++},function(){google.load(a,b,d)})};var m=0;})();google.kCSI={};var _gjwl=location;function _gjuc(){var a=_gjwl.href.indexOf("#");if(0<=a&&(a=_gjwl.href.substring(a),0<a.indexOf("&q=")||0<=a.indexOf("#q="))&&(a=a.substring(1),-1==a.indexOf("#"))){for(var d=0;d<a.length;){var b=d;"&"==a.charAt(b)&&++b;var c=a.indexOf("&",b);-1==c&&(c=a.length);b=a.substring(b,c);if(0==b.indexOf("fp="))a=a.substring(0,d)+a.substring(c,a.length),c=d;else if("cad=h"==b)return 0;d=c}_gjwl.href="/search?"+a+"&cad=h";return 1}return 0}
function _gjh(){!_gjuc()&&window.google&&google.x&&google.x({id:"GJH"},function(){google.nav&&google.nav.gjh&&google.nav.gjh()})};window._gjh&&_gjh();</script><style>#gbar,#guser{font-size:13px;padding-top:1px !important;}#gbar{height:22px}#guser{padding-bottom:7px !important;text-align:right}.gbh,.gbd{border-top:1px solid #c9d7f1;font-size:1px}.gbh{height:0;position:absolute;top:24px;width:100%}@media all{.gb1{height:22px;margin-right:.5em;vertical-align:top}#gbar{float:left}}a.gb1,a.gb4{text-decoration:underline !important}a.gb1,a.gb4{color:#00c !important}.gbi .gb4{color:#dd8e27 !important}.gbf .gb4{color:#900 !important}</style><style>body,td,a,p,.h{font-family:arial,sans-serif}body{margin:0;overflow-y:scroll}#gog{padding:3px 8px 0}td{line-height:.8em}.gac_m td{line-height:17px}form{margin-bottom:20px}.h{color:#36c}.q{color:#00c}.ts td{padding:0}.ts{border-collapse:collapse}em{font-weight:bold;font-style:normal}.lst{height:25px;width:496px}.gsfi,.lst{font:18px arial,sans-serif}.gsfs{font:17px arial,sans-serif}.ds{display:inline-box;display:inline-block;margin:3px 0 4px;margin-left:4px}input{font-family:inherit}a.gb1,a.gb2,a.gb3,a.gb4{color:#11c !important}body{background:#fff;color:black}a{color:#11c;text-decoration:none}a:hover,a:active{text-decoration:underline}.fl a{color:#36c}a:visited{color:#551a8b}a.gb1,a.gb4{text-decoration:underline}a.gb3:hover{text-decoration:none}#ghead a.gb2:hover{color:#fff !important}.sblc{padding-top:5px}.sblc a{display:block;margin:2px 0;margin-left:13px;font-size:11px}.lsbb{background:#eee;border:solid 1px;border-color:#ccc #999 #999 #ccc;height:30px}.lsbb{display:block}.ftl,#fll a{display:inline-block;margin:0 12px}.lsb{background:url(/images/srpr/nav_logo80.png) 0 -258px repeat-x;border:none;color:#000;cursor:pointer;height:30px;margin:0;outline:0;font:15px arial,sans-serif;vertical-align:top}.lsb:active{background:#ccc}.lst:focus{outline:none}</style><script></script></head><body bgcolor="#fff"><script>(function(){var src='/images/nav_logo176.png';var iesg=false;document.body.onload = function(){window.n && window.n();if (document.images){new Image().src=src;}
if (!iesg){document.f&&document.f.q.focus();document.gbqf&&document.gbqf.q.focus();}
}
})();</script><div id="mngb"> <div id=gbar><nobr><b class=gb1>Search</b> <a class=gb1 href="https://www.google.com/imghp?hl=en&tab=wi">Images</a> <a class=gb1 href="https://maps.google.com/maps?hl=en&tab=wl">Maps</a> <a class=gb1 href="https://play.google.com/?hl=en&tab=w8">Play</a> <a class=gb1 href="https://www.youtube.com/?tab=w1">YouTube</a> <a class=gb1 href="https://news.google.com/nwshp?hl=en&tab=wn">News</a> <a class=gb1 href="https://mail.google.com/mail/?tab=wm">Gmail</a> <a class=gb1 href="https://drive.google.com/?tab=wo">Drive</a> <a class=gb1 style="text-decoration:none" href="http://www.google.com/intl/en/options/"><u>More</u> »</a></nobr></div><div id=guser width=100%><nobr><span id=gbn class=gbi></span><span id=gbf class=gbf></span><span id=gbe></span><a href="http://www.google.com/history/optout?hl=en" class=gb4>Web History</a> | <a href="/preferences?hl=en" class=gb4>Settings</a> | <a target=_top id=gb_70 href="https://accounts.google.com/ServiceLogin?hl=en&continue=https://www.google.com/" class=gb4>Sign in</a></nobr></div><div class=gbh style=left:0></div><div class=gbh style=right:0></div> </div><center><span id="prt" style="display:block"> <div><style>.pmoabs{background-color:#fff;border:1px solid #E5E5E5;color:#666;font-size:13px;padding-bottom:20px;position:absolute;right:2px;top:3px;z-index:986}#pmolnk{border-radius:2px;-moz-border-radius:2px;-webkit-border-radius:2px}.kd-button-submit{border:1px solid #3079ed;background-color:#4d90fe;background-image:-webkit-gradient(linear,left top,left bottom,from(#4d90fe),to(#4787ed));background-image:-webkit-linear-gradient(top,#4d90fe,#4787ed);background-image:-moz-linear-gradient(top,#4d90fe,#4787ed);background-image:-ms-linear-gradient(top,#4d90fe,#4787ed);background-image:-o-linear-gradient(top,#4d90fe,#4787ed);background-image:linear-gradient(top,#4d90fe,#4787ed);filter:progid:DXImageTransform.Microsoft.gradient(startColorStr='#4d90fe',EndColorStr='#4787ed')}.kd-button-submit:hover{border:1px solid #2f5bb7;background-color:#357ae8;background-image:-webkit-gradient(linear,left top,left bottom,from(#4d90fe),to(#357ae8));background-image:-webkit-linear-gradient(top,#4d90fe,#357ae8);background-image:-moz-linear-gradient(top,#4d90fe,#357ae8);background-image:-ms-linear-gradient(top,#4d90fe,#357ae8);background-image:-o-linear-gradient(top,#4d90fe,#357ae8);background-image:linear-gradient(top,#4d90fe,#357ae8);filter:progid:DXImageTransform.Microsoft.gradient(startColorStr='#4d90fe',EndColorStr='#357ae8')}.kd-button-submit:active{-webkit-box-shadow:inset 0 1px 2px rgba(0,0,0,0.3);-moz-box-shadow:inset 0 1px 2px rgba(0,0,0,0.3);box-shadow:inset 0 1px 2px rgba(0,0,0,0.3)}#pmolnk a{color:#fff;display:inline-block;font-weight:bold;padding:5px 20px;text-decoration:none;white-space:nowrap}.xbtn{color:#999;cursor:pointer;font-size:23px;line-height:5px;padding-top:5px}.padi{padding:0 8px 0 10px}.padt{padding:5px 20px 0 0;color:#444}.pads{text-align:left;max-width:200px}</style> <div class="pmoabs" id="pmocntr2" style="behavior:url(#default#userdata);display:none"> <table border="0"> <tr> <td colspan="2"> <div class="xbtn" onclick="google.promos&&google.promos.toast&& google.promos.toast.cpc()" style="float:right">×</div> </td> </tr> <tr> <td class="padi" rowspan="2"> <img src="/images/icons/product/chrome-48.png"> </td> <td class="pads">A faster way to browse the web</td> </tr> <tr> <td class="padt"> <div class="kd-button-submit" id="pmolnk"> <a href="/chrome/index.html?hl=en&brand=CHNG&utm_source=en-hpp&utm_medium=hpp&utm_campaign=en" onclick="google.promos&&google.promos.toast&& google.promos.toast.cl()">Install Google Chrome</a> </div> </td> </tr> </table> </div> <script type="text/javascript">(function(){var a={o:{}};a.o.Pa=50;a.o.Oa=10;a.o.ca="body";a.o.La=!0;a.o.Ea=function(b,c){var d=a.o.Ja();a.o.Ka(d,b,c);a.o.Na(d);a.o.La&&a.o.Ma(d)};a.o.Na=function(b){(b=a.o.ba(b))&&0<b.forms.length&&b.forms[0].submit()};a.o.Ja=function(){var b=document.createElement("iframe");b.height=0;b.width=0;b.style.overflow="hidden";b.style.top=b.style.left="-100px";b.style.position="absolute";document.body.appendChild(b);return b};a.o.ba=function(b){return b.contentDocument||b.contentWindow.document};a.o.Ka=function(b,c,d){b=a.o.ba(b);b.open();d=["<",a.o.ca,'><form method=POST action="',d,'">'];for(var e in c)c.hasOwnProperty(e)&&d.push('<textarea name="',e,'">',c[e],"</textarea>");d.push("</form></",a.o.ca,">");b.write(d.join(""));b.close()};a.o.aa=function(b,c){c>a.o.Oa?google&&google.ml&&google.ml(Error("ogcdr"),!1,{cause:"timeout"}):b.contentWindow?a.o.Qa(b):window.setTimeout(function(){a.o.aa(b,c+1)},a.o.Pa)};a.o.Qa=function(b){document.body.removeChild(b)};a.o.Ma=function(b){a.o.Ra(b,"load",function(){a.o.aa(b,0)})};a.o.Ra=function(b,c,d){b.addEventListener?b.addEventListener(c,d,!1):b.attachEvent&&b.attachEvent("on"+c,d)};var m={Va:0,D:1,F:2,S:5};a.k={};a.k.T={Ha:"i",X:"d",Ia:"l"};a.k.A={R:"0",H:"1"};a.k.U={O:1,X:2,P:3};a.k.w={ta:"a",wa:"g",C:"c",za:"u",ya:"t",R:"p",xa:"pid",va:"eid",Aa:"at"};a.k.Ca=window.location.protocol+"//www.google.com/_/og/promos/";a.k.Ba="g";a.k.Da="z";a.k.Q=function(b,c,d,e){var f=null;switch(c){case m.D:f=window.gbar.up.gpd(b,d,!0);break;case m.S:f=window.gbar.up.gcc(e)}return null==f?0:parseInt(f,10)};a.k.ia=function(b,c,d){return c==m.D?null!=window.gbar.up.gpd(b,d,!0):!1};a.k.V=function(b,c,d,e,f,h,k,l){var g={};g[a.k.w.R]=b;g[a.k.w.wa]=c;g[a.k.w.ta]=d;g[a.k.w.Aa]=e;g[a.k.w.va]=f;g[a.k.w.xa]=1;k&&(g[a.k.w.C]=k);l&&(g[a.k.w.za]=l);if(h)g[a.k.w.ya]=h;else return google.ml(Error("knu"),!1,{cause:"Token is not found"}),null;return g};a.k.W=function(b,c,d){if(b){var e=c?a.k.Ba:a.k.Da;c&&d&&(e+="?authuser="+d);a.o.Ea(b,a.k.Ca+e)}};a.k.Ga=function(b,c,d,e,f,h,k){b=a.k.V(c,b,a.k.T.X,a.k.U.X,d,f,null,e);a.k.W(b,h,k)};a.k.Fa=function(b,c,d,e,f,h,k){b=a.k.V(c,b,a.k.T.Ha,a.k.U.O,d,f,e,null);a.k.W(b,h,k)};a.k.la=function(b,c,d,e,f,h,k,l,g,n){switch(c){case m.S:window.gbar.up.dpc(e,f);break;case m.D:window.gbar.up.spd(b,d,1,!0);break;case m.F:g=g||!1,l=l||"",h=h||0,k=k||a.k.A.H,n=n||0,a.k.Ga(e,h,k,f,l,g,n)}};a.k.ja=function(b,c,d,e,f){return c==m.D?0<d&&a.k.Q(b,c,e,f)>=d:!1};a.k.ga=function(b,c,d,e,f,h,k,l,g,n){switch(c){case m.S:window.gbar.up.iic(e,f);break;case m.D:c=a.k.Q(b,c,d,e)+1;window.gbar.up.spd(b,d,c.toString(),!0);break;case m.F:g=g||!1,l=l||"",h=h||0,k=k||a.k.A.R,n=n||0,a.k.Fa(e,h,k,1,l,g,n)}};a.k.na=function(b,c,d,e,f,h){b=a.k.V(c,b,a.k.T.Ia,a.k.U.P,d,e,null,null);a.k.W(b,f,h)};var p={Ta:"a",Wa:"l",Ua:"c",ka:"d",P:"h",O:"i",gb:"n",H:"x",cb:"ma",eb:"mc",fb:"mi",Xa:"pa",Ya:"pc",$a:"pi",bb:"pn",ab:"px",Za:"pd",hb:"gpa",jb:"gpi",kb:"gpn",lb:"gpx",ib:"gpd"};a.i={};a.i.v={oa:"hplogo",Sa:"pmocntr2"};a.i.A={ea:"0",H:"1",ma:"2"};a.i.p=document.getElementById(a.i.v.Sa);a.i.pa=16;a.i.qa=2;a.i.ra=20;google.promos=google.promos||{};google.promos.toast=google.promos.toast||{};a.i.G=function(b){a.i.p&&(a.i.p.style.display=b?"":"none",a.i.p.parentNode&&(a.i.p.parentNode.style.position=b?"relative":""))};a.i.$=function(b){try{if(a.i.p&&b&&b.es&&b.es.m){var c=window.gbar.rtl(document.body)?"left":"right";a.i.p.style[c]=b.es.m-a.i.pa+a.i.qa+"px";a.i.p.style.top=a.i.ra+"px"}}catch(d){google.ml(d,!1,{cause:a.i.s+"_PT"})}};google.promos.toast.cl=function(){try{a.i.I==m.F&&a.k.na(a.i.J,a.i.B,a.i.A.ma,a.i.N,a.i.L,a.i.M),window.gbar.up.sl(a.i.B,a.i.s,p.P,a.i.K(),1)}catch(b){google.ml(b,!1,{cause:a.i.s+"_CL"})}};google.promos.toast.cpc=function(){try{a.i.p&&(a.i.G(!1),a.k.la(a.i.p,a.i.I,a.i.v.Y,a.i.J,a.i.da,a.i.B,a.i.A.H,a.i.N,a.i.L,a.i.M),window.gbar.up.sl(a.i.B,a.i.s,p.ka,a.i.K(),1))}catch(b){google.ml(b,!1,{cause:a.i.s+"_CPC"})}};a.i.Z=function(){try{if(a.i.p){var b=276,c=document.getElementById(a.i.v.oa);c&&(b=Math.max(b,c.offsetWidth));var d=parseInt(a.i.p.style.right,10)||0;a.i.p.style.visibility=2*(a.i.p.offsetWidth+d)+b>document.body.clientWidth?"hidden":""}}catch(e){google.ml(e,!1,{cause:a.i.s+"_HOSW"})}};a.i.fa=function(){var b=["gpd","spd","aeh","sl"];if(!window.gbar||!window.gbar.up)return!1;for(var c=0,d;d=b[c];c++)if(!(d in window.gbar.up))return!1;return!0};a.i.ha=function(){return a.i.p.currentStyle&&"absolute"!=a.i.p.currentStyle.position};google.promos.toast.init=function(b,c,d,e,f,h,k,l,g,n,q,r){try{a.i.fa()?a.i.p&&(e==m.F&&!l==!g?(google.ml(Error("tku"),!1,{cause:"zwieback: "+g+", gaia: "+l}),a.i.G(!1)):(a.i.v.C="toast_count_"+c+(q?"_"+q:""),a.i.v.Y="toast_dp_"+c+(r?"_"+r:""),a.i.s=d,a.i.B=b,a.i.I=e,a.i.J=c,a.i.da=f,a.i.N=l?l:g,a.i.L=!!l,a.i.M=k,a.k.ia(a.i.p,e,a.i.v.Y,c)||a.k.ja(a.i.p,e,h,a.i.v.C,c)||a.i.ha()?a.i.G(!1):(a.k.ga(a.i.p,e,a.i.v.C,c,f,a.i.B,a.i.A.ea,a.i.N,a.i.L,a.i.M),n||(window.gbar.up.aeh(window,"resize",a.i.Z),window.lol=
a.i.Z,window.gbar.elr&&a.i.$(window.gbar.elr()),window.gbar.elc&&window.gbar.elc(a.i.$),a.i.G(!0)),window.gbar.up.sl(a.i.B,a.i.s,p.O,a.i.K())))):google.ml(Error("apa"),!1,{cause:a.i.s+"_INIT"})}catch(t){google.ml(t,!1,{cause:a.i.s+"_INIT"})}};a.i.K=function(){var b=a.k.Q(a.i.p,a.i.I,a.i.v.C,a.i.J);return"ic="+b};})();</script> <script type="text/javascript">(function(){var sourceWebappPromoID=144002;var sourceWebappGroupID=5;var payloadType=5;var cookieMaxAgeSec=2592000;var dismissalType=5;var impressionCap=25;var gaiaXsrfToken='';var zwbkXsrfToken='';var kansasDismissalEnabled=false;var sessionIndex=0;var invisible=false;window.gbar&&gbar.up&&gbar.up.r&&gbar.up.r(payloadType,function(show){if (show){google.promos.toast.init(sourceWebappPromoID,sourceWebappGroupID,payloadType,dismissalType,cookieMaxAgeSec,impressionCap,sessionIndex,gaiaXsrfToken,zwbkXsrfToken,invisible,'0612');}
});})();</script> </div> </span><br clear="all" id="lgpd"><div id="lga"><img alt="Google" height="95" src="/images/srpr/logo9w.png" style="padding:28px 0 14px" width="269" id="hplogo" onload="window.lol&&lol()"><br><br></div><form action="/search" name="f"><table cellpadding="0" cellspacing="0"><tr valign="top"><td width="25%"> </td><td align="center" nowrap=""><input name="ie" value="ISO-8859-1" type="hidden"><input value="en" name="hl" type="hidden"><input name="source" type="hidden" value="hp"><div class="ds" style="height:32px;margin:4px 0"><input style="color:#000;margin:0;padding:5px 8px 0 6px;vertical-align:top" autocomplete="off" class="lst" value="" title="Google Search" maxlength="2048" name="q" size="57"></div><br style="line-height:0"><span class="ds"><span class="lsbb"><input class="lsb" value="Google Search" name="btnG" type="submit"></span></span><span class="ds"><span class="lsbb"><input class="lsb" value="I'm Feeling Lucky" name="btnI" onclick="if(this.form.q.value)this.checked=1; else top.location='/doodles/'" type="submit"></span></span></td><td class="fl sblc" align="left" nowrap="" width="25%"><a href="/advanced_search?hl=en&authuser=0">Advanced search</a><a href="/language_tools?hl=en&authuser=0">Language tools</a></td></tr></table><input id="gbv" name="gbv" type="hidden" value="1"></form><div id="gac_scont"></div><div style="font-size:83%;min-height:3.5em"><br></div><span id="footer"><div style="font-size:10pt"><div style="margin:19px auto;text-align:center" id="fll"><a href="/intl/en/ads/">Advertising Programs</a><a href="/services/">Business Solutions</a><a href="https://plus.google.com/116899029375914044550" rel="publisher">+Google</a><a href="/intl/en/about.html">About Google</a></div></div><p style="color:#767676;font-size:8pt">© 2015 - <a href="/intl/en/policies/privacy/">Privacy</a> - <a href="/intl/en/policies/terms/">Terms</a></p></span></center><div id="xjsd"></div><div id="xjsi" data-jiis="bp"><script>(function(){function c(b){window.setTimeout(function(){var a=document.createElement("script");a.src=b;document.getElementById("xjsd").appendChild(a)},0)}google.dljp=function(b,a){google.xjsu=b;c(a)};google.dlj=c;})();(function(){window.google.xjsrm=[];})();if(google.y)google.y.first=[];if(!google.xjs){window._=window._||{};window._._DumpException=function(e){throw e};if(google.timers&&google.timers.load.t){google.timers.load.t.xjsls=new Date().getTime();}google.dljp('/xjs/_/js/kx3dxjs.hp.en_US.4dB-kXZgo4g.O/mx3dsb_he,d/rtx3dj/dx3d1/tx3dzcms/rsx3dACT90oFyTgnV60GhNLdstOIcFET3IVANCA','/xjs/_/js/kx3dxjs.hp.en_US.4dB-kXZgo4g.O/mx3dsb_he,d/rtx3dj/dx3d1/tx3dzcms/rsx3dACT90oFyTgnV60GhNLdstOIcFET3IVANCA');google.xjs=1;}google.pmc={"sb_he":{"agen":true,"cgen":true,"client":"heirloom-hp","dh":true,"ds":"","exp":"msedr","fl":true,"host":"google.com","jam":0,"jsonp":true,"msgs":{"cibl":"Clear Search","dym":"Did you mean:","lcky":"Iu0026#39;m Feeling Lucky","lml":"Learn more","oskt":"Input tools","psrc":"This search was removed from your u003Ca href="/history"u003EWeb Historyu003C/au003E","psrl":"Remove","sbit":"Search by image","srch":"Google Search"},"ovr":{},"pq":"","refoq":true,"scd":10,"sce":5,"stok":"iXw-xWnUXlH7Fp6SrUErmgr3X8g"},"d":{}};google.y.first.push(function(){if(google.med){google.med('init');google.initHistory();google.med('history');}});if(google.j&&google.j.en&&google.j.xi){window.setTimeout(google.j.xi,0);}
</script></div></body></html>
UPDATE 2021: Looks like this no longer works. Google deprecated this a while back. They instead recommend you use their new Custom Search API but this now requires you to input an API key, which is less convenient than the old public endpoint.
Replace the beginning of your url with http://ajax.googleapis.com/ajax/services/search/web?v=1.0
It should now looks like this:
http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=urban+outfitters+facebook
Using the Python json parser you can retrieve the first url.
import requests
import json
url = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=urban+outfitters+facebook"
google_result = json.loads(requests.get(url).text)
print google_result["responseData"]["results"][0]["url"]
Since Laurent’s old answer didn’t work (and I didn’t want to use the API) I searched for another method and found the one from here: https://hackernoon.com/how-to-scrape-google-with-python-bo7d2tal
import requests
from bs4 import BeautifulSoup
query = "hackernoon How To Scrape Google With Python"
query = query.replace(' ', '+')
URL = f"https://google.com/search?q={query}"
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0"
headers = {"user-agent" : USER_AGENT}
resp = requests.get(URL, headers=headers)
if resp.status_code == 200:
soup = BeautifulSoup(resp.content, "html.parser")
Using the ‘soup’ everything else can be extracted from the html.
Make sure you’re using user-agent because Google treats your script as python-requests. That was a problem you faced. You need to fake a real user visit with user-agent. List of user-agents.
Code and example in online IDE:
from bs4 import BeautifulSoup
import requests, lxml
headers = {
'User-agent':
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}
html = requests.get('https://www.google.com/search?q=urban outfitters facebook', headers=headers).text
soup = BeautifulSoup(html, 'lxml')
for result in soup.select('.yuRUbf'):
title = result.select_one('.DKV0Md').text
link = result.select_one('a')['href']
print(f'{title}n{link}n')
Output:
Urban Outfitters - Verified Page | Facebook
https://www.facebook.com/urbanoutfitters/
Urban Outfitters - Home | Facebook
https://www.facebook.com/urbanoutfitterseurope/
Urban Outfitters - Facebook - Urban Outfitters - Blog
https://blog.urbanoutfitters.com/facebook
Contact Us - Urban Outfitters
https://www.urbanoutfitters.com/en-gb/help/contact-us
Frank & Funny Facebook Card | Urban Outfitters
https://www.urbanoutfitters.com/shop/frank--funny-facebook-card?color=&parentid=SALE_APT&quantity=1&type=REGULAR
Urban Outfitters
https://www.urbanoutfitters.com/
Urban Outfitters (@urbanoutfitters) • Instagram photos and ...
https://www.instagram.com/urbanoutfitters/
Alternatively, you can achieve these results using Google Search Engine Results API from SerpApi. It’s a paid API with a free plan to test out.
If you want to get just a RAW-HTML link, you can call ['search_metadata']['raw_html_file'] or just print(results) to get all data.
Code to integrate:
import os
from serpapi import GoogleSearch
params = {
"engine": "google",
"q": "urban outfitters facebook",
"api_key": os.getenv("API_KEY"),
}
search = GoogleSearch(params)
results = search.get_dict()
html = results['search_metadata']['raw_html_file']
print(f'Raw HTML: {html}')
print(json.dumps(results, indent = 2, ensure_ascii = False))
Output:
https://serpapi.com/searches/f4b6c93572fd22f8/609cb4f5c640d28919b34dad.html
...
"organic_results": [
{
"position": 1,
"title": "Urban Outfitters - Verified Page | Facebook",
"link": "https://www.facebook.com/urbanoutfitters/",
"displayed_link": "https://www.facebook.com › ... › Clothing (Brand)",
"snippet": "Urban Outfitters. 2187043 likes · 4157 talking about this · 169423 were here. Visit us at www.urbanoutfitters.com. Always open, always awesome.",
"sitelinks": {
"expanded": [
{
"title": "Urban Outfitters",
"link": "https://en-gb.facebook.com/urbanoutfitters",
"snippet": "Urban Outfitters. 2187043 likes · 4259 talking about this ..."
},
{
"title": "Instagram",
"link": "https://www.facebook.com/urbanoutfitters/app/168188869963563/",
"snippet": "Block Page. More. Send Message. See more of Urban Outfitters on ..."
},
{
"title": "About",
"link": "https://www.facebook.com/urbanoutfitters/about/",
"snippet": "The official Facebook page of Urban Outfitters. Questions ..."
},
{
"title": "Events",
"link": "https://www.facebook.com/urbanoutfitters/events/",
"snippet": "Urban Outfitters does not have any upcoming events. Past Events ..."
}
]
}
}
]...
Disclaimer, I work for SerpApi.