Using Python and BeautifulSoup (saved webpage source codes into a local file)
Question:
I am using Python 2.7 + BeautifulSoup 4.3.2.
I am trying to use Python and BeautifulSoup to pick up information on a webpage. Because the webpage is in the company website and requires login and redirection, I copied the target page’s source code page into a file and saved it as “example.html” in C: for the convenience of practicing.
This is a part of the original code:
<tr class="ghj">
<td><span class="city-sh"><sh src="./citys/1.jpg" alt="boy" title="boy" /></span><a href="./membercity.php?mode=view&u=12563">port_new_cape</a></td>
<td class="position"><a href="./search.php?id=12563&sr=positions" title="Search positions">452</a></td>
<td class="details"><div>South</div></td>
<td>May 09, 1997</td>
<td>Jan 23, 2009 12:05 pm </td>
</tr>
The code I worked out so far is:
from bs4 import BeautifulSoup
import re
import urllib2
url = "C:example.html"
page = urllib2.urlopen(url)
soup = BeautifulSoup(page.read())
cities = soup.find_all('span', {'class' : 'city-sh'})
for city in cities:
print city
This is just the first stage of testing, so it’s somewhat incomplete.
However, when I run it, it gives an error message. Seems it’s improper to use urllib2.urlopen
to open a local file.
Traceback (most recent call last):
File "C:Python27Testing.py", line 8, in <module>
page = urllib2.urlopen(url)
File "C:Python27liburllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "C:Python27liburllib2.py", line 404, in open
response = self._open(req, data)
File "C:Python27liburllib2.py", line 427, in _open
'unknown_open', req)
File "C:Python27liburllib2.py", line 382, in _call_chain
result = func(*args)
File "C:Python27liburllib2.py", line 1247, in unknown_open
raise URLError('unknown url type: %s' % type)
URLError: <urlopen error unknown url type: c>
How can I practice using a local file?
Answers:
With Chandan’s help, the problem has been solved. All the credits shall go to him. 🙂
the “urllib2.url” is useless here.
from bs4 import BeautifulSoup
import re
# import urllib2
url = "C:example.html"
page = open(url)
soup = BeautifulSoup(page.read())
cities = soup.find_all('span', {'class' : 'city-sh'})
for city in cities:
print city
You can try using lxml parser also. Here is an example for your html data.
from lxml.html import fromstring
import lxml.html as PARSER
data = open('example.html').read()
root = PARSER.fromstring(data)
for ele in root.getiterator():
if ele.tag == "td":
print ele.text_content()
o/p:
port_new_cape
452
South
May 09, 1997
Jan 23, 2009 12:05 pm
The best way to open a local file with BeautifulSoup is to pass it a file handler directly. http://www.crummy.com/software/BeautifulSoup/bs4/doc/#making-the-soup
from bs4 import BeautifulSoup
with open("C:\example.html") as fp:
soup = BeautifulSoup(fp, 'html.parser')
for city in soup.find_all('span', {'class' : 'city-sh'}):
print(city)
I am using Python 2.7 + BeautifulSoup 4.3.2.
I am trying to use Python and BeautifulSoup to pick up information on a webpage. Because the webpage is in the company website and requires login and redirection, I copied the target page’s source code page into a file and saved it as “example.html” in C: for the convenience of practicing.
This is a part of the original code:
<tr class="ghj">
<td><span class="city-sh"><sh src="./citys/1.jpg" alt="boy" title="boy" /></span><a href="./membercity.php?mode=view&u=12563">port_new_cape</a></td>
<td class="position"><a href="./search.php?id=12563&sr=positions" title="Search positions">452</a></td>
<td class="details"><div>South</div></td>
<td>May 09, 1997</td>
<td>Jan 23, 2009 12:05 pm </td>
</tr>
The code I worked out so far is:
from bs4 import BeautifulSoup
import re
import urllib2
url = "C:example.html"
page = urllib2.urlopen(url)
soup = BeautifulSoup(page.read())
cities = soup.find_all('span', {'class' : 'city-sh'})
for city in cities:
print city
This is just the first stage of testing, so it’s somewhat incomplete.
However, when I run it, it gives an error message. Seems it’s improper to use urllib2.urlopen
to open a local file.
Traceback (most recent call last):
File "C:Python27Testing.py", line 8, in <module>
page = urllib2.urlopen(url)
File "C:Python27liburllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "C:Python27liburllib2.py", line 404, in open
response = self._open(req, data)
File "C:Python27liburllib2.py", line 427, in _open
'unknown_open', req)
File "C:Python27liburllib2.py", line 382, in _call_chain
result = func(*args)
File "C:Python27liburllib2.py", line 1247, in unknown_open
raise URLError('unknown url type: %s' % type)
URLError: <urlopen error unknown url type: c>
How can I practice using a local file?
With Chandan’s help, the problem has been solved. All the credits shall go to him. 🙂
the “urllib2.url” is useless here.
from bs4 import BeautifulSoup
import re
# import urllib2
url = "C:example.html"
page = open(url)
soup = BeautifulSoup(page.read())
cities = soup.find_all('span', {'class' : 'city-sh'})
for city in cities:
print city
You can try using lxml parser also. Here is an example for your html data.
from lxml.html import fromstring
import lxml.html as PARSER
data = open('example.html').read()
root = PARSER.fromstring(data)
for ele in root.getiterator():
if ele.tag == "td":
print ele.text_content()
o/p:
port_new_cape
452
South
May 09, 1997
Jan 23, 2009 12:05 pm
The best way to open a local file with BeautifulSoup is to pass it a file handler directly. http://www.crummy.com/software/BeautifulSoup/bs4/doc/#making-the-soup
from bs4 import BeautifulSoup
with open("C:\example.html") as fp:
soup = BeautifulSoup(fp, 'html.parser')
for city in soup.find_all('span', {'class' : 'city-sh'}):
print(city)