How to scrape only visible webpage text with BeautifulSoup?
Question:
Basically, I want to use BeautifulSoup to grab strictly the visible text on a webpage. For instance, this webpage is my test case. And I mainly want to just get the body text (article) and maybe even a few tab names here and there. I have tried the suggestion in this SO question that returns lots of <script> tags and html comments which I don’t want. I can’t figure out the arguments I need for the function findAll() in order to just get the visible texts on a webpage.
So, how should I find all visible text excluding scripts, comments, css etc.?
Answers:
The title is inside an <nyt_headline> tag, which is nested inside an <h1> tag and a <div> tag with id “article”.
soup.findAll('nyt_headline', limit=1)
Should work.
The article body is inside an <nyt_text> tag, which is nested inside a <div> tag with id “articleBody”. Inside the <nyt_text> element, the text itself is contained within <p> tags. Images are not within those <p> tags. It’s difficult for me to experiment with the syntax, but I expect a working scrape to look something like this.
text = soup.findAll('nyt_text', limit=1)[0]
text.findAll('p')
Try this:
from bs4 import BeautifulSoup
from bs4.element import Comment
import urllib.request
def tag_visible(element):
if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']:
return False
if isinstance(element, Comment):
return False
return True
def text_from_html(body):
soup = BeautifulSoup(body, 'html.parser')
texts = soup.findAll(text=True)
visible_texts = filter(tag_visible, texts)
return u" ".join(t.strip() for t in visible_texts)
html = urllib.request.urlopen('http://www.nytimes.com/2009/12/21/us/21storm.html').read()
print(text_from_html(html))
The approved answer from @jbochi does not work for me. The str() function call raises an exception because it cannot encode the non-ascii characters in the BeautifulSoup element. Here is a more succinct way to filter the example web page to visible text.
html = open('21storm.html').read()
soup = BeautifulSoup(html)
[s.extract() for s in soup(['style', 'script', '[document]', 'head', 'title'])]
visible_text = soup.getText()
I completely respect using Beautiful Soup to get rendered content, but it may not be the ideal package for acquiring the rendered content on a page.
I had a similar problem to get rendered content, or the visible content in a typical browser. In particular I had many perhaps atypical cases to work with such a simple example below. In this case the non displayable tag is nested in a style tag, and is not visible in many browsers that I have checked. Other variations exist such as defining a class tag setting display to none. Then using this class for the div.
<html>
<title> Title here</title>
<body>
lots of text here <p> <br>
<h1> even headings </h1>
<style type="text/css">
<div > this will not be visible </div>
</style>
</body>
</html>
One solution posted above is:
html = Utilities.ReadFile('simple.html')
soup = BeautifulSoup.BeautifulSoup(html)
texts = soup.findAll(text=True)
visible_texts = filter(visible, texts)
print(visible_texts)
[u'n', u'n', u'nn lots of text here ', u' ', u'n', u' even headings ', u'n', u' this will not be visible ', u'n', u'n']
This solution certainly has applications in many cases and does the job quite well generally but in the html posted above it retains the text that is not rendered. After searching SO a couple solutions came up here BeautifulSoup get_text does not strip all tags and JavaScript and here Rendered HTML to plain text using Python
I tried both these solutions: html2text and nltk.clean_html and was surprised by the timing results so thought they warranted an answer for posterity. Of course, the speeds highly depend on the contents of the data…
One answer here from @Helge was about using nltk of all things.
import nltk
%timeit nltk.clean_html(html)
was returning 153 us per loop
It worked really well to return a string with rendered html. This nltk module was faster than even html2text, though perhaps html2text is more robust.
betterHTML = html.decode(errors='ignore')
%timeit html2text.html2text(betterHTML)
%3.09 ms per loop
import urllib
from bs4 import BeautifulSoup
url = "https://www.yahoo.com"
html = urllib.urlopen(url).read()
soup = BeautifulSoup(html)
# kill all script and style elements
for script in soup(["script", "style"]):
script.extract() # rip it out
# get text
text = soup.get_text()
# break into lines and remove leading and trailing space on each
lines = (line.strip() for line in text.splitlines())
# break multi-headlines into a line each
chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
# drop blank lines
text = 'n'.join(chunk for chunk in chunks if chunk)
print(text.encode('utf-8'))
While, i would completely suggest using beautiful-soup in general, if anyone is looking to display the visible parts of a malformed html (e.g. where you have just a segment or line of a web-page) for whatever-reason, the the following will remove content between < and > tags:
import re ## only use with malformed html - this is not efficient
def display_visible_html_using_re(text):
return(re.sub("(<.*?>)", "",text))
Using BeautifulSoup the easiest way with less code to just get the strings, without empty lines and crap.
tag = <Parent_Tag_that_contains_the_data>
soup = BeautifulSoup(tag, 'html.parser')
for i in soup.stripped_strings:
print repr(i)
If you care about performance, here’s another more efficient way:
import re
INVISIBLE_ELEMS = ('style', 'script', 'head', 'title')
RE_SPACES = re.compile(r's{3,}')
def visible_texts(soup):
""" get visible text from a document """
text = ' '.join([
s for s in soup.strings
if s.parent.name not in INVISIBLE_ELEMS
])
# collapse multiple spaces to two spaces.
return RE_SPACES.sub(' ', text)
soup.strings is an iterator, and it returns NavigableString so that you can check the parent’s tag name directly, without going through multiple loops.
The simplest way to handle this case is by using getattr(). You can adapt this example to your needs:
from bs4 import BeautifulSoup
source_html = """
<span class="ratingsDisplay">
<a class="ratingNumber" href="https://www.youtube.com/watch?v=oHg5SJYRHA0" target="_blank" rel="noopener">
<span class="ratingsContent">3.7</span>
</a>
</span>
"""
soup = BeautifulSoup(source_html, "lxml")
my_ratings = getattr(soup.find('span', {"class": "ratingsContent"}), "text", None)
print(my_ratings)
This will find the text element,"3.7", within the tag object <span class="ratingsContent">3.7</span> when it exists, however, default to NoneType when it does not.
getattr(object, name[, default])
Return the value of the named attribute of object. name must be a string. If the string is the name of one of the object’s attributes, the result is the value of that attribute. For example, getattr(x, ‘foobar’) is equivalent to x.foobar. If the named attribute does not exist, default is returned if provided, otherwise, AttributeError is raised.
from bs4 import BeautifulSoup
from bs4.element import Comment
import urllib.request
import re
import ssl
def tag_visible(element):
if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']:
return False
if isinstance(element, Comment):
return False
if re.match(r"[n]+",str(element)): return False
return True
def text_from_html(url):
body = urllib.request.urlopen(url,context=ssl._create_unverified_context()).read()
soup = BeautifulSoup(body ,"lxml")
texts = soup.findAll(text=True)
visible_texts = filter(tag_visible, texts)
text = u",".join(t.strip() for t in visible_texts)
text = text.lstrip().rstrip()
text = text.split(',')
clean_text = ''
for sen in text:
if sen:
sen = sen.rstrip().lstrip()
clean_text += sen+','
return clean_text
url = 'http://www.nytimes.com/2009/12/21/us/21storm.html'
print(text_from_html(url))
UPDATE
To get all the human readable text of the HTML <body> you can use .get_text(), to get rid of redundant whitespaces, etc. set strip parameter and join/separate all by a single whitespace:
import bs4, requests
response = requests.get('https://www.nytimes.com/interactive/2022/09/13/us/politics/congress-stock-trading-investigation.html',headers={'User-Agent': 'Mozilla/5.0','cache-control': 'max-age=0'}, cookies={'cookies':''})
soup = bs4.BeautifulSoup(response.text)
soup.article.get_text(' ', strip=True)
In newer code avoid old syntax findAll() instead use find_all() or select() with css selectors – For more take a minute to check docs
Basically, I want to use BeautifulSoup to grab strictly the visible text on a webpage. For instance, this webpage is my test case. And I mainly want to just get the body text (article) and maybe even a few tab names here and there. I have tried the suggestion in this SO question that returns lots of <script> tags and html comments which I don’t want. I can’t figure out the arguments I need for the function findAll() in order to just get the visible texts on a webpage.
So, how should I find all visible text excluding scripts, comments, css etc.?
The title is inside an <nyt_headline> tag, which is nested inside an <h1> tag and a <div> tag with id “article”.
soup.findAll('nyt_headline', limit=1)
Should work.
The article body is inside an <nyt_text> tag, which is nested inside a <div> tag with id “articleBody”. Inside the <nyt_text> element, the text itself is contained within <p> tags. Images are not within those <p> tags. It’s difficult for me to experiment with the syntax, but I expect a working scrape to look something like this.
text = soup.findAll('nyt_text', limit=1)[0]
text.findAll('p')
Try this:
from bs4 import BeautifulSoup
from bs4.element import Comment
import urllib.request
def tag_visible(element):
if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']:
return False
if isinstance(element, Comment):
return False
return True
def text_from_html(body):
soup = BeautifulSoup(body, 'html.parser')
texts = soup.findAll(text=True)
visible_texts = filter(tag_visible, texts)
return u" ".join(t.strip() for t in visible_texts)
html = urllib.request.urlopen('http://www.nytimes.com/2009/12/21/us/21storm.html').read()
print(text_from_html(html))
The approved answer from @jbochi does not work for me. The str() function call raises an exception because it cannot encode the non-ascii characters in the BeautifulSoup element. Here is a more succinct way to filter the example web page to visible text.
html = open('21storm.html').read()
soup = BeautifulSoup(html)
[s.extract() for s in soup(['style', 'script', '[document]', 'head', 'title'])]
visible_text = soup.getText()
I completely respect using Beautiful Soup to get rendered content, but it may not be the ideal package for acquiring the rendered content on a page.
I had a similar problem to get rendered content, or the visible content in a typical browser. In particular I had many perhaps atypical cases to work with such a simple example below. In this case the non displayable tag is nested in a style tag, and is not visible in many browsers that I have checked. Other variations exist such as defining a class tag setting display to none. Then using this class for the div.
<html>
<title> Title here</title>
<body>
lots of text here <p> <br>
<h1> even headings </h1>
<style type="text/css">
<div > this will not be visible </div>
</style>
</body>
</html>
One solution posted above is:
html = Utilities.ReadFile('simple.html')
soup = BeautifulSoup.BeautifulSoup(html)
texts = soup.findAll(text=True)
visible_texts = filter(visible, texts)
print(visible_texts)
[u'n', u'n', u'nn lots of text here ', u' ', u'n', u' even headings ', u'n', u' this will not be visible ', u'n', u'n']
This solution certainly has applications in many cases and does the job quite well generally but in the html posted above it retains the text that is not rendered. After searching SO a couple solutions came up here BeautifulSoup get_text does not strip all tags and JavaScript and here Rendered HTML to plain text using Python
I tried both these solutions: html2text and nltk.clean_html and was surprised by the timing results so thought they warranted an answer for posterity. Of course, the speeds highly depend on the contents of the data…
One answer here from @Helge was about using nltk of all things.
import nltk
%timeit nltk.clean_html(html)
was returning 153 us per loop
It worked really well to return a string with rendered html. This nltk module was faster than even html2text, though perhaps html2text is more robust.
betterHTML = html.decode(errors='ignore')
%timeit html2text.html2text(betterHTML)
%3.09 ms per loop
import urllib
from bs4 import BeautifulSoup
url = "https://www.yahoo.com"
html = urllib.urlopen(url).read()
soup = BeautifulSoup(html)
# kill all script and style elements
for script in soup(["script", "style"]):
script.extract() # rip it out
# get text
text = soup.get_text()
# break into lines and remove leading and trailing space on each
lines = (line.strip() for line in text.splitlines())
# break multi-headlines into a line each
chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
# drop blank lines
text = 'n'.join(chunk for chunk in chunks if chunk)
print(text.encode('utf-8'))
While, i would completely suggest using beautiful-soup in general, if anyone is looking to display the visible parts of a malformed html (e.g. where you have just a segment or line of a web-page) for whatever-reason, the the following will remove content between < and > tags:
import re ## only use with malformed html - this is not efficient
def display_visible_html_using_re(text):
return(re.sub("(<.*?>)", "",text))
Using BeautifulSoup the easiest way with less code to just get the strings, without empty lines and crap.
tag = <Parent_Tag_that_contains_the_data>
soup = BeautifulSoup(tag, 'html.parser')
for i in soup.stripped_strings:
print repr(i)
If you care about performance, here’s another more efficient way:
import re
INVISIBLE_ELEMS = ('style', 'script', 'head', 'title')
RE_SPACES = re.compile(r's{3,}')
def visible_texts(soup):
""" get visible text from a document """
text = ' '.join([
s for s in soup.strings
if s.parent.name not in INVISIBLE_ELEMS
])
# collapse multiple spaces to two spaces.
return RE_SPACES.sub(' ', text)
soup.strings is an iterator, and it returns NavigableString so that you can check the parent’s tag name directly, without going through multiple loops.
The simplest way to handle this case is by using getattr(). You can adapt this example to your needs:
from bs4 import BeautifulSoup
source_html = """
<span class="ratingsDisplay">
<a class="ratingNumber" href="https://www.youtube.com/watch?v=oHg5SJYRHA0" target="_blank" rel="noopener">
<span class="ratingsContent">3.7</span>
</a>
</span>
"""
soup = BeautifulSoup(source_html, "lxml")
my_ratings = getattr(soup.find('span', {"class": "ratingsContent"}), "text", None)
print(my_ratings)
This will find the text element,"3.7", within the tag object <span class="ratingsContent">3.7</span> when it exists, however, default to NoneType when it does not.
getattr(object, name[, default])Return the value of the named attribute of object. name must be a string. If the string is the name of one of the object’s attributes, the result is the value of that attribute. For example, getattr(x, ‘foobar’) is equivalent to x.foobar. If the named attribute does not exist, default is returned if provided, otherwise, AttributeError is raised.
from bs4 import BeautifulSoup
from bs4.element import Comment
import urllib.request
import re
import ssl
def tag_visible(element):
if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']:
return False
if isinstance(element, Comment):
return False
if re.match(r"[n]+",str(element)): return False
return True
def text_from_html(url):
body = urllib.request.urlopen(url,context=ssl._create_unverified_context()).read()
soup = BeautifulSoup(body ,"lxml")
texts = soup.findAll(text=True)
visible_texts = filter(tag_visible, texts)
text = u",".join(t.strip() for t in visible_texts)
text = text.lstrip().rstrip()
text = text.split(',')
clean_text = ''
for sen in text:
if sen:
sen = sen.rstrip().lstrip()
clean_text += sen+','
return clean_text
url = 'http://www.nytimes.com/2009/12/21/us/21storm.html'
print(text_from_html(url))
UPDATE
To get all the human readable text of the HTML <body> you can use .get_text(), to get rid of redundant whitespaces, etc. set strip parameter and join/separate all by a single whitespace:
import bs4, requests
response = requests.get('https://www.nytimes.com/interactive/2022/09/13/us/politics/congress-stock-trading-investigation.html',headers={'User-Agent': 'Mozilla/5.0','cache-control': 'max-age=0'}, cookies={'cookies':''})
soup = bs4.BeautifulSoup(response.text)
soup.article.get_text(' ', strip=True)
In newer code avoid old syntax findAll() instead use find_all() or select() with css selectors – For more take a minute to check docs