BeautifulSoup not returning links

Question

For my python bootcamp I am trying to create a log of the articles from this site, and return the highest upvoted. The rest of the code works, but I cannot get it to return the href properly. I get "none." I have tried everything I know to do… can anyone provide any guidance?

from bs4 import BeautifulSoup
import requests


response = requests.get("https://news.ycombinator.com/")
yc_web_page = response.text


soup = BeautifulSoup(yc_web_page, "html.parser")
articles = soup.find_all(name="span", class_="titleline")

article_texts = []
article_links = []

for article_tag in articles:

    article_text = article_tag.get_text()
    article_texts.append(article_text)

    article_link = article_tag.get("href")
    article_links.append(article_link)



article_upvotes = [int(score.getText().split()[0]) for score in soup.find_all(name="span", class_="score")]


largest_number = max(article_upvotes)
largest_index = article_upvotes.index(largest_number)

print(article_texts[largest_index])
print(article_links[largest_index])
print(article_upvotes[largest_index])`

I have tried to change the ‘href’ to just an ‘a’ tag and it returned the same value of "none"

Asked By: Thomas Hudson

||

Source

Answer 1

Try:


...

    article_link = article_tag.a.get("href")    # <--- put .a here

...

from bs4 import BeautifulSoup
import requests


response = requests.get("https://news.ycombinator.com/")
yc_web_page = response.text


soup = BeautifulSoup(yc_web_page, "html.parser")
articles = soup.find_all(name="span", class_="titleline")

article_texts = []
article_links = []

for article_tag in articles:

    article_text = article_tag.get_text()
    article_texts.append(article_text)

    article_link = article_tag.a.get("href")   # <--- put .a here
    article_links.append(article_link)


article_upvotes = [
    int(score.getText().split()[0])
    for score in soup.find_all(name="span", class_="score")
]


largest_number = max(article_upvotes)
largest_index = article_upvotes.index(largest_number)

print(article_texts[largest_index])
print(article_links[largest_index])
print(article_upvotes[largest_index])

Prints:

Fred Brooks has died (twitter.com/stevebellovin)
Sad news from @unccs — Fred P. Brooks, the founder and long-time chair of the department (and a major influence on my professional outlook) passed away a few hours ago.
— Steven M. Bellovin (@SteveBellovin) November 18, 2022

1368

Answered By: Andrej Kesely

Answer 2

Here’s a bit shorter approach:

import requests
from bs4 import BeautifulSoup

url = "https://news.ycombinator.com/"

soup = BeautifulSoup(requests.get(url).text, "lxml")

all_scores = [
    [
        int(x.getText().replace(" points", "")),
        x["id"].replace("score_", ""),
    ]
    for x in soup.find_all("span", class_="score")
]

votes, tr_id = sorted(all_scores, key=lambda x: x[0], reverse=True)[0]

table_row = soup.find("tr", id=tr_id)
text = table_row.select_one("span a").getText()
link = table_row.select_one("span a")["href"]

print(f"{text}n{link}n{votes} votes")

Output:

Fred Brooks has died
Sad news from @unccs — Fred P. Brooks, the founder and long-time chair of the department (and a major influence on my professional outlook) passed away a few hours ago.
— Steven M. Bellovin (@SteveBellovin) November 18, 2022

1377 votes

Answered By: baduker

BeautifulSoup not returning links

Question:

Answers: