How to make output from web scraping python selenium in div class to output text

Question

This is my Code `

from attr import attr
import requests
from bs4 import BeautifulSoup
import csv

datas = []
key = 'sepatu'
jenis = 'teplek'
url = 'https://website.com/search/?term={}+{}'.format(key,jenis)
headers = {
    'user-agent' : 'Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0'
}
req = requests.get(url, headers=headers)
soup = BeautifulSoup (req.text, 'html.parser')
sepatu = soup.find_all('div', 'element_1')
for it in sepatu:
    harga = it.find('div','element').__str__
    datas.append([harga])
hasil = ['Harga'] 
write = csv.writer(open('result/{}_{}.csv'.format(key,jenis), 'w', newline=''))
write.writerow (hasil)
for d in datas: write.writerow(d)

this is Output from this code

Column A
<bound method Tag.unicode of Rp 88.000 >
<bound method Tag.unicode of Rp 200.000 >

how to turn that output to this output

Column A
Rp 88.000
Rp 200.000

i was try
harga = it.find('div','element').__str__ to harga = it.find('div','element').text but i got error AttributeError: 'NoneType' object has no attribute 'text'

i try to learn web scraping python selenium but i got block by output into text and i expecting i want all the output into text

Asked By: Rezky

||

Source

Answers:

Answer 1

Looks like the problem is with the commands you use to locate the elements. If element is not found, BS returns None for them.
What mean element_1 and element in this lines?:

sepatu = soup.find_all('div', 'element_1')

and

harga = it.find('div','element').__str__

If these are IDs, you need to set it this way:

sepatu = soup.find_all('div', {'id': 'element_1'})

If it’s a class name:

sepatu = soup.find_all('div', {'class': 'element_1'})

Or any other attribute:

sepatu = soup.find_all('div', {'attr_name': 'element_1'})

When an element is found you can use .text property to get its text

Answered By: Eugeny Okulik

Answer 2

You can add .text in this line

harga = soup.find("div", {"class": "db gM ei b hE be f16-360-o ff vb uT ellipsis-1"}).text

Then you will get an output like this

Nama Sepatu	Harga
Sepatu A	Rp.24.000

Answered By: indrarasendriya

How to make output from web scraping python selenium in div class to output text

Question:

Answers: