Webscraping a discussion board
Question:
I need to scrape all the questions from the first 5 pages of the following website: https://www.dell.com/community/Inspiron/bd-p/Inspiron
I need to return the discussion title, author, time of question, and who responded last into a csv file.
Could someone get me started? I don’t have any code as of yet. I’m also not sure where to start. There are too many pieces in this problem for me to figure out.
Thank you! It doesn’t have to be the full code, but a starting place would be very helpful. I know I’m supposed to use requests and BeautifulSoup
Edit: I don’t have a TA to ask for help. That’s why I’ve posted here.
Answers:
You should try Scrapy, a free and open-source web-crawling framework.
You can check it out here: https://scrapy.org/
The solution might look like the example bellow:
# -*- coding: utf-8 -*-
import scrapy
class ToScrapeSpiderXPath(scrapy.Spider):
name = 'toscrape-xpath'
start_urls = [
'http://quotes.toscrape.com/',
]
def parse(self, response):
for quote in response.xpath('//div[@class="quote"]'):
yield {
'text': quote.xpath('./span[@class="text"]/text()').extract_first(),
'author': quote.xpath('.//small[@class="author"]/text()').extract_first(),
'tags': quote.xpath('.//div[@class="tags"]/a[@class="tag"]/text()').extract()
}
next_page_url = response.xpath('//li[@class="next"]/a/@href').extract_first()
if next_page_url is not None:
yield scrapy.Request(response.urljoin(next_page_url))
Example from: https://github.com/scrapy/quotesbot
You can use requests to retrieve the page source, and BeautifulSoup to parse the results. To find the question, iterate over each tr with a class containing the question id:
from bs4 import BeautifulSoup as soup
import requests, re
def scrape_page(page_num):
d = soup(requests.get('https://www.dell.com/community/Inspiron/bd-p/Inspiron/page/{}'.format(page_num)).text, 'html.parser')
q = d.find_all('tr', {'class':re.compile('lia-js-data-messageUid-d+')}) #find all question trs
result = [[i.find('a', {'class':'page-link'}).text]+[c.text for c in i.find_all('a', {'class':'lia-user-name-link'})]+[i.find('span', {'class':'lia-component-common-widget-date'}).text] for i in q]
return [[re.sub('n|s{2,}', '', c) for c in i] for i in result]
The result for each page is a list of lists containing the title, author, last comment poster, and the date the question was asked. To find the first five pages:
results = [scrape_page(i) for i in range(1, 6)]
Output:
[[['Inspiron gaming laptops, battery drain is normal', 'DELL-Chris M', 'u200e11-30-201808:36 AM'], ['Where are my Inspiron Laptop manuals?', 'DELL-Chris M', 'u200e05-02-201809:53 AM'], ['Where are my Inspiron Laptop drivers?', 'DELL-Chris M', 'u200e05-02-201809:50 AM'], ['New Inspiron Laptop Usage and Troubleshooting Guides (3/2/18)', 'DELL-Chris M', 'u200e03-02-201810:28 AM'], ['Can not install dell mobile connect aplication from windows store', 'GeoNeo77', 'Mary G', 'u200e04-11-201908:20 AM'], ['Inspiron 13 7000 (7370) Black / Blank Screen / Display', 'brianms', 'Erutan409', 'u200e11-14-201805:46 PM'], ['Boot From External USB Drive or USB Stick', 'borntodeal', 'u200e04-11-201910:01 AM'], ['WiFi suddenly disappeared', 'Anagno7', 'u200e04-11-201909:50 AM'], ['Task bar ikons move quickly back and forth', 'ralphspajka', 'ralphspajka', 'u200e04-11-201907:06 AM'], ['Does an Intel Dual Band Wireless-AC 8265 or 9260 will work on a Inspiron 5555', 'Cauchons', 'jphughan', 'u200e04-07-201910:43 AM'], ['Fan not working', 'Tunay', 'JOcean', 'u200e04-10-201903:35 PM'], ['Upgraded Wireless N 7260 WIFI Card', 'fadsarmy', 'fadsarmy', 'u200e03-30-201909:13 PM'], ['XPS 13-9365, battery issues!', 'rickmanhatan', 'DELL-Chris M', 'u200e04-11-201903:18 AM'], ['Inspiron 17 7779, turns black after playing games', 'Prime7105', 'Teetertotter', 'u200e04-09-201908:12 PM'], ['New 7559 BIOS is corrupted', 'Suncatcher_13', 'Teetertotter', 'u200e02-26-201906:25 PM'], ['Inspiron 15 7559 compatible con SSD Samsung 970 EVO Plus?', 'defto', 'Saltgrass', 'u200e04-10-201908:40 PM'], ['Laptop wont turn on...', 'TonicWhisper', 'JOcean', 'u200e04-10-201911:40 PM'], ['Why is my Dell G7 buzzing when charging', 'Andre Dolar', 'Jey_31', 'u200e09-24-201806:09 AM'], ['Clone only OS partition to SSD/Install windows 10 on new SSD', 'HemantSangle', 'hezi', 'u200e09-26-201803:02 AM'], ['ERROR Alienware Command Center en G7 17 7790', 'Matafleur', 'SERGIO.MAGNO', 'u200e03-16-201908:03 PM'], ['Inspiron 15-7567, M.2 NVMe SSD, failed boot device', 'JohnJakeLindert', 'Saltgrass', 'u200e03-31-201902:02 PM'], ['Dell adapter not detected. Plugged in, not charging.', 'spbone', 'acmoo1964', 'u200e01-23-201802:11 PM'], ['Inspiron 5482 keyboard skipping characters, missing strokes. HELP! :(', 'NajibOzy', 'NajibOzy', 'u200e04-10-201902:43 AM'], ['Bluetooth Inspiron 14 3000 Demora demais para conectar com o fone.', 'VitorVitalino', 'Dell-SreejithR', 'u200e04-08-201912:45 PM'], ['Inspiron 13-7000 7359', 'jjrapy', 'dell-robert p', 'u200e04-08-201903:49 PM'], ['DELL INSPIRON i5 5570 CPU OVERHEATING', 'ryanmcgrath', 'Dell-SreejithR', 'u200e04-05-201911:30 PM'], ['Inspiron 24 5000 problem', 'cm_a', 'DELL-Chris M', 'u200e04-04-201912:07 AM'], ['Inspiron G5 15-5587, WiFi drops, then connects itself', 'abhar22', 'abhar22', 'u200e04-02-201907:16 PM'], ['Dell G5 15 5587 Overheating Issue', 'eldarinn', 'Dell-SreejithR', 'u200e03-06-201902:38 AM'], ['Inspiron 15, WiFi not detected', 'savaman', 'DELL-Chris M', 'u200e04-05-201911:35 AM'], ['Dell Inspiron 13 7000 P83G Will Not Wake After Sleep', 'random0o', 'dell-robert p', 'u200e04-06-201905:51 PM'], ['Inspiron 14, 2.1 USB sound kit', 'GuilhermeBil', 'DELL-Chris M', 'u200e04-05-201902:32 PM'], ['Help with display', 'dannyetlv', 'Mary G', 'u200e04-09-201902:00 PM'], ['Inspiron 15 5567 battery drains when shutdown', 'Razaaque', 'Dell-SreejithR', 'u200e03-21-201903:28 AM']], [['If the charger is plugged in at 100% battery life will it harm my battery life in any way?', 'Burlex', 'dell-robert p', 'u200e04-05-201911:48 AM'], ['Audio problems after Dell update........ EVERY TIME', 'Skaboudles', 'Saltgrass', 'u200e04-08-201909:50 AM'], ['Dell Inspiron 7548 Ram', 'Foofuu', 'JOcean', 'u200e04-09-201905:27 PM'], ['Inspiron 17 5770 max size of RAM (memory)', 'leks12lk', 'JOcean', 'u200e04-10-201912:16 AM'], ['Inspiron G3 15-3579, randomly freezes', 'livictor', 'Mike_Space', 'u200e10-17-201806:59 PM'], ['audio renderer error', 'Shoun0305', 'anttisum', 'u200e01-24-201911:41 PM'], ['Dell G5 5587 Sound issues', 'NickBk', 'Cptntrips', 'u200e02-15-201911:49 AM'], ['Inspiron G7 15-7588, speaker is not plugged after updating drivers', 'Thai Hung', 'Thai Hung', 'u200e04-07-201908:30 PM'], ["Dell Inspiron 15 5000 fan on constantly when connected to power but won't boot", '996sleepless', 'u200e04-10-201901:22 AM'], ['Inspiron 17 7000 - Headset not an option?', 'Vildpear', 'Vildpear', 'u200e02-14-201902:07 AM'], ['Dell Inspiron 5575 ryzen 5 vega 8 keeps throttling', 'FIDERIKS', 'u200e04-10-201912:49 AM'], ['Inspiron 15 5575 AMD M.2 NVME slow speed', 'Billdag', 'jphughan', 'u200e04-09-201905:16 PM'], ['hdd not detected inspiron 7573 after BIOs upgrade', 'abylan', 'abylan', 'u200e04-06-201902:11 PM'], ['Dell Inspiron 14 3421-Dell Wireless 1703 Wlan Card compatibility', '5GHzstarve', '5GHzstarve', 'u200e04-09-201907:43 AM'], ['Dell Inspiron 15 5000 laptop super slow', 'DH9500', 'u200e04-09-201905:55 PM'], ['USB Issue with TB18DC', 'AmazingDell', 'u200e04-09-201905:50 PM'], ['Nvidia 1060 max-q temperature in Dell Inspiron 7577', 'Villn52', 'u200e04-09-201905:49 PM'], ['Wifi issues with dell Inspiron 15 7000', 'Desperate4help', 'Zcarcath', 'u200e11-15-201804:23 AM'], ['Inspiron 15 7559 - Battery light randomly flickering', 'AstralC', 'AstralC', 'u200e04-09-201910:37 AM'], ['Intermittent noises from hard drive', 'SV126CBC', 'nyc10036', 'u200e04-09-201902:03 PM'], ['Smartbyte reinstalls itself?! How to make it stop?', 'krgman101', 'RivetNetworks', 'u200e04-08-201905:21 AM'], ['Mouse/Keyboard input overlap issue', 'Goodnight guy', 'Sgtsparkles', 'u200e04-08-201904:56 PM'], ['Inspiron 15-7567, reed switch?', 'shawanish8', 'Furrydaus', 'u200e04-01-201911:29 PM'], ['inspiron 15-5580', 'GOWTHAMAN RAVI', 'u200e04-09-201910:29 AM'], ['Inspiron 7580, CPU temperature', 'pl66', 'pl66', 'u200e04-02-201910:50 AM'], ['Ranura m2 en dell inspiron 15 3000 3567', 'Nico.eguia', 'Dell-SreejithR', 'u200e04-03-201909:10 AM'], ['Dell PC keeps booting up to diagnostic boot test', 'a__biee', 'Tiburon66', 'u200e10-09-201804:17 AM'], ['How To Disable Adaptive Brightness in DELL Insipiron 7580 laptop', 'Addyzaf', 'pl66', 'u200e04-09-201910:18 AM'], ['G3 3579 Fan makes clicking noise at low rpm', 'a2375658', 'Dell-SreejithR', 'u200e02-20-201910:27 PM'], ["Which Inspiron's have DBC?", 'Vortex--5', 'DELL-Chris M', 'u200e04-03-201905:27 AM']], [['backlit keyboard assembly for Inspiron 11-3185', 'mrsbranj', 'dell-robert p', 'u200e04-03-201903:57 AM'], ['Re: Inspiron 7580 fan noise and heating issue on m', 'Addyzaf', 'dell-robert p', 'u200e04-07-201910:41 AM'], ['Inspiron G7 15-7588, second monitor not working', 'Movie1011', 'DELL-Chris M', 'u200e04-08-201911:40 AM'], ['The display is okay?', 'dannyetlv', 'u200e04-09-201910:45 AM'], ['bios update problem', 'yodas91', 'JLJM', 'u200e02-08-201808:13 AM'], ["Inspiron 13 7000 (7370) Display Won't Wake Up", 'brianms', 'brianms', 'u200e11-15-201809:27 AM'], ['2019 G7 7590 Fan Control Abysmal', 'msbtt94', 'u200e04-08-201905:41 PM'], ['Dell G5 Startup problem on battery', 'bob8me', 'Dell-SreejithR', 'u200e02-04-201903:02 PM'], ['Monitor going black', 'uffda123', 'Dell-SreejithR', 'u200e04-07-201912:10 PM'], ['how to system rewrite on Inspiron 3153', 'HD72', 'Philip_Yip', 'u200e04-08-201911:45 AM'], ['I would like to upgrade my Dell Inspiron 17R, any advice?', 'careb', 'nyc10036', 'u200e04-08-201912:32 PM'], ['dell G5 8875 lagging or freezing during playing a video or surfing the internet and fan noise', 'zeyadmourad', 'Dell-SreejithR', 'u200e04-07-201903:42 AM'], ['dell G5 5587 over heating', 'zeyadmourad', 'Dell-SreejithR', 'u200e04-08-201908:49 PM'], ['Took a gamble... WD15 USB-C dock works with Inspiron 17 5770', 'Frozen Mitchell', 'leks12lk', 'u200e03-22-201908:44 PM'], ['Inspiron 7572, Intel 620, Nvidia MX 150 4GB', 'Teetertotter', 'Teetertotter', 'u200e04-06-201909:45 AM'], ['BIOS update will not run on Inspiron 15-3542', 'ai4gk', 'ai4gk', 'u200e04-08-201910:26 PM'], ['Dell inspiron 3558 low wifi signal', 'Gaffar', 'JOcean', 'u200e04-08-201912:45 PM'], ['Inspiron 7359 no video and 2 beeps', 'mek33', 'JOcean', 'u200e04-08-201906:56 PM'], ['Memory and SDD Upgrade for dell N3542', 'bitsrdx', 'JOcean', 'u200e04-09-201904:30 AM'], ['are Dell Inspiron 14" 5458 and 5459 parts interchangable?', 'tristan012345', 'u200e04-09-201903:47 AM'], ['G5 5587 Latest Realtek audio driver breaks sound entirely', 'BarnyTrubble', 'Mark3434', 'u200e02-06-201911:01 PM'], ['G3 3579, IRST Drivers causing data loss on SATA SSD.', 'ariaperdana', 'u200e04-08-201910:28 PM'], ['Inspiron 15-5580.', 'GOWTHAMAN RAVI', 'u200e04-08-201910:04 PM'], ['Dell Inspiron 7370 sleep / wake. Lid closure issues.', 'Harish_Vanguard', 'Nambabwe', 'u200e05-23-201803:50 AM'], ['Inspiron 13 7000', 'jjrapy', 'nyc10036', 'u200e04-08-201903:54 PM'], ['Computer not connecting to network after getting static IP', 'MChe', 'irishgeoff', 'u200e03-04-201805:34 AM'], ['HID Complaint touch screen driver not available', 'VarunDell', 'shahrishit9', 'u200e08-10-201811:35 AM'], ['5559 Touchscreen driver', 'Natalie Settoon', 'kroppykat', 'u200e03-30-201806:55 PM'], ['Support Assist Not installing', 'Kannans243', 'PT8888', 'u200e05-18-201802:06 AM'], ['Keyboard issue', 'SteveTree', 'SteveTree', 'u200e04-07-201904:09 AM']], [['Please fix the C-State bug in Inspiron 11 3000.', 'saudiqbal', 'leemell', 'u200e08-26-201810:27 PM'], ['Dell 7577 overheating', 'qaiser96', 'Dell-SreejithR', 'u200e04-06-201903:57 AM'], ['Dell G3 (3579) BSOD when is plugged into power supply', 'Brssvk', 'Dell-SreejithR', 'u200e04-06-201902:22 AM'], ['Power surge on USB port', 'Lazysocks', 'Mary G', 'u200e04-08-201912:29 PM'], ['Laptop Warranty', 'sophie_ep', 'Mary G', 'u200e04-08-201910:01 AM'], ['Dell updates fail to install after warranty expires', 'JWall61', 'Mary G', 'u200e04-08-201907:31 AM'], ['Inspiron 13 5378 - Touchpad missing from hardware and gestures do not work - Windows 10', 'bjames8787', 'u200e04-08-201910:56 AM'], ['Inspiron 7580, thermals, cooling, fan control', 'pl66', 'pl66', 'u200e04-06-201902:33 AM'], ['RAM upgrade inspiron 5521-6527 my service tag', 'Cihat Eren', 'JOcean', 'u200e04-08-201904:09 AM'], ["New hard drive, dell recovery disk won't go to install menu", 'yt1300don', 'yt1300don', 'u200e04-07-201911:49 PM'], ['Dell Inspiron i5 7000 gaming turning on for a few seconds and immediadtly back off', 'Cmac123', 'speedstep', 'u200e04-07-201909:10 PM'], ['Recover BIOS for Inspiron 5755', 'laltobelli', 'SHColf57', 'u200e03-15-201807:08 PM'], ['Inspiron 7380, loud fan', 'ladrian', 'u200e04-07-201906:43 PM'], ['Power Adapter Error', 'GMMister', 'dell-robert p', 'u200e01-25-201908:35 PM'], ['Inspiron 17 3721 Four Beeps on booting', 'Eddythealien2003', 'fireberd', 'u200e04-08-201902:38 AM'], ['inspiron', 'cosmicdancer', 'evegreen', 'u200e03-11-201903:48 AM'], ['RAM upgrade for Dell Inspiron 5570', 'asadsultanawan', 'Sunray001', 'u200e08-13-201806:28 AM'], ['ssd upgrade dell inspiron 7559', 'Mr Wob', 'Ning-123', 'u200e04-06-201901:32 PM'], ['AC Power Adapter not reading', 'Monemi', 'u200e04-07-201910:46 PM'], ['Battery replacement for Inspiron 5558', 'Chxrlotte', 'ron564848', 'u200e01-02-201905:38 AM'], ['Can Inspiron 5570 boot from NVMe M.2 SSD?', 'valentin_b', 'jamquam', 'u200e07-11-201809:37 AM'], ['Using laptop on direct power without using battery power [built in battery ]', 'RAJR11', 'u200e04-07-201908:56 PM'], ['charger wire colors', 'Shaul G', 'nyc10036', 'u200e04-07-201904:29 AM'], ['Dell Inspiron 13 5378 pen support', 'Joeldamole', 'Joeldamole', 'u200e04-03-201903:49 PM'], ['Dell OS Recovery Tool', 'vedantjoshi', 'Philip_Yip', 'u200e04-07-201910:07 AM'], ['Inspiron N5050 Upgrade', 'Forkman', 'u200e04-07-201901:45 PM'], ['Inspiron N5050 Upgrade', 'Forkman', 'u200e04-07-201901:42 PM'], ['Include XPS thermal management system to G3 BIOS', 'Evsign', 'JeffersonX', 'u200e02-26-201908:37 AM'], ['keyboard stopped working after upgrading the RAM', 'OudMaster', 'OudMaster', 'u200e04-06-201901:02 PM'], ['G3 3579 temperature problem', 'TheLonelySqrt3', 'JeffersonX', 'u200e12-01-201808:55 PM']], [['Latitude E6530 graphics card problem', 'Anujpandey324', 'u200e04-07-201912:35 PM'], ['Inspiron n5010 3 beeps & black screen', 'stayaway.fgs', 'JOcean', 'u200e04-07-201907:48 AM'], ['INSPIRON 15 5000 SERIES, POSSIBLY OTHERS BEWARE', 'louie.bitweiser', 'u200e04-07-201911:30 AM'], ['Lap tops', 'oakfans', 'u200e04-07-201911:02 AM'], ['USB 3.0 Ethernet Adapter', 'winterboy8', 'jphughan', 'u200e04-06-201911:54 AM'], ['Dell g5 fans not spinning at all', 'Ryinix', 'Ryinix', 'u200e04-05-201903:16 PM'], ['upgrade wlan in Dell Inspiron 3567', 'Jesse__G', 'Jesse__G', 'u200e03-26-201910:14 AM'], ['UPS for Dell Inspiron 15R 5520 laptop', 'Hsyn', 'nyc10036', 'u200e04-07-201901:58 AM'], ['Pressing 8 shows 0. Replaced keyboard, worked for a few weeks, same issue returned?', 'AW2', 'AW2', 'u200e04-07-201903:13 AM'], ['7437 Keyboard issues after A13 BIOS upgrade. Downgrade fixes', 'AW2', 'AW2', 'u200e03-06-201902:18 AM'], ['Heat', 'Prestonwhhhite', 'Afflospark', 'u200e10-13-201805:38 PM'], ['Inspiron 7586 screen issues', 'lharmon1922', 'vamp4200', 'u200e11-29-201801:22 PM'], ['Insiroin 5570 battery time', 'jurek224466', 'Afflospark', 'u200e04-05-201912:31 AM'], ['Dell inspiron 7577 battery discharges while gamimg with charger pluged in', 'George7577', 'Afflospark', 'u200e03-10-201809:50 PM'], ['Inspiron 3000 Dead after 1 year??', 'Eastwaangel', 'u200e04-06-201908:52 PM'], ['True Color App', 'Jukebox8691', 'dsfingers', 'u200e03-09-201803:58 PM'], ['Inspiron 7577, Games refusing to use Dedicated Graphics', 'InternetInfantry', 'JOcean', 'u200e04-06-201912:55 AM'], ['Inspiron N 4010 1600MHz sticks', 'sree harsha', 'Mary G', 'u200e04-06-201905:23 PM'], ['Inspiron I7 7737 video small after driver upgrade', 'Irish2204', 'Irish2204', 'u200e04-06-201904:08 PM'], ["Inspirion 13 7348 Won't Turn On", 'packer96', 'robbb9119', 'u200e02-25-201807:34 PM'], ['Screen issues with Dell Inspiron 15 7000', 'GDMTCaboose', 'Mr Wob', 'u200e06-17-201806:20 AM'], ['Inspiron 15-7???, not detecting headphones', 'Tensins', 'Greenphase', 'u200e05-23-201802:49 PM'], ['Inspiron 15-3521, ram upgrade', 'AymaneHTF', 'Pipibrasci', 'u200e02-07-201807:53 AM'], ['Leaving my dell 15 7559 connected to the AC when using?', 'LeoTwixx', 'Teetertotter', 'u200e04-05-201905:14 PM'], ['G3 3779 Slow wifi download speed around 400 kb/s', 'Teodor Yanchev', 'fireberd', 'u200e04-05-201907:56 PM'], ['how can i change boot order from sata 0 to sata 1 ????', 'roami', 'Mary G', 'u200e04-05-201903:24 PM'], ['Need to replace only display cable or entire display?', 'dannyetlv', 'dannyetlv', 'u200e04-06-201912:44 AM'], ['Inspiron 17r 5737 BIOS update AC Power Adapter type cannot be determined', 'kozle', 'SheTriedIt', 'u200e03-24-201807:08 AM'], ['13 7378 2-in-1 Can I replace the 256 GB with a 1 TB SSD', 'Biker 2017', 'Mike61986', 'u200e11-25-201806:57 AM'], ['Inspiron 15 7000 series case separating', 'MikeD927', 'u200e04-06-201904:10 AM']]]
Of course, there’s a far simpler solution to webscraping…it’s to pull in the RSS feed directly :). This is the legal approach, and technically simpler. It’s a one-liner to pull in the data, already in the format you expect.
https://www.dell.com/community/s/vjauj58549/rss/boardmessages?board.id=Inspiron&count=100
Example code:
import requests
rssFeed = "https://www.dell.com/community/s/vjauj58549/rss/boardmessages?board.id=Inspiron&count=100"
xmlData = requests.get(rssFeed)
... parse xmlData ( i.e., xml to dict) ) ...
This would be preferred to using webscraping, because the static html website depends on this.
this is a simple method and easy method i hope it will help you.
import requests
from bs4 import BeautifulSoup
import urllib
questions_list =list()
count = 1
# here you can change condition to scrape as many pages you want
while count < 6:
print()
print("page no. {}".format(count))
print()
html_page = requests.get("https://www.dell.com/community/Inspiron/bd-p/Inspiron/page/{}".format(count))
soup = BeautifulSoup(html_page.text, "html.parser")
page = soup.findAll('h2', class_="message-subject")
user_name = soup.find_all('a', class_="lia-user-name-link")
date = soup.find_all('span',class_='lia-component-common-widget-date')
for i,j,k in zip(page,user_name,date):
mydict = dict()
ques = (i.text.strip())
user_name = j.text.strip()
date = k.text.strip()
mydict[ques] = user_name,date
questions_list.append(mydict)
print(ques)
print(user_name)
print(date)
print()
print('*********************************************************')
count = count + 1
I need to scrape all the questions from the first 5 pages of the following website: https://www.dell.com/community/Inspiron/bd-p/Inspiron
I need to return the discussion title, author, time of question, and who responded last into a csv file.
Could someone get me started? I don’t have any code as of yet. I’m also not sure where to start. There are too many pieces in this problem for me to figure out.
Thank you! It doesn’t have to be the full code, but a starting place would be very helpful. I know I’m supposed to use requests and BeautifulSoup
Edit: I don’t have a TA to ask for help. That’s why I’ve posted here.
You should try Scrapy, a free and open-source web-crawling framework.
You can check it out here: https://scrapy.org/
The solution might look like the example bellow:
# -*- coding: utf-8 -*-
import scrapy
class ToScrapeSpiderXPath(scrapy.Spider):
name = 'toscrape-xpath'
start_urls = [
'http://quotes.toscrape.com/',
]
def parse(self, response):
for quote in response.xpath('//div[@class="quote"]'):
yield {
'text': quote.xpath('./span[@class="text"]/text()').extract_first(),
'author': quote.xpath('.//small[@class="author"]/text()').extract_first(),
'tags': quote.xpath('.//div[@class="tags"]/a[@class="tag"]/text()').extract()
}
next_page_url = response.xpath('//li[@class="next"]/a/@href').extract_first()
if next_page_url is not None:
yield scrapy.Request(response.urljoin(next_page_url))
Example from: https://github.com/scrapy/quotesbot
You can use requests to retrieve the page source, and BeautifulSoup to parse the results. To find the question, iterate over each tr with a class containing the question id:
from bs4 import BeautifulSoup as soup
import requests, re
def scrape_page(page_num):
d = soup(requests.get('https://www.dell.com/community/Inspiron/bd-p/Inspiron/page/{}'.format(page_num)).text, 'html.parser')
q = d.find_all('tr', {'class':re.compile('lia-js-data-messageUid-d+')}) #find all question trs
result = [[i.find('a', {'class':'page-link'}).text]+[c.text for c in i.find_all('a', {'class':'lia-user-name-link'})]+[i.find('span', {'class':'lia-component-common-widget-date'}).text] for i in q]
return [[re.sub('n|s{2,}', '', c) for c in i] for i in result]
The result for each page is a list of lists containing the title, author, last comment poster, and the date the question was asked. To find the first five pages:
results = [scrape_page(i) for i in range(1, 6)]
Output:
[[['Inspiron gaming laptops, battery drain is normal', 'DELL-Chris M', 'u200e11-30-201808:36 AM'], ['Where are my Inspiron Laptop manuals?', 'DELL-Chris M', 'u200e05-02-201809:53 AM'], ['Where are my Inspiron Laptop drivers?', 'DELL-Chris M', 'u200e05-02-201809:50 AM'], ['New Inspiron Laptop Usage and Troubleshooting Guides (3/2/18)', 'DELL-Chris M', 'u200e03-02-201810:28 AM'], ['Can not install dell mobile connect aplication from windows store', 'GeoNeo77', 'Mary G', 'u200e04-11-201908:20 AM'], ['Inspiron 13 7000 (7370) Black / Blank Screen / Display', 'brianms', 'Erutan409', 'u200e11-14-201805:46 PM'], ['Boot From External USB Drive or USB Stick', 'borntodeal', 'u200e04-11-201910:01 AM'], ['WiFi suddenly disappeared', 'Anagno7', 'u200e04-11-201909:50 AM'], ['Task bar ikons move quickly back and forth', 'ralphspajka', 'ralphspajka', 'u200e04-11-201907:06 AM'], ['Does an Intel Dual Band Wireless-AC 8265 or 9260 will work on a Inspiron 5555', 'Cauchons', 'jphughan', 'u200e04-07-201910:43 AM'], ['Fan not working', 'Tunay', 'JOcean', 'u200e04-10-201903:35 PM'], ['Upgraded Wireless N 7260 WIFI Card', 'fadsarmy', 'fadsarmy', 'u200e03-30-201909:13 PM'], ['XPS 13-9365, battery issues!', 'rickmanhatan', 'DELL-Chris M', 'u200e04-11-201903:18 AM'], ['Inspiron 17 7779, turns black after playing games', 'Prime7105', 'Teetertotter', 'u200e04-09-201908:12 PM'], ['New 7559 BIOS is corrupted', 'Suncatcher_13', 'Teetertotter', 'u200e02-26-201906:25 PM'], ['Inspiron 15 7559 compatible con SSD Samsung 970 EVO Plus?', 'defto', 'Saltgrass', 'u200e04-10-201908:40 PM'], ['Laptop wont turn on...', 'TonicWhisper', 'JOcean', 'u200e04-10-201911:40 PM'], ['Why is my Dell G7 buzzing when charging', 'Andre Dolar', 'Jey_31', 'u200e09-24-201806:09 AM'], ['Clone only OS partition to SSD/Install windows 10 on new SSD', 'HemantSangle', 'hezi', 'u200e09-26-201803:02 AM'], ['ERROR Alienware Command Center en G7 17 7790', 'Matafleur', 'SERGIO.MAGNO', 'u200e03-16-201908:03 PM'], ['Inspiron 15-7567, M.2 NVMe SSD, failed boot device', 'JohnJakeLindert', 'Saltgrass', 'u200e03-31-201902:02 PM'], ['Dell adapter not detected. Plugged in, not charging.', 'spbone', 'acmoo1964', 'u200e01-23-201802:11 PM'], ['Inspiron 5482 keyboard skipping characters, missing strokes. HELP! :(', 'NajibOzy', 'NajibOzy', 'u200e04-10-201902:43 AM'], ['Bluetooth Inspiron 14 3000 Demora demais para conectar com o fone.', 'VitorVitalino', 'Dell-SreejithR', 'u200e04-08-201912:45 PM'], ['Inspiron 13-7000 7359', 'jjrapy', 'dell-robert p', 'u200e04-08-201903:49 PM'], ['DELL INSPIRON i5 5570 CPU OVERHEATING', 'ryanmcgrath', 'Dell-SreejithR', 'u200e04-05-201911:30 PM'], ['Inspiron 24 5000 problem', 'cm_a', 'DELL-Chris M', 'u200e04-04-201912:07 AM'], ['Inspiron G5 15-5587, WiFi drops, then connects itself', 'abhar22', 'abhar22', 'u200e04-02-201907:16 PM'], ['Dell G5 15 5587 Overheating Issue', 'eldarinn', 'Dell-SreejithR', 'u200e03-06-201902:38 AM'], ['Inspiron 15, WiFi not detected', 'savaman', 'DELL-Chris M', 'u200e04-05-201911:35 AM'], ['Dell Inspiron 13 7000 P83G Will Not Wake After Sleep', 'random0o', 'dell-robert p', 'u200e04-06-201905:51 PM'], ['Inspiron 14, 2.1 USB sound kit', 'GuilhermeBil', 'DELL-Chris M', 'u200e04-05-201902:32 PM'], ['Help with display', 'dannyetlv', 'Mary G', 'u200e04-09-201902:00 PM'], ['Inspiron 15 5567 battery drains when shutdown', 'Razaaque', 'Dell-SreejithR', 'u200e03-21-201903:28 AM']], [['If the charger is plugged in at 100% battery life will it harm my battery life in any way?', 'Burlex', 'dell-robert p', 'u200e04-05-201911:48 AM'], ['Audio problems after Dell update........ EVERY TIME', 'Skaboudles', 'Saltgrass', 'u200e04-08-201909:50 AM'], ['Dell Inspiron 7548 Ram', 'Foofuu', 'JOcean', 'u200e04-09-201905:27 PM'], ['Inspiron 17 5770 max size of RAM (memory)', 'leks12lk', 'JOcean', 'u200e04-10-201912:16 AM'], ['Inspiron G3 15-3579, randomly freezes', 'livictor', 'Mike_Space', 'u200e10-17-201806:59 PM'], ['audio renderer error', 'Shoun0305', 'anttisum', 'u200e01-24-201911:41 PM'], ['Dell G5 5587 Sound issues', 'NickBk', 'Cptntrips', 'u200e02-15-201911:49 AM'], ['Inspiron G7 15-7588, speaker is not plugged after updating drivers', 'Thai Hung', 'Thai Hung', 'u200e04-07-201908:30 PM'], ["Dell Inspiron 15 5000 fan on constantly when connected to power but won't boot", '996sleepless', 'u200e04-10-201901:22 AM'], ['Inspiron 17 7000 - Headset not an option?', 'Vildpear', 'Vildpear', 'u200e02-14-201902:07 AM'], ['Dell Inspiron 5575 ryzen 5 vega 8 keeps throttling', 'FIDERIKS', 'u200e04-10-201912:49 AM'], ['Inspiron 15 5575 AMD M.2 NVME slow speed', 'Billdag', 'jphughan', 'u200e04-09-201905:16 PM'], ['hdd not detected inspiron 7573 after BIOs upgrade', 'abylan', 'abylan', 'u200e04-06-201902:11 PM'], ['Dell Inspiron 14 3421-Dell Wireless 1703 Wlan Card compatibility', '5GHzstarve', '5GHzstarve', 'u200e04-09-201907:43 AM'], ['Dell Inspiron 15 5000 laptop super slow', 'DH9500', 'u200e04-09-201905:55 PM'], ['USB Issue with TB18DC', 'AmazingDell', 'u200e04-09-201905:50 PM'], ['Nvidia 1060 max-q temperature in Dell Inspiron 7577', 'Villn52', 'u200e04-09-201905:49 PM'], ['Wifi issues with dell Inspiron 15 7000', 'Desperate4help', 'Zcarcath', 'u200e11-15-201804:23 AM'], ['Inspiron 15 7559 - Battery light randomly flickering', 'AstralC', 'AstralC', 'u200e04-09-201910:37 AM'], ['Intermittent noises from hard drive', 'SV126CBC', 'nyc10036', 'u200e04-09-201902:03 PM'], ['Smartbyte reinstalls itself?! How to make it stop?', 'krgman101', 'RivetNetworks', 'u200e04-08-201905:21 AM'], ['Mouse/Keyboard input overlap issue', 'Goodnight guy', 'Sgtsparkles', 'u200e04-08-201904:56 PM'], ['Inspiron 15-7567, reed switch?', 'shawanish8', 'Furrydaus', 'u200e04-01-201911:29 PM'], ['inspiron 15-5580', 'GOWTHAMAN RAVI', 'u200e04-09-201910:29 AM'], ['Inspiron 7580, CPU temperature', 'pl66', 'pl66', 'u200e04-02-201910:50 AM'], ['Ranura m2 en dell inspiron 15 3000 3567', 'Nico.eguia', 'Dell-SreejithR', 'u200e04-03-201909:10 AM'], ['Dell PC keeps booting up to diagnostic boot test', 'a__biee', 'Tiburon66', 'u200e10-09-201804:17 AM'], ['How To Disable Adaptive Brightness in DELL Insipiron 7580 laptop', 'Addyzaf', 'pl66', 'u200e04-09-201910:18 AM'], ['G3 3579 Fan makes clicking noise at low rpm', 'a2375658', 'Dell-SreejithR', 'u200e02-20-201910:27 PM'], ["Which Inspiron's have DBC?", 'Vortex--5', 'DELL-Chris M', 'u200e04-03-201905:27 AM']], [['backlit keyboard assembly for Inspiron 11-3185', 'mrsbranj', 'dell-robert p', 'u200e04-03-201903:57 AM'], ['Re: Inspiron 7580 fan noise and heating issue on m', 'Addyzaf', 'dell-robert p', 'u200e04-07-201910:41 AM'], ['Inspiron G7 15-7588, second monitor not working', 'Movie1011', 'DELL-Chris M', 'u200e04-08-201911:40 AM'], ['The display is okay?', 'dannyetlv', 'u200e04-09-201910:45 AM'], ['bios update problem', 'yodas91', 'JLJM', 'u200e02-08-201808:13 AM'], ["Inspiron 13 7000 (7370) Display Won't Wake Up", 'brianms', 'brianms', 'u200e11-15-201809:27 AM'], ['2019 G7 7590 Fan Control Abysmal', 'msbtt94', 'u200e04-08-201905:41 PM'], ['Dell G5 Startup problem on battery', 'bob8me', 'Dell-SreejithR', 'u200e02-04-201903:02 PM'], ['Monitor going black', 'uffda123', 'Dell-SreejithR', 'u200e04-07-201912:10 PM'], ['how to system rewrite on Inspiron 3153', 'HD72', 'Philip_Yip', 'u200e04-08-201911:45 AM'], ['I would like to upgrade my Dell Inspiron 17R, any advice?', 'careb', 'nyc10036', 'u200e04-08-201912:32 PM'], ['dell G5 8875 lagging or freezing during playing a video or surfing the internet and fan noise', 'zeyadmourad', 'Dell-SreejithR', 'u200e04-07-201903:42 AM'], ['dell G5 5587 over heating', 'zeyadmourad', 'Dell-SreejithR', 'u200e04-08-201908:49 PM'], ['Took a gamble... WD15 USB-C dock works with Inspiron 17 5770', 'Frozen Mitchell', 'leks12lk', 'u200e03-22-201908:44 PM'], ['Inspiron 7572, Intel 620, Nvidia MX 150 4GB', 'Teetertotter', 'Teetertotter', 'u200e04-06-201909:45 AM'], ['BIOS update will not run on Inspiron 15-3542', 'ai4gk', 'ai4gk', 'u200e04-08-201910:26 PM'], ['Dell inspiron 3558 low wifi signal', 'Gaffar', 'JOcean', 'u200e04-08-201912:45 PM'], ['Inspiron 7359 no video and 2 beeps', 'mek33', 'JOcean', 'u200e04-08-201906:56 PM'], ['Memory and SDD Upgrade for dell N3542', 'bitsrdx', 'JOcean', 'u200e04-09-201904:30 AM'], ['are Dell Inspiron 14" 5458 and 5459 parts interchangable?', 'tristan012345', 'u200e04-09-201903:47 AM'], ['G5 5587 Latest Realtek audio driver breaks sound entirely', 'BarnyTrubble', 'Mark3434', 'u200e02-06-201911:01 PM'], ['G3 3579, IRST Drivers causing data loss on SATA SSD.', 'ariaperdana', 'u200e04-08-201910:28 PM'], ['Inspiron 15-5580.', 'GOWTHAMAN RAVI', 'u200e04-08-201910:04 PM'], ['Dell Inspiron 7370 sleep / wake. Lid closure issues.', 'Harish_Vanguard', 'Nambabwe', 'u200e05-23-201803:50 AM'], ['Inspiron 13 7000', 'jjrapy', 'nyc10036', 'u200e04-08-201903:54 PM'], ['Computer not connecting to network after getting static IP', 'MChe', 'irishgeoff', 'u200e03-04-201805:34 AM'], ['HID Complaint touch screen driver not available', 'VarunDell', 'shahrishit9', 'u200e08-10-201811:35 AM'], ['5559 Touchscreen driver', 'Natalie Settoon', 'kroppykat', 'u200e03-30-201806:55 PM'], ['Support Assist Not installing', 'Kannans243', 'PT8888', 'u200e05-18-201802:06 AM'], ['Keyboard issue', 'SteveTree', 'SteveTree', 'u200e04-07-201904:09 AM']], [['Please fix the C-State bug in Inspiron 11 3000.', 'saudiqbal', 'leemell', 'u200e08-26-201810:27 PM'], ['Dell 7577 overheating', 'qaiser96', 'Dell-SreejithR', 'u200e04-06-201903:57 AM'], ['Dell G3 (3579) BSOD when is plugged into power supply', 'Brssvk', 'Dell-SreejithR', 'u200e04-06-201902:22 AM'], ['Power surge on USB port', 'Lazysocks', 'Mary G', 'u200e04-08-201912:29 PM'], ['Laptop Warranty', 'sophie_ep', 'Mary G', 'u200e04-08-201910:01 AM'], ['Dell updates fail to install after warranty expires', 'JWall61', 'Mary G', 'u200e04-08-201907:31 AM'], ['Inspiron 13 5378 - Touchpad missing from hardware and gestures do not work - Windows 10', 'bjames8787', 'u200e04-08-201910:56 AM'], ['Inspiron 7580, thermals, cooling, fan control', 'pl66', 'pl66', 'u200e04-06-201902:33 AM'], ['RAM upgrade inspiron 5521-6527 my service tag', 'Cihat Eren', 'JOcean', 'u200e04-08-201904:09 AM'], ["New hard drive, dell recovery disk won't go to install menu", 'yt1300don', 'yt1300don', 'u200e04-07-201911:49 PM'], ['Dell Inspiron i5 7000 gaming turning on for a few seconds and immediadtly back off', 'Cmac123', 'speedstep', 'u200e04-07-201909:10 PM'], ['Recover BIOS for Inspiron 5755', 'laltobelli', 'SHColf57', 'u200e03-15-201807:08 PM'], ['Inspiron 7380, loud fan', 'ladrian', 'u200e04-07-201906:43 PM'], ['Power Adapter Error', 'GMMister', 'dell-robert p', 'u200e01-25-201908:35 PM'], ['Inspiron 17 3721 Four Beeps on booting', 'Eddythealien2003', 'fireberd', 'u200e04-08-201902:38 AM'], ['inspiron', 'cosmicdancer', 'evegreen', 'u200e03-11-201903:48 AM'], ['RAM upgrade for Dell Inspiron 5570', 'asadsultanawan', 'Sunray001', 'u200e08-13-201806:28 AM'], ['ssd upgrade dell inspiron 7559', 'Mr Wob', 'Ning-123', 'u200e04-06-201901:32 PM'], ['AC Power Adapter not reading', 'Monemi', 'u200e04-07-201910:46 PM'], ['Battery replacement for Inspiron 5558', 'Chxrlotte', 'ron564848', 'u200e01-02-201905:38 AM'], ['Can Inspiron 5570 boot from NVMe M.2 SSD?', 'valentin_b', 'jamquam', 'u200e07-11-201809:37 AM'], ['Using laptop on direct power without using battery power [built in battery ]', 'RAJR11', 'u200e04-07-201908:56 PM'], ['charger wire colors', 'Shaul G', 'nyc10036', 'u200e04-07-201904:29 AM'], ['Dell Inspiron 13 5378 pen support', 'Joeldamole', 'Joeldamole', 'u200e04-03-201903:49 PM'], ['Dell OS Recovery Tool', 'vedantjoshi', 'Philip_Yip', 'u200e04-07-201910:07 AM'], ['Inspiron N5050 Upgrade', 'Forkman', 'u200e04-07-201901:45 PM'], ['Inspiron N5050 Upgrade', 'Forkman', 'u200e04-07-201901:42 PM'], ['Include XPS thermal management system to G3 BIOS', 'Evsign', 'JeffersonX', 'u200e02-26-201908:37 AM'], ['keyboard stopped working after upgrading the RAM', 'OudMaster', 'OudMaster', 'u200e04-06-201901:02 PM'], ['G3 3579 temperature problem', 'TheLonelySqrt3', 'JeffersonX', 'u200e12-01-201808:55 PM']], [['Latitude E6530 graphics card problem', 'Anujpandey324', 'u200e04-07-201912:35 PM'], ['Inspiron n5010 3 beeps & black screen', 'stayaway.fgs', 'JOcean', 'u200e04-07-201907:48 AM'], ['INSPIRON 15 5000 SERIES, POSSIBLY OTHERS BEWARE', 'louie.bitweiser', 'u200e04-07-201911:30 AM'], ['Lap tops', 'oakfans', 'u200e04-07-201911:02 AM'], ['USB 3.0 Ethernet Adapter', 'winterboy8', 'jphughan', 'u200e04-06-201911:54 AM'], ['Dell g5 fans not spinning at all', 'Ryinix', 'Ryinix', 'u200e04-05-201903:16 PM'], ['upgrade wlan in Dell Inspiron 3567', 'Jesse__G', 'Jesse__G', 'u200e03-26-201910:14 AM'], ['UPS for Dell Inspiron 15R 5520 laptop', 'Hsyn', 'nyc10036', 'u200e04-07-201901:58 AM'], ['Pressing 8 shows 0. Replaced keyboard, worked for a few weeks, same issue returned?', 'AW2', 'AW2', 'u200e04-07-201903:13 AM'], ['7437 Keyboard issues after A13 BIOS upgrade. Downgrade fixes', 'AW2', 'AW2', 'u200e03-06-201902:18 AM'], ['Heat', 'Prestonwhhhite', 'Afflospark', 'u200e10-13-201805:38 PM'], ['Inspiron 7586 screen issues', 'lharmon1922', 'vamp4200', 'u200e11-29-201801:22 PM'], ['Insiroin 5570 battery time', 'jurek224466', 'Afflospark', 'u200e04-05-201912:31 AM'], ['Dell inspiron 7577 battery discharges while gamimg with charger pluged in', 'George7577', 'Afflospark', 'u200e03-10-201809:50 PM'], ['Inspiron 3000 Dead after 1 year??', 'Eastwaangel', 'u200e04-06-201908:52 PM'], ['True Color App', 'Jukebox8691', 'dsfingers', 'u200e03-09-201803:58 PM'], ['Inspiron 7577, Games refusing to use Dedicated Graphics', 'InternetInfantry', 'JOcean', 'u200e04-06-201912:55 AM'], ['Inspiron N 4010 1600MHz sticks', 'sree harsha', 'Mary G', 'u200e04-06-201905:23 PM'], ['Inspiron I7 7737 video small after driver upgrade', 'Irish2204', 'Irish2204', 'u200e04-06-201904:08 PM'], ["Inspirion 13 7348 Won't Turn On", 'packer96', 'robbb9119', 'u200e02-25-201807:34 PM'], ['Screen issues with Dell Inspiron 15 7000', 'GDMTCaboose', 'Mr Wob', 'u200e06-17-201806:20 AM'], ['Inspiron 15-7???, not detecting headphones', 'Tensins', 'Greenphase', 'u200e05-23-201802:49 PM'], ['Inspiron 15-3521, ram upgrade', 'AymaneHTF', 'Pipibrasci', 'u200e02-07-201807:53 AM'], ['Leaving my dell 15 7559 connected to the AC when using?', 'LeoTwixx', 'Teetertotter', 'u200e04-05-201905:14 PM'], ['G3 3779 Slow wifi download speed around 400 kb/s', 'Teodor Yanchev', 'fireberd', 'u200e04-05-201907:56 PM'], ['how can i change boot order from sata 0 to sata 1 ????', 'roami', 'Mary G', 'u200e04-05-201903:24 PM'], ['Need to replace only display cable or entire display?', 'dannyetlv', 'dannyetlv', 'u200e04-06-201912:44 AM'], ['Inspiron 17r 5737 BIOS update AC Power Adapter type cannot be determined', 'kozle', 'SheTriedIt', 'u200e03-24-201807:08 AM'], ['13 7378 2-in-1 Can I replace the 256 GB with a 1 TB SSD', 'Biker 2017', 'Mike61986', 'u200e11-25-201806:57 AM'], ['Inspiron 15 7000 series case separating', 'MikeD927', 'u200e04-06-201904:10 AM']]]
Of course, there’s a far simpler solution to webscraping…it’s to pull in the RSS feed directly :). This is the legal approach, and technically simpler. It’s a one-liner to pull in the data, already in the format you expect.
https://www.dell.com/community/s/vjauj58549/rss/boardmessages?board.id=Inspiron&count=100
Example code:
import requests
rssFeed = "https://www.dell.com/community/s/vjauj58549/rss/boardmessages?board.id=Inspiron&count=100"
xmlData = requests.get(rssFeed)
... parse xmlData ( i.e., xml to dict) ) ...
This would be preferred to using webscraping, because the static html website depends on this.
this is a simple method and easy method i hope it will help you.
import requests
from bs4 import BeautifulSoup
import urllib
questions_list =list()
count = 1
# here you can change condition to scrape as many pages you want
while count < 6:
print()
print("page no. {}".format(count))
print()
html_page = requests.get("https://www.dell.com/community/Inspiron/bd-p/Inspiron/page/{}".format(count))
soup = BeautifulSoup(html_page.text, "html.parser")
page = soup.findAll('h2', class_="message-subject")
user_name = soup.find_all('a', class_="lia-user-name-link")
date = soup.find_all('span',class_='lia-component-common-widget-date')
for i,j,k in zip(page,user_name,date):
mydict = dict()
ques = (i.text.strip())
user_name = j.text.strip()
date = k.text.strip()
mydict[ques] = user_name,date
questions_list.append(mydict)
print(ques)
print(user_name)
print(date)
print()
print('*********************************************************')
count = count + 1