Scrapy xpath not able to extract
Question:
I am trying to extract data from link, I used
scrapy shell "https://www.newegg.com/Product/Product.aspx?Item=06T-0045-00045"
I got correct response, but can’t get the Xpath to work,i.e getting the price with response.xpath('//li[@class="price-current"]') returns empty, I tried also response.xpath('//*[@id="landingpage-price"]/div/div/ul') but also empty, When I use response.xpath('//*[@id="landingpage-price"]') it works but anything deeper returns empty.
Answers:
You always need to check source HTML (Ctrl+U). There is <meta itemprop='price' content='78.23' /> in the source. So simple:
response.xpath('//meta[@itemprop="price"]/@content').extract_first()
will work.
I was searching for the same question for the whole day and find this answer perfect for this:
response.xpath('//meta[@itemprop="price"]/@content').get()
I am trying to extract data from link, I used
scrapy shell "https://www.newegg.com/Product/Product.aspx?Item=06T-0045-00045"
I got correct response, but can’t get the Xpath to work,i.e getting the price with response.xpath('//li[@class="price-current"]') returns empty, I tried also response.xpath('//*[@id="landingpage-price"]/div/div/ul') but also empty, When I use response.xpath('//*[@id="landingpage-price"]') it works but anything deeper returns empty.
You always need to check source HTML (Ctrl+U). There is <meta itemprop='price' content='78.23' /> in the source. So simple:
response.xpath('//meta[@itemprop="price"]/@content').extract_first()
will work.
I was searching for the same question for the whole day and find this answer perfect for this:
response.xpath('//meta[@itemprop="price"]/@content').get()