Get href using css selector with Scrapy

Question

I want to get the href value:

<span class="title">
  <a href="https://www.example.com"></a>
</span>

I tried this:

Link = Link1.css('span[class=title] a::text').extract()[0]

But I just get the text inside the <a>. How can I get the link inside the href?

Asked By: Marco Dinatsoli

||

Source

Answer 1

What you’re looking for is:

Link = Link1.css('span[class=title] a::attr(href)').extract()[0]

Since you’re matching a span “class” attribute also, you can even write

Link = Link1.css('span.title a::attr(href)').extract()[0]

Please note that ::text pseudo element and ::attr(attributename) functional pseudo element are NOT standard CSS3 selectors. They’re extensions to CSS selectors in Scrapy 0.20.

Edit (2017-07-20): starting from Scrapy 1.0, you can use .extract_first() instead of .extract()[0]

Link = Link1.css('span[class=title] a::attr(href)').extract_first()
Link = Link1.css('span.title a::attr(href)').extract_first()

Answered By: paul trmbrth

Answer 2

Link = Link1.css('span.title a::attr(href)').extract_first()

you can get more infomation from this

Answered By: Eddy

Answer 3

This will do the job:

Link = Link1.css('span.title a::attr(href)').extract()

Link will have the value : https://www.example.com

Answered By: Jorgesys

Get href using css selector with Scrapy

Question:

Answers: