scrapy Link Extractor Python Pagination

Question

Link: https://www.softsurroundings.com/clothing/tops/

When I scroll the page the Link gets changed to https://www.softsurroundings.com/clothing/tops/page-2

on furthur scrolling, The link changes to https://www.softsurroundings.com/clothing/tops/page-3 and so on.

The Link Extractor is only remaining scraps that get displayed only in first url i.e. https://www.softsurroundings.com/clothing/tops/.

The Link Extractor Rule is as follows:

 rules = [
    Rule(LinkExtractor(restrict_css='.flexWrap.flexStart.prw .thumb a[class="thmNm"]'),
         callback='parse_items',follow=True)
]

When the main link is calles using

  start_urls = [
    "https://www.softsurroundings.com/clothing/tops/"
]

Only 50 Products get scrapped which are on page 1. Now there is no button or link for next for pagination.. The link gets updates on scrolling only and scrapy Link Extractor is not following the products that appear on scrolling and link changes i.e. When I scroll the page the Link gets changed to https://www.softsurroundings.com/clothing/tops/page-2
and so on..

How to do this? WHat do I need to change in Link Extractor Rule?

Asked By: Malik Zohaib Mustafa

||

Source

Answer 1

Así puedes simular el scroll (especificamente para esta página)

start_urls= ['https://www.softsurroundings.com/clothing/tops/page-'+str(x)+'/' for x in range(1,3)]

De nada papu

Answered By: LJ FI

scrapy Link Extractor Python Pagination

Question:

Answers: