Why does my scraper return empty result set when I run it?

Question

When i run this code block, it runs successfully but returns empty result for the name field. please help me check, what am I missing?

Here’s my block code

import scrapy


class TruckspiderSpider(scrapy.Spider):
      name = 'truckspider'
      allowed_domains = ['www.quicktransportsolutions.com']
      start_urls = ['https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php']

      def parse(self, response):

          containers = response.css('div.col-md-4') 

          for container in containers:
    
          yield
          {
             'name': container.css('a::text').get(),
          }

Here is the result I get

2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': None}
2023-02-01 19:17:42 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>    
{'name': None}
2023-02-01 19:17:42 [scrapy.core.engine] INFO: Closing spider (finished)
2023-02-01 19:17:42 [scrapy.extensions.feedexport] INFO: Stored json feed (55 items) in: truck.json
2023-02-01 19:17:42 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 265,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 4912,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'elapsed_time_seconds': 1.916927,
 'feedexport/success_count/FileFeedStorage': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2023, 2, 1, 18, 17, 42, 902380),
 'httpcompression/response_bytes': 22809,
 'httpcompression/response_count': 1,
 'item_scraped_count': 55,
 'log_count/DEBUG': 57,
 'log_count/INFO': 11,
 'log_count/WARNING': 1,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2023, 2, 1, 18, 17, 40, 985453)}
2023-02-01 19:17:42 [scrapy.core.engine] INFO: Spider closed (finished)

Asked By: Miracle

||

Source

Answer 1

You are getting empty ResultSet because CSS selectors div.col-md-4 isn’t in correct way instead use [class="col-md-4 column"] and grab the desired output.

import scrapy

class TruckspiderSpider(scrapy.Spider):
      name = 'truckspider'
      allowed_domains = ['www.quicktransportsolutions.com']
      start_urls = ['https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php']

      def parse(self, response):
        containers = response.css('[class="col-md-4 column"]')
        for container in containers:
            yield {
                'name': container.css('a::text').get()}

Output:

{'name': 'Alabama Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Alaska Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Arizona Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Arkansas Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'California Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Colorado Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Connecticut Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Delaware Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Florida Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Georgia Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Hawaii Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Idaho Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Illinois Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Indiana Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Iowa Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Kansas Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Kentucky Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Louisiana Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Maine Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Massachusetts Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Maryland Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Michigan Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Minnesota Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Mississippi Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Missouri Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Montana Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Nevada Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>
{'name': 'Nebraska Trucking Companies'}
2023-02-02 00:56:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php>

Runthe following code without scrapy project

from scrapy.crawler import CrawlerProcess

import scrapy

class TruckspiderSpider(scrapy.Spider):
      name = 'truckspider'
      allowed_domains = ['www.quicktransportsolutions.com']
      start_urls = ['https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php']

      def parse(self, response):
        containers = response.css('[class="col-md-4 column"]')
        for container in containers:
            yield {
                'name': container.css('a::text').get()}

if __name__ == "__main__":  
    process = CrawlerProcess()
    process.crawl(TruckspiderSpider)
    process.start()

Answered By: Md. Fazlul Hoque

Answer 2

@Md. Fazlul Hoque thanks for all your time and responses.

I finally got my way around it. I gave my local machine access to this file: 2023-02-02 10:54:17 [filelock] DEBUG: Attempting to acquire lock 2733195998640 on C:UsersChiNeduAppDataLocalProgramsPythonPython310libsite-packagestldextract.suffix_cache/publicsuffix.org-tldsde84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock in my AppData folder and it worked like magic. So happy.

Answered By: Miracle

Why does my scraper return empty result set when I run it?

Question:

Answers: