Scrapy Python Set up User Agent

Question

I tried to override the user-agent of my crawlspider by adding an extra line to the project configuration file. Here is the code:

[settings]
default = myproject.settings
USER_AGENT = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36"


[deploy]
#url = http://localhost:6800/
project = myproject

But when I run the crawler against my own web, I notice the spider did not pick up my customized user agent but the default one “Scrapy/0.18.2 (+http://scrapy.org)”.
Can any one explain what I have done wrong.

Note:

(1). It works when I tried to override the user agent globally:

scrapy crawl myproject.com -o output.csv -t csv -s USER_AGENT="Mozilla...."

(2). When I remove the line “default = myproject.setting” from the configuration file, and run scrapy crawl myproject.com, it says “cannot find spider..”, so I feel like the default setting should not be removed in this case.

Thanks a lot for the help in advance.

Asked By: B.Mr.W.

||

Source

Answer 1

Move your USER_AGENT line to the settings.py file, and not in your scrapy.cfg file. settings.py should be at same level as items.py if you use scrapy startproject command, in your case it should be something like myproject/settings.py

Answered By: paul trmbrth

Answer 2

Just in case anyone lands here that manually controls the scrapy crawl. i.e. you do not use the scrapy crawl process from the shell…

$ scrapy crawl myproject

But insted you use CrawlerProcess() or CrawlerRunner()…

process = CrawlerProcess()

or

process = CrawlerRunner()

then the user agent, along with other settings, can be passed to the crawler in a dictionary of configuration variables.

Like this…

    process = CrawlerProcess(
            {
                'USER_AGENT': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
            }
    )

Answered By: Bletch

Scrapy Python Set up User Agent

Question:

Answers: