How to use PyCharm to debug Scrapy projects
Question:
I am working on Scrapy 0.20 with Python 2.7. I found PyCharm has a good Python debugger. I want to test my Scrapy spiders using it. Anyone knows how to do that please?
What I have tried
Actually I tried to run the spider as a script. As a result, I built that script. Then, I tried to add my Scrapy project to PyCharm as a model like this:
File->Setting->Project structure->Add content root.
But I don’t know what else I have to do
Answers:
I am also using PyCharm, but I am not using its built-in debugging features.
For debugging I am using ipdb. I set up a keyboard shortcut to insert import ipdb; ipdb.set_trace() on any line I want the break point to happen.
Then I can type n to execute the next statement, s to step into a function, type any object name to see its value, alter execution environment, type c to continue execution…
This is very flexible, works in environments other than PyCharm, where you don’t control the execution environment.
Just type in your virtual environment pip install ipdb and place import ipdb; ipdb.set_trace() on a line where you want the execution to pause.
UPDATE
You can also pip install pdbpp and use the standard import pdb; pdb.set_trace instead of ipdb. PDB++ is nicer in my opinion.
The scrapy command is a python script which means you can start it from inside PyCharm.
When you examine the scrapy binary (which scrapy) you will notice that this is actually a python script:
#!/usr/bin/python
from scrapy.cmdline import execute
execute()
This means that a command like
scrapy crawl IcecatCrawler can also be executed like this: python /Library/Python/2.7/site-packages/scrapy/cmdline.py crawl IcecatCrawler
Try to find the scrapy.cmdline package.
In my case the location was here: /Library/Python/2.7/site-packages/scrapy/cmdline.py
Create a run/debug configuration inside PyCharm with that script as script. Fill the script parameters with the scrapy command and spider. In this case crawl IcecatCrawler.
Like this:
Put your breakpoints anywhere in your crawling code and it should workâ˘.
You just need to do this.
Create a Python file on crawler folder on your project. I used main.py.
- Project
- Crawler
- Crawler
- Spiders
- …
- main.py
- scrapy.cfg
Inside your main.py put this code below.
from scrapy import cmdline
cmdline.execute("scrapy crawl spider".split())
And you need to create a “Run Configuration” to run your main.py.
Doing this, if you put a breakpoint at your code it will stop there.
To add a bit to the accepted answer, after almost an hour I found I had to select the correct Run Configuration from the dropdown list (near the center of the icon toolbar), then click the Debug button in order to get it to work. Hope this helps!
I am running scrapy in a virtualenv with Python 3.5.0 and setting the “script” parameter to /path_to_project_env/env/bin/scrapy solved the issue for me.
intellij idea also work.
create main.py:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#coding=utf-8
import sys
from scrapy import cmdline
def main(name):
if name:
cmdline.execute(name.split())
if __name__ == '__main__':
print('[*] beginning main thread')
name = "scrapy crawl stack"
#name = "scrapy crawl spa"
main(name)
print('[*] main thread exited')
print('main stop====================================================')
show below:
According to the documentation https://doc.scrapy.org/en/latest/topics/practices.html
import scrapy
from scrapy.crawler import CrawlerProcess
class MySpider(scrapy.Spider):
# Your spider definition
...
process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})
process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished
As of 2018.1 this became a lot easier. You can now select Module name in your project’s Run/Debug Configuration. Set this to scrapy.cmdline and the Working directory to the root dir of the scrapy project (the one with settings.py in it).
Like so:
Now you can add breakpoints to debug your code.
I use this simple script:
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
process = CrawlerProcess(get_project_settings())
process.crawl('your_spider_name')
process.start()
Extending @Rodrigo’s version of the answer I added this script and now I can set spider name from configuration instead of changing in the string.
import sys
from scrapy import cmdline
cmdline.execute(f"scrapy crawl {sys.argv[1]}".split())
Might be a bit late, but maybe it helps somebody:
Since the latest PyCharm-versions it’s actually pretty straight forward, you can call Scrapy directly – see attached picture of runtime config (Scrapy tutorial).
Tested with PyCharm 2022.1.4.
I am working on Scrapy 0.20 with Python 2.7. I found PyCharm has a good Python debugger. I want to test my Scrapy spiders using it. Anyone knows how to do that please?
What I have tried
Actually I tried to run the spider as a script. As a result, I built that script. Then, I tried to add my Scrapy project to PyCharm as a model like this:
File->Setting->Project structure->Add content root.
But I don’t know what else I have to do
I am also using PyCharm, but I am not using its built-in debugging features.
For debugging I am using ipdb. I set up a keyboard shortcut to insert import ipdb; ipdb.set_trace() on any line I want the break point to happen.
Then I can type n to execute the next statement, s to step into a function, type any object name to see its value, alter execution environment, type c to continue execution…
This is very flexible, works in environments other than PyCharm, where you don’t control the execution environment.
Just type in your virtual environment pip install ipdb and place import ipdb; ipdb.set_trace() on a line where you want the execution to pause.
UPDATE
You can also pip install pdbpp and use the standard import pdb; pdb.set_trace instead of ipdb. PDB++ is nicer in my opinion.
The scrapy command is a python script which means you can start it from inside PyCharm.
When you examine the scrapy binary (which scrapy) you will notice that this is actually a python script:
#!/usr/bin/python
from scrapy.cmdline import execute
execute()
This means that a command like
scrapy crawl IcecatCrawler can also be executed like this: python /Library/Python/2.7/site-packages/scrapy/cmdline.py crawl IcecatCrawler
Try to find the scrapy.cmdline package.
In my case the location was here: /Library/Python/2.7/site-packages/scrapy/cmdline.py
Create a run/debug configuration inside PyCharm with that script as script. Fill the script parameters with the scrapy command and spider. In this case crawl IcecatCrawler.
Like this:
Put your breakpoints anywhere in your crawling code and it should workâ˘.
You just need to do this.
Create a Python file on crawler folder on your project. I used main.py.
- Project
- Crawler
- Crawler
- Spiders
- …
- main.py
- scrapy.cfg
- Crawler
- Crawler
Inside your main.py put this code below.
from scrapy import cmdline
cmdline.execute("scrapy crawl spider".split())
And you need to create a “Run Configuration” to run your main.py.
Doing this, if you put a breakpoint at your code it will stop there.
To add a bit to the accepted answer, after almost an hour I found I had to select the correct Run Configuration from the dropdown list (near the center of the icon toolbar), then click the Debug button in order to get it to work. Hope this helps!
I am running scrapy in a virtualenv with Python 3.5.0 and setting the “script” parameter to /path_to_project_env/env/bin/scrapy solved the issue for me.
intellij idea also work.
create main.py:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#coding=utf-8
import sys
from scrapy import cmdline
def main(name):
if name:
cmdline.execute(name.split())
if __name__ == '__main__':
print('[*] beginning main thread')
name = "scrapy crawl stack"
#name = "scrapy crawl spa"
main(name)
print('[*] main thread exited')
print('main stop====================================================')
show below:
According to the documentation https://doc.scrapy.org/en/latest/topics/practices.html
import scrapy
from scrapy.crawler import CrawlerProcess
class MySpider(scrapy.Spider):
# Your spider definition
...
process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})
process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished
As of 2018.1 this became a lot easier. You can now select Module name in your project’s Run/Debug Configuration. Set this to scrapy.cmdline and the Working directory to the root dir of the scrapy project (the one with settings.py in it).
Like so:
Now you can add breakpoints to debug your code.
I use this simple script:
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
process = CrawlerProcess(get_project_settings())
process.crawl('your_spider_name')
process.start()
Extending @Rodrigo’s version of the answer I added this script and now I can set spider name from configuration instead of changing in the string.
import sys
from scrapy import cmdline
cmdline.execute(f"scrapy crawl {sys.argv[1]}".split())
Might be a bit late, but maybe it helps somebody:
Since the latest PyCharm-versions it’s actually pretty straight forward, you can call Scrapy directly – see attached picture of runtime config (Scrapy tutorial).
Tested with PyCharm 2022.1.4.