How can I use different pipelines for different spiders in a single Scrapy project


I have a scrapy project which contains multiple spiders.
Is there any way I can define which pipelines to use for which spider? Not all the pipelines i have defined are applicable for every spider.


Asked By: CodeMonkeyB



I can think of at least four approaches:

  1. Use a different scrapy project per set of spiders+pipelines (might be appropriate if your spiders are different enough warrant being in different projects)
  2. On the scrapy tool command line, change the pipeline setting with scrapy settings in between each invocation of your spider
  3. Isolate your spiders into their own scrapy tool commands, and define the default_settings['ITEM_PIPELINES'] on your command class to the pipeline list you want for that command. See line 6 of this example.
  4. In the pipeline classes themselves, have process_item() check what spider it’s running against, and do nothing if it should be ignored for that spider. See the example using resources per spider to get you started. (This seems like an ugly solution because it tightly couples spiders and item pipelines. You probably shouldn’t use this one.)
Answered By: Francis Avila

Building on the solution from Pablo Hoffman, you can use the following decorator on the process_item method of a Pipeline object so that it checks the pipeline attribute of your spider for whether or not it should be executed. For example:

def check_spider_pipeline(process_item_method):

    def wrapper(self, item, spider):

        # message template for debugging
        msg = '%%s %s pipeline step' % (self.__class__.__name__,)

        # if class is in the spider's pipeline, then use the
        # process_item method normally.
        if self.__class__ in spider.pipeline:
            spider.log(msg % 'executing', level=log.DEBUG)
            return process_item_method(self, item, spider)

        # otherwise, just return the untouched item (skip this step in
        # the pipeline)
            spider.log(msg % 'skipping', level=log.DEBUG)
            return item

    return wrapper

For this decorator to work correctly, the spider must have a pipeline attribute with a container of the Pipeline objects that you want to use to process the item, for example:

class MySpider(BaseSpider):

    pipeline = set([

    def parse(self, response):
        # insert scrapy goodness here
        return item

And then in a file:

class Save(object):

    def process_item(self, item, spider):
        # do saving here
        return item

class Validate(object):

    def process_item(self, item, spider):
        # do validating here
        return item

All Pipeline objects should still be defined in ITEM_PIPELINES in settings (in the correct order — would be nice to change so that the order could be specified on the Spider, too).

Answered By: mstringer

You can use the name attribute of the spider in your pipeline

class CustomPipeline(object)

    def process_item(self, item, spider)
         if == 'spider1':
             # do something
             return item
         return item

Defining all pipelines this way can accomplish what you want.

Answered By: pad

The other solutions given here are good, but I think they could be slow, because we are not really not using the pipeline per spider, instead we are checking if a pipeline exists every time an item is returned (and in some cases this could reach millions).

A good way to completely disable (or enable) a feature per spider is using custom_setting and from_crawler for all extensions like this:

from scrapy.exceptions import NotConfigured

class SomePipeline(object):
    def __init__(self):

    def from_crawler(cls, crawler):
        if not crawler.settings.getbool('SOMEPIPELINE_ENABLED'):
            # if this isn't specified in settings, the pipeline will be completely disabled
            raise NotConfigured
        return cls()

    def process_item(self, item, spider):
        # change my item
        return item

   'myproject.pipelines.SomePipeline': 300,
SOMEPIPELINE_ENABLED = True # you could have the pipeline enabled by default

class Spider1(Spider):

    name = 'spider1'

    start_urls = [""]

    custom_settings = {

As you check, we have specified custom_settings that will override the things specified in, and we are disabling SOMEPIPELINE_ENABLED for this spider.

Now when you run this spider, check for something like:

[scrapy] INFO: Enabled item pipelines: []

Now scrapy has completely disabled the pipeline, not bothering of its existence for the whole run. Check that this also works for scrapy extensions and middlewares.

Answered By: eLRuLL

Just remove all pipelines from main settings and use this inside spider.

This will define the pipeline to user per spider

class testSpider(InitSpider):
    name = 'test'
    custom_settings = {
        'ITEM_PIPELINES': {
            'app.MyPipeline': 400
Answered By: Mirage

I am using two pipelines, one for image download (MyImagesPipeline) and second for save data in mongodb (MongoPipeline).

suppose we have many spiders(spider1,spider2,………..),in my example spider1 and spider5 can not use MyImagesPipeline

ITEM_PIPELINES = {'scrapycrawler.pipelines.MyImagesPipeline' : 1,'scrapycrawler.pipelines.MongoPipeline' : 2}
IMAGES_STORE = '/var/www/scrapycrawler/dowload'

And bellow complete code of pipeline

import scrapy
import string
import pymongo
from scrapy.pipelines.images import ImagesPipeline

class MyImagesPipeline(ImagesPipeline):
    def process_item(self, item, spider):
        if not in ['spider1', 'spider5']:
            return super(ImagesPipeline, self).process_item(item, spider)
           return item 

    def file_path(self, request, response=None, info=None):
        image_name = string.split(request.url, '/')[-1]
        dir1 = image_name[0]
        dir2 = image_name[1]
        return dir1 + '/' + dir2 + '/' +image_name

class MongoPipeline(object):

    collection_name = 'scrapy_items'

    def __init__(self, mongo_uri, mongo_db):
        self.mongo_uri = mongo_uri
        self.mongo_db = mongo_db

    def from_crawler(cls, crawler):
        return cls(
            mongo_db=crawler.settings.get('MONGO_DATABASE', 'scraping')

    def open_spider(self, spider):
        self.client = pymongo.MongoClient(self.mongo_uri)
        self.db = self.client[self.mongo_db]

    def close_spider(self, spider):

    def process_item(self, item, spider):
        collection_name=item.get( 'collection_name', self.collection_name )
        data = {}
        data['base_id'] = item['base_id']
            'base_id': item['base_id']
        }, {
            '$set': {
            'image_download': 1
        }, upsert=False, multi=True)
        return item
Answered By: Nanhe Kumar

we can use some conditions in pipeline as this

    # -*- coding: utf-8 -*-
from scrapy_app.items import x

class SaveItemPipeline(object):
    def process_item(self, item, spider):
        if isinstance(item, x,):
        return item
Answered By: Wade

You can just set the item pipelines settings inside of the spider like this:

class CustomSpider(Spider):
    name = 'custom_spider'
    custom_settings = {
        'ITEM_PIPELINES': {
            '__main__.PagePipeline': 400,
            '__main__.ProductPipeline': 300,

I can then split up a pipeline (or even use multiple pipelines) by adding a value to the loader/returned item that identifies which part of the spider sent items over. This way I won’t get any KeyError exceptions and I know which items should be available.

    def scrape_stuff(self, response):
        pageloader = PageLoader(
                PageItem(), response=response)

        pageloader.add_xpath('entire_page', '/html//text()')
        pageloader.add_value('item_type', 'page')
        yield pageloader.load_item()

        productloader = ProductLoader(
                ProductItem(), response=response)

        productloader.add_xpath('product_name', '//span[contains(text(), "Example")]')
        productloader.add_value('item_type', 'product')
        yield productloader.load_item()

class PagePipeline:
    def process_item(self, item, spider):
        if item['item_type'] == 'product':
            # do product stuff

        if item['item_type'] == 'page':
            # do page stuff
Answered By: Ryan Stefan

Simple but still useful solution.

Spider code

    def parse(self, response):
        item = {}
        ... do parse stuff
        item['info'] = {'spider': 'Spider2'}

pipeline code

    def process_item(self, item, spider):
        if item['info']['spider'] == 'Spider1':
            logging.error('Spider1 pipeline works')
        elif item['info']['spider'] == 'Spider2':
            logging.error('Spider2 pipeline works')
        elif item['info']['spider'] == 'Spider3':
            logging.error('Spider3 pipeline works')

Hope this save some time for somebody!

Answered By: NashGC

The most simple and effective solution is to set custom settings in each spider itself.

custom_settings = {'ITEM_PIPELINES': {'project_name.pipelines.SecondPipeline': 300}}

After that you need to set them in the file

   'project_name.pipelines.FistPipeline': 300,
   'project_name.pipelines.SecondPipeline': 400

in that way each spider will use the respective pipeline.

Answered By: default_settings

Overriding ‘ITEM_PIPELINES’ with custom settings per spider, as others have suggested, works well. However, I found I had a few distinct groups of pipelines I wanted to use for different categories of spiders. I wanted to be able to easily define the pipeline for a particular category of spider without a lot of thought, and I wanted to be able to update a pipeline category without editing each spider in that category individually.

So I created a new file called in the same directory as contains functions like this:

def episode_pipelines():
    return {
        'radio_scrape.pipelines.SaveEpisode': 100,

def show_pipelines():
    return {
        'radio_scrape.pipelines.SaveShow': 100,

Then in each spider I would import the specific function relevant for the spider:

from radio_scrape.pipeline_definitions import episode_pipelines

I then use that function in the custom settings assignment:

class RadioStationAEspisodesSpider(scrapy.Spider):
    name = 'radio_station_A_episodes'        
    custom_settings = {
        'ITEM_PIPELINES': episode_pipelines()
Answered By: shawn caza
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.