How to replace 'FEEDS' in scrapy version 2.X with 'FEED_URI & 'FEED_FORMAT'? GUI app using Python Tkinter

Question

I am trying to learn to make a Scrapy Desktop app using Python Tk inter. So, I’m trying to run this prebuild Scrapy desktop app using Python Tkinter. While saving the output, there is an issue with the "FEEDS". The below-mentioned function does save the output file by choosing the specific extension and saving it to a specific location. This script was developed based Scrapy version up to 1.8. So in Scrapy version 2.X the ‘FEED_URI & ‘FEED_FORMAT’ are not working as they have been replaced with "FEEDS".

def execute_spider():
    if dataset_entry.get() == '' or chosen_feed not in ['CSV', 'JSON']:
        messagebox.showerror('Error', 'All entries are required')
        return
    try:
        feed_uri = f'file:///{folder_path}/{dataset_entry.get()}.{chosen_feed}'
    except:
        messagebox.showerror('Error', 'All entries are required')

    settings = project.get_project_settings()
    settings. Set('FEED_URI', feed_uri)
    settings. Set('FEED_FORMAT', chosen_feed)

    configure_logging()
    runner = CrawlerRunner(settings)
    runner.crawl(chosen_spider)

    reactor.run(installSignalHandlers=False)

To see the full script, you can also visit here

So, how can I use ‘FEEDS’ instead of these particular 2 lines?

settings. Set('FEED_URI', feed_uri)
settings. Set('FEED_FORMAT', chosen_feed)

Asked By: Raisul Islam

||

Source

Answer 1

You would instead use


settings.set("FEEDS", {
   feed_uri: {
       'format': chosen_format,
       'encoding': 'utf8'
 }
})

and include any other settings that are appropriate.

Refer to the scrapy docs for available options: FEEDS

Answered By: Alexander

How to replace 'FEEDS' in scrapy version 2.X with 'FEED_URI & 'FEED_FORMAT'? GUI app using Python Tkinter

Question:

Answers: